Developer & Vendor Tools for Data Integration

Imagine a scenario where you launch a new project management tool, and it’s really the best one out there, but no one wants to use it. This could happen for several reasons, one of which would be the lack of integrations with other types of software your potential users are using. They might need integration with Google Calendar, or with a variety of CRMs, or ticketing systems, or bug tracking systems, or version control services, and so on and so forth. Eventually, you’re forced to start providing built-in integrations with any and every other tool out there. With each new integration offered, the potential user base of your app grows bigger, which is good, but this scenario has it’s disadvantages:

  • There are infinite potential integrations, so the work is unending.
  • This diverts the attention of your development staff from what they do well (project management, in the case described above) to other types of work.
  • Bugs are usually hard to control because you’re working against systems that you don’t host, and whose code you can’t see.

The need for integration tools aimed at software vendors derives from the scenario described just now, which I like to call the integration pitfall. Application providers are faced with pressure, having to offer built-in integrations with a growing subset of related products, so that their own product would be useful to new target audiences. Some tools were developed with the aim of assisting these software providers, instead of targeting their end users directly.

Elastic.io is one such tool. It lets you connect with a variety of third party services and map fields from these data sources to your own application with a simple drag and drop interface. In sum, Elastic.io is meant for non-technical users that want to extend the application that they offer with built-in integrations. A very different approach is that of Nectil, a French service provider that developed their own software for web application integrations, and that offer small businesses tailored solutions on top of that framework.

Most other tools aim at the developer crowd, trying to help them consume APIs with greater ease and speed. Temboo provides SDKs for server side API consumption. It supports a very extensive amount of APIs and provides access to them through a unified interface. The service is SaaS based, though cheap, and meant for low levels of consumption (anything really big would require custom pricing). Webshell is an API that lets you integrate different APIs from third party services and create new functionality. It’s a JS library, which handles authentication for you and lets you construct your custom APIs using an editor. DERI Pipes is a software that lets you create automated processes that transform and mash together web content (much like Yahoo! Pipes, but more technically complex).

Cumula is a PHP application framework that takes the idea even further. Based on the assumption that modern web applications no longer sit on top of a single on-premise database, but instead connect with a multiplicity of services, they’ve contrived a framework that lets developers build applications with a unified interface for local and remote data sources. It’s modular, with the use of components for describing parts of the application. Each component has its own routing and templating. A collection of DataStore and DataService packages are available as dependencies, included in the same Github repo. As of now, Github’s popularity metrics indicate very low acceptance rates, and the same goes for the associated Google Group. Nevertheless, this is a relatively new project, and they’ve already received some media coverage.

Other platforms aim to help developers find and consume APIs, without altering the way they work or requiring the utilization of a specific framework. APIHub (renamed to the anypoint portal after being acquired by Mulesoft) and the ProgrammableWeb projects are API explorers, within which one can quickly search and learn about new APIs. Mashape does the same, although focused more on developer APIs and less on connecting with third party applications. Mashape acts as a middleman between providers and consumers, relieving API consumers from having to maintain separate authentication processes with a multiplicity of remote services, and thus enabling the proliferation of usage of such services.

Edit: Since the time of writing this, a new contestant has emerged, called CloudElements, that provides a uniform API for accessing multiple cloud based applications.


*This article is part of a business plan I decided to edit into a series of blog posts. You can find the rest of the content here