Overview of Data Integration Tools

Integration for SMBs

OneSaas is a data integration application that automates common tasks for specific business users, mostly in eCommerce and online marketing. It features a variety of integrated services and a simple interface to connect to these third party user accounts. Then, there are a few options for each type of data, mainly, whether or not to add and delete items according to changes in other systems that handle the same data type – Magento with eBay, Google Contacts with a CRM, etc. It also has an interface that displays all the data from the third party vendors in tables.

Cazoomi SyncApps is an integration cloud meant for small businesses. It features a visual studio for composing integrations, much like Enterprise solutions do, but are aimed at SMBs, both with regard to pricing and available integrations. Cazoomi’s website reports them having 10,000 customers, and a team of 13 members (at the time of writing). It appears that Cazoomi’s solution is meant more towards application vendors that want to create integrations than it is for end users themselves. Another type of tools are web page scrapers meant to ease extraction of data from websites without the use of APIs that sometimes aren’t available. The prominent tool in this respect is Dapper, and a newer contestant is import.io.

UPDATE: Since this text was written, a new service came out that scrapes websites automatically, and let’s you build apps out of the data it extracts, called Kimono

Industry Specific Integration Tools

Each industry has an array of specific tools, either meant for integrating data, or that integrate data as a byproduct of their main function (dashboards, process automation, etc.). These solutions are thus somewhat parallel in their function to data integration software, and could suffice in any scenario in which integration needs are within this specific set of third party applications. However, a common scenario among small businesses is an overdependence on one such specific tool, and an inability to make that tool work with services meant for other aspects of running a business. For instance, a business in eCommerce might utilize an application that optimizes and automates some of the aspects of selling online. Such a tool would surely connect seamlessly with any and every marketplace, comparison shopping engine, shopping cart software, payment gateway and shipping service. However, the ability to make this tool work with a newsletter management system, or a PPC ad platform, is of limited capacity.

Examples of such niche products vary widely from one industry to another. Ad management tools include Datorama, Kenshoo and IgnitionOne. In eCommerce, common tools include Channeladvisor, eSellerPro, Vendio and Auctiva. mHealth apps can be connected with Validic. Data from different analytics solutions can be pulled together with Segment.io. Social media campaigns can be managed from a single place with Hootsuite.

Enterprise Integration Tools

The Enterprise world covers a very wide array of solutions at this point in time. As part of this overview, only SaaS products meant at Enterprise customers will be examined and not hybrid and on-premise solutions. A key character of the following solutions is that they’re meant for IT departments to deploy, and not for the end users themselves.

Informatica Cloud is a high end service for Enterprise clients that offers a large set of tools for integrations and data processing. Jitterbit offers an open source platform for the creation and management of integrations, both legacy and SOA based. They make use of reusable templates for common integration needs so that developers have less work. Extol offers a design tool for building integrations between Enterprise applications. JackBe Presto is a visualization and BI tool that connects to external sources – data warehouses, spreadsheets, feeds, existing BI systems, etc., using a graphical assembly tool. SnapLogic is an Enterprise-grade solution for data integration that makes use of modular pieces of connectivity or functionality called ‘snaps’. Snaps can connect to third party services, local files or anything else, and can provide atomic bits of functionality used for ETL tasks. They have a marketplace for snaps with high end pricing. Mulesoft offers a set of products meant for data integration. Their visual editing tools are meant to ease the work of developers, not to provide a EUP solution. It offers a variety of pre-built connectors and integration apps, the ability to develop new ones, and visual tools for integrating services and for building data transformation processes. Dell Boomi is an Enterprise API integration tool that uses a flowchart interface and offers both pre-built integration types as well as the ability to create new ones or have Dell do the customization for the end users. Scribesoft offers the same standard set of features, as well as the ability to connect SaaS products with on premise software. Other solutions in this sphere include IBM Infosphere DataStage, IBM CastIron, and Talend’s Integration Suite.

How to Serve Cloudinary with HTTPS on Django

Cloudinary’s documentation states that their client side libraries can automatically detect the protocol used in a page and generate HTTP/S urls for the image tags accordingly. This, of course, doesn’t apply to server side code, for which there’s no apt documentation on this matter.

In Cloudinary’s Django library, there are three ways in which you can define your site specific parameters:

  • A dictionary called CLOUDINARY that you can set in your settings.py.
  • The CLOUDINARY_CLOUD_NAME environment variable.
  • The CLOUDINARY_URL environment variable.

Although not mentioned in the documentation, the CLOUDINARY_URL environment variable can take query paramters, and these would be added to the cloudinary object that generates the urls within the Django templates. So to make sure that all requests to Cloudinary are made with HTTPS, all you have to do is set the CLOUDINARY_URL variable with the query parameter ‘secure’ and the value ‘True’, like so:

export CLOUDINARY_URL=cloudinary://123456789012345:abcdefghijklmnop-qrstuvwxyz@abcdefgh?secure=True

Just make sure to replace the fake API key, API secret and cloud name I typed above with your own.

This approach is not so great, though. It forces us to make sure we include the extra query param if we’ll ever need to change the Cloudinary URL in the future, and because it’s not in the code, we can only have all our pages transferred securely or none at all.

An alternative would be to use the CLOUDINARY dict that we can set under settings.py, but that method has a disadvantage as well – it doesn’t play nicely with the other environment variables. In fact, Cloudinary’s Django library regards the three approaches specified above for entering site preferences as exclusionary, so if we set the CLOUDINARY dict under settings.py, the CLOUDINARY_URL will not even be looked at. Considering the fact that under platforms such as Heroku, an environment variable is the preferable route, we’re basically left with nothing but bad choices.

Luckily, the cloudinary object used to generate image links is just python code, the intantiation method of which takes keyword arguments that would be added to the same dict used for storing the site specific values from the inputs mentioned above. So to set Cloudinary to use secure URLs, all that’s required are the following two lines in your settings.py:

import cloudinary

That’s it! Your Cloudinary URLs will now always be loaded with HTTPS. Because this is just python code, you can alter it to use secure or unsecure URLs according to different views (just make sure to do that inside view functions and not in settings.py).

Overview of Online Databases & Business App Solutions

These platforms are meant to allow non-technical end users to define data structures, according to their specific needs, and then make use of an online database that stores data according to those definitions. They usually also allow one to build business apps, which are small productivity apps for web or mobile, meant to ease tasks such as data entry and reporting.

The challenge they all face is the technical complexities involved with data modeling, which in this case is left to non-technical end users. The traditional way of solving this is the route taken by MS Access, that is, leaving it to the end user to plan and manually build table schemas and relations. Once the user completes this process, the system can automatically create forms for data entry, based on the schemas he supplied. A more modern approach to solve the same problem goes the exact opposite path, by allowing the user to build the form itself, with a drag and drop interface, then automatically generating a matching schema behind the scenes.

In general, there was a surge of app authoring tools in ’06-’07, and some of those attempts to empower non-programmers to build tools for and by themselves were halted by the ‘08 crisis. Under these circumstances, DabbleDB was acquired by Twitter in ‘10 and shut down in ‘11, Lazybase disappeared in thin air, Teqlo got shut down, and Coghead became part of SAP’s River platform.

The main tool used today for authoring small business apps is Caspio Bridge. It allows one to create databases, web forms, and apps, all without writing any code. With regard to the UI, it works similarly to MS Excel and Access, but is online, with integration support through a SOAP API, POST requests, Datahub, a JS library and a plugin for MS office. Another common solution is Intuit’s QuickBase, which allows the same standard app creation, as well as managing data through a spreadsheet UI. They differentiate by offering a large set of database templates for common industry-specific needs. Both these tools have pricing levels meant for SMBs (but not micro businesses), meaning their plans start at $250-300/mo. Zoho Creator is a third alternative with lower pricing (for small usage, whereas a high usage would cost a lot more than what’s offered by other services). It’s an online database and business apps service, with a form creation wizard, business rules engine, customizable reports, multiple view types (including an editable grid), alerting, rebranding and styles, collaboration, backups, APIs and data exports.

Other contestants in this field would be TrackVia, LongJump, FileMaker Pro and Viravis. They all offer database and app creators meant for non-programmers, as well as custom reporting and database templates. LongJump provides an SDK meant for adding functionality to the system by code. FileMaker Pro offers the ability to build apps for tablets and mobile. Eccentex is another solution, that provides a platform on top of which developers can build productivity apps, and end users can make use of them.

A newer generation of tools is emerging for the past two years or so. Apart from Dreamface, which is aimed at IBM BPM software customers, all of these solutions are meant for micro to small businesses as well as the early adopter consumer market, and are priced accordingly. Ragic! Builder is an app authoring tool that offers a spreadsheet UI meant to allow the end user to define the applications’ form fields. It then offers reporting, full text search, a query builder, embedding results onto one’s website, importing and exporting, access control, versioning, customized scripting and an API. Knack is another solution with really easy data management, user management, and that’s customizable with an open API as well as with CSS and JS editors. SodaDB is a donationware product that offers a simple and customizable database, importing and exporting, a form builder, full text search, and the ability to work without signing in.

Another brand of solutions is that of online spreadsheets, which sit at a crossroad between accounting software and online databases. These are meant to compete with MS Excel on specific vectors, and offer features that make them more attractive for these specific target audiences. Smartsheet is a paid-only online spreadsheet, with a modern UI, mobile versions, and an emphasis on collaboration. Zoho Sheet is a MS Excel clone that mostly fits existing customers of Zoho. Glide Crunch is a desktop spreadsheet application meant for large spreadsheets (the kind that won’t fit into an online tool), and that syncs automatically with your storage on Glide’s cloud office suite. AirXcell is a web based spreadsheet tool meant for scientific and statistical heavy-load work. It uses R language syntax for formulas and functions, and comes built in with a few financial applications. Flextory is a web application that acts as a sort of administration panel for your own data. It’s looks and acts like a standard web admin panel, with item editing forms, filtering, sorting and choosing table fields to view, but the data is really the kind of data you’d use MS Excel to manage. SecureSheet is an online spreadsheet that offers collaboration and security. BinaryThumb is a spreadsheet for iPhones and iPads that contains not only numbers but any kind of media (slightly pivoted since this text was written). Sumwise is a spreadsheet solution that offers smart features like grouping, reusable cells, etc. CollateBox offers collaboration on lists based on MS Excel data. Other solutions would include Editgrid and Skysheet.

UPDATE: Since the time of writing this text in 2012, a new online database solution emerged in the scene, called Team Desk.