Imagine a scenario where you launch a new project management tool, and it’s really the best one out there, but no one wants to use it. This could happen for several reasons, one of which would be the lack of integrations with other types of software your potential users are using. They might need integration with Google Calendar, or with a variety of CRMs, or ticketing systems, or bug tracking systems, or version control services, and so on and so forth. Eventually, you’re forced to start providing built-in integrations with any and every other tool out there. With each new integration offered, the potential user base of your app grows bigger, which is good, but this scenario has it’s disadvantages:
- There are infinite potential integrations, so the work is unending.
- This diverts the attention of your development staff from what they do well (project management, in the case described above) to other types of work.
- Bugs are usually hard to control because you’re working against systems that you don’t host, and whose code you can’t see.
The need for integration tools aimed at software vendors derives from the scenario described just now, which I like to call the integration pitfall. Application providers are faced with pressure, having to offer built-in integrations with a growing subset of related products, so that their own product would be useful to new target audiences. Some tools were developed with the aim of assisting these software providers, instead of targeting their end users directly.
Elastic.io is one such tool. It lets you connect with a variety of third party services and map fields from these data sources to your own application with a simple drag and drop interface. In sum, Elastic.io is meant for non-technical users that want to extend the application that they offer with built-in integrations. A very different approach is that of Nectil, a French service provider that developed their own software for web application integrations, and that offer small businesses tailored solutions on top of that framework.
Most other tools aim at the developer crowd, trying to help them consume APIs with greater ease and speed. Temboo provides SDKs for server side API consumption. It supports a very extensive amount of APIs and provides access to them through a unified interface. The service is SaaS based, though cheap, and meant for low levels of consumption (anything really big would require custom pricing). Webshell is an API that lets you integrate different APIs from third party services and create new functionality. It’s a JS library, which handles authentication for you and lets you construct your custom APIs using an editor. DERI Pipes is a software that lets you create automated processes that transform and mash together web content (much like Yahoo! Pipes, but more technically complex).
Cumula is a PHP application framework that takes the idea even further. Based on the assumption that modern web applications no longer sit on top of a single on-premise database, but instead connect with a multiplicity of services, they’ve contrived a framework that lets developers build applications with a unified interface for local and remote data sources. It’s modular, with the use of components for describing parts of the application. Each component has its own routing and templating. A collection of DataStore and DataService packages are available as dependencies, included in the same Github repo. As of now, Github’s popularity metrics indicate very low acceptance rates, and the same goes for the associated Google Group. Nevertheless, this is a relatively new project, and they’ve already received some media coverage.
Other platforms aim to help developers find and consume APIs, without altering the way they work or requiring the utilization of a specific framework. APIHub (renamed to the anypoint portal after being acquired by Mulesoft) and the ProgrammableWeb projects are API explorers, within which one can quickly search and learn about new APIs. Mashape does the same, although focused more on developer APIs and less on connecting with third party applications. Mashape acts as a middleman between providers and consumers, relieving API consumers from having to maintain separate authentication processes with a multiplicity of remote services, and thus enabling the proliferation of usage of such services.
Edit: Since the time of writing this, a new contestant has emerged, called CloudElements, that provides a uniform API for accessing multiple cloud based applications.
*This article is part of a business plan I decided to edit into a series of blog posts. You can find the rest of the content here
For most of us, this is already a given truth, and looks like a rather obvious progression from previous technological advancements. However, at the same time, we also take for granted that the web applications we use for our day-to-day trade are contained silos, with their own special UIs and terminologies. Each of these contained silos holds a segment of our data, and forces us to sign in so we can view it, and to adapt to their idea of how to present our data to us.
Gradually, businesses are adopting data aggregators that simplify specific segments of their day to day work. For instance, a business that relies on PPC-based marketing, might start using a PPC management tool, from which to manage their entire ad spend from a single user interface. Now, they no longer have to sign in to multiple services, and cope with the varied complexities of each system.
With the adoption of such tools, comes a new expectation – that other aspects of their business be managed with the same simplicity. Each aspect of a business’s operation deserves a specific tool that would abstract away the complexities of the underlying systems. But then arises a new multiplicity of tools, which suffer from the same sorts of problems – multiple sign-ins, multiple UIs, and data that’s contained within specific apps, not usable anywhere else.
I propose a bigger revolution – one of a super-distribution of data at large. The principle logic behind this change is the recognition that building a truly one-stop-shop for any and every need of a business is unachievable. Instead, one should focus on enabling the distributed model, in which multiple complementing services are used in parallel. With this recognition in mind, I wouldn’t suggest to anyone to try to replace the PPC management tool mentioned above, but to complement it by enabling it to communicate with other tools meant for other specific aspects of a business’s operation.
* This article is part of a business plan I decided to edit into a series of blog posts. You can find the rest of the content here
The need to combine different data sources that exist only within cloud applications forces many small business owners to turn to working with spreadsheets. These, have several advantages that complement the standard ‘menus and tables’ UI common in most web applications:
- The ability to view all the information from one screen.
- The ability to change values on a whole column in bulk, instead of having to go through each item separately.
- A UI that non-technical people are familiar with and understand.
- The ability to merge data from different sources onto a single spreadsheet.
In spite of the apparent comforts of managing a business with spreadsheets, this has the disadvantage of being a completely manual process, which has to be repeated at intervals, including a preparatory stage of merging separate spreadsheets into one. The outcome of this work is what I like to call ‘Excel Spaghetti’, as more and more files are accumulated on the desktops of the employees responsible for the manual labor, slowly contributing to an unmanageable work process. The apparent disadvantages of such a manual process would be:
- The lack of version control mechanisms, and a lack of a single source of truth.
- Work is saved locally and transferred by means such as email, instead of being concentrated in a single location.
- Data wrangling, as well as re-formatting the different sources of data, is needed each time an update is required, usually over a dataset that’s mostly the same as before.
- Data is never really up-to-date by the time it’s downloaded and re-formatted for presentation or analysis.
*This article is part of a business plan I decided to edit into a series of blog posts. You can find the rest of the content here | MS Excel is the go-to tool for most data-related day to day tasks in businesses, small and large alike. For any task that repeats itself long enough, this leads to unmanageable work processes.
Process automation tools, meant for either small businesses and consumers, or for Enterprises, allow non-programmers to combine services from different vendors into a new piece of functionality. Enterprise and consumer tools differ from one another in the level of complexity and technical know-how that they require. Most tools meant for consumers can be configured with no more than a few simple steps. They allow people that don’t write code to define automated behaviors between cloud-based services, such as sending yourself an email when this or that happens, adding a new contact to your mailing list or CRM when a form is submitted somewhere, or sending new content directly to your Dropbox folder. These can be regarded as productivity micro-tools, and are very limited in the scope of possibilities that they offer.
Zapier (YCombinator alumni) is a prominent tool, meant for very small businesses and consumers, which offers an extensive list of third party services to integrate with. IFTTT (a Betaworks profile company) is another prominent contestant, meant mostly for consumers and free of charge. Cloudwork (sister company of GetApp) covers the same space as well, and are working on a set of SDKs, and a visual editor, to allow anyone to create new integration types. Yahoo! Pipes is a tool to aggregate, manipulate and mash up feeds from third party services, that offers lots of useful functions for processing the data and is all assembled visually. Other alternatives in this sphere include RunMyProcess, Foxweave & WeWiredWeb.
BPM software for Enterprise customers are the other end of the spectrum. They normally provide form designers, visual process editors, rule engines and customizable dashboards for monitoring workflows. These solutions are relatively complex to operate, and usually require a learning period to adapt to. Their pricing levels are clearly meant for the Enterprise world. Competitors in this sphere include Appian, Skelta, Rollbase, Iceberg and ProcessMaker, to name a few.
*This article is part of a business plan I decided to edit into a series of blog posts. You can find the rest of the content here
The tools surveyed here offer an integration of data into a single platform, but do so only in one direction. By this, I mean that these solutions don’t offer the possibility of sending updates back to the applications from which the data originally came from. One such tool is Sumall, which offers a marketing-focused set of metrics based on third party data, consumed via APIs. A competitor in the same vector is Geckoboard, offering an API itself for end users to add their own services on top of those provided out of the box.
A slightly more configurable tool is a
Every BI software provider offers some level of dashboard building capabilities. Notwithstanding, these solutions are usually out of reach for small businesses. Jaspersoft, for instance, offers a community edition of its solution as an open source project. However, that open source version lacks the dashboard capabilities that paying users are offered.
A subset of dashboard providers is that of analytics solutions. These are less likely to integrate data from different sources and are therefore of relatively minor significance for this overview. In spite of that, some tools are still worth noting. The first of which is Chartbeat, which gathers data from the users’ websites, but integrates with social media data as well, to generate a more insightful perspective, all without requiring the end users themselves to do the heavy lifting data work. Another tool takes that approach slightly further – ThinkUp pulls together data from users’ social networks, but doesn’t offer graphs and dashboards at all. Instead of pouring immense amounts of data onto the screen, ThinkUp presents a feed, much like that of most social networks, that highlights actionable insights based on the sum of the data that was gathered.
*This article is part of a business plan I decided to edit into a series of blog posts. You can find the rest of the content here
Integration for SMBs
OneSaas is a data integration application that automates common tasks for specific business users, mostly in eCommerce and online marketing. It features a variety of integrated services and a simple interface to connect to these third party user accounts. Then, there are a few options for each type of data, mainly, whether or not to add and delete items according to changes in other systems that handle the same data type – Magento with eBay, Google Contacts with a CRM, etc. It also has an interface that displays all the data from the third party vendors in tables.
Cazoomi SyncApps is an integration cloud meant for small businesses. It features a visual studio for composing integrations, much like Enterprise solutions do, but are aimed at SMBs, both with regard to pricing and available integrations. Cazoomi’s website reports them having 10,000 customers, and a team of 13 members (at the time of writing). It appears that Cazoomi’s solution is meant more towards application vendors that want to create integrations than it is for end users themselves. Another type of tools are web page scrapers meant to ease extraction of data from websites without the use of APIs that sometimes aren’t available. The prominent tool in this respect is Dapper, and a newer contestant is import.io.
Since this text was written, a new service came out that scrapes websites automatically, and let’s you build apps out of the data it extracts, called Kimono
Industry Specific Integration Tools
Each industry has an array of specific tools, either meant for integrating data, or that integrate data as a byproduct of their main function (dashboards, process automation, etc.). These solutions are thus somewhat parallel in their function to data integration software, and could suffice in any scenario in which integration needs are within this specific set of third party applications. However, a common scenario among small businesses is an overdependence on one such specific tool, and an inability to make that tool work with services meant for other aspects of running a business. For instance, a business in eCommerce might utilize an application that optimizes and automates some of the aspects of selling online. Such a tool would surely connect seamlessly with any and every marketplace, comparison shopping engine, shopping cart software, payment gateway and shipping service. However, the ability to make this tool work with a newsletter management system, or a PPC ad platform, is of limited capacity.
Examples of such niche products vary widely from one industry to another. Ad management tools include Datorama, Kenshoo and IgnitionOne. In eCommerce, common tools include Channeladvisor, eSellerPro, Vendio and Auctiva. mHealth apps can be connected with Validic. Data from different analytics solutions can be pulled together with Segment.io. Social media campaigns can be managed from a single place with Hootsuite.
Enterprise Integration Tools
The Enterprise world covers a very wide array of solutions at this point in time. As part of this overview, only SaaS products meant at Enterprise customers will be examined and not hybrid and on-premise solutions. A key character of the following solutions is that they’re meant for IT departments to deploy, and not for the end users themselves.
Informatica Cloud is a high end service for Enterprise clients that offers a large set of tools for integrations and data processing. Jitterbit offers an open source platform for the creation and management of integrations, both legacy and SOA based. They make use of reusable templates for common integration needs so that developers have less work. Extol offers a design tool for building integrations between Enterprise applications. JackBe Presto is a visualization and BI tool that connects to external sources – data warehouses, spreadsheets, feeds, existing BI systems, etc., using a graphical assembly tool. SnapLogic is an Enterprise-grade solution for data integration that makes use of modular pieces of connectivity or functionality called ‘snaps’. Snaps can connect to third party services, local files or anything else, and can provide atomic bits of functionality used for ETL tasks. They have a marketplace for snaps with high end pricing. Mulesoft offers a set of products meant for data integration. Their visual editing tools are meant to ease the work of developers, not to provide a EUP solution. It offers a variety of pre-built connectors and integration apps, the ability to develop new ones, and visual tools for integrating services and for building data transformation processes. Dell Boomi is an Enterprise API integration tool that uses a flowchart interface and offers both pre-built integration types as well as the ability to create new ones or have Dell do the customization for the end users. Scribesoft offers the same standard set of features, as well as the ability to connect SaaS products with on premise software. Other solutions in this sphere include IBM Infosphere DataStage, IBM CastIron, and Talend’s Integration Suite.
*This article is part of a business plan I decided to edit into a series of blog posts. You can find the rest of the content here | This is a sumarry of tools for both SMBs and enterprises meant for integrating data from business related applications.
Cloudinary’s documentation states that their client side libraries can automatically detect the protocol used in a page and generate HTTP/S urls for the image tags accordingly. This, of course, doesn’t apply to server side code, for which there’s no apt documentation on this matter.
In Cloudinary’s Django library, there are three ways in which you can define your site specific parameters:
- A dictionary called CLOUDINARY that you can set in your settings.py.
- The CLOUDINARY_CLOUD_NAME environment variable.
- The CLOUDINARY_URL environment variable.
Although not mentioned in the documentation, the CLOUDINARY_URL environment variable can take query paramters, and these would be added to the cloudinary object that generates the urls within the Django templates. So to make sure that all requests to Cloudinary are made with HTTPS, all you have to do is set the CLOUDINARY_URL variable with the query parameter ‘secure’ and the value ‘True’, like so:
Just make sure to replace the fake API key, API secret and cloud name I typed above with your own.
This approach is not so great, though. It forces us to make sure we include the extra query param if we’ll ever need to change the Cloudinary URL in the future, and because it’s not in the code, we can only have all our pages transferred securely or none at all.
An alternative would be to use the CLOUDINARY dict that we can set under settings.py, but that method has a disadvantage as well – it doesn’t play nicely with the other environment variables. In fact, Cloudinary’s Django library regards the three approaches specified above for entering site preferences as exclusionary, so if we set the CLOUDINARY dict under settings.py, the CLOUDINARY_URL will not even be looked at. Considering the fact that under platforms such as Heroku, an environment variable is the preferable route, we’re basically left with nothing but bad choices.
Luckily, the cloudinary object used to generate image links is just python code, the intantiation method of which takes keyword arguments that would be added to the same dict used for storing the site specific values from the inputs mentioned above. So to set Cloudinary to use secure URLs, all that’s required are the following two lines in your settings.py:
import cloudinary cloudinary.config(secure=True)
That’s it! Your Cloudinary URLs will now always be loaded with HTTPS. Because this is just python code, you can alter it to use secure or unsecure URLs according to different views (just make sure to do that inside view functions and not in settings.py).| The missing configuration statement that’s required to get Cloudinary’s Python library to work on HTTPS-enabled websites.
These platforms are meant to allow non-technical end users to define data structures, according to their specific needs, and then make use of an online database that stores data according to those definitions. They usually also allow one to build business apps, which are small productivity apps for web or mobile, meant to ease tasks such as data entry and reporting.
The challenge they all face is the technical complexities involved with data modeling, which in this case is left to non-technical end users. The traditional way of solving this is the route taken by MS Access, that is, leaving it to the end user to plan and manually build table schemas and relations. Once the user completes this process, the system can automatically create forms for data entry, based on the schemas he supplied. A more modern approach to solve the same problem goes the exact opposite path, by allowing the user to build the form itself, with a drag and drop interface, then automatically generating a matching schema behind the scenes.
In general, there was a surge of app authoring tools in ’06-’07, and some of those attempts to empower non-programmers to build tools for and by themselves were halted by the ‘08 crisis. Under these circumstances, DabbleDB was acquired by Twitter in ‘10 and shut down in ‘11, Lazybase disappeared in thin air, Teqlo got shut down, and Coghead became part of SAP’s River platform.
The main tool used today for authoring small business apps is Caspio Bridge. It allows one to create databases, web forms, and apps, all without writing any code. With regard to the UI, it works similarly to MS Excel and Access, but is online, with integration support through a SOAP API, POST requests, Datahub, a JS library and a plugin for MS office. Another common solution is Intuit’s QuickBase, which allows the same standard app creation, as well as managing data through a spreadsheet UI. They differentiate by offering a large set of database templates for common industry-specific needs. Both these tools have pricing levels meant for SMBs (but not micro businesses), meaning their plans start at $250-300/mo. Zoho Creator is a third alternative with lower pricing (for small usage, whereas a high usage would cost a lot more than what’s offered by other services). It’s an online database and business apps service, with a form creation wizard, business rules engine, customizable reports, multiple view types (including an editable grid), alerting, rebranding and styles, collaboration, backups, APIs and data exports.
Other contestants in this field would be TrackVia, LongJump, FileMaker Pro and Viravis. They all offer database and app creators meant for non-programmers, as well as custom reporting and database templates. LongJump provides an SDK meant for adding functionality to the system by code. FileMaker Pro offers the ability to build apps for tablets and mobile. Eccentex is another solution, that provides a platform on top of which developers can build productivity apps, and end users can make use of them.
A newer generation of tools is emerging for the past two years or so. Apart from Dreamface, which is aimed at IBM BPM software customers, all of these solutions are meant for micro to small businesses as well as the early adopter consumer market, and are priced accordingly. Ragic! Builder is an app authoring tool that offers a spreadsheet UI meant to allow the end user to define the applications’ form fields. It then offers reporting, full text search, a query builder, embedding results onto one’s website, importing and exporting, access control, versioning, customized scripting and an API. Knack is another solution with really easy data management, user management, and that’s customizable with an open API as well as with CSS and JS editors. SodaDB is a donationware product that offers a simple and customizable database, importing and exporting, a form builder, full text search, and the ability to work without signing in.
Another brand of solutions is that of online spreadsheets, which sit at a crossroad between accounting software and online databases. These are meant to compete with MS Excel on specific vectors, and offer features that make them more attractive for these specific target audiences. Smartsheet is a paid-only online spreadsheet, with a modern UI, mobile versions, and an emphasis on collaboration. Zoho Sheet is a MS Excel clone that mostly fits existing customers of Zoho. Glide Crunch is a desktop spreadsheet application meant for large spreadsheets (the kind that won’t fit into an online tool), and that syncs automatically with your storage on Glide’s cloud office suite. AirXcell is a web based spreadsheet tool meant for scientific and statistical heavy-load work. It uses R language syntax for formulas and functions, and comes built in with a few financial applications. Flextory is a web application that acts as a sort of administration panel for your own data. It’s looks and acts like a standard web admin panel, with item editing forms, filtering, sorting and choosing table fields to view, but the data is really the kind of data you’d use MS Excel to manage. SecureSheet is an online spreadsheet that offers collaboration and security. BinaryThumb is a spreadsheet for iPhones and iPads that contains not only numbers but any kind of media (slightly pivoted since this text was written). Sumwise is a spreadsheet solution that offers smart features like grouping, reusable cells, etc. CollateBox offers collaboration on lists based on MS Excel data. Other solutions would include Editgrid and Skysheet.
Since the time of writing this text in 2012, a new online database solution emerged in the scene, called Team Desk.
*This article is part of a business plan I decided to edit into a series of blog posts. You can find the rest of the content here | Solutions that allow non-programmers to build simple form-based apps have a long history. There’s always the question of the MS Access vs. Google Forms approach – whether to let the user build a database and generate the form automatically, or to let him build the form, and generate the database automatically. While most solutions in this sphere that cater to early adopters, tech folk and micro businesses never seem to really catch on, there are solid solutions out there meant for SMBs and large businesses.
Following my decision to try and refrain from from startups for the time being, I’ve decided to publish parts of the 40-pages-long business plan I wrote for my last venture.
Here’s the list of posts published so far (more will come within the following few weeks):
- Zapier Usage Research
- The Fragmented ERP
- Overview of Online Databases & Business App Solutions
- Overview of Data Integration Tools
- Overview of Dashboard Solutions for Data Integration
- Overview of Process Automation & BPM Solutions
- Excel Spaghetti
- The Super Distribution of Data
- Developer & Vendor Tools for Data Integration