Recently I did a small survey on to find out whether testing community prefers open source tools or commercial tools. After around one month, 81% participants voted for open source tools. I have been using open source testing tools for around four years now so I wasn’t surprised with the result. But still, this result got me interested and made me think about why so many people prefer open source tools? What are / were the problems with vendor tools? How open source tools have affected testers and the way we work?
I am sure some respondent might have voted for open source because of moral reasons but for me, and probably many others it is the value that we have got from using open source tools. I have been using Firefox, Selenium, FitNesse, WATIR, Selendion, Concordion and many such tools and was benefited greatly with the rich feature set and support. I have used vendor tools like Silk Test, Rational Robot, Rational Functional Tester, Quality Centre etc in past and have the first-hand experience of pain / problems of using them.
So let us start with the brief understanding of what is open source / free software? OSS (Open Source Software)/FS (Free Software) programs are programs whose licenses give users the freedom to run the program for any purpose, to study and modify the program, and to redistribute copies of either the original or modified program (without having to pay royalties to previous developers). It is important to understand that free does not refer to price, it refers to liberty. Free here refers to the freedom to understand, discuss, repair, and modify the technological devices / software user own.
In this paper, the author has done an in-depth study and given quantitative reasons to support open source adoption. This paper gives us very good understanding and (to some people) confidence that open source software works by capturing information on trend, reliability, performance, scalability and so on. The price might be a factor when adopting open source, but if this is the only reason it might be difficult to justify it sometimes. As mentioned earlier, Free in OSS is liberty, which allows users to have fundamental control and flexibility on the software, since they can modify and maintain their software to their liking. This is probably one of the most important reasons why open source works, in almost every sphere.
But what is making open source tools work specifically for the testing community? Let us discuss various factors which are important for adopting any tool and compare open source and commercial tools against them. I am taking a narrow view and summarizing these factors for the automated test execution tools only, some factors listed below might or might not be relevant for other tools.
1. Automation Language
Language in which you write your automation has a big influence on how maintainable and robust your automation suite is. If you use right language ( depending on your context), chances of getting the right support, technical know-how and libraries for the typical task are much higher. Historically, vendors have always given their own specific language which was good for only a single tool. In my opinion, this was one of the major drawbacks of tool vendors. This situation is changing with RFT (Rational Functional Tester) supporting JAVA / .NET and TestComplete supporting many languages. Open Source Software, scores really well on this front. Tools like Selenium support almost every major language, and because of their open nature, if support for any language is not present someone from the community might develop that.
It can be argued that support from tool vendors should be better because they are getting paid for it. However, In my experience, support from tool vendors have fallen short of support that we receive from the motivated bunch of people working as a community on any open source project. These folks are probably much more responsive than the various level of support sold by tool vendors. Tool vendors can probably guarantee that support will be available, but if you choose popular open source project, chances of getting right support will be higher. One important point to remember here is, with tool vendors, support is demanded (because you paid them.) and with open source, support is requested (Because you need them).
3. Feedback Loop I started working with test automation tools in 2001 and for initial 4-5
I started working with test automation tools in 2001 and for initial 4-5 years, they were more or less static. There were changes / improvements but more or less they were not radically different from one another or from the previous versions. One reason for this slow development was probably long feedback. Users and developers in these cases were two different entities and were oblivious to each other’s pain. On the other hand, in OSS users and developers are same folks, so feedback loop is extremely fast. That is the reason why tools like Selenium, WATIR etc. have become so popular and feature rich in such a short time.
4. Short evaluation period
Usually, if time is short for the evaluation process, wrong decisions will be made about the tool and eventually they will become shelfware. Good evaluation is essential, but most of the time evaluation period given by tool vendors is not good enough to find out if the tool is the right tool for the application under test? OSS on the other hand, give us long (evaluation) period so when we decide to use any tool, we make that decision with good knowledge on limitation, capabilities, and its applicability in the context of the application under test. This increases chances of succeeding with the tool and so its reputation.
5. Selling is driven by Marketing
Normally tools are sold by the sales people to the managers and not to the users. Tools are sold with wrong practices like record / playback, quick automation and so on. This gives the wrong impression to management in terms of what is good automation, how it should be approached and so on. Most of the time, practices like this result in unmaintainable automation suite which becomes useless very soon. Normally tools or testers are blamed in these cases, but most of the time it is not the tool or tester but the approach in which tool was sold /used. Most of the time its the reputation of the tool which is damaged in such instances.
6. Vendor locking
Historically, tools have always tried to lock users with their offerings. QTP is integrated with the quality center, Rational robot was integrated with test manager and so on. Thier internal formats, integration everything is coupled with other tools from their stack making it impossible to migrate from one tool to another without substantial rework once you are trapped. OSS, on the other hand, are like an open book and hardly have any motive to lock users.
7. Lack of choices
There are not many tools from the vendors and consolidation in the tool market means choices will be even lesser in future. There are only 4 or 5 major players in the field which hardly gives you any choice. Lack of choice is bad because it increases the possibility of monopoly and features are not delivered based on what user is expecting but based on what competitors are doing.
8. Community Feeling
Sure there are some products like Mac which can instill the feeling of community in users. On the software front, this behavior is more common for OS than vendors. This community acts as a support center, as a platform to discuss the problems and give direction in which development should be carried on. This community usually provides information to anyone who is seeking information, facing difficulty in implementing anything and so on, without any self-interest. I have not done any formal study but was never disappointed with the community of many OSS tools.
9. Specialization & Generalization
One of the main motives of tool vendors is to sell more products and make money. They increase their chances of sales by combining more features, providing support for many languages / platforms etc. Most of the time these products end up in a very bulky product, which is not easy to operate / understand and reduces the speed at which tools can be adopted. OSS on the other hand, try to solve one problem and they try to do it well. The Motive in OSS is not to sell more products, but to solve a specific problem and that takes them much closer to users than vendors.
10. Reduces difference between dev / test team IMO, with the usage of OSS,
IMO, with the usage of OSS, the testing community is much closer to development team than tools from the vendors. OSS has allowed developers and testers to talk in the same language, use the same set of tools and allowed tighter integration. They have also allowed testers to leverage the work from developers and understand their work more closely. In the old world of vendor tools, testers and developers were effectively in their own silos, OSS has helped in reducing that gap
These, IMO are the top 10 reasons why open source tools are more popular in the testing community than tools from vendors. One interesting reason which I have omitted is the emergence of the web as a platform. Are you using OSS because of any other reason? Please leave your reasons in the comments and we can discuss.