Freely subscribe to our NEWSLETTER

Many open source products have a small forum of contributors that can be asked questions, furthermore some of these contributors and their organisations will be happy to deliver commercial consultancy. Open source can often in fact be a marketing strategy for many organisations to enable them to sell services or add-ons.

There is an African proverb that has been applied to Open Source: “If you want to go fast, travel alone; but if you want to go far, travel together”.
The most useful and successful open source software is frequently used by many people and in some cases has significant corporate sponsorship. In this case open source can have very active development, however when the software is infrequently used, specialist and with few and rare contributions the model starts to fail.

Performance testing remains a niche discipline and hence the tools used for it remain relatively little used and less so developed when compared to something like an operating system (e.g. Ubuntu), FTP client (e.g. FileZilla) or word processing package (e.g. OpenOffice) which have huge user and contribution bases. Furthermore there is not just one open source performance testing tool, so development is further diluted. Effort needs to be sponsored and focussed onto one tool for any chance of success. Perhaps we’ll see this in the future, but at the moment it’s not the case.

Performance testers often find that open source tools are not able to work with the latest technology, are very time consuming to use and require a great degree of technical ability. A quote from an email we had from a prospect we’re working with right now is all too common in theme:
“We are struggling to find a happy cost medium at the moment. Load testing tools in the open source community seem mostly under-developed”.

So how do we work out the true cost of open source?
As a performance testing tool vendor we work with testing consultancies that specialise in performance testing frequently. When asked about this subject, Alan Gordon, Principal Consultant, SQS UK stated:

“There are two main parts to the differences – creating the tests & analysing the results after execution.
Creating the tests could mean an increase in effort anywhere between double and fourfold depending on the scripting language and complexity of the test case scenario.

Analysing the results can take five times as long using open source tools, but if effort is channelled into creating a framework that effort can be reduced greatly but will still usually be greater that required when using a reputable commercial tool”

Following this comment we decided to make a poll our network of performance test experts on the LinkedIn professional networking site and as of January 2010 this is what it the poll results look like (see over page):

Along with the poll interaction some interesting comments were made such as:

Aside from the cost there is the question of how reliable are the results that are generated. Load generated by one tool can be different from the load generated by another, even when we are talking about commercial tools, the level of accuracy and relevance of the load footprint per virtual user can be greatly varied.

There are many things to question such as the number of thread connections per user, whether keep alive is simulated, whether GZip compression is supported and lots more – all things that can vary the load footprint.
If a vendor gets this wrong then they are answerable to their customers, if an open source collaboration gets this wrong then eventually the bug will be fixed, but ultimately the individual that made the choice to use the open source tool will be held responsible for choosing the tool in the first place.

So to evaluate the cost of a typical performance testing project based on 15 days at £500 a day for a performance test consultant using a commercial tool vs. an open source tool:

Technology Required Effort Effort Cost Software Cost Total Cost Comments
Commercial 15 days £500 £5000 £12,500
Open Source 45 days £500 £0 £22,500 Additional risk and limitations

So what happens as the number of users increases? Well of course the commercial software model increases in price with the number of virtual users as most license model are based on virtual users, however at a certain point the open source model fails all together as many of these tools cannot be scaled as far as reputable commercial software can.

What happens when the number of test iterations increases? The commercial model becomes even more appealing as while the resource time (consultant or staff) increases the software costs remain static and become less of a proportion on the overall cost. In fact some features in commercial software can reduce the level of expertise required which can drive down the cost of resource or staff.

Quotium is exhibiting at 360°IT, the IT Infrastructure Event held 22nd – 23rd September 2010, at Earl’s Court, London. The event provides an essential road map of technologies for the management and development of a flexible, secure and dynamic IT infrastructure. For further information please visit www.360itevent.com