Monday, August 8, 2011

ESB Performance Testing.. Getting ready for the next round

Its been sometime since we've compared the performance of free and open source ESBs.. and now its time for the next round..

Amazon EC2 AMI to be published!
At the end of this round, we will publish an Amazon EC2 machine image (AMI) with the selected ESBs so that any end-user can simply fire up an instance of this image, and re-run the performance suite to compare the ESBs of his choice!

This will also make it easier to extend the performance test to include any custom test cases of the user etc, before selecting an ESB for use. But most of all, it will allow a true apples-to-apples comparison of the ESBs of choice, without any vendor bias, as each ESB could be tested on the same EC2 instance, and only allowed to talk to itself on the localhost - preventing a requirement for stringent network isolation.

In the past different vendors used the performance benchmark on different hardware configurations, and also it was not easy for a newbie to re-run the test suite on his own without investing some time to learn things and setup the various ESBs. However, the soon to be published EC2 AMI will contain everything required to fire-up the different ESB's of choice - preventing anyone from having to spend hours to learn and set-up the different ESBs for comparison. A vendor or another end user can also modify the configurations of the ESBs to better tune them, and republish the new image again, making the process simpler and easier for everyone!

The free and open source ESB's evaluated in this round include:

  • UltraESB v1.5.1
  • WSO2 ESB v4.0.0
  • Mule CE v3.1.2
  • ServiceMix v4.3.0
It has not been easy to set-up all these different ESB's, and we've found that its pretty difficult to get some of the ESB's to execute all test scenarios.. More on this will be published soon as we try to overcome these difficulties.. so stay tuned!!

If you would like any other ESB to be included into this round, do let us know, and share a compatible configuration.

Visit the ESB Performance site for more details..