ESB performance testing - Round 5 Results for 8 Open Source ESBs

AdroitLogic Private Ltd. announced today the results of the fifth round of ESB Performance testing. This round compares the WSO2 ESB, Mule ESB CE, Fuse ESB, Apache ServiceMix, Talend ESB, Petals ESB, JBoss ESB and the UltraESB. The ESB performance benchmark has been in use since June 2007 by vendors such as WSO2, AdoritLogic, Mulesoft and BEA to compare ESBs across different loads and message sizes for six common use cases. This round also publishes an EC2 AMI to easily reproduce the results by average users, and also publishes all source configuration for the 8 different ESBs in a public BitBucket repository

What's significant about this test suite is that it has now been accepted as a de-facto ESB performance test suite by vendors such as WSO2, BEA and Mulesoft in the past, who have reported results against their implementations publicly. This has also been used extensively by end users for ESB comparison, and load testing of configurations before production deployments.

Although 8 open source ESBs were considered, 4 of these failed without being able to complete the load test. One ESB had a 3-4% error rate, while three completed successfully. The test includes two variants of the free and open source UltraESB. The enhanced version is supercharged with the power of VTD XML which allows the UltraESB to perform XPath evaluation on XML payloads without XML parsing. This works nicely with the RAM Disk based file cache used by the UltraESB to provide both Zero-Copy and Java Non-Blocking IO for the extreme levels of performance.


Full information available at http://esbperformance.org

ESB Performance Testing.. Getting ready for the next round

Its been sometime since we've compared the performance of free and open source ESBs.. and now its time for the next round..

Amazon EC2 AMI to be published!
At the end of this round, we will publish an Amazon EC2 machine image (AMI) with the selected ESBs so that any end-user can simply fire up an instance of this image, and re-run the performance suite to compare the ESBs of his choice!

This will also make it easier to extend the performance test to include any custom test cases of the user etc, before selecting an ESB for use. But most of all, it will allow a true apples-to-apples comparison of the ESBs of choice, without any vendor bias, as each ESB could be tested on the same EC2 instance, and only allowed to talk to itself on the localhost - preventing a requirement for stringent network isolation.

In the past different vendors used the performance benchmark on different hardware configurations, and also it was not easy for a newbie to re-run the test suite on his own without investing some time to learn things and setup the various ESBs. However, the soon to be published EC2 AMI will contain everything required to fire-up the different ESB's of choice - preventing anyone from having to spend hours to learn and set-up the different ESBs for comparison. A vendor or another end user can also modify the configurations of the ESBs to better tune them, and republish the new image again, making the process simpler and easier for everyone!

The free and open source ESB's evaluated in this round include:

• UltraESB v1.5.1
• WSO2 ESB v4.0.0
• Mule CE v3.1.2
• ServiceMix v4.3.0

It has not been easy to set-up all these different ESB's, and we've found that its pretty difficult to get some of the ESB's to execute all test scenarios.. More on this will be published soon as we try to overcome these difficulties.. so stay tuned!!

If you would like any other ESB to be included into this round, do let us know, and share a compatible configuration.

Visit the ESB Performance site for more details..

The ESB Performance Testing Framework & Results of Round 4

I developed the original ESB Performance Test Framework - in June 2007 while actively working on the Apache Synapse and the WSO2 ESBs. Since then, we've run three rounds of testing, comparing both Proprietary and Open Source ESBs - including Mule, Apache ServiceMix, Apache Synapse/WSO2 ESB, a leading proprietary ESB and the proprietary version of an open source ESB.

It was interesting to see other vendors such as Mule and BEA picking up this test framework in addition to WSO2, in publishing results. However, due to different hardware configurations being used, and other advanced tuning or optimizations performed by vendors - the results were left a bit questionable and could not be compared in a fair manner.

This latest round - Round 4 - takes this framework to the Amazon EC2 - and is designed to let end-users run the tests on an EC2 node for less than $2 of computing time! This will allow differences in hardware to not make any difference, and allow the users to see how exactly the tests have been configured, tuned and run.

I hope Mule, ServiceMix, JBoss, OpenESB, Petals, BEA/Oracle, IBM, WSO2 and any other ESB vendors I've missed will make use of this opportunity and publicly share the necessary configurations so that we could all rely on an accurate and fair benchmark - which can be verified independently on demand by any end-user!

Hence - unlike in the past, I will refrain from naming any competitor - unless its vendor or the open source project team requests that it be included and the results publicly shared. Thus it is now left to the users of ESBs - to demand the configurations necessary for them to run these tests from vendors - and themselves decide which one to select after executing the test on Amazon EC2. I hope to make this process even simpler at the next round - with a custom AMI and more automation

The Round 4 results includes the three test cases conducted earlier (i.e. Direct Proxy, Content Based Routing [CBR] Proxy, and XSLT Proxy), and adds a new scenario for WS-Security processing.

http://adroitlogic.org/samples-articles-and-tutorials/15-tutorials/48-esb-performance.html