ESB performance testing - Round 5 Results for 8 Open Source ESBs

AdroitLogic Private Ltd. announced today the results of the fifth round of ESB Performance testing. This round compares the WSO2 ESB, Mule ESB CE, Fuse ESB, Apache ServiceMix, Talend ESB, Petals ESB, JBoss ESB and the UltraESB. The ESB performance benchmark has been in use since June 2007 by vendors such as WSO2, AdoritLogic, Mulesoft and BEA to compare ESBs across different loads and message sizes for six common use cases. This round also publishes an EC2 AMI to easily reproduce the results by average users, and also publishes all source configuration for the 8 different ESBs in a public BitBucket repository

What's significant about this test suite is that it has now been accepted as a de-facto ESB performance test suite by vendors such as WSO2, BEA and Mulesoft in the past, who have reported results against their implementations publicly. This has also been used extensively by end users for ESB comparison, and load testing of configurations before production deployments.

Although 8 open source ESBs were considered, 4 of these failed without being able to complete the load test. One ESB had a 3-4% error rate, while three completed successfully. The test includes two variants of the free and open source UltraESB. The enhanced version is supercharged with the power of VTD XML which allows the UltraESB to perform XPath evaluation on XML payloads without XML parsing. This works nicely with the RAM Disk based file cache used by the UltraESB to provide both Zero-Copy and Java Non-Blocking IO for the extreme levels of performance.


Full information available at http://esbperformance.org

Transactions spanning multiple resources - an Example with JMS, JDBC and File systems with JTA

We've written a sample that shows use of multiple JTA transactions spanning JMS, JDBC and File system resources. The sample is a solution in reply to a real user question as presented here.

The solution is presented using the Atomikos JTA libraries that allows the execution of the sample without deploying the UltraESB on a JEE application server such as JBoss. The original post requested for a sample using WebSphere MQ, and Oracle as the database, on top of JBoss.

The sample we've implemented uses an "embedded" ActiveMQ JMS server - started with just a couple of lines of Spring configuration within the UltraESB. The complete mediation is just a few lines of extremely powerful code - using the robust Spring framework JDBCTemplates. This coupled with the use of Java allows one to write powerful integration scripts, with the full power of the Java programming language or any other JSR 223 scripting language such as Ruby, Groovy, Javascript etc., without learning vendor or product specific configuration languages based on XML.

The sample shows the happy day scenarios as well as rollback of transactions, using the SOA ToolBox to fire sample JMS messages for testing.

The ESB Performance Testing Framework & Results of Round 4

I developed the original ESB Performance Test Framework - in June 2007 while actively working on the Apache Synapse and the WSO2 ESBs. Since then, we've run three rounds of testing, comparing both Proprietary and Open Source ESBs - including Mule, Apache ServiceMix, Apache Synapse/WSO2 ESB, a leading proprietary ESB and the proprietary version of an open source ESB.

It was interesting to see other vendors such as Mule and BEA picking up this test framework in addition to WSO2, in publishing results. However, due to different hardware configurations being used, and other advanced tuning or optimizations performed by vendors - the results were left a bit questionable and could not be compared in a fair manner.

This latest round - Round 4 - takes this framework to the Amazon EC2 - and is designed to let end-users run the tests on an EC2 node for less than $2 of computing time! This will allow differences in hardware to not make any difference, and allow the users to see how exactly the tests have been configured, tuned and run.

I hope Mule, ServiceMix, JBoss, OpenESB, Petals, BEA/Oracle, IBM, WSO2 and any other ESB vendors I've missed will make use of this opportunity and publicly share the necessary configurations so that we could all rely on an accurate and fair benchmark - which can be verified independently on demand by any end-user!

Hence - unlike in the past, I will refrain from naming any competitor - unless its vendor or the open source project team requests that it be included and the results publicly shared. Thus it is now left to the users of ESBs - to demand the configurations necessary for them to run these tests from vendors - and themselves decide which one to select after executing the test on Amazon EC2. I hope to make this process even simpler at the next round - with a custom AMI and more automation

The Round 4 results includes the three test cases conducted earlier (i.e. Direct Proxy, Content Based Routing [CBR] Proxy, and XSLT Proxy), and adds a new scenario for WS-Security processing.

http://adroitlogic.org/samples-articles-and-tutorials/15-tutorials/48-esb-performance.html