ESB Performance Testing - Round 7 Published!

We have just published the results for the long awaited Round 7 - of ESB performance benchmarking. This round compares 4 of the leading free and open source ESBs, Mule CE 3.4.0, Talend ESB SE 5.3.1, WSO2 ESB 4.7.0 and the UltraESB 2.0.0

Although at first glance the above image may indicate that the WSO2 ESB and the UltraESB have very similar performance characteristics, the Devil is indeed in the detail.

During the testing we discovered several issues with the previously published article Round 6.5 from WSO2, including a severe response corruption for messages over 16,384 bytes when the default pass-through transport is being used. However, what's most surprising is that this issue remains in versions 4.6.0 and 4.7.0 of the WSO2 ESB, as well as the latest Milestone 4 of the soon to be released version 4.8.0.

Sadly, the WSO2 engineers conducting the Round 6.5 also failed to notice a complete failure of all of the XSLT test cases, but nevertheless published numbers obtained for the failed test cases as high performance numbers over the other ESBs.

Read about these and other flaws of the Round 6.5 in the article Why the Round 6.5 results published by WSO2 is flawed

ESB performance testing - Round 5 Results for 8 Open Source ESBs

AdroitLogic Private Ltd. announced today the results of the fifth round of ESB Performance testing. This round compares the WSO2 ESB, Mule ESB CE, Fuse ESB, Apache ServiceMix, Talend ESB, Petals ESB, JBoss ESB and the UltraESB. The ESB performance benchmark has been in use since June 2007 by vendors such as WSO2, AdoritLogic, Mulesoft and BEA to compare ESBs across different loads and message sizes for six common use cases. This round also publishes an EC2 AMI to easily reproduce the results by average users, and also publishes all source configuration for the 8 different ESBs in a public BitBucket repository

What's significant about this test suite is that it has now been accepted as a de-facto ESB performance test suite by vendors such as WSO2, BEA and Mulesoft in the past, who have reported results against their implementations publicly. This has also been used extensively by end users for ESB comparison, and load testing of configurations before production deployments.

Although 8 open source ESBs were considered, 4 of these failed without being able to complete the load test. One ESB had a 3-4% error rate, while three completed successfully. The test includes two variants of the free and open source UltraESB. The enhanced version is supercharged with the power of VTD XML which allows the UltraESB to perform XPath evaluation on XML payloads without XML parsing. This works nicely with the RAM Disk based file cache used by the UltraESB to provide both Zero-Copy and Java Non-Blocking IO for the extreme levels of performance.


Full information available at http://esbperformance.org

ESB Performance Testing.. Getting ready for the next round

Its been sometime since we've compared the performance of free and open source ESBs.. and now its time for the next round..

Amazon EC2 AMI to be published!
At the end of this round, we will publish an Amazon EC2 machine image (AMI) with the selected ESBs so that any end-user can simply fire up an instance of this image, and re-run the performance suite to compare the ESBs of his choice!

This will also make it easier to extend the performance test to include any custom test cases of the user etc, before selecting an ESB for use. But most of all, it will allow a true apples-to-apples comparison of the ESBs of choice, without any vendor bias, as each ESB could be tested on the same EC2 instance, and only allowed to talk to itself on the localhost - preventing a requirement for stringent network isolation.

In the past different vendors used the performance benchmark on different hardware configurations, and also it was not easy for a newbie to re-run the test suite on his own without investing some time to learn things and setup the various ESBs. However, the soon to be published EC2 AMI will contain everything required to fire-up the different ESB's of choice - preventing anyone from having to spend hours to learn and set-up the different ESBs for comparison. A vendor or another end user can also modify the configurations of the ESBs to better tune them, and republish the new image again, making the process simpler and easier for everyone!

The free and open source ESB's evaluated in this round include:

• UltraESB v1.5.1
• WSO2 ESB v4.0.0
• Mule CE v3.1.2
• ServiceMix v4.3.0

It has not been easy to set-up all these different ESB's, and we've found that its pretty difficult to get some of the ESB's to execute all test scenarios.. More on this will be published soon as we try to overcome these difficulties.. so stay tuned!!

If you would like any other ESB to be included into this round, do let us know, and share a compatible configuration.

Visit the ESB Performance site for more details..

Mulesoft publishes new results against the ESB Performance Testing Framework

Mulesoft recently published new results on the performance of the Mule ESB v 3.1 over Mule 2.2.7 benchmarked in July 2008.

"Mule 3.1 performance is in average 10% better than its predecessor version 2.2.7, performing better when the number of concurrent consumers gets bigger and much better when dealing with XSLT transformations (around 15% better)."

A clear difference from the previous results from Mulesoft is that they are only testing concurrency levels of 20,40,80 & 160 this time, while dropping the 320, 640, 1280 and 2560 concurrent user test scenarios from this round! They have not published results against the 100K message sizes either - or for the WS-Security test case introduced by AdroitLogic last February [See http://esbperformance.org].

Its interesting to note both Mule and WSO2 now follows a pattern of performance testing only against previous versions of their own ESBs - when previously both published performance figures over each other; other open source ESBs and even commercial equivalents.

AdroitLogic first published the figures for its UltraESB in February 2010, and took a decision to compare its performance against competition - but not to release the names of the other products tested. Instead, it ships the complete ESB Performance Test Framework including scripts to run and convert results into CSVs, so that an end user can easily compare its performance against virtually any other comparative ESB (even commercial alternatives) on an exact same hardware and software configuration for a true apples-to-apples comparison.

Enterprise Service Bus [ESB] Comparison: Poll a directory for a file and upload it with an ...

Enterprise Service Bus [ESB] Comparison: Poll a directory for a file and upload it with a HTTP PUT: "Here is an example of a File polling service, which then sends the file to an authenticated HTTP service as a PUT request"

What happened to the Cat set amongst the Pigeons? - ESB Performance comparison

WSO2 Published an article titled "WSO2 ESB Performance (New)" today, after 2 and a half years since the last article on ESB performance - published in June 2008.

What's interesting to note is that unlike in the last three rounds of performance testing since June 2007, WSO2 does not compare its numbers against those of any other ESB.

The last time WSO2 published performance numbers, its CTO Paul Fremantle wrote the blog post Setting the Cat amongst the Pigeons stating "Now, I'm hoping the Mule guys will get their act together and provide a decent contest, because we couldn't get Mule to do HTTP KeepAlive without failing"

Previously WSO2 compared its performance against a proprietary ESB, Mule, and ServiceMix. A complete list of resources, and tools were shared under an Apache License to encourage other ESB vendors to run the same tests which were "fair" and open.

The folks at Mule did take the challenge, and published an article that showed performance of the Mule ESB, although it was not a true apples-to-apples comparison.

This time, WSO2 has not shared the configurations or the resources either, and for some unknown reason, have explicitly turned off HTTP keep-alives which generally improves performance. They compare against the WSO2 ESB v2.1.3 - which incidentally was not benchmarked publicly at any point in the past. Hence a 'comparison' becomes meaningless, irrespective of the fact that in quite many scenarios, the performance of the v3.0.1 has indeed decreased!

Note that on earlier occasions concurrency levels from 20, 40, 80, 160, 320, 640, 1280, 2560 were run, with each selected message size. However, deviating from this "fair" and "open" norm, this time a different approach was taken: "For each scenario each message size load is generated with concurrency varying from 20 to 300(increasing by 40 at each stage). Then the maximum transactions per second(TPS) achieved during this concurrency range is recorded as the Transactions per second(TPS) for the corresponding scenarios message size."

This begs the question - did the WSO2 ESB v3.0.1 crash after 300 concurrent users? when previous versions could handle up-to 2560 concurrent users??


I guess like some "other" companies that grows too large these days, WSO2 has finally closed its openness on the performance benchmarking as well. Its refraining from publishing any comparisons, as it cannot compare against the Free and Open Source UltraESB from AdroitLogic, which introduced Zero-copy proxying coupled with Java NIO for extreme performance levels, not yet achieved by others - although some have tried to.

AdroitLogic published performance metrics of the UltraESB in February 2010, and launched the ESB Performance site http://esbperformance.org to ensure that a "open" and "free" ESB Performance Benchmark will be available for anyone. The UltraESB "ships" this complete benchmark in its 30MB download - and makes it easy for anyone to compare the performance of the UltraESB with any other ESB - on any selected hardware and environment under "identical conditions"!

Finally, I know that WSO2 ran these tests over 4 months back, and since then they have possibly been busy to find a way to publish these results with minimum damage..

But I guess the Cat set amongst the Pigeons - is now no more! ..
and now, I'm expecting some fireworks!!

Download Email and save attachments into the file system - Compare the UltraESB with Mule

For over two weeks, a user on the Mule mailing list has been looking for a way to download Email messages over POP3/IMAP and save attachments into the file system. However, as of today, the last post indicates a blocker issue MULE-5138 which needs a solution before IMAP maybe used..

I've written a simple solution over the Free and Open Source UltraESB to do the same, and it works well, and easily, with only a few lines of configuration as presented in the new ESB Comparison blog article titled "Download Email and save attachments into the file system". This example will be available as Sample #503 with future versions of the UltraESB

The ESB Performance Testing Framework & Results of Round 4

I developed the original ESB Performance Test Framework - in June 2007 while actively working on the Apache Synapse and the WSO2 ESBs. Since then, we've run three rounds of testing, comparing both Proprietary and Open Source ESBs - including Mule, Apache ServiceMix, Apache Synapse/WSO2 ESB, a leading proprietary ESB and the proprietary version of an open source ESB.

It was interesting to see other vendors such as Mule and BEA picking up this test framework in addition to WSO2, in publishing results. However, due to different hardware configurations being used, and other advanced tuning or optimizations performed by vendors - the results were left a bit questionable and could not be compared in a fair manner.

This latest round - Round 4 - takes this framework to the Amazon EC2 - and is designed to let end-users run the tests on an EC2 node for less than $2 of computing time! This will allow differences in hardware to not make any difference, and allow the users to see how exactly the tests have been configured, tuned and run.

I hope Mule, ServiceMix, JBoss, OpenESB, Petals, BEA/Oracle, IBM, WSO2 and any other ESB vendors I've missed will make use of this opportunity and publicly share the necessary configurations so that we could all rely on an accurate and fair benchmark - which can be verified independently on demand by any end-user!

Hence - unlike in the past, I will refrain from naming any competitor - unless its vendor or the open source project team requests that it be included and the results publicly shared. Thus it is now left to the users of ESBs - to demand the configurations necessary for them to run these tests from vendors - and themselves decide which one to select after executing the test on Amazon EC2. I hope to make this process even simpler at the next round - with a custom AMI and more automation

The Round 4 results includes the three test cases conducted earlier (i.e. Direct Proxy, Content Based Routing [CBR] Proxy, and XSLT Proxy), and adds a new scenario for WS-Security processing.

http://adroitlogic.org/samples-articles-and-tutorials/15-tutorials/48-esb-performance.html