WSO2 Enterprise Service Bus (ESB) Performance Testing - Round 3

We have just concluded the third round of Enterprise Service Bus (ESB) performance testing here at WSO2!

Its great to hear that even BEA / Oracle was very much interested about the performance numbers we demonstrated last year, and wanted to beat our numbers with their latest release of AquaLogic Service Bus 3.0 on Weblogic 10. I think this demonstrates a level of trust by the Enterprise Service Bus (ESB) vendors and users alike, on the tests we conducted. Several users re-ran some of the tests themselves, during the past year; while some vendors promised to re-run the tests and publish their response, but have so far failed to talk about the results they obtained. Ross Mason from Mulesource has been wanting to publish performance test results for Mule for the last 2 years! The fact that we made the configurations/tuning and the tools used openly available, and asked for help from the vendors to optimally configure their ESB's for the scenarios, and reporting back any problems we encountered adds to the level of openness we demonstrated. Additionally this round of testing shows that even the WSO2 Enterprise Service Bus (ESB) / Apache Synapse Enterprise Service Bus (ESB) has room for improvement against a proprietary ESB that we benchmarked against, but also shows a very clear lead against the other open source alternatives.


It should be noted that the advantages of the WSO2 Enterprise Service Bus (ESB) does not lie only on its low resource usage foot print, excellent performance or scalability or its free and open source Apache License v2.0 alone; but in its ease of use, the ability to define a configuration graphically, and the fact that it ships over 55 working samples and documentation that demonstrates various features to help users become effective from day 1. Add to this the excellent level of free support that users always talk about, and compare this to some of the mailing lists for other open source ESB's where you hardly would get a reply to a technically challenging question.

Let me just highlight some of the observations and conclusions here

- Mule CE 2.0.1 couldn't handle the cases where we used a concurrency level of 80; while other ESB's scaled to support to over 2500 concurrent connections. This was after tuning the maximum active thread count to 100 from its default value, which limited Mule to a very few concurrent connections.

- A proprietary version of an open source ESB had the same problem described above. However, we cannot name this ESB due to its restrictive license which states that we cannot : “..publicly disseminate performance information or analysis (including, without limitation, benchmarks) from any source relating to the Software.”. One of my friends commented that the above line is a “Nice way to "performance tune" your slow app :P”

- Mule CE 2.0.1 also dropped 1% of ALL requests it received

- Apache ServiceMix 3.2.1 failed to forward the incoming SOAPAction for proxy services, and this was now a known issue (I would consider this a blocker, and would suggest that ServiceMix folks follow up a 3.2.2 release just to fix this critical issue)

- A proprietary ESB we benchmarked and beat last year, did some major improvements to their performance, and did 1.6~1.9 times better than us for some of the scenarios

- The WSO2 Enterprise Service Bus (ESB) / Apache Synapse Enterprise Service Bus (ESB) shows a clear lead and dominates the open source ESB space


Read about it all here: http://wso2.org/library/3740 and run the benchmarks yourself!