WSO2 Published an article titled "WSO2 ESB Performance (New)" today, after 2 and a half years since the last article on ESB performance - published in June 2008.
What's interesting to note is that unlike in the last three rounds of performance testing since June 2007, WSO2 does not compare its numbers against those of any other ESB.
The last time WSO2 published performance numbers, its CTO Paul Fremantle wrote the blog post Setting the Cat amongst the Pigeons stating "Now, I'm hoping the Mule guys will get their act together and provide a decent contest, because we couldn't get Mule to do HTTP KeepAlive without failing"
Previously WSO2 compared its performance against a proprietary ESB, Mule, and ServiceMix. A complete list of resources, and tools were shared under an Apache License to encourage other ESB vendors to run the same tests which were "fair" and open.
The folks at Mule did take the challenge, and published an article that showed performance of the Mule ESB, although it was not a true apples-to-apples comparison.
This time, WSO2 has not shared the configurations or the resources either, and for some unknown reason, have explicitly turned off HTTP keep-alives which generally improves performance. They compare against the WSO2 ESB v2.1.3 - which incidentally was not benchmarked publicly at any point in the past. Hence a 'comparison' becomes meaningless, irrespective of the fact that in quite many scenarios, the performance of the v3.0.1 has indeed decreased!
Note that on earlier occasions concurrency levels from 20, 40, 80, 160, 320, 640, 1280, 2560 were run, with each selected message size. However, deviating from this "fair" and "open" norm, this time a different approach was taken: "For each scenario each message size load is generated with concurrency varying from 20 to 300(increasing by 40 at each stage). Then the maximum transactions per second(TPS) achieved during this concurrency range is recorded as the Transactions per second(TPS) for the corresponding scenarios message size."
This begs the question - did the WSO2 ESB v3.0.1 crash after 300 concurrent users? when previous versions could handle up-to 2560 concurrent users??
I guess like some "other" companies that grows too large these days, WSO2 has finally closed its openness on the performance benchmarking as well. Its refraining from publishing any comparisons, as it cannot compare against the Free and Open Source UltraESB from AdroitLogic, which introduced Zero-copy proxying coupled with Java NIO for extreme performance levels, not yet achieved by others - although some have tried to.
AdroitLogic published performance metrics of the UltraESB in February 2010, and launched the ESB Performance site http://esbperformance.org to ensure that a "open" and "free" ESB Performance Benchmark will be available for anyone. The UltraESB "ships" this complete benchmark in its 30MB download - and makes it easy for anyone to compare the performance of the UltraESB with any other ESB - on any selected hardware and environment under "identical conditions"!
Finally, I know that WSO2 ran these tests over 4 months back, and since then they have possibly been busy to find a way to publish these results with minimum damage..
But I guess the Cat set amongst the Pigeons - is now no more! ..
and now, I'm expecting some fireworks!!
2 comments:
Found figures and analysis below as useful. Thanks. Do you have any direct performance comparison figures between WSO2/UltraESB/Mule etc ?
http://adroitlogic.org/samples-articles-and-tutorials/15-tutorials/48-esb-performance.html
We plan to execute another round of performance testing around January 2010, and will include other free and open source ESBs although we will not name them directly.
We encourage users to run the performance test themselves, between the UltraESB and any others, and thus we include the complete performance test framework and tools with our 30MB binary distribution to make this task easy
Post a Comment