We have just published the results for the long awaited Round 7 - of ESB performance benchmarking. This round compares 4 of the leading free and open source ESBs, Mule CE 3.4.0, Talend ESB SE 5.3.1, WSO2 ESB 4.7.0 and the UltraESB 2.0.0
Although at first glance the above image may indicate that the WSO2 ESB and the UltraESB have very similar performance characteristics, the Devil is indeed in the detail.
During the testing we discovered several issues with the previously published article Round 6.5 from WSO2, including a severe response corruption for messages over 16,384 bytes when the default pass-through transport is being used. However, what's most surprising is that this issue remains in versions 4.6.0 and 4.7.0 of the WSO2 ESB, as well as the latest Milestone 4 of the soon to be released version 4.8.0.
Sadly, the WSO2 engineers conducting the Round 6.5 also failed to notice a complete failure of all of the XSLT test cases, but nevertheless published numbers obtained for the failed test cases as high performance numbers over the other ESBs.
Read about these and other flaws of the Round 6.5 in the article Why the Round 6.5 results published by WSO2 is flawed
During the testing we discovered several issues with the previously published article Round 6.5 from WSO2, including a severe response corruption for messages over 16,384 bytes when the default pass-through transport is being used. However, what's most surprising is that this issue remains in versions 4.6.0 and 4.7.0 of the WSO2 ESB, as well as the latest Milestone 4 of the soon to be released version 4.8.0.
Sadly, the WSO2 engineers conducting the Round 6.5 also failed to notice a complete failure of all of the XSLT test cases, but nevertheless published numbers obtained for the failed test cases as high performance numbers over the other ESBs.
Read about these and other flaws of the Round 6.5 in the article Why the Round 6.5 results published by WSO2 is flawed
6 comments:
First of all I should appreciate your effort to run these performance bench marks which in fact lead to performance enhancement in esbs. But I would like to point out some inconsistence of the result obtained as I see. What are your explanations about following cases?
a) if we take UltraESB, it has shown 3,895 for a 500b case while there are over 4000 cases for 10k. How a 10k shows a better performance than 500b? b) if we look at the average TPS values for a one message size (say 1k,10k) I can see a TPS variation with concurrency. But this is not possible. Once the system gets to a saturated sate it should shows same TPS irrespective of concurrency. c) Lets take the lk 2560 case. There the maximum TPS is 7,327 while average is 5,475. This means average for low TPS values is 4,549. That is at least 61% different between best case and worst case.
My explanation of all these cases are not having enough number of messages to run at each scenario. Typically each scenario would have run at least 4-5 minits. I have done lot of performance tests including tests given in this benchmark and never see those problems when we have higher number of messages.
Another thing is the analyzing part. How fair it is to take an average through all message sizes and show one performance figure? In this case the esbs perform better in higher messages sizes have a disadvantage. Again good example is UltraESB. It has show better result in 100k messages but has not reflected that in final results. In this case we need to draw graph for each and every case and show how performance varied with the messages size. This way readers can get a much better idea about the behavior of the esb according to their scenarios and pick the best.
You have indicated you are going to change the benchmark for next testings. So what I would like to suggest is to fix those as well.
Hi Amila
Thanks for your comment.. and good to hear from you. Let me address the points in brief.
a)3,895 was for the 640 concurrency with 500 bytes payload, the corresponding value for 10K at 640 concurrency is 3,554. So I am not sure exactly what you point out in this one, but in general the UltraESB has a zero-copy overhead for small payloads, but it starts to pay off significantly as the payload size increases. b) true, I agree. We used these test cases from many years back - esp when some of the other ESB failed to complete the test or took a very very long time. That's why the number of iterations were done this way - its historical. This is the reason why we would update the benchmark to be better, and your suggestions are definitely considered. c) these values are between three different runs, and yes, there is a variance, and I guess the low number of messages used (if I remember - 10) is the culprit.
You are correct in your analysis that the number of messages should be increased, the reason why we ran this exact same test was to wrap up this benchmark one final time before starting the new one, and also to highlight some of the flaws which would otherwise be thought as facts from the 6.5 article. Again, on the analysis, we used the same style of analysis - just to be the same as our last round. We will improve on these aspects in the new benchmark.
As someone I know who has really run valid performance benchmarks, I would be happy to work with you in future to define the new benchmark. Check the contributions to the bitbucket project so far, and you will realize that many others also would help in this endeavor.
regards
asankha
a)3,895 was for the 640 concurrency with 500 bytes payload, the corresponding value for 10K at 640 concurrency is 3,554
Here you have taken the same concurrency level. But there is no need for that. What I have mentioned here is that there 10k messages with 4,598 and 4,361. For any system there is no reason to show low performance numbers with low payload. At least it should be same.
As I mentioned this is not a problem of UltraESB or any other ESB (you can see the same thing with WSO2 ESB as well) but lack of messages to run the performance test. Therefore as you also has mentioned better to increase the number of messages to avoid these.
Amila
We will definitely improve on the message numbers next time. Hope you will help with your contributions too
regards
asankha
Asanka, Thanking you for analyzing and finding bugs in WSO2 ESB Products..Hope you will find more ..keep up your good work..will come hard with 4.8.0 (thanks to you..:) to you we have resolved the payload corruption issue...the bottom line is ...WSO2 ESB is keep getting faster with added features..(many features).. hope you will do many rounds of testigs with ESBs to come up
Cheers
Dushan
Asanka,
Thanks for analyzing WSO2 ESB, and again you did managed find bugs as always you do.. where we never intent to find any problematic cases for any other products. and as professionals we never intent do in such way..may be you have that intention which flows through the years to get over WSO2 ESB..but I would like emphasis this..we are purely open source product.(as you know which is version driven and we do releases chunks regularly so if something wrong you CAN NOT say it's a disaster) you may have proven something against WSO2 ESB..such as highlighting payload corruption.. which we have fixed just in seconds..and you can not emphases in any article saying, WSO2 ESB corrupt payload and will do For ever,chill dude it wont happen..xtream.xpath is not something which ESB shipped by default and default setup will never corrupt any payload..and as always it requires enhancements..anyway the botom line is WSO2 ESB is faster (is faster in many cases compare to Ultra ESB) and has many features and many many more features to added with fantastic tooling capabilities compare any OPEN source products (which is valid to your product as well)..
Cheers
D.S
Post a Comment