ESB Performance Testing - Round 7 Published!

We have just published the results for the long awaited Round 7 - of ESB performance benchmarking. This round compares 4 of the leading free and open source ESBs, Mule CE 3.4.0, Talend ESB SE 5.3.1, WSO2 ESB 4.7.0 and the UltraESB 2.0.0

Although at first glance the above image may indicate that the WSO2 ESB and the UltraESB have very similar performance characteristics, the Devil is indeed in the detail.

During the testing we discovered several issues with the previously published article Round 6.5 from WSO2, including a severe response corruption for messages over 16,384 bytes when the default pass-through transport is being used. However, what's most surprising is that this issue remains in versions 4.6.0 and 4.7.0 of the WSO2 ESB, as well as the latest Milestone 4 of the soon to be released version 4.8.0.

Sadly, the WSO2 engineers conducting the Round 6.5 also failed to notice a complete failure of all of the XSLT test cases, but nevertheless published numbers obtained for the failed test cases as high performance numbers over the other ESBs.

Read about these and other flaws of the Round 6.5 in the article Why the Round 6.5 results published by WSO2 is flawed

Digital Content and Service Provider Jesta Digital Migrates to the UltraESB

Zero-Copy proxying and Non-Blocking IO allows three node cluster to process ~80 million messages a day at just 5% CPU utilization and 300MB heap memory usage

AdroitLogic, developer of the high performance Open Source Enterprise Service Bus (ESB) UltraESB, today announced that Jesta Digital, a leading global provider of next generation entertainment content and services for the digital consumer, successfully migrated their ESB clusters to the UltraESB. Jesta Digital is behind Bitbop wireless subscription service to deliver on-demand commercial-free television and films to personal computers, tablets and smartphones, and home to a number of well-known and established brands including Jamba and Jamster, which deliver branded content, music, games and apps to millions of mobile consumers.

"In a very demanding and rapidly changing market it is of utmost importance to be able to quickly adjust the technical platform to support product innovation and change. The UltraESB was able to demonstrate to the Jesta Digital Technology team in Berlin that its simplicity, testability, extensibility and performance is made for a matching foundation" said Eric Hubert, the Executive Director of Strategy and Architecture at Jesta Digital.

On the main ESB cluster, Jesta Digital processes around 80 million messages a day with a peak 3,000 TPS across three nodes. The average CPU utilization has been just 5% with the heap memory usage at 300MB with very low GC overheads due to the efficient use of a RAM disk coupled with Zero-Copy and Non-Blocking IO. Prior to the migration, Jesta Digital customized the ESB Performance Benchmark (http://esbperformance.org) to compare the performance of the UltraESB for both SOAP and Hessian messages over HTTP/S for a sample scenario. Benchmark results similar to those published recently on the ESB Performance site were independently verified by Jesta Digital during this exercise. Furthermore the UltraESB showed extreme stability during stress tests executed over multiple days.

"We have a top notch technical team at Jesta Digital in Berlin and it was a great pleasure to work with subject-matter experts who combined first-hand knowledge, passionate work on their product and dedication to tackle the customer's challenges" said Eric.

During the migration process, AdroitLogic also worked with the Jesta Digital team to include some key features for even better enterprise deployment support. Among these is the ability to switch a live configuration with zero down time while messages are being processed. Previously servers had to be gracefully shutdown for updates, resulting in an unequal distribution of traffic across the nodes after a configuration switch, leading to connection related issues. In addition, AdroitLogic developed a feature to easily manage and monitor services and endpoints via automatic registration against a Zabbix monitoring server, using predefined or customized templates via JMX.

The UltraESB is clustered using the Apache ZooKeeper framework, and thus a single node, or the whole cluster can be managed from any of the nodes via a remote web based console, command line interface or via JMX/jconsole.

AdroitLogic was founded in January 2010 by Asankha Perera, the former lead contributor of the Apache Synapse ESB and the original architect of the WSO2 ESB. In September 2011, Ruwan Linton who initially succeeded Asankha at WSO2 as its next ESB architect, also joined AdroitLogic as its Director of Engineering. AdroitLogic published a new round of ESB benchmark results in October across 8 open source ESBs, which showed a very clear lead in performance against its competitors. In addition to performance, the UltraESB is also one of the simplest ESB's to use effectively, with support for IDE based step-through debugging, Unit testing and a light weight deployment with over 70 samples. AdroitLogic is also one of the very few - if not the only - ESB vendors that publishes code coverage statistics along with continuous builds and code quality reports publicly.

"It was a great pleasure to work with subject-matter experts who combined first-hand knowledge, passionate work on their product and dedication to tackle the customer's challenges"

Read the full Press Release and Download the Case Study on Migration to the Free and Open Source UltraESB

Former WSO2 ESB and Apache Synapse ESB Architect Ruwan Linton Joins AdroitLogic

It gives me great pleasure to announce that Ruwan Linton has joined AdroitLogic!

Ruwan was the former architect and product manager of the WSO2 ESB, taking over the position when I left WSO2 in October 2008. We've worked in the past together for over 2 years and contributed over 80% of the Apache Synapse ESB which is also the core of the WSO2 ESB.

Ruwan is a brilliant architect, and a good friend. We both understand each other perfectly well, and thus he will be a great addition to AdroitLogic as its new Director of Engineering!

Here is the full press release: Former WSO2 ESB and Apache Synapse ESB Architect Ruwan Linton Joins AdroitLogic

ESB Performance Testing.. Getting ready for the next round

Its been sometime since we've compared the performance of free and open source ESBs.. and now its time for the next round..

Amazon EC2 AMI to be published!
At the end of this round, we will publish an Amazon EC2 machine image (AMI) with the selected ESBs so that any end-user can simply fire up an instance of this image, and re-run the performance suite to compare the ESBs of his choice!

This will also make it easier to extend the performance test to include any custom test cases of the user etc, before selecting an ESB for use. But most of all, it will allow a true apples-to-apples comparison of the ESBs of choice, without any vendor bias, as each ESB could be tested on the same EC2 instance, and only allowed to talk to itself on the localhost - preventing a requirement for stringent network isolation.

In the past different vendors used the performance benchmark on different hardware configurations, and also it was not easy for a newbie to re-run the test suite on his own without investing some time to learn things and setup the various ESBs. However, the soon to be published EC2 AMI will contain everything required to fire-up the different ESB's of choice - preventing anyone from having to spend hours to learn and set-up the different ESBs for comparison. A vendor or another end user can also modify the configurations of the ESBs to better tune them, and republish the new image again, making the process simpler and easier for everyone!

The free and open source ESB's evaluated in this round include:

• UltraESB v1.5.1
• WSO2 ESB v4.0.0
• Mule CE v3.1.2
• ServiceMix v4.3.0

It has not been easy to set-up all these different ESB's, and we've found that its pretty difficult to get some of the ESB's to execute all test scenarios.. More on this will be published soon as we try to overcome these difficulties.. so stay tuned!!

If you would like any other ESB to be included into this round, do let us know, and share a compatible configuration.

Visit the ESB Performance site for more details..

Mulesoft publishes new results against the ESB Performance Testing Framework

Mulesoft recently published new results on the performance of the Mule ESB v 3.1 over Mule 2.2.7 benchmarked in July 2008.

"Mule 3.1 performance is in average 10% better than its predecessor version 2.2.7, performing better when the number of concurrent consumers gets bigger and much better when dealing with XSLT transformations (around 15% better)."

A clear difference from the previous results from Mulesoft is that they are only testing concurrency levels of 20,40,80 & 160 this time, while dropping the 320, 640, 1280 and 2560 concurrent user test scenarios from this round! They have not published results against the 100K message sizes either - or for the WS-Security test case introduced by AdroitLogic last February [See http://esbperformance.org].

Its interesting to note both Mule and WSO2 now follows a pattern of performance testing only against previous versions of their own ESBs - when previously both published performance figures over each other; other open source ESBs and even commercial equivalents.

AdroitLogic first published the figures for its UltraESB in February 2010, and took a decision to compare its performance against competition - but not to release the names of the other products tested. Instead, it ships the complete ESB Performance Test Framework including scripts to run and convert results into CSVs, so that an end user can easily compare its performance against virtually any other comparative ESB (even commercial alternatives) on an exact same hardware and software configuration for a true apples-to-apples comparison.

What happened to the Cat set amongst the Pigeons? - ESB Performance comparison

WSO2 Published an article titled "WSO2 ESB Performance (New)" today, after 2 and a half years since the last article on ESB performance - published in June 2008.

What's interesting to note is that unlike in the last three rounds of performance testing since June 2007, WSO2 does not compare its numbers against those of any other ESB.

The last time WSO2 published performance numbers, its CTO Paul Fremantle wrote the blog post Setting the Cat amongst the Pigeons stating "Now, I'm hoping the Mule guys will get their act together and provide a decent contest, because we couldn't get Mule to do HTTP KeepAlive without failing"

Previously WSO2 compared its performance against a proprietary ESB, Mule, and ServiceMix. A complete list of resources, and tools were shared under an Apache License to encourage other ESB vendors to run the same tests which were "fair" and open.

The folks at Mule did take the challenge, and published an article that showed performance of the Mule ESB, although it was not a true apples-to-apples comparison.

This time, WSO2 has not shared the configurations or the resources either, and for some unknown reason, have explicitly turned off HTTP keep-alives which generally improves performance. They compare against the WSO2 ESB v2.1.3 - which incidentally was not benchmarked publicly at any point in the past. Hence a 'comparison' becomes meaningless, irrespective of the fact that in quite many scenarios, the performance of the v3.0.1 has indeed decreased!

Note that on earlier occasions concurrency levels from 20, 40, 80, 160, 320, 640, 1280, 2560 were run, with each selected message size. However, deviating from this "fair" and "open" norm, this time a different approach was taken: "For each scenario each message size load is generated with concurrency varying from 20 to 300(increasing by 40 at each stage). Then the maximum transactions per second(TPS) achieved during this concurrency range is recorded as the Transactions per second(TPS) for the corresponding scenarios message size."

This begs the question - did the WSO2 ESB v3.0.1 crash after 300 concurrent users? when previous versions could handle up-to 2560 concurrent users??


I guess like some "other" companies that grows too large these days, WSO2 has finally closed its openness on the performance benchmarking as well. Its refraining from publishing any comparisons, as it cannot compare against the Free and Open Source UltraESB from AdroitLogic, which introduced Zero-copy proxying coupled with Java NIO for extreme performance levels, not yet achieved by others - although some have tried to.

AdroitLogic published performance metrics of the UltraESB in February 2010, and launched the ESB Performance site http://esbperformance.org to ensure that a "open" and "free" ESB Performance Benchmark will be available for anyone. The UltraESB "ships" this complete benchmark in its 30MB download - and makes it easy for anyone to compare the performance of the UltraESB with any other ESB - on any selected hardware and environment under "identical conditions"!

Finally, I know that WSO2 ran these tests over 4 months back, and since then they have possibly been busy to find a way to publish these results with minimum damage..

But I guess the Cat set amongst the Pigeons - is now no more! ..
and now, I'm expecting some fireworks!!

WSO2 falsely claims support for Zero copy

A few minutes ago, Asanka Abeysinghe the Director of Solutions Architecture of WSO2 claimed that the WSO2 ESB supports "Zero Copy", at the WSO2Con held in Colombo, 14th September 2010

However, this statement is false, and can be easily proved by just looking at the implementation source code of the BinaryRelayBuilder, which copies the complete input stream into a byte[] in Heap memory via traditional CPU copying into user space.

In-fact, when WSO2 announced this "Binary Relay" in December 2009, even the author of this "feature" did not state that its anything near "Zero Copy". Even a search on Google for WSO2 and Zero copy returns no results!

"Zero-copy" describes computer operations in which the CPU does not perform the task of copying data from one memory area to another." - Wikipedia

The AdroitLogic UltraESB is the FIRST and still the ONLY Open Source ESB (or most possibly "the only ESB") that supports True Zero-Copy proxying.

Thats why its the fastest ESB out there!

The ESB Performance Testing Framework & Results of Round 4

I developed the original ESB Performance Test Framework - in June 2007 while actively working on the Apache Synapse and the WSO2 ESBs. Since then, we've run three rounds of testing, comparing both Proprietary and Open Source ESBs - including Mule, Apache ServiceMix, Apache Synapse/WSO2 ESB, a leading proprietary ESB and the proprietary version of an open source ESB.

It was interesting to see other vendors such as Mule and BEA picking up this test framework in addition to WSO2, in publishing results. However, due to different hardware configurations being used, and other advanced tuning or optimizations performed by vendors - the results were left a bit questionable and could not be compared in a fair manner.

This latest round - Round 4 - takes this framework to the Amazon EC2 - and is designed to let end-users run the tests on an EC2 node for less than $2 of computing time! This will allow differences in hardware to not make any difference, and allow the users to see how exactly the tests have been configured, tuned and run.

I hope Mule, ServiceMix, JBoss, OpenESB, Petals, BEA/Oracle, IBM, WSO2 and any other ESB vendors I've missed will make use of this opportunity and publicly share the necessary configurations so that we could all rely on an accurate and fair benchmark - which can be verified independently on demand by any end-user!

Hence - unlike in the past, I will refrain from naming any competitor - unless its vendor or the open source project team requests that it be included and the results publicly shared. Thus it is now left to the users of ESBs - to demand the configurations necessary for them to run these tests from vendors - and themselves decide which one to select after executing the test on Amazon EC2. I hope to make this process even simpler at the next round - with a custom AMI and more automation

The Round 4 results includes the three test cases conducted earlier (i.e. Direct Proxy, Content Based Routing [CBR] Proxy, and XSLT Proxy), and adds a new scenario for WS-Security processing.

http://adroitlogic.org/samples-articles-and-tutorials/15-tutorials/48-esb-performance.html