Monday, August 8, 2011

ESB Performance Testing.. Getting ready for the next round

Its been sometime since we've compared the performance of free and open source ESBs.. and now its time for the next round..

Amazon EC2 AMI to be published!
At the end of this round, we will publish an Amazon EC2 machine image (AMI) with the selected ESBs so that any end-user can simply fire up an instance of this image, and re-run the performance suite to compare the ESBs of his choice!

This will also make it easier to extend the performance test to include any custom test cases of the user etc, before selecting an ESB for use. But most of all, it will allow a true apples-to-apples comparison of the ESBs of choice, without any vendor bias, as each ESB could be tested on the same EC2 instance, and only allowed to talk to itself on the localhost - preventing a requirement for stringent network isolation.

In the past different vendors used the performance benchmark on different hardware configurations, and also it was not easy for a newbie to re-run the test suite on his own without investing some time to learn things and setup the various ESBs. However, the soon to be published EC2 AMI will contain everything required to fire-up the different ESB's of choice - preventing anyone from having to spend hours to learn and set-up the different ESBs for comparison. A vendor or another end user can also modify the configurations of the ESBs to better tune them, and republish the new image again, making the process simpler and easier for everyone!

The free and open source ESB's evaluated in this round include:

  • UltraESB v1.5.1
  • WSO2 ESB v4.0.0
  • Mule CE v3.1.2
  • ServiceMix v4.3.0
It has not been easy to set-up all these different ESB's, and we've found that its pretty difficult to get some of the ESB's to execute all test scenarios.. More on this will be published soon as we try to overcome these difficulties.. so stay tuned!!

If you would like any other ESB to be included into this round, do let us know, and share a compatible configuration.

Visit the ESB Performance site for more details..

6 comments:

Alp Şehiç said...

Hi there,

would you like also include;

- Talend ESB SE 4.2.0
=> After Sopera acqusition, they became a serious player in OS ESB field ( + also same comment for Fuse ESB)

- Fuse ESB 4
=> yes, it is ServiceMix, but there is some modification ;-)

- JBoss ESB 4

- Spagic 3.3.0
=> As they became the main Eclipse SOA movement, I think this also a considerable one

- JOnAS + Camel/Fuse Mediation Router/Talend Integration Factory
=> As this solution address all the keypoints which define what an ESB is + its serious production references in
government/telco sector, this alternative shall be also considered

- Petals ESB 3.1.3

- (Similar to JOnAS + Camel) Camel + a proven JEE/OSGi runtime which can be considered as an ESB ;-)...

thanks for your work...

Alp Şehiç said...

Sorry, forgotten;

- Apache Tuscany Java 2.0-Beta3

Asankha said...

Alp, Thanks for the list of other ESBs we should look at..

- We will include Fuse ESB for sure

- We will try to include Talend ESB, JBoss ESB and Petals ESB if we can get them to work in this round.

- We will have to postpone the other options for a future round

Federica said...

And you also should include GreenVulcano ESB.
Thanks.
Federica

Asankha said...

Federica, thanks for the option, I had not heard about it before..

The Wiki being in Italian is not that good, especially since I was looking for some easy examples (e.g. a SOAP proxy service) to get started.. Do you know any English links to get started? Or, would you be able to help us develop the use cases we want to test?

Paucus3d said...

I'd like to see Open ESB in there...