Thursday, November 7, 2013

Nausea with Tomcat after biting too much?

About an year back I wrote a post "Does Tomcat bite more than it can chew" about how Tomcat accepts more TCP connections that it can handle, and resets them later on. I also wrote a follow-up post "How to stop biting, when you cant chew more.." and explained how the UltraESB does it using Apache HttpComponents NIO underneath. I raised the issue on the Tomcat mailing list too, but found that most of the folks over there did not like what I had to say, and refused to change Tomcat.

This week one of our clients was running a load test using the UltraESB in front of a Tomcat instance, and wanted to measure the performance of the UltraESB to use WS-Security signatures over a Tomcat service. However, at 1000 concurrent users, the UltraESB started to report an XML parse exception with a "Premature end of file" for the responses.

After detailed investigations to reproduce the issue in our environment, we found the culprit to be Tomcat. However, proof of the issue was not that simple, and we used a standalone Java client to issue fixed sized asynchronous HTTP post requests to Tomcat, and measured the size of the responses received. The message size used was 5120 bytes, and thus a response of equal size was expected.

However, under load, Tomcat accepted a 5K request, but replied with an HTTP status code of 200, but without any payload. It sent the Content-Length header value as "0". Here is what Wireshark reported:

Here are the response headers that Tomcat sent back:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Length: 0
Date: Thu, 07 Nov 2013 10:15:14 GMT
Connection: close
It's also interesting to note that Tomcat sometimes returns error responses which states that an internal error occurred; but with a HTTP status code of 200, instead of the expected 500 error code. Here is an example of such a payload returned:
<html><head><title>Apache Tomcat/7.0.28 - Error report</title><style><!--H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;} {color : black;}HR {color : #525D76;}--></style> </head><body><h1>HTTP Status 500 - </h1><HR size="1" noshade="noshade"><p><b>type</b> Status report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The server encountered an internal error () that prevented it from fulfilling this request.</u></p><HR size="1" noshade="noshade"><h3>Apache Tomcat/7.0.28</h3></body></html>
It would have been so much better for many, if Tomcat did not bite fast, when it cannot chew!

Monday, October 14, 2013

ESB Performance Testing - Round 7 Published!

We have just published the results for the long awaited Round 7 - of ESB performance benchmarking. This round compares 4 of the leading free and open source ESBs, Mule CE 3.4.0, Talend ESB SE 5.3.1, WSO2 ESB 4.7.0 and the UltraESB 2.0.0
Although at first glance the above image may indicate that the WSO2 ESB and the UltraESB have very similar performance characteristics, the Devil is indeed in the detail.

During the testing we discovered several issues with the previously published article Round 6.5 from WSO2, including a severe response corruption for messages over 16,384 bytes when the default pass-through transport is being used. However, what's most surprising is that this issue remains in versions 4.6.0 and 4.7.0 of the WSO2 ESB, as well as the latest Milestone 4 of the soon to be released version 4.8.0.

Sadly, the WSO2 engineers conducting the Round 6.5 also failed to notice a complete failure of all of the XSLT test cases, but nevertheless published numbers obtained for the failed test cases as high performance numbers over the other ESBs.

Read about these and other flaws of the Round 6.5 in the article Why the Round 6.5 results published by WSO2 is flawed

Tuesday, August 27, 2013

UltraESB 2.0.0 is released!

Yeah, its been quite sometime since we did a major release of the UltraESB! Although the last GA release was extremely stable and widely used, we've made significant improvements for the long awaited 2.0.0 release.

Since the beginning, we have been privileged to work closely with some of the best architects actually using our software in production. The suggestions and improvements proposed by them helped us implement key features that set the UltraESB apart. These were real requirements that meant a lot for large scale production deployments running 24 x 7 x 365 with zero downtime. For the 2.0.0 release, we've implemented many features that have been similarly inspired by the requirements of a top Fortune #10 company. These include a built-in metrics management and alerting functionality, and a programmatic instance management capability - coupled with annotation driven extensibility, especially for custom message interception at various stages.

Ruwan has already blogged about some of the metrics graphs generated by the UConsole in the 2.0.0 release, and you could find more documentation at The 2.0.0 release also migrates to Apache HttpComponents/Core 4.2.4 and Spring framework / Spring security 3.1.1 and introduces a concept of a 'Deployment Unit' that can be deployed/updated or removed at runtime. Deployment units now has the ability to re-load dependency JAR files into the runtime as well, and support atomic updates as before, which allows all changes to be deployed into ESB nodes servicing client requests - without causing inconsistencies.

The Mediation API has also been cleaned up and streamlined with the introduction of 'Support Interfaces'. While being backwards compatible, the changes will help users utilize the UltraESB mediation API more easily.

Check out the full news release here, and refer to the documentation to get started!

Saturday, February 23, 2013

What drives talent out?

A few days back I saw the famous presentation from Netflix about its culture, and three interesting slides from it was about what drives talent out of a company.

Today I came across a poster created by හිච්චි කොලුවා which shows another aspect from a more Sri Lankan point of view, and it says its the 'තිත්ත ඇත්ත' .. interesting points to think about!

PS: If you aim the moon, believe that you WILL get there. If a planes' limit is the clouds.. get onto a rocket.. do not deviate from achieving your goal!