Blog

The IETF HTTPbis Working Group recently called for expressions of interest in the development of the HTTP/2.0 protocol, with SPDY being one of the candidates to use as a basis.

As a HTTP server and an early implementer of the SPDY protocol, the Jetty project certainly has an interest in HTTP/2.0 and this blog contains the text of our response below.  However it is also very interest to have a read all of the expressions of interest received from industry heavy hitters like:

• Jetty
• Firefox
• Facebook
• Microsoft
• Google
• F5
• haproxy
• Squid
• Akamai
• Alibaba
• curl
• varnish

Reading through these and the thousands of replies, it is clear that there is significant interest and some momentum towards replacing HTTP/1.1, but that the solution is not quite as simple as s/SPDY/HTTP/2.0/.

There is a lot of heat around the suggestion of mandatory encryption (even though no proposal actually has mandatory encryption), and it looks like there is a big divide in the community.

I also think that many of the concerns of the intermediaries (F5, haproxy, squid) are not being well addressed.  This is a mistake often made in previous protocol iterations and we would be well served by taking the time to listen and understand their concerns.  Even simple features such as easy access to host headers for quick routing may have significant benefits.

The Jetty Expression of Interest in HTTP/2.0

(see also the original post and responses)

I’m the project leader of the Jetty project (http://eclipse.org/jetty) and am making this initial response on behalf of the project and not for eclipse as a whole (although we will solicit further feedback from other projects within eclipse). I work for Webtide|Intalio who sell support services around Jetty.

Jetty is an open source server and client written in java that supports the 3.0 Servlet API, HTTP/1.1, Websocket 1.0 and SPDY v3. We have a reasonable market share of java servers (>10% < 30%) and are deployed on everything from tiny embedded servers to very large deployments with over 100k connections per server.

The Jetty project is very interested in the development and standardisation of HTTP/2.0 and intend to be contributors to the WG and early implementers. We are well acquainted with the limitations of HTTP/1.1 and have a desire to see the problems of pipelining and multiple connections (>2) resolved.

The Jetty project SPDY effort is lead by Simone Bordet and it has implemented SPDY v3 with flow control and push. This is available in the main releases of our Jetty-7 and jetty-8 servers (we also have a java SPDY client). The project has also provided an extension to the JVM to implement the TLS NPN extension needed by SPDY, and we understand that several other java SPDY implementations are using this.

We chose SPDY to implement rather than any other HTTP/2.0 proposal mainly because of the support available in deployed browsers, so that we can achieve real world feedback. However, we were also encouraged in our adoption of SPDY by the open, methodical and congenial approach of the SPDY project at Google (not always our experience with projects at Google or elsewhere).

We definitely see the potential of SPDY and it is already being used by some sites. However we still lack the feedback from widespread deployment (it is early days) or from large deployments. We are actively seeking significant sites who are interested in working with us to deploy SPDY.

There are several key features of SPDY that we see as promising:

Header compression greatly improves data density. In our use of Ajax and Comet over HTTP/1.1 we have often hit scalability limits due to network saturation with very poor data density of small massages in large HTTP framing. While websocket is doing a lot to resolve this, we are hoping that SPDY will provide improvement without the need to redevelop applications.

Multiplexing of multiple streams over a single connection is also a good development. Reducing the number of connections that the server must handle is key to scalability, specially as modern HTTP browsers are now exceeding the 2 connection limit. The ability to send out of order responses is good and we also suspect that receiving messages from a single client over a single connection may help reduce some of the non deterministic behaviours that can develop as multiple connections from the same client set cookies or update session state. It will also avoid the issue of load balancers directing connections from the same client to different nodes in a cluster. We recognise the additional cost of multiplexing (extra copies and flow control), but currently believe that it is worth the effort.

We see the potential of server push for content, but are struggling with the lack of meta data knowledge available to know what to push and when. We are currently working on strategies that use the referrer header to identify associated resources that can be pushed together. We also check for if-modified-since headers as an indication that associated content may already be cached and thus a push is not required. We see the challenge of push as not being the protocol to send the content, but in working out the standards for meta data, cache control etc so that we know what to push and when.

We have not yet implemented websocket over SPDY, but do intend to do so if it is supported by the browsers. We see a lot of similarities in the base framing of these two protocols and would hope that eventually only one would need to be well supported.

We are bit ambivalent about the use of NPN and TLS only connections. There is a question to be asked regarding if we should be sending any web content in the clear, and how intermediaries should be able to (or not) filter/inspect/mutate content. However, I personally feel that this is essentially a non technical issue and we should not use a protocol to push any particular agenda. The RTT argument for not supporting in the clear connections is weak as there are several easy technical solutions available. Furthermore, the lack of support for NPN is a barrier to adoption (albeit one that we have broken down for some JVMs at least). Debugging over TLS is and will always be difficult. We would like to HTTP/2.0 support standardised non encrypted connections (at least from TLS offload to server). If a higher level debate determines that web deployments only accept TLS connections, then we are fine with that non technical determination.

I repeat that we selected SPDY to implement because it’s availability in the browsers and not as the result of a technical review against other alternatives. However we are generally pleased with the direction and results obtained so far and look forward to gaining more experience and feedback as it is more widely deployed.

However we do recognise that much of the “goodness” of SPDY can be provided by the other proposals. I’m particularly interested in the HTTP/speed/mobility’s use of websockets as it’s framing layer (as that addressed the concern I raised above). But we currently do not have any plans to implement the alternatives, mainly because of resource limitations and lack of browser support. So currently we are advocates of the SPDY approach in the starship troopers sense: ie we support SPDY until it is dead or we find something better. Of course Jetty is an open platform and we would really welcome and assist any contributors who would like to build on our websocket support to implement HTTP/SM.

We believe that there is high demand for a significant improvement over HTTP/1.1 and that the environment is ripe for a rapid rollout of an alternative/improved protocol and expect that HTTP/1.1 can quickly be replaced. Because of this, we have begun development of jetty-9 which replaces the HTTP protocol centric architecture of jetty-7/8 with something that is much better suited to multiple protocols and multiplexed HTTP semantics. SPDY, Websocket and HTTP/1.1 are true peers in Jetty-9 rather than the newer protocols being implemented as HTTP facades. We believe that jetty-9 will be the ideal platform on which to develop and deploy HTTP/2.0 and we invite anybody with an interest to come contribute to the project.

Jetty has long had a rich set of JMX mbeans that give very detailed status, configuration and control over the server and applications, which can now simply be accessed with the JMiniX web console:

The usability of JMX has been somewhat let down due to a lack of quality JMX management consoles.  JConsole and JVirtualVM do give good access to MBeans, but they rely on a RMI connection which can be tricky to setup to a remote machine.   JMiniX avoids the RMI by allowing access to the MBeans via a servlet you can add to your webapplication.

The instructions were straight forward to follow and the steps were simply:

1. Add dependency to your pom
2. Add a repository to your pom (bummer – needs restlet.org which is not in maven central – if it was I’d consider adding JMiniX to our released test webapp)
3. Define the servlet in your web.xml
4. Build and run!

You can see by the screen shot above that the console gives a nice rendering of the available mbeans from the JVM and Jetty (and cometd if running). Attributes can be viewed and updated, and operations can be called – all the normal stuff.   It only gives direct mbean access and does not provide any higher level management functions, but this is not a big problem if the mbeans are well designed and self documented.

Also if you wanted to develop more advanced management functions, then the restful nature of JMiniX should make this fairly straight forward.  For example attributes can be retrieved with simple requests like:

http://localhost:8080/jmx/servers/0/domains/org.eclipse.jetty.server/
mbeans/type=server,id=0/attributes/startupTime/

That returns JSON like:

{"value":"1339059648877","label":"startupTime"}

JMiniX looks like a great tool to improve the management of your servers and applications and to leverage the value already built into the Jetty JMX mbeans.

We had been working on a similar effort for restful access to JMX, but JMiniX is more advanced.  It does lack some of the features that we had been working on like aggregate access to repeated attributes, but considering the state of JMiniX, we may consider contributing those features to that project instead.

One of my pet peeves is misleading benchmarks, as discussed in my Lies, Damned Lies and Benchmarks blog.  Recently there has been a bit of interest in Vert.x, some of it resulting from apparently good benchmark results against node.js. The author gave a disclaimer that the tests were non-rigorous and just for fun, but they have already lead some people to ask if Jetty can scale like Vert.x.

I know absolutely nothing about Vert.x, but I do know that their benchmark is next to useless to demonstrate any kind of scalability of a server.  So I’d like to analyse their benchmarks and compare them to how we benchmark jetty/cometd to try to give some understanding about how benchmarks should be designed and interpreted.

The benchmark

The vert.x benchmark uses 6 clients, each with 10 connections, each with up to 2000 pipelines HTTP requests for a trivial 200 OK or tiny static file. The tests were run for a minute and the average request rate was taken. So lets break this down:

6 Clients of 10 connections!

However you look at this (6 users each with a browser with 10 connections, or 60 individual users), 6 or 60 users does not represent any significant scalability.  We benchmark jetty/comet with 10,000 to 200,000 connections and have production sites that run with similar numbers.

Testing 60 connections does not tell you anything about scalability. So why do so many benchmarks get performed on low numbers of connections?  It’s because it is really really hard to generate realistic load for hundreds of thousands of connections.  To do so, we use the jetty asynchronous HTTP client, which has been designed specifically for this purpose, and we still need to use multiple load generating machines to achieve high numbers of connections.

2000 pipelined requests!

Really? HTTP pipelining is not turned on by default in most web browsers, and even if it was, I cannot think of any realistic application that would be generate 2000 requests in a pipeline. Why is this important?  Because with pipelined requests a server that does:

byte[] buffer = new byte[8192];
socket.getInputStream().read(buffer);

will read many requests into that buffer in a single read.  A trivial HTTP request is a few 10s of bytes (and I’m guessing they didn’t send any of the verbose complex headers that real browsers do), so the vert.x benchmark would be reading 30 or more requests on each read.  Thus this benchmark is not really testing any IO performance, but simply how fast they can iterate over a buffer and parse simple requests. At best it is telling you about the latency in their parsing and request handling.

Handling reads is not the hard part of scaling IO.  It is handling the idle pauses between the reads that is difficult.  It is these idle periods that almost all real load profiles have that requires the server to carefully allocate resources so that idle connections do not consume resources that could be better used by non idle connections.    2000 connections each with 6 pipelined requests would be more realistic, or better yet 20000 connections with 6 requests that are sent with 10ms delays between them.

Trivial 200 OK or Tiny static resource

Creating a scalable server for non trivial applications is all about trying to ensure that maximal resources are applied to performing real business logic in preparing dynamic responses.   If all the responses are trivial or static, then the server is free to be more wasteful.  Worse still for realistic benchmarks, a trivial response generation can probably be in-lined by the hotspot compiler is a way that no real application ever could be.

Run for a minute

A minute is insufficient time for a JVM to achieve steady state.  For the first few minutes of a run the Hotspot JIT compiler will be using CPU to analyse and compile code. A trivial application might be able to be hotspot compiled in a minute, but any reasonably complex server/application is going to take much longer.  Try watching your application with jvisualvm and watch the perm generation continue to grow for many minutes while more and more classes are compiled. Only after the JVM has warmed up your application and CPU is no longer being used to compile, can any meaningful results be obtained.

The other big killer of performance are full garbage collections that can stop the entire VM for many seconds.  Running fast for 60 seconds does not do you much good if a second later you pause for 10s while collecting the garbage from those fast 60 seconds.

Benchmark result need to be reported for steady state over longer periods of time and you need to consider GC performance.  The jetty/cometd benchmark tools specifically measures and reports both JIT and GC actions during the benchmark runs and we can perform many benchmark runs in the same JVM.  Below is example output showing that for a 30s run some JIT was still performed, so the VM is not fully warmed up yet:

Statistics Started at Mon Jun 21 15:50:58 UTC 2010
Operative System: Linux 2.6.32-305-ec2 amd64
JVM : Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server
VM runtime 16.3-b01 1.6.0_20-b02
Processors: 2
System Memory: 93.82409% used of 7.5002174 GiB
Used Heap Size: 2453.7236 MiB
Max Heap Size: 5895.0 MiB
Young Generation Heap Size: 2823.0 MiB
- - - - - - - - - - - - - - - - - - - -
Testing 2500 clients in 100 rooms
Sending 3000 batches of 1x50B messages every 8000µs
- - - - - - - - - - - - - - - - - - - -
Statistics Ended at Mon Jun 21 15:51:29 UTC 2010
Elapsed time: 30164 ms
        Time in JIT compilation: 12 ms
        Time in Young Generation GC: 0 ms (0 collections)
        Time in Old Generation GC: 0 ms (0 collections)
Garbage Generated in Young Generation: 1848.7974 MiB
Garbage Generated in Survivor Generation: 0.0 MiB
Garbage Generated in Old Generation: 0.0 MiB
Average CPU Load: 109.96191/200

Conclusion

I’m sure the vert.x guys had every good intent when doing their micro-benchmark, and it may well be that vert.x scales really well.  However I wish that when developers consider benchmarking servers, that instead of thinking: “let’s send a lot of requests at it”, that their first thought was “let’s open a lot of connections at it”.  Better yet, a benchmark (micro or otherwise) should be modelled on some real application and the load that it might generate.

The jetty/cometd benchmark is of a real chat application, that really works and has real features like member lists, private messages etc.  Thus the results that we achieve in benchmarks are able to be reproduced by real applications in production.