Author: admin

The IETF HTTPbis Working Group recently called for expressions of interest in the development of the HTTP/2.0 protocol, with SPDY being one of the candidates to use as a basis.

As a HTTP server and an early implementer of the SPDY protocol, the Jetty project certainly has an interest in HTTP/2.0 and this blog contains the text of our response below.  However it is also very interest to have a read all of the expressions of interest received from industry heavy hitters like:

• Jetty
• Firefox
• Facebook
• Microsoft
• Google
• F5
• haproxy
• Squid
• Akamai
• Alibaba
• curl
• varnish

Reading through these and the thousands of replies, it is clear that there is significant interest and some momentum towards replacing HTTP/1.1, but that the solution is not quite as simple as s/SPDY/HTTP/2.0/.

There is a lot of heat around the suggestion of mandatory encryption (even though no proposal actually has mandatory encryption), and it looks like there is a big divide in the community.

I also think that many of the concerns of the intermediaries (F5, haproxy, squid) are not being well addressed.  This is a mistake often made in previous protocol iterations and we would be well served by taking the time to listen and understand their concerns.  Even simple features such as easy access to host headers for quick routing may have significant benefits.

The Jetty Expression of Interest in HTTP/2.0

(see also the original post and responses)

I’m the project leader of the Jetty project (http://eclipse.org/jetty) and am making this initial response on behalf of the project and not for eclipse as a whole (although we will solicit further feedback from other projects within eclipse). I work for Webtide|Intalio who sell support services around Jetty.

Jetty is an open source server and client written in java that supports the 3.0 Servlet API, HTTP/1.1, Websocket 1.0 and SPDY v3. We have a reasonable market share of java servers (>10% < 30%) and are deployed on everything from tiny embedded servers to very large deployments with over 100k connections per server.

The Jetty project is very interested in the development and standardisation of HTTP/2.0 and intend to be contributors to the WG and early implementers. We are well acquainted with the limitations of HTTP/1.1 and have a desire to see the problems of pipelining and multiple connections (>2) resolved.

The Jetty project SPDY effort is lead by Simone Bordet and it has implemented SPDY v3 with flow control and push. This is available in the main releases of our Jetty-7 and jetty-8 servers (we also have a java SPDY client). The project has also provided an extension to the JVM to implement the TLS NPN extension needed by SPDY, and we understand that several other java SPDY implementations are using this.

We chose SPDY to implement rather than any other HTTP/2.0 proposal mainly because of the support available in deployed browsers, so that we can achieve real world feedback. However, we were also encouraged in our adoption of SPDY by the open, methodical and congenial approach of the SPDY project at Google (not always our experience with projects at Google or elsewhere).

We definitely see the potential of SPDY and it is already being used by some sites. However we still lack the feedback from widespread deployment (it is early days) or from large deployments. We are actively seeking significant sites who are interested in working with us to deploy SPDY.

There are several key features of SPDY that we see as promising:

Header compression greatly improves data density. In our use of Ajax and Comet over HTTP/1.1 we have often hit scalability limits due to network saturation with very poor data density of small massages in large HTTP framing. While websocket is doing a lot to resolve this, we are hoping that SPDY will provide improvement without the need to redevelop applications.

Multiplexing of multiple streams over a single connection is also a good development. Reducing the number of connections that the server must handle is key to scalability, specially as modern HTTP browsers are now exceeding the 2 connection limit. The ability to send out of order responses is good and we also suspect that receiving messages from a single client over a single connection may help reduce some of the non deterministic behaviours that can develop as multiple connections from the same client set cookies or update session state. It will also avoid the issue of load balancers directing connections from the same client to different nodes in a cluster. We recognise the additional cost of multiplexing (extra copies and flow control), but currently believe that it is worth the effort.

We see the potential of server push for content, but are struggling with the lack of meta data knowledge available to know what to push and when. We are currently working on strategies that use the referrer header to identify associated resources that can be pushed together. We also check for if-modified-since headers as an indication that associated content may already be cached and thus a push is not required. We see the challenge of push as not being the protocol to send the content, but in working out the standards for meta data, cache control etc so that we know what to push and when.

We have not yet implemented websocket over SPDY, but do intend to do so if it is supported by the browsers. We see a lot of similarities in the base framing of these two protocols and would hope that eventually only one would need to be well supported.

We are bit ambivalent about the use of NPN and TLS only connections. There is a question to be asked regarding if we should be sending any web content in the clear, and how intermediaries should be able to (or not) filter/inspect/mutate content. However, I personally feel that this is essentially a non technical issue and we should not use a protocol to push any particular agenda. The RTT argument for not supporting in the clear connections is weak as there are several easy technical solutions available. Furthermore, the lack of support for NPN is a barrier to adoption (albeit one that we have broken down for some JVMs at least). Debugging over TLS is and will always be difficult. We would like to HTTP/2.0 support standardised non encrypted connections (at least from TLS offload to server). If a higher level debate determines that web deployments only accept TLS connections, then we are fine with that non technical determination.

I repeat that we selected SPDY to implement because it’s availability in the browsers and not as the result of a technical review against other alternatives. However we are generally pleased with the direction and results obtained so far and look forward to gaining more experience and feedback as it is more widely deployed.

However we do recognise that much of the “goodness” of SPDY can be provided by the other proposals. I’m particularly interested in the HTTP/speed/mobility’s use of websockets as it’s framing layer (as that addressed the concern I raised above). But we currently do not have any plans to implement the alternatives, mainly because of resource limitations and lack of browser support. So currently we are advocates of the SPDY approach in the starship troopers sense: ie we support SPDY until it is dead or we find something better. Of course Jetty is an open platform and we would really welcome and assist any contributors who would like to build on our websocket support to implement HTTP/SM.

We believe that there is high demand for a significant improvement over HTTP/1.1 and that the environment is ripe for a rapid rollout of an alternative/improved protocol and expect that HTTP/1.1 can quickly be replaced. Because of this, we have begun development of jetty-9 which replaces the HTTP protocol centric architecture of jetty-7/8 with something that is much better suited to multiple protocols and multiplexed HTTP semantics. SPDY, Websocket and HTTP/1.1 are true peers in Jetty-9 rather than the newer protocols being implemented as HTTP facades. We believe that jetty-9 will be the ideal platform on which to develop and deploy HTTP/2.0 and we invite anybody with an interest to come contribute to the project.

Jetty has long had a rich set of JMX mbeans that give very detailed status, configuration and control over the server and applications, which can now simply be accessed with the JMiniX web console:

The usability of JMX has been somewhat let down due to a lack of quality JMX management consoles.  JConsole and JVirtualVM do give good access to MBeans, but they rely on a RMI connection which can be tricky to setup to a remote machine.   JMiniX avoids the RMI by allowing access to the MBeans via a servlet you can add to your webapplication.

The instructions were straight forward to follow and the steps were simply:

1. Add dependency to your pom
2. Add a repository to your pom (bummer – needs restlet.org which is not in maven central – if it was I’d consider adding JMiniX to our released test webapp)
3. Define the servlet in your web.xml
4. Build and run!

You can see by the screen shot above that the console gives a nice rendering of the available mbeans from the JVM and Jetty (and cometd if running). Attributes can be viewed and updated, and operations can be called – all the normal stuff.   It only gives direct mbean access and does not provide any higher level management functions, but this is not a big problem if the mbeans are well designed and self documented.

Also if you wanted to develop more advanced management functions, then the restful nature of JMiniX should make this fairly straight forward.  For example attributes can be retrieved with simple requests like:

http://localhost:8080/jmx/servers/0/domains/org.eclipse.jetty.server/
mbeans/type=server,id=0/attributes/startupTime/

That returns JSON like:

{"value":"1339059648877","label":"startupTime"}

JMiniX looks like a great tool to improve the management of your servers and applications and to leverage the value already built into the Jetty JMX mbeans.

We had been working on a similar effort for restful access to JMX, but JMiniX is more advanced.  It does lack some of the features that we had been working on like aggregate access to repeated attributes, but considering the state of JMiniX, we may consider contributing those features to that project instead.

One of my pet peeves is misleading benchmarks, as discussed in my Lies, Damned Lies and Benchmarks blog.  Recently there has been a bit of interest in Vert.x, some of it resulting from apparently good benchmark results against node.js. The author gave a disclaimer that the tests were non-rigorous and just for fun, but they have already lead some people to ask if Jetty can scale like Vert.x.

I know absolutely nothing about Vert.x, but I do know that their benchmark is next to useless to demonstrate any kind of scalability of a server.  So I’d like to analyse their benchmarks and compare them to how we benchmark jetty/cometd to try to give some understanding about how benchmarks should be designed and interpreted.

The benchmark

The vert.x benchmark uses 6 clients, each with 10 connections, each with up to 2000 pipelines HTTP requests for a trivial 200 OK or tiny static file. The tests were run for a minute and the average request rate was taken. So lets break this down:

6 Clients of 10 connections!

However you look at this (6 users each with a browser with 10 connections, or 60 individual users), 6 or 60 users does not represent any significant scalability.  We benchmark jetty/comet with 10,000 to 200,000 connections and have production sites that run with similar numbers.

Testing 60 connections does not tell you anything about scalability. So why do so many benchmarks get performed on low numbers of connections?  It’s because it is really really hard to generate realistic load for hundreds of thousands of connections.  To do so, we use the jetty asynchronous HTTP client, which has been designed specifically for this purpose, and we still need to use multiple load generating machines to achieve high numbers of connections.

2000 pipelined requests!

Really? HTTP pipelining is not turned on by default in most web browsers, and even if it was, I cannot think of any realistic application that would be generate 2000 requests in a pipeline. Why is this important?  Because with pipelined requests a server that does:

byte[] buffer = new byte[8192];
socket.getInputStream().read(buffer);

will read many requests into that buffer in a single read.  A trivial HTTP request is a few 10s of bytes (and I’m guessing they didn’t send any of the verbose complex headers that real browsers do), so the vert.x benchmark would be reading 30 or more requests on each read.  Thus this benchmark is not really testing any IO performance, but simply how fast they can iterate over a buffer and parse simple requests. At best it is telling you about the latency in their parsing and request handling.

Handling reads is not the hard part of scaling IO.  It is handling the idle pauses between the reads that is difficult.  It is these idle periods that almost all real load profiles have that requires the server to carefully allocate resources so that idle connections do not consume resources that could be better used by non idle connections.    2000 connections each with 6 pipelined requests would be more realistic, or better yet 20000 connections with 6 requests that are sent with 10ms delays between them.

Trivial 200 OK or Tiny static resource

Creating a scalable server for non trivial applications is all about trying to ensure that maximal resources are applied to performing real business logic in preparing dynamic responses.   If all the responses are trivial or static, then the server is free to be more wasteful.  Worse still for realistic benchmarks, a trivial response generation can probably be in-lined by the hotspot compiler is a way that no real application ever could be.

Run for a minute

A minute is insufficient time for a JVM to achieve steady state.  For the first few minutes of a run the Hotspot JIT compiler will be using CPU to analyse and compile code. A trivial application might be able to be hotspot compiled in a minute, but any reasonably complex server/application is going to take much longer.  Try watching your application with jvisualvm and watch the perm generation continue to grow for many minutes while more and more classes are compiled. Only after the JVM has warmed up your application and CPU is no longer being used to compile, can any meaningful results be obtained.

The other big killer of performance are full garbage collections that can stop the entire VM for many seconds.  Running fast for 60 seconds does not do you much good if a second later you pause for 10s while collecting the garbage from those fast 60 seconds.

Benchmark result need to be reported for steady state over longer periods of time and you need to consider GC performance.  The jetty/cometd benchmark tools specifically measures and reports both JIT and GC actions during the benchmark runs and we can perform many benchmark runs in the same JVM.  Below is example output showing that for a 30s run some JIT was still performed, so the VM is not fully warmed up yet:

Statistics Started at Mon Jun 21 15:50:58 UTC 2010
Operative System: Linux 2.6.32-305-ec2 amd64
JVM : Sun Microsystems Inc. Java HotSpot(TM) 64-Bit Server
VM runtime 16.3-b01 1.6.0_20-b02
Processors: 2
System Memory: 93.82409% used of 7.5002174 GiB
Used Heap Size: 2453.7236 MiB
Max Heap Size: 5895.0 MiB
Young Generation Heap Size: 2823.0 MiB
- - - - - - - - - - - - - - - - - - - -
Testing 2500 clients in 100 rooms
Sending 3000 batches of 1x50B messages every 8000µs
- - - - - - - - - - - - - - - - - - - -
Statistics Ended at Mon Jun 21 15:51:29 UTC 2010
Elapsed time: 30164 ms
        Time in JIT compilation: 12 ms
        Time in Young Generation GC: 0 ms (0 collections)
        Time in Old Generation GC: 0 ms (0 collections)
Garbage Generated in Young Generation: 1848.7974 MiB
Garbage Generated in Survivor Generation: 0.0 MiB
Garbage Generated in Old Generation: 0.0 MiB
Average CPU Load: 109.96191/200

Conclusion

I’m sure the vert.x guys had every good intent when doing their micro-benchmark, and it may well be that vert.x scales really well.  However I wish that when developers consider benchmarking servers, that instead of thinking: “let’s send a lot of requests at it”, that their first thought was “let’s open a lot of connections at it”.  Better yet, a benchmark (micro or otherwise) should be modelled on some real application and the load that it might generate.

The jetty/cometd benchmark is of a real chat application, that really works and has real features like member lists, private messages etc.  Thus the results that we achieve in benchmarks are able to be reproduced by real applications in production.

 
 
 
 
 

With the release of Jetty 7.5.0 and the latest draft 13 of the WebSocket protocol, the API for the client has be re-factored a little since my last blog on WebSocket: Server, Client and Load Test.

WebSocketClientFactory

When creating many instances of the java WebSocketClient, there is much that can be shared between multiple instances: buffer pools, thread pools and NIO selectors.  Thus the client API has been updated to use a factory pattern, where the factory can hold the configuration and instances of the common infrastructure:

WebSocketClientFactory factory = new WebSocketClientFactory();
factory.setBufferSize(4096);
factory.start();

WebSocketClient

Once the WebSocketClientFactory is started, WebSocketClient instances can be created and configured:

WebSocketClient client = factory.newWebSocketClient();
client.setMaxIdleTime(30000);
client.setMaxTextMessageSize(1024);
client.setProtocol("chat");

The WebSocketClient does not need to be started and the configuration set is copied to the connection instances as they are opened.

WebSocketClient.open(…)

A websocket connection can be created from a WebSocketClient by calling open and passing the URI and the websocket instance that will handle the call backs (eg onOpen, onMessage etc.):

Future future = client.open(uri,mywebsocket);
WebSocket.Connection connection = future.get(10,TimeUnit.SECONDS);

The open call returns a Future to the WebSocket.Connection.  Like the NIO.2 API in JDK7, calling get with a timeout imposes a connect time on the connection attempt and the connection will be aborted if the get times out.   If the connection is successful, the connection returned by the get is the same object passed to the WebSocket.onOpen(Connection) callback, so it may be access and used in either way.

WebSocket.Connection

The connection instance accessed via the onOpen callback or Future.get() is used to send messages and also to configure the connection:

connection.setMaxIdleTime(10000);
connection.setMaxTextMessageSize(2*1024);
connection.setMaxBinaryMessageSize(64*1024);

The  maximum message sizes are used to control how large messages can grow when they are being aggregated from multiple websocket frames.  Small max message sizes protect a server against DOS attack.

The Jetty Overlay Deployer allows multiple WAR files to be overlayed so that a web application can be customised, configured and deployed without the need to unpack, modify and repack the WAR file. This has the benefits of:

• The WAR file may be kept immutable, even signed, so that it is clear which version has been deployed.
• All modifications made to customise/configure the web application are kept in a separate wars and thus are easily identifiable for review and migration to new versions.
• A parameterised template overlay can be created that contains common customisations and configuration that apply to many instances of the web application (eg for multi-tenant deployment).
• Because the layered deployment clearly identifies the common and instance specific components, then Jetty is able to share classloaders and static resource caches for the template, greatly reducing the memory footprint of multiple instances

This blog is a tutorial of how to configure Jetty to use the Overlay deployer, and how to deploy multiple instances of the JTrac web application.

Overview

The customisation, configuration and deployment a web application bundled as a WAR file frequently includes some or all of:

• Editing the WEB-INF/web.xml file to set init parameters, add filters/servlets or to configure JNDI resources.
• Editing other application specific configuration files in WEB-INF/*
• Editing container specific configuration files in WEB-INF/* (eg jetty-web.xml or jboss-web.xml)
• Adding/modifying static content such as images and css to style/theme the webapplication
• Adding jars to the container classpath for Datasource and other resources
• Modifying the container configuration to provide JNDI resources

The result is that the customisations and configurations are blended into both the container and the WAR file. If either the container or the base WAR file are upgraded to a new version, it can be a very difficult and error prone task to identify all the changes that have been made and to reapply them to a new version.

Overlays

To solve the problems highlighted above, jetty 7.4 introduces WAR overlays (which are a concept borrowed from the maven war plugin). An overlay is basically just another WAR files, whose contents are merged on top of the original WAR so that files may be added or replaced.

However,  jetty overlays also allow mixin fragments of web.xml, so the configuration can be modified without being replaced

Jtrac Overlay Example

The jtrac issue tracking webapplication is a good example of a typical web application, as it uses the usual suspects of libs: spring, hibernate, dom4j, commons-*, wicket, etc. So I’ve used it as the basis of this example.

The files for this demonstration are available in overlays-demo.tar.gz. This could be expanded on top of the jetty distribution, but for this tutorial we will expand it to /tmp and install the components step by step:

cd /tmp
wget http://webtide.intalio.com/wp-content/uploads/2011/05/overlays-demo.tar.gz
tar xfvz overlays-demo.tar.gz
export OVERLAYS=/tmp/overlays

Configuring Jetty for Overlays

Overlays support is included in jetty distributions from 7.4.1-SNAPSHOT onwards, so you can download a distribution from oss.sonatype.org or maven central(once 7.4.1 is released) and unpack into a directory.    The start.ini file then needs to be edited so that it includes the overlay option and configuration file.   The resulting file should look like:

OPTIONS=Server,jsp,jmx,resources,websocket,ext,overlay
etc/jetty.xml
etc/jetty-deploy.xml
etc/jetty-overlay.xml

The smarts of this are in etc/jetty-deploy.xml files, which installs the OverlayedAppProvider into the DeploymentManager. Jetty can then be started normally:

java -jar start.jar

Jetty will now be listening on port 8080, but with no webapp deployed.   The rest of the tutorial should be conducted in another window with the JETTY_HOME environment set to the jetty distribution directory.

Installing the WebApp

The WAR file for this demo can be downloaded and deployed using the following commands, which essentially downloads and extracts the WAR file to the $JETTY_HOME/overlays/webapps directory

cd /tmp
wget -O jtrac.zip http://sourceforge.net/projects/j-trac/files/jtrac/2.1.0/jtrac-2.1.0.zip/download
jar xfv jtrac.zip jtrac/jtrac.war
mv jtrac/jtrac.war $JETTY_HOME/overlays/webapps

When you have run these commands (or equivalent), you will see in the jetty server window a message saying that the OverlayedAppProvider has extracted and loaded the war file:

2011-05-06 10:31:54.678:INFO:OverlayedAppProvider:Extract jar:file:/tmp/jetty-distribution-7.4.1-SNAPSHOT/overlays/webapps/jtrac-2.1.0.war!/ to /tmp/jtrac-2.1.0_236811420856825222.extract
2011-05-06 10:31:55.235:INFO:OverlayedAppProvider:loaded jtrac-2.1.0@1304641914666

Unlike the normal webapps dir, loading a war file from the overlays/webapp dir does not deploy the webapplication.  It simply makes it available to be used as the basis for templates and overlays.

Installing a Template Overlay

A template overlay is a WAR structured directory/archive that contains just the files that have been added or modified to customize/configure the webapplication for all instances that will be deployed.

The demo template can be installed from the downloaded files with the command:

mv $OVERLAYS/jtracTemplate=jtrac-2.1.0 $JETTY_HOME/overlays/templates/

In the Jetty server window, you should see the template loaded with a message like:

2011-05-06 11:00:08.716:INFO:OverlayedAppProvider:loaded jtracTemplate=jtrac-2.1.0@1304643608715

The contents of the loaded template is as follows:

templates/jtracTemplate=jtrac-2.1.0
└── WEB-INF
    ├── classes
    │   └── jtrac-init.properties
    ├── log4j.properties
    ├── overlay.xml
    ├── template.xml
    └── web-overlay.xml

The name of the template directory (or it could be a war) uses the ‘=’ character in jtracTemplate=jtrac-2.1.0 to separates the name of the template from the name of the WAR file in webapps that it applies to.  If  = is a problem, then -- may also be used.

WEB-INF/classes/jtrac-init.properties – replaces the jtrac properties file with an empty file, as the properties contained within it are configured elsewhere

WEB-INF/log4j.properties – configures the logging for all instances of the template.

WEB-INF/overlay.xml – a Jetty XML formatted IoC file that is used to inject/configure the ContextHandler for each instances. In this case it just sets up the context path:

  /

WEB-INF/template.xml – a Jetty XML formatted IoC file that is used to inject/configure the resource cache and classloader that is shared by all instances of the template. It is run only once per load of the template:

  
    true
    10000000
    1000
    64000000
  

WEB-INF/web-overlay.xml – a web.xml fragment that is overlayed on top of the web.xml from the base WAR file, that can set init parameters and add/modify filters and servlets. In this it sets the application home and springs rootKey:

  
    jtrac.home
    /tmp/jtrac-${overlay.instance.classifier}
  
  
    webAppRootKey
    jtrac-${overlay.instance.classifier}
  
  

Note the use of parameterisation of values such as ${overlays.instance.classifier}, as this allows the configuration to be made in the template and not customised for each instance.

Without the overlayed deployer, all the configurations above would still need to have been made, but rather that being in a single clear structure they would have been either in the servers common directory, the servers webdefaults.xml (aka server.xml), or baked into the WAR file of each application instance using copied/modified files from the original. The overlayed deployer allows us to make all these changes in one structure, more over it allows some of the configuration to be parameterised to facilitate easy multi-tenant deployment.

Installing an Instance Overlay

Now that we have installed a template, we can install one or more instance overlays, which deploy the actual web applications:

mv /tmp/overlays/instances/jtracTemplate=blue $JETTY_HOME/overlays/instances/
mv /tmp/overlays/instances/jtracTemplate=red $JETTY_HOME/overlays/instances/
mv /tmp/overlays/instances/jtracTemplate=blue $JETTY_HOME/overlays/instances/

As each instance is moved into place, you will see the jetty server window react and deploy that instance. Within each instance, there is the structure:

instances/jtracTemplate=red/
├── WEB-INF
│   └── overlay.xml
├── favicon.ico
└── resources
    └── jtrac.css

WEB-INF/overlay.xml – a Jetty XML format IoC file that injects/configures the context for the instance. In this case it sets up a virtual host for the instance:

  
    
      127.0.0.2
      red.myVirtualDomain.com
    
  

favicon.ico – replaces the icon in the base war with one themed for the instance colour.

resources/jtrac.css – replaces the style sheet from the base war with one themed for the instance colour

The deployed instances can now be viewed by pointing your browser at http://127.0.0.1:8080, http://127.0.0.2:8080 and http://127.0.0.3:8080. The default username/password for jtrac is admin/admin.

Things to know and notice

• Each instance is themed with images and styles sheets from the instance overlay.
• Each instance is running with it’s own application directory (eg. /tmp/jtrac-red), that is set templates web-overlay.xml.
• The instances are distinguished by virtual host that is set in the instance overlay.xml
• The static content from the base war and template are shared between all instances. Specifically there is a shared ResourceCache so only a single instance of each static content is loaded into memory.
• The classloader at the base war and template level is shared between all instances, so that only a single instance of common classes is loaded into memory. Classes with non shared statics can be configured to load in the instances classloader.
• All overlays are hot deployed and dependencies tracked. If an XML is touched in an instance, it is redeployed. If an XML is touched in a template, then all instances using it are redeployed. If a WAR file is touched, then all templates and all instances dependant on it are redeployed.
• New versions can easily be deployed. Eg when jtrac-2.2.0.war becomes available, it can just be dropped into overlays/webapps and then rename jtracTemplate=jtrac-2.1.0 to jtracTemplate=jtrac-2.2.0
• There is a fuller version of this demo in overlays-demo-jndi.tar.gz, that uses JNDI (needs options=jndi,annotations and jetty-plus.xml in start.ini) and shows how extra jars can be added in the overlays.