This short 5-minute survey is being presented to the Eclipse Jetty user community to validate conjecture the Jetty developers have for how users will leverage JakartaEE servlets and the Jetty project. We have some features we are gauging interest in before supporting in Jetty 12 and your responses will help shape its forthcoming release.
We will summarize results in a future blog.
With the release of the Servlet API 5.0 as part of Eclipse Jakarta EE 9.0 the standardization process has completed its move from the now-defunct Java Community Process (JCP) to being fully open source at the Eclipse Foundation, including the new Eclipse EE Specification Process (JESP) and the transition of the APIs from the javax.* to the jakarta.* namespace. The move represents a huge amount of work from many parties, but ultimately it was all meta work, in that Servlet 5.0 API is identical to the 4.0 API in all regards but name, licenses, and process, i.e. nothing functional has changed.
But now with the transition behind us, the Servlet API project is now free to develop the standard into a 5.1 or 6.0 release. So in this blog, I will put forward my ideas for how we should evolve the Servlet specification, specifically that I think that before we add new features to the API, it is time to remove some.
Version 1.0 was created in 1997 and it is amazing that over 2 decades later, a Servlet written against that version should still run in the very latest EE container. So why with such a great backward compatible record should we even contemplate introducing breaking changes to future Servlet API specification? Let’s consider some of the reasons that a developer might choose to use EE Servlets over other available technologies:
byte[] to ByteBuffers or Collections, or it may be more fundamental integration of things such as dependency injection, coding by convention, asynchronous, reactive, etc. The specification has done a reasonable job supporting such features over the years, but mistakes have been made and some features now clash, causing ambiguity and complexity. Ultimately feature integration can be an N2 problem, so reducing or simplifying existing features can greatly reduce the complexity of introducing new features.If the Servlet API is to continue to be relevant, then it needs to be able to compete with start-of-the-art HTTP servers that do not support decades of EE legacy. Legacy can be both a strength and a weakness, and I believe now is the time to focus on the former. The namespace break from java.* to jakarta.* has already introduced a discontinuity in backward compatibility. Keeping 5.0 identical in all but name to 4.0 was the right thing to do to support automatic porting of applications. However, it has also given developers a reason to consider alternatives, so now is the time to act to ensure that Servlet 6.0 a good basis for the future of EE Servlets.
Let’s just all agree upfront, without going into the details, that cross-context dispatch is a bad thing. For the purposes of the rest of this blog, I’m ignoring the many issues of cross-context dispatch. I’ll just say that every issue I will discuss below becomes even more complex when cross-context dispatch is considered, as it introduces: additional class loaders; different session values in the same session ID space; different authentication realms; authorization bypass. Don’t even get me started on the needless mind-bending complexities of a context that forwards to another then forwards back to the original…
Modern web applications are now often broken up into many microservices, so the concept of one webapp invoking another is not in itself bad, but the idea of those services being co-located in the same container instance is not very general nor flexible assumption. By all means, the Servlet API should support a mechanism to forward or include other resources, but ideally, this should be done in a way that works equally for co-resident, co-located, and remote resources.
So let’s just assume cross-context dispatch is already dead.
The concept of including another resource in a response should be straight forward, but the specification of RequestDispatcher.include(...) is just bizarre!
@WebServlet(urlPatterns = {"/servletA/*"})
public static class ServletA extends HttpServlet
{
@Override protected void doGet(HttpServletRequest request,
HttpServletResponse response) throws IOException
{
request.getRequestDispatcher("/servletB/infoB").include(request, response);
}
}
The ServletA above includes ServletB in its response. However, whilst within ServletB any calls to getServletPath() or getPathInfo(),will still return the original values used to call ServletA, rather than the “/servletB” or “/infoB” values for the target Servlet (as is done for a call to forward(...)). Instead the container must set an ever-growing list of Request attributes to describe the target of the include and any non trivial Servlet that acts on the actual URI path must do something like:
public boolean doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{
String servletPath;
String pathInfo;
if (request.getAttribute(RequestDispatcher.INCLUDE_REQUEST_URI) != null)
{
servletPath = (String)
request.getAttribute(RequestDispatcher.INCLUDE_SERVLET_PATH);
pathInfo = (String)
request.getAttribute(RequestDispatcher.INCLUDE_PATH_INFO);
}
else
{
servletPath = request.getServletPath();
pathInfo = request.getPathInfo();
}
String pathInContext = URIUtil.addPaths(servletPath, pathInfo);
// ...
}
Most Servlets do not do this, so they are unable to be correctly be the target of an include. For the Servlets that do correctly check, they are more often than not wasting CPU cycles needlessly for the vast majority of requests that are not included.
Meanwhile, the container itself must set (and then reset) at least 5 attributes, just in case the target resource might lookup one of them. Furthermore, the container must disable most of the APIs on the response object during an include, to prevent the included resource from setting the headers. So the included Servlet must be trusted to know that it is being included in order to serve the correct resource, but is then not trusted to not call APIs that are inconsistent with that knowledge. Servlets should not need to know the details of how they were invoked in order to generate a response. They should just use the paths and parameters of the request passed to them to generate a response, regardless of how that response will be used.
Ultimately, there is no need for an include API given that the specification already has a reasonable forward mechanism that supports wrapping. The ability to include one resource in the response of another can be provided with a basic wrapper around the response:
@WebServlet(urlPatterns = {"/servletA/*"})
public static class ServletA extends HttpServlet
{
@Override
protected void doGet(HttpServletRequest request,
HttpServletResponse response) throws IOException
{
request.getRequestDispatcher("/servletB/infoB")
.forward(request, new IncludeResponseWrapper(response));
}
}
Such a response wrapper could also do useful things like ensuring the included content-type is correct and better dealing with error conditions rather than ignoring an attempt to send a 500 status. To assist with porting, the include can be deprecated it’s implementation replaced with a request wrapper that reinstates the deprecated request attributes:
@Deprecated
default void include(ServletRequest request, ServletResponse response)
throws ServletException, IOException
{
forward(new Servlet5IncludeAttributesRequestWrapper(request),
new IncludeResponseWrapper(response));
}
The inclusion of the method Request.getDispatcherType()in the Servlet API is almost an admission of defeat that the specification got it wrong in so many ways that required a Servlet to know how and/or why it is being invoked in order to function correctly. Why must a Servlet know its DispatcherType? Probably so it knows it has to check the attributes for the corresponding values? But what if an error page is generated asynchronously by including a resource that forwards to another? In such a pathological case, the request will contain attributes for ERROR, ASYNC, and FORWARD, yet the type will just be FORWARD.
The concept of DispatcherType should be deprecated and it should always return REQUEST. Backward compatibility can be supported by optionally applying a wrapper that determines the deprecated DispatcherType only if the method is called.
A key feature that really needs to be revised is 6.2.2 Wrapping Requests and Responses, introduced in Servlet 2.3. The core concept of wrappers is sound, but the requirement of Wrapper Object Identity (see Object Identity Crisis below) has significant impacts. But first let’s look at a simple example of a request wrapper:
public static class ForcedUserRequest extends HttpServletRequestWrapper
{
private final Principal forcedUser;
public ForcedUserRequest(HttpServletRequest request, Principal forcedUser)
{
super(request);
this.forcedUser = forcedUser;
}
@Override
public Principal getUserPrincipal()
{
return forcedUser;
}
@Override
public boolean isUserInRole(String role)
{
return forcedUser.getName().equals(role);
}
}
This request wrapper overrides the existing getUserPrincipal() and isUserInRole(String)methods to forced user identity. This wrapper can be applied in a filter or in a Servlet as follows:
@WebServlet(urlPatterns = {"/servletA/*"})
public static class ServletA extends HttpServlet
{
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{
request.getServletContext()
.getRequestDispatcher("/servletB" + req.getPathInfo())
.forward(new ForcedUserRequest(req, new UserPrincipal("admin")),
response);
}
}
Such wrapping is an established pattern in many APIs and is mostly without significant problems. For Servlets there are some issues: it should be better documented if the wrapped user identity is propagated if ServletB makes any EE calls (I think no?); some APIs have become too complex to sensibly wrap (e.g HttpInputStream with non-blocking IO). But even with these issues, there are good safe usages for this wrapping to override existing methods.
Object Identity Crisis!
The Servlet specification allows for wrappers to do more than just override existing methods! In 6.2.2, the specification says that:
“… the developer not only has the ability to override existing methods on the request and response objects, but to provide new API… “
So the example above could introduce new API to access the original user principal:
public static class ForcedUserRequest extends HttpServletRequestWrapper
{
// ... getUserPrincipal & isUserInRole as above
public Principal getOriginalUserPrincipal()
{
return super.getUserPrincipal();
}
public boolean isOriginalUserInRole(String role)
{
return super.isUserInRole(role);
}
}
In order for targets to be able to use these new APIs then they must be able to downcast the passed request/response to the known wrapper type:
@WebServlet(urlPatterns = {"/servletB/*"})
public static class ServletB extends HttpServlet
{
@Override
protected void doGet(HttpServletRequest req, HttpServletResponse resp)
throws ServletException, IOException
{
MyWrappedRequest myr = (MyWrappedRequest)req;
resp.getWriter().printf("user=%s orig=%s wasAdmin=%b%n",
req.getUserPrincipal(),
myr.getOriginalUserPrincipal(),
myr.isOriginalUserInRole("admin"));
}
}
This downcast will only work if the wrapped object is passed through the container without any further wrapping, thus the specification requires “wrapper object identity”:
… the container must ensure that the request and response object that it passes to the next entity in the filter chain, or to the target web resource if the filter was the last in the chain, is the same object that was passed into the
doFiltermethod by the calling filter. The same requirement of wrapper object identity applies to the calls from a Servlet or a filter toRequestDispatcher.forwardorRequestDispatcher.include, when the caller wraps the request or response objects.
This “wrapper object identity” requirement means that the container is unable to itself wrap requests and responses as they are passed to filters and servlets. This restriction has, directly and indirectly, a huge impact on the complexity, efficiency, and correctness of Servlet container implementations, all for very dubious and redundant benefits:
ClassCastException. This defeats the whole point of an API specification like Servlets, which is to define good software components that can be variously assembled according to their API contracts.ServletContainerInitializers.RequestDispatcher.forward(...) then the container must make the original request object mutable so that it changes the value returned from getServletPath() to reflect the target of the dispatch. It is this impact that has significant impacts on complexity, efficiency, and correctness:
isOriginalUserInRole(String) incorrect because it calls super.isUserInRole(String) whose result can be mutated if the target Servlet has a run-as configuration. Thus this method will inadvertently return the target rather than the original role.Map operations just to make available the original state, even though it is very seldom required. Also, some aspects of the event history of a request are not recoverable from the attributes: the isUserInRolemethod; the original target of an include that does another include.The “wrapper object identity” requirement has little utility yet significant impacts on the correctness and performance of implementations. It significantly impairs the implementation of the container for a feature that can be rendered unusable by a wrapper applied by another filter. It should be removed from Servlet 6.0 and requests passed in by the container should be immutable.
A bit of history
Jetty continuations were a non-standard feature introduced in Jetty-6 (around 2005) to support thread-less waiting for asynchronous events (e.g. typically another HTTP request in a chat room). Because the Servlet API had not been designed for thread-safe access from asynchronous processes, the continuations feature did not attempt to let arbitrary threads call the Servlet API. Instead, it has a suspend/resume model that once the asynchronous wait was over, the request was re-dispatched back into the Servlet container to generate a response, using the normal blocking Servlet API from a well-defined context.
When the continuation feature was standardized in the Servlet 3.0 specification, the Jetty suspend/resume model was supported with the APIs ServletRequest.startAsync() and AsyncContext.dispatch() methods. However (against our strongly given advice), a second asynchronous model was also enabled, as represented by ServletRequest.startAsync() followed by AsyncContext.complete(). With the start/complete model, instead of generating a response by dispatching a container-managed thread, serialized on the request, to the Servlet container, arbitrary asynchronous threads could generate the response by directly accessing the request/response objects and then call the AsyncContext.complete() method when the response had been fully generated to end the cycle. The result is that the entire API, designed not to be thread safe, was now exposed to concurrent calls. Unfortunately there was (and is) very little in the specification to help resolve the many races and ambiguities that resulted.
Off to the Races
The primary race introduced by start/complete is that described above caused by mutable requests that are forced by “wrapper object identity”. Consider the following asynchronous Servlet:
@WebServlet(urlPatterns = {"/async/*"}, asyncSupported = true)
@RunAs("special")
public static class AsyncServlet extends HttpServlet
{
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response)
throws ServletException, IOException
{
AsyncContext async = request.startAsync();
PrintWriter out = response.getWriter();
async.start( () ->
{
response.setStatus(HttpServletResponse.SC_OK);
out.printf("path=%s special=%b%n",
request.getServletPath(),
request.isUserInRole("special"));
async.complete();
});
}
}
If invoked via a RequestDispatcher.forward(...), then the result produced by this Servlet is a race: will the thread dispatched to execute the lambda execute before or after the thread returns from the `doGet` method (and any applied filters) and the pre-forward values for the path and role are restored? Not only could the path and role be reported either for the target or caller, but the race could even split them so they are reported inconsistently. To avoid this race, asynchronous Servlets must copy any value that they may use from the request before starting the asynchronous thread, which is needless complexity and expense. Many Servlets do not actually do this and just rely on happenstance to work correctly.
This problem is the result of the start/complete lifecycle of asynchronous Servlets permitting/encouraging arbitrary threads to call the existing APIs that were not designed to be thread-safe. This issue is avoided if the request object passed to doGet is immutable and if it is the target of a forward, it will always act as that target. However, there are other issues of the asynchronous lifecycle that cannot be resolved just with immutability.
Out of Time
The example below is a very typical race that exists in many applications between a timeout and asynchronous processing:
@Override
protected void doGet(HttpServletRequest request,
HttpServletResponse response) throws IOException
{
AsyncContext async = request.startAsync();
PrintWriter out = response.getWriter();
async.addListener(new AsyncListener()
{
@Override
public void onTimeout(AsyncEvent asyncEvent) throws IOException
{
response.setStatus(HttpServletResponse.SC_BAD_GATEWAY);
out.printf("Request %s timed out!%n", request.getServletPath());
out.printf("timeout=%dms%n ", async.getTimeout());
async.complete();
}
});
CompletableFuture<String> logic = someBusinessLogic();
logic.thenAccept(answer ->
{
response.setStatus(HttpServletResponse.SC_OK);
out.printf("Request %s handled OK%n", request.getServletPath());
out.printf("The answer is %s%n", answer);
async.complete();
});
}
Because the handling of the result of the business logic may be executed by a non-container-managed thread, it may run concurrently with the timeout callback. The result can be an incorrect status code and/or the response content being interleaved. Even if both lambdas grab a lock to mutually exclude each other, the results are sub-optimal, as both will eventually execute and one will ultimately throw an IllegalStateException, causing extra processing and a spurious exception that may confuse developers/deployers.
The current specification of the asynchronous life cycle is the worst of both worlds for the implementation of the container. On one hand, they must implement the complexity of request-serialized events, so that for a given request there can only be a single container-managed thread in service(...), doFilter(...), onWritePossible(), onDataAvailable(), onAllDataRead()and onError(), yet on the other hand an arbitrary application thread is permitted to concurrently call the API, thus requiring additional thread-safety complexity. All the benefits of request-serialized threads are lost by the ability of arbitrary other threads to call the Servlet APIs.
Request Serialized Threads
The fix is twofold: firstly make more Servlet APIs immutable (as discussed above) so they are safe to call from other threads; secondly and most importantly, any API that does mutate state should only be able to be called from request-serialized threads! The latter might seem a bit draconian as it will make the lambda passed to thenAccept in the example above throw an IllegalStateException when it tries to setStatus(int) or call complete(), however, there are huge benefits in complexity and correctness and only some simple changes are needed to rework existing code.
Any code running within a call to service(...), doFilter(...), onWritePossible(), onDataAvailable(), onAllDataRead()and onError() will already be in a request-serialized thread, and thus will require no change. It is only code executed by threads managed by other asynchronous components (e.g. the lambda passed to thenAccept() above) that need to be scoped. There is already the method AsyncContext.start(Runnable) that allows a non-container thread to access the context (i.e. classloader) associated with the request. An additional similar method AsyncContext.dispatch(Runnable) can be provided that not only scopes the execution but mutually excludes it and serializes it against any call to the methods listed above and any other dispatched Runnable. The Runnables passed may be executed within the scope of the dispatch call if possible (making the thread momentarily managed by the container and request serialized) or scheduled for later execution. Thus calls to mutate the state of a request can only be made from threads that are serialized.
To make accessing the dispatch(Runnable) method more convenient, an executor can be provided with AsyncContext.getExecutor() which provides the same semantic. The example above can now be simply updated:
@Override
protected void doGet(HttpServletRequest request,
HttpServletResponse response) throws IOException
{
AsyncContext async = request.startAsync();
PrintWriter out = response.getWriter();
async.addListener(new AsyncListener()
{
@Override
public void onTimeout(AsyncEvent asyncEvent) throws IOException
{
response.setStatus(HttpServletResponse.SC_BAD_GATEWAY);
out.printf("Request timed out after %dms%n ", async.getTimeout());
async.complete();
}
});
CompletableFuture<String> logic = someBusinessLogic();
logic.thenAcceptAsync(answer ->
{
response.setStatus(HttpServletResponse.SC_OK);
out.printf("The answer is %s%n", answer);
async.complete();
}, async.getExecutor());
}
Because the AsyncContext.getExecutor() is used to invoke the business logic consumer, then the timeout and business logic response methods are mutually excluded. Moreover, because they are serialized by the container, the request state can be checked between each, so that if the business logic has completed the request, then the timeout callback will never be called, even if the underlying timer expires while the response is being generated. Conversely, if the business logic result is generated after the timeout, then the lambda to generate the response will never be called. Because both of the tasks in this example call complete, then only one of them will ever be executed.
And Now You’re Complete
In the example below, a non-blocking read listener has been set on the request input stream, thus a callback to onDataAvailable() has been scheduled to occur at some time in the future. In parallel, an asynchronous business process has been initiated that will complete the response:
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException
{
AsyncContext async = request.startAsync();
request.getInputStream().setReadListener(new MyReadListener());
CompletableFuture<String> logicB = someBusinessLogicB();
PrintWriter out = response.getWriter();
logicB.thenAcceptAsync(b ->
{
out.printf("The answer for %s is B=%s%n", request.getServletPath(), b);
async.complete();
}, async.getExecutor());
}
The example uses the proposed APIs above so that any call to complete is mutually excluded and serialized with the call to doGet and onDataAvailable(...). Even so, the current spec is unclear if the complete should prevent any future callback to onDataAvailable(...) or if the effect of complete() should be delayed until the callback is made (or times out). Given that the actions can now be request-serialized, the spec should require that once a request serialized thread that has called complete returns, then the request cycle is complete and there will be no other callbacks other than onComplete(...), thus cancelling any non-blocking IO callbacks.
Before extending the Servlet specification, I believe the following existing features should be removed or deprecated:
getServletContext() method will return the same value no matter what state the request is in.RequestDispatcher.include(...) is deprecated and replaced with utility response wrappers. The existing API can be deprecated and its implementation changed to use a request wrapper to simulate the existing attributes.getDispatcherType() method is deprecated and returns REQUEST, unless a utility wrapper is used to replicate the old behavior.IllegalStateException. New AsyncContext.dispatch(Runnable) and AsyncContext.getExecutor() methods will provide access to request-serialization for arbitrary threads/lambdas/RunnablesWith these changes, I believe that many web applications will not be affected and most of the remainder could be updated with minimal effort. Furthermore, utility filters can be provided that apply wrappers to obtain almost all deprecated behaviors other than Wrapper Object Identity. In return for the slight break in backward compatibility, the benefit of these changes would be significant simplifications and efficiencies of the Servlet container implementations. I believe that only with such simplifications can we have a stable base on which to build new features into the Servlet specification. If we can’t take out the cruft now, then when?
The plan is to follow this blog up with another proposing some more rationalisation of features (I’m looking at you sessions and authentication), before another blog proposing some new features an future directions.
The Jetty Project just released the Jetty Load Generator, a Java 11+ library to load-test any HTTP server, that supports both HTTP/1.1 and HTTP/2.
The project was born in 2016, with specific requirements. At the time, very few load-test tools had support for HTTP/2, but Jetty’s HttpClient did. Furthermore, few tools supported web-page like resources, which were important to model in order to compare the multiplexed HTTP/2 behavior (up to ~100 concurrent HTTP/2 streams on a single connection) against the HTTP/1.1 behavior (6-8 connections). Lastly, we were more interested in measuring quality of service, rather than throughput.
The Jetty Load Generator generates requests asynchronously, at a specified rate, independently from the responses. This is the Jetty Load Generator core design principle: we wanted the request generation to be constant, and measure response times independently from the request generation. In this way, the Jetty Load Generator can impose a specific load on the server, independently of the network round-trip and independently of the server-side processing time. Adding more load generators (on the same machine if it has spare capacity, or using additional machines) will allow the load against the server to increase linearly.
Using this core principle, you can setup the load testing by having N load generator loaders that impose the load on the server, and 1 load generator probe that imposes a very light load and measures response times.
For example, you can have 4 loaders that impose 20 requests/s each, for a total of 80 requests/s seen by the server. With this load on the server, what would be the experience, in terms of response times, of additional users that make requests to the server? This is exactly what the probe measures.
If the load on the server is increased to 160 requests/s, what would the probe experience? The same response times? Worse? And what are the probe response times if the load on the server is increased to 240 requests/s?
Rather than trying to measure some form of throughput (“what is the max number of requests/s the server can sustain?”), the Jetty Load Generator measures the quality of service seen by the probe, as the load on the server increases. This is, in practice, what matters most for HTTP servers: knowing that, when your server has a load of 1024 requests/s, an additional user can still see response times that are acceptable. And knowing how the quality of service changes as the load increases.
The Jetty Load Generator builds on top of Jetty’s HttpClient features, and offers:
ab or wrk2, with histogram reporting, for ease of use, scripting, and integration with CI servers.Download the latest command-line tool uber-jar from: https://repo1.maven.org/maven2/org/mortbay/jetty/loadgenerator/jetty-load-generator-starter/
$ cd /tmp $ curl -O https://repo1.maven.org/maven2/org/mortbay/jetty/loadgenerator/jetty-load-generator-starter/1.0.2/jetty-load-generator-starter-1.0.2-uber.jar
Use the --help option to display the available command line options:
$ java -jar jetty-load-generator-starter-1.0.2-uber.jar --help
Then run it, for example:
$ java -jar jetty-load-generator-starter-1.0.2-uber.jar --scheme https --host your_server --port 443 --resource-rate 1 --iterations 60 --display-stats
You will obtain an output similar to the following:
----------------------------------------------------
------------- Load Generator Report --------------
----------------------------------------------------
https://your_server:443 over http/1.1
resource tree : 1 resource(s)
begin date time : 2021-02-02 15:38:39 CET
complete date time: 2021-02-02 15:39:39 CET
recording time : 59.657 s
average cpu load : 3.034/1200
histogram:
@ _ 37 ms (0, 0.00%)
@ _ 75 ms (0, 0.00%)
@ _ 113 ms (0, 0.00%)
@ _ 150 ms (0, 0.00%)
@ _ 188 ms (0, 0.00%)
@ _ 226 ms (0, 0.00%)
@ _ 263 ms (0, 0.00%)
@ _ 301 ms (0, 0.00%)
@ _ 339 ms (46, 76.67%) ^50%
@ _ 376 ms (7, 11.67%) ^85%
@ _ 414 ms (5, 8.33%) ^95%
@ _ 452 ms (1, 1.67%)
@ _ 489 ms (0, 0.00%)
@ _ 527 ms (0, 0.00%)
@ _ 565 ms (0, 0.00%)
@ _ 602 ms (0, 0.00%)
@ _ 640 ms (0, 0.00%)
@ _ 678 ms (0, 0.00%)
@ _ 715 ms (0, 0.00%)
@ _ 753 ms (1, 1.67%) ^99% ^99.9%
response times: 60 samples | min/avg/50th%/99th%/max = 303/335/318/753/753 ms
request rate (requests/s) : 1.011
send rate (bytes/s) : 189.916
response rate (responses/s): 1.006
receive rate (bytes/s) : 41245.797
failures : 0
response 1xx group: 0
response 2xx group: 60
response 3xx group: 0
response 4xx group: 0
response 5xx group: 0
----------------------------------------------------
Use the Jetty Load Generator for your load testing, and report comments and issues at https://github.com/jetty-project/jetty-load-generator. Enjoy!
With the recent release of JakartaEE9, the future for Java has never been brighter. In addition to headline projects moving forward into the new jakarta.* namespace, there has been a tremendous amount of work done throughout the community to stay at the forefront of the changing landscape. These efforts are the summation of hundreds of hours by just as many developers and highlight the vibrant ecosystem in the Jakarta workspace.
The Jakarta EE contributors and committers came together to shape the 9 release. They chose to create a reality that benefits the entire Jakarta EE ecosystem. Sometimes, we tend to underestimate our influence and the power of our actions. Now that open source is the path of Jakarta EE, you, me, all of us can control the outcome of this technology.
Such examples that are worthy of emulation include the following efforts. In their own words:
Eclipse Jetty – The Jetty project recently released Jetty 11, which has worked towards full compatibility with JakartaEE9 (Servlet, JSP, and WebSocket). We are driven by a mission statement of “By Developers, For Developers”, and the Jetty team has worked since the announcement of the so-called “Big Bang” approach to move Jetty entirely into the jakarta.* namespace. Not only did this position Jetty as a platform for other developers to push their products into the future, but also allowed the project to quickly adapt to innovations that are sure to come.
[Michael Redich] The Road to Jakarta EE 9, an InfoQ news piece, was published this past October to highlight the efforts by Kevin Sutter, Jakarta EE 9 Release Lead at IBM, and to describe the progress made this past year in making this new release a reality. The Java community should be proud of their contributions to Jakarta EE 9, especially implementing the “big bang,” and discussions have already started for Jakarta EE 9.1 and Jakarta EE 10. The Q&A with Kevin Sutter in the news piece includes the certification and voting process for all the Jakarta EE specifications, plans for upcoming releases of Jakarta EE, and how Java developers can get involved in contributing to Jakarta EE. Personally, I am happy to have been involved in Jakarta EE having authored 14 Jakarta EE-related InfoQ news items for the three years, and I look forward to taking my Jakarta EE contributions to the next level. I have committed to contributing to the Jakarta NoSQL specification which is currently under development. The Garden State Java User Group (in which I serve as one of its directors) has also adopted Jakarta NoSQL. I challenge anyone who still thinks that the Java programming language is dead because these past few years have been an exciting time to be part of this amazing Java community!
WildFly 22 Beta1 contains a tech preview EE 9 variant called WildFly Preview that you can download from the WildFly download page. The WildFly team is still working on passing the needed (Jakarta EE 9) TCKs (watch for updates via the wildfly.org site.) WildFly Preview includes a mix of native EE 9 APIs and implementations (i.e. ones that use the jakarta.* namespace) along with many APIs and implementations from EE 8 (i.e. ones that use the java.* namespace). This mix of namespaces is made possible by using the Eclipse community’s excellent Eclipse Transformer project to bytecode transformer legacy EE 8 artifacts to EE 9 when the server is provisioned. Applications that are written for EE 8 can also run on WildFly Preview, as a similar transformation is performed on any deployments managed by the server.
Apache TomEE is a Jakarta EE application server based on Apache Tomcat. The project main focus is the Web Profile up until Jakarta EE 8. However, with Jakarta EE 9 and some parts being optional or pruned, the project is considering the full platform for the future. TomEE is so far a couple of tests down (99% coverage) before it reaches compatibility with Jakarta EE 8 (See Introducing TCK Work and how it helps the community jump into the effort). For Jakarta EE 9, the Community decided to pick a slightly different path than other implementations. We have already produced a couple of Apache TomEE 9 milestones for Jakarta EE 9 based on a customised version of the Eclipse Transformer. It fully supports the new jakarta.* namespace. Not to forget, the project also implements MicroProfile.
Open Liberty is in the process of completing a Compatible Implementation for Jakarta EE 9. For several months, the Jakarta EE 9 implementation has been rolling out via the “monthly” Betas. Both of the Platform and Web Profile TCK testing efforts are progressing very well with 99% success rates. The expectation is to declare one (or more) of the early Open Liberty 2021 Betas as a Jakarta EE 9 Compatible Implementation. Due to Open Liberty’s flexible architecture and “zero migration” goal, customers can be assured that their current Java EE 7, Java EE 8, and Jakarta EE 8 applications will continue to execute without any changes required to the application code or server configuration. But, with a simple change to their server configuration, customers can easily start experimenting with the new “jakarta” namespace in Jakarta EE 9.
Jelastic PaaS is the first cloud platform that has already made Jakarta EE 9 release available for the customers across a wide network of distributed hosting service providers. For the last several months Jelastic team has been actively integrating Jakarta EE 9 within the cloud platform and in December made an official release. The certified container images with the following software stacks are already updated and available for customers across over 100 data centers: Tomcat, TomEE, GlassFish, WildFly and Jetty. Jelastic PaaS provides an easy way to create environments with new Jakarta EE 9 application servers for deep testing, compatibility checks and running live production environments. It’s also possible now to redeploy existing containers with old versions to the newest ones in order to reduce the necessary migration efforts, and to expedite adoption of cutting-edge cloud native tools and products.
[Amelia Eiras] Pull Request 923- Jakarta EE 9 Contributors Card is a formidable example of eleven-Jakartees coming together to create, innovate and collaborate on an Integration-Feature that makes it so that no contributor, who helped on Jakarta EE 9 release, be forgotten in the new landing page for the EE 9 Release. Who chose those Contributors? None. That is the sole point of the existence of PR923.I chose to lead the work on the PR and worked openly by prompt communications delivered the day that Tomitribe submitted the PR – Jakarta EE Working Group message to the forum to invite other Jakartees to provide input in the creation of the new feature. With Triber Andrii, who wrote the code and the feedback of those involved, the feature is active and used in the EE 9 contributors cards, YOU ROCK WALL!
The Integration-Feature will be used in future releases. We hope that it is also adopted by any project, community, or individual in or outside the Eclipse Foundation to say ThankYOU with actions to those who help power & maintain any community.
- PR logistics: 11 Jakartees came together and produced 116 exchanges that helped merge the code. Thank you, Chris (Eclipse WebMaster) for helping check the side of INFRA. The PR’s exchanges lead us to choose the activity from 2 GitHub sources: 1) https://github.com/jakartaee/specifications/pulls all merged pulls and 2) https://github.com/eclipse-ee4j all repositories.
- PR Timeframe: the Contributors’ work accomplished from October 31st, 2019 to November 20th, 2020, was boxed and is frozen. The result is that the Contributor Cards highlight 6 different Jakartees at a time every 15 seconds. A total of 171 Jakartee Contributors (committers and contributors, leveled) belong to the amazing people behind EE 9 code. While working on that PR, other necessary improvements become obvious. A good example is the visual tweaks PR #952 we submitted that improved the landing page’s formatting, cards’ visual, etc.
Via actions, we chose to not “wait & see”, saving the project $budget, but also enabling openness to tackle the stuff that could have been dropped into “nonsense”.
In open-source, our actions project a temporary part of ourselves, with no exceptions. Those actions affect positively or negatively any ecosystem. Thank you for taking the time to read this #SharingIsCaring blog.
“Project Loom aims to drastically reduce the effort of writing, maintaining, and observing high-throughput concurrent applications that make the best use of available hardware. … The problem is that the thread, the software unit of concurrency, cannot match the scale of the application domain’s natural units of concurrency — a session, an HTTP request, or a single database operation. … Whereas the OS can support up to a few thousand active threads, the Java runtime can support millions of virtual threads. Every unit of concurrency in the application domain can be represented by its own thread, making programming concurrent applications easier. Forget about thread-pools, just spawn a new thread, one per task.” – Ron Pressler, State of Loom, May 2020
In this series of blogs, we are examining the new Loom virtual thread features now available in OpenJDK 16 early access releases. In part 1 we saw that Loom’s claim of 1,000,000 virtual threads was true, but perhaps a little misleading, as that only applies to threads with near-empty stacks. If threads actually have deep stacks, then the achieved number of virtual threads is bound by memory and is back to being the same order of magnitude as kernel threads. In this part, we will further examine the claims and ramifications of Project Loom, specifically if we can now forget about Thread Pools. Spoiler: Cheap threads can do expensive things!
All the code from this blog is available in our loom-trial project and has been run on my dev machine (Intel® Core™ i7-6820HK CPU @ 2.70GHz × 8, 32GB memory, Ubuntu 20.04.1 LTS 64-bit, OpenJDK Runtime Environment (build 16-loom+9-316)) with no specific tuning and default settings unless noted otherwise.
Project Loom makes the claim that applications need threads because kernel threads “cannot match the scale of the application domain’s natural units of concurrency”!
Really??? We’ve seen that without tuning, we can achieve 32k of either type of thread on my laptop. We think it would be fair to assume that with careful tuning, that could be stretched to beyond 100k for either technology. Is this really below the natural scale of most applications? How many applications have a natural scale of more than 32k simultaneous parallel tasks? Don’t get me wrong, there are many apps that do exceed those scales and Jetty has users that put an extra 0 on that, but they are the minority and in reality very few applications are ever going to see that demand for concurrency.
So if the vast majority of applications would be covered by blocking code with a concurrency of 32k, then what’s the big deal? Why do those apps need Loom? Or, by the same argument, why would they need to be written in asynchronous style?
The answer is that you rarely see any application deployed with 10,000s of threads; instead, threads are limited by a thread pool, typically to 100s or 1000s of threads. The default thread pool size in jetty is 200, which we sometimes see increased to 1000s, but we have never seen a 32k thread pool even though my un-tuned laptop could supposedly support it!
So what’s going on? Why are thread pools typically so limited and what about the claim that Loom means we can “Forget about thread pools”?
One reason we are told that thread pools are used is because kernel threads are slow to start, thus having a bunch of them pre-started, waiting for a task in a pool improves latency. Loom claims their virtual threads are much faster to start, so let’s test that with StartThreads, which reports:
kStart(ns) ave:137,903 from:1,000 min:47,466 max:6,048,540 vStart(ns) ave: 10,881 from:1,000 min: 4,648 max: 486,078
So that claim checks out. Virtual threads start an order of magnitude faster than kernel threads. If start time was the only reason for thread pools, then Loom’s claim of forgetting about thread pools would hold.
But start time only explains why we have thread pools, but it doesn’t explain why thread pools are frequently sized far below the systems capacity for threads: 100s instead of 10,000s? What is the reason that thread pools are sized as they are?
Giving a thread a task to do is a resource commitment. It is saying that a flow of control may proceed to consume CPU, memory and other resources that will be needed to run to completion or at least until a blocking point, where it can wait for those resources. Most of those resources are not on the stack, thus limiting the number of available threads is a way to limit a wide range of resource consumption and give quality of service:
Having a limited thread pool allows an application to be tested to that limit so that it can be proved that an application has the memory and other resources necessary to service all of those threads. Traditional thinking has been that if the configured number of threads is insufficient for the load presented, then either the excess load must wait, or the application should start using asynchronous techniques to more efficiently use those threads (rather than increase the number of threads beyond the resource capacity of the machine).
A limited thread pool is a coarse grained limit on all resources, not only threads. Limiting the number of threads puts a limit on concurrent lock contention, memory consumption and CPU usage.
Having established that there might be some good reasons to use thread pools, let’s see if Loom gives us any good reasons not to use them? So we have created a FakeDataBase class which simulates a JDBC connection pool of 100 connections with a semaphore and then in ManyTasks we run 100,000 tasks that do 2 selects and 1 insert to the database, with a small amount of CPU consumed both with and without the semaphore acquired. The core of the thread pool test is:
for (int i = 0; i < tasks; i++) pool.execute(newTask(latch));
and this is compared against the Loom virtual thread code of:
for (int i = 0 ; i < tasks; i++) Thread.builder().virtual().task(newTask(latch)).start();
And the results are…. drum roll… pretty much the same for both types of thread:
Pooled K Threads 33,729ms Spawned V Threads 34,482ms
The pooled kernel thread does appear to be consistently a little bit better, but this test is not that rigorous so let’s call it the same, which is kind of expected as the total duration is pretty much going to be primarily constrained by the concurrency of the database.
So were there any difference at all? Here is the system monitor graph during both runs: kernel threads with a pool are the left hand first period (60-30s) and then virtual threads after a change over peak (30s – 0s):
Kernel threads with thread pool do not stress the CPU at all, but virtual threads alone use almost twice as much CPU! There is also a hint of more memory being used.
The thread pool has 100k tasks in the thread pool queue, 100 kernel threads that take tasks, 100 at a time, and each task takes one of 100 semaphores permits 3 times, with little or no contention.
The Loom approach has 100k independent virtual threads that each contend 3 times for the 100 semaphore permits, with up to 99,900 threads needing to be added then removed 3 times from the semaphore’s wake up queue. The extra queuing for virtual threads could easily explain the excess CPU needed, but more investigation is needed to be definitive.
However, tasks limited by a resource like JDBC are not really the highly concurrent tasks that Loom is targeted at. To truly test Loom (and async), we need to look at a type of task that just won’t scale with blocking threads dispatched from a thread pool.
One highly concurrent work load that we often see on Jetty is chat room style interaction (or games) written on CometD and/or WebSocket. Such applications often have many 10,000s or even 100,000s of connections to the server that are mostly idle, waiting for a message to receive or an event to send. Currently we achieve these scales only by asynchronous threadless waiting, with all its ramifications of complex async APIs into the application needing async callbacks. Luckily, CometD was originally written when there was only async servlets and not async IO, thus it still has the option to be deployed using blocking I/O reads and writes. This gives it good potential to be a like for like comparison between async pooled kernel threads vs blocking virtual threads.
However, we still have a concern that this style of application/load will not be suitable for Loom because each message to a chat room will fan out to the 10s, 100s or even 1000s of other users waiting in that room. Thus a single read could result in many blocking write operations, which are typically done with deep stacks (parsing, framework, handling, marshalling, then writing) and other resources (buffers, locks etc). You can see in the following flame graph from a CometD load test using Loom virtual threads, that even with a fast client the biggest block of time is spent in the blue peak on the left, that is writing with deep stacks. It is this part of the graph that needs to scale if we have either more and/or slower clients:
To fairly test Loom, it is not sufficient to just replace the limited pool of kernel threads with infinite virtual threads. Jetty goes to lots of effort with its eat what you kill scheduling using reserved threads to ensure that whenever a selector thread calls a potentially blocking task, another selector thread has been executed. We can’t just put Loom virtual threads on top of this, else it will be paying the cost and complexity of core Jetty plus the overheads of Loom. Moreover, we have also learnt the risk of Thread Starvation that can result in highly concurrent applications if you defer important tasks (e.g. HTTP/2 flow control). Since virtual threads can be postponed (potentially indefinitely) by CPU bound applications or the use of non-Loom-aware locks (such as the synchronized keyword), they are not suitable for all tasks within Jetty.
Thus we think a better approach is to keep the core of Jetty running on kernel threads, but to spawn a virtual thread to do the actual work of reading, parsing, and calling the application and writing the response. If we flag those tasks with InvocationType.NON_BLOCKING, then they will be called directly by the selector thread, with no executor overhead. These tasks can then spawn a new virtual thread to proceed with the reading, parsing, handling, marshalling, writing and blocking. Thus we have created the jetty-10.0.x-loom branch, to use this approach and hopefully give a good basis for fair comparisons.
Our initial runs with our CometD benchmark with just 20 clients resulted in long GCs followed by out of memory failures! This is due to the usage of ThreadLocal for gathering latency statistics and each virtual thread was creating a latency capture data structure, only to use it once and then throw it away! While this problem is solvable by changing the CometD benchmark code, it reaffirms that threads use resources other than stack and that Loom virtual threads are not a drop in replacement for kernel threads.
We are aware that the handling of ThreadLocal is a well known problem in Loom, but until solved it may be a surprisingly hard problem to cope with, since you don’t typically know if a library your application depends on uses ThreadLocal or not.
With the CometD benchmark modified to not use ThreadLocal, we can now take Loom/Jetty/CometD to a moderate number of clients (1000 which generated the flame graph above) with the following results:
CLIENT: Async Jetty/CometD server
========================================
Testing 1000 clients in 100 rooms, 10 rooms/client
Sending 1000 batches of 10x50 bytes messages every 10000 µs
Elapsed = 10015 ms
- - - - - - - - - - - - - - - - - - - -
Outgoing: Rate = 990 messages/s - 99 batches/s - 12.014 MiB/s
Incoming: Rate = 99829 messages/s - 35833 batches/s(35.89%) - 26.352 MiB/s
@ _ 3,898 µs (112993, 11.30%)
@ _ 7,797 µs (141274, 14.13%)
@ _ 11,696 µs (136440, 13.65%)
@ _ 15,595 µs (139590, 13.96%) ^50%
@ _ 19,493 µs (142883, 14.29%)
@ _ 23,392 µs (130493, 13.05%)
@ _ 27,291 µs (112283, 11.23%) ^85%
@ _ 31,190 µs (59810, 5.98%) ^95%
@ _ 35,088 µs (12968, 1.30%)
@ _ 38,987 µs (4266, 0.43%) ^99%
@ _ 42,886 µs (2150, 0.22%)
@ _ 46,785 µs (1259, 0.13%)
@ _ 50,683 µs (910, 0.09%)
@ _ 54,582 µs (752, 0.08%)
@ _ 58,481 µs (567, 0.06%)
@ _ 62,380 µs (460, 0.05%) ^99.9%
@ _ 66,278 µs (365, 0.04%)
@ _ 70,177 µs (232, 0.02%)
@ _ 74,076 µs (82, 0.01%)
@ _ 77,975 µs (13, 0.00%)
@ _ 81,873 µs (2, 0.00%)
Messages - Latency: 999792 samples
Messages - min/avg/50th%/99th%/max = 209/15,095/14,778/35,815/78,184 µs
Messages - Network Latency Min/Ave/Max = 0/14/78 ms
SERVER: Async Jetty/CometD server
========================================
Operative System: Linux 5.8.0-33-generic amd64
JVM: Oracle Corporation OpenJDK 64-Bit Server VM 16-ea+25-1633 16-ea+25-1633
Processors: 12
System Memory: 89.26419% used of 31.164349 GiB
Used Heap Size: 73.283676 MiB
Max Heap Size: 2048.0 MiB
- - - - - - - - - - - - - - - - - - - -
Elapsed Time: 10568 ms
Time in Young GC: 5 ms (2 collections)
Time in Old GC: 0 ms (0 collections)
Garbage Generated in Eden Space: 3330.0 MiB
Garbage Generated in Survivor Space: 4.227936 MiB
Average CPU Load: 397.78314/1200
========================================
Jetty Thread Pool:
threads: 174
tasks: 302146
max concurrent threads: 34
max queue size: 152
queue latency avg/max: 0/11 ms
task time avg/max: 1/3316 ms
CLIENT: Loom Jetty/CometD server
========================================
Testing 1000 clients in 100 rooms, 10 rooms/client
Sending 1000 batches of 10x50 bytes messages every 10000 µs
Elapsed = 10009 ms
- - - - - - - - - - - - - - - - - - - -
Outgoing: Rate = 990 messages/s - 99 batches/s - 13.774 MiB/s
Incoming: Rate = 99832 messages/s - 41201 batches/s(41.27%) - 27.462 MiB/s
@ _ 2,718 µs (99690, 9.98%)
@ _ 5,436 µs (116281, 11.64%)
@ _ 8,155 µs (115202, 11.53%)
@ _ 10,873 µs (108572, 10.87%)
@ _ 13,591 µs (106951, 10.70%) ^50%
@ _ 16,310 µs (117139, 11.72%)
@ _ 19,028 µs (114531, 11.46%)
@ _ 21,746 µs (94080, 9.42%) ^85%
@ _ 24,465 µs (71479, 7.15%)
@ _ 27,183 µs (34358, 3.44%) ^95%
@ _ 29,901 µs (11526, 1.15%) ^99%
@ _ 32,620 µs (4513, 0.45%)
@ _ 35,338 µs (2123, 0.21%)
@ _ 38,056 µs (988, 0.10%)
@ _ 40,775 µs (562, 0.06%)
@ _ 43,493 µs (578, 0.06%) ^99.9%
@ _ 46,211 µs (435, 0.04%)
@ _ 48,930 µs (187, 0.02%)
@ _ 51,648 µs (31, 0.00%)
@ _ 54,366 µs (27, 0.00%)
@ _ 57,085 µs (1, 0.00%)
Messages - Latency: 999254 samples
Messages - min/avg/50th%/99th%/max = 192/12,630/12,476/29,704/54,558 µs
Messages - Network Latency Min/Ave/Max = 0/12/54 ms
SERVER: Loom Jetty/CometD server
========================================
Operative System: Linux 5.8.0-33-generic amd64
JVM: Oracle Corporation OpenJDK 64-Bit Server VM 16-loom+9-316 16-loom+9-316
Processors: 12
System Memory: 88.79622% used of 31.164349 GiB
Used Heap Size: 61.733116 MiB
Max Heap Size: 2048.0 MiB
- - - - - - - - - - - - - - - - - - - -
Elapsed Time: 10560 ms
Time in Young GC: 23 ms (8 collections)
Time in Old GC: 0 ms (0 collections)
Garbage Generated in Eden Space: 8068.0 MiB
Garbage Generated in Survivor Space: 3.6905975 MiB
Average CPU Load: 413.33084/1200
========================================
Jetty Thread Pool:
threads: 14
tasks: 0
max concurrent threads: 0
max queue size: 0
queue latency avg/max: 0/0 ms
task time avg/max: 0/0 ms
The results here are a bit mixed, but there are some positives for Loom:
This is an interesting but inconclusive result. It is at a low scale on a fast loopback network with a client unlikely to cause blocking, so not really testing either approach. We now need to scale this test to many 10,000s of clients on a real network, which will require multiple load generation machines and careful measurement. This will be the subject of part 3 (probably some weeks away).
It is good that Project Loom adds inexpensive and fast spawning/blocking virtual threads to the JVM. But cheap threads can do expensive things!
Having 1,000,000 concurrent application entities is going to take memory, CPU and other resources, no matter if they block or use async callbacks. It may be that entirely different programming styles are needed for Loom, as is suggested by Loom Structured Concurrency, however we have not yet seen anything that provides limitations on resources that can be used by unlimited spawning of virtual threads. There are also indications that Loom’s flexible stack management comes with a CPU cost. However, it has been moderately simple to update Jetty to experiment with using Loom to call a blocking application and we’d very much encourage others to load test their application on the jetty-10.0.x-loom branch.
Many of Loom’s claims have stacked up: blocking code is much easier to write, virtual threads are very fast to start and cheap to block. However, other key claims either do not hold up or have yet to be substantiated: we do not think virtual threads give natural scaling as threads themselves are not the limiting factor, rather it is the resources that are used that determines the scaling. The suggestion to “Forget about thread-pools, just spawn a new thread…” feels like an invitation to create unstable applications unless other substantive resource management strategies are put into place.
Given that Duke’s “new clothes” woven by Loom are not one-size-fits-all, it would be a mistake to stop developing asynchronous APIs for things such as DNS and JDBC on the unsubstantiated suggestion that Loom virtual threads will make them unnecessary.
Following the releases of Eclipse Jetty 10.0.0 and 11.0.0, the CometD project has released versions 5.0.3, 6.0.0 and 7.0.0.
CometD 5.0.x, of which the latest is the newly released 5.0.3, require at least Java 8 and it is based on Jetty 9.4.x.
This version will be maintained as long as Jetty 9.4.x is maintained, likely many more years, to allow migration away from Java 8.
CometD 6.0.x, with the newly released 6.0.0, requires at least Java 11 and it is based on Jetty 10.0.x.
In turn, Jetty 10.0.x is based on Java EE 8 / Jakarta EE 8, which provide Servlet and WebSocket APIs under the javax.* packages.
This version of CometD provides a smooth transition from CometD 5 (or earlier) to Java 11 and Jetty 10.
In this way, you can leverage new Java 11 language features in your application source code, as well as the few new APIs made available in the Servlet 4.0 Specification without having to change much of your code, since you will still be using javax.* Servlet and WebSocket APIs (see the next section on CometD 7.0.x for the migration to the jakarta.* APIs).
However, server-side CometD applications should not depend much on Servlet or WebSocket APIs, so the migration from CometD 5 (or earlier) should be pretty straightforward.
CometD 6.0.x, as Jetty 10.0.x, are transition releases to the new Jakarta EE 9 APIs in the jakarta.* packages.
CometD 6.0.x will be maintained as long as Jetty 10.0.x is maintained.
Since CometD applications do not depend much on Servlet or WebSocket APIs, consider moving directly to CometD 7.0.x if you don’t depend on other libraries (for example, Spring) that require javax.* classes.
CometD 7.0.x, with the newly released 7.0.0, requires at least Java 11 and it is based on Jetty 11.0.x.
In turn, Jetty 11.0.x is based on Jakarta EE 9, which provide Servlet and WebSocket APIs under the jakarta.* packages.
Migrating from CometD 6.0.x (or earlier) to CometD 7.0.x should be very easy if your applications depend mostly on the CometD APIs (few or no changes there) and depend very little on the Servlet or WebSocket APIs.
Dependencies on the javax.* APIs should be migrated to the jakarta.* APIs.
If your applications depend on third-party libraries, you need to make sure to update the third-party library to a version that supports — if necessary — jakarta.* APIs.
Migrating to CometD 7.0.x allows you to leverage Java 11 language features and the benefit that you can base your applications on the new Jakarta EE Specifications, therefore abandoning the now-dead Java EE Specifications.
This migration keeps your applications up-to-date with the current state-of-the-art of EE Specifications, reducing the technical debt to a minimum.
CometD 7.0.x will be maintained as long as Jetty 11.0.x is maintained.
The evolution of the Jakarta EE Specifications will be implemented by future Jetty versions, and future CometD versions will keep the pace.
If your applications depend on third-party libraries that use or depend on Jetty 9.4.x (such as Spring / Spring Boot), use CometD 5.0.x and Jetty 9.4.x until the third party libraries update to newer Jetty versions.
If your applications depend on third-party libraries that depend on javax.* APIs and that have not been updated to Jakarta EE 9 yet (and therefore to jakarta.* APIs), use CometD 6.0.x and Jetty 10.0.x until the third party libraries update to Jakarta EE 9.
If your applications depend on third-party libraries that have already been updated to use Jakarta EE 9 APIs, for example, Jakarta Restful Web Services (previously known as JAX-RS) using Eclipse Jersey 3.0, use CometD 7.0.x and Jetty 11.0.x.
If you are migrating from earlier CometD versions, skip entirely CometD 6.0.x: for example move from CometD 5.0.x and Jetty 9.4.x directly to CometD 7.0.x and Jetty 11.0.x.
Following the releases of Eclipse Jetty 10.0.0 and 11.0.0, the Reactive HttpClient project — introduced back in 2017 — has released versions 1.1.5, 2.0.0 and 3.0.0.
Reactive HttpClient Versions 1.1.x, of which the latest is the newly released 1.1.5, requires at least Java 8 and it is based on Jetty 9.4.x.
This version will be maintained as long as Jetty 9.4.x is maintained, likely many more years, to allow migration away from Java 8.
Reactive HttpClient Versions 2.0.x, with the newly released 2.0.0, requires at least Java 11 and it is based on Jetty 10.0.x.
The Reactive HttpClient 2.0.x series is incompatible with the 1.1.x series, since the Jetty HttpClient APIs changed between Jetty 9.4.x and Jetty 10.0.x.
This means that projects such as Spring WebFlux, at the time of this writing, are not compatible with the 2.0.x series of Reative HttpClient.
Reactive HttpClient Versions 3.0.x, with the newly released 3.0.0, requires at least Java 11 and it is based on Jetty 11.0.x.
In turn, Jetty 11.0.x is based on the Jakarta EE 9 Specifications, which means jakarta.servlet and not javax.servlet.
The Reactive HttpClient 3.0.x series is fundamentally identical to the 2.0.x series, apart from the Jetty dependency.
While the HttpClient APIs do not change between Jetty 10 and Jetty 11, if you are using Jakarta EE 9 it will be more convenient to use the Reactive HttpClient 3.0.x series.
For example when using Reactive HttpClient to call a third party service from within a REST service, it will be natural to use Reactive HttpClient 2.0.x if you use javax.ws.rs, and Reactive HttpClient 3.0.x if you use jakarta.ws.rs.
Enjoy the new releases and tell us which series you use by adding a comment here!
For further information, refer to the project page on GitHub.
The Jetty HTTP client internally uses a connection pool to recycle HTTP connections, as they are expensive to create and dispose of. This is a well-known pattern that has proved to work well.
While this pattern brings great benefits, it’s not without its shortcomings. If the Jetty client is used by concurrent threads (like it’s meant to be) then one can start seeing some contention building up onto the pool as it is a central point that all threads have to go to to get a connection.
Here’s a breakdown of what the client contains:
HttpClient ConcurrentMap<Origin, HttpDestination> Origin scheme host port HttpDestination ConnectionPool
The client contains a Map of Origin -> HttpDestination. The origin basically describes where the remote service is and the destination is a wrapper for the connection pool with bells and whistles.
So, when you ask the client to do a GET on https://www.google.com/abc the client will use the connection pool contained in the destination keyed by “https|www.google.com|443”. Asking the client to GET https://www.google.com/xyz would make it use the exact same connection pool as the origins of both URLs are the same. Asking the same client to also do a GET on https://www.facebook.com/ would make it use a different connection pool as the origin this time around is “https|www.facebook.com|443”.
It is quite common for a client to exchange with a handful, or sometimes even a single destination so it is expected that if multiple threads are using the client in parallel, they will all ask the same connection pool for a connection.
There are three important implementations of the ConnectionPool interface:
DuplexConnectionPool is the default. It guarantees that a connection will never be shared among threads, i.e.: when a connection is acquired by one thread, no other thread will be able to acquire it for as long as it hasn’t been released. It also tries to maximize re-use of the connections such as only a minimal amount of connections have to be kept open.
MultiplexConnectionPool This one allows up to N threads (N being configurable) to acquire the exact same connection. While this is of no use for HTTP/1 connections as HTTP/1.1 does not allow running concurrent requests, HTTP/2 does so when the Jetty client connects to an HTTP/2 server; a single connection can be used to send multiple requests in parallel. Just like DuplexConnectionPool, it also tries to maximize the re-use of connections.
RoundRobinConnectionPool Similar MultiplexConnectionPool as it allows multiple threads to share the connections. But unlike it, it does not try to maximize the re-use of connections. Instead, it tries to spread the load evenly across all of them. It can be configured to not multiplex the connections so that it can also be used with HTTP/1 connections.
Some users are heavily using the Jetty client with many concurrent threads, and they noticed that their threads were bottlenecked in the code that tries to get a connection from the connection pool. A quick investigation later revealed the problem comes from the fact that the connection pool implementations all use a java.util.Deque protected with a java.util.concurrent.locks.ReentrantLock. A single lock + multiple threads makes it rather obvious why this bottlenecks: all threads are contending on the lock which is easily provable with a simple microbenchmark ran under a profiler.
Here is the benchmarked code:
@Benchmark
public void testPool()
{
Connection connection = pool.acquire(true);
Blackhole.consumeCPU(ThreadLocalRandom.current().nextInt(10, 20));
pool.release(connection);
}
And here is the report of the benchmark, running on a 12 cores CPU with 12 parallel threads and 12 connections in the pool:
Benchmark (POOL_TYPE) Mode Cnt Score Error Units CPU ConnectionPoolsBenchmark.testPool duplex thrpt 3 3168609.140 ± 1703378.453 ops/s 15% ConnectionPoolsBenchmark.testPool multiplex thrpt 3 2284937.900 ± 191568.815 ops/s 15% ConnectionPoolsBenchmark.testPool round-robin thrpt 3 1403693.845 ± 219405.841 ops/s 25%
It was quite apparent that while the benchmark was running, the reported CPU consumption never reached anything close to 100% no matter how many threads were configured to use the connection pool in parallel.
There are two fundamental problems. The most obvious one is the lock that protects all access to the connection pool. The second is more subtle, it’s the fact that a queue is an ill-suited data structure for writing performant concurrent algorithms.
To be more precise: there are excellent concurrent queue algorithms out there that do a terrific job, and sometimes you have no choice but to use a queue to implement your logic correctly. But when you can write your concurrent code with a different data structure than a queue, you’re almost always going to win. And potentially win big.
Here is an over-simplified explanation of why:
Queue head tail [c1]-[c2]-[c3]-[c4] c1-4 are the connection objects.
All threads dequeue from the head and enqueue at the bottom, so that creates natural contention on these two spots. No matter how the queue is implemented, there is a natural set of data describing the head that must be read and modified concurrently by many threads so some form of mutual exclusion or Compare-And-Set retry loop has to be used to touch the head. Said differently: reading from a queue requires modifying the shape of the queue as you’re removing an entry from it. The same is true when writing to the queue.
Both DuplexConnectionPool and MultiplexConnectionPool exacerbate the problem because they use the queue as a stack: the re-queuing is performed at the top of the queue as these two implementations want to maximize usage of the connections. RoundRobinConnectionPool mitigates that a bit by dequeuing from the head and re-queuing at the tail to maximize load spreading over all connections.
The idea was to come up with a data structure that does not need to modify its shape when a connection is acquired or released. So instead of directly storing the connections in the data structure, let’s wrap it in some metadata holder object (let’s call its class Entry) that can tell if the connection is in use or not. Let’s pick a base data structure that is quick and easy to iterate, like an array.
All threads iterate the array and try to reserve the visited connections up until one can be successfully reserved. The reservation is done by trying to flip the flag atomically which isn’t retried as a failure meaning you
need to try the next connection. Because of multiplexing, the free/in-use flag actually has to be a counter that can go from 0 to configured max multiplex.
Array [e1|e2|e3|e4] a1 a2 a3 a4 c1 c2 c3 c4 e1-4 are the Entry objects. a1-4 are the "acquired" counters. c1-4 are the connection objects.
Both DuplexConnectionPool and MultiplexConnectionPool still have some form of contention as they always start iterating at the array’s index 0. This is unavoidable if you want to maximize the usage of connections.
RoundRobinConnectionPool uses an atomic counter to figure out where to start iterating. This relieves the contention on the first elements in the array at the expense of contending on the atomic counter itself.
The results of this algorithm are speaking for themselves:
Benchmark (POOL_TYPE) Mode Cnt Score Error Units CPU ConnectionPoolsBenchmark.testPool duplex thrpt 3 15661516.143 ± 104590.027 ops/s 100% ConnectionPoolsBenchmark.testPool multiplex thrpt 3 6172145.313 ± 459510.509 ops/s 100% ConnectionPoolsBenchmark.testPool round-robin thrpt 3 15446647.061 ± 3544105.965 ops/s 100%
Clearly, this time we’re using all the CPU cycles we can. And we get 3 times more throughput for the multiplex pool, 5 times for the duplex pool, and 10 times for the round-robin pool.
This is quite an improvement in itself, but we can still do better.
The above explanation describes the core algorithm that the new connection pools were built upon. But this is a bit of an over-simplification as reality is a bit more complex than that.
There is one extra step we can take that can improve the performance of the duplex and multiplex connection pools even further. In an ideal case, there are as many connections in the pool as there are threads using them. What if we could make the threads stick to the same connection every time? This can be done by storing a reference to the successfully acquired connection into a thread-local variable which could then be reused the next time(s) a connection is acquired. This way, we could bypass both the iteration of the array and contention on the “acquired” counters. Of course, life isn’t always ideal so we would still need to keep the iteration and the counters as a fallback in case we drift away from the ideal case. But the closer the usage of the pool would be from the ideal case, the more this thread-local cache would avoid iterations and contention on the counters.
Here are the results you get with this extra addition:
Benchmark (POOL_TYPE) Mode Cnt Score Error Units CPU ConnectionPoolsBenchmark.testPool cached/duplex thrpt 3 65338641.629 ± 42469231.542 ops/s 100% ConnectionPoolsBenchmark.testPool cached/multiplex thrpt 3 64245269.575 ± 142808539.722 ops/s 100%
The duplex pool is over four times faster again, and the multiplex pool over 10 times faster. Again, the benchmark reflects the ideal case so this is really the best one can get but these improvements are certainly worth considering some tuning of the pool to try to get as much of those multipliers as possible.
All this logic has been moved to the org.eclipse.jetty.util.Pool class so that it can be re-used across all connection pool implementations, as well as potentially for pooling other objects. A good example is the server’s XmlConfiguration parser that uses expensive-to-create XML parsers. Another example is the Inflater and Deflater classes that are expensive to instantiate but are heavily used for compressing/decompressing requests and responses. Both are great candidates to pool objects and already are, using custom never-reused pooling code.
So we could replace those two pools with the new pool created for the Jetty client’s HTTP connection pool. Except that none of those need to maximize re-use of the pooled objects, and they don’t need to rotate through them either, so none of the two iterations discussed above do the proper job, and both come with some overhead.
Hence, StrategyType was introduced to tell the pool from where the iteration should start:
public enum StrategyType
{
/**
* A strategy that looks for an entry always starting from the first entry.
* It will favour the early entries in the pool, but may contend on them more.
*/
FIRST,
/**
* A strategy that looks for an entry by iterating from a random starting
* index. No entries are favoured and contention is reduced.
*/
RANDOM,
/**
* A strategy that uses the {@link Thread#getId()} of the current thread
* to select a starting point for an entry search. Whilst not as performant as
* using the {@link ThreadLocal} cache, it may be suitable when the pool is substantially smaller
* than the number of available threads.
* No entries are favoured and contention is reduced.
*/
THREAD_ID,
/**
* A strategy that looks for an entry by iterating from a starting point
* that is incremented on every search. This gives similar results to the
* random strategy but with more predictable behaviour.
* No entries are favoured and contention is reduced.
*/
ROUND_ROBIN,
}
FIRST being the one used by duplex and multiplex connection pools and ROUND_ROBIN the one used, well, by the round-robin connection pool. The two new ones, RANDOM and THREAD_ID, are about spreading the load across all entries like ROUND_ROBIN, but unlike it, they use a cheap way to find that start index that does not require any form of contention.
The Jetty Project provided to the Java community support for NPN first (the precursor of ALPN) in Java 7, and then support for ALPN in Java 8.
The ALPN support was implemented by modifying sun.security.ssl classes, and this required that the modified classes were prepended to the bootclasspath, so the command line would like like this:
java -Xbootclasspath/p:/path/to/alpn-boot-8.1.13.v20181017.jar ...
However, every different OpenJDK version could require a different version of the alpn-boot jar. ALPN support is there, but cumbersome to use.
Things improved a bit with the Jetty ALPN agent. The agent could detect the Java version and redefine on-the-fly the sun.security.ssl classes (using the alpn-boot jars). However, a new OpenJDK version could generate a new alpn-boot jar, which in turn generates a new agent version, and therefore for every new OpenJDK version applications needed to verify whether the agent version needed to be updated. A bit less cumbersome, but still another hoop to jump through.
The Jetty Project worked with the OpenJDK project to introduce proper ALPN APIs, and this was introduced in Java 9 with JEP 244.
On April 14th 2020, OpenJDK 8u252 released (the Oracle version is named 8u251, and it may be equivalent to 8u252, but there is no source code available to confirm this; we will be referring only to the OpenJDK version in the following sections).
OpenJDK 8u252 contains the ALPN APIs backported from Java 9.
This means that for OpenJDK 8u252 the alpn-boot jar is no longer necessary, because it’s no longer necessary to modify sun.security.ssl classes as now the official OpenJDK ALPN APIs can be used instead of the Jetty ALPN APIs.
The Jetty ALPN agent version 2.0.10 does not perform class redefinition if it detects that the OpenJDK version is 8u252 or later. It can still be left in the command line (although we recommend not to), but will do nothing.
Libraries or applications that are directly using the Jetty ALPN APIs (the org.eclipse.jetty.alpn.ALPN and related classes) should now be modified to detect whether the backported OpenJDK ALPN APIs are available. If the backported OpenJDK ALPN APIs are available, libraries or applications must use them; otherwise they must fall back to use the Jetty ALPN APIs (like they were doing in the past).
For example, the Jetty project is using the Jetty ALPN APIs in artifact jetty-alpn-openjdk8-[client|server]. The classes in that Jetty artifact have been changed like described above, starting from Jetty version 9.4.28.v20200408.
Applications that are using Java 8 and that depend on libraries that provide ALPN support (such as the jetty-alpn-openjdk8-[client|server] artifact described above) must modify the way they are started.
For an application that is still using an OpenJDK version prior to 8u252, the typical command line requires the alpn-boot jar in the bootclasspath and a library that uses the Jetty ALPN APIs (here as an example, jetty-alpn-openjdk8-server) in the classpath:
/opt/openjdk-8u242/bin/java -Xbootclasspath/p:/path/to/alpn-boot-8.1.13.v20181017.jar -classpath jetty-alpn-openjdk8-server-9.4.27.v20200227:...
For the same application that wants to use OpenJDK 8u252 or later, the command line becomes:
/opt/openjdk-8u252/bin/java -classpath jetty-alpn-openjdk8-server-9.4.28.v20200408:...
That is, the -Xbootclasspath option must be removed and the library must be upgraded to a version that supports the backported OpenJDK ALPN APIs.
Alternatively, applications can switch to use the Jetty ALPN agent as described below.
Yes, because Java vendors are planning to support Java 8 until (at least) 2030 and this would have required to provide alpn-boot for another 10 years.
Now instead, applications and libraries will be able to use the official OpenJDK ALPN APIs provided by Java for as long as Java 8 is supported.
Yes, if your application does not need ALPN (in particular, it does not need artifacts jetty-alpn-openjdk8-[client|server]).
No, if your application needs ALPN (in particular, it needs artifacts jetty-alpn-openjdk8-[client|server]). You must upgrade to Jetty 9.4.28.v20200402 or later.
You must verify that library X has been updated to OpenJDK 8u252.
In practice, library X must detect whether the backported OpenJDK ALPN APIs are available; if so, it must use the backported OpenJDK ALPN APIs; otherwise, it must use the Jetty ALPN APIs.
You need to make a version of your application that works with OpenJDK 8u252 and earlier, see the previous question.
For example, if you depend on Jetty, you need to depend on jetty-alpn-openjdk8-[client|server]-9.4.28.v20200402 or later.
Then you must add the Jetty ALPN agent to the command line, since it supports all Java 8 versions.
/opt/openjdk-8u242/bin/java -javaagent:/path/to/jetty-alpn-agent-2.0.10.jar -classpath jetty-alpn-openjdk8-server-9.4.28.v20200408:...
/opt/openjdk-8u252/bin/java -javaagent:/path/to/jetty-alpn-agent-2.0.10.jar -classpath jetty-alpn-openjdk8-server-9.4.28.v20200408:...
The command lines are identical apart the Java version.
In this way, if your application is run with with OpenJDK 8u242 or earlier, the Jetty ALPN agent will apply the sun.security.ssl class redefinitions, and jetty-alpn-openjdk8-[client|server]-9.4.28.v20200402 will use the Jetty ALPN APIs.
Otherwise your application is run with OpenJDK 8u252 or later, the Jetty ALPN agent will do nothing (no class redefinition is necessary), and the jetty-alpn-openjdk8-[client|server]-9.4.28.v20200402 will use the OpenJDK ALPN APIs.
The Eclipse Jakarta EE project has not obtained the rights from Oracle to extend the Java EE APIs living in the javax.* package. As such, the Java community is faced with a choice between continuing to use the frozen javax.* APIs or transitioning to a new jakarta.* namespace where these specifications could continue to be evolved and/or significantly changed.
This name change is now a “fait accompli” (a.k.a. “done deal”), so it will happen no matter what. However, how the Eclipse Jakarta EE specification process handles the transition and how vendors like Webtide respond are yet to be determined.
This blog discusses some of the options for how Webtide can evolve the Jetty project to deal with this renaming.
A release of Jakarta EE 8 will happen first, and that will include:
javax.* namespace.For example, the current Maven coordinates for the Servlet jar are (groupId:artifactId:version):
javax.servlet:javax.servlet-api:4.0.1
With Jakarta EE 8, these coordinates will change to:
jakarta.servlet:jakarta.servlet-api:4.0.2
The content of these 2 jars will be identical: both will contain javax.servlet.Servlet.
The Eclipse Jakarta EE project is currently having a debate about what the future of Jakarta looks like. Options include:
Jakarta EE 9 would rename every API and implementation to use the jakarta.* namespace API.
This means that javax.servlet.Servlet will become jakarta.servlet.Servlet.
No other significant changes would be made (including not removing any deprecated methods or behaviours). This requires applications to update the package imports from javax.* to jakarta.* en mass to update possible dependencies to versions that use the jakarta.* APIs (e.g. REST frameworks, etc.) and recompile the application.
Alternatively, backward compatibility would be required (provided by Containers) in some form so that legacy javax.* code could be deployed (at least initially) with static and/or dynamic translation and/or adaption.
For example, Container vendors could provide tools that pre-process your javax.* application, transforming it into a jakarta.* application. Exact details of how these tools will work (or even exist) are still uncertain.
Jakarta EE 9 would maintain any javax.* API that did not require any changes. Only those APIs that have been modified/updated/evolved/replaced would be in the jakarta.* namespace.
For example, if the Servlet 4.0 Specification is not updated, it will remain in the javax.* package.
However, if the Servlet Specification leaders decide to produce a Servlet 5.0 Specification, it will be in the jakarta.* package.
So far, there appears very little user demand for iterations on the Servlet API.
The current stable Jetty release series, 9.4.x, is on Servlet 3.1, which has not changed since 2013.
Even though Servlet 4.0 was released in 2017, we have yet to finalize our Jetty 10 release using it, in part waiting for new eclipse Jakarta artifacts for the various EE specs, eg: servlet-api, jsp-api, el-api, websocket-api, and mail-api. Despite being 2 years late with Servlet 4.0, we have not received a single user request asking for Servlet 4.0 features or a release date.
On the other hand, there has been interest from our users in more significant changes, some of which we have already started supporting in Jetty specific APIs:
Jetty 10, implementing the Servlet 4.0 Specification, will be released once the frozen Jakarta EE 8 artifacts are available. These artifacts will have a Maven groupId of jakarta.*, but will contain classes in the javax.* packages.
It is envisioned that Jetty 10 will soon after become our primary stable release of Jetty and will be enhanced for several years and maintained for many more. In the mid-term, it is not Webtide’s intention to force any user to migrate to the new jakarta.* APIs purely due to the lack of availability of a javax.* implementation. Any innovations or developments done in Jetty 10 will have to be non standard extensions.
However, we are unable to commit to long term support for the external dependencies bundled with a Jetty release that use the javax.* package (eg. JSP, JSTL, JNDI, etc.) unless we receive such commitments from their developers.
Jetty 11 would be our first release that uses the Jakarta EE 9 APIs. These artifacts will have a Maven groupId of jakarta.* and contain classes also in the jakarta.* packages.
We are currently evaluating if we will simply do a rename (aka Big Bang) or instead take the opportunity to evolve the server to be able to run independently of the EE APIs and thus be able to support both javax.* and jakarta.*
If the Eclipse foundation determines that the existing javax.* APIs are to be renamed to jakarta.* APIs at once and evolved, then the following options are available for the development of future Jetty versions.
Note that current discussions around this approach will not change anything about the functionality present in the existing APIs (eg: no removal of deprecated methods or functions. no cleanup or removal of legacy behaviors. no new functionality, etc)
We keep developing Jetty against the javax.* APIs which become frozen in time.
We would continue to add new features but via Jetty specific APIs. The resources that would have otherwise been used supporting the rename to jakarta.* will be used to improve and enhance Jetty instead.
For users wishing to stay with javax.* this is what we plan to do with Jetty 10 for many years, so “Do Nothing” will be an option for some time. That said, if Jetty is to continue to be a standards-based container, then we do need to explore additional options.
Jetty 11 would be created by branching Jetty 10 and renaming all javax.* code to use jakarta.* APIs from Jakarta EE9. Any users wishing to run javax.* code would need to do so on a separate instance of Jetty 10 as Jetty 11 would only run the new APIs.
New features implemented in future releases of Jakarta EE APIs would only be available in Jetty 11 or beyond. Users that wish to use the latest Jetty would have to do the same rename in their entire code base and in all their dependencies.
The transition to the new namespace would be disruptive but is largely a one-time effort. However, significant new features in the APIs would be delayed by the transition and then constrained by the need to start from the existing APIs.
This option is essentially the same as Option 1, except that Jetty 11 would include tools/features to dynamically rename (or adapt – the technical details to be determined) javax.* code usage, either as it is deployed or via pre-deployment tooling. This would allow existing code to run on the new Jetty 11 releases. However, as the Jakarta APIs evolve, there is no guarantee that this could continue to be done, at least not without some runtime cost. The Jakarta EE project is currently discussing backward compatibility modes and how (or if) these will be specified.
This approach minimizes the initial impact of transitioning, as it allows legacy code to continue to be deployed. The downside, however, is that these impacts may be experienced for many years until that legacy code is refactored. These effects will include the lost opportunities for new features to be developed as development resources are consumed implementing and maintaining the binary compatibility feature.
We could take the opportunity forced on us by the renaming to make the core of Jetty 11 independent of any Jakarta EE APIs.
A new modern lightweight core Jetty server would be developed, based on the best parts of Jetty 10, but taking the opportunity to update, remove legacy concerns, implement current best practices and support new features.
Users could develop to this core API as they do now for embedded Jetty usage. Current embedded Jetty code would need to be ported to the new API (as it would with any javax.* rename to jakarta.* ).
Standard-based deployments would be supported by adaption layers providing the servlet container and implementations would be provided for both javax.* and jakarta.*. A single server would be able to run both old and new APIs, but within a single context, the APIs would not be able to be mixed.
This is a significant undertaking for the Jetty Project but potentially has the greatest reward as we will obtain new features, not just a rename.
These are options if the Eclipse foundation determines that the jakarta.* package will be used only for new or enhanced APIs.
This option is conceptually similar to Option 3, except that the new Jetty core API will be standardized by Jakarta EE 9 (which will hopefully be a lightweight core) to current best practices and include new features.
Users could develop to this new standard API. Legacy deployments would be supported by adaption layers providing the javax.* servlet container, which may also be able to see new APIs and thus mix development.
This option is substantially the same as Option 3 as Jetty 11 would be based around an independent lightweight core API.
Users could develop to this core API as they do now for embedded Jetty usage. Standard-based deployments would be supported by adaption layers providing the servlet container and implementations would be provided for both javax.* and jakarta.*.
As with any large-scale change, community feedback is paramount. We invite all users of Jetty, regardless of how you consume it, to make your voice heard on this issue.
For Jetty-specific feedback, we have opened messages on both the jetty-users and jetty-dev mailing lists. These discussions can be found below:
https://www.eclipse.org/lists/jetty-users/msg08908.html
https://www.eclipse.org/lists/jetty-dev/msg03307.html
To provide feedback on the broader change from the javax.* to jakarta.* namespace, you can find a discussion on the jakartaee-platform-dev mailing list:
https://www.eclipse.org/lists/jakartaee-platform-dev/msg00029.html