Chapter 7. EventLoop and threading model

This chapter covers

• Threading model overview
• Event loop concept and implementation
• Task scheduling
• Implementation details

Simply stated, a threading model specifies key aspects of thread management in the context of an OS, programming language, framework, or application. How and when threads are created obviously has a significant impact on the execution of application code, so developers need to understand the trade-offs associated with different models. This is true whether they choose the model themselves or acquire it implicitly via the adoption of a language or framework.

In this chapter we’ll examine Netty’s threading model in detail. It’s powerful but easy to use and, as usual with Netty, aims to simplify your application code and maximize performance and maintainability. We’ll also discuss the experiences that led to the choice of the current model.

If you have a good general understanding of Java’s concurrency API (java.util.concurrent), you should find the discussion in this chapter straightforward. If you’re new to these concepts or need to refresh your memory, Java Concurrency in Practice by Brian Goetz, et al. (Addison-Wesley Professional, 2006) is an excellent resource.

7.1. Threading model overview

In this section we’ll introduce threading models in general and then discuss Netty’s past and present threading models, reviewing the benefits and limitations of each.

As we pointed out at the start of the chapter, a threading model specifies how code is going to be executed. Because we must always guard against the possible side effects of concurrent execution, it’s important to understand the implications of the model being applied (there are single-thread models as well). Ignoring these matters and merely hoping for the best is tantamount to gambling—with the odds definitely against you.

Because computers with multiple cores or CPUs are commonplace, most modern applications employ sophisticated multithreading techniques to make efficient use of system resources. By contrast, our approach to multithreading in the early days of Java wasn’t much more than creating and starting new Threads on demand to execute concurrent units of work, a primitive approach that works poorly under heavy load. Java 5 then introduced the Executor API, whose thread pools greatly improved performance through Thread caching and reuse.

The basic thread pooling pattern can be described as:

• A Thread is selected from the pool’s free list and assigned to run a submitted task (an implementation of Runnable).
• When the task is complete, the Thread is returned to the list and becomes available for reuse.

This pattern is illustrated in figure 7.1.

Figure 7.1. Executor execution logic

Pooling and reusing threads is an improvement over creating and destroying a thread with each task, but it doesn’t eliminate the cost of context switching, which quickly becomes apparent as the number of threads increases and can be severe under heavy load. In addition, other thread-related problems can arise during the lifetime of a project simply because of the overall complexity or concurrency requirements of an application.

In short, multithreading can be complex. In the next sections we’ll see how Netty helps to simplify it.

7.2. Interface EventLoop

Running tasks to handle events that occur during the lifetime of a connection, a basic function of any networking framework. The corresponding programming construct is often referred to as an event loop, a term Netty adopts with interface io.netty.channel.EventLoop.

The basic idea of an event loop is illustrated in the following listing, where each task is an instance of Runnable (as in figure 7.1).

Listing 7.1. Executing tasks in an event loop

Netty’s EventLoop is part of a collaborative design that employs two fundamental APIs: concurrency and networking. First, the package io.netty.util.concurrent builds on the JDK package java.util.concurrent to provide thread executors. Second, the classes in the package io.netty.channel extend these in order to interface with Channel events. The resulting class hierarchy is seen in figure 7.2.

Figure 7.2. EventLoop class hierarchy

In this model, an EventLoop is powered by exactly one Thread that never changes, and tasks (Runnable or Callable) can be submitted directly to EventLoop implementations for immediate or scheduled execution. Depending on the configuration and the available cores, multiple EventLoops may be created in order to optimize resource use, and a single EventLoop may be assigned to service multiple Channels.

Note that Netty's EventLoop, while it extends ScheduledExecutorService, defines only one method, parent().^[1] This method, shown in the following code snippet, is intended to return a reference to the EventLoopGroup to which the current EventLoop implementation instance belongs.

¹

This method overrides the EventExecutor method EventExecutorGroup parent().

public interface EventLoop extends EventExecutor, EventLoopGroup {
    @Override
    EventLoopGroup parent();
}

Event/task execution order

Events and tasks are executed in FIFO (first-in-first-out) order. This eliminates the possibility of data corruption by guaranteeing that byte contents are processed in the correct order.

7.2.1. I/O and event handling in Netty 4

As we described in detail in chapter 6, events triggered by I/O operations flow through a ChannelPipeline that has one or more installed ChannelHandlers. The method calls that propagate these events can then be intercepted by the ChannelHandlers and the events processed as required.

The nature of an event usually determines how it is to be handled; it may transfer data from the network stack into your application, do the reverse, or do something entirely different. But event-handling logic must be generic and flexible enough to handle all possible use cases. Therefore, in Netty 4 all I/O operations and events are handled by the Thread that has been assigned to the EventLoop.

This differs from the model that was used in Netty 3. In the next section we’ll discuss the earlier model and why it was replaced.

7.2.2. I/O operations in Netty 3

The threading model used in previous releases guaranteed only that inbound (previously called upstream) events would be executed in the so-called I/O thread (corresponding to Netty 4’s EventLoop). All outbound (downstream) events were handled by the calling thread, which might be the I/O thread or any other. This seemed a good idea at first but was found to be problematical because of the need for careful synchronization of outbound events in ChannelHandlers. In short, it wasn’t possible to guarantee that multiple threads wouldn’t try to access an outbound event at the same time. This could happen, for example, if you fired simultaneous downstream events for the same Channel by calling Channel.write() in different threads.

Another negative side effect occurred when an inbound event was fired as a result of an outbound event. When Channel.write() causes an exception, you need to generate and fire an exceptionCaught event. But in the Netty 3 model, because this is an inbound event, you wound up executing code in the calling thread, then handing the event over to the I/O thread for execution, with a consequent additional context switch.

The threading model adopted in Netty 4 resolves these problems by handling everything that occurs in a given EventLoop in the same thread. This provides a simpler execution architecture and eliminates the need for synchronization in the ChannelHandlers (except for any that might be shared among multiple Channels).

Now that you understand the role of the EventLoop, let’s see how tasks are scheduled for execution.

7.3. Task scheduling

Occasionally you’ll need to schedule a task for later (deferred) or periodic execution. For example, you might want to register a task to be fired after a client has been connected for five minutes. A common use case is to send a heartbeat message to a remote peer to check whether the connection is still alive. If there is no response, you know you can close the channel.

In the next sections, we’ll show you how to schedule tasks with both the core Java API and Netty’s EventLoop. Then, we’ll examine the internals of Netty’s implementation and discuss its advantages and limitations.

7.3.1. JDK scheduling API

Before Java 5, task scheduling was built on java.util.Timer, which uses a background Thread and has the same limitations as standard threads. Subsequently, the JDK provided the package java.util.concurrent, which defines the interface ScheduledExecutorService. Table 7.1 shows the relevant factory methods of java.util.concurrent.Executors.

Table 7.1. The java.util.concurrent.Executors factory methods

Methods	Description
newScheduledThreadPool( int corePoolSize) newScheduledThreadPool( int corePoolSize, ThreadFactorythreadFactory)	Creates a ScheduledThreadExecutor-Service that can schedule commands to run after a delay or to execute periodically. It uses the argument corePoolSize to calculate the number of threads.
newSingleThreadScheduledExecutor() newSingleThreadScheduledExecutor( ThreadFactorythreadFactory)	Creates a ScheduledThreadExecutor-Service that can schedule commands to run after a delay or to execute periodically. It uses one thread to execute the scheduled tasks.

Although there are not many choices,^[2] those provided are sufficient for most use cases. The next listing shows how to use ScheduledExecutorService to run a task after a 60-second delay.

²

The only concrete implementation of this interface provided by the JDK is java.util.concurrent.ScheduledThreadPoolExecutor.

Listing 7.2. Scheduling a task with a ScheduledExecutorService

Although the ScheduledExecutorService API is straightforward, under heavy load it can introduce performance costs. In the next section we’ll see how Netty provides the same functionality with greater efficiency.

7.3.2. Scheduling tasks using EventLoop

The ScheduledExecutorService implementation has limitations, such as the fact that extra threads are created as part of pool management. This can become a bottleneck if many tasks are aggressively scheduled. Netty addresses this by implementing scheduling using the channel’s EventLoop, as shown in the following listing.

Listing 7.3. Scheduling a task with EventLoop

After 60 seconds have elapsed, the Runnable instance will be executed by the EventLoop assigned to the channel. To schedule a task to be executed every 60 seconds, use scheduleAtFixedRate(), as shown next.

Listing 7.4. Scheduling a recurring task with EventLoop

As we noted earlier, Netty’s EventLoop extends ScheduledExecutorService (see figure 7.2), so it provides all of the methods available with the JDK implementation, including schedule() and scheduleAtFixedRate(), used in the preceding examples. The complete list of all the operations can be found in the Javadocs for ScheduledExecutorService.^[3]

³

Java Platform, Standard Edition 8 API Specification, java.util.concurrent, Interface ScheduledExecutorService, http://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ScheduledExecutorService.html.

To cancel or check the state of an execution, use the ScheduledFuture that’s returned for every asynchronous operation. This listing shows a simple cancellation operation.

Listing 7.5. Canceling a task using ScheduledFuture

These examples illustrate the performance gain that can be achieved by taking advantage of Netty’s scheduling capabilities. These depend, in turn, on the underlying threading model, which we’ll examine next.

7.4. Implementation details

This section examines in greater detail the principal elements of Netty’s threading model and scheduling implementation. We’ll also mention limitations to be aware of, as well as areas of ongoing development.

7.4.1. Thread management

The superior performance of Netty’s threading model hinges on determining the identity of the currently executing Thread; that is, whether or not it is the one assigned to the current Channel and its EventLoop. (Recall that the EventLoop is responsible for handling all events for a Channel during its lifetime.)

If the calling Thread is that of the EventLoop, the code block in question is executed. Otherwise, the EventLoop schedules a task for later execution and puts it in an internal queue. When the EventLoop next processes its events, it will execute those in the queue. This explains how any Thread can interact directly with the Channel without requiring synchronization in the ChannelHandlers.

Note that each EventLoop has its own task queue, independent of that of any other EventLoop. Figure 7.3 shows the execution logic used by EventLoop to schedule tasks. This is a critical component of Netty’s threading model.

Figure 7.3. EventLoop execution logic

We stated earlier the importance of not blocking the current I/O thread. We’ll say it again in another way: “Never put a long-running task in the execution queue, because it will block any other task from executing on the same thread.” If you must make blocking calls or execute long-running tasks, we advise the use of a dedicated EventExecutor. (See the sidebar “ChannelHandler execution and blocking” in section 6.2.1.)

Leaving aside such a limit case, the threading model in use can strongly affect the impact of queued tasks on overall system performance, as can the event-processing implementation of the transport employed. (And as we saw in chapter 4, Netty makes it easy to switch transports without modifying your code base.)

7.4.2. EventLoop/thread allocation

The EventLoops that service I/O and events for Channels are contained in an EventLoopGroup. The manner in which EventLoops are created and assigned varies according to the transport implementation.

Asynchronous transports

Asynchronous implementations use only a few EventLoops (and their associated Threads), and in the current model these may be shared among Channels. This allows many Channels to be served by the smallest possible number of Threads, rather than assigning a Thread per Channel.

Figure 7.4 displays an EventLoopGroup with a fixed size of three EventLoops (each powered by one Thread). The EventLoops (and their Threads) are allocated directly when the EventLoopGroup is created to ensure that they will be available when needed.

Figure 7.4. EventLoop allocation for non-blocking transports (such as NIO and AIO)

The EventLoopGroup is responsible for allocating an EventLoop to each newly created Channel. In the current implementation, using a round-robin approach achieves a balanced distribution, and the same EventLoop may be assigned to multiple Channels. (This may change in future versions.)

Once a Channel has been assigned an EventLoop, it will use this EventLoop (and the associated Thread) throughout its lifetime. Keep this in mind, because it frees you from worries about thread safety and synchronization in your ChannelHandler implementations.

Also, be aware of the implications of EventLoop allocation for ThreadLocal use. Because an EventLoop usually powers more than one Channel, ThreadLocal will be the same for all associated Channels. This makes it a poor choice for implementing a function such as state tracking. However, in a stateless context it can still be useful for sharing heavy or expensive objects, or even events, among Channels.

Blocking transports

The design for other transports such as OIO (old blocking I/O) is a bit different, as illustrated in figure 7.5.

Figure 7.5. EventLoop allocation of blocking transports (such as OIO)

Here one EventLoop (and its Thread) is assigned to each Channel. You may have encountered this pattern if you’ve developed applications that use the blocking I/O implementation in the java.io package.

But just as before, it is guaranteed that the I/O events of each Channel will be handled by only one Thread—the one that powers the Channel’s EventLoop. This is another example of Netty’s consistency of design, and it is one that contributes strongly to Netty’s reliability and ease of use.

7.5. Summary

In this chapter you learned about threading models in general and Netty’s threading model in particular, whose performance and consistency advantages we discussed in detail.

You saw how to execute your own tasks in the EventLoop (I/O Thread) just as the framework itself does. You learned how to schedule tasks for deferred execution, and we examined the question of scalability under heavy load. You also saw how to verify whether a task has executed and how to cancel it.

This information, augmented by our study of the framework’s implementation details, will help you to maximize your application’s performance while simplifying its code base. For more information about thread pools and concurrent programming in general, we recommend Java Concurrency in Practice by Brian Goetz. His book will give you a deeper understanding of even the most complex multithreading use cases.

We’ve reached an exciting point—in the next chapter we’ll discuss bootstrapping, the process of configuring and connecting all of Netty’s components to bring your application to life.