Chapter 13. Broadcasting events with UDP
This chapter covers
- An overview of UDP
- A sample broadcasting application
Most of the examples you’ve seen so far have used connection-based protocols such as TCP. In this chapter we’ll focus on a connectionless protocol, User Datagram Protocol (UDP), which is often used when performance is critical and some packet loss can be tolerated.[1]
One of the best-known UDP-based protocols is the Domain Name Service (DNS), which maps fully qualified names to numeric IP addresses.
We’ll start with an overview of UDP, its characteristics and limitations. Following that we’ll describe this chapter’s sample application, which will demonstrate how to use the broadcasting capabilities of UDP. We’ll also make use of an encoder and a decoder to handle a POJO as the broadcast message format. By the end of the chapter, you’ll be ready to make use of UDP in your own applications.
13.1. UDP basics
Connection-oriented transports (like TCP) manage the establishment of a connection between two network endpoints, the ordering and reliable transmission of messages sent during the lifetime of the connection, and finally, orderly termination of the connection. By contrast, in a connectionless protocol like UDP, there’s no concept of a durable connection and each message (a UDP datagram) is an independent transmission.
Furthermore, UDP doesn’t have the error-correcting mechanism of TCP, where each peer acknowledges the packets it receives and unacknowledged packets are retransmitted by the sender.
By analogy, a TCP connection is like a telephone conversation, where a series of ordered messages flows in both directions. UDP, conversely, resembles dropping a bunch of postcards in a mailbox. You can’t know the order in which they will arrive at their destination, or even if they all will arrive.
These aspects of UDP may strike you as serious limitations, but they also explain why it’s so much faster than TCP: all overhead of handshaking and message management has been eliminated. Clearly, UDP is a good fit for applications that can handle or tolerate lost messages, unlike those that handle financial transactions.
13.2. UDP broadcast
All of our examples so far have utilized a transmission mode called unicast,[2] defined as the sending of messages to a single network destination identified by a unique address. This mode is supported by both connected and connectionless protocols.
UDP provides additional transmission modes for sending a message to multiple recipients:
- Multicast —Transmission to a defined group of hosts
- Broadcast —Transmission to all of the hosts on a network (or a subnet)
The example application in this chapter will demonstrate the use of UDP broadcast by sending messages that can be received by all the hosts on the same network. For this purpose we’ll use the special limited broadcast or zero network address 255.255.255.255. Messages sent to this address are destined for all the hosts on the local network (0.0.0.0) and are never forwarded to other networks by routers.
Next we’ll discuss the design of the application.
13.3. The UDP sample application
Our example application will open a file and broadcast each line as a message to a specified port via UDP. If you’re familiar with UNIX-like OSes, you may recognize this as a very simplified version of the standard syslog utility. UDP is a perfect fit for such an application because the occasional loss of a line of a log file can be tolerated, given that the file itself is stored in the file system. Furthermore, the application provides the very valuable capability of effectively handling a large volume of data.
What about the receiver? With UDP broadcast, you can create an event monitor to receive the log messages simply by starting up a listener program on a specified port. Note that this ease of access raises a potential security concern, which is one reason why UDP broadcast tends not to be used in insecure environments. For the same reason, routers often block broadcast messages, restricting them to the network where they originate.
Publish/subscribe
Applications like syslog are typically classified as publish/subscribe: a producer or service publishes the events, and multiple clients subscribe to receive them.
Figure 13.1 presents a high-level view of the overall system, which consists of a broadcaster and one or more event monitors. The broadcaster listens for new content to appear, and when it does, transmits it as a broadcast message via UDP.
Figure 13.1. Broadcast system overview
All event monitors listening on the UDP port receive the broadcast messages.
To keep things simple, we won’t be adding authentication, verification, or encryption to our sample application. But it would not be difficult to incorporate these features to make this a robust, usable utility.
In the next section we’ll start to explore the design and implementation details of the broadcaster component.
13.4. The message POJO: LogEvent
In messaging applications, data is often represented by a POJO, which may hold configuration or processing information in addition to the actual message content. In this application we’ll handle a message as an event, and because the data comes from a log file, we’ll call it LogEvent. Listing 13.1 shows the details of this simple POJO.
Listing 13.1. LogEvent message
With the message component defined, we can implement the application’s broadcasting logic. In the next section we’ll examine the Netty framework classes that are used to encode and transmit LogEvent messages.
13.5. Writing the broadcaster
Netty provides a number of classes to support the writing of UDP applications. The primary ones we’ll be using are the message containers and Channel types listed in table 13.1.
Table 13.1. Netty UDP classes used in broadcaster
|
Name |
Description |
|---|---|
|
interface AddressedEnvelope <M, A extends SocketAddress> extends ReferenceCounted |
Defines a message that wraps another message with sender and recipient addresses. M is the message type; A is the address type. |
|
class DefaultAddressedEnvelope <M, A extends SocketAddress> implements AddressedEnvelope<M,A> |
Provides a default implementation of interface AddressedEnvelope. |
|
class DatagramPacket extendsDefaultAddressedEnvelope <ByteBuf, InetSocketAddress> implements ByteBufHolder |
Extends DefaultAddressedEnvelope to use ByteBuf as the message data container. |
|
interface DatagramChannel extends Channel |
Extends Netty’s Channel abstraction to support UDP multicast group management. |
|
class NioDatagramChannnel extends AbstractNioMessageChannel implements DatagramChannel |
Defines a Channel type that can send and receive AddressedEnvelope messages. |
Netty’s DatagramPacket is a simple message container used by DatagramChannel implementations to communicate with remote peers. Like the postcards we referred to in our earlier analogy, it carries the address of the recipient (and optionally, the sender) as well as the message payload itself.
To convert EventLog messages to DatagramPackets, we’ll need an encoder. But there’s no need to write our own from scratch. We’ll extend Netty’s MessageToMessageEncoder, which we used in chapters 9 and 10.
Figure 13.2 shows the broadcasting of three log entries, each one via a dedicated DatagramPacket.
Figure 13.2. Log entries sent via DatagramPackets
Figure 13.3 represents a high-level view of the ChannelPipeline of the LogEventBroadcaster, showing how LogEvents flow through it.
Figure 13.3. LogEventBroadcaster: ChannelPipeline and LogEvent flow
As you’ve seen, all data to be transmitted is encapsulated in LogEvent messages. The LogEventBroadcaster writes these to the channel, sending them through the ChannelPipeline where they’re converted (encoded) into DatagramPacket messages. Finally, they are broadcast via UDP and picked up by remote peers (monitors).
The next listing shows our customized version of MessageToMessageEncoder, which performs the conversion just described.
Listing 13.2. LogEventEncoder
With LogEventEncoder implemented, we’re ready to bootstrap the server, which includes setting various ChannelOptions and installing the needed ChannelHandlers in the pipeline. This will be done by the main class, LogEventBroadcaster, shown next.
Listing 13.3. LogEventBroadcaster
This completes the broadcaster component of the application. For initial testing you can use the netcat program. On UNIX/Linux systems you should find it installed as nc. A version for Windows is available at http://nmap.org/ncat.
netcat is perfect for basic testing of this application; it just listens on a specified port and prints all data received to standard output. Set it to listen for UDP data on port 9999 as follows:
$ nc -l -u 9999
Now we need to start our LogEventBroadcaster. Listing 13.4 shows how to compile and run the broadcaster using mvn. The configuration in pom.xml points to a file that is frequently updated, /var/log/messages (assuming a UNIX/Linux environment), and sets the port to 9999. The entries in the file will be broadcast to that port via UDP and printed to the console on which you started netcat.
Listing 13.4. Compile and start the LogEventBroadcaster
$ chapter14> mvn clean package exec:exec LogEventBroadcaster [INFO] Scanning for projects... [INFO] [INFO] -------------------------------------------------------------------- [INFO] Building UDP Broadcast 1.0-SNAPSHOT [INFO] -------------------------------------------------------------------- ... ... [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ netty-in-action --- [INFO] Building jar: target/chapter14-1.0-SNAPSHOT.jar [INFO] [INFO] --- exec-maven-plugin:1.2.1:exec (default-cli) @ netty-in-action – LogEventBroadcaster running
To change the file and port values, specify them as System properties when invoking mvn. The next listing shows how to set the logfile to /var/log/mail.log and the port to 8888.
Listing 13.5. Compile and start the LogEventBroadcaster
$ chapter14> mvn clean package exec:exec -PLogEventBroadcaster / -Dlogfile=/var/log/mail.log –Dport=8888 –.... .... [INFO] [INFO] --- exec-maven-plugin:1.2.1:exec (default-cli) @ netty-in-action – LogEventBroadcaster running
When you see LogEventBroadcaster running, you’ll know it started up successfully. If there are errors, an exception message will be printed. Once the process is running, it will broadcast any new log messages that are added to the logfile.
Using netcat is adequate for testing purposes but it would not be suitable for a production system. This brings us to the second part of our application—the broadcast monitor we’ll implement in the next section.
13.6. Writing the monitor
Our goal is to replace netcat with a more complete event consumer, which we’ll call EventLogMonitor. This program will
1. Receive UDP DatagramPackets broadcast by the LogEventBroadcaster
2. Decode them to LogEvent messages
3. Write the LogEvent messages to System.out
As before, the logic will be implemented by custom ChannelHandlers—for our decoder we’ll extend MessageToMessageDecoder. Figure 13.4 depicts the ChannelPipeline of the LogEventMonitor and shows how LogEvents will flow through it.
Figure 13.4. LogEventMonitor
The first decoder in the pipeline, LogEventDecoder, is responsible for decoding incoming DatagramPackets to LogEvent messages (a typical setup for any Netty application that transforms inbound data). The following listing shows the implementation.
Listing 13.6. LogEventDecoder
The job of the second ChannelHandler is to perform some processing on the LogEvent messages created by the first. In this case, it will simply write them to System.out. In a real-world application you might aggregate them with events originating from a different log file or post them to a database. This listing, which shows the LogEventHandler, illustrates the basic steps to follow.
Listing 13.7. LogEventHandler
The LogEventHandler prints the LogEvents in an easy-to-read format that consists of the following:
- The received timestamp in milliseconds
- The InetSocketAddress of the sender, which consists of the IP address and port
- The absolute name of the file the LogEvent was generated from
- The actual log message, which represents one line in the log file
Now we need to install our handlers in the ChannelPipeline, as seen in figure 13.4. This listing shows how it is done by the LogEventMonitor main class.
Listing 13.8. LogEventMonitor
13.7. Running the LogEventBroadcaster and LogEventMonitor
As before, we’ll use Maven to run the application. This time you’ll need to open two console windows, one for each of the programs. Each will keep running until you stop it with Ctrl-C.
First you need to start the LogEventBroadcaster. Because you’ve already built the project, the following command will suffice (using the default values):
$ chapter14> mvn exec:exec -PLogEventBroadcaster
As before, this will broadcast the log messages via UDP.
Now, in a new window, build and start the LogEventMonitor to receive and display the broadcast messages.
Listing 13.9. Compiling and starting the LogEventBroadcaster
$ chapter13> mvn clean package exec:exec -PLogEventMonitor [INFO] Scanning for projects... [INFO] [INFO] -------------------------------------------------------------------- [INFO] Building UDP Broadcast 1.0-SNAPSHOT [INFO] -------------------------------------------------------------------- [INFO] [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ netty-in-action --- [INFO] Building jar: target/chapter14-1.0-SNAPSHOT.jar [INFO] [INFO] --- exec-maven-plugin:1.2.1:exec (default-cli) @ netty-in-action --- LogEventMonitor running
When you see LogEventMonitor running, you’ll know it started up successfully. If there is an error, an exception message will be printed.
The console will display any events as they are added to the log file, as shown next. The format of the messages is that created by the LogEventHandler.
Listing 13.10. LogEventMonitor output
1364217299382 [/192.168.0.38:63182] [/var/log/messages] : Mar 25 13:55:08
dev-linux dhclient: DHCPREQUEST of 192.168.0.50 on eth2 to 192.168.0.254
port 67
1364217299382 [/192.168.0.38:63182] [/var/log/messages] : Mar 25 13:55:08
dev-linux dhclient: DHCPACK of 192.168.0.50 from 192.168.0.254
1364217299382 [/192.168.0.38:63182] [/var/log/messages] : Mar 25 13:55:08
dev-linux dhclient: bound to 192.168.0.50 -- renewal in 270 seconds.
1364217299382 [/192.168.0.38:63182] [[/var/log/messages] : Mar 25 13:59:38
dev-linux dhclient: DHCPREQUEST of 192.168.0.50 on eth2 to 192.168.0.254
port 67
1364217299382 [/192.168.0.38:63182] [/[/var/log/messages] : Mar 25 13:59:38
dev-linux dhclient: DHCPACK of 192.168.0.50 from 192.168.0.254
1364217299382 [/192.168.0.38:63182] [/var/log/messages] : Mar 25 13:59:38
dev-linux dhclient: bound to 192.168.0.50 -- renewal in 259 seconds.
1364217299383 [/192.168.0.38:63182] [/var/log/messages] : Mar 25 14:03:57
dev-linux dhclient: DHCPREQUEST of 192.168.0.50 on eth2 to 192.168.0.254
port 67
1364217299383 [/192.168.0.38:63182] [/var/log/messages] : Mar 25 14:03:57
dev-linux dhclient: DHCPACK of 192.168.0.50 from 192.168.0.254
1364217299383 [/192.168.0.38:63182] [/var/log/messages] : Mar 25 14:03:57
dev-linux dhclient: bound to 192.168.0.50 -- renewal in 285 seconds.
If you don’t have access to a UNIX syslog, you can create a custom file and supply content manually to see the application in action. The steps shown next use UNIX commands, starting with touch to create an empty file.
$ touch ~/mylog.log
Now start up the LogEventBroadcaster again and point it to the file by setting the system property:
$ chapter14> mvn exec:exec -PLogEventBroadcaster -Dlogfile=~/mylog.log
Once the LogEventBroadcaster is running, you can manually add messages to the file to see the broadcast output in the LogEventMonitor console. Use echo and redirect the output to the file as shown here:
$ echo 'Test log entry' >> ~/mylog.log
You can start as many instances of the monitor as you like; each will receive and display the same messages.
13.8. Summary
In this chapter we provided an introduction to connectionless protocols using UDP as an example. We built a sample application that converts log entries to UDP datagrams and broadcasts them to be picked up by subscribed monitor clients. Our implementation made use of a POJO to represent the log data and a custom encoder to convert from this message format to Netty’s DatagramPacket. The example illustrates the ease with which a Netty UDP application can be developed and extended to support specialized uses.
In the next two chapters we’ll look at case studies presented by users from well-known companies who have built industrial-strength applications with Netty.