Operation Modes

XDP has three operation modes to accommodate easily testing functions, custom hardware from vendors, and commonly built kernels without custom hardware. Let’s go over each of them.

# Clone the linux-stable repository
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
linux-stable

# Checkout the tag for your current kernel version
cd linux-stable
git checkout tags/v4.18

# Check the available drivers
git grep -l XDP_SETUP_PROG drivers/

drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
drivers/net/ethernet/cavium/thunder/nicvf_main.c
drivers/net/ethernet/intel/i40e/i40e_main.c
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
drivers/net/ethernet/mellanox/mlx4/en_netdev.c
drivers/net/ethernet/mellanox/mlx5/core/en_main.c
drivers/net/ethernet/netronome/nfp/nfp_net_common.c
drivers/net/ethernet/qlogic/qede/qede_filter.c
drivers/net/netdevsim/bpf.c
drivers/net/tun.c
drivers/net/virtio_net.c

git grep -l XDP_SETUP_PROG_HW drivers/

include/linux/netdevice.h
866:    XDP_SETUP_PROG_HW,

net/core/dev.c
8001:           xdp.command = XDP_SETUP_PROG_HW;

drivers/net/netdevsim/bpf.c
200:    if (bpf->command == XDP_SETUP_PROG_HW && !ns->bpf_xdpoffload_accept) {
205:    if (bpf->command == XDP_SETUP_PROG_HW) {
560:    case XDP_SETUP_PROG_HW:

drivers/net/ethernet/netronome/nfp/nfp_net_common.c
3476:   case XDP_SETUP_PROG_HW:

The Packet Processor

The actor that makes it possible to execute BPF programs on XDP packets and that coordinates the interaction between them and the network stack is the XDP packet processor. The packet processor is the in-kernel component for XDP programs that processes packets on the receive (RX) queue directly as they are presented by the NIC. It ensures that packets are readable and writable and allows you to attach post-processing verdicts in the form of packet processor actions. Atomic program updates and new program loads to the packet processor can be done at runtime without any service interruption in terms of networking and associated traffic. While operating, XDP can be used in “busy polling” mode, allowing you to reserve the CPUs that will have to deal with each RX queue; this avoids context switches and allows immediate packet reactivity upon arrival regardless of IRQ affinities. The other mode XDP can be used in is the “interrupt driven” mode that, on the other hand, does not reserve the CPU but instructs an interrupt acting as an event medium to inform the CPU that it has to deal with a new event while still doing normal processing.

In Figure 7-1 you can see in the interaction points between RX/TX, applications, the packet processor, and the BPF programs applied to its packets.

Notice that there are a few squares with a string prepended by XDP_ in Figure 7-1. Those are the XDP result codes, which we cover next.

Figure 7-1. The packet processor

XDP result codes (packet processor actions)

After a decision is made about the packet in the packet processor, it can be expressed using one of the five return codes that then can instruct the network driver on how to process the packet. Let’s dive into the actions that the packet processor performs:

Drop (XDP_DROP)

Drops the packet. This happens at the earliest RX stage in the driver; dropping a packet simply implies recycling it back into the RX ring queue it just “arrived” on. Dropping the packet as early as possible is key for the denial-of-service (DoS) mitigation use cases. This way, dropped packets use as little CPU processing time and power as possible.

Forward (XDP_TX)

Forwards the packet. This can happen before or after the packet has been modified. Forwarding a packet implies bouncing the received packet page back out the same NIC it arrived on.

Redirect (XDP_REDIRECT)

Similar to XDP_TX in that it is able to transmit the XDP packet, but it does so through another NIC or into a BPF cpumap. In the case of a BPF cpumap, the CPUs serving XDP on the NIC’s receive queues can continue to do so and push the packet for processing the upper kernel stack to a remote CPU. This is similar to XDP_PASS, but with the ability that the XDP BPF program can keep serving the incoming high load as opposed to temporarily spending work on the current packet for pushing into the upper layers.

Pass (XDP_PASS)

Passes the packet to the normal network stack for processing. This is equivalent to the default packet handling behavior without XDP. This can be done in one of two ways:

• Normal receive allocates metadata (sk_buff), receives the packet onto the stack, and steers the packet to another CPU for processing. It allows for raw interfaces to user-space. This can happen before or after the packet has been modified.

• Generic receive offload (GRO) can perform a receive of large packets and combines packets of the same connection. GRO eventually passes the packet through the “normal receive” flow after processing.

Code error (XDP_ABORTED)

Denotes an eBPF program error and results in the packet being dropped. It is not something a functional program should ever use as a return code. For example, XDP_ABORTED would be returned if the program divided by zero. XDP_ABORTED’s value will always be zero. It passes the trace_xdp_exception tracepoint, which can be additionally monitored to detect misbehavior.

These action codes are expressed in the linux/bpf.h header file as follows:

enum xdp_action {
    XDP_ABORTED = 0,
    XDP_DROP,
    XDP_PASS,
    XDP_TX,
    XDP_REDIRECT,
};

Because XDP actions determine different behaviors and are an internal mechanism of the packet processor, you can look at a simplified version of Figure 7-1 focused on only the return actions (see Figure 7-2).

Figure 7-2. XDP action codes

An interesting thing about XDP programs is that you don’t usually need to write a loader to load them. There is a good loader in most Linux machines implemented by the ip command. The next section describes how to use it.

XDP and iproute2 as a Loader

The ip command, available in iproute2, has the ability to act as a frontend to load XDP programs compiled into an ELF file and has full support for maps, map relocation, tail call and object pinning.

Because loading an XDP program can be expressed as a configuration of an existing network interface, the loader is implemented as part of the ip link command, which is the one that does network device configuration.

The syntax to load the XDP program is simple:

# ip link set dev eth0 xdp obj program.o sec mysection

Let’s analyze this command parameter by parameter:

ip

This invokes the ip command.

link

Configures network interfaces.

set

Changes device attributes.

dev eth0

Specifies the network device on which we want to operate and load the XDP program.

xdp obj program.o

Loads an XDP program from the ELF file (object) named program.o. The xdp part of this command tells the system to use the native driver when it is available and fallback to generic otherwise. You can force using a mode or another by using a more specific selector:

• xdpgeneric to use generic XDP

• xdpdrv to use native XDP

• xdpoffload to use offloaded XDP

sec mysection

Specifies the section name mysection containing the BPF program to use from the ELF file; if this is not specified, the section named prog will be used. If no section is specified in the program, you have to specify sec .text in the ip invocation.

Let’s see a practical example.

The scenario is that we have a system with a web server on port 8000 for which we want to block any access to its pages on the public-facing NIC of the server by disallowing all the TCP connections to it.

The first thing that we will need is the web server in question; if you don’t already have one, you can start one with python3.

$ python3 -m http.server

After your webserver is started, its open port will be shown in the open sockets using ss. As you can see the webserver is bound to any interface, *:8000, so as of now, any external caller with access to our public interfaces will be able to see its content!

$  ss -tulpn
Netid  State      Recv-Q Send-Q Local Address:Port   Peer Address:Port
tcp    LISTEN     0      5      *:8000                *:*

At this point, we can inspect the network interfaces on the machine that’s running our HTTP server:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defau
lt qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP g
roup default qlen 1000
    link/ether 02:1e:30:9c:a3:c0 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 84964sec preferred_lft 84964sec
    inet6 fe80::1e:30ff:fe9c:a3c0/64 scope link
       valid_lft forever preferred_lft forever
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP g
roup default qlen 1000
    link/ether 08:00:27:0d:15:7d brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.11/24 brd 192.168.33.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe0d:157d/64 scope link
       valid_lft forever preferred_lft forever

Notice that this machine has three interfaces, and the network topology is simple:

lo

This is just the loopback interface for internal communication.

enp0s3

This is the management network tier; administrators will use this interface to connect to the web server to do their operations.

enp0s8

This is the interface open to the public, our web server will need to be hidden from this interface.

Now, before loading any XDP program, we can check open ports on the server from another server that can access its network interface, in our case, with IPv4 192.168.33.11.

You can check open ports on a remote host by using nmap as follows:

# nmap -sS 192.168.33.11
Starting Nmap 7.70 ( https://nmap.org ) at 2019-04-06 23:57 CEST
Nmap scan report for 192.168.33.11
Host is up (0.0034s latency).
Not shown: 998 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
8000/tcp open  http-alt

Good! Port 8000 is right there, at this point we need to block it!

Our program will consist of a single source file named program.c, so let’s see what we need to write.

It needs to use the IPv4 iphdr and Ethernet Frame ethhdr header structs and also protocol constants and other structs. Let’s include the needed headers, as shown here:

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/in.h>
#include <linux/ip.h>

After the headers are included, we can declare the SEC macro we already met in the previous chapters, used to declare ELF attributes.

#define SEC(NAME) __attribute__((section(NAME), used))

Now we can declare the main entry point for our program, myprogram, and its ELF section name, mysection. Our program takes as input context an xdp_md struct pointer, the BPF equivalent of the in-driver xdp_buff. By using that as the context, we then define the variables we will use next such as the data pointers, the Ethernet, and IP layer structs:

SEC("mysection")
int myprogram(struct xdp_md *ctx) {
  int ipsize = 0;
  void *data = (void *)(long)ctx->data;
  void *data_end = (void *)(long)ctx->data_end;
  struct ethhdr *eth = data;
  struct iphdr *ip;

Because data contains the Ethernet frame, we can now extract the IPv4 layer from it. We also check that the offset where we look for the IPv4 layer doesn’t exceed the whole pointer space so that the static verifier stays happy. When the address space is exceeded we just drop the packet:

  ipsize = sizeof(*eth);
  ip = data + ipsize;
  ipsize += sizeof(struct iphdr);
  if (data + ipsize > data_end) {
    return XDP_DROP;
  }

Now, after all the verifications and setup, we can implement the real logic for the program, which basically drops every TCP packet while allowing anything else:

  if (ip->protocol == IPPROTO_TCP) {
    return XDP_DROP;
  }

  return XDP_PASS;
}

Now that our program is done, we can save it as program.c.

The next step is to compile the ELF file program.o out of our program using Clang. We can do this compilation step outside the target machine because BPF ELF binaries are not platform dependent:

$ clang -O2 -target bpf -c program.c -o program.o

Now back on the machine hosting our web server, we can finally load program.o against the public network interface enp0s8 using the ip utility with the set command, as described earlier:

# ip link set dev enp0s8 xdp obj program.o sec mysection

As you might notice, we select the section mysection as the entry point for the program.

At this stage, if that command returned zero as the exit code with no errors, we can check the network interface to see whether the program had been loaded correctly:

# ip a show enp0s8
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric/id:32
    qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:0d:15:7d brd ff:ff:ff:ff:ff:ff
    inet 192.168.33.11/24 brd 192.168.33.255 scope global enp0s8
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe0d:157d/64 scope link
       valid_lft forever preferred_lft forever

As you can see, our output for ip a now has a new detail; after the MTU, it shows xdpgeneric/id:32, which is showing two interesting bits of information:

• The driver that had been used, xdpgeneric

• The ID of the XDP program, 32

The last step is to verify that the loaded program is in fact doing what it is supposed to do. We can verify that by executing nmap again on an external machine to observe that port 8000 is no longer reachable:

# nmap -sS 192.168.33.11
Starting Nmap 7.70 ( https://nmap.org ) at 2019-04-07 01:07 CEST
Nmap scan report for 192.168.33.11
Host is up (0.00039s latency).
Not shown: 998 closed ports
PORT    STATE SERVICE
22/tcp  open  ssh

Another test to verify that it all works can be trying to access the program through a browser or doing any HTTP request. Any kind of test should fail when targeting 192.168.33.11 as the destination. Good job and congratulations on loading your first XDP program!

If you followed all of those steps on a machine that you need to restore to its original state, you can always detach the program and turn off XDP for the device:

# ip link set dev enp0s8 xdp off

Interesting! Loading XDP programs seems easy, doesn’t it?

At least when using iproute2 as the loader, you can skip the part of having to write a loader yourself. In this example, our focus was on iproute2, which already implements a loader for XDP programs. However, the programs are in fact BPF programs, so even if iproute2 can be handy sometimes, you should always remember that you can load your programs using BCC, as shown in the next section, or you can use the bpf syscall directly. Having a custom loader has the advantage of allowing you to manage the lifecycle of the program and its interactions with user-space.

XDP Testing Using the Python Unit Testing Framework

Writing XDP tests with BPF_PROG_TEST_RUN and integrating them with the Python unit testing framework unittest is a good idea for several reasons:

• You can load and execute BPF programs using the Python BCC library.

• Python has one of the best packet crafting and introspection libraries available: scapy.

• Python integrates with C structs using ctypes.

As said, we need to import all of the needed libraries; that’s the first thing we will do in a file named test_xdp.py:

from bcc import BPF, libbcc
from scapy.all import Ether, IP, raw, TCP, UDP

import ctypes
import unittest


class XDPExampleTestCase(unittest.TestCase):
    SKB_OUT_SIZE = 1514  # mtu 1500 + 14 ethernet size
    bpf_function = None

After all the needed libraries are imported, we can proceed and create a test case class named XDPExampleTestCase. This test class will contain all of our test cases and a member method (_xdp_test_run) that we will use to do assertions and call bpf_prog_test_run.

In the following code you can see what _xdp_test_run looks like:

    def _xdp_test_run(self, given_packet, expected_packet, expected_return):
        size = len(given_packet)

        given_packet = ctypes.create_string_buffer(raw(given_packet), size)
        packet_output = ctypes.create_string_buffer(self.SKB_OUT_SIZE)

        packet_output_size = ctypes.c_uint32()
        test_retval = ctypes.c_uint32()
        duration = ctypes.c_uint32()
        repeat = 1
        ret = libbcc.lib.bpf_prog_test_run(self.bpf_function.fd,
                                           repeat,
                                           ctypes.byref(given_packet),
                                           size,
                                           ctypes.byref(packet_output),
                                           ctypes.byref(packet_output_size),
                                           ctypes.byref(test_retval),
                                           ctypes.byref(duration))
        self.assertEqual(ret, 0)
        self.assertEqual(test_retval.value, expected_return)

        if expected_packet:
            self.assertEqual(
                packet_output[:packet_output_size.value], raw(expected_packet))

It takes three arguments:

given_packet

This is the packet we test our XDP program against; it is the raw packet received by the interface.

expected_packet

This is the packet we expect to receive back after the XDP program processes it; when the XDP program returns an XDP_DROP or XDP_ABORT, we expect this to be None; in all the other cases, the packet remains the same as given_packet or can be modified.

expected_return

This is the expected return of the XDP program after processing our given_packet.

Besides the arguments, the body of this method is simple. It does conversion to C types using the ctypes library, and then it calls the libbcc equivalent of BPF_PROG_TEST_RUN, libbcc.lib.bpf_prog_test_run, using as test arguments our packets and their metadata. Then it does all of the assertions based on the results from the test call along with the given values.

After we have that function we can basically just write test cases by crafting different packets to test how they behave when passing through our XDP program, but before doing that, we need to do a setUp method for our test.

This part is crucial because the setup does the actual load of our BPF program named myprogram by opening and compiling a source file named program.c (that’s the file where our XDP code will be):

    def setUp(self):
        bpf_prog = BPF(src_file=b"program.c")
        self.bpf_function = bpf_prog.load_func(b"myprogram", BPF.XDP)

After the setup is done, the next step is to write the first behavior we want to observe. Without being too imaginative, we want to test that we will drop all TCP packets.

So we craft a packet in given_packet, which is just a TCP packet over IPv4. Then, using our assertion method, _xdp_test_run, we just verify that given our packet, we will get back an XDP_DROP with no return packet:

    def test_drop_tcp(self):
        given_packet = Ether() / IP() / TCP()
        self._xdp_test_run(given_packet, None, BPF.XDP_DROP)

Because that is not enough, we also want to explicitly test that all UDP packets are allowed. We then craft two UDP packets, one for given_packet and one for expected_packet, that are essentially the same. In that way we are also testing that UDP packets are not modified while being allowed with XDP_PASS:

    def test_pass_udp(self):
        given_packet = Ether() / IP() / UDP()
        expected_packet = Ether() / IP() / UDP()
        self._xdp_test_run(given_packet, expected_packet, BPF.XDP_PASS)

To make things a bit more complicated, we decided that this system will then allow TCP packets on the condition that they go to port 9090. When they do, they will also be rewritten to change their destination MAC address to redirect to a specific network interface with address 08:00:27:dd:38:2a.

Here’s the test case to do that. The given_packet has 9090 as a destination port, and we require the expected_packet with the new destination and port 9090 again:

    def test_transform_dst(self):
        given_packet = Ether() / IP() / TCP(dport=9090)
        expected_packet = Ether(dst='08:00:27:dd:38:2a') / 
            IP() / TCP(dport=9090)
        self._xdp_test_run(given_packet, expected_packet, BPF.XDP_TX)

With plenty of test cases, we now write the entry point for our test program, which will just call unittest.main() that then loads and executes our tests:

if __name__ == '__main__':
    unittest.main()

We have now written tests for our XDP program first! Now that we have the test acting as a specific example of what we want to have, we can write the XDP program that implements it by creating a file named program.c.

Our program is simple. It just contains the myprogram XDP function with the logic we just tested. As always, the first thing we need to do is to include the needed headers. Those headers are self-explainatory. We have a BPF program that will process TCP/IP flowing over Ethernet:

#define KBUILD_MODNAME "kmyprogram"

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/tcp.h>
#include <linux/in.h>
#include <linux/ip.h>

Again, as with the other programs in this chapter, we need to check offsets and fill variables for the three layers of our packet: ethhdr, iphdr, and tcphdr, respectively, for Ethernet, IPv4, and TCP:

int myprogram(struct xdp_md *ctx) {
  int ipsize = 0;
  void *data = (void *)(long)ctx->data;
  void *data_end = (void *)(long)ctx->data_end;
  struct ethhdr *eth = data;
  struct iphdr *ip;
  struct tcphdr *th;

  ipsize = sizeof(*eth);
  ip = data + ipsize;
  ipsize += sizeof(struct iphdr);
  if (data + ipsize > data_end) {
    return XDP_DROP;
  }

Once we have the values we can implement our logic.

The first thing we do is to check whether the protocol is TCP ip->protocol == IPPROTO_TCP. When it is, we always do an XDP_DROP; otherwise, we do an XDP_PASS for everything else.

In the check for the TCP protocol, we do another control to check whether the destination port is 9090, th->dest == htons(9090); if it is, we change the destination MAC address at the Ethernet layer and return XDP_TX to bounce the packet through the same NIC:

  if (ip->protocol == IPPROTO_TCP) {
    th = (struct tcphdr *)(ip + 1);
    if ((void *)(th + 1) > data_end) {
      return XDP_DROP;
    }

    if (th->dest == htons(9090)) {
      eth->h_dest[0] = 0x08;
      eth->h_dest[1] = 0x00;
      eth->h_dest[2] = 0x27;
      eth->h_dest[3] = 0xdd;
      eth->h_dest[4] = 0x38;
      eth->h_dest[5] = 0x2a;
      return XDP_TX;
    }

    return XDP_DROP;
  }

  return XDP_PASS;
}

Amazing! Now we can just run our tests:

sudo python test_xdp.py

The output of it will just report that the three tests passed:

...
--------------------------------
Ran 3 tests in 4.676s

OK

At this point, breaking things is easier! We can just change the last XDP_PASS to XDP_DROP in program.c and observe what happens:

.F.
======================================================================
FAIL: test_pass_udp (__main__.XDPExampleTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_xdp.py", line 48, in test_pass_udp
    self._xdp_test_run(given_packet, expected_packet, BPF.XDP_PASS)
  File "test_xdp.py", line 31, in _xdp_test_run
    self.assertEqual(test_retval.value, expected_return)
AssertionError: 1 != 2

----------------------------------------------------------------------
Ran 3 tests in 4.667s

FAILED (failures=1)

Our test failed—the status code did not match, and the test framework reported an error. That’s exactly what we wanted! This is an effective testing framework to write XDP programs with confidence. We now have the ability to make assertions on specific steps and change them accordingly to the behavior that we want to obtain. Then we write the matching code to express that behavior in the form of an XDP program.