8
CONFIGURING NETWORKING

Now that you know enough networking to be dangerous, you can configure a network connection. While FreeBSD supports many different protocols, we’ll focus on the nearly ubiquitous Ethernet connection, generally delivered over CAT5 or CAT6 cables.¹

We’ll start with the essentials for getting a host on the network and able to access other internet hosts. Raw TCP/IP connectivity isn’t enough, however; you also need the ability to resolve host names to IP addresses, so we’ll cover that next. Then we’ll talk about measuring network activity, performance, VLANs, and aggregating links.

Before you can do any of that, though, you need some information.

Network Prerequisites

If your network offers Dynamic Host Configuration Protocol (DHCP), you can connect to the network as a client without knowing anything about the network. A static IP address makes much more sense on a server, however. While the installer will configure the network for you, eventually every server needs changes. Both IPv4 and IPv6 require the following information:

• An IP address
• The netmask for that IP address and protocol
• The IP address of the default gateway

Armed with this information, attach your system to the network with ifconfig(8) and route(8) and then make the configuration permanent in /etc/rc.conf.

Configuring Changes with ifconfig(8)

The ifconfig(8) program displays the interfaces on your computer and lets you configure them. Start by listing the existing interfaces on your system by running ifconfig(8) without any arguments:

# ifconfig
➊em0: flags=8843<➋UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=85259b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,
LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO>
        inet ➌203.0.113.43 netmask 0xfffffff0 broadcast 198.51.100.47
        inet6 ➍fe80::225:90ff:fee8:1270%em0 prefixlen 64 scopeid 0x1
        inet6 ➎2001:db8::bad:c0de:cafe prefixlen 64
        ether ➏00:25:90:db:d5:94
        media: ➐Ethernet autoselect (1000baseTX <full-duplex>)
        status: ➑active

rl0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
        options=8<VLAN_MTU>
        ether 00:20:ed:72:3b:5f
        media: Ethernet autoselect (10baseT/UTP)
        status: ➒no carrier
➓lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Our first network interface is em0 ➊, or the first network card that uses the em(4) driver. The em(4) man page reveals that this is an Intel PRO/1000 card. You’ll then see basic information about this card ➋, including that it is in the UP state, meaning it’s either working or trying to work. It’s assigned the IPv4 address 203.0.113.43 ➌ and the netmask 0xfffffff0 (or 255.255.255.240, per Table 7-2). This card has two IPv6 addresses, the link-local address (beginning with fe80) ➍ and the global IPv6 address ➎. You’ll also see the MAC address ➏ and the connection speed ➐. Finally, the status entry shows that this card is active ➑: a cable is plugged in and we have a link light.

The second card, rl0, has almost none of this information associated with it. One key fact is the no carrier signal ➒: it’s not plugged in and there is no link light. This card is not in use.

Finally we have the interface lo0 ➓, the loopback. This interface has the IPv4 address 127.0.0.1 and IPv6 address ::1 on every machine. This loopback address is used when the machine talks to itself. This is a standard software interface, which does not have any associated physical hardware. Do not attempt to delete the loopback interface, and do not change its IP address—things will break in an amusing way if you do so. FreeBSD supports other software interfaces, such as disc(4), tap(4), gif(4), and many more.

Adding an IP to an Interface

The install process will configure any network cards you have working at install time. If you didn’t configure the network for all of your cards during the setup process, or if you add or remove network cards after finishing the install, you can assign an IP address to your network card with ifconfig(8). You need the card’s assigned IP address and netmask.

# ifconfig interface-name inet IP-address netmask

For example, if your network card is em0, your IP address is 203.0.113.250, and your netmask is 255.255.255.0, you would type:

# ifconfig em0 inet 203.0.113.250 255.255.255.0

Specify the netmask in dotted-quad notation as above or in hex format (0xffffff00). Perhaps simplest of all is to use slash notation, like this:

# ifconfig em0 inet 203.0.113.250/24

To configure an IPv6 address, add the inet6 keyword between the interface name and the address.

# ifconfig em0 inet6 2001:db8::bad:c0de:cafe/64

The ifconfig(8) program can also perform any other configuration your network cards require, letting you work around hardware bugs in features such as the various sorts of checksum offloading, like setting media type and duplex mode for sub-gigabit interfaces. You’ll find supported options in the man pages for the driver and ifconfig(8). Here, I disable checksum offloading and TCP segmentation offloading on my em0 interface, even while I set the IP address.

# ifconfig em0 inet 203.0.113.250/24 -tso -rxcsum

To make this persist across reboots, add an entry to /etc/rc.conf that tells the system to configure the card at boot. An IPv4 entry has the form ifconfig_interfacename="ifconfig arguments". For example, configuring the idle rl0 card would require an entry much like this:

ifconfig_rl0 ="inet 203.0.113.250/24"

An IPv6 entry has the form ifconfig_interfacename_ipv6="ifconfig arguments".

ifconfig_rl0_ipv6="2001:db8::bad:c0de:cafe/64"

Once you have a working configuration for your interface, copy your ifconfig(8) arguments into a /etc/rc.conf entry.

Testing Your Interface

Now that your interface has an IP address, try to ping the IPv4 address of your default gateway. If you get a response, as shown in the following example, you’re on the local network. Interrupt the ping with CTRL-C.

# ping 203.0.113.1
PING 203.0.113.1 (203.0.113.1): 56 data bytes
64 bytes from 203.0.113.1: icmp_seq=0 ttl=64 time=1.701 ms
64 bytes from 203.0.113.1: icmp_seq=1 ttl=64 time=1.436 ms
^C
--- 203.0.113.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.436/1.569/1.701/0.133 ms

For IPv6, use ping6(8) instead of ping(8). If you use router discovery, the default route will almost always be a link-local address.

# ping6 2001:db8::1
PING6(56=40+8+8 bytes) 2001:db8::bad:c0de:cafe --> 2001:db8::1
16 bytes from 2001:db8::1, icmp_seq=0 hlim=64 time=0.191 ms
16 bytes from 2001:db8::1, icmp_seq=1 hlim=64 time=0.186 ms
16 bytes from 2001:db8::1, icmp_seq=2 hlim=64 time=0.197 ms
--snip--

If you don’t get any answers, your network connection isn’t working. Either you have a bad connection (check your cables and link lights) or you have misconfigured your card.

Set Default Route

The default route is the address where your system sends all traffic that’s not on the local network. If you can ping the default route’s IPv4 address, set it via route(8).

# route add default 203.0.113.1

That’s it! You should now be able to ping any public IPv4 address on the internet.

Adding the default IPv6 route is much the same, but you need to add the -6 command line flag to change the IPv6 routing table.

# route -6 add default 2001:db8::1

If you didn’t choose nameservers during the system install, you’ll have to use the IP address rather than the hostname.

Once you have a working default router, make it persist across reboots by adding the proper defaultrouter and ipv6_defaultrouter entries in /etc/rc.conf:

defaultrouter="203.0.113.1"
ipv6_defaultrouter="2001:db8::1"

Multiple IP Addresses on One Interface

A FreeBSD system can respond to multiple IP addresses on one interface. This is especially useful for jails (see Chapter 22). Specify additional IPv4 addresses for an interface with ifconfig(8) and the keywords inet and alias. The netmask on an IPv4 alias is always /32, regardless of the size of the network address block the main address uses.

# ifconfig em0 inet alias 203.0.113.225/32

IPv6 aliases use the actual prefix length (slash) of the subnet they’re on. Be sure you use the inet6 keyword.

# ifconfig em0 inet6 alias 2001:db8::bad:c0de:caff/64

Once you add an alias to the interface, the additional IP address appears in ifconfig(8) output. The main IP always appears first, and aliases follow.

   # ifconfig fxp0
   fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
           options=b<RXCSUM,TXCSUM,VLAN_MTU>
           inet6 fe80::225:90ff:fee8:1270%vtnet0 prefixlen 64 scopeid 0x1
➊          inet6 2001:db8::bad:c0de:cafe prefixlen 64
           inet6 2001:db8::bad:c0de:caff prefixlen 64
           inet 203.0.113.250 netmask 0xffffff00 broadcast 203.0.113.255
➋          inet 203.0.113.225 netmask 0xffffffff broadcast 203.0.113.255
           ether 00:02:b3:63:e4:1d
   --snip--

Here we see our brand new IPv4 ➊ and IPv6 ➋ aliases. Hosts that ping your aliased addresses will get a response from this server.

Once you have the aliases working as you like, make them persist across reboots by adding additional ifconfig statements in /etc/rc.conf:

ifconfig_em0_alias0="inet 203.0.113.225/32"
ifconfig_em0_alias1="inet6 2001:db8::bad:c0de:caff/64"

The only real difference between this entry and the standard rc.conf ’s “here’s my IP address” entry are the alias0 and alias1 chunks. The alias keyword tells FreeBSD that this is an aliased IP, and the 0 and 1 are unique numbers assigned to each alias. Every alias set in /etc/rc.conf must have a unique number, and this number must be sequential. If you skip a number, aliases after the gap won’t be installed at boot. This is the most common interface misconfiguration I’ve seen.

Many daemons, such as inetd(8) and sshd(8), can be bound to a single address (see Chapter 20), so you can run multiple instances of the same program on the same server using multiple addresses.

Renaming Interfaces

FreeBSD names its network interfaces after the device driver used by the network card. This is a fine old tradition in the Unix world and common behavior among most industrial operating systems. Some operating systems name their network interfaces by the type of interface—for example, Linux calls its Ethernet interfaces eth0, eth1, and so on. At times, it makes sense to rename an interface, either to comply with an internal standard or to make its function more apparent. For example, I have one device with 12 network interfaces, each plugged into a different network. Each network has a name such as test, QA, and so on. Renaming these network interfaces to match the attached networks makes sense.

While FreeBSD is flexible on interface names, some software isn’t—it assumes that a network interface name is a short word followed by a number. This isn’t likely to change any time in the near future, so it’s best practice to use a short interface name ending in a digit. Use ifconfig(8)’s name keyword to rename an interface. For example, to rename em1 to test1, you would run:

# ifconfig em1 name test1

Running ifconfig(8) without arguments shows that you have renamed that interface.

--snip--
test1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
--snip--

Make this change permanent with the ifconfig_interface_name option in /etc/rc.conf.

ifconfig_em1_name="test1"

FreeBSD renames interfaces early in the boot process, before setting IP addresses or other values. This means that any further interface configuration must reference the new interface name rather than the old. Full configuration of a renamed interface with IP addresses and aliases would look something like this:

ifconfig_em1_name="dmz2"
ifconfig_dmz2="inet 203.0.113.2 netmask 255.255.255.0"
ifconfig_dmz2_alias0="inet 203.0.113.3"

DHCP

Very few networks use DHCP for everything, including servers. A DHCP server will set the server’s IP address, netmask, nameservers, and default gateway for you. If your network administrator configures servers via DHCP, you can tell the network card to take its configuration via DHCP with the following:

ifconfig_em0="DHCP"

Reboot!

Now that you have your network interfaces fully configured, be sure to reboot to test any changes you made to /etc/rc.conf. If FreeBSD finds an error in /etc/rc.conf, especially in network configuration, you’ll have problems accessing the system remotely. It’s much better to learn that you made a typo under controlled conditions as opposed to the middle of your sleeping hours.

If you feel like living dangerously, you can run service netif restart with the interface name to reconfigure only a single interface.

# service netif restart em0

Skip the interface name, and this will restart all interfaces. It’s not a perfect test, but it will catch a bunch of daftness. A reboot is always the best test.

The Domain Name Service

The Domain Name Service, or DNS, is one of those quiet, behind-the-scenes services that doesn’t get half the attention it deserves. Although most users have never heard of it, DNS makes the internet as we know it work. Also called name service, DNS provides a map between hostnames and IP addresses. It also provides the reverse map, of IP addresses to hostnames. Without DNS, web browsers and email programs couldn’t use the nice and convenient hostnames like www.michaelwlucas.com or www.nostarch.com; instead, you’d have to browse the web by typing in appalling things like https://2001:19f0:5c00:9041:225:90ff:fee8:1270. This would greatly reduce the internet’s popularity.² To most end users, a DNS failure is an internet failure, end of story. While we won’t discuss building your own authoritative nameserver, we must cover configuring your server to use DNS.

A host that trawls the internet to dig up DNS mappings is called a nameserver, or DNS server. DNS servers aren’t difficult to run, but most individuals don’t need one. DNS servers are needed only by organizations who run their own servers (and lunatics who have dozens of hosts in their basement, like me). Nameservers come in two varieties, authoritative and recursive.

Authoritative nameservers provide DNS mappings for the public to find an organization’s nameservers. As the operator of michaelwlucas.com, I must provide authoritative nameservers for that domain and let the public query them. These authoritative nameservers answer queries only about the domains I manage. Configuring an authoritative nameserver is beyond the scope of this book.

Recursive nameservers service client requests. When you try to browse to https://www.michaelwlucas.com, your local recursive nameserver searches the internet for my authoritative nameserver. Once the recursive nameserver retrieves the hostname-to-IP mapping, it returns that response to your client. This book shows you how to use recursive nameservers and how to enable your own recursive nameserver.

The system’s resolver is responsible for configuring how the host performs DNS queries and relaying the responses to programs. Configuring the resolver is a vital part of system administration. Even DNS servers need a configured resolver, because the host won’t know it’s a nameserver unless you tell it so. Configuring a resolver requires answering a few questions:

• Where does the server look for DNS information?
• What local overrides do you want?
• What are the local domain names?
• Which nameservers should be queried?

The answers to these questions are configured in /etc/nsswitch.conf and /etc/resolv.conf.

Host/IP Information Sources

This should be easy. A server gets its host information from a nameserver, right? I just spent a few paragraphs telling you that, didn’t I?

The real world isn’t quite that simple, though. Perhaps you have a small home network with only three machines. You want each machine to be able to find each other by hostname, but you don’t want to run a local authoritative nameserver. Or maybe you’re on a large corporate network where completing DNS changes takes weeks, and you have a couple test systems that need to talk to each other. FreeBSD, like all Unix-like operating systems, can get information from both DNS and from the plaintext hosts file /etc/hosts.

When FreeBSD needs to know the address of a host (or the hostname of an address), by default the query goes first to the hosts file and then the configured nameservers. This means that you can locally override nameserver results, which is very useful for hosts behind a NAT or on large corporate networks with odd requirements. In some cases, you might need to reverse this order to query DNS first and the hosts file second. Set this order in /etc/nsswitch.conf.

Each entry in /etc/nsswitch.conf is a single line containing the name of the name service, a colon, and a list of information sources. Here’s the hostname service lookup configuration:

hosts: files dns

The resolver queries the information sources in the order listed. If you have an additional information source, such as nscd(8), list it here. The documentation sources for these add-ons should include the name of the service.

Local Names with /etc/hosts

The /etc/hosts file matches internet addresses to hostnames. Once upon a time, before the Domain Name Service, the internet had a single hosts file that provided the hostnames and IP addresses of every node on the internet. Sysadmins submitted their host changes to a central maintainer, who issued a revised hosts file every few months. Sysadmins would then download the hosts file and install it on all of their machines. This worked fine when the whole internet had four systems on it, and was even acceptable when there were hundreds of hosts. As soon as the internet began its exponential growth, however, this scheme became totally unmaintainable.

While the hosts file is very effective, it works only on the machine it’s installed on and must be maintained by the sysadmin. The public DNS has largely supplanted /etc/hosts, but it’s still useful in environments where you don’t want to run local authoritative DNS³ or you’re behind an IPv4 NAT device. Using the hosts file makes perfect sense if you have one or two servers at home, or if someone else manages your authoritative nameservers. Once you have enough hosts that the thought of updating the hosts file makes you ill, it’s time to learn to build an authoritative nameserver.

Each line in /etc/hosts represents one host. The first entry on each line is an IP address, and the second is the fully qualified domain name of the host, such as mail.michaelwlucas.com. Following these two entries, you can list an arbitrary number of aliases for that host.

For example, a small company might have a single server handling email, serving FTP, web pages, and DNS, as well as performing a variety of other functions. A desktop on that network might have a hosts file entry like this:

203.0.113.3     mail.mycompany.com    mail ftp www dns

With this /etc/hosts entry, the desktop could find the server with either the full domain name or any of the brief aliases listed. This won’t get you to Facebook, however. For that, you need nameservice.

Configuring Nameservice

Tell your host how to query nameservers with the file /etc/resolv.conf. You probably want to provide a local domain or a domain search list and then list the nameservers.

Local Domain and Search List

If your organization has many machines, typing out complete hostnames can quickly get old. If you’re doing maintenance and need to log into every web server, by the time you get to www87.BertJWRegeerHasTooManyBlastedComputers.com you’ll need treatment for impending carpal tunnel syndrome. You can either provide a local domain or a list of domains to search on the first line of /etc/resolv.conf.

The domain keyword tells the resolver which local domain to check, by default, for all hostnames. All of my test hosts are in the domain michaelwlucas.com, so I could set that as the default domain.

domain  michaelwlucas.com

Once you specify a local domain, the resolver will automatically append the domain to any short hostname. If I type ping www, the resolver will append the local domain and send ping(8) to www.michaelwlucas.com. If I give a complete hostname, such as www.bertjwregeer.com, though, the resolver doesn’t add the default domain.

Maybe I have more than one domain I’d like to search. Use the search keyword to give a list of domain names to try, in order. Like domain, search must be the first line of resolv.conf.

search  michaelwlucas.com bertjwregeer.com mwl.io

When you use a brief hostname, such as www, the resolver appends the first domain name in the search list. If there’s no answer, it repeats the query with the second domain name, and then the third. If I run ping petulance, the resolver searches for petulance.michaelwlucas.com, petulance.bertwjregeer.com, and petulance.mwl.io. If no such host exists in any of these domains, the search fails.

If you have neither domain nor search entries in /etc/resolv.conf, but the machine’s hostname includes a domain name, the resolver uses the local machine’s domain name.

The Nameserver List

Now that your resolver knows which domains to try, tell it which nameservers to query. List each in /etc/resolv.conf on its own line, in order of preference. Use the keyword nameserver and the DNS server’s IP address. The resolver queries the listed nameservers in order. A complete resolv.conf might look like this:

domain mwl.io
nameserver 127.0.0.1
nameserver 203.0.113.8
nameserver 192.0.2.8

This resolver is ready to rock.

Note the first nameserver entry, though. The address 127.0.0.1 is always attached to the local host. This machine is running a local recursive nameserver. You can too!

Caching Nameserver

Your host needs to perform a DNS lookup every single time it must contact a host. A busy server makes a whole bunch of queries, and by itself, the resolver doesn’t cache any of these responses. A host needs to connect to Google 500 times in a minute? That’s 500 DNS lookups. While setting up an authoritative DNS server requires a specific skill set, configuring a local recursive server requires only one line in /etc/rc.conf. This lets your FreeBSD host cache its DNS responses while reducing network congestion and improving performance.

Enable the local nameserver with the rc.conf variable local_unbound_enable.

# sysrc local_unbound_enable=YES
local_unbound_enable: NO -> YES

You can now start the local nameserver.

# service local_unbound start

When you start the service the first time, unbound configures itself. It extracts your system’s nameservers from /etc/resolv.conf and configures itself to forward all queries to those nameservers. The setup process then edits /etc/resolv.conf to point all queries at the local nameserver, running on the IP address 127.0.0.1.

When your host makes a DNS query, the resolver queries unbound. The local nameserver checks its cache to see whether it has a valid and unexpired answer for the query. If it doesn’t have a cached response, unbound queries your preferred nameservers.

I recommend enabling local_unbound on every server that isn’t a DNS server.

Network Activity

Now that you’re on the network, how can you see what’s going on? There are several ways to look at the network, and we’ll consider each in turn. Unlike many commercial operating systems, FreeBSD commands such as netstat(8) and sockstat(1) give you more information about the network than can possibly be healthy.

Current Network Activity

The general-purpose network management program netstat(8) displays different information depending on the flags it’s given. One common question people have is, “How much traffic is my system pushing right now?” The netstat(8) -w (for wait) option displays how many packets and bytes your system is processing. The -w flag takes one argument, the number of seconds between updates. Adding the -d (for drop) flag tells netstat(8) to include information about packets that never made it to the system. Here, we ask netstat(8) to update its display every five seconds:

# netstat -w 5 -d
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls drops
       ➊34   ➋0     ➌0    ➍44068        ➎23   ➏0      ➐1518    ➑0   ➒0
         33    0       0     42610          23    0        1518      0    0
--snip--

Nothing appears to happen when you enter this command, but in a few seconds, the display prints a single line of information. The first three columns describe inbound traffic, while the next three describe outbound traffic. We see the number of packets received since the last update ➊, the number of interface errors for inbound traffic since the last update ➋, and the number of inbound dropped packets ➌. The input information ends with the number of bytes received since the last update ➍. The next three columns show the number of packets the machine transmitted since the last update ➎, the number of errors in transmission since the last update ➏, and how many bytes we sent ➐. We then see the number of network collisions that have occurred since the last update ➑, and the number of packets that have been dropped ➒. For example, in this display, the system received 34 packets ➊ since netstat -w 5 -d started running.

Five seconds later, netstat(8) prints a second line describing the activity since the first line was printed.

You can make the output as detailed as you want and run it as long as you like. If you’d like to get updates every second, just run netstat -w 1 -d. If once a minute is good enough for you, netstat -w 60 -d will do the trick. I find a five-second interval most suitable when I’m actively watching the network, but you’ll quickly learn what best fits your network and your problems.

Hit CTRL-C to stop the report once you’ve had enough.

What’s Listening on Which Port?

Another popular question is, “Which ports are open and what programs are listening on them?” FreeBSD includes sockstat(1), a friendly tool to answer this question. It shows both active connections and ports available for client use.

The sockstat(1) program not only lists ports listening to the network, but also any other ports (or sockets) on the system. Use the -4 flag to see IPv4 sockets and -6 to view IPv6. Here’s trimmed sockstat(1) output from a very small server:

The first column gives us the username that’s running the program attached to the port in question. The second column is the name of the command. We then have the process ID of the program and the file descriptor number attached to the socket. The next column shows what transport protocol the socket uses—either tcp4 for TCP on TCP/IP version 4, or udp4 for UDP on TCP/IP version 4. We then list the local IP address and port number, and finally the remote IP address and port number for each existing connection.

Take a look at our very first entry. I’m running the program sshd ➊. A man page search takes you to sshd(8), the SSH daemon. The main sshd(8) daemon forked a child process on my behalf to handle my connection, so we see multiple instances of sshd(8) with different process IDs. I’m connected to the local IP address 203.0.113.43 ➋ on TCP port 22. The remote end of this connection is at the IP address 24.192.127.92 ➌ on port 62937. This is an SSH connection from a remote system to the local computer.

Other available connections include Sendmail ➍, the mail server, running on port 25. Note that this entry doesn’t have any IP address listed as the foreign address. This socket is listening for incoming connections. Our httpd process ➎ is listening for incoming connections on port 80.

The astute among you might notice that this server has two SSH daemons available for incoming connections, one on port 23 ➏ and one on port 22 ➐. As /etc/services shows, SSH normally runs on port 22 while port 23 is reserved for telnet. Anyone who telnets to this machine will be connected to an SSH daemon, which won’t work as they expect. The suspicious among you might suspect that this SSH server was set up to waltz around firewalls that only filter traffic based on source and destination ports and not the actual protocol. (I have no comment on such allegations.)

The last two entries are for a nameserver, named, awaiting incoming connections on port 53. This nameserver is listening for both UDP ➑ and TCP ➒ connections and is attached to the single IP address 203.0.113.8.

Port Listeners in Detail

While sockstat(1) provides a nice high-level view of network service availability, you can get a little more detailed information about individual connections with netstat(8). To view open network connections, use netstat(8)’s -a flag. The -n flag tells netstat(8) not to bother translating IP addresses to hostnames; not only can this translation slow down the output, it can cause ambiguous output. Finally, the -f inet option tells netstat(8) to worry only about IPv4 network connections, while -f inet6 addresses IPv6. Here’s matching netstat output from the same machine we just ran sockstat(1) on:

# netstat -na -f inet
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4       0     48  203.0.113.43.22        24.192.127.92.62937    ESTABLISHED
tcp4       0      0  *.25                   *.*                    LISTEN
tcp4       0      0  *.23                   *.*                    LISTEN
tcp4       0      0  *.80                   *.*                    LISTEN
tcp4       0      0  *.22                   *.*                    LISTEN
tcp4       0      0  203.0.113.8.53         *.*                    LISTEN
udp4       0      0  203.0.113.8.53         *.*  

Here, we get no idea of what program is attached to any port. The first entry in each column is the transport protocol used by the socket—mostly TCP, but the last line shows UDP.

The Recv-Q and Send-Q columns show the number of bytes waiting to be handled by this connection. If you see nonzero Recv-Q numbers for some connection most of the time, you know that the program listening on that port can’t process incoming data quickly enough to keep up with the network stack. Similarly, if the Send-Q column keeps having nonzero entries, you know that either the network or the remote system can’t accept data as quickly as you’re sending it. Occasional queued packets are normal, but if they don’t go away, you might want to investigate why things are slow. You must watch your own system to learn what’s normal.

The Local Address is, as you might guess, the IP address and network port number on the local system that the network connection is listening on. The network port appears at the end of the entry and is separated from the IP address by a dot. For example, 203.0.113.43.22 is the IP address 203.0.113.43, port 22. If the entry is an asterisk followed by a period and a port number, that means that the system is listening on that port on all available IP addresses. The system is ready to accept a connection on that port.

The Foreign Address column shows the remote address and port number of any connection.

Finally, the (state) column shows the status of the TCP handshake. You don’t need to know all of the possible TCP connection states, so long as you learn what’s normal. ESTABLISHED means that a connection exists and that data is probably flowing. LAST_ACK, FIN_WAIT_1, and FIN_WAIT_2 mean that the connection is being closed. SYN_RCVD, ACK, and SYN+ACK are all parts of connection creation (the three-way handshake from Chapter 7). LISTEN indicates that the port is ready for incoming connections. In the preceding example, one TCP connection is running and four are ready to accept clients. As UDP is stateless, those connections list no state information.

By reading this output and combining it with information provided by sockstat(1), you can learn exactly which programs are behaving well and which are suffering bottlenecks.

If you’re not interested in listening sockets but only those with active connections, use netstat(8)’s -b option instead of -a. Running netstat -nb -f inet displays only connections with foreign systems.

You can also use netstat -T to display TCP retransmits and out-of-order packets on individual connections. Retransmits and misordered packets are symptoms of dropped packets.

Network Capacity in the Kernel

The FreeBSD kernel handles network memory by using mbufs. An mbuf is a chunk of kernel memory used for networking. You’ll keep tripping across mentions of mbufs throughout the FreeBSD network stack documentation, so it’s important to have at least a vague idea of them.

FreeBSD automatically allocates network capacity at boot time based on the amount of physical RAM in the system. We assume that if you have a system with 64GB RAM, you want to use more memory for networking than on a little box with 1GB RAM. View how FreeBSD uses its resources with netstat -s and netstat -m. Let’s look at the shortest one first.

To get a generic view of kernel memory used for networking, run netstat -m. The output can be divided into two general categories: how much is used and how many requests failed. The following output is trimmed to include only a few examples of these, but they all follow the same general format:

# netstat -m
--snip--
➊32/372/404/➋25600 mbuf clusters in use (current/cache/total/max)
--snip--
0/0/0 requests for mbufs ➌denied (mbufs/clusters/mbuf+clusters)
--snip--

Here we see how many mbuf clusters are used ➊. You’d probably guess that these are related to mbufs, and you’d be right. You don’t have to know exactly what mbuf clusters are; the important thing is that you know how many you can allocate ➋ and can see that you’re under that limit.

Similarly, we can see how many different requests for mbufs the kernel has denied ➌. This system hasn’t rejected any requests for mbufs, which means that we aren’t having performance problems due to memory shortages. If your system starts rejecting mbuf requests because it’s out of memory, you’re in trouble. See “Optimizing Network Performance” next.

While netstat -m produces a dozen lines of output, netstat -s runs for pages and pages. It provides per-protocol performance statistics. Much like netstat -m, you can break up these statistics into categories of how much was done and how many problems you had. Run both of these commands occasionally on your systems and review the results so you know what passes for normal on your servers and can recognize abnormal numbers when you have problems.

Optimizing Network Performance

Now that you can see what’s going on, how could you improve FreeBSD’s network performance? There’s a simple rule of thumb when considering optimizing: don’t. Network performance is generally limited only by your hardware. Many applications can’t process data as quickly as your network can provide. If you think that you need to optimize your performance, you’re probably looking in the wrong spot. Check Chapter 21 for hints on investigating performance bottlenecks.

Generally speaking, network performance should be adjusted only when you experience network problems. This means that you should have output from netstat -m or netstat -s indicating that the kernel is having resource problems. If the kernel starts denying requests for resources or dropping connection requests, look at the hints in this section. If you have issues or if you think you should be getting better performance, look at the hardware first.

Optimizing Network Hardware

Not all network hardware is created equal. While anyone in IT hears this frequently, FreeBSD’s open nature makes this obvious. For example, here’s a comment from the source code of the rl(4) network card driver:

The RealTek 8139 PCI NIC redefines the meaning of 'low end.' This is
probably the worst PCI ethernet controller ever made, with the possible
exception of the FEAST chip made by SMC. The 8139 supports bus-master
DMA, but it has a terrible interface that nullifies any performance
gains that bus-master DMA usually offers.

This can be summarized as, “This card sucks and blows at the same time. Buy another card.” While this is the most vitriolic comment that I’ve seen in the FreeBSD source code, and this particular hardware is very hard to find today, the drivers for certain other cards say the same thing in a more polite manner. Optimizing network performance with low-end hardware is like putting a high-performance racing transmission in your 1974 Gremlin. Replacing your cheap network card will probably fix your problems. Generally speaking, Intel makes decent network cards; they maintain a FreeBSD driver for their wired network cards and provide support so that the FreeBSD community can help maintain the drivers. (Wireless cards are another story.) Similarly, many companies that build server-grade machines make a point of using server-grade network cards. Some companies provide a FreeBSD driver but do not provide documentation for their hardware. This means that the driver probably works, but you’re entirely dependent upon the vendor’s future fondness of FreeBSD for your updates. Companies that specialize in inexpensive consumer network equipment are not your best choice for high-performance cards—after all, the average home user has no idea how to pick a network card, so they go by price alone. If in doubt, check the FreeBSD-questions mailing list archives for recent network card recommendations.

Similarly, switch quality varies wildly. The claim that a switch speaks the protocol used in gigabit connections doesn’t mean that you can actually push gigabit speed through every port! I have a 100Mb switch that bottlenecks at 15Mbps and a “gigabit” switch that seems to choke at about 50Mbps. I recommend that you think of a switch’s speed as a protocol or a language: I could claim that I speak Russian, but 30 years after my studies ceased, my speech bottlenecks at about three words a minute. My Russian language interface is of terrible quality. Again, switches designed for home use are not your best choice in a production environment.

If getting decent hardware doesn’t solve your problems, read on.

Memory Usage

FreeBSD uses the amount of memory installed in a system to decide how much memory space to reserve for mbufs. Don’t adjust the number of mbufs you create unless netstat -m tells you that you’re short on mbuf space. If you have an mbuf problem, the real fix is to add memory to your machine. This will make FreeBSD recompute the number of mbufs created at boot and solve your problem. Otherwise, you’ll just shift the problem to a different part of the system or a different application. You might configure gobs of memory for network connections and find that you’ve smothered your database server. If you’re sure you want to proceed, though, here’s how you do it.

Two sysctl values control mbuf allocation, kern.maxusers and kern.ipc.nmbclusters. The first, kern.maxusers, is a boot-time tunable. Your system automatically determines an appropriate kern.maxusers value from the system hardware at boot time. Adjusting this value is probably the best way to scale your system as a whole. In older versions of FreeBSD, kern.maxusers preallocated memory for networking and refused to release it for other tasks, so increasing kern.maxusers could badly impact other parts of the system. Modern FreeBSD does not preallocate network memory, however, so this is just an upper limit on networking memory. If kern.maxusers is too small, you’ll get warnings in /var/log/messages (see Chapter 21).

The sysctl kern.ipc.nmbclusters specifically controls the number of mbufs allocated by the system for data sitting in socket buffers, waiting to be sent to or read by an application. Although this is runtime tunable, it’s best to set it early at boot by defining it in /etc/sysctl.conf (see Chapter 6). If you set this too high, however, you can actually starve the kernel of memory for other tasks and panic the machine.

# sysctl kern.ipc.nmbclusters
kern.ipc.nmbclusters: 25600

Mbufs are allocated in units called nmbclusters (sometimes called mbuf clusters). While the size of an mbuf varies, one cluster is about 2KB. You can use simple math to figure out how much RAM your current nmbcluster setting requires and then calculate sensible values for your system and applications. This example machine has 25,600 nmbclusters, which means the kernel can use up to about 50MB RAM for networking purposes. This is negligible on my test laptop’s gig of RAM, but it might be unsuitable on an embedded system.

To calculate an appropriate number of mbuf clusters, run netstat -m when the server is really busy. The second line of the output will give you the number of mbufs in use and the total number available. If your server at its busiest doesn’t use nearly as many nmbclusters as it has available, you’re barking up the wrong tree—stop futzing with mbufs and replace your hardware already.⁴ For example:

➊32/➋372/➌404/➍25600 mbuf clusters in use (current/cache/total/max)

This system is currently using 32 nmbclusters ➊ on this machine and has cached 372 previously used nmbclusters ➋. With this total of 404 clusters ➌ in memory at this time, our capacity of 25,600 clusters ➍ is 1.5 percent utilized. If this is your real system load, actually reducing the number of nmbclusters might make sense.

My personal rule of thumb is that a server should have enough mbufs to handle twice its standard high load. If your server uses 25,000 nmbclusters during peak hours, it should have at least 50,000 available to handle those brief irregular peaks.

Maximum Incoming Connections

The FreeBSD kernel provides capacity to handle a certain number of incoming new TCP connections. This doesn’t refer to connections that the server previously received and is handling, but rather to clients who are attempting to initiate connections simultaneously. For example, the web pages currently being delivered to clients don’t count, but the incoming requests that have reached the kernel but not the web server process do. It’s a very narrow window, but some sites do overflow it.

The sysctl kern.ipc.somaxconn dictates how many simultaneous connection attempts the system will try to handle. This defaults to 128, which might not be enough for a highly loaded web server. If you’re running a high-capacity server where you expect more than 128 new requests to be arriving simultaneously, you probably need to increase this sysctl. If users start complaining that they can’t connect, this might be your culprit. Of course, very few applications will accept that many simultaneous new connections; you’ll probably have to tune your app well before you hit this point.

Polling

Some gigabit cards can improve their performance with polling. Polling takes the time-honored idea of interrupts and IRQs and boots it out the window, replacing it with regular checks for network activity. In the classic interrupt-driven model, whenever a packet arrives at the network card, the card demands attention from the CPU by generating an interrupt. The CPU stops whatever it’s doing and handles that data. This is grand, and even desirable, when the card doesn’t process a huge amount of traffic. Once a system starts handling large amounts of data, however, the card generates interrupts continuously. Instead of constantly interrupting, the system is more efficient if the kernel grabs network data from the card at regular intervals. This regular checking is called polling. Generally speaking, polling is useful only if you push large amounts of traffic.

Polling isn’t available as a kernel module as of this writing, since it requires modifications to device drivers. This also means that not all network cards support polling, so be sure to check polling(4) for the complete list. Enable polling by adding DEVICE_POLLING to your kernel configuration. After your reboot, enable polling on a per-interface basis with ifconfig(8).

# ifconfig em0 polling

Similarly, disable polling with the argument -polling. The ifconfig(8) command also displays if polling is enabled on an interface. As you can enable and disable polling on the fly, enable polling when your system is under a heavy load and see whether performance improves.

Polling is used only on older cards. 10GB cards and faster can’t poll.

Other Optimizations

FreeBSD has about 200 networking-related sysctls. You have all the tools you need to optimize your system so greatly that it no longer passes any traffic at all. Be very careful when playing with network optimizations. Many settings that seem to fix problems actually fix only one set of problems while introducing a whole new spectrum of issues.

Some software vendors (i.e., Samba) recommend particular network sysctl changes. Try them cautiously, and watch for unexpected side effects on other programs before accepting them as your new default. TCP/IP is a terribly, terribly complicated protocol, and FreeBSD’s defaults and autotuning reflect years of experience, testing, and sysadmin suffering.

Also remember that FreeBSD is over two decades old. Mailing list and forum posts from more than a few years ago are probably not useful in network tuning.

Network Adapter Teaming

As network servers become more and more vital to business, redundancy becomes more important. We have redundant hard drives in a server and redundant bandwidth into a data center, but what about redundant bandwidth into a server? On a smaller scale, as you move around your office, you might move your laptop between wired and wireless connections. It would be really nice not to lose your existing SSH sessions because you unplugged a cable.

FreeBSD can treat two network cards as a single entity, allowing you to have multiple connections with a single switch. This is commonly called network adapter teaming, bonding, or link aggregation. FreeBSD implements adapter teaming through lagg(4), the link aggregation interface.

The kernel module lagg(4) provides a lagg0 virtual interface. You assign physical interfaces to the lagg0 interface, making them part of the aggregated link. While you could use lagg(4) with only one physical interface, aggregating links only makes sense when you have two or more physical interfaces to assign to the aggregated link. The lagg(4) module allows you to implement seamless roaming between wired and wireless networks, failover, and several different aggregation protocols.

Aggregation Protocols

Not all network switches support all link aggregation protocols. FreeBSD has basic implementation of some complicated high-end protocols and also includes very basic failover setups. The three I recommend are Fast EtherChannel, LACP, and failover. (There are more schemes, which you can read about in lagg(4).)

Cisco’s Fast EtherChannel (FEC) is a reliable link aggregation protocol, but it works only on high- to medium-end Cisco switches running particular versions of Cisco’s operating system. If you have an unmanaged switch, Fast EtherChannel is not a viable choice. Fast EtherChannel is complicated to configure (on the switch), so I recommend FEC only when it is already your corporate standard for link aggregation.

The Link Aggregation Control Protocol (LACP) is an industry standard for link aggregation. The physical interfaces are bonded into a single virtual interface with approximately the same bandwidth as all of the individual links combined. LACP provides excellent fault tolerance, and almost all switches support it. I recommend LACP unless you have a specific requirement for Fast EtherChannel or a switch that chokes when you use LACP.

If you do have a switch that chokes on LACP, use failover. The failover method sends traffic through one physical interface at a time. If that interface goes down, the connection fails over to the next connection in the pool. While you don’t get aggregated bandwidth, you do get the ability to attach your server to multiple switches for fault tolerance. Use failover to let your laptop roam between wired and wireless connections.

Configuring lagg(4)

The lagg interface is virtual, meaning there is no physical part of the machine that you could point to and say, “That is interface lagg0.” Before you can configure the interface, you must create it. FreeBSD lets you create interfaces with ifconfig interfacename create, but you can also do this in /etc/rc.conf with the cloned_interfaces statement.

Configuring a lagg(4) interface in rc.conf has three steps: creating the interface, bringing up the physical interfaces, and aggregating them. Here, we create a single lagg0 interface out of two Intel gigabit Ethernet cards, em0 and em1.

cloned_interfaces="lagg0"
ifconfig_em0="up"
ifconfig_em1="up"
ifconfig_lagg0="laggproto lacp laggport em0 laggport em1 inet 203.0.113.1/24"

First, you list lagg0 as a cloned interface, so FreeBSD will create this interface at boot. Then, bring interfaces em0 and em1 up, but don’t configure them. Finally, tell the lagg0 interface what aggregation protocol to use (LACP), what physical interfaces belong to it, and its network information. These few lines of configuration give you a high-availability Ethernet connection.

Virtual LANs

A virtual LAN, or VLAN, lets you get multiple Ethernet segments on a single piece of wire. You’ll sometimes see VLANs called 802.1q, tagging, or a combination of these terms. You can use these multiple networks by configuring additional logical interfaces attached to a physical interface. The physical wire can still carry only so much data, however, so all VLANs and the regular network (or native VLAN ) that share the wire use a common pool of bandwidth. If you need a FreeBSD host on multiple Ethernet segments simultaneously, this is one way to do it without running more cable.

VLAN frames that arrive at your network card are like regular Ethernet frames, tagged with an additional header that says “This is part of VLAN number whatever.” Each VLAN is identified by a tag from 1 to 4096. The native VLAN arrives without any tagging whatsoever. The network often (but not always) calls this VLAN 1 internally.

Configuring a VLAN on your FreeBSD host doesn’t magically connect the host to the VLAN. The network must be configured to send those VLANs to your host. You must work with the network team to get access to the VLANs.

Configuring VLAN Devices

Use ifconfig(8) to create VLAN interfaces. You must know the physical interface and the VLAN tag.

# ifconfig interface.tag create vlan tag vlandev interface

Here, I create an interface for VLAN 2 and attach it to the interface em0.

# ifconfig em0.2 create vlan 2 vlandev em0

I can now configure interface em0.2 as I would a physical interface.

# ifconfig em0.2 inet 192.0.2.236/28

In reality, I’d probably do all of this in a single command.

# ifconfig em0.2 create vlan 2 vlandev em0 inet 192.0.2.236/28

That’s everything. Now use ifconfig(8) to display your new interface.

# ifconfig em0.2
em0.2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=503<RXCSUM,TXCSUM,TSO4,LRO>
        ether 00:25:90:db:d5:94
        inet 198.22.65.236 netmask 0xffffff00 broadcast 255.255.255.240
        inet6 fe80::225:90ff:fedb:d594%em0.2 prefixlen 64 scopeid 0x6
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
     ➊ media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
     ➋ vlan: 2 vlanpcp: 0 parent interface: em0
     ➌ groups: vlan

This looks almost exactly like any other physical interface. The media information ➊ comes directly from the underlying interface. You’ll see a label with VLAN information ➋ and a note that this is grouped with the other VLAN interfaces ➌.

Configuring VLANs at Boot

Configure VLANs at boot with rc.conf variables. First, use a vlan_ variable tagged with the interface name to list the VLANs attached to that interface. Here, I tell FreeBSD to enable VLAN 2 and 3 on interface em0 and assign an IP configuration to each.

vlans_em0="2 3"
ifconfig_em0_2="inet 192.0.2.236/28"
ifconfig_em0_3="inet 198.51.100.50/24"

If the underlying interface has no configuration, you need to at least bring it up. The VLAN interfaces won’t work unless the physical interface is on.

ifconfig_em0="up"

You now have virtual LANs at boot. Congratulations!

Now that you have a working network, let’s get a little more local and look at basic system security.