19
ADVANCED SECURITY FEATURES

FreeBSD includes a variety of tools for securing network traffic and users. Some of these tools are invisible to sysadmins but work behind the scenes to increase security, such as the sandboxing API capsicum(4). Packet filtering lets you control who can access your system. You can also use blacklisting to block network addresses that keep poking at your host. In addition, FreeBSD has a whole bunch of optional security features you can enable either in the installation process or later. In this chapter, we’ll examine these tools and techniques, look at monitoring your system’s security, and discuss how to react if you suffer an intrusion.

Let’s start with a core security topic: unprivileged users.

Unprivileged Users

An unprivileged user is a specific user for a specific task. He has only the rights necessary to perform that limited task. Many programs run as unprivileged users or use unprivileged users to perform specific duties.

“Only the rights needed to perform its duties” sounds like every user account, doesn’t it? That’s true, but the account used by the least privileged human being still has more rights than many programs need. Anyone with shell access has a home directory. The normal user may create files in their home directory, run text editors, or process email. Your average shell user needs these minimal privileges, but programs do not. By having a program, particularly a network daemon, run as a very restricted user, you control the amount of damage an intruder can do to either the program or the user.

FreeBSD includes several unprivileged users. Take a look at /etc/passwd and you’ll see accounts like audit, bind, uucp, and www. These are all unprivileged accounts for use by specific server daemons. See what they have in common.

Unprivileged users don’t have normal home directories. Many have a home directory of /nonexistent, while others, such as sshd, have a special home directory such as /var/empty. Having a home directory where you may not write or read files makes the account less flexible but good enough for a server daemon. These users do own files on the system, but they usually can’t write to those files.

Similarly, nobody should ever log into these accounts. If the account bind is reserved for the DNS system, nobody should actually log into the system as that user! Such an account must have a user shell that specifically denies logging in, like /usr/sbin/nologin. How does all this enhance system security? Let’s look at an example.

Whatever web server you’re using, it generally runs under the unprivileged account www. Suppose that an intruder discovered a security flaw in the version of the web server program you’re using and can make the web server execute arbitrary code. This is among the worst types of security problems, where an intruder can make the server program do absolutely anything within its power. What is within this program’s power?

The intruder probably wants a command prompt on the system. A command prompt on a Unix-like system is the door to so much more access, after all. The unprivileged user has an assigned shell that specifically disallows logins. This really annoys intruders and requires them to work much harder to reach that command prompt.

If she’s really clever, though, the nologin shell won’t stop the intruder. Let’s assume that through clever trickery she makes the web server execute a simple shell, such as /bin/sh, and offer her the prompt. She’s in and can wreak untold damage . . . or can she?

She has no home directory and doesn’t have permissions to create one. That means that any files she wants to store must go in a globally accessible directory, such as /tmp or /var/tmp, increasing her visibility. The Apache configuration file is owned by root or by your web server administration group, and the www user isn’t part of that group. The intruder might have a path into the web server, but she can’t reconfigure it. She can’t change the website files, as the www user doesn’t own them. The www user doesn’t have access to anything on the system except the web server itself. A sufficiently skilled intruder can make the web server serve up different pages or redirect to another site, at least until a reboot. Penetrating the application running on the server, or the host itself, requires another whole set of security breaches.

An unprivileged user doesn’t solve all security problems, mind you. Our compromised www user can view web application source files. If your application is badly written or has database passwords hardcoded into hidden files, you’re still in a lot of trouble. Still, if you’ve kept your system updated and all your packages up-to-date, an intruder will have a very hard time penetrating FreeBSD itself.

The nobody Account

For years, system administrators used the account nobody as a generic unprivileged user. They’d run web servers, proxy servers, and whatever else as nobody. This was better than running those programs as root, but not as good as having separate users for each daemon. If an intruder successfully penetrated one of these programs, he had access to them all. Our hypothetical web server intruder would abruptly have access not only to the web server but also to whatever other programs run as that same user! If you’re using NFS, remember that NFS defaults to mapping remote root accounts to nobody. The whole point of using unprivileged users is to minimize the possible damage from a successful intrusion.

While you might test with the nobody account, never deploy production services with it. Use separate unprivileged accounts liberally.

A Sample Unprivileged User

Here are parameters useful for a generic unprivileged user:

Username Assign a username related to the user’s function. For example, the default user for web servers is www.

Home directory Unprivileged users should deliberately not have a home directory, so use /nonexistent. Do not create a /nonexistent directory either; the whole point is that it doesn’t exist!

Shell Unpriviliged users must not have a shell that can execute commands, so use /usr/sbin/nologin.

UID/GID Choose a special range of user and group IDs for unprivileged users.

Full name Assign a name describing the user’s function.

Password Use chpass(1) to assign the user a single asterisk as their encrypted password. This disables the account password. (Note that chpass(1) stands for change password file, not change password!)

These settings make your unprivileged user very unprivileged indeed. You can set all of this easily with adduser(8), giving the account no password, the correct home directory, and an appropriate shell.

Many ports and packages have assigned unprivileged users and groups, listed in /usr/ports/UIDs and /usr/ports/GIDs. Don’t be afraid to add more. Use UIDs above 1,000, so as not to conflict with those assigned by packages and FreeBSD’s core.

Network Traffic Control

Sysadminis must have the ability to control traffic to and from their systems. Unwanted visitors must be stopped while legitimate users get access. FreeBSD provides a variety of tools that allow you to control outside access to your systems, including TCP wrappers, packet filtering, and blacklisting.

The TCP wrappers, or simply wrappers, control access to network daemons. While the program must be written to support TCP wrappers, most modern software has supported wrappers for many years. Wrappers are fairly simple to configure and don’t require much networking knowledge. As access controls go, however, wrappers are fairly limited. Wrappers do let you do interesting things with connections and with daemons offering connections, though, which is why we’ll discuss it.

Packet filtering controls which traffic the system allows to pass through it and which traffic it rejects. Most firewalls are packet filters with a pretty GUI on top, but you can use FreeBSD packet filtering and proxy software to build a solid firewall in and of itself. A rejected connection request never reaches any userland program; it’s blocked in the network stack. Packet filtering can control access to any program, service, or network port but does require more networking knowledge.

Blacklisting is useful when you want a program to be able to decide to stop listening to a remote host. The most common tool for blacklisting is fail2ban (https://www.fail2ban.org/), which is flexible but requires much special configuration. FreeBSD includes blacklistd, an easier-to-configure blacklisting tool that requires integration with programs that use it.

Which should you use? For basic TCP/IP access control, I recommend always using a packet filter. Only use TCP wrappers if you need their specific features. I discuss blocking and allowing connections with TCP wrappers only as a prerequisite to those advanced features. If you want a service to block clients after a certain number of failed connection attempts, consider blacklistd.

With wrappers or packet filtering, you must decide whether you want a default accept or default deny traffic control policy.

Default Accept vs. Default Deny

One of the essential decisions in any security policy is between default accept and default deny. A default accept security stance means that you allow any type of connection except what you specifically disallow. A default deny stance means that you allow connections only from specified parts of the internet and/or to specified services and that you refuse all other connections. The default is used unless you make a specific rule dictating otherwise. Once you’ve chosen your default security stance, you create exceptions one way or another to either provide or block services as necessary. The choice is really between whether you offer services to the world (default accept) or only to a select few (default deny).

For example, company policy might dictate that the intranet web server must be accessible only from within the company. If so, adopt a default deny stance and explicitly list who may access the server. Alternatively, if you have a public website but want to block certain parts of the internet from accessing it for whatever reason, adopt a default accept stance.

I always recommend a default deny stance. If you don’t make a choice, however, you’ve chosen default accept.

Choosing a default doesn’t mean that the default must be implemented without exceptions. My public web servers have a default deny security stance, but I specifically allow the world to access the websites. The machine rejects attempts to connect to other programs unless they come from one of a few specified IP addresses. This is a perfectly acceptable default deny stance.

Different security tools implement these stances in different ways. For example, with TCP wrappers, the first matching rule is applied. If your last rule denies everything, you’ve established a policy that says, “Unless I’ve specifically created a rule earlier to permit this traffic, block it.” On the other hand, with the PF packet filter, the last matching rule applies. If your first rule says, “Block all traffic,” you’ve implemented a policy that says, “Unless I specifically create a later rule to permit this traffic, block it.”

Both default accept and default deny annoy the sysadmin. If you have a default accept policy, you’ll spend your time continually plugging holes. If you choose a default deny policy, you’ll spend your time opening access for people. You’ll repeatedly apologize for either choice. With default deny, you’ll say things like, “I’ve just activated service for you. I apologize for the inconvenience.” With default accept, you’ll say things like, “. . . and that’s why the intruders were able to access our internal accounting database and why we lost millions of dollars.” In the latter case, “I apologize for the inconvenience” really doesn’t suffice.

TCP Wrappers

Remember from Chapter 7 that network connections are made to various programs that listen for connection requests. When a program is built with TCP wrappers support, the program checks the incoming request against the wrappers configuration. If the wrappers configuration says to reject the connection, the program immediately drops the request. Despite the name, TCP wrappers work with both TCP and UDP connections. Wrappers are a long-running Unix standard that have been incorporated into FreeBSD. Individual programs might or might not work with wrappers; while just about everything in the base FreeBSD system does, some third-party software doesn’t.

TCP wrappers are implemented as a shared library, called libwrap. As seen in Chapter 17, shared libraries are small chunks of code that can be shared between programs. Any program that links with libwrap may use the TCP wrappers functions.

Wrappers most commonly protect inetd(8), the super server that handles network requests for smaller programs. We’ll discuss inetd in Chapter 20. While our examples cover inetd(8), you can protect any other program that supports wrappers in exactly the same way. While wrappers help protect inetd(8), make sure inetd(8) doesn’t offer any unnecessary services, just as you do for the main system.

Configuring Wrappers

Wrappers check each incoming connection request against the rules in /etc/hosts.allow, in order. The first matching rule is applied, and processing stops immediately. This makes rule order very important. Each rule is on a separate line and is made up of three parts separated by colons: a daemon name, a client list, and a list of options. Here’s a sample rule:

ftpd : all : deny

The daemon name is ftpd; the client list is all, meaning all hosts; and the option is deny, telling wrappers to reject all connections. Nobody can connect to the FTP server on this host unless an earlier rule explicitly grants access.

In the early examples, I refer to only two options: accept and deny. They allow and reject connections, respectively. We’ll discuss the additional options later.

Daemon Name

The daemon name is the program’s name as it appears on the command line. For example, inetd(8) starts the ftpd(8) program when it receives an incoming FTP request. The Apache web server starts a program called httpd, so if your version of Apache supports wrappers, give the daemon name as httpd. (Note that Apache doesn’t run out of inetd, but it can support wrappers anyway.) One special daemon name, ALL, matches all daemons that support wrappers.

If your system has multiple IP addresses, you can specify, as part of the daemon name, different wrapper rules for each IP address that a daemon listens on:

[email protected] : ALL : deny
[email protected] : ALL : accept

In this example, we have two daemon names, [email protected] and [email protected]. Each has a separate TCP wrapper rule.

The Client List

The client list is a list of specific IP addresses, network address blocks, hostnames, domain names, and keywords, separated by spaces. Hostnames and IP addresses are simple: just list them.

ALL : netmanager.absolutefreebsd.com 203.0.113.5 : allow

With this rule at the top of /etc/hosts.allow, wrappers allow my netmanager machine and any host with an IP address of 203.0.113.5 to connect to any service on this host. (I could block this access by other means, mind you.)

Specify network numbers in the client list with a slash between the IP address and the netmask, as discussed in Chapter 7. For example, if script kiddies attack your server from a bunch of addresses that begin with 192.0.2, you could block them like this:

ALL : 192.0.2.0/255.255.255.0 : deny

You can also use domain names in client lists by prefacing them with a dot. This works through reverse DNS, which means that anyone who controls the DNS server for a block of addresses can evade this restriction.

ALL : .mycompany.com : allow

If you have a long list of clients, you can even list them in a file and put the full path to the file in the client space in /etc/hosts.allow. I’ve been on networks with large numbers of widely scattered hosts, such as an ISP or corporate network environment with network management workstations scattered across the world. Each workstation shared the same wrapper rules as every other workstation and appeared on half a dozen lines in hosts.allow. By maintaining a single file with a workstation list, I could centralize all changes.

In addition to specifically listing client addresses and names, wrappers provide several special client keywords to add groups of clients to your list. Table 19-1 shows the keywords and their usage.

Most of the client keywords listed in Table 19-1 require a working DNS server. If you use these keywords, you must have a very reliable DNS service, and you must remember the vital link between DNS and the rest of your programs. If your DNS server fails, daemons that use wrappers and those keywords can’t identify any hosts. This means that everything matches your UNKNOWN rule, which probably denies the connection. Also, broken DNS on the client end can deny remote users access to your servers, as your DNS servers won’t be able to get proper information from the client’s servers. Finally, if you use DNS-based wrapping extensively, an intruder needs only to overload your nameserver or otherwise interrupt your nameserver to create a very effective denial-of-service attack against your network.

Table 19-1: TCP Wrapper Keywords

Keyword	Usage
ALL	This matches every possible host.
LOCAL	This matches every machine whose hostname does not include a dot. Generally, this means machines in the local domain. Machines on the other side of the world who happen to share your domain name are considered “local” under this rule.
UNKNOWN	This matches machines with unidentifiable hostnames or usernames. As a general rule, any host making an IP connection has a known IP address. Tracing hostnames, however, requires DNS, and tracking usernames requires identd(8). Be very careful using this option because transitory DNS issues can make even local hostnames unresolvable and most hosts don’t run identd(8) by default. You don’t want a service to become unusable just because your nameserver was misconfigured—especially if that machine is your nameserver!
KNOWN	This matches any host with a determinable hostname and IP address. Be very careful using this, as DNS outages can interrupt service.
PARANOID	This matches any host whose name does not match its IP address. You might receive a connection from a host with an IP address of 192.168.84.3 that claims to be called mail.michaelwlucas.com. Wrappers turn around and check the IP address of mail.michaelwlucas.com. If wrappers get a different IP address, the host matches this rule. Sysadmins who do not have time to maintain their DNS are the most likely to have unpatched, insecure systems.

TCP wrappers provide additional keywords, but they’re not as useful or secure as these. For example, it’s possible to allow connections based on the username on the remote machine. You don’t want to rely on a client username on a remote machine, however. For example, if I set up wrappers to allow only someone with a username of mwlucas to connect to my home system, someone could easily add an account of that name to his FreeBSD system and get right in. Also, this relies on the same rarely used identd(1) protocol that was mentioned earlier. You can find a few other obscure keywords of similar usefulness in hosts_access(5).

The ALL and ALL EXCEPT Keywords

Both daemon names and client lists can use the ALL and ALL EXCEPT keywords. The ALL keyword matches absolutely everything. For example, the default hosts.allow starts with a rule that permits all connections, from all locations, to any daemon:

ALL : ALL : accept

This matches all programs and all clients. You can limit this by giving a specific name to either the client list or the daemon list.

ALL : 203.0.113.87 : deny

In this example, we reject all connections from the host 203.0.113.87.

Categorically blocking access to all hosts isn’t that great an idea, but remember that TCP wrappers follow rules in order and quit when they reach the first matching rule. The ALL keyword lets you set a default stance quite easily. Consider the following ruleset:

ALL : 192.168.8.3 192.168.8.4 : accept
ftpd : ALL : accept
ALL : ALL : deny

Our workstations 192.168.8.3 and 192.168.8.4 (probably the sysadmin’s workstations) may access anything they want. Anyone in the world may access the FTP server. Finally, we drop all other connections. This is a useful default deny stance.

Use the ALL EXCEPT keyword to compress rules. ALL EXCEPT allows you to list hosts by exclusion; what isn’t listed matches. Here, we write the same rules with ALL EXCEPT:

ALL : 192.168.8.3 192.168.8.4 : accept
ALL EXCEPT ftpd : ALL : deny

Of course, this rule relies on having a default accept policy that permits the FTP connection later.

Some people find rules more clear when written with ALL, others prefer ALL EXCEPT. The important thing to remember is that the first matching rule ends the check, so be careful slinging ALL around.

It’s a good idea to allow any connections from the local host; you’re likely to discover a number of programs that break when they can’t talk to the local machine. Put a rule like this early in your hosts.allow:

ALL : localhost : allow

Options

We’ve already seen two options: allow and deny. While allow permits the connection, deny blocks it. The first rule in the default hosts.allow applies to all daemons and clients, and it matches and allows all possible connections. This rule can’t be first in your hosts.allow if you want to wrap your services, but it’s a good final rule in a default accept security stance. Similarly, an ALL:ALL:deny rule is a good final rule in a default deny security stance. TCP wrappers support other options besides the simple allow and deny, however, giving you a great deal of flexibility.

Logging

Once you’ve decided to accept or reject the connection attempt, you can also log the connection. Suppose you want to permit but specifically log all incoming requests from a competitor. Similarly, you might want to know how many connections your server rejects because of DNS problems when using the PARANOID client keyword. Logging is good. More logging is better. Disk space is cheaper than your time.

The severity option sends a message to the system log, syslogd(8). You can configure syslogd to direct these messages to an arbitrary file based on the syslogd facility and level you choose (see Chapter 21).

sshd : ALL : severity local0.info : allow

This example permits all SSH connections but also logs them using the local0 facility.

Twisting

The twist option allows you to run arbitrary shell commands and scripts when someone attempts to connect to a wrapped TCP daemon and returns the output to the remote user. The twist option works properly only with TCP connections. (Remember, UDP is connectionless; there’s no connection to return the response over, so you must jump through very sophisticated and annoying hoops to make twist work with UDP. Also, protocols that transmit over UDP frequently don’t expect such a response and aren’t usually equipped to receive or interpret it. Using twist with UDP isn’t worth the trouble.) The twist option takes a shell command as an argument and acts as a deny-plus-do-this rule. You must know basic shell scripting to use twist; very complicated uses of twist are possible, but we’ll stick with the simple ones.

The twist option is useful for a final rule in a default deny stance. Use twist to return an answer to the person attempting to connect as follows:

ALL : ALL : twist /bin/echo "You cannot use this service."

If you want to deny just a particular service to a particular host, you can use more specific daemon and client lists with twist:

sendmail : .spammer.com : twist /bin/echo
    "You cannot use this service, spam-boy."

This isn’t effective against spam, but it might make you feel better. Legit customers that encounter rude messages might trigger meetings, however.

If you’re feeling friendly, you can tell people why you’re rejecting their connection. The following twist rejects all connections from people whose hostname doesn’t match their IP address and tells them why:

ALL : PARANOID : twist /bin/echo
    "Your DNS is broken. When you fix it, try again."

Using twist holds the network connection open until the shell command finishes. If your command takes a long time to finish, you could find that you’re holding open more connections than you like. This can impact system performance. A script kiddie can use twist to overload your system, creating a very simple DoS attack. Make twist simple and quick-finishing.

Spawning

Like twist, the spawn option denies the connection and runs a specified shell command. Unlike twist, spawn doesn’t return the results to the client. Use spawn when you want your FreeBSD system to take an action upon a connection request but you don’t want the client to know about it. Spawned commands run in the background. The following example allows the connection but logs the client’s IP address to a file:

ALL : PARANOID : spawn (/bin/echo %a >> /var/log/misconfigured )
    : allow

Wait a minute—where did the %a come from? TCP wrappers support several variables for use in twist and spawn commands, so you can easily customize your responses. This particular variable, %a, stands for client address. It expands into the client’s IP address in the shell command before the command is run. Table 19-2 lists other variables.

Table 19-2: Variables for twist and spawn Scripts

Variable	Description
%a	Client address.
%A	Server IP address.
%c	All available client information.
%d	Name of the daemon connected to.
%h	Client hostname, or IP address if hostname not available.
%H	Server hostname, or IP address if hostname not available.
%n	Client hostname, or UNKNOWN if no hostname is found. If the hostname and the IP address don’t match, this returns PARANOID.
%N	Server hostname, but if no hostname is found, this returns either UNKNOWN or PARANOID.
%p	Daemon’s process ID.
%s	All available server information.
%u	Client’s username.
%%	A single % character.

Use these variables anywhere you’d use the information they represent in a shell script. For example, to log all available client information to a file whenever anyone connects to a wrapped program, you could use this:

ALL : ALL : spawn (/bin/echo %c >> /var/log/clients) : allow

Spaces and backslashes are illegal characters in shell commands and might cause problems. While neither appears in hostnames under normal circumstances, the internet is almost by definition not normal. TCP wrappers replace any character that might confuse the command shell with an underscore (_). Check for underscores in your logs; they might indicate possible intrusion attempts or just someone who doesn’t know what they’re doing.

Wrapping Up Wrappers

Let’s take all the examples given so far in this section and build a complete /etc/hosts.allow to protect a hypothetical network system. We must first inventory the network resources this system offers, the IP addresses we have on the network, and the remote systems we wish to allow to connect.

While these requirements are fairly complicated, they boil down to a very simple ruleset:

#reject all connections from hosts with invalid DNS and from our competitor
ALL : PARANOID 198.51.100.0/24 : deny
#localhost can talk to itself
ALL : localhost : allow
#our local network may access portmap, but no others
portmap : ALL EXCEPT 203.0.113.0/24 : allow
#allow SSH, pop3, and ftp, deny everything else
sshd, POP3, ftpd : ALL : allow
ALL : ALL : deny

You can find many more commented-out examples in /etc/hosts.allow or in hosts_allow(5) and hosts_access(5).

Packet Filtering

To control access to networked programs that don’t support TCP wrappers, or whenever your needs exceed what wrappers provide, use one of FreeBSD’s kernel-level packet filtering tools. If you need a packet filter, it’s best to entirely replace your TCP wrappers implementation with packet filtering. Using both tools at once on the same machine will simply confuse you.

A packet filter compares every network packet that enters the system to a list of rules. When a rule matches the packet, the kernel acts based upon that rule. Rules can tell the system to allow, drop, or alter the packet. You can’t use the nifty options provided by TCP wrappers, however; instead of spitting a comparatively friendly rejection message back at the client, the connection is severed at the network level before the client even reaches the application.

While the idea of packet filtering is straightforward enough, your first implementation will be a complete nightmare—er, I mean, a “valuable learning experience.” Be prepared to spend hours experimenting and don’t be discouraged by failures. In my experience, it’s ignorance of basic TCP/IP that causes grief with packet filtering, rather than the packet filter itself. Trying to filter network traffic without understanding the network is frustrating and pointless. The only way to really understand TCP/IP is to do real work with it, however. Go study Chapter 7 again. If that doesn’t suffice, dig into the books recommended there.

FreeBSD suffers from a wealth of packet filters: IPFW, IP Filter, and PF.

IPFW is the primordial FreeBSD packet filtering software. It’s tightly integrated with FreeBSD; in fact, the generically named files /etc/rc.firewall and /etc/rc.firewall6 are purely for IPFW. While quite powerful and very popular with more experienced FreeBSD administrators, it’s a little difficult for a beginner.

The second packet filter, IP Filter, is not a FreeBSD-specific firewall program but is supported on several Unix-like operating systems. It’s primarily the work of one individual, Darren Reed, who has by heroic effort developed the overwhelming majority of the code and ported it to all those operating systems. IP Filter is most useful if you want to share one firewall configuration among multiple operating systems.

We’ll focus on the imaginatively named PF, or packet filter. PF originated in OpenBSD and was designed to be featureful, flexible, and easy to use. The average FreeBSD administrator can use PF to achieve almost any effect possible with the other two packet filters.

Enabling PF

PF includes the packet filtering kernel module, pf.ko, and the userland program pfctl(8). Before using PF, you must load the kernel module. The simplest way is to enable PF in rc.conf:

pf_enable="YES"

PF defaults to the accept all stance, which means that you won’t lock yourself out of your server merely by enabling the firewall.

Default Accept and Default Deny in Packet Filtering

The security stances (default accept and default deny) are critical in packet filtering. If you use the default accept stance and want to protect your system or network, you need numerous rules to block every possible attack. If you use the default deny stance, you must explicitly open holes for every little service you offer. In almost all cases, default deny is preferable; while it can be more difficult to manage, its increased security more than makes up for that difficulty.

When using a default deny stance, it’s very easy to lock yourself out of remotely accessing your machine. When you have an SSH connection to a remote machine and accidentally break the rule that allows SSH access, you’re in trouble. Everybody does this at least once, so don’t be too embarrassed when it happens to you. The point is, it’s best not to learn about packet filtering on a remote machine; start with a machine that you can console into so you can recover easily. I’ve cut my own access many times, generally because I’m not thinking straight when solving an unrelated packet filtering problem. Without a remote console or IPMI, the only fix is to kick myself as I climb into the car, drive to the remote location, and apologize profusely to the people I’ve inconvenienced as I fix the problem. Fortunately, as I grow older, this happens less and less.¹

Still, in almost all circumstances, a default deny stance is correct. As a new administrator, the only way you can reasonably learn packet filtering is if you have convenient access to the system console. If you’re not entirely confident in your configuration, don’t set up a packet filtering system across the country unless you have remote console and power access, a competent local administrator, or a serial console.

Basic Packet Filtering and Stateful Inspection

Recall from Chapter 7 that a TCP connection can be in a variety of states, such as opening, open, closing, and so on. For example, every connection opens when the client sends a SYN packet to the server to request connection synchronization. If the server is listening on the requested port, it responds with a SYN-ACK, meaning, “I’ve received your request, and here’s basic information for our connection.” The client acknowledges receipt of the information with an ACK packet, meaning, “I acknowledge receipt of the connection information.” Each part of this three-way handshake must complete for a connection to occur. Your packet filtering ruleset must permit all parts of the handshake, as well as the actual data transmission, to occur. Allowing your server to receive incoming connection requests is useless if your packet filter rules don’t permit transmitting that SYN-ACK.

In the early 1990s, packet filters checked each packet individually. If a packet matched a rule, it was allowed to pass. The system didn’t record what it had previously passed and had no idea whether a packet was part of a legitimate transaction or not. For example, if a packet arrived marked SYN-ACK with a destination address inside the packet filter, the packet filter generally decided that the packet had to be the response to a packet it had previously approved. Such a packet had to be approved to complete the three-way handshake. As a result, intruders forged SYN-ACK packets and used them to circumvent seemingly secure devices. Since the packet filter didn’t know who had previously sent a SYN packet, it couldn’t reject illegitimate SYN-ACK packets. Once an intruder gets packets inside a network, he can usually trigger a response from a random device and start to worm his way in.

Modern packet filters use stateful inspection to counteract this problem. Stateful inspection means keeping track of every connection and its current condition. If an incoming SYN-ACK packet appears to be part of an ongoing connection, but nobody sent a corresponding SYN request, the packet is rejected. While this complicates the kernel, writing stateful inspection packet filter rules is easier than writing old-fashioned rules. The packet filter must track many, many possible states, so this is harder to program than it might seem—especially when you add in problems such as packet fragmentation, antispoofing, and so on.

PF performs stateful inspection by default. You don’t need to specify it in a rule.

If you’ve started to think, “Hey, packet filtering sounds like a firewall,” you’re right, to a point. The word firewall is applied to a variety of network protection devices. Some of these devices are very sophisticated; some lose intelligence contests to cinderblocks. These days, the term firewall is nothing more than a marketing buzzword with very little concrete meeting. The word firewall is like the word car: do you mean a rusty 1972 Gremlin with a 6-horsepower engine and an exhaust system that emits enough fumes to breach the Kyoto Accords, or a shiny Tesla Roadster with a 500-horsepower engine, a fancy tricolor paintjob, and the Stereo System of The Apocalypse? Both have their uses, but one is obviously designed for performance. While the Gremlins of firewalls might have their place, it’s preferable to get the best you can afford.

Having said that, FreeBSD can be made as solid a firewall as you desire. Packet filtering is only the beginning. The packages collection contains a variety of application proxies that can let your FreeBSD system go up against Checkpoint or a PIX and come out on top—for tens of thousands of dollars less.

Configuring PF

Configure PF in /etc/pf.conf. This file contains statements and rules whose formats vary with the features they configure. Not only is the rule order extremely important but also the order in which features are configured. If you try to do stateful inspection before you reassemble fragmented packets, for example, connections won’t work properly.

The default /etc/pf.conf has the sample rules in the proper order, but if you’re in the slightest danger of becoming confused, I suggest that you put large comment markers between the sections, in capital letters if necessary. (Use hash marks to comment pf.conf.) The features must be entered in this exact order:

1. Macros
2. Tables
3. Options
4. Packet normalization
5. Bandwidth management
6. Translation
7. Redirection
8. Packet filtering

Yes, PF does more than just filter packets. It’s a general-purpose TCP/IP manipulation tool. We won’t cover all of its features here; go read Peter’s book.

Macros

A macro lets you define variables to make writing and reading rules easier. For example, here are macros to define your network interface and your IP address:

interface="em0"
serveraddr="203.0.113.2"

Later in your rules, you may describe your network interface as $interface and your server’s IP address as $serveraddr. This means that if you renumber your server or change your network card, making one change in your pf.conf fully updates your rules.

Sometimes you’ll want a rule to refer to “all IP addresses currently on this interface.” You don’t care which address the traffic arrives at, you just want either to accept or reject traffic to that interface. PF provides shorthand for this. Enclose the interface name in parentheses, as we’ll see later. (You can use the interface name without parentheses, but then PF won’t notice any IP changes since the last reload or restart.)

Tables and Options

PF can store long lists of addresses through tables. That’s a more sophisticated use of PF than we’re going to use, but you should know the capability exists.

Similarly, PF has a variety of options that control network connection timing, table sizes, and other internal settings. The default settings are generally adequate for normal (and most abnormal) use.

Packet Normalization

TCP/IP packets can be broken up in transit, and processing these shards of data increases system load and the amount of work your server must do both to serve the request and filter the packets. A system must reassemble these fragments before handing them on to your client software, while deciding what to do with any other random crud that arrives. PF refers to this reassembly as scrubbing. For example, to reassemble all fragments coming in your network interface, drop all fragments too small to possibly be legitimate, and otherwise sensibly sanitize your incoming data stream, use the following rule:

scrub in

This affects all packets entering the computer.

While scrubbing seems like a “nice to have,” it’s actually quite important since PF filters are based on whole packets. Fragments are much more difficult to filter and require special handling unless reassembled. Not scrubbing your traffic causes connectivity problems.

Bandwidth, Translation, and Redirection

PF includes other features vital for firewalls and performs other functions normally associated with network devices. Through queueing, PF can control how much traffic the host transmits on a per-IP or even per-port basis. PF includes a whole bunch of features to support Network Address Translation (NAT) and port redirection, two critical firewall features. The support exceeds that found in many commercial offerings.

All of this would fill another book. Literally. Peter Hansteen wrote The Book of PF. Go read that and build a firewall. Every sysadmin should build a firewall out of a raw operating system at least once in her life. Even if you revert to using a commercial offering, a little embedded device, or a product like pfSense or OPNsense, you’ll learn a whole bunch.²

Small-Server PF Rule Sample

Here’s a sample set of PF rules for protecting a small internet server. Start from here and edit this to match your server’s requirements.

➊ ext_if="em1"
➋ set skip on lo0
➌ scrub in

➍ block in
➎ pass out

➏ pass in on $ext_if proto tcp from any to ($ext_if) port {22, 53, 80, 443}
➐ pass in on $ext_if proto udp to ($ext_if) port 53
➑ pass in on $ext_if inet proto icmp to ($ext_if) icmp-type { unreach, redir,
       timex, echoreq }

We start by defining a macro for our interface name ➊ so that if we change network cards, we won’t need to rewrite all our rules.

The second line instructs PF not to filter on the lo0 interface ➋. The loopback interface is local to the machine. The only host that can communicate over it is the local machine.

Then, we scrub incoming traffic ➌, reassembling packets into a coherent whole and throwing away what can’t be reassembled.

Now that we have a sensible stream of incoming data, we can filter it. This policy starts by blocking all incoming traffic ➍, setting a default deny policy. Everything not explicitly permitted is forbidden.

Outbound traffic gets a default allow policy ➎.

The final three rules in this policy address TCP, UDP, and ICMP. They have a similar format, which we’ll dissect shortly.

First, we permit TCP traffic to ports 22, 53, 80, and 443 ➏.

Next, we permit UDP traffic to port 53 ➐. If this host offered more services than DNS, we’d have a longer list of ports.

The final rule allows vital ICMP traffic to our host and permits the host to respond ➑.

Let’s take a closer look at the TCP rule.

➊pass in➋on $ext_if➌proto tcp➍from any➎to ($ext_if)➏port {22, 53, 80, 443}

This host has a default deny policy on inbound traffic, so with the pass in statement ➊, we’re carving out an exception to that policy.

The next chunk of the rule specifies which interface this rule applies to ➋. This rule applies to the interface defined by the macro $ext_if, or em1.

Then, we specify a protocol ➌. This rule applies to TCP connections.

You can write PF rules that apply only to specific source or destination addresses. This rule applies to traffic from any host ➍. You can drop this part of the rule if you’re permitting any source address.

We then specify a destination address ➎. The destination is the interface name in parentheses, which means “any IP address on this interface.”

Lastly, define the ports this rule applies to ➏. The braces allow you to group several entities together. The filter permits connections to port 22 (ssh), 53 (DNS), 80 (HTTP), and 443 (HTTPS). You could specify a port by its name (from /etc/services), but I find numbers to be more reliable. Editing /etc/services shouldn’t break your firewall! Deploying a new TCP service on this host requires only adding a port to the list and reloading the firewall rules.

The UDP rule is very slightly different.

pass in on $ext_if➊proto udp to ($ext_if)➋port 53

The most obvious change is defining UDP protocol instead of TCP ➊. One less obvious change is that this rule drops the source address. It applies to packets from any address. This packet filter allows only one port, 53 ➋. Rules with a single port don’t need braces.

The ICMP rule looks a little tricky, but it’s really just the same.

pass in on $ext_if inet➊proto icmp to ($ext_if)➋icmp-type➌{ unreach, redir, timex echoreq }

Specifying that this rule applies to ICMP is straightforward ➊. And this rule also doesn’t list a source address, so it applies to traffic from anywhere.

Where the TCP and UDP rules specify a destination port, this ICMP rule lists an icmp-type ➋. ICMP doesn’t have ports, but it does have different types of traffic. For our purposes, though, ICMP types are much like ports. Types have numerical codes, but the names are easier.

This rule specifies four different types of ICMP traffic ➌.

Taken as a whole, this rule permits ICMP traffic that’s generally necessary for proper internet functioning. Your environment might need other ICMP types. Your organization’s security policy might specify what ICMP you can and cannot pass. But these four are a reasonable combination for an internet-facing server.

This simple policy defines basic rules for communicating with our server. While it’s not perfect, it can raise barriers for intruders. That jerk who broke into your web server and started a command prompt on port 10000? If your firewall rules don’t allow incoming connections on that port, all their hard work will be wasted. Such a tragedy.

Managing PF

Manage PF with pfctl(8). If your rules have no errors, pfctl(8) runs silently; it produces output only when you have errors. You’ll want to test, activate, view, and remove rules.

Testing Rules

As a firewall error can cause you much grief, it’s best to check your rules before activating them. While a rule check only parses the file, checking for grammatical errors in the rules themselves, activating rules with grammatical errors either leaves your system unprotected, locks you out, or both. Use the -n flag to check a file for problems and -f to specify the PF rules file.

# pfctl -nf /etc/pf.conf

If you get errors, fix them and try again.

Activating Rules

Once your syntax check runs silently, activate the new rules by removing the -n flag.

# pfctl -f /etc/pf.conf

Changing PF configuration is very quick. This means you can have several PF configurations for different times or situations. Perhaps you want to allow access only to certain services at certain parts of the day; you could schedule a pfctl(8) run to install appropriate rules for those times. Or maybe you have separate rules for disaster situations and want to install a special ruleset when you lose your internet connection. Using pfctl(8) makes all these configurations simple.

View Rules

If you want to see the rules currently running on your firewall, use pfctl -sr.

# pfctl -sr
scrub in all fragment reassemble
block drop in all
pass in on em1 proto tcp from any to (em1) port = ssh flags S/SA keep state
pass in on em1 proto tcp from any to (em1) port = domain flags S/SA keep state
pass in on em1 proto tcp from any to (em1) port = http flags S/SA keep state
pass in on em1 proto tcp from any to (em1) port = https flags S/SA keep state
pass in on em1 proto udp from any to (em1) port = domain keep state
pass in on em1 inet proto icmp from any to (em1) icmp-type unreach keep state
pass in on em1 inet proto icmp from any to (em1) icmp-type redir keep state
pass in on em1 inet proto icmp from any to (em1) icmp-type timex keep state
pass in on em1 inet proto icmp from any to (em1) icmp-type echoreq keep state
pass out all flags S/SA keep state

You can write PF rules in exactly the format shown here.

Note that while we specified multiple TCP ports in the configuration file, in the packet filter each TCP and UDP port gets its own rule. Likewise, each ICMP type gets its own rule.

Removing Rules

Finally, remove all rules from your running configuration with the -Fa (flush all) flags. (You could use flags other than a to remove parts of your firewall config, but that can leave your system in an inconsistent state.)

# pfctl -Fa

You’ll see PF systematically erase all rules, NAT configurations, and anything else in your configuration. Do not manually clear the configuration before loading a new configuration; just load the new rules file to erase the old rules.

PF is terribly powerful, very flexible, and can abuse TCP/IP in almost any way you like (and some ways you won’t like). We’ve barely scratched the surface. Check out some of the resources listed at the start of “Packet Filtering” on page 462 to explore PF in depth.

Blacklistd(8)

Sometimes you want more thoughtful packet filtering than a simple allow or deny permits. I often have SSH servers open to the public internet so that I can log in from anywhere. I do rather resent botnets thinking that I’d be sufficiently daft to permit logins without a password, though. That’s where blacklistd(8) comes in.

Blacklistd lets a daemon report, “Hey, this IP address is bugging me.” Once blacklistd receives a sufficient number of complaints about an address, it tells the firewall to block that address. Those bots eternally poking at your SSH server? They’re history.

This sort of blacklisting is only marginally useful against distributed botnets like the Hail Mary Cloud, but even then, you might be able to configure sensitivity to block out the most annoying clients. It all depends on just how intrusive each botnet member is.

To use blacklistd, you must set up the packet filter to accept input from blacklistd, set tolerance levels for each service, and configure the service to use blacklistd.

PF and Blacklistd

PF handles dynamic rules through anchors. You can use pfctl(8) to edit an active anchor, letting you insert rules at a specific point in the policy. Add the blacklistd anchor to your rules right before your first block and pass statements. Using the policy from the previous section, your rules would look like this:

--snip--
anchor "blacklistd/*" in on $ext_if
block in
pass out
--snip--

You must include the quotes around the anchor name, and you must specify the interface.

The packet filter is now ready for dynamic blacklisting.

Configuring Blacklistd

Blacklistd gets its configuration from /etc/blacklistd.conf. While most of its configuration goes in this file, you can also modify the service’s behavior with command line options.

Start by enabling blacklistd in /etc/rc.conf.

# sysrc blacklistd_enable=YES
blacklistd_enable: NO -> YES

The daemon won’t start until you either reboot or start it manually, so you can configure it now.

/etc/blacklistd.conf

Blacklistd rules each support a single service, port, or group of addresses. Put your rules into /etc/blacklistd.conf, one rule per line. Blacklistd rules come in two groups, local and remote.

Local blacklistd rules apply to items local to the machine running blacklistd. This is where you set rules for the local SSH service, or port 99, or anything else local. The section of local rules is prefaced with [local].

Remote blacklistd rules apply to items not local to the machine. Here, you might define rules like “this block gets reduced tolerance” or “disable these addresses for shorter times” or “never block these addresses.” The section of remote rules gets prefaced with [remote]. We’ll talk about local rules first and then the additions supported by remote rules.

Here’s a sample blacklistd.conf entry:

[local]
ssh             stream  *       *               *       3       24h

The first line is a [local] statement. Every rule that appears after this applies to the local machine, until we hit a [remote] entry.

Each rule has seven fields. The first four fields identify traffic to be blacklisted, while the last three fields define the blacklist behavior. An asterisk (*) is a wildcard, saying anything matches this field.

The first field is the location. For local rules, this gives the network port that this rule applies to. Entries like ssh and ftp are slightly deceiving. They don’t apply to the programs named sshd and ftpd, but rather to the network ports listed in /etc/services. While you can list a specific IP address and port in local rules, blacklistd ignores the address. Only the port applies. The sample rule blocks on ssh, or port 22.

The second field gives the socket type. TCP sockets use type stream, while UDP sockets need dgram. At this time, all services that support blacklistd use TCP. You can safely use an asterisk here to say “any socket type.” Our sample rule uses stream, so it’s for TCP connections.

The third field defines the protocol. Supported options include tcp, udp, tcp6, udp6, or numeric, or you can just use a wildcard and say “any protocol.” The only reason not to use a wildcard here is if you want to specifically match only one version of IP, such as using a different blacklist setting for TCP over IPv4 than for TCP over IPv6.

The fourth field gives the owner of the daemon complaining about the traffic. This can be a wildcard, a username, or a UID. Again, wildcards are the most common entry here. For blacklisting purposes, I don’t care which user runs the server running on port 22; I care that it gets protected from random poking.

The fifth field, the packet filter rule name, is the first entry that determines how the block works. Blacklistd defaults to putting all blocks under an anchor called blacklistd, which we put into pf.conf in the previous section. If you want separate blacklists to use different anchors, you can define an anchor name in this field; otherwise, just use the wildcard for the default.

If you start a name with a hyphen (-), it means “use an anchor with the default name prepended.”

ssh             stream  *       *               -ssh       3       24h

This entry adds any new blacklist rules to an anchor called blacklistd-ssh.

Using a slash (/) in the name field and the length of the netmask tells blacklistd to block entire subnets using prefix notation.

22              stream  tcp       *               */24    3       24h

When one host in a network misbehaves, we block everything in the adjoining /24. A /24 means very different things in IPv4 versus IPv6. Be sure to specify which protocol this rule applies to!

The sixth column, nfail, sets the number of login failures needed to blacklist the remote IP. Here, a wildcard means never. Our example rule sets a limit of 3, which is how many chances OpenSSH gives you to log in on one connection.

The last column, disable, says how long to blacklist the host for. The default unit is seconds, but you can use m, h, and d for minutes, hours, and days, respectively. Our example rule is set to 24 hours.

So, with this rule in place, failing to authenticate to SSH three times will result in the client being blocked for 24 hours.

Once you have local rules set up, you can configure remote rules.

blacklistd.conf Remote Rules

Use remote rules to specify how blacklistd varies its behavior depending on the remote host. Each of the fields in a remote rule is the same as that in the local rules, but how blacklistd uses them changes. Here’s a sample remote rule:

[remote]
203.0.113.128/25 *      *       *               =/25    =       48h

The address column is an IP (either IPv4 or IPv6) address, a port, or both. This lets you set special rules for a specific remote address range. Our sample rule applies to the address range 203.0.113.128/25.

The type, protocol, and owner columns are interpreted identical to the local rules.

The name column gets interesting. The equal sign in a remote rule means “use the value from the local rule you’re matching.” This rule says to take the firewall rule name entry and add the network prefix /25 (a 255.255.255.128 netmask) to it. If a connection from this address range gets blacklisted, it will affect the entire subnet. If you put a PF anchor name here, the blacklistd adds rules for this address block to the named anchor. A wildcard reverts to the default table.

The nfail column lets you set a custom number of failures for this address. Maybe you want to offer that one customer that just can’t figure out how to type their password the first 30 times extra attempts to fail. Setting this column to an asterisk disables blocking.

The disable column lets you set a custom block time for this address block. Using a wildcard here disables blocking.

Remote rules let you enforce stricter limits on people you don’t like, while telling blacklistd(8) never to blacklist your office.

You can now start blacklistd. It won’t do anything, though, because programs don’t know they should complain to it. But once you configure them, it’ll be ready.

Configuring Blacklistd Clients

FreeBSD includes a few blacklistd-aware clients. The two you’re most likely to use are ftpd(8) and sshd(8).

To enable blacklistd in your SSH server, add the following line to /etc/ssh/sshd_config.

UseBlacklist yes

Restart sshd.

Enable blacklisting in ftpd(8) with the -B command line option, either in /etc/inetd.conf or in the standalone process’s /etc/rc.conf flags.

ftpd_flags="-B"

These programs will now whinge to blacklistd(8) any time someone fails to log in.

Managing Blacklistd

Blacklisting annoying clients that have no right to poke at your services cuts down on the amount of log analysis you need to do, but you’ll probably want to see exactly what the blacklist is blocking. You want blacklistctl(8).

The blacklistctl(8) program has only one function: to display addresses and networks blocked by blacklistd. You always want the blacklistctl dump command.

By default, blacklistctl dump shows hosts that are in the list of candidates to be blocked but are not yet blocked. Add the -b flag to see all blocked hosts.

# blacklistctl dump -b
        address/ma:port id      nfail   last access
  203.0.113.128/25:22   OK      6/3     2018/08/28 16:30:09

Here, we see that the address range 203.0.113.128/25 attempted 6 out of 3 permitted login attempts. How did it achieve this? SSH lets a client try multiple logins on a single TCP/IP connection. Blacklisting doesn’t stop a live connection. The last time the guilty host attempted to access this service was at the date shown in last access.

You might find the time remaining more useful than the time of last access. Add the -r flag.

# blacklistctl dump -br
        address/ma:port id      nfail   remaining time
  203.0.113.128/25:22   OK      4/3     36s

Too soon, this subnet will be free to harass and harry my innocent SSH server. Maybe I need to increase the blacklist duration.

De-Blacklisting

Despite your best efforts, one day you’ll need to pull an address from the blacklist before it expires naturally. The blacklistctl(8) program offers no way to do this: you must manually delete the address from the PF table. Doing so requires understanding how blacklistd manages addresses inside PF.

Each blocked port has a child anchor inside the blacklistd anchor. This anchor is named after the port. The child anchor that blocks port 22 would be called blacklistd/22. Inside that child anchor, you’ll find a table containing the blocked addresses. The table is named port, followed by the port number. Hosts that can no longer connect to port 22 appear in a table called port22.

Here, I use the packet filter control program pfctl(8) to examine the contents of the port22 table inside child anchor blacklistd/22. I’m not going to explain all of this; just substitute your table and child anchor names. (Read Hansteen’s The Book of PF to let anchors drag you under. Far, far under.)

# pfctl -a blacklistd/22 -t port22 -T show
--snip--
   203.0.113.128/25
--snip--

Yes, our problem address is in there. Removing it requires a fairly arcane pfctl(8) command.

# pfctl -a blacklistd/22 -t port22 -T delete 203.0.113.128/25

The blacklist is maintained in a database outside of PF, though, so the blacklisted address will still show up in blacklistctl(8). That database entry will eventually expire harmlessly. If the host misbehaves again, it will get blocked again.

Public-Key Encryption

Many server daemons rely upon public-key encryption to ensure confidentiality, integrity, and authenticity of communications. Many different internet services also use public-key encryption. You need a basic grasp of public-key encryption to run services like secure websites (https) and secure POP3 mail (pop3ssl). If you’re already familiar with public-key encryption, you can probably skip this section. If not, gird your loins for a highly compressed introduction to the topic.

Encryption systems use a key to transform messages between readable (cleartext) and encoded (ciphertext) versions. Although the words cleartext and ciphertext include the word text, they aren’t restricted to text; they can also include graphics files, binaries, and any other data you might want to send.

All cryptosystems have three main purposes: integrity, confidentiality, and nonrepudiation. Integrity means that the message hasn’t been tampered with. Confidentiality means that the message can be read only by the intended audience. And nonrepudiation means that the author can’t later claim that he or she didn’t write that message.

Older ciphers relied on a single key, and anyone with the key could both encrypt and decrypt messages. You might have had to do a lot of work to transform the message, as with the Enigma engine that drove the Allies nuts during World War II, but the key made the transformation possible. A typical example is any code that requires a key or password. The one-time message pads popular in spy novels are the ultimate single-key ciphers, impossible to break unless you have that exact key.

Unlike single-key ciphers, public-key (or asymmetric) encryption systems use two keys: a private key and a public key. Messages are encrypted with one key and decrypted with the other, and digital signatures ensure the message isn’t tampered with en route. The math to explain this is really quite horrendous, but it does work—just accept that really, really large numbers behave really, really oddly. Generally, the key owner keeps the private key secret but hands the public key out to the world at large, for anyone’s use. The key owner uses the private key, while everyone else uses the public key. The key owner can encrypt messages that anyone can read, while anyone in the public can send a message that only the key owner can read.

Public-key cryptography fills our need for integrity, confidentiality, and nonrepudiation. If an author wants anyone to be able to read his message, while ensuring that it isn’t tampered with, he can encrypt the message with his private key. Anyone with the public key (that is, the world) can read the message, but tampering with the message renders it illegible. (Depending on the use, he might choose to sign the message digitally instead.)

Encrypting messages this way also ensures that the author of the message has the private key. If someone wants to send a message that can be read only by a particular person, he can encrypt the message with the desired audience’s public key. Only the person with the matching private key can read the message.

This works well so long as the private key is kept private. Once the private key is stolen, lost, or made public, the security is lost. A careless person who has his private key stolen could even find others signing documents for him. Be careful with your keys, unless you want to learn that someone used your private key to order half a million dollars’ worth of high-end graphics workstations and have them overnighted to an abandoned-house maildrop in inner-city Detroit.³

The standard toolkit for all of these operations is OpenSSL.

OpenSSL

FreeBSD includes the OpenSSL toolkit for handling public-key cryptography. OpenSSL lets you perform a full range of encryption operations. While many programs use OpenSSL functionality, the sysadmin doesn’t need OpenSSL directly very often.

While OpenSSL works fine out of the box, I find it worthwhile to set a few defaults to make my life easier down the road. Configure OpenSSL with the file /etc/ssl/openssl.cnf. Almost all of the settings in this file are correct as they are, and you shouldn’t change them unless you’re a cryptographer. The few things useful to change are the defaults for generating cryptographic signatures. Each default value is marked by the string _default. You’d be most interested in the following settings for common OpenSSL operations, which I’ve adjusted to fit my needs:

➊ countryName_default            = US
➋ stateOrProvinceName_default    = Michigan
➌ 0.organizationName_default     = Burke and Hare Word Mine, LLC

The countryName_default ➊ is the two-letter code for your nation—in my case, US. The stateOrProvinceName_default ➋ is the name of your local state and can be of any length. I would set this to Michigan. The 0.organizationName_default field ➌ is your company name. If I’m buying a signed certificate, I’d put the same thing here that I want to appear on the certificate. If I’m just testing how programs work with SSL and don’t have a real company name, I might use the name of the company I work for or something that I make up.

The following values don’t show up in openssl.cnf, but if you set them, they appear as defaults in the OpenSSL command prompts. I find these useful, even though they change more frequently than the previous defaults—they remind me of the correct format of these answers, if nothing else.

➊ localityName_default            = Detroit
➋ organizationalUnitName_default  = Pen-Monkey Division
➌ commonName_default              = www.michaelwlucas.com
➍ emailAddress_default            = [email protected]

The localityName_default ➊ is the name of your city. The organizationalUnitName_default ➋ is the part of your company this certificate is for. One of the most commonly misunderstood values in OpenSSL, commonName_default ➌, is the hostname of the machine this certificate is for, as it appears in reverse DNS. Remember, reverse DNS isn’t necessarily the same as the hostname! Your web server might have a nice friendly name, but the hosting company might assign it a totally different name in reverse DNS. Finally, emailAddress_default ➍ is the email address of the site administrator.

These values all show up in prompts in the OpenSSL command as default choices. Setting them in the configuration file will save you annoyance later.

Certificates

One interesting thing about public-key encryption is that the author and the audience don’t have to be people. They can be programs. Secure Shell (SSH) and the Secure Sockets Layer (SSL) are two different ways programs can communicate without fear of intruders listening in. Public-key cryptography is a major component of the digital certificates used by secure websites and secure mail services. When you open Firefox to buy something online, you might not realize that the browser is frantically encrypting and decrypting web pages. This is why your computer might complain about “invalid certificates;” someone’s public key has either expired or the certificate is self-signed. Today’s protocols encrypt and decrypt with Transport Layer Security (TLS) and use TLS certificates.

Many companies, such as VeriSign, provide a public-key signing service. These companies are called Certificate Authorities (CAs), as they provide TLS certificates. Other companies that need a certificate signed provide proof of their identity, such as corporate papers and business records, and those public-key signing companies sign the applicant’s certificate with their CA certificate. By signing the certificate, the CA says, “I have inspected this person’s credentials and he, she, or it has proven their identity to my satisfaction.” They’re not guaranteeing anything else, however. A TLS certificate owner can use the certificate to run a website that sells fraudulent or dangerous products or use it to encrypt a ransom note. Signed TLS certificates guarantee certain types of technical security, not personal integrity or even unilateral technical security. Certificates don’t magically apply security patches for you.

Web browsers and other certificate-using software include certificates for the major CAs. When the browser receives a certificate signed by a CA, it recognizes the certificate as legitimate. Essentially, the web browser says, “I trust the Certificate Authority, and the Certificate Authority trusts this company, so I will trust the company.” So long as you trust the CA, everything works.

The package ca_root_nss contains the CA certificates recognized by the Mozilla Project. If a piece of software fails attempting to validate certificates, make sure you installed this package.

Most CAs are big commercial companies. No matter the size of your organization, though, I encourage you to investigate Let’s Encrypt (https://www.letsencrypt.org/). Let’s Encrypt is a CA that provides free, globally valid TLS certificates.

Using a certificate that’s not signed by any CA is perfectly fine for testing. It might also suffice for applications within a company, where you can install the certificate in the client web browser or tell your users to trust the certificate. We’ll look at both ways.

Both uses of the certificate require a host key.

TLS Host Key

Both signed and self-signed certificates require a private key for the host. The host key is just a carefully crafted random number. The following command creates a 2,048-bit host key and places it in the file host.key:

# openssl genrsa 2048 > host.key

You’ll see a statement that OpenSSL is creating a host key and dots crossing the screen as key generation proceeds. In only a few seconds, you’ll have a file containing a key. The key is a plaintext file that contains the words BEGIN RSA PRIVATE KEY and a bunch of random characters.

Protect your host key! Make it owned by root and readable only by root. Once you place your certificate in production, anyone who has that key can use it to eavesdrop on your private communications.

# chown root host.key
# chmod 400 host.key

Place this host key in a directory with the same permissions that we placed on the key file itself.

Create a Certificate Request

You need a certificate request for either a signed or self-signed certificate. We don’t do much with OpenSSL, so we won’t dissect this command. Go to the directory with your host key and enter this verbatim:

# openssl req -new -key host.key -out csr.pem

In response, you’ll see instructions and then a series of questions. By hitting ENTER, you’ll take the default answers. If you’ve configured OpenSSL, the default answers are correct.

➊ Country Name (2 letter code) [US]:
➋ State or Province Name (full name) [Michigan]:
➌ Locality Name (eg, city) [Detroit]:
➍ Organization Name (eg, company) [Burke and Hare Word Mine, LLC]:
➎ Organizational Unit Name (eg, section) [Pen-Monkey Division]:
➏ Common Name (eg, YOUR name) [www.michaelwlucas.com]:
➐ Email Address [[email protected]]:

The two-letter code for the country ➊ is defined in the ISO 3166 standard, so a quick web search will find this for you. If you don’t know the state ➋ and city ➌ you live in, ask someone who occasionally leaves the server room. The organization name ➍ is probably your company, and you list the department or division name ➎ as well. If you don’t have a company, list your family name or some other way to uniquely identify yourself, and for a self-signed certificate, you can list anything you want. Different CAs have different standards for noncorporate entities, so check the CA’s instructions.

The common name ➏ is frequently misunderstood. It’s not your name; it’s the name of the server as shown in reverse DNS. You must have a server name here, or the request will be useless.

I suggest using a generic email address ➐ rather than an individual’s email address. In this case, I am michaelwlucas.com, so I might as well use my address. You don’t want your organization’s certificates tied to an individual who might leave the company for whatever reason.

   Please enter the following 'extra' attributes
   to be sent with your certificate request
➊ A challenge password []:
➋ An optional company name []:

The challenge password ➊ is also known as a passphrase. Again, keep this secret because anyone with the passphrase can use your certificate. Use of a certificate passphrase is optional, however. If you use one, you must type it when your server starts. That means that if your web server crashes, the website won’t work until someone enters the passphrase. While passphrase use is highly desirable, this might be unacceptable. Hit ENTER to use a blank passphrase.

You’ve already entered quite a few company names, so a third ➋ is probably unnecessary.

Once you return to a command prompt, you’ll see the file csr.pem in the current directory. It looks much like your host key, except that the top line says BEGIN CERTIFICATE REQUEST instead of BEGIN RSA PRIVATE KEY.

Submit csr.pem to your Certificate Authority, who will return the actual certificate. I recommend saving the certificate in a file named after the host, such as www.mwl.io.crt. This signed certificate is good for any TLS service, including web pages, pop3ssl, or any other TLS-capable daemon.

Some CAs require you use an intermediate certificate with your cert. While most daemons have a configuration option to specify an intermediate certificate, if yours doesn’t, you can append the signed certificate to the end of the intermediate cert.

Sign a Certificate Yourself

A self-signed certificate is technically identical to a signed certificate, but it’s not submitted to a Certificate Authority. Instead, you provide the signature yourself. Most customers won’t accept a self-signed certificate on a production service, but it’s perfectly suitable for testing. To sign your own CSR, run the following:

# openssl x509 -req -days➊365 -in csr.pem -signkey host.key
-out➋selfsigned.crt
Signature ok
subject=/C=US/ST=Michigan/L=Detroit/O=Burke and Hare Word Mine, LLC/OU=Pen-
Monkey Division/CN=michaelwlucas.com/[email protected]
Getting Private key
#

That’s it! You now have a self-signed certificate good for 365 days ➊ in the file selfsigned.crt ➋. You can use this key exactly like a signed certificate, so long as you’re willing to ignore the warnings your application displays.

If you sign your own certificates, client software generates warnings that the “certificate signer is unknown.” This is expected—after all, people outside my office have no idea who Michael W. Lucas is or why he’s signing web certificates. For some reason, people trust Symantec and other big-company CAs. I’m trusted by the people who know me,⁴ but not trusted by the world at large. For this reason, don’t use self-signed certificates anywhere the public will see them because the warnings will confuse, annoy, or even scare them away.

But before you go drop any amount of money on a CA certificate, definitely check out Let’s Encrypt. It really will change your system administration practice.

TLS Trick: Connecting to TLS-Protected Ports

I said we wouldn’t do much with OpenSSL, and that’s correct. There’s one facility the software offers that’s too useful to pass up, however, and once you know it, you’ll use this one trick at least once a month and be glad you have it.

Throughout this book, we test network services by using telnet(1) to connect to the daemon running on that port and issuing commands. This works well for plaintext services such as SMTP, POP3, and HTTP. It doesn’t work for encrypted services such as HTTPS. You need a program to manage the encryption for you when you connect to these services. OpenSSL includes the openssl s_client command, which is intended for exactly this sort of client debugging. While you’ll see a lot of cryptographic information, you’ll also get the ability to issue plaintext commands to the daemon and view its responses. Use the command openssl s_client -connect with a hostname and port number, separated by a colon. Here, we connect to the secure web server at www.absolutefreebsd.com:

# openssl s_client -connect www.michaelwlucas.com:443
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
--snip--

You’ll see lots of stuff about chains of trust and limitations of liability, as well as lines and lines of the random-looking digital certificates. After all that, however, you’ll see a blank line with no command prompt. You’re speaking directly to the server daemon. As this is a web server, let’s try an HTTP command:

GET /

The system responds with:

HTTP/1.1 400 Bad Request

The HTTP protocol has changed since the last time I tried this, I guess. But I’m definitely connected to the web server. The network works.

Some of you are probably wondering why we encrypt the service if it’s so easy to talk to the encrypted service. The encryption doesn’t protect the daemon; it protects the data stream between the client and the server. TLS encryption prevents someone from eavesdropping your network conversation in transit—it doesn’t protect either the server or the client. TLS can’t save you if someone breaks into your desktop.

From this point on, I’ll assume that you understand this OpenSSL command and what happens when we use it.

Global Security Settings

FreeBSD supports many optional security settings. These settings change basic FreeBSD behavior, making it differ from the common Unix experience. Some other operating systems provide these settings by default, however, so they’re not unique to FreeBSD.

Should you turn all these features on in the name of improved security? There’s no universally correct answer here. If restricting access to part of the system to the root account means that you’ll need to give more people root access, maybe you shouldn’t impose that restriction. A couple of these should be activated on all systems, though.

Install-Time Options

The FreeBSD installer provides an option for enabling each of these settings on first boot. You can enable and disable them later with the given sysctl setting.

Many of these features are especially useful on servers that don’t have many users. If your application server doesn’t have unprivileged users other than those used by applications, you should probably enable features that restrict unprivileged users. If you have unprivileged users, though, consider the situation more closely. Most of my unprivileged users⁵ shouldn’t be looking at server processes or other users, so I lock them down.

Hiding Other UIDs’ Processes

Normally, commands like ps -ax display all processes running on the system. When you set the sysctl security.bsd.see_other_uids to 0, users can see only their own processes. Root can see all processes, no matter how you set this.

Hiding Other GIDs’ Processes

Similarly, users can normally see processes owned by other groups. Disable that ability by setting the sysctl security.bsd.see_other_gids to 0. Again, root can see every process, no matter how this is set.

Hiding Jailed Processes

Users on a host can usually see all processes running in jails. By setting security.bsd.see_jail_proc to 0, unprivileged nonjailed users can’t see jailed processes. This feature appeared in FreeBSD 12.

Hide Message Buffer

Unprivileged users can normally see the system message buffer, available through dmesg(8). Disable that access by setting the sysctl security.bsd.unprivileged_read_msgbuf to 0.

Disable Process Debugging

A debugger can tell users a whole bunch of useful information. Setting security.bsd.unprivileged_proc_debug to 0 disallows unprivileged users from using the debugger on processes.

Randomize Process IDs

Traditional Unix systems create process IDs in sequential order, allowing attackers a chance at guessing what the next PID will be. Randomize process IDs by setting the sysctl kern.randompid to a random large integer. If you set it to 1, the kernel picks a fresh random number between 100 and 1,123 at each boot.

Clean /tmp

All sensible Unix-like systems clean /tmp at boot to dispose of temporary files. Somewhere in the last few years, FreeBSD turned this behavior off by default. You might use tmpfs(5) for /tmp, which gets destroyed at every power-down. If your /tmp is on disk, though, well . . . as you’re all sensible and wholesome sysadmins, always set clear_tmp_enable to YES in /etc/rc.conf.

Disable Syslogd Networking

By default, syslogd(8) creates a half-open socket on UDP port 514. Nobody can connect to this socket; it’s used only as a placeholder so nothing else binds to that port. Some people consider this half-open socket problematic. I’d say it’s a feature; you don’t want something else binding to port 514, claiming to be syslogd, and sending either worrisome or falsely soothing messages to your logging host. But to disable that half-open socket, set syslogd_flags to -ss in /etc/rc.conf.

Disable Sendmail

A default FreeBSD install doesn’t accept email from the network, but it does run a sendmail(8) daemon to sent outgoing messages. To completely disable sending mail from this host, set sendmail_enable to NONE in /etc/rc.conf.

Disabling outbound mail won’t prevent the daily, weekly, and monthly maintenance tasks from running. It’ll prevent you from receiving the output of those messages unless you log directly onto the host, however. For people with multiple hosts, disabling outbound mail is unwise. Disabling Sendmail makes sense if you use an alternative mail agent, such as dma(8) (see Chapter 20).

Secure Console

Most Unix systems consider the physical console secure. Anyone who has access to the physical machine can do anything to the host that they want, including changing the root password. By changing all of the /etc/ttys entries that say secure to insecure, you tell FreeBSD to demand the root password even in single-user mode.⁶ This won’t prevent someone from physical access gaining access to your operating system, but it’ll mean that they’ll have to do slightly more work to subvert your machine. Very slightly more work.

Nonexecutable Stack and Stack Guard

One basic exploit mitigation technique is the nonexecutable stack. Once a program is loaded into memory, each page of memory allocated to that program should be either writable or executable, but not both.

A common exploit technique is to trick a program into writing information to memory and then executing that memory. An attacker might convince a program to write to a chunk of memory, but with the nonexecutable stack, the kernel won’t execute it.

The stack defaults to nonexecutable on modern versions of FreeBSD. The only reason to disable this is if you have a badly written program that relies on executing and writing the same chunk of memory. Most such defective software has been rightfully purged from the open source ecosystem in the last 15 years. If you’re very unlucky and can’t avoid running a program that can’t handle a nonexecutable stack, you can disable this by setting the sysctls kern.elf32.nxstack (for 32-bit programs) or kern.elf64.nxstack (for 64-bit programs) to 0.

Related to the nonexecutable stack, a stack guard page adds a random-sized shred of extra memory between parts of a program’s memory allocation. This makes it harder for an attacker to guess memory addresses. FreeBSD allocates a stack guard page by default, but you can turn it off by setting the sysctl security.bsd.stack_guard_page to 0.

Other Security Settings

Most of FreeBSD’s other kernel-level security settings are available in the security.bsd sysctl tree. More get added every few months. Run sysctl -d security.bsd to display your hosts’ available options. I’ve described many of these earlier in this section, but you might find some of the others useful. Options include disabling the root account’s privileges (security.bsd.suser_enabled), allowing nonroot users to set an idle priority (security.bsd.unprivileged_idprio), and blocking unprivileged users from using mlock(2) (security.bsd.unprivileged_mlock). Take a look at the current options and see what might be useful.

Preparing for Intrusions with mtree(1)

One of the worst things to happen to a sysadmin is something that makes him think that his system could’ve been penetrated. If you find mysterious files in /tmp or extra commands in /usr/local/sbin, or if things “just don’t feel right,” you’ll be left wondering whether someone has compromised your system. The worst thing about this feeling is that there’s no way to prove it hasn’t happened. A skilled attacker can replace system binaries with her own customized versions, so that her actions are never logged and your attempts to find her will fail. Having Sherlock Holmes examine your server with a magnifying glass is useless when the magnifying glass has been provided by the criminal and includes the special criminal-cloaking feature! People have even hijacked the system compiler so that freshly built binaries include the hijacker’s backdoor.⁷ What makes matters worse is that computers do weird things all the time. Operating systems are terribly complicated, and applications are worse. Maybe that weird file in /tmp is something your text editor barfed up when you hit the keys too fast, or perhaps it’s a leftover from a sloppy intruder.

The only way to recover a compromised system is to reinstall it from scratch, restore the data from backup, and hope that the security hole that led to the compromise is fixed. That’s a thin hope, and doubt is so easy to acquire that many sysadmins eventually stop caring or lie to themselves rather than live with the constant worry.

Most intruders change files that already exist on the system. FreeBSD’s mtree(1) can record the permissions, size, dates, and cryptographic checksums of files on your system. (While freebsd-update(8) includes similar features, and you don’t have to gather data beforehand, it covers only the base system.) If you record these characteristics when your system is freshly installed, you have a record of what those files look like intact. When an intruder changes those files, a comparison will highlight the differences. When you have even the vaguest feeling you’ve been hacked, you can check that same information on the existing files to see whether any have changed.

Running mtree(1)

The following command runs mtree(1) across your root partition and stores SHA512 and SHA256 cryptographic checksums, placing them in a file for later analysis:

# mtree➊-x➋-ic➌-K sha512➍-K sha256➎-p /➏-X /home/mwlucas/mtree-exclude >➐/tmp/mtree.out

While you can use mtree(1) across the entire server, most people use -x ➊ to run it once per partition. You don’t want to record checksums on frequently changing files, such as the database partition on your database server. Collecting checksums on NFS mounts has the twin features of running really slowly and increasing network congestion. The -ic flag ➋ tells mtree to print its results to the screen, with each subsequent layer in the filesystem indented. This format matches the system mtree files in /etc/mtree. The -K flag accepts several optional keywords; in this case, we want to generate SHA512 checksums ➌ and SHA256 checksums ➍. The -p flag ➎ tells mtree which partition to check. Almost every partition has files or directories that change on a regular basis and that you therefore don’t want to record checksums for. Use -X ➏ to specify an exclusion file, a file containing a list of paths not to match. Finally, redirect the output of this command to the file /tmp/mtree.out ➐.

mtree(1) Output: The Spec File

mtree(1)’s output is known as a specification, or spec. While this specification was originally intended for use in installing software, we’re using it to verify a software install. Your spec starts with comments showing the user who ran the command, the machine the command ran on, the filesystem analyzed, and the date. The first real entry in the spec sets the defaults for this host and begins with /set.

/set type=file uid=0 gid=0 mode=0755 nlink=1 flags=uarch

The mtree(1) program picked these settings as defaults based on its analysis of the files in the partition. The default filesystem object is a file, owned by UID 0 and GID 0, with permissions of 0755, with one hard link and the user archive flag. After that, every file and directory on the system has a separate entry. Here’s the entry for the root directory:

➊.        ➋type=dir➌nlink=19➍time=1504101311.033742000

This file is the dot (.) ➊, or the directory we’re in right now. It’s a directory ➋, and it has 19 hard links ➌ to it. This directory was modified 1,504,101,311.033742000 seconds into Unix epochal time ➍. The Unix epoch began January 1, 1970.

In some ways, the entry for the directory is rather boring. An intruder can’t realistically replace the directory itself, after all! Here’s an entry for an actual file in the root directory:

.cshrc      mode=0644 nlink=2➊size=950 time=1499096179.000000000
➋ sha256digest=20d2a78c9773c159bac1df5585227c7b64b6aab6b77bccadbe4c65f1be474e8c
➌ sha512=24d4330e327f75f10101cd7c0d6a5e59163336ade5b9eb04b0d96ea43d221c5eea4c71a89dfe85a...

We see the filename and the same mode, link, and time information as in the root directory, but also get the file size ➊. Additionally, there’s the SHA256 ➋ and SHA512 ➌ cryptographic hashes computed from the files.

While it’s theoretically possible for an intruder to craft a file that matches a particular cryptographic hash, and while cryptographers are constantly trying to find practical ways to create files that match arbitrary SHA256 and SHA512 checksums, it’s extremely unlikely that an intruder can create a fake file that matches both checksums, contains his backdoor, and still functions well enough that the system owner won’t immediately notice a problem. By the time this happens, we will have additional checksum algorithms resistant to those methods and will switch to them.

The Exclusion File

The exclusion file (given with -X) lists filesystems you don’t want mtree(1) to analyze. Lots of filesystems will change without malicious intervention. Log files and user home directories should change. Directories like /tmp and /var/db/entropy better change on a functional system. List each directory you don’t want checked on its own line in the exclusion file, with a leading dot.

./tmp
./var/db/entropy
./var/log
./usr/home

Wait a day or so, and then run mtree(1) again to generate a new spec file. Differences between the two mtree files will let you improve your exclusion file. You’ll do the exact same thing when you suspect a system intrusion.

Saving the Spec File

The spec file contains the information needed to verify the integrity of your system after a suspected intrusion. Leaving the spec file on the server you want to verify means that an intruder can edit the file and conceal his wrongdoing. You must not save the file on the system itself! Now and then someone will suggest that you checksum the mtree spec file but keep it on the server. That’s not useful; if someone tampers with the mtree file and the checksum, how would you know? Or worse—if someone tampered with the spec file and you caught it, you couldn’t tell what change had been made! Copy your spec file to a safe location, preferably on an offline media, such as a flash drive or an optical disk.

Finding System Differences

When something raises your suspicions and you begin to think that you might have suffered an intrusion, create a new mtree spec file and compare it with the “known good” spec file you stored offline. Use mtree(1) to check for differences between spec files.

# mtree -f mtree.suspect -f mtree.good > mtree.differences

Every entry in the file is something that has changed. My exclusion file is finely tuned, eliminating files I expect to have changed. This particular run generates two lines of output.

                bin/sh file➊size=161672
➋ sha256digest=a4a85ca3563d8f3bda449711c6b591b37093e668fc136f8829eb188b955f56ab
➌ sha512=011793e3e6cacd99b4261e0a0f3a0b9bd6a6842f3ccd55da1ce2070b568e3c49ae7b0e51d33bb59eff...
                bin/sh file size=➍10489808
➎ sha256digest=45856525d4251b43d68df1429cf1fe0f4adb6640f06d7f995aace5b7ca0c03c2
➏ sha512=a4f0e83e5fb12d615721fd7d57cb6a120068d1aa71fc305b7b86927391f33bec822cf14ce8a8a9db14...

The file /bin/sh ➊ has changed size ➍ between mtree runs. This isn’t good. Also, note the two different SHA256 hashes ➋ ➎ and the two different SHA512 hashes ➌ ➏. Don’t hit the panic button yet, but start asking your fellow sysadmins pointed, hard questions. If you can’t get a good answer as to why this binary changed, you might look for your installation media.

Or, perhaps you need to update your exclusion file. But if /bin/sh changed, probably not.

Monitoring System Security

So, you think your server is secure. Maybe it is . . . for now.

Unfortunately, there’s a class of intruders with nothing better to do than to keep up on the latest security holes and try them out on systems they think might be vulnerable. Even if you read FreeBSD-security religiously and apply every single patch, you still might get hacked one day. While there’s no way to be absolutely sure you haven’t been hacked, the following hints will help you find out when something does happen:

• Be familiar with your servers. Run ps -axx on them regularly, and learn what processes normally run on them. If you see a process you don’t recognize, investigate.
• Examine your open network ports with netstat -na and sockstat. What TCP and UDP ports should your server be listening on? If you don’t recognize an open port, investigate. Perhaps it’s innocent, but it might be an intruder’s backdoor.
• Unexplained system problems are hints. Many intruders are ham-fisted klutzes with poor sysadmin skills, who use click-and-drool attacks. They’ll crash your system and think that they’re the cyber incarnation of Samuel L. Jackson.
• Truly skilled intruders not only clean up after themselves but also ensure that the system has no problems that might alert you. Therefore, systems that are unusually stable are also suspicious.
• Unexplained reboots might indicate someone illicitly installing a new kernel. They might also be a sign of failing hardware or bad configuration, so investigate them anyway.
• FreeBSD sends you emails every day giving basic system status information. Read them. Save them. If something looks suspicious, investigate. Look at old messages to see when something has changed.

I particularly recommend the lsof package to increase your familiarity with your system. The lsof program lists all open files on your system. Reading lsof(8) output is an education in and of itself; you probably had no idea that your web server opened so much crud. Seeing strange files open indicates either that you’re not sufficiently familiar with your system or that someone’s doing something improper.

Package Security

The FreeBSD Project provides a database of security vulnerabilities in the ports and packages system. This database is made available in Vulnerability and eXposure Markup Language (VuXML). When someone volunteers to maintain a port, they’re also volunteering to watch out for security problems with that port.

An internet-connected FreeBSD host with pkg(8) installed downloads the latest VuXML file during the periodic(8) run (see Chapter 21) and stores it in /var/db/pkg/vuln.xml. It then compares the installed packages with that database. If one of your packages has a vulnerability, you’ll be notified in the daily status email. (You are reading your daily status emails, right?)

If your packages are insecure, upgrade them as per Chapter 15.

If need be, you can set a different location to fetch the vuln.xml file with the VULNXML_SITE option in pkg.conf. You might do this if you maintain your own package repository and vulnerability databases.

If You’re Hacked

After all this, what do you do if your system is hacked? There’s no easy answer. Huge books are written on the subject. Here are a few general suggestions, however.

First and foremost: A hacked system can’t be trusted. If someone has gained root access on your internet server, he could have replaced any program on the system. Even if you close the hole he broke in through, he could have installed a hacked version of login(8) that sends your username and password to an IRC channel somewhere every time you log in. Do not trust this system. An upgrade can’t cleanse it, as even freebsd-update(8) and the compiler are suspect.

While rootkit-hunting software might help you verify the presence of intruders, nothing can verify that the intruder isn’t there. Feel free to write [email protected] for advice. Describe what you’re seeing and why you think you’re hacked. Be prepared for the ugly answer, though: completely reinstall your computer from known secure media, and restore your data from backup. You did read Chapter 5, right?

Good security practices reduce your chances of being hacked, just as safe driving reduces your chances of being in a car wreck. Eventually you’ll total your wheels anyway and wonder why you bothered. Good luck!