22
JAILS
Virtualization separates an operating system instance from the underlying hardware. Virtualization allows you to move operating system installs from one piece of hardware to another by copying a file. Virtualization needs an operating system installed on the hardware, but that install is normally very minimal, has no public-facing services, and is easily reproduced on new hardware. It’s perhaps the biggest change in system administration in decades.
Virtualization is something like a client-server environment. The hardware and its core operating system instance is the host, while the clients are all virtualized operating system instances. The clients rely on the host to provide basic services, such as storage, processor power, and memory. Changes to the host can be reflected in the virtualized clients, but changes on the client have no effect on the host beyond consuming resources.
FreeBSD supports two types of virtualization: jails and bhyve.
Jails are a lightweight virtualization method, sometimes called OS-level virtualization. A jail normally contains a complete operating system userland that runs on top of an existing FreeBSD system. The jail relies on the host’s filesystem but is limited to a subset of the directory tree. It might even have a chunk of dedicated space in a ZFS pool. A jail doesn’t have its own kernel and instead runs in a restricted portion of the host’s kernel. The host can manage jailed processes without entering the jail, or it can run processes inside the jail if preferred. Jails don’t get a graphical console. Use jails to virtualize FreeBSD installs of the same version or older, or to run simple virtual Linux systems.
Bhyve is a heavier virtualization system. Rather than using the host’s kernel and filesystem, bhyve simulates hardware. The host provides a chunk of disk space for the virtual machine to use as a disk. A bhyve virtual machine must bring along its own filesystem, kernel, and supporting infrastructure. Bhyve virtual machines require more resources than jails, but they also offer a console via virtual network computing (VNC) and can run truly foreign operating systems, like Microsoft Windows. Bhyve is changing rapidly thanks to its rapid development, so this book doesn’t cover it. I’ll write about bhyve once it stabilizes.
Before considering bhyve, see whether a jail will meet your needs.
Jail Basics
A jail is a supercharged chroot that applies not only to the filesystem but also to processes and the network stack. A jailed system can access only a narrow part of the filesystem and can’t see processes outside the jail. Traditionally, each jail is assigned a dedicated IP address, and the jail can view only traffic to that particular IP. Each jail even has its own user accounts. The root account in a jail completely controls that jail but has no access to anything beyond the jail.
To a user given root access to a jail, the jail looks like a nearly complete FreeBSD system, missing only a few device nodes. The user can install whatever software she likes without interfering with the host or other jails. All processes running in the jail can affect only the jail’s files and processes. The jailed user has no visibility of anything beyond the jail; she’s confined. If the jail is hacked, the intruder is also confined to the jail.
Jails can use a virtual network stack, based on vnet(9). That’s an advanced use we won’t cover here, but if you need to provide a jail with its own routing table, that’s how you do it.
As of FreeBSD 9, multiple jails can share a single IP address, but the sysadmin needs to configure each jail to use unique TCP/IP ports for every network service. You can’t run multiple SSH instances on port 22 of a single IP! For simplicity, the following examples use a single IP for each jail, but remember that you have other options.
Many people put all of their services in jails, even when a host is dedicated to a particular purpose. A ZFS snapshot of the jail dataset, or a tarball of the directory tree, is a complete backup of the jail. Restoration after a failed software upgrade becomes a simple matter of extracting a tarball or rolling back to the snapshot.
A jail is also useful for software development and testing. Deploying a new service often requires installing and testing quite a few packages. Doing the testing in a jail before selecting a solution and proceeding to production prevents polluting the host with abandoned files and unneeded software.
Depending on your hardware and the system load, a single FreeBSD host can support dozens or even hundreds of jails. If you want to seriously run that many jails, though, make sure your host has two separate network interfaces. Dedicate one to jails and the other to managing the host.
Everything starts with configuring your jail host.
Jail Host Server Setup
A server meant as a jail host must work within a few annoying constraints. Configure your host correctly before building your first jail.
The jail system has its own sysctl tree, security.jail. You can change these sysctls only from the host system. Some sysctls affect all jails running on the host. Sysctls that begin with security.jail.param can be set on a per-jail basis. We’ll touch on these throughout this chapter.
Jail Host Storage
I strongly advise you to use your jail host only for the purpose of running jails and to put all services inside a jail. Start by configuring the host’s storage to separate jails and the host operating system.
Many hosts intended for virtualization include SATA DOM flash drives on the mainboard for the operating system. These drives are usually less than 100GB in size, but a base install of FreeBSD fits in much less than a gigabyte. If you have a SATA DOM or similar, use it for the host operating system. If you have multiple sets of redundant hard drives, use a pair to mirror the operating system and dedicate everything else to jails.
If you don’t have such hardware, dedicate space to the host operating system. Use either a partition for UFS filesystems or a dataset reservation for ZFS. In either case, 10GB of space should be sufficient. If you need additional space for an emergency, you can borrow some from the jail space.
While ZFS is highly useful for jails, it’s not necessary. I ran jails on UFS for many years. Use what works for you and fits your environment.
Once you have your host partitioned and the operating system installed, look at the network.
Jail Networking
There are two seemingly conflicting aspects of jail networking: first, each jail expects full control of any IP addresses assigned to it; and second, jails can share IP addresses with other jails and even the host. You can start a jail using any IP address on the host, but that jail can’t coordinate any network-facing services with other services running on that IP. If your jail shares the host’s IP address, and the host runs SSH on port 22, the jail can’t use port 22. If you try to start sshd(8) in the jail, the program will complain that it can’t use port 22 and crash. Sharing IP addresses between jails, or even between jails and the host, requires the sysadmin to coordinate which ports belong to which hosts and to configure everything accordingly.
The simplest way to configure jails is to assign each its own IP address and give the host its own IP address. Each jail can then completely control its own IP address. Once you get the hang of this, you can start sharing addresses between jails. That means the host can’t have daemons listening on IP addresses assigned to jails. Having a host’s daemons listening on the jail’s IP won’t prevent the jail from starting, but it will prevent the jail from starting its own services on that port. Users like Bert will complain if they can’t SSH to their private jails!
The cleanest way to configure a jail host is to decide that the host only provides jails. Any services run on the host must be in a jail. If you need simple services, like a nameserver or a mail exchanger, configure them in a jail. Not only is this easier than properly reconfiguring all these servers to attach only to the selected IP address; it also provides an additional layer of security for your other jails. An intrusion on the host automatically grants the intruder access to all of your jails, while an intrusion on a single jail confines the intruder to that jail.
Use sockstat(1) to identify programs listening on your network, as discussed in Chapter 9. Add the -46 flags to show only IPv4 and IPv6 traffic, and -l to show only listening sockets.
# sockstat -46l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
root ntpd 19776 20 udp6 *:123 *:*
root ntpd 19776 21 udp4 *:123 *:*
--snip--
root sshd 2846 3 tcp6 *:22 *:*
root sshd 2846 4 tcp4 *:22 *:*
This fairly default FreeBSD install has two programs listening to the network: ntpd and sshd. Both are listening to all IP addresses. We must configure all of these daemons to listen only to the main server address.
Here are some common daemons that cause problems on host servers. In all of these, I’ll assume that the jail host has an IP of 198.51.100.50.
syslogd
The system logger syslogd(8) opens a UDP socket so that it can send messages to other hosts. If you don’t log remotely, or if you use a different logging solution, use the -ss flag in rc.conf to turn off the network component.
syslogd_flags="-ss"
If you need to send syslogd messages, use the -b flag to force syslogd to attach to only a single IP address.
syslogd_flags="-b 198.51.100.50"
Either solution lets your jails’ administrators individually decide whether or not they’re going to log across the network. See Chapter 21 for a full discussion of syslogd.
inetd
If you need inetd (see Chapter 20), you should almost certainly run it from within a jail rather than the host. If you can’t weasel out of running inetd on the host, though, use the -a flag to restrict it to a single IP address, as in the following rc.conf snippet.
inetd_flags="-a 198.51.100.50 -wW -C 60"
If I had only specified the -a flag and the IP address, it would’ve overwritten inetd’s default flags from /etc/defaults/rc.conf. Every release of FreeBSD from the last few decades has used the default flags of -wW -C 60; I added my -a and the IP address to those flags.
sshd
The option ListenAddress in /etc/ssh/sshd_config tells sshd(8) which addresses to bind to. Restrict it to only your host IP.
ListenAddress 198.51.100.50
If the only service your jail host offers is sshd(8), you’ve done well.
NFS
Network filesystem programs, such as rpcbind(8) and nfsd(8), bind to all IP addresses on a host, no matter what you do. Don’t run these programs inside a jail, and don’t run NFS from within a jail. If your clients need NFS mounts, have the host run these programs and provide the NFS mounts.
Network Time Protocol
The most problematic service on a jail host is timekeeping. All jails get their system clock from the host. FreeBSD’s included time daemon, ntpd(8), listens to all IP addresses on the host—including the jailed ones. As the lone rare exception, though, I’m going to tell you to go ahead and run ntpd on the host.
A jail lacks the proper access to change the kernel’s time. While you could run ntpd in a jail, it couldn’t actually do anything. Go ahead and run ntpd on your jail host, and don’t worry about it1 listening to all IP addresses. Anyone who tries to run a UDP-based service other than ntpd on port 123 is probably trying to evade a packet filter. Make them work harder.
If you want to avoid even the chance of a collision, install the openntpd package. Unlike the base system ntpd(8), OpenNTPD can be configured to listen to a single IP address.
IP Addresses
Each jail can have one or more IP addresses. These addresses must be attached to the host before you start the jail. A jail will run without any networking, but it won’t be accessible beyond the host. Add any necessary IP addresses as aliases in /etc/rc.conf.
Jails at Boot
To have FreeBSD start your jails at boot, set jail_enable in rc.conf.
jail_enable=YES
FreeBSD defaults to starting all jails listed in /etc/jail.conf. If you want the system to start only a subset of those jails at boot, use the jail_list rc.conf option. Here, I have two jails, called mariadb and httpd. I want them started in this order so that my database jail is running before the web server that calls it.
jail_list="mariadb httpd"
During system shutdown, FreeBSD stops jails in the same order it starts them. Your application might not like that. In my example, I want the web server to turn off before the database backend. I’d rather have a website be flat-out unavailable than have users see the dreaded “database server is kaput” error.
jail_reverse_stop=YES
If the jail startup order is unimportant, you can start and stop all jails simultaneously.
jail_parallel_start="YES"
Now you can configure a jail.
Jail Setup
Now that I have a host, I can install some jails. I’ll start with a jail called mariadb, for running . . . wait for it . . . MariaDB.
Each jail needs a dedicated root directory. All of my example jails live under /jail. I normally put each jail in a directory named after the jail name—in this case, /jail/mariadb.
Each jail needs a primary IP. It can also have other IPs, as we’ll see later, but let’s start with one. The jail mariadb gets 203.0.113.51.
Each jail needs an internet hostname, just as if it were a real host. This jail will become mariadb.mwl.io.
Now we can put a userland in the jail.
Jail Userland
While you can install any userland components in a jail, all a jail requires is the base system. Grab the base.txz distribution set for your desired FreeBSD release and extract it in your jail’s root directory.
# tar -xpf base.txz -C /jail/mariadb
That’s a complete install of the base operating system. If you want additional distribution sets, such as the debugging symbols, extract them the same way.
If you’ve built your own FreeBSD base system, you can install it in the jail.
# cd /usr/src
# make installworld DESTDIR=/jail/mariadb
Jails also need supporting directories and assorted detritus created by the install process, but not by make installworld. The make distribution command creates those files. If you already have these directories and files, though, don’t rerun make distribution: it’ll overwrite any local changes. And don’t forget the DESTDIR setting, unless you like resetting the host’s configuration!
# make distribution DESTDIR=/jail/mariadb
You can also build a custom userland with only enough binaries for running a single program, much as you would for a traditionally chrooted program. For most of us, that’s too much work, but if you want to break out ldd(1) and go wild, don’t let me stop you.
Once you have a jail userland, tell FreeBSD about your jail in /etc/jail.conf.
/etc/jail.conf
Traditionally, FreeBSD configured jails in /etc/rc.conf. This was clunky and unwieldy. While FreeBSD still supports rc.conf configuration of jails, I recommend using the more flexible /etc/jail.conf instead. This file isn’t in UCL, although it looks like something UCL could support. Define each jail by a name. Give the jail parameters in braces after the jail name. Each parameter definition ends in a semicolon.
Many jail parameters have an equal sign, where we assign a parameter a value. Here, I set the parameter path to the value /jail/mariadb:
path="/jail/mariadb";
Other parameters enable or disable a feature with their mere presence. Here, I tell this jail to turn on the mount.devfs feature:
mount.devfs;
Jails support a whole bunch of “mount” parameters, with subparameters for different filesystems. This particular parameter specifically addresses mounting devfs.
Toggles can be turned off for a jail by adding no in front of the specific parameter. If I don’t want to enable devfs, I wouldn’t turn off the whole mount parameter; I’d put the no in front of the devfs.
mount.nodevfs;
Here’s how I define a jail named mariadb:
mariadb {
host.hostname="mariadb.mwl.io";
ip4.addr="203.0.113.51";
path="/jail/mariadb";
mount.devfs;
exec.clean;
exec.start="sh /etc/rc";
exec.stop="sh /etc/rc.shutdown";
}
The parameter host.hostname gives the jail’s hostname. While the jail name is mariadb, this host identifies itself by the internet hostname mariadb.mwl.io.
The IP address is in ip4.addr. I’ve assigned the address 203.0.113.51 to this jail. This IP must be on the host first.
The jail’s root directory goes in the path variable. Here, it’s set to /jail/mariadb.
Almost every jail needs access to specific device nodes in /dev, which requires mounting devfs (see Chapter 13) in the jail. Enable devfs with the mount.devfs setting. A jail defaults to getting only a few very specific device nodes. An untrusted user can sometimes use device nodes to escape a jail, so don’t add additional devices without careful research. You can allow other device nodes with a custom devfs ruleset. Assign a custom devfs ruleset to the jail with the devfs_ruleset jail.conf parameter. I strongly recommend using the default jail devfs rules as a base and unhiding the additional devices this jail needs, rather than trying to build a custom devfs ruleset from scratch.
A jailed process can inherit parts of its environment from the parent process. The exec.clean option tells jail(8) to strip away all of the environment except for $TERM. The environment variables $HOME, $USER, and $SHELL get set to the target environment, normally that of the jail’s root account. You’ll almost always want exec.clean.
The exec.start and exec.stop options tell FreeBSD how to start and stop the jail.
In-Jail Startup
Jails can emulate a full-running FreeBSD userland, run a single process, or anything in between. You must either use the exec.start jail.conf parameter to tell FreeBSD what process to run in the jail or the persist parameter to declare you want the jail to exist even without any processes in it. Here, I start a full FreeBSD userland, using the normal FreeBSD startup script:
exec.start="/bin/sh /etc/rc"
If you need only a single command to run inside the jail, you can write your own startup script and use exec.start to run it when the jail boots. Your brand-new jail won’t have an rc.conf yet, so it won’t start any additional processes.
Instead of exec.start, you could set the persist option. This tells FreeBSD that a jail can exist without any processes running inside it. Including both persist and exec.start means that FreeBSD will start a process for the jail, but when the process stops running, the jail won’t shut itself down.
You can tell the jail to run an additional command after it starts with the exec.poststart option. Any command or script listed with exec.poststart gets run in the host once the normal /etc/rc startup process (including any enabled packages) finishes. This lets you write scripts to glue jails together.
Similarly, you can use the exec.prestop option to run a command on the host before stopping the jail. When the sysadmin turns the jail off, the host first runs this command, and then the jail runs the normal shutdown command.
The exec.stop command tells FreeBSD what command to run inside the jail to shut the jail down. If you’re simulating a full jail, you’ll probably run /bin/sh /etc/rc.shutdown as in our example in the previous section.
Jail Defaults
You’ll find that many of your jails share common settings. You can define those settings in the front of the configuration. All jails will use those settings unless you override them. This doesn’t seem to make much sense when using a single jail.
exec.start="/bin/sh /etc/rc";
exec.stop="/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
mariadb {
host.hostname="mariadb.mwl.io";
ip4.addr="203.0.113.221";
path="/jail/mariadb";
}
Given this configuration, though, adding another jail becomes five lines including the braces.
httpd {
host.hostname="httpd.mwl.io";
ip4.addr="203.0.113.222";
path="/jail/httpd";
}
Over dozens of jails, it saves a lot of trouble.
You can override the defaults within a jail’s definition. If I don’t want to mount devfs(5) in a jail, I would set mount.nodevfs for that specific jail.
jail.conf Variables
You can use variable substitutions in jails. While you can define some of these variables, you can also pull some from the jail’s settings. Variables are expanded in double quotes and in unquoted strings, but not in single-quoted strings.
Here, I define a variable for the directory that contains all of my jails and use that inside my jail definition:
$j="/jail";
mariadb {
path="$j/mariadb";
--snip--
}
If I must move my jails to a new filesystem or pool, I can update jail.conf by changing the one variable rather than editing every definition.
Parameters as Variables
Once you define a jail parameter, you can use it as a variable. Every jail has at least one parameter, name. You can use these parameters to further expand default settings.
$j="/jail";
path="$j/$name";
host.hostname="$name.mwl.io";
mariadb {
ip4.addr="203.0.113.221";
}
By setting the global default path to $j/$name, I’ve removed the need to define path for each individual jail.
You can use multiterm parameters with a period in them by enclosing the parameter in braces. While this doesn’t make sense for parameters like mount.devfs, it’s useful for per-jail parameters, like host.hostname.
path = "/jail/${host.hostname}";
I prefer to put my jails in directories named after the shorter name, rather than the hostname, but feel free to indulge your own biases.
Combining parameters and variables with a coherent directory layout lets you squeeze each jail definition down into a single configuration statement.
Testing and Configuring a Jail
Once you have files for a jail, lock yourself in. Run the jail(8) command to run a single command inside the jail. You’ll need four arguments: the path, the jail name, the primary IP, and the command to run.
# jail <path to jail> <jail name> <jail IP> <command>
Here, I use the jail in /jail/mariadb, named mariadb, with the IP address 203.0.113.51, to run the command /bin/sh:
# jail /jail/mariadb/ mariadb 203.0.113.51 /bin/sh
#
Run ls(1). You’re in the root directory of your jail filesystem. This jail isn’t quite in single-user mode, but no programs other than /bin/sh are running here. You can do some basic setup, but not even devfs(5) is mounted.
# ps
ps: /dev/null: No such file or directory
Yes, the normal jail startup process would mount /dev—but the jail has no user accounts, no root password, no daemons running, and absolutely nothing optional. Configure the jail before starting it.
Stuff to Steal from the Host
Some host setup information is also useful within the jail. You can copy this information from the host to the jail, but you must do this from the host, not the jail.
Each jail performs its own DNS resolution. You can probably copy the host’s /etc/resolv.conf into the jail.
Your jail probably shares the same time zone as the host. Copy the host’s /etc/localtime into the jail or run tzsetup(8) inside the jail to select a new time zone.
Create /etc/fstab
Many programs and scripts, including /etc/rc, expect to find /etc/fstab and have a tantrum if it’s not there. Requiring /etc/fstab is perfectly sensible in a real server, but a jailed machine has no need for a filesystem table. Create an empty filesystem table.
# touch /etc/fstab
I don’t mind unhappy programs. I just don’t want to listen to them whinge.
Create /etc/rc.conf
Either you’ll do all jail management from the host, or you’ll manage jails via SSH. You’ll need an /etc/rc.conf entry for sshd.
sshd_enable="YES"
Add any other settings you want while creating this file. If you know some of the settings packages will need, it won’t hurt to set them before they’re needed.
User Accounts and Root Password
You can add user accounts and change passwords only from within the jail. Set a root password with passwd(1) and run adduser(8) to add at least one user, for SSH. While SSH is not the only way to access the host, it’s far easier in most cases.
Jail Startup and Shutdown
The host considers each jail an independent service, much like sshd(8), a web server, or any other daemon. Yes, each jail might run a whole bunch of services that need managing independently, but from the host’s perspective, each jail is a single entity containing a group of processes. That’s part of the separation between the host and the jail.
Use service(8) to start, stop, and restart jails. You’ll need to provide one additional argument, the name of the jail. FreeBSD automatically starts them at boot, but you can stop, start, and restart them individually once the system is running. Let’s shut down my database jail and fire it up again.
# service jail stop mariadb
# service jail start mariadb
I could use the restart command, but that wouldn’t look nearly so impressive here on the page.
If you omit the jail name, the service(8) command affects all jails that FreeBSD starts at boot.
# service jail restart
Stopping jails: httpd mariadb.
Starting jails: mariadb httpd
This lets you coherently reinitialize your production jail infrastructure.
FreeBSD defaults to starting all jails listed in /etc/jail.conf. As discussed in “Jails at Boot” on page 568, you can change that in /etc/rc.conf. The service(8) command can control jails that aren’t autostarted, but you must specify them by name.
Jail Dependencies
If you have a whole bunch of jails, listing the start order in /etc/rc.conf can get tedious. You’ll most often need to set a start order to maintain service dependencies. Rather than defining the order in rc.conf, though, you can tell a jail that it requires another jail with the depend option.
httpd {
ip4.addr="203.0.113.232";
depend=mariadb;
}
The jail httpd won’t start until the jail mariadb is running. A depend statement overrides a rc.conf jail_list entry.
Managing Jails
Virtualization doesn’t make system administration tasks evaporate; it only adds options for performing typical sysadmin tasks. Here’s some of those options.
Viewing Jails and Jail IDs
Use jls(8) to see all jails currently running on the system.
# jls
JID IP Address Hostname Path
29 203.0.113.221 mariadb.mwl.io /jail/mariadb
30 203.0.113.222 httpd.mwl.io /jail/httpd
31 203.0.113.223 test.mwl.io /jail/test
Each jail has a unique jail ID, or JID. The JID is much like a process ID; while each jail has one, the exact JID issued to a jail changes each time the jail is started. We’ll use the jail ID or name to execute various jail-management tasks.
We also get each jail’s IP address, hostname, and the path to the jail’s files. You don’t get the jail name, but those of us who use a hostname based on the jail name have no trouble figuring it out.
Jailed Processes
Jailed processes all get a process ID, like any other Unix process. Process IDs are not unique to jails; they’re shared between the host, the jail, and all the other jails. You won’t find repeated process IDs.
Jailed processes show up in ps(1) with the -J flag.
# ps -ax
PID TT STAT TIME COMMAND
0 - DLs 2:56.24 [kernel]
1 - ILs 0:00.07 /sbin/init –
--snip--
35002 - SsJ 0:00.01 /usr/sbin/syslogd -s
35129 - IsJ 0:00.00 /usr/sbin/sshd
--snip--
Process IDs 35002 and 35129 are jailed.
View a particular jail’s processes with ps(1) using the -J flag and the jail name.
# ps -ax -J test
PID TT STAT TIME COMMAND
35561 - IsJ 0:00.01 /usr/sbin/syslogd -s
35652 - IsJ 0:00.00 /usr/sbin/sshd
35661 - SsJ 0:00.03 sendmail: accepting connections (sendmail)
--snip--
Using -J 0 excludes all jailed processes from ps(1) output, letting you more easily debug the host.
Commands like pgrep(1), pkill(1), and killall(1) all accept a -j argument to let you specify a jail. If you prefer using pgrep(1) to view process information, use pgrep -lfj and the jail name or JID.
# pgrep -lf -j mariadb
PID TT STAT TIME COMMAND
35002 - SsJ 0:00.01 /usr/sbin/syslogd -s
35129 - IsJ 0:00.00 /usr/sbin/sshd
35158 - SsJ 0:00.02 sendmail: accepting connections (sendmail)
--snip--
Why is Sendmail running inside this jail? Let’s kill it.
# pkill -9 -j mariadb sendmail
Running pgrep again shows that Sendmail is dead.
This works well if you want to get information about which processes are running in a jail, but sometimes you have a process ID and must identify which jail it belongs to. That’s where you need the -O option to ps(1). This option supports a bunch of keywords that adjust the output of ps(1) in ways not supported by the regular command line flags—specifically, -O jail adds a column for the name of the jail the process is running in.
# ps -ax -O jail | grep 39415
39415 mariadb - IsJ 0:00.00 /usr/local/libexec/mysqld
This process is running inside the jail mariadb.
Running Commands in Jails
The jexec(8) command lets the jail host administrator execute commands within a jail without going to the trouble of logging into the jail. This helps preserve the jail owner’s sense of privacy.2 When jail owner Bert calls to beg for help, I don’t need his root password or even an account on his system. Using jexec requires knowing the jail’s name or JID. Here, I use the host’s root account to run ps -ax inside my jail mariadb.
# jexec mariadb ps -ax
PID TT STAT TIME COMMAND
35002 - SsJ 0:00.00 /usr/sbin/syslogd -s
35129 - IsJ 0:00.00 /usr/sbin/sshd
--snip--
This command runs as root inside the jail. I might want to run the command as another jailed user, though. Give that username with the -U flag.
# jexec -U xistence mariadb ps
jexec: xistence: no such user
Well, that’s not good. I’m expecting Bert to run my database. Let’s make him a user account.
# jexec mariadb adduser
Username: xistence
Full name: Bert JW Regeer
Uid (Leave empty for default):
Login group [bert]:
Login group is bert. Invite bert into other groups? []: wheel
--snip--
This jail now has an account for Bert, using his preferred username and everything. I’ve added it to the wheel group within the jail. Remember, root access within a jail doesn’t equal root access on the host. That’s the whole point of jails.
I can now run commands as that user in that jail.
# jexec -U xistence mariadb sh
$
I’m locked up in jail! Specifically, in Bert’s jail cell.
This jailed process will behave a little oddly, though. A process retains its environment. In this case, while I’m running as the user xistence, I retain all the environment settings I had in my nonjailed process. This includes stuff like $SSH_AUTH_SOCK, my IRC server setting, and more. I don’t want this stuff in my jailed environment. If I’m logged in as Bert, I want to be Bert.
To strip your environment before entering a jail, use jexec’s -l flag. This simulates a clean login.
# jexec -lU xistence mariadb sh
Should you always strip your environment before running a command in a jail? No, not always. It depends entirely on what you’re doing.
Many commands include support for running them on the host but targeting a jail. Always check the man page for such an option. One good example is sysrc(8), which lets you specify a jail with -j. Here, I enable MariaDB on the jail mariadb. MariaDB has chosen to continue to use MySQL naming conventions, so it’s enabled with the rc.conf option mysql_enable.
# sysrc -j mariadb mysql_enable=YES
mysql_enable: -> YES
This jail is now ready to run MariaDB.
Except for the bit where MariaDB isn’t installed, of course. Let’s take care of that next.
Installing Jail Packages
FreeBSD’s package tools let you manage software either from within the jail or from the host. If the host administrator has allocated you a jail to configure, you probably want to manage packages from the jail. Jail packages work exactly as packages on any other FreeBSD host, as discussed in Chapter 15. If you’re responsible for the whole system, including the host and all the jails on that host, you probably want to manage each jail’s packages from the host rather than logging into each jail. Let’s spend some time on that.
The pkg(8) command’s -j flag lets you specify a jail to manage. You’ll need one argument, the jail’s name or JID. The -j flag must be given before the pkg(8) subcommand. Here, I install the MariaDB server on its dedicated jail:
# pkg -j mariadb install mariadb101-server
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 9 package(s) will be affected (of 0 checked):
--snip--
Note that pkg(8) offers no notice that it’s installing packages within a jail. It assumes that if you’re using -j, you know you’re working in a jail.
When you manage a jail’s packages from the host, the package tools don’t get installed on the jail. The jail has its own package database, stored within the jail, but the jail has no way to use that database directly.
Don’t switch between managing packages on the host and from within the jail. Choose one method and stick with it.
Updating Jails
So you have umpteen bajillion jails on your host, each dedicated to performing its own task in perfect isolation. That’s grand, until you have to apply security patches to all of the hosts. If you’ve built your FreeBSD from source, you’ll need to install a new world in each jail. If you’re running releases, though, freebsd-update(8) (see Chapter 18) can handle jails.
You can’t use freebsd-update(8) inside a jail. The same things that isolate a jail from compromising the host system disallow some of the functionality freebsd-update(8) needs. Instead, you update the jail from the host.
Any time you need to update your system, update your host before updating your jails. The host must be running a version of FreeBSD equal to or newer than any jail.
Start by copying /etc/freebsd-update.conf to an alternate file, such as /etc/jail-freebsd-update.conf. Remove all components that aren’t installed on the jail. Jails don’t have kernels, and most of them don’t have source code, so you’ll probably wind up with an entry like this:
Components world
When you run freebsd-update(8), it checks the version of FreeBSD you’re running on. It does this by querying the kernel. If you have FreeBSD 12.0 jails on a FreeBSD 13.0 system, the update program gets confused, chokes, and dies with mysterious errors. You need freebsd-update to use the release installed in the jail, not the version the host is running. Use the --currently-running option to tell freebsd-update(8) what version the jail is running. You must use the jail’s release, including the patch level. While you could easily enough extract that information from the jail, I encourage you to let freebsd-update ask the jail what version it’s currently running. Do this by using jexec(8) to query the version of FreeBSD running in the jail.
You’ll also use the -b flag to tell freebsd-update(8) the directory the jail lives in.
Here, I update the jail called test. The jail’s files are in /jail/test. I use a jexec(8) command in backticks to check the current FreeBSD version.
# freebsd-update -f /etc/jail-freebsd-update.conf -b /jail/test/ --currently-running `jexec -l test freebsd-version` fetch install
Once freebsd-update finishes running, restart your jail. It’s upgraded.
Many people have written scripts to run through /etc/jail.conf and upgrade all of their jails. If you have more than a couple jails, you find or write such a script.
More Jail Options
You can customize jails in all sorts of ways. The jail(8) man page includes the current list of jail options, but here’s a few features I commonly use.
Rather than letting jail(8) assign jail IDs, you can assign each jail a permanent ID with the jid option.
jid=101;
The securelevel option lets you raise the securelevel (see Chapter 9) within a jail. The jail’s securelevel can never be lower than that of the host.
You can view a jail’s startup messages by manually running a /etc/rc with a jail command. That’s inconvenient for routine troubleshooting, though. Direct the jail’s console messages to a file with the exec.consolelog option.
exec.consolelog="$j/logs/$name.log";
In addition to mounting /dev, a jail can have its own fdescfs(5) and procfs(5) with the mount.fdescfs and mount.procfs options.
Jailing Ancient FreeBSD
In my experience, the phrase enterprise network is synonymous with “we have lots of ancient stuff that nobody dares touch.” Jails can help you cope with some of these systems. In 2014, I worked at a company that ran a critical custom-built PHP and MySQL application on FreeBSD 4.10. I don’t know when this server was installed, but FreeBSD 4.11 came out in January 2005, so: before then. This application used ancient versions of Perl, PHP, OpenSSL, and more.
Worse, this application lived on a repurposed desktop machine. With a standard-issue, high-quality desktop hard drive. I hid a spare desktop machine of the same vintage under my desk, just so I had a hope of getting FreeBSD 4.10 onto it. The most proper solution was to rewrite or replace this application. Several sysadmins had faced that task—and failed. I decided to virtualize it. FreeBSD 4.10 doesn’t run well on VMWare—yes, you can find de(4) and fxp(4) drivers, but they’re for versions of those cards over a decade old. Here’s how I got this ancient FreeBSD system into a jail.
Drop to single-user mode. Unmount /proc—yes, FreeBSD 4 still used /proc. Those were the days. Tar up the entire filesystem, including temporary directories, like /usr/obj, /usr/ports, /var/tmp, and suchforth. By modern standards, they won’t use much space at all, and you have no way to know what files you might need later. You can probably find an old PHP 5.0.whatever tarball out on the internet, but that would involve work.
Copy the tar file to your jail host and extract it in your jail directory.
# tar -C /jail/oldserver -xvpf oldserver.tgz
Be sure to use the -p flag to preserve permissions.
Now look at /etc/rc.conf. The jail host will handle all networking functions, so turn off any statements that set IP addresses or set routes. Get rid of daemons that provide services the jail host offers, such as time, packet filters, and SSH. Your jailed host needs only the functions that directly support the application. In this case, I needed Apache and MySQL.
Consider the jail’s /etc/fstab. Do you need a NFS filesystem or some other special mount? Remove everything you don’t need. If this application needs /proc, provide it with the jail option mount.procfs.
Remove the old /dev. You can’t use FreeBSD 4 device nodes on a modern FreeBSD.
Configure the host to protect the jail. While people can write perfectly fine applications in Apache and PHP, not even the most ardent Apache and MySQL fan would encourage you to expose 15-year-old versions of these servers to the internet. Use the host’s packet filter to protect the jail. Don’t even consider using the migrated host’s OpenSSH server.
You won’t be able to use some FreeBSD 4 commands inside the jail, as the interfaces have diverged too far. A FreeBSD 4 ps(1) can’t successfully query a modern FreeBSD kernel. You can copy statically linked versions of most of those programs from the host’s /rescue and copy them into the jail, however.
Is it this simple? No, not really. The older the source system is, the more problems you’ll have. Most of the problems I had in this particular migration meant changing a configuration file to account for the new underlying filesystem. You’ll need to perform your usual sysadmin debugging. But it’s one way to get a modern network interface on a system lacking a device driver for it, and it’s the only way to get ZFS on a FreeBSD 4 system.
Last Jail Notes
People have evolved many ways of using jails. Fully covering all of these features would pretty much require a book of its own, but here are some pointers.
You can use ZFS features to delegate a dataset entirely to the jail administrator so that jail owners can take their own snapshots and create their own child datasets. With the VIMAGE kernel option, you can give a jail its own routing table. If you’re brave, nullfs(5) lets you recycle an operating system install and minimize disk utilization. You can establish per-jail resource limits with the RCTL kernel option.
If you have many jails, you might prefer using a jail management program, such as iocage or ezjail. Both are available in the Ports Collection.
Successfully using jails requires automating your maintenance. Each jail requires separate security patches, both for the userland and for installed packages. The more you can automate this process, the more likely it is that you’ll actually perform such maintenance. I recommend using Ansible’s jail modules or at least writing your own shell script to apply patches.
The jail(8) command will let you modify, create, and destroy jails without a command line. If you’re doing extensive jail work, definitely read the man page.
Now let’s look at some of FreeBSD’s less well-known corners.