4
START ME UP! THE BOOT PROCESS

While FreeBSD boots easily and automatically when you turn on the power, understanding exactly what happens at each stage will make you a better system administrator. Intervention during the boot process is rarely necessary, but one day you’ll be glad you know how to do it. And once you’re comfortable with adjusting the boot process, you’ll find you can solve problems you’ve previously accepted and endured.

We’ll start by discussing how the system loader starts and gathering information from the loader. You can use the loader to change the early boot process, including booting alternate kernels and starting in single-user mode. We’ll cover serial consoles, a standard system management tool. The FreeBSD multiuser startup process is responsible for starting all the various services that make your computer useful, and we’ll give attention to that as well. In addition, we’ll cover the information FreeBSD records about the boot process and how FreeBSD turns itself off without corrupting data.

The boot process itself can be divided into three main parts: the loader, single-user startup, and multiuser startup.

Power-On

A computer needs enough brains to find and load its operating system. For many years, this facility came from the basic input/output system (BIOS). Newer systems use the Unified Extensible Firmware Interface (UEFI) instead of the BIOS. New installs should use UEFI. Other hardware platforms have console firmware or bootroms that perform the same function, but we’re focused on commodity hardware, so we’ll cover UEFI and BIOS.

Unified Extensible Firmware Interface

UEFI is a replacement for the three-decades-old BIOS. Any new system will come with UEFI enabled and will expect to use it.

UEFI searches the boot drive for a partition marked as a UEFI boot partition. Despite what the special mark might imply, that partition contains only a FAT filesystem with a specific directory and file layout. UEFI executes the file /EFI/BOOT/BOOTX64.EFI. That file might be a fancy multi-OS boot loader, or it might dump you straight into an operating system. In FreeBSD, the UEFI boot fires up the boot loader, /boot/loader.efi.

UEFI is comparatively new. If your new system has trouble booting FreeBSD, you might try enabling a BIOS or “legacy” mode. If the system boots FreeBSD in BIOS mode but not with UEFI, please file a bug, as discussed in Chapter 24.¹

Basic Input/Output System

The primordial Intel PC used a basic input/output system (BIOS) with just enough brains to look for an operating system somewhere on a disk. A BIOS searches for a disk partition marked active and then executes the first section of that partition. For FreeBSD, that chunk of data is called the loader. Every FreeBSD system has a reference copy of the loader as /boot/loader.

A BIOS has all sorts of limitations. The boot loader must reside in a very specific section of the disk. BIOS can’t boot from disks larger than 2.2TB. The target boot loader must be smaller than 512KB—huge by 1980 standards, yes, but paltry today. The installed loader is a binary, not a filesystem, so even minor changes require recompiling. UEFI has none of these limitations and offers modern features, like mouse support.

Ultimately, though, a BIOS and UEFI both have the goal of getting your system to the FreeBSD loader.

The Loader

The loader, or boot blocks, loads the FreeBSD kernel and presents you with a menu before starting that kernel. The loader(8) program offers a menu of seven options on the left. A new FreeBSD system presents these options:

1. Boot Multi User [Enter]
2. Boot Single User
3. Escape to loader prompt
4. Reboot
5. Kernel: default/kernel (1 of 2)
6. Configure Boot Options…
7. Select Boot Environment…

Each option highlights certain words or characters, such as S in “Boot Single User” and ESC in “Escape to loader prompt.” Select an option by pressing the highlighted character or the number.

The options at the top of the menu control how FreeBSD boots. We’ll look at each option in turn. If you wait 10 seconds, the loader automatically boots FreeBSD by default.

The options at the bottom half let you fine-tune the boot process. You can tweak how you want the system to boot, as we’ll discuss later, and then choose one of the preceding booting options.

Boot Multi User [Enter]

This is a normal boot. Hit ENTER to boot immediately, skipping the 10-second delay.

Boot FreeBSD in Single-User Mode

Single-user mode is a minimal startup mode that’s very useful on damaged systems, especially when the damage was self-inflicted. It’s the earliest point where FreeBSD can provide a command prompt, and it’s important enough to have its own section later in this chapter.

Escape to Loader Prompt

The loader includes a command line interpreter, where you can issue commands to tweak your system to boot exactly the way you need. We’ll cover this in detail in “The Loader Prompt” on page 55.

Reboot

Once more, this time with feeling!

Of these options, the most important are single-user mode and the loader prompt.

Single-User Mode

FreeBSD can perform a minimal boot, called single-user mode, that loads the kernel and finds devices but doesn’t automatically set up your filesystems, start the network, enable security, or run any standard Unix services. Single-user mode is the earliest the system can possibly give you a command prompt.

Why use single-user mode? If a badly configured daemon hangs the boot, you can enter single-user mode to prevent it from starting. If you’ve lost your root password, you can boot into single-user mode to change it. If you need to shuffle critical filesystems around, again, single-user mode is the place to do it.

When you choose a single-user mode boot, you’ll see the regular system startup messages flow past. Before any programs start, however, the kernel offers you a chance to choose a shell. You can enter any shell on the root partition; I usually just take the default /bin/sh, but use /bin/tcsh if you prefer.

Disks in Single-User Mode

In single-user mode, the root partition is mounted read-only and no other disks are mounted. (We’ll discuss mounting filesystems in Chapter 10, but for now just follow along.) Many of the programs that you’ll want to use are on partitions other than the root, so you’ll want them all mounted read-write and available. The way to do this varies depending on whether you’re using UFS or ZFS.

UFS in Single-User Mode

To make all the filesystems listed in the filesystem table /etc/fstab usable, run the following commands:

# fsck -p
# mount -o rw /
# mount -a

The fsck(8) program “cleans” the filesystems and confirms that they’re internally consistent and that all the files that a disk thinks it has are actually present and accounted for.

The root filesystem is mounted read-only. Whatever drove us to single-user mode probably requires changing the root filesystem. Remount the root filesystem read-write.

Finally, the -a flag to mount(8) activates every filesystem listed in /etc/fstab (see Chapter 10). If one of these filesystems is causing you problems, you can mount the desired filesystems individually by specifying them on the command line (for example, mount /usr). If you’re an advanced user with NFS filesystems configured (see Chapter 13), you’ll see error messages for those filesystems at this point because the network isn’t up yet. If the host has network filesystems in /etc/fstab, mount only the UFS filesystems as shown next.

If you have trouble mounting partitions by name, try using the device name instead. The device name for the root partition is probably /dev/ad0s1a. You’ll also need to specify a mount point for this partition. For example, to mount your first IDE disk partition as root, enter the command:

# mount /dev/ad0s1a /

If you have network filesystems on your server but your network isn’t up yet, you can mount all your local partitions by specifying the filesystem type. Here, we mount all of the local filesystems of type UFS, which is FreeBSD’s default filesystem type:

# mount -a -t ufs

You can now access your UFS filesystems.

ZFS in Single-User Mode

To make all of your ZFS datasets available, use zfs mount. You can either mount individual datasets by name or mount everything that’s marked as mountable with -a.

# zfs mount -a

ZFS will perform its usual integrity checks before mounting the datasets.

Most of the datasets will be exactly as accessible as in multiuser mode, but the dataset mounted as root will still be read-only. Turn that off. Here, I’m setting the root dataset to read-write on a default FreeBSD install.

# zfs set readonly=off zroot/ROOT/default

You can now change the filesystem.

Programs Available in Single-User Mode

The commands available for your use depend on which partitions are mounted. Some basic commands are available on the root partition in /bin and /sbin, and they’re available even if root is mounted read-only. Others live in /usr and are inaccessible until you mount that partition. (Take a look at /bin and /sbin on your system to get an idea of what you’ll have to work with when things go bad.)

If you’ve scrambled your shared library system (see Chapter 17), none of these programs will work. If you’re that unlucky, FreeBSD provides statically linked versions of many core utilities in the /rescue directory.

The Network in Single-User Mode

If you want to have network connectivity in single-user mode, use the shell script /etc/netstart. This script calls the appropriate scripts to start the network, gives IP addresses to interfaces, and enables packet filtering and routing. If you want some, but not all, of these services, you’ll need to read that shell script and execute the appropriate commands manually.

Uses for Single-User Mode

In single-user mode, your access to the system is limited only by your knowledge of FreeBSD and Unix.

For example, if you’ve forgotten your root password, you can reset it from single-user mode:

# passwd
Changing local password for root
New Password:
Retype New Password:
#

Or, if you find that there’s a typo in /etc/fstab that confuses the system and makes it unbootable, you can mount the root partition with the device name and then edit /etc/fstab to resolve the issue.

If you have a program that panics the system on boot and you need to stop that program from starting again, you can either edit /etc/rc.conf to disable the program or set the permissions on the startup script so that it can’t execute.

# chmod a-x /usr/local/etc/rc.d/program.sh

We’ll discuss third-party programs (ports and packages) in Chapter 15.

You need to understand single-user mode to be a successful sysadmin, and we’ll refer to it throughout this book. For now, though, let’s look at the loader prompt.

The Loader Prompt

The loader prompt allows you to make basic changes to your computer’s boot environment and the variables that must be configured early in the boot process. It’s not a Unix-like environment; it’s cramped and supports only a minimal feature set. When you escape to a loader prompt (the third option in the boot menu), you’ll see the following:

Type '?' for a list of commands, 'help' for more detailed help.
OK

This is the loader prompt. While the word OK might be friendly and reassuring, it’s one of the few friendly things about the loader environment. This isn’t a full-featured operating system; it’s a tool for configuring a system boot that’s not intended for the ignorant nor the faint of heart. Any changes you make at the loader prompt affect only the current boot. To undo changes, reboot again. (We’ll see how to make loader changes permanent in the next section.)

To see all available commands, enter a question mark.

OK ?
Available commands:
  heap             show heap usage
  reboot           reboot the system
  lszfs            list child datasets of a zfs dataset
--snip--

Many loader commands aren’t useful to anyone except a developer, so we’ll focus on the commands useful to a system administrator.

Viewing Disks

To view the disks that the loader knows about, use lsdev.

   OK lsdev
➊ cd devices:
   disk devices:
➋ disk0:    BIOS drive C (33554432 X 512):
      ➌ disk0p1: FreeBSD boot
         disk0p2: FreeBSD swap
       disk0p3: FreeBSD ZFS
➍ zfs devices:
       zfs:zroot

The loader checks for CD drives ➊ and doesn’t find any. (The loader finds CD drives only if you boot from a CD, so don’t be alarmed by this.) It finds a hard drive, known to the BIOS as drive C ➋. It then describes the partitions on that hard drive. As we’ll see in Chapter 10, GPT partitions identify partitions with the letter p and a number. The partition disk0p1 ➌ is a FreeBSD boot partition used to bootstrap FreeBSD from the BIOS. You might find this knowledge useful on an unfamiliar system that’s having trouble booting. The loader can also identify the ZFS pools ➍ on the host.

Loader Variables

The loader has variables set within the kernel and by a configuration file. View these variables and their settings with the show command, and use the spacebar to advance to the next page.

OK show
LINES=24
acpi.oem=VBOX
acpi.revision=2
acpi.rsdp=0x000e0000
--snip--

These values include low-level kernel tunables and information gleaned from the hardware BIOS or UEFI. We’ll see a partial list of loader variables in “Loader Configuration” on page 57, and additional values will be brought up throughout the book in the appropriate sections.

You can show specific variables by name. Sadly, you can’t show all of a keyword’s sub-variables. A command like show acpi.oem works, but show acpi or show acpi.* doesn’t.

Change a value for a single boot with the set command. For example, to change the console setting to comconsole, you’d enter:

OK set console=comconsole

The loader lets you change variables that really shouldn’t change. Setting acpi.revision to 4 won’t suddenly upgrade your system to ACPI version 4, and you can’t change hard drives with a software setting.

Reboot

You didn’t mean to get into the loader? Start over.

Booting from the Loader

Now that you’ve twiddled your system’s low-level settings, you probably want to boot the system. Use the boot(8) command. You can adjust the boot further using the boot flags discussed in the man page.

Once your system boots just the way you need it to, you’ll probably want to make those settings permanent. FreeBSD lets you do this through the loader configuration file.

Loader Configuration

Make loader setting changes permanent with the configuration file /boot/loader.conf. Settings in this file are fed directly into the boot loader at system startup. Of course, if you enjoy being at your console every time the system boots, then don’t bother with this!

The loader has a default configuration file, /boot/defaults/loader.conf. We override many of the values here.

If you look at the default loader configuration, you’ll see many options that resemble variables listed in the loader. For example, here we can set the name of the console device:

console="vidconsole"

Throughout the FreeBSD documentation, you’ll see references to boot-time tunables and loader settings. All of these are set in loader.conf, which includes many sysctl values that are read-only once the system is up and kicking. (For more on tunables and sysctls, see Chapter 6.) Here, we force the kernel variable kern.maxusers to 32.

kern.maxusers="32"

Some of these variables don’t have a specific value set in loader.conf; instead, they appear as empty quotes. This means that the loader normally lets the kernel set this value, but if you want to override the kernel’s setting, you can.

kern.nbuf=""

The kernel has an idea of what the value of kern.nbuf should be, but you can have the loader dictate a different value if you must.

We’ll discuss system tuning via the boot loader in the appropriate section—for example, kernel values will be discussed in Chapter 6, where they’ll make something resembling sense—but here are some commonly used loader values that affect the appearance and operation of the loader itself and basic boot functionality. As FreeBSD matures, the developers introduce new loader values and alter the functionality of old ones, so be sure to check /boot/defaults/loader.conf on your installation for the current list.

boot_verbose="NO"

This value toggles the verbose boot mode that you can reach through the boot menu. In a standard boot, the kernel prints out a few basic notes about each device as it identifies system hardware. When you boot in verbose mode, the kernel tells each device driver to print out any and all information it can about each device as well as display assorted kernel-related setup details. Verbose mode is useful for debugging and development, but not generally for day-to-day use.

autoboot_delay="10"

This value indicates the number of seconds between the display of the boot menu and the automatic boot. I frequently turn this down to 2 or 3 seconds, as I want my machines to come up as quickly as possible.

beastie_disable="NO"

This value controls the appearance of the boot menu (originally, an ASCII art image of the BSD “Beastie” mascot decorated the boot menu). If set to YES, the boot menu will not appear.

loader_logo="fbsdbw"

This value allows you to choose which logo appears to the right of the boot menu. The fbsdbw option gives you the default FreeBSD logo in ASCII art. Other options include beastiebw (the original logo), beastie (the logo in color), and none (no logo).

Boot Options

The boot menu also presents three options: choosing a kernel, setting boot options, and selecting a boot environment. We’ll discuss each of these in an appropriate section, but here’s a bit to orient you.

A host can have multiple kernels in its /boot directory. Hitting the Kernel option tells the loader to cycle between the available options. To have a kernel appear as an option, list it in loader.conf in the kernels variable.

KERNELS="kernel kernel.old kernel.GENERIC"

The menu recognizes kernels only in directories beginning with /boot/kernel. If you have a kernel in /boot/gerbil, you’ll have to load it from the loader prompt.

FreeBSD supports a number of boot options. Selecting the Configure Boot Options item brings up the most popular.

Load System Defaults

You mucked with your settings and want to undo all that? Choose this. You can at least boot the system to single-user mode and fix your loader.conf.

ACPI Support

ACPI is the Advanced Configuration and Power Interface, an Intel/Toshiba/Microsoft standard for hardware configuration. It replaces and subsumes a whole bunch of obscure standards. ACPI has been a standard for many years now, but if a particular piece of hardware has trouble running FreeBSD, you can turn it off and see what happens. If you even think of trying this option, also read Chapter 24 and file a bug report.

Safe Mode

FreeBSD’s safe mode turns on just about every conservative option in the operating system. It turns off DMA and write caching on hard disks, limiting their speed but increasing their reliability. It turns off ACPI. 32-bit systems disable SMP. USB keyboards no longer work in safe mode. This option is useful for debugging older hardware.

Verbose

The FreeBSD kernel probes every piece of hardware as it boots. Most of the information discovered is irrelevant to day-to-day use, so the boot loader doesn’t display it. When you boot in verbose mode, FreeBSD prints all the details it can about every system setting and attached device. The information will be available later in /var/run/dmesg.boot, as discussed in the next section. I encourage you to try verbose mode on new machines, just to glimpse the system’s complexity.

Finally, the Select Boot Environment option lets you choose between ZFS boot environments, as discussed in Chapter 12.

Startup Messages

A booting FreeBSD system displays messages indicating the hardware attached to the system, the operating system version, and the status of various programs and services as they start. These messages are important when you first install your system and when you do troubleshooting. The boot messages always start off the same way, with a statement listing the copyrights for the FreeBSD Project and the Regents of the University of California:

Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.0-CURRENT #3 r320502: Fri Jun 30 13:48:50 EDT 2017
    root@storm:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)

In addition, you get a notice of the version of FreeBSD that’s booting, along with the date and time it was compiled and the compiler used. You can also see who compiled this kernel, what machine it was built on, and even where in the filesystem this kernel was built. If you build a lot of kernels, this information can be invaluable when trying to identify exactly what system features are available.

WARNING: WITNESS option enabled, expect reduced performance.

The kernel will print out diagnostic messages throughout the boot process. The preceding message means that I have debugging and fault-identifying code enabled in this particular kernel, and my performance will suffer as a result. In this case, I don’t care about the performance impact, for reasons which will become clear momentarily.

Timecounter "i8254" frequency 1193182 Hz quality 100

This message identifies a particular piece of hardware. The timecounter, or hardware clock, is a special piece of hardware, and while your computer needs one, it’s such a low-level device that the end user really can’t do much with it directly. Now and then, you’ll see messages like this for hardware that isn’t directly visible to the user but is vital to the system. The boot messages dance between showing too much detail and obscuring details that might be critical. For example, it’ll also show all the information it can about the CPU in the system:

CPU: Intel(R) Xeon(R) CPU E5-1620 v2 @ ➊3.70GHz (3700.08-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
              ➋ Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,
CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,
xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics

You probably didn’t know that a simple CPU could have so many details and features, did you? But when you file a trouble report that advanced features don’t work, a developer might respond by asking whether your CPU has a particular feature.

Here’s why I’m not worried about the performance hit caused by the WITNESS option shown earlier: this box is pretty darn fast ➊ and supports a whole bunch of features important to modern CPUs ➋. While I certainly want all the performance I paid for, I also want to catch any problems when they happen. I want to be able to file good bug reports on those problems, so the developers will listen to my problem report. That’s why I’m running a development version of FreeBSD that ships with WITNESS enabled, after all!

FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs

Here, the kernel announces that it’s found all eight CPU cores and is ready to manage them. I have CPU power to spare and a fair amount of memory as well.

➊ real memory  = 34359738368 (32768 MB)
➋ avail memory = 33207656448 ➌(31669 MB)

The real memory ➊ is the amount of RAM physically installed in the computer, while the avail memory ➋ is the amount of memory left over after the kernel is loaded. I have 31,669MB of RAM ➌ available for real work, which more than suffices for the load on this system.

➊ioapic0 <Version 2.0> ➋irqs 0-23 on motherboard
➌ioapic1 <Version 2.0> irqs 24-47 on motherboard

Here’s a fairly typical device entry. This device is known as ioapic, and the kernel has found that this hardware is version 2.0 and has extra information associated with it ➋. What’s more, we’ve found two devices of that type, numbered 0 ➊ and 1 ➌. (All devices are numbered starting with zero.) You can find out more about the device by reading the man page for the device driver. Almost all—but not all—device drivers have man pages.

usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0

Not all device drivers print all their information on a single line. Here, we have a single device, usbus, that takes up three lines with just a single instance of the device. The only way to know that this is a single USB bus rather than three separate ones is to check the number of the device. All of these are for device number zero, so it’s a single device.

➊ pci0: <simple comms> at device 22.0 (no driver attached)
➋ pcib8: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on ➌pci0
➍ pci8: <ACPI PCI bus> on ➎pcib8

One interesting thing about the boot messages is that they display how your computer’s components are attached to one another. Here, we have pci0 ➊, a PCI interface directly on the mainboard. Then, there’s pcib8 ➋, PCI bridge number eight attached to pci0 ➌. We also find PCI bus pci8 ➍ attached to that PCI bridge ➎. As you read on, you’ll find individual devices attached to that bus. You might not be equipped to do much with this information now, but you’ll find that having it available will be valuable when you have to troubleshoot a problem.

➊em0: <Intel(R) PRO/➋1000 Network Connection> port 0xd000-0xd01f mem
0xfba00000-0xfba1ffff,0xfba20000-0xfba23fff irq 18 at device 0.0 on pci9

This entry shows em0, a network card of type em(4) ➊, and indicates that the card speaks gigabit Ethernet ➋. We also see all sorts of information about its memory address, IRQ, and PCI bus attachment.

Every device on your computer has one or more entries like these. Taken as a whole, they describe your computer’s hardware in reasonable detail. If you boot in verbose mode, you’ll see even more detail—probably far more than you want.

One key thing that the kernel displays in the boot messages is the device name for each piece of hardware. This is critical information for managing your system. Every piece of hardware has a device node name, and to configure it, you’ll need to know that name. For example, earlier we saw an entry for an Ethernet card called em0. The card uses the em(4) driver, and the first device controlled by this driver has number zero. Your second device of this type would be em1, then em2, and so on.

Most devices that can be configured or managed have a device node entry somewhere under /dev. For example, the first optical drive is represented by the file /dev/cd0. These files are called device nodes, and they’re a convenient way to address a particular piece of hardware. Most device nodes can’t be directly accessed as a regular file; you can’t cat(1) a device node or copy another file to it. However, device nodes are used as arguments to specialized programs. For example, the hard drive that showed up at boot as ada4 is the same as the device node /dev/ada4. When you want to mount that hard drive, you can use the device node name and be sure you’re getting that exact piece of hardware.

Multiuser Startup

Beyond single-user mode, you’ll find multiuser mode. This is the standard operating mode for a Unix-like OS. If you’re doing real work, your system is in multiuser mode.

When FreeBSD finishes inspecting the hardware and attaching all the device drivers appropriately, it runs the shell script /etc/rc. This script mounts all filesystems, brings up the network interfaces, configures device nodes, identifies available shared libraries, and does all the other work necessary to make a system ready for normal work. Most systems have different startup requirements; while almost every server needs to mount a hard drive, a web server’s operating requirements are very different from those of a database server, even if it’s running on absolutely identical hardware. This means that /etc/rc must be extremely flexible. It achieves flexibility by delegating everything to other shell scripts responsible for specific aspects of the system.

The /etc/rc script is controlled by the files /etc/defaults/rc.conf and /etc/rc.conf.

/etc/rc.conf, /etc/rc.conf.d, and /etc/defaults/rc.conf

Much like the loader configuration file, the configuration of /etc/rc is split between two files: the default settings file, /etc/defaults/rc.conf, and the local settings file, /etc/rc.conf. Settings in /etc/rc.conf override any values given in /etc/defaults/rc.conf, exactly as with the loader.

The /etc/defaults/rc.conf file is huge and contains quite a few variables, frequently called knobs, or tunables. We aren’t going to discuss all of them, not only because knobs are added continually and such a list would be immediately obsolete but also because quite a few knobs aren’t commonly used on servers. Almost everything in a standard FreeBSD system has one or more rc.conf knobs, from your keyboard map to TCP/IP behavior. For a complete, up-to-date list, read rc.conf(5). To change rc.conf settings, you can either use a text editor or sysrc(8).

sysrc(8)

While editing rc.conf by hand works just fine, in this age of cloud computing, it’s not sustainable across large numbers of machines. If you must change dozens of servers, you need a reliable way to alter the system without either manually editing each server’s config or resorting to sed/awk hackery.²

FreeBSD includes sysrc(8), a command line program to consistently and safely alter /etc/rc.conf and friends from the command line. Additionally, sysrc(8) can display information about your system’s non-default settings.

Start by using -a to ask sysrc(8) what it knows about your host.

# sysrc -a
clear_tmp_enable: YES
defaultrouter: 203.0.113.1
dumpdev: AUTO
keymap: us.dvorak.kbd
--snip--

You’ll get a list of all non-default /etc/rc.conf settings.

To have sysrc(8) enable a service, give it the variable name, an equals sign, and the new value.

# sysrc rc_startmsgs=NO
rc_startmsgs: YES -> NO

The variable rc_startmsgs is now set to no.

Remember that sysrc(8) is a tool for changing rc.conf, not for configuring FreeBSD. It does no validity checking. One of my very junior sysadmins really doesn’t want Bert logging in, and he took some bad advice on how to prevent that.

# sysrc bert=no

While this code sets bert="no" in /etc/rc.conf, this variable doesn’t do anything. Remove it with the -x flag.

# sysrc -x bert

Many FreeBSD configuration files closely resemble rc.conf. You can use sysrc(8) to manage them by adding the -f flag and the file name.

# sysrc -af /boot/loader.conf

Should you edit rc.conf or use sysrc(8)? If you’re making manual changes, then use whichever you prefer. Automation should err on the side of sysrc(8). This book mixes examples of both.

/etc/rc.conf.d/

If you use a server configuration system such as Puppet or Ansible, you might trust copying entire files more than editing them. Use /etc/rc.conf.d/ files to enable services through such tools.

To manage a service in /etc/rc.conf.d/, create a file named after the service. That is, to manage bsnmpd(8) you’d create /etc/rc.conf.d/bsnmpd. Enable or disable that service in this file.

bsnmpd_enable=YES

I normally use Ansible’s service enabling features that directly alter /etc/rc.conf rather than /etc/rc.conf.d, but use whatever you prefer.

The next few sections illustrate the types of things you can enable and disable in /etc/rc.conf. Each appears in /etc/defaults/rc.conf and can be overridden by an /etc/rc.conf entry. Each variable appears with its default setting.

Startup Options

The following rc.conf options control how FreeBSD configures itself and starts other programs. These far-reaching settings affect how all other system programs and services run.

If you’re having a problem with the startup scripts themselves, you might enable debugging on /etc/rc and its subordinate scripts. This can provide additional information about why a script is or isn’t starting.

rc_debug="NO"

If you don’t need the full debugging output but would like some additional information about the /etc/rc process, enable informational messages with rc_info:

rc_info="NO"

When the boot process hits multiuser startup, it prints out a message for each daemon it starts. Remove those messages with the rc_startmsgs option.

rc_startmsgs="NO"

Filesystem Options

FreeBSD can use memory as a filesystem, as we’ll discuss in Chapter 13. One common use for this feature is to make /tmp really fast by using memory rather than a hard drive as its backend. Once you’ve read Chapter 13, you might consider implementing this. Variables in rc.conf let you enable a memory-backed /tmp and set its size transparently and painlessly. You can also choose the options FreeBSD will use to complete the filesystem. (The impatient among you are probably wondering what the -S flag means. It means disable soft updates. If you have no idea what this means, either, wait for Chapter 11.) If you want to use a memory filesystem /tmp, set tmpmfs to YES and set tmpsize to the desired size of your /tmp.

tmpmfs="AUTO"
tmpsize="20m"
tmpmfs_flags="-S"

Another popular FreeBSD filesystem feature is its integrated encrypted partitions. FreeBSD supports two different filesystem encryption systems out of the box: GBDE and GELI. GEOM-Based Disk Encryption (GBDE) was FreeBSD’s first encrypted filesystem designed for military-grade use. GELI is a little more friendly and complies with different standards than GBDE. (You definitely want to read Chapter 23 before enabling either of these!)

geli_devices=""
geli_tries=""
geli_default_flags=""
geli_autodetach="YES"

By default, FreeBSD mounts the root partition read-write upon achieving multiuser mode. If you want to run in read-only mode instead, you can set the following variable to NO. Many people consider this more secure, but a read-only root can interfere with operation of certain software, and it’ll certainly prevent you from editing any files on the root partition!

root_rw_mount="YES"

When a booting FreeBSD attempts to mount its filesystems, it checks them for internal consistency. If the kernel finds major filesystem problems, it can try to fix them automatically with fsck -y. While this is necessary in certain situations, it’s not entirely safe. (Be sure to read Chapter 11 very carefully before enabling this!)

fsck_y_enable="NO"

The kernel might also find minor filesystem problems, which it resolves on the fly using a background fsck while the system is running in multiuser mode, as discussed in Chapter 11. There are legitimate concerns about the safety of using this feature in certain circumstances. You can control the use of background fsck and set how long the system will wait before beginning the background fsck.

background_fsck="YES"
background_fsck_delay="60"

Miscellaneous Network Daemons

FreeBSD includes many smaller programs, or daemons, that run in the background to provide specific services. We’ll cover quite a few of these integrated services throughout the book, but here are a few specific ones that’ll be of interest to experienced system administrators. One popular daemon is syslogd(8). Logs are a Good Thing. Logs are so very, very good that large parts of Chapter 21 are devoted to the topic of logging with, for, by, and on FreeBSD.

syslogd_enable="YES"

Once you’ve decided to run the logging daemon, you can choose exactly how it’ll run by setting command line flags for it. FreeBSD will use these flags when starting the daemon. For all the programs included in rc.conf that can take command line flags, the flags are given in this format:

syslogd_flags="-s"

Another popular daemon is inetd(8), the server for small network services. (We cover inetd in Chapter 20.)

inetd_enable="NO"

Most systems use the Secure Shell (SSH) daemon for remote logins. If you want to connect to your system remotely over the network, you’ll almost certainly need SSH services.

sshd_enable="NO"

While the SSH daemon can be configured via the command line, you’re generally better off using the configuration files in /etc/ssh/. See Chapter 20 for details.

sshd_flags=""

FreeBSD also incorporates extensive time-keeping software that functions to ensure the system clock remains synchronized with the rest of the world. You’ll need to configure this for it to be useful; we’ll cover that in Chapter 20.

ntpd_enable="NO"
ntpd_flags="-p /var/run/ntpd.pid -f /var/db/ntpd.drift"

In addition, FreeBSD includes a small SNMP daemon for use in facilities with SNMP-based management tools. We’ll cover configuring SNMP in Chapter 21.

bsnmpd_enable="NO"

Network Options

These knobs control how FreeBSD configures its network facilities during boot. We’ll discuss networking in Chapter 7.

Every machine on the internet needs a hostname. The hostname is the fully qualified domain name of the system, such as www.absolutefreebsd.org. Many programs won’t run properly without this.

hostname=""

FreeBSD includes a few different integrated firewall packages. We’re going to briefly cover the packet filter (PF) in Chapter 19. Enable and disable PF in rc.conf.

pf_enable="NO"

You might be interested in failed attempts to connect to your system over the network. This will help detect port scans and network intrusion attempts, but it’ll also collect a lot of garbage. It’s interesting to set this for a short period of time just to see what really happens on your network. (Then again, knowing what’s really going on tends to cause heartburn.) Set this to 1 to log failed connection attempts.

log_in_vain="0"

Routers use ICMP redirects to inform client machines of the proper network gateways for particular routes. While this is completely legitimate, on some networks intruders can use this to capture data. If you don’t need ICMP redirects on your network, you can set this option for an extremely tiny measure of added security. If you’re not sure whether you’re using them, ask your network administrator.

icmp_drop_redirect="NO"

If you are the network administrator and you’re not sure whether your network uses ICMP redirects, there’s an easy way to find out—just log all redirects received by your system to /var/log/messages.³ Note that if your server is under attack, this can fill your hard drive with redirect logs fairly quickly.

icmp_log_redirect="NO"

To get on the network, you’ll need to assign each interface an IP address. We’ll discuss this in some detail in Chapter 8. You can get a list of your network interfaces with the ifconfig(8) command. List each network interface on its own line, with its network configuration information in quotes. For example, to give your em0 network card an IP address of 172.18.11.3 and a netmask of 255.255.254.0, you would use:

ifconfig_em0="inet 172.18.11.3 netmask 255.255.254.0"

If your network uses DHCP, use the value dhcp as an IP address.

ifconfig_em0="dhcp"

Similarly, you can assign aliases to a network card. An alias is not the card’s actual IP address, but the card answers for that IP address, as discussed in Chapter 8. FreeBSD supports hundreds of aliases on a single card, with rc.conf entries in the following form:

ifconfig_em0_aliasnumber="address netmask 255.255.255.255"

The alias numbers must be continuous, starting with 0. If there’s a break in numbering, aliases above the break won’t be installed at boot time. (This is a common problem, and when you see it, check your list of aliases.) For example, an alias of 192.168.3.4 would be listed as:

ifconfig_em0_alias0="192.168.3.4 netmask 255.255.255.255"

Network Routing Options

FreeBSD’s network stack includes many features for routing internet traffic. These start with the very basic, such as configuring an IP for your default gateway. While assigning a valid IP address to a network interface gets you on the local network, a default router will give you access to everything beyond your LAN.

defaultrouter=""

Network control devices, such as firewalls, must pass traffic between different interfaces. While FreeBSD won’t do this by default, it’s simple to enable. Just tell the system that it’s a gateway and it’ll connect multiple networks for you.

gateway_enable="NO"

Console Options

The console options control how the monitor and keyboard behave. You can change the language of your keyboard, the monitor’s font size, or just about anything else you like. For example, the keyboard map defaults to the standard US keyboard, frequently called QWERTY. You’ll find all sorts of keymaps in the directory /usr/share/syscons/keymaps. I prefer the Dvorak keyboard layout, which has an entry there as us.dvorak. By changing the keymap knob to us.dvorak, my system will use a Dvorak keyboard once it boots to multiuser mode.

keymap="NO"

FreeBSD turns the monitor dark when the keyboard has been idle for a time specified in the blanktime knob. If you set this to NO, FreeBSD won’t dim the screen. Mind you, new hardware will dim the monitor after some time as well, to conserve power. If your screen goes blank even if you’ve set the blanktime knob to NO, check your BIOS and your monitor manual.

blanktime="300"

FreeBSD can also use a variety of fonts on the console. While the default font is fine for servers, you might want a different font on your desktop or laptop. My laptop has one of those 17-inch screens proportioned for watching movies, and the default fonts look kind of silly at that size. You can choose a new font from the directory /usr/share/syscons/fonts. Try a few to see how they look on your systems. The font’s name includes the size, so you can set the appropriate variable. For example, the font swiss-8x8.fnt is the Swiss font, 8 pixels by 8 pixels. To use it, you would set the font8x8 knob.

font8x16="NO"
font8x14="NO"
font8x8="YES"

You can use a mouse on the console, even without a GUI. By default, FreeBSD will try to autodetect your mouse type. If you have a PS/2 or USB mouse, chances are that it’ll just work when you enable the mouse daemon, without any special configuration. Some older and more unusual types of mice require manual configuration, as documented in moused(8).

moused_enable="NO"
moused_type="AUTO"

You can also change the display on your monitor to fit your needs. If you have an odd-sized monitor, you can change the number of lines of text and their length to fit, change text colors, change your cursor and cursor behavior, and do all sorts of other little tweaks. You can get a full list of different options in man vidcontrol(1).

allscreens_flags=""

Similarly, you can adjust your keyboard behavior almost arbitrarily. Everything from key repeat speed to the effect of function keys can be configured, as documented in kbdcontrol(1).

allscreens_kbdflags=""

Other Options

This final potpourri of knobs might or might not be useful in any given environment, but they’re needed frequently enough to deserve mention. For example, not all systems have access to a printer, but those that do will want to run the printing daemon lpd(8). We brush up against printer configuration in Chapter 20.

lpd_enable="NO"

The sendmail(8) daemon manages transmission and receipt of email between systems. While almost all systems need to transmit email, most FreeBSD machines don’t need to receive email. The sendmail_enable knob specifically handles incoming mail, while sendmail_outbound_enable allows the machine to transmit mail. See Chapter 20 for more details.

sendmail_enable="NO"
sendmail_submit_enable="YES"

One of FreeBSD’s more interesting features is its ability to run software built for Linux. We discuss this feature in Chapter 17. Running Linux software isn’t quite as easy as throwing this toggle, so don’t enable Linux compatibility modes without reading that chapter first!

linux_enable="NO"

A vital part of any Unix-like operating system is shared libraries. You can control where FreeBSD looks for shared libraries. Although the default setting is usually adequate, if you find yourself regularly setting the LD_LIBRARY_PATH environment variable for your users, you should consider adjusting the library path instead. See Chapter 17 for more advice on the library path.

ldconfig_paths="/usr/lib /usr/local/lib"

FreeBSD has a security profile system that allows the administrator to control basic system features. You can globally disallow mounting hard disks, accessing particular TCP/IP ports, and even changing files. See Chapter 9 for details on how to use these.

kern_securelevel_enable="NO"
kern_securelevel="-1"

Now that you know a smattering of the configuration knobs FreeBSD supports out of the box, let’s see how they’re used.

The rc.d Startup System

FreeBSD bridges the gap between single-user mode and multiuser mode via the shell script /etc/rc. This script reads in the configuration files /etc/defaults/rc.conf and /etc/rc.conf, and runs a collection of other scripts based on what it finds there. For example, if you’ve enabled the network time daemon, /etc/rc runs a script written specifically for starting that daemon. FreeBSD includes scripts for starting services, mounting disks, configuring the network, and setting security parameters.

These scripts live in /etc/rc.d and /usr/local/etc/rc.d. I’d recommend reading a few of them if only to see how the rc.d system works.

Control these scripts with service(8).

The service(8) Command

All of the rc.d scripts are readable, and the way they fit together is pretty straightforward. When you have a problem, you can read the scripts to see how they work and what they do. But that’s a lot like work, and most sysadmins have more interesting work to do. The service(8) command provides a friendly frontend to the rc.d scripts. You can use service(8) to see which scripts run automatically; to stop, start, and restart services; to check the status of a service; and more.

Listing and Identifying Enabled Services

Use the -e flag to service(8) to see the full path of all scripts that’ll be run at system boot, in the order they’ll be run.

# service -e
/etc/rc.d/hostid
/etc/rc.d/zvol
/etc/rc.d/hostid_save
/etc/rc.d/zfsbe
--snip--
/etc/rc.d/sshd
/etc/rc.d/sendmail
--snip--

This tiny host runs 23 scripts at boot.

One important detail here is the script name. You’ll use the script name in other commands, like starting, stopping, and restarting services.

Managing Services

While it’s entirely possible to restart, say, sshd(8) at the command line, a production host needs everything to run consistently. Best practice calls for using service(8) to manage processes. You’ll need the script name as shown earlier, but without the directory path.

# service name command

For example, suppose I want to restart the sshd(8) service. According to the service -e output shown earlier, there’s a script /etc/rc.d/sshd. I strongly suspect this script manages sshd(8), but I want to be certain I don’t accidentally restart the Stupidly Similarly named Harassment Daemon. This is where the describe command comes in. Let’s ask service(8) to describe the sshd script.

# service sshd describe
Secure Shell Daemon

It’s the right daemon. Let’s restart it.

   # service sshd restart
➊ Performing sanity check on sshd configuration.
➋ Stopping sshd.
➌ Performing sanity check on sshd configuration.
➍ Starting sshd.

Restarting a service is a combination of “stop the service” and “start the service.” This particular service does more than that, though. It starts by verifying the configuration file ➊ and then stopping the daemon ➋. It then reverifies the configuration ➌ and starts the daemon ➍. Why?

SSH handles remote access to this host. If the SSH service breaks, nobody can log into the host to fix the SSH service. Yes, you could use a remote KVM or IPMI or drive to the colocation facility, but any of these prolongs the outage. It’s much better to verify that sshd(8) can be restarted before shutting it down. Many service scripts include this kind of safety check. If a service complains that it can’t stop, read the output carefully to find out why.

The commands each service supports vary. The easiest way to get the full list of commands a particular service supports is to give the service a bogus argument. Something like “bert” is pretty bogus.

# service sshd bert
/etc/rc.d/sshd: unknown directive 'bert'.
Usage: /etc/rc.d/sshd [fast|force|one|quiet](start|stop|restart|rcvar|enabled|
describe|extracommands|configtest|keygen|reload|status|poll)

You get a full list of commands this service supports, in two groups.

The first group, in square brackets, contains options for the commands. Here are the standard options. Use them as prefixes for the commands in the second group.

fast Do no checking (used during startup).

force Try harder.

one Start this service despite not being enabled in rc.conf.

quiet Only print service name (used during startup).

The second group, in parentheses, contains the following commands:

start Start the service.

stop Stop the service.

restart Stop and restart the service.

rcvar Print the rc.conf variables for this service.

enabled Return true in shell if enabled (for script use).

describe Print service description.

extracommands Show service-specific commands.

The extracommands command is very specific to the service and lists only the additional commands this service accepts. By default, the extra commands appear after the default commands. Here are some common extra commands:

configtest Parse the service’s configuration file and stop if there’s an error.

reload Perform a soft reload (usually via SIGHUP) rather than a restart.

status Determine whether service is running.

To determine exactly what a service’s extra commands do, you need to read the service script.

We’ll look at rc.d in more detail in Chapter 17, when we discuss customizing and writing your own rc.d scripts.

System Shutdown

FreeBSD makes the rc.d startup system do double duty; not only must it handle system startup, it must also shut all those programs down when it’s time to power down. Something has to unmount all those hard drives, shut down the daemons, and clean up after doing all the work. Some programs don’t care whether they’re unceremoniously killed when the system closes up for the night—after all, after the system goes down, any clients connected over SSH will be knocked off and any half-delivered web pages remain incomplete. Database software, however, cares very much about how it’s turned off, and unceremoniously killing the process will damage your data. Many other programs that manage actual data are just as particular, and if you don’t let them clean up after themselves, you’ll regret it.

When you shut down FreeBSD with the shutdown(8), halt(8), or reboot(8) commands, the system calls the shell script /etc/rc.shutdown. This script calls each rc.d script in turn with the stop option, reversing the order they were called during startup, thereby allowing server programs to terminate gracefully and disks to tidy themselves up before the power dies.

Serial Consoles

All this console stuff is nice, but when your FreeBSD system is in a colocation facility on the other side of the country or on another continent, you can’t just walk up to the keyboard and start typing. Many data centers won’t have room for a keyboard or monitor. And how do you reset the machine remotely when it won’t respond to the network? Using a serial console to redirect the computer’s keyboard and video to the serial port instead of the keyboard and monitor helps with all of these problems.

Serial consoles can be physical, such as a serial port on the back of a computer. By hooking up a standard null modem cable to the serial port and attaching the other end to another computer’s serial port, you can access the first system’s boot messages from the second computer.

They might also be virtual, as provided by IPMI’s Serial-over-LAN (SOL) protocol. Rather than a null modem cable, you’ll need to set up the IPMI interface and use special software to configure and access the virtual serial port.

Before we set up a port, though, let’s talk about serial port protocol.

Serial Protocol

Some of the first computer consoles were serial ports connected to teletypes. Serial has been around a long time and has evolved over the decades. Unlike modern protocols, serial lines do not autonegotiate. You must configure both sides of a serial link to the exact same settings. A configuration mismatch will cause either a blank screen or gibberish.

Original serial lines worked at low speeds. Many of the serial cables remain basically the same, but we’ve developed better software and hardware to stick at each end that allows us to transmit data much faster. Where old serial connections ran at 300 bits per second (baud), a whole bunch of modern hardware can run at 115,200 baud. Across hardware platforms, though, the common standard is 9600 baud, which is FreeBSD’s default console speed. A baud rate of 9600 is enough to carry whole screens of text at a comfortable speed.

Stick with 9600 baud for physical connections, unless you can’t. Some modern hardware doesn’t support 9600 baud. Some claim to support 9600 baud, but don’t. I’ve worked with devices hardcoded to 115,200 baud. Anything that fails or flat-out refuses to do 9600 baud is busted by design, but we often don’t control the choice of hardware. Changing the serial console speed for reasons other than hardware limitations makes your connection more fragile, and if you’re using the console, you’re in no mood for fragility. When I mention changing the port speed, that’s for use only when you have to.

SOL connections aren’t physical wires, so you don’t have to worry about line noise. You can safely run them at higher speeds.

Serial protocols also include a whole bunch of settings beyond their speed. It’s possible to muck with them, but the standard settings of 8 data bits, no parity, and 1 stop bit are the most widely used. You can’t change these in FreeBSD without recompiling the kernel, so don’t muck with them.

With that in mind, let’s set up a console.

Physical Serial Console Setup

No matter what sort of serial console you have, you’ll need to plug into it correctly to make it work. You’ll need a null modem cable, available at any computer store or from online vendors. While the gold-plated serial cables are not worth the money, don’t buy the cheapest cable you can find either; if you have an emergency and need the serial console, you’re probably not in the mood to endure line noise!⁴

Plug one end of the null modem cable into the serial console port on your FreeBSD server—by default the first serial port (COM1 or uart0, depending on what operating system you’re used to). You can change this with a server.

Plug the other end of your null modem cable into an open serial port on another system. I recommend either another FreeBSD (or other Unix) system or a terminal server, but you can use a Windows box if that’s all you have.

If you have two FreeBSD machines at a remote location, make sure that they each have two serial ports. Get two null modem cables and plug the first serial port on each box into the second serial port of the other machine. That way, you can use each machine as the console client for the other. If you have three machines, daisy-chain them into a loop. By combining twos and threes, you can get serial consoles on any number of systems. I’ve worked data centers with 30 or 40 FreeBSD machines, where installing monitors was simply not practical, and we used serial consoles to great effect. Once you have a rack or two of servers, however, investing in a terminal server is a really good idea. You can find them cheaply on eBay.

Another option is to use two DB9-to-RJ45 converters, one standard and one crossover. These allow you to run your console connections over a standard CAT5 cable. If you have a lights-out data center where human beings are not allowed, you can have your serial consoles come out near your desk, in your warm room, or anywhere else your standard Ethernet-style patch panels reach. Most modern data facilities are better equipped to handle Ethernet than serial cables.

IPMI Serial Console Setup

The Intelligent Platform Management Interface (IPMI) is a standard for managing computer systems at a hardware level. IPMI runs separately from the operating system, using a small device called a baseboard management controller (BMC). Essentially, the BMC acts as your remote hands and eyes to control the server. To use an IPMI console, you’ll need to configure both the BMC and the host’s BIOS or UEFI.

I’ll try to orient you here, but the best resource for configuring BMC or UEFI is your hardware manual.⁵

BMC Setup

A server’s BMC has its own IP address and normally gets a dedicated Ethernet port on the mainboard. Each vendor gets to design its own BMC in a way that conforms to its own biases. This means that configuring the BMC is way, way beyond the scope of this book, but here are a few hints.

You configure most BMCs through a web interface. Before you can access the web interface, though, the BMC needs an IP address. Set most BMC IP information in the BIOS or UEFI firmware’s setup menu. Once you get in the management interface, configure a username and password. Remember them.

A usual BMC also includes functions such as power cycling the main system, remote console access via some sort of downloaded application (often Java), virtual media, and more.

Never forget that the BMC is a small embedded computer running a web server and that it was written by some overworked corporate employee charged with building the minimum viable product. The BMC wasn’t tested for how it performed after several months of uptime. If it gives you even a sneeze of trouble, reboot it. No, you don’t have to power cycle the whole computer; there’s usually a “BMC Reset” or “Unit Reboot” menu option somewhere in the web interface.

If the BMC supports an applet-based console, why use a serial console? Because the BMC console is applet-based and BMC firmwares are rarely updated. I have quite a few BMC consoles that work only with obsolete, insecure⁶ versions of Java. Using them requires overriding security warnings and repeatedly clicking the “Yes, I know I’m an idiot, do it anyway” box. I have to keep a virtual machine with this insecure Java version specifically to access those consoles. The applet-based console doesn’t support copy and paste, and is often very laggy.

IPMI works better than the Java console applet over slower connections. I can copy and paste. Also, I can use the IPMI console from the command line, from any modern operating system.

While you’re in the BMC setup, locate the option to launch SOL. That brings up an applet to connect to the host’s SOL interface, which will help you test your serial console configuration.

UEFI/BIOS Serial Console Configuration

Once your BMC is ready, you must configure the server hardware to direct a serial port to the BMC. Go to the hardware’s Setup menu, where you configure your UEFI or BIOS. Somewhere in that maze of twisty little options, you’ll find something like “Serial Port Console Redirection.”

A vital question here is, how many serial ports does your host have? Maybe it has none. Maybe it has several. You can choose to redirect one of those ports or add an additional, virtual port. I encourage you to leave your existing serial ports alone and add a virtual port dedicated to SOL. It’s probably called something like “SOL Console Redirection.” Enable it, and go into the settings for that port.

Here are some settings I find helpful for FreeBSD and SOL:

Terminal type vt100

Data bits 8

Parity none

Stop bits 1

Flow control none

The tricky part is the baud, speed, or bits-per-second setting. Stick with the default speed, but make a note of it. You’ll need the speed to connect.

Now that you have a serial console, set up FreeBSD.

Configuring FreeBSD’s Serial Console

As FreeBSD boots, the loader decides where to print console messages and where to accept input from. While this defaults to the monitor and keyboard, with a few tweaks, you can redirect the console to a serial port. The serial console won’t grant BIOS access, but you can tweak the FreeBSD boot itself in almost any way. You can configure a serial console in either the first- or second-stage boot loader.

A first-stage boot loader gets you console access at the earliest possible moment but requires you use the first serial port as a console. Changing the port requires recompiling the kernel. The first-stage boot loader allows you to perform tasks like choose which disk you’re going to load the second-stage loader from—essentially, to boot from a disk other than the disk the BIOS or UEFI selected. This is undeniably useful, but very few users need this.

The second-stage boot loader can use any serial port as a console, but the first bit of output you’ll get is the boot menu discussed in “The Loader Prompt” on page 55. For most of us, that’s perfectly acceptable.

Console Options

FreeBSD’s default configuration uses the monitor and keyboard as the console. You can choose to switch to only the serial console or to use a dual console. Choose which with the /boot/loader.conf option console.

A serial-only console prevents some random colocation employee from power cycling your box, plugging in a monitor, and dinking with the menu. Yes, they could still work mayhem from the first-stage loader or boot off of USB, but that requires greater skill. Set the console variable to comconsole to use only the serial port as a console.

console="comconsole"

For most deployments, I recommend a dual console. Dual consoles show console activity on both the serial port and the monitor. You can use either the standard or the serial console as needed. Specify a dual-console configuration by listing both comconsole and vidconsole.

console="comconsole vidconsole"

If you’re in a server-room situation, you might want to switch back and forth between a standard console and a serial console. I generally manage large arrays of FreeBSD systems via the serial console but leave the video console in place in case of trouble.

The console won’t be effective until after a reboot. You can see whether FreeBSD put its console on a serial port by checking the boot messages.

uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)

The second line shows that the serial port uart0 is configured as a console, using the default settings. We’ll look at those settings in “Using Serial Consoles” on page 79.

Advanced Console Options

In addition to enabling the console, you can adjust the console’s port and the speed.

Maybe I need to use the second serial port for the console. Perhaps the first serial port has something plugged into it, or maybe the second port is the virtual SOL port. Serial ports use the uart(4) device driver. Remember that FreeBSD devices start numbering at zero, while COM ports start numbering at 1. COM1 is uart0, COM2 is uart1, and so on. You’ll need the port’s base I/O port, which you can get from the system bootup messages.

# grep uart /var/run/dmesg.boot
uart0: <16550 or compatible> port ➊0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port ➋0x2f8-0x2ff irq 3 on acpi0

The first number after the word port is the base I/O port ➊. The base address of COM2, or uart1, is 0x2f8. Set comconsole_port to this value ➋.

comconsole_port="0x2f8"

Your console is now on serial port COM2.

If my serial connection won’t do 9600 baud, I can change the port speed with the comconsole_speed option.

comconsole_speed="115200"

On a physical port, don’t increase the port speed just because you can.

Using Serial Consoles

Now that you have both physical and software set up, configure your client to access the serial console. The key to using a serial console is to remember the following settings:

• Speed (9600 baud, or whatever your hardware is set to)
• 8 bits
• No parity
• 1 stop bit

The way you access a serial line depends on whether it’s a physical line or an IPMI SOL connection.

Physical Serial Lines

Connect your client to the other end of the serial line. You can find terminal emulators for Microsoft platforms (PuTTY being the most famous), macOS, and almost any other operating system. Once upon a time, I used a Palm handheld with a serial cable to access serial consoles. Enter the correct value settings into the terminal emulator, and the serial console will “just work.”

FreeBSD accesses serial lines with tip(1), a program that allows you to connect to remote systems in a manner similar to telnet. To run tip, do this as root:

# tip portname

A port name is shorthand for specifying the serial port number and speed to be used on a serial port. The file /etc/remote contains a list of port names. Most of the entries in this file are relics of the eon when UUCP was the major data transfer protocol and serial lines were the norm instead of the exception.⁷ At the end of this file, you’ll see a few entries like:

# Finger friendly shortcuts
uart0|com1:dv=/dev/cuau0:br#9600:pa=none:
uart1|com2:dv=/dev/cuau1:br#9600:pa=none:
--snip--

The uart entries are the standard Unix-type device names, while the com names were added for the convenience of people who grew up on x86 hardware.

Assume that you have two FreeBSD boxes wired back-to-back, with each one’s serial port 1 null-modemed into serial port 2. Both machines are configured to use a serial console. You’ll want to connect to your local serial port 2 to talk to the other system’s serial console:

# tip uart1
connected

You’re in!

To disconnect the serial console, press ENTER and then type the disconnect sequence “tilde-dot” at any time.

~.

You’ll be gracefully disconnected. (This also works in the OpenSSH client.)

The tip(1) program uses the tilde (~) as a control character. Read the man page for a full list of things you can do with it.

IPMI SOL Connections

You’ll need a SOL client to connect to your IPMI serial port. The quickest way to test your configuration is probably with the SOL client applet included in your BMC. While that client has most of the disadvantages of the console applet, it’s a good place to test. If the BMC SOL client doesn’t work, check your SOL settings and FreeBSD configuration. Verify that the SOL client is set to use the same speed you set in the hardware and in FreeBSD. If it doesn’t work but all your settings appear to match, reboot the BMC. Once it works, you can use SOL from another host.

The standard IPMI SOL client is IPMItool (https://sourceforge.net/projects/ipmitool/), available as the ipmitool package. (Chapter 15 discusses packages.) IPMItool can interact with your BMC over the network, granting you all of the BMC functions without logging into a clunky web interface. You can reboot the host, check hardware alarms and sensors, and more, all from the command line. But for the moment, we’ll stick with the SOL console. Use the BMC’s hostname or IP, the username, and the password to log into SOL.

# ipmitool -H bmc -U username -I lanplus sol activate

Here, I log into my web server’s BMC, with the hostname www-bmc, using the username “bert.”

# ipmitool -H www-kvm -U bert -I lanplus sol activate

Enter the password at the prompt, and the SOL will acknowledge your login.

[SOL Session operational.  Use ~? for help]

We have a console. Probably. Let’s do the final test.

Working at the Console

The real test of a serial console is whether or not you can get data across it. Once you have your console connected, hit ENTER.

FreeBSD/amd64 (www) (ttyu2)

login:

FreeBSD permits logins on serial consoles by default. Log in to the host and reboot it, and you’ll get the usual console messages.

Jul 13 11:48:24 Stopping cron.
Stopping sshd.
Stopping devd.
Writing entropy file:.
Writing early boot entropy file:.
Terminated
.
Jul 13 11:48:24 zfs1 syslogd: exiting on signal 15
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `bufdaemon' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 done
All buffers synced.

There will be a long pause while the system runs its BIOS routines and hands control over to the serial console. Just about the time you decide that the machine is never coming back up, you’ll get the loader menu. Congratulations! You’re using a serial console. Press the spacebar to interrupt the boot just as if you were at the keyboard.

It doesn’t matter how far away the system is; you can change your booting kernel, get a verbose boot, bring it up in single-user mode, or manually fsck the hard drive—whatever. A software serial console might not show you the BIOS, but chances are that’s set up correctly already. Once you’ve used a serial console for a while, it won’t matter whether the machine is on the other side of the world or the other side of the room; getting out of your chair merely to access the console will feel like too much work.

If a system in a remote location entirely locks up, you can connect to your serial console and have the “remote hands” at the colocation facility power-cycle the system. It might not be good for your computer, but it’s also not good for it to be locked up. With the serial console, you can boot into single-user mode and fix the problem by digging through the logs and whatever other troubleshooting you feel capable of. We’ll discuss troubleshooting this sort of problem in Chapter 24.

Now that you understand how FreeBSD starts up and shuts down, let’s look at some basic tools you can use to ensure that your system will continue to run even after you’ve been experimenting with it.