6
KERNEL GAMES

If you’re new to Unix administration, the word kernel might intimidate you. After all, the kernel is one of those secret parts of a computer that mere mortals are not meant to dabble in. In some versions of Unix, kernel tampering is unthinkable. Microsoft doesn’t advertise that its operating systems even have kernels, which is like glossing over the fact that human beings have brains.¹ While high-level users can access the kernel through a variety of methods, this isn’t widely acknowledged or encouraged. In many parts of the open source Unix-like world, however, meddling with the kernel is a very viable and expected way to change system behavior. It would probably be an excellent way to adjust other operating systems, if you were allowed to do so.

The FreeBSD kernel can be dynamically tuned or changed on the fly, and most aspects of system performance can be adjusted as needed. We’ll discuss the kernel’s sysctl interface and how you can use it to alter a running kernel.

At the same time, some parts of the kernel can be altered only while the system is in the early stages of booting. The boot loader lets you adjust the kernel before the host even finds its filesystems.

Some kernel features require extensive reconfiguration. You can custom-build kernels for really tiny systems or build a kernel tuned precisely for the hardware you’re running. The best way to do this is to build your own kernel.

FreeBSD has a modular kernel, meaning that entire chunks of the kernel can be loaded or unloaded from the operating system, turning entire subsystems on or off as desired. This is highly useful in this age of removable hardware, such as PC cards and USB devices. Loadable kernel modules can impact performance, system behavior, and hardware support.

Finally, we’ll cover basic debugging of your kernel, including some of the scary-looking messages it gives out as well as when and how to boot alternate kernels.

What Is the Kernel?

You’ll hear many different definitions of a kernel. Many are just flat-out confusing, some are technically correct but bewilder the novice, while others are wrong. The following definition isn’t complete, but it’ll do for most people most of the time and it’s comprehensible: the kernel is the interface between the hardware and the software.

The kernel lets the software write data to disk drives and to the network. When a program wants memory, the kernel handles all the low-level details of accessing the physical memory chip and allocating resources for the job. Once your MP3 file passes through the codec software, the kernel translates the codec output into a stream of zeros and ones that your particular sound card understands. When a program requests CPU time, the kernel schedules a time slot for it. In short, the kernel provides all the software interfaces that programs need in order to access hardware resources.

While the kernel’s job is easy to define (at least in this simplistic manner), it’s actually a complicated task. Different programs expect the kernel to provide different interfaces to the hardware, and different types of hardware provide interfaces differently. For example, FreeBSD supports a few dozen families of Ethernet cards, each with its own requirements that the kernel must handle. If the kernel can’t talk to the network card, the system isn’t on the network. Different programs request memory to be arranged in different ways, and if you have a program that requests memory in a manner the kernel doesn’t support, you’re out of luck. The way your kernel investigates some hardware during the boot sequence defines how the hardware behaves, so you have to control that. Some devices identify themselves in a friendly manner, while others lock up if you dare to ask them what they’re for.

The kernel and any modules included with FreeBSD are files in the directory /boot/kernel. Third-party kernel modules go in /boot/modules. Files elsewhere in the system are not part of the kernel. Nonkernel files are collectively called the userland, meaning they’re intended for users even if they use kernel facilities.

Since a kernel is just a set of files, you can have alternative kernels on hand for special situations. On systems where you’ve built your own kernel, you will find /boot/kernel.old, a directory containing the kernel that was installed before your current kernel. I habitually copy the kernel installed with the system into /boot/kernel.install. You can also create your own special kernels. The FreeBSD team makes configuring and installing kernels as simple as possible. The simplest and best-supported way to alter a kernel is through the sysctl interface.

Kernel State: sysctl

The sysctl(8) program allows you to peek at the values used by the kernel and, in some cases, to set them. Just to make things more confusing, these values are also sometimes known as sysctls. The sysctl interface is a powerful feature because, in many cases, it will let you solve performance issues without rebuilding the kernel or reconfiguring an application. Unfortunately, this power also gives you the ability to sweep the legs out from under a running program and make your users really, really unhappy.

The sysctl(8) program handles all sysctl operations. Throughout this book, I’ll point out how particular sysctls change system behavior, but first, you need to understand sysctls in general. Start by grabbing all the human-visible sysctls on your system and saving them to a file so you can study them easily.

# sysctl -o -a > sysctl.out

The file sysctl.out now contains hundreds of sysctl variables and their values, most of which will look utterly meaningless. A few of them, however, you can interpret without knowing much:

kern.hostname: storm

This particular sysctl, called kern.hostname, has the value storm. Oddly enough, the system I ran this command on has a hostname of storm, and the sysctl hints that this is the kernel’s name for the system it’s running on. See these sysctls with the -a flag. Most sysctls are meant to be read this way, but a few, called opaque sysctls, can only be interpreted by userland programs. Show opaque sysctls with the -o flag.

net.local.stream.pcblist: Format:S,xunpcb Length:5488 Dump:0x20000000000000001
1000000dec0adde...

I could guess that the variable net.local.stream.pcblist represents something for the network stack. I can’t even guess what the value means. Userland programs like netstat(1) pull information from these opaque sysctls.

sysctl MIBs

The sysctls are organized in a tree format called a management information base (MIB) with several broad categories, such as net (network), kern (kernel), and vm (virtual memory). Table 6-1 lists the roots of the sysctl MIB tree on a system running the GENERIC kernel.

Table 6-1: Roots of the sysctl MIB Tree

sysctl	Function
kern	Core kernel functions and features
vm	Virtual memory system
vfs	Filesystem
net	Networking
debug	Debugging
hw	Hardware
machdep	Machine-dependent settings
user	Userland interface information
p1003_1b	POSIX behavior
kstat	Kernel statistics
dev	Device-specific information
security	Security-specific kernel features

Each of these categories is divided further. For example, the net category, covering all networking sysctls, is divided into categories such as IP, ICMP, TCP, and UDP. The concept of a management information base is used in several other parts of system administration, as we’ll see in Chapter 21 and you’ll see throughout your career. The terms sysctl MIB and sysctl are frequently used interchangeably. Each category is named by stringing together the parent category and all of its children to create a unique variable name, such as:

--snip--
kern.maxfilesperproc: 11095
kern.maxprocperuid: 5547
kern.ipc.maxsockbuf: 262144
kern.ipc.sockbuf_waste_factor: 8
kern.ipc.max_linkhdr: 16
--snip--

Here we have five sysctls plucked from the middle of the kern category. The first two are directly beneath the kern label and have no sensible grouping with other values other than the fact that they’re kernel-related. The remaining three all begin with kern.ipc; they’re part of the IPC (interprocess communication) section of kernel sysctls. If you keep reading the sysctls you saved, you’ll see that some sysctl variables are several categories deep.

sysctl Values and Definitions

Each MIB has a value that represents a buffer, setting, or characteristic used by the kernel. Changing the value changes how the kernel operates. For example, the kernel handles transmitting and receiving packets, but by default won’t send a packet from one interface to another. You can change a sysctl to permit this forwarding, thereby turning your host into a router.

Each sysctl value is either a string, an integer, a binary value, or an opaque. Strings are free-form texts of arbitrary length; integers are ordinary whole numbers; binary values are either 0 (off) or 1 (on); and opaques are pieces of machine code that only specialized programs can interpret.

Many sysctl values are not well documented; there is no single document listing all available sysctl MIBs and their functions. A MIB’s documentation generally appears in a man page for the corresponding function, or sometimes only in the source code. For example, the original documentation for the MIB kern.securelevel (discussed in Chapter 9) is in security(7). Although sysctl documentation has expanded in recent years, many MIBs still have no documentation.

Fortunately, some MIBs have obvious meanings. For example, as we discuss later in this chapter, this is an important MIB if you frequently boot different kernels:

kern.bootfile: /boot/kernel/kernel

If you’re debugging a problem and have to reboot with several different kernels in succession, you can easily forget which kernel you’ve booted (not that this has ever happened to me, really). A reminder can therefore be helpful.

An easy way to get some idea of what a sysctl does is to use the -d switch with the full MIB. This prints a brief description of the sysctl:

# sysctl -d kern.maxfilesperproc
kern.maxfilesperproc: Maximum files allowed open per process

This brief definition tells you that this sysctl controls exactly what you might think it does. Unfortunately, not all sysctls provide definitions with -d. While this example is fairly easy, other MIBs might be much more difficult to guess.

Viewing sysctls

To view all the MIBs available in a particular subtree of the MIB tree, use the sysctl command with the name of the part of the tree you want to see. For example, to see everything under kern, enter this command:

# sysctl kern
kern.ostype: FreeBSD
kern.osrelease: 12.0-CURRENT
kern.osrevision: 199506
kern.version: FreeBSD 12.0-CURRENT #0 r322672: Fri Aug 18 16:31:34 EDT 2018
    root@storm:/usr/obj/usr/src/sys/GENERIC
--snip--

This list goes on for quite some time. If you’re just becoming familiar with sysctls, you might use this to see what’s available. To get the exact value of a specific sysctl, give the full MIB name as an argument:

# sysctl kern.securelevel
kern.securelevel: -1

The MIB kern.securelevel has the integer value -1. We’ll discuss the meaning of this sysctl and its value in Chapter 9.

Changing sysctls

Some sysctls are read-only. For example, take a look at the hardware MIBs:

hw.model: Intel(R) Xeon(R) CPU E5-1620 v2 @ 3.70GHz

The FreeBSD Project has yet to develop the technology to change Intel hardware into ARM64 hardware via a software setting, so this sysctl is read-only. If you were able to change it, all you’d do is crash your system. FreeBSD protects you by not allowing you to change this value. An attempt to change it won’t hurt anything, but you’ll get a warning. On the other hand, consider the following MIB:

vfs.usermount: 0

This MIB determines whether users can mount removable media, such as CDROM and floppy drives, as discussed in Chapter 13. Changing this MIB requires no extensive tweaks within the kernel or modifications to hardware; it’s only an in-kernel permissions setting. To change this value, use the sysctl(8) command, the sysctl MIB, an equal sign, and the desired value:

# sysctl vfs.usermount=1
vfs.usermount: 0 -> 1

The sysctl(8) program responds by showing the sysctl name, the old value, and the new value. This sysctl is now changed. A sysctl that can be tuned on the fly like this is called a runtime tunable sysctl.

Setting sysctls Automatically

Once you’ve tweaked your kernel’s settings to your whim, you’ll want those settings to remain after a reboot. Use the file /etc/sysctl.conf for this. List each sysctl you want to set and the desired value in this file. For example, to set the vfs.usermount sysctl at boot, add the following on its own line in /etc/sysctl.conf:

vfs.usermount=1

The Kernel Environment

The kernel is a program started by the boot loader. The boot loader can hand environment variables to the kernel, creating the kernel environment. The kernel environment is also a MIB tree, much like the sysctl tree. Many, but not all, of these environment variables later get mapped onto read-only sysctls.

Viewing the Kernel Environment

Use kenv(8) to view the kernel environment. Give it the name of a kernel environment variable to see just that variable, or run it without arguments to see the whole tree.

# kenv
LINES="24"
acpi.oem="SUPERM"
acpi.revision="2"
acpi.rsdp="0x000f04a0"
acpi.rsdt="0x7dff3028"
--snip--

These variables look an awful lot like the loader variables. Because they are the loader variables. They frequently relate to initial hardware probes. If your serial port uses an unusual memory address, the kernel needs to know about that before trying to probe it.

These environment settings are also called boot-time tunable sysctls, or tunables, frequently related to low-level hardware settings. As an example, when the kernel first probes a hard drive, it must decide whether it’s going to provide ident-based or GPT ID-based labels. This decision must be made before anything in the kernel accesses the hard drive, and you can’t change your mind without rebooting the machine.

Kernel environment variables can be set only from the loader. You can make changes manually at boot time or set them in /boot/loader.conf to take effect at the next boot (see Chapter 4).

Much like sysctl.conf, setting tunable values in loader.conf will let you really mess up a machine. The good news is that these values are easily unset.

Dropping Hints to Device Drivers

You can use environment variables to tell device drivers needed settings. You’ll learn about these settings by reading the driver man pages and other documentation. Additionally, much ancient hardware requires the kernel to address it at very specific IRQ and memory values. If you’re old enough to remember plug-and-pray, “hardware configuration” floppy disks, and special slots for bus master cards, you know what I’m talking about and probably have one of these systems polluting your hardware closet even today. (If you’re too young for that, buy one of us geezers a drink and listen to our horror stories.²) You can tell FreeBSD to probe for such hardware at any IRQ or memory address you specify, which is very useful when you have a card with a known configuration but the floppy that can change that configuration biodegraded years ago.

If you’re truly unfortunate, you might have a machine with a built-in floppy disk drive. Look in /boot/device.hints to find entries that configure this hardware:

hint.fdc.0.at="isa"
hint.➊fdc.➋0.➌port=➍"0x3F0"
hint.fdc.0.irq=➎"6"
hint.fdc.0.drq=➏"2"

These entries are all hints for the fdc(4) device driver ➊. The entry is used for fdc device number zero ➋. If you enable this device, a booting kernel will probe for a card at memory address (or port ➌) 0x3F0 ➍, IRQ 6 ➎, and DRQ 2 ➏. If it finds a device with these characteristics, it gets assigned the fdc(4) driver. If that device isn’t a floppy drive, you’ll have amusing crashes.³

Boot-time tunables and sysctl let you adjust how a kernel behaves, but kernel modules let you add functionality to a running kernel.

Kernel Modules

Kernel modules are parts of a kernel that can be started, or loaded, when needed and unloaded when unused. Kernel modules can be loaded when you plug in a piece of hardware and removed with that hardware. This greatly expands the system’s flexibility. Plus, a kernel with all possible functions compiled into it would be rather large. Using modules, you can have a smaller, more efficient kernel and load rarely used functionality only when it’s required.

Just as the default kernel is held in the file /boot/kernel/kernel, kernel modules are the other files under /boot/kernel. Take a look in that directory to see hundreds of kernel module files. Each kernel module name ends in .ko. Generally speaking, the file is named after the functionality contained in the module. For example, the file /boot/kernel/wlan.ko handles the wlan(4) wireless layer. FreeBSD needs this module for wireless networking.

Viewing Loaded Modules

The kldstat(8) command shows modules loaded into the kernel.

   # kldstat
   Id Refs Address           Size     Name
➊ 1   36 0xffffffff80200000 204c3e0  kernel
➋ 2    1 0xffffffff8224e000 3c14f0   zfs.ko
➌ 3    2 0xffffffff82610000 d5f8     opensolaris.ko
➍ 5    1 0xffffffff82821000 ac15     linprocfs.ko
   --snip--

This desktop has three kernel modules loaded. The first is the kernel proper ➊; then, modules to support ZFS ➋ and the OpenSolaris kernel functions needed by ZFS ➌ follow. I experiment with Linux software on this host (see Chapter 17), so finding the linprocfs(5) module ➍ loaded is not a surprise.

Each module contains one or more submodules, which you can view using kldstat -v, but the kernel itself has a few hundred submodules—so be ready for a lot of output.

Loading and Unloading Modules

Loading and unloading kernel modules is done with kldload(8) and kldunload(8). For example, suppose I’m experimenting with IPMI on a test host. This requires the ipmi(4) kernel module. While I’d normally load this automatically at boot using loader.conf, I’m in the lab. I use the kldload command and the name of the kernel module or the file containing the kernel module for that feature:

# kldload /boot/kernel/ipmi.ko

If I happen to remember the name of the module, I can just use that. The module name doesn’t need the .ko at the end of the file. I happen to recall the name of the IPMI module.

# kldload ipmi

Most often, my feeble brain relies on tab completion in my shell to remind me of the module’s full and proper name.

Once I finish experimenting, I’ll unload the module.⁴ Specify the name of the kernel module as it appears in kldstat(8).

# kldunload ipmi

Any module that’s actively in use, such as the opensolaris.ko module loaded whenever you use ZFS, will not be permitted to unload. Attempting to unload an active module gives you an error like this:

# kldunload opensolaris
kldunload: can't unload file: Device busy

Sysadmins load modules much more often than they unload them. Unloading modules is expected to work, and it works the overwhelming majority of the time, but it’s arguably the most common way to panic a system. If unloading a module triggers a panic, file a bug report as per Chapter 24.

Loading Modules at Boot

Use /boot/loader.conf to load modules at boot. The default loader.conf includes many examples of loading kernel modules, but the syntax is always the same. Take the name of the kernel module, chop off the trailing .ko, and add the string _load="YES". For example, to load the module /boot/kernel/procfs.ko automatically at boot, add this to loader.conf:

procfs_load="YES"

The hard part, of course, is knowing which module to load. The easy ones are device drivers; if you install a new network or SCSI card that your kernel doesn’t support, you can load the driver module instead of reconfiguring the kernel. In this case, you’ll need to find out which driver supports your card; the man pages and Google are your friends there. I’ll be giving specific pointers to kernel modules to solve particular problems throughout this book.

Wait a minute, though—why would FreeBSD make you load a device driver to recognize hardware if it recognizes almost everything at boot? That’s an excellent question! The answer is that you may have built your own custom kernel and removed support for hardware you’re not using. You don’t know how to build a kernel? Well, let’s fix that right now.

Build Your Own Kernel

Eventually, you’ll find that you can’t tweak your kernel as much as you like using only sysctl(8) and modules, and your only solution will be to build a customized kernel. This sounds much harder than it is; we’re not talking about writing code here—just editing a text file and running a couple of commands. If you follow the process, it’s perfectly safe. If you don’t follow the process, well, it’s like driving on the wrong side of the road. (Downtown. During rush hour.) But the recovery from a bad kernel isn’t that bad, either.

The kernel shipped in a default install is called GENERIC. GENERIC is configured to run on a wide variety of hardware, although not necessarily optimally. GENERIC boots nicely on most hardware from the last 15 years or so, and I frequently use it in production. When you customize your kernel, you can add support for specific hardware, remove support for hardware you don’t need, or enable features not included in GENERIC.

Preparations

You must have the kernel source code before you can build a kernel. If you followed my advice back in Chapter 3, you’re all set. If not, you can either go back into the installer and load the kernel sources, download the source code from a FreeBSD mirror, or jump ahead to Chapter 18 and use svnlite(1). If you don’t remember whether you installed the source code, look into your /usr/src directory. If it contains a bunch of files and directories, you have the kernel sources.

Before building a new kernel, you must know what hardware your system has. This can be difficult to determine; the brand name on a component doesn’t necessarily describe the device’s identity or abilities. Many companies use rebranded generic components—I remember one manufacturer that released four different network cards under the same model name and didn’t even put a version number on the first three. The only way to tell the difference was to keep trying different device drivers until one of them worked. This has been going on for decades—many different companies manufactured NE2000-compatible network cards. The outside of the box had a vendor’s name on it, but the circuits on the card said NE2000. Fortunately, some vendors use a standard architecture for their drivers and hardware; you can be fairly sure that an Intel network card will be recognized by the Intel device driver.

The best place to see what hardware FreeBSD found on your system is the file /var/run/dmesg.boot, discussed in Chapter 4. Each entry represents either a hardware or software feature in the kernel. As you work on a new kernel for a system, keep the dmesg.boot of that system handy.

Buses and Attachments

Every device in the computer is attached to some other device. If you read your dmesg.boot carefully, you can see these chains of attachments. Here’s an edited set of boot messages to demonstrate:

➊ acpi0: <SUPERM SMCI--MB> on motherboard
➋ acpi0: Power Button (fixed)
➌ cpu0: <ACPI CPU> on acpi0
   cpu1: <ACPI CPU> on acpi0
➍ attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
➎ pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
➏ pci0: <ACPI PCI bus> on pcib0

Our first device on this system is acpi0 ➊. You might not know what that is, but you could always read man acpi to find out. (Or, if you must, you could read the rest of this chapter.) There’s a power button ➋ on the acpi0 device. The CPUs ➌ are also attached to acpi0, as is a timekeeping device ➍. Eventually we have the first PCI bridge, pcib0 ➎, attached to the acpi0 device. The first PCI bus ➏ is in turn attached to the PCI bridge.

So, your common PCI devices connect to a hierarchy of buses that, in turn, attach to a PCI bridge to talk to the rest of the computer. You could read dmesg.boot and draw a tree of all the devices on the system; while that isn’t necessary, understanding what’s attached where makes configuring a kernel much more likely to succeed.

If you’re in doubt, use pciconf(8) to see what’s actually on your system. pciconf -lv will list every PCI device attached to the system, whether or not the current kernel found a driver for it.

Back Up Your Working Kernel

A bad kernel can render your system unbootable, so you absolutely must keep a good kernel around at all times. The kernel install process keeps your previous kernel around for backup purposes, in the directory /boot/kernel.old. This is nice for being able to fall back, but I recommend that you go further. See Chapter 4 for details on booting alternate kernels.

If you don’t keep a known good backup, here’s what can happen. If you build a new kernel, find that you made a minor mistake, and have to rebuild it again, the system-generated backup kernel is actually the first kernel you made—the one with that minor mistake. Your working kernel has been deleted. When you discover that your new custom kernel has the same problem, or an even more serious error, you’ll deeply regret the loss of that working kernel.

A common place to keep a known good kernel is /boot/kernel.good. Back up your working, reliable kernel like this:

# cp -a /boot/kernel /boot/kernel.good

If you’re using ZFS, a boot environment might make more sense than copying (see Chapter 12).

Don’t be afraid to keep a variety of kernels on hand. Disk space is cheaper than time. I know people who keep kernels in directories named by date so that they can fall back to earlier versions if necessary. Many people also keep a current copy of the GENERIC kernel in /boot/kernel.GENERIC for testing and debugging purposes. The only way to have too many kernels is to fill up your hard drive.

Configuration File Format

FreeBSD’s kernel is configured via text files. There’s no graphical utility or menu-driven system for kernel configuration; it’s still much the same as in 4.4 BSD. If you’re not comfortable with text configuration files, building a kernel is just not for you.

Each kernel configuration entry is on a single line. You’ll see a label to indicate what sort of entry this is, and then a term for the entry. Many entries also have comments set off with a hash mark, much like this entry for the FreeBSD filesystem FFS:

options         FFS                     # Berkeley Fast Filesystem

Every complete kernel configuration file is made up of five types of entries: cpu, ident, makeoptions, options, and devices. The presence or absence of these entries dictates how the kernel supports the associated feature or hardware:

cpu This label indicates what kind of processor this kernel supports. The kernel configuration file for the boring old PC hardware includes several CPU entries to cover processors such as the 486 (I486_CPU), Pentium (I586_CPU), and Pentium Pro through modern Pentium 4 CPUs (I686_CPU). The kernel configuration for amd64/EM64T hardware includes only one CPU type, HAMMER, as that architecture has only one CPU family. While a kernel configuration can include multiple CPU types, they must be of similar architectures; a kernel can run on 486 and Pentium CPUs, but you can’t have a single kernel run on both Intel-compatible and ARM processors.

ident Every kernel has a single ident line, giving a name for the kernel. That’s how the GENERIC kernel gets its name; it’s an arbitrary text string.

makeoptions This string gives instructions to the kernel-building software. The most common option is DEBUG=-g, which tells the compiler to build a debugging kernel. Debugging kernels help developers troubleshoot system problems.

options These are kernel functions that don’t require particular hardware. This includes filesystems, networking protocols, and in-kernel debuggers.

devices Also known as device drivers, these provide the kernel with instructions on how to speak to certain devices. If you want your system to support a piece of hardware, the kernel must include the device driver for that hardware. Some device entries, called pseudodevices, aren’t tied to particular hardware, but instead support whole categories of hardware—such as Ethernet, random number generators, or memory disks. You might wonder what differentiates a pseudodevice from an option. The answer is that pseudodevices appear to the system as devices in at least some ways, while options have no device-like features. For example, the loopback pseudodevice is a network interface that connects to only the local machine. While no hardware exists for it, software can connect to the loopback interface and send network traffic to other software on the same machine.

Here’s another snippet of a configuration file—the part that covers ATA controllers:

# ATA controllers
device          ahci     # AHCI-compatible SATA controllers
device          ata      # Legacy ATA/SATA controllers
device          mvs      # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA
device          siis     # SiliconImage SiI3124/SiI3132/SiI3531 SATA

Each of these devices is a different type of ATA controller. Compare these entries to a couple of our ATA entries in /var/run/dmesg.boot :

atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170
-0x177,0x376,0xc160-0xc16f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
ada0 at ata0 bus 0 scbus0 target 0 lun 0
cd0 at ata1 bus 0 scbus1 target 0 lun 0

The kernel configuration has an ATA bus, device ata. It’s a “legacy” ATA bus, whatever the word “legacy” means today. The dmesg snippet here starts with the atapci device, the controller where ATA meets PCI. We then have two ATA buses, ata0 and ata1. Disk ada0 is on ata0, while CD drive cd0 is on ata1.

Without device ata in the kernel configuration, the kernel would not recognize the ATA bus. Even if the system figured out that the system has a DVD drive, the kernel wouldn’t know the route to get information to and from it. Your kernel configuration must include all the intermediary devices for the drivers that rely on them. On the other hand, if your system doesn’t have ATA RAID drives, floppy drives, or tape drives, you can remove those device drivers from your kernel.

If this host had an AHCI, MVS, or SIIS controller, those device names would show up in dmesg instead of ata.

Configuration Files

Fortunately, you don’t normally create a kernel configuration file from scratch; instead, you build on an existing one. Start with the GENERIC kernel for your hardware architecture. It can be found in /sys/<arch>/conf —for example, the i386 kernel configuration files are in /sys/i386/conf, the amd64 kernel configuration files are in /sys/amd64/conf, and so on. This directory contains several files, of which the most important are DEFAULTS, GENERIC, GENERIC.hints, MINIMAL, and NOTES:

DEFAULTS This is a list of options and devices that are enabled by default for a given architecture. That doesn’t mean that you can compile and run DEFAULTS, but it is a starting point should you want to build a kernel by adding devices. Using GENERIC is easier, though.

GENERIC This is the configuration for the standard kernel. It contains all the settings needed to get standard hardware of that architecture up and running; this is the kernel configuration used by the installer.

GENERIC.hints This is the hints file that is later installed as /boot/device.hints. This file provides configuration information for older hardware.

MINIMAL This configuration excludes anything that can be loaded from a module.

NOTES This is an all-inclusive kernel configuration for that hardware platform. Every platform-specific feature is included in NOTES. Find platform-independent kernel features in /usr/src/sys/conf/NOTES.

Many architectures also have architecture-specific configurations, needed only for that hardware. The i386 architecture includes the PAE kernel configuration, which lets you use more than 4GB of RAM on a 32-bit system. The arm architecture includes dozens of configurations, one for each of the many different platforms FreeBSD supports.

Sometimes, you’ll find a kernel configuration that does exactly what you want. I want the smallest possible kernel. The MINIMAL kernel looks like a good place to start. Let’s build it.

Building a Kernel

A base install of FreeBSD, combined with the operating system source code, includes all the infrastructure you need to easily build a kernel. All you need to do is tell the system which kernel configuration to build through the KERNCONF variable. You can set KERNCONF in /etc/src.conf (or /etc/make.conf, if you’re really old-school).

KERNCONF=MINIMAL

If you’re experimenting with building and running different kernels, though, it’s best to set the configuration file on the command line when you build the kernel. Build the kernel with the make buildkernel command.

# cd /usr/src
# make KERNCONF=MINIMAL buildkernel

The build process first runs config(8) to find syntactical configuration errors. If config(8) detects a problem, it reports the error and stops. Some errors are blatantly obvious—for example, you might have accidentally deleted support for the Unix File System (UFS) but included support for booting off of UFS. One requires the other, and config(8) will tell you exactly what’s wrong. Other messages are strange and obscure; those that may take the longest to figure out are like this:

MINIMAL: unknown option "NET6"

NET6 is the IPv6 option, isn’t it? No, that’s I NET6. Apparently some doofus examined the config file in a text editor and accidentally deleted a letter. The error is perfectly self-explanatory—once you’re familiar with all the supported kernel options. Read these errors carefully!

Once config(8) validates the configuration, the kernel build process takes a few minutes on a modern machine. A successful build ends with a message like this.

--------------------------------------------------------------
>>> Kernel build for MINIMAL completed on Tue Sep 12 14:27:08 EDT 2017
--------------------------------------------------------------

After building the kernel, install it. Running make installkernel moves your current kernel to /boot/kernel.old and installs the new kernel in /boot/kernel. Installing a kernel is much faster than building it.

Once the install completes, reboot your server and watch the boot messages. If everything worked, you’ll get something like the following, showing exactly what kernel is running and when it was built.

Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD storm 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r323136: Sat Sep  2
21:46:53 EDT 2018     root@storm:/usr/obj/usr/src/sys/MINIMAL  amd64
--snip--

The catch is, the MINIMAL kernel doesn’t boot all hardware. It doesn’t boot most hardware. And of the hardware MINIMAL can boot, it won’t boot most FreeBSD installations on that hardware.

MINIMAL leaves everything that can be a module in a module. Disk partitioning methods, both GPT and MBR, can be modules. You must load either geom_part_gpt.ko or geom_part_mbr.ko via loader.conf to boot MINIMAL. Filesystems are modules too, so you have to load those. In short, you have to load every stupid module required by the hardware and your installation decisions. MINIMAL is a good reference of what all kernels need, and a decent place to start designing your own kernel, but insufficient for production.

Booting an Alternate Kernel

So, what to do if your new kernel doesn’t work, or if it works badly? Perhaps you forgot a device driver or accidentally cut out the INET option and can’t access the internet. Sometimes it’ll hang up so early in the boot process that the only thing you can do is reboot the host. Don’t panic! You did keep your old kernel, right? Here’s what to do.

Start by recording the error message. You’ll need to research that message to find out how your new kernel failed you.⁵ To fix the error, though, you’ll need to boot a working kernel so you can build an improved kernel.

Back in Chapter 4, we discussed the mechanics of booting an alternate kernel. We’ll go through the process of what to type here, but to see some of the in-depth details of loader management, you’ll want to go back to the earlier section. For now, we’ll focus on the reasons to boot an alternate kernel and on how to do it correctly.

Start by deciding which kernel you want to boot. Your old kernel should be in a directory under /boot; in this section, we’ll assume that you want to boot the kernel in /boot/kernel.good. Reboot and interrupt the boot to get to the boot menu. The fifth option lets you choose a different kernel. The menu displays every kernel directory listed in the kernels option in loader.conf. While it lists kernel and kernel.old by default, I’ll add kernel.good.

Once you install another new kernel, though, remember: the existing /boot/kernel gets copied to /boot/kernel.old, so your new kernel can be placed in /boot/kernel. If that kernel doesn’t boot, and your new kernel also doesn’t boot, you’ll be left without a working kernel. This kind of sucks. Be sure you keep a known good kernel on hand.

Custom Kernel Configuration

Maybe none of the provided kernel configurations are suitable for you. You need something different. FreeBSD lets you create whatever you want. It’s easiest to modify an existing configuration, however. You can either copy an existing file or use include options. We’ll start by modifying an existing file. Be sure you use the correct architecture directory, probably either /sys/amd64/conf or /sys/i386/conf.

Do not edit any of the files in the configuration directory directly. Instead, copy GENERIC to a file named after your machine or the kernel’s function and then edit the copy. For this example, I’m building a minimal kernel to support VirtualBox systems. I copy the file GENERIC to a file called VBOX and open VBOX in my preferred text editor.

Trimming a Kernel

Once upon a time, memory was far more expensive than it is today and was available only in smaller quantities. When a system has 128MB of RAM, you want every bit of that to be available for work, not holding useless device drivers. Today, when a cheap laptop somehow suffers through the day with a paltry 64GB RAM, kernel size is almost irrelevant.

For most of us, stripping unnecessary drivers and features out of a kernel to shrink it is a waste of time and energy, but I would encourage you to do it once. It will teach you how to build a kernel so that when you have to test a kernel patch or something, you won’t need to learn kernel building along with coping with the problem compelling the rebuild. It’ll also help when you start experimenting with FreeBSD on tiny hosts like a BeagleBone or Raspberry Pi.

I want to build a kernel that supports VirtualBox kernels. I boot a working FreeBSD install on VirtualBox so I can get at dmesg.boot. I’ll be going back and forth between the dmesg and the configuration, commenting out unneeded entries.

CPU Types

On most architectures, FreeBSD supports only one or two types of CPU. The amd64 platform supports only one, HAMMER. The i386 platform supports three, but two of those—the 486 and the original Pentium—are wildly obsolete outside the embedded market.

cpu             I486_CPU
cpu             I586_CPU
cpu             I686_CPU

You need to include only the CPU you have. If you’re not sure of the CPU in your hardware, check dmesg.boot. I have an ancient laptop that shows:

CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ (2200.10-MHz 686-class CPU)
  Origin = "AuthenticAMD"  Id = 0x20fb1  Stepping = 1
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,
CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
--snip--

As shown in bold, this is a 686-class CPU, which means that I can remove the I486_CPU and I586_CPU statements to make my kernel smaller.

Core Options

Following the CPU type configuration entries, we have a whole list of options for basic FreeBSD services, such as TCP/IP and filesystems. An average system won’t require all of these, but having them present provides a great deal of flexibility. You’ll also encounter options rarely used in your environment as well as those you can remove from your custom kernel configuration. We won’t discuss all possible kernel options but will cover specific examples of different option types. I’ll specifically mention those that can be trimmed from an internet server. The LINT file, man pages, and your favorite internet search engine can fill you in on the other options. If you’re in doubt about an option, keep it. Or disable it and see what breaks.

Consider the following network-related options:

options         INET                 # InterNETworking
options         INET6                # IPv6 communications protocols
options         IPSEC                # IP (v4/v6) security
options         IPSEC_SUPPORT        # Allow kldload of ipsec and tcpmd5
options         TCP_OFFLOAD          # TCP offload
options         TCP_HHOOK            # hhook(9) framework for TCP
options         SCTP                 # Stream Control Transmission Protocol

These options support networking. INET is the standard old-fashioned TCP/IP, while INET6 supports IPv6. Much Unix-like software depends on TCP/IP, so you certainly require both of these. IPSEC and IPSEC_SUPPORT let you use the IPSec VPN protocol. I certainly won’t use these on my virtual machines, so I’ll comment them out.

The TCP_OFFLOAD option lets the network stack offload TCP/IP computations to the network card. That sounds good, except the vnet(4) network interfaces on virtual machines don’t perform that function. Off with its head!

The TCP_HHOOK option gives you a convenient man page to read. Would I use this option? Maaaybe. More importantly, I don’t know what software I’m running will need it. I’ll keep it.

The SCTP transport protocol is nifty, but totally useless to the virtual machines running on my laptop. Bye-bye.

options         FFS             # Berkeley Fast Filesystem
options         SOFTUPDATES     # Enable FFS soft updates support
options         UFS_ACL         # Support for access control lists
options         UFS_DIRHASH     # Improve performance on big directories
options         UFS_GJOURNAL    # Enable gjournal-based UFS journaling

The FFS option provides the standard FreeBSD filesystem, UFS. Even a ZFS host needs UFS support. Keep it. The other options are all related to FFS. We discuss FFS and its options in more detail than you care for in Chapter 11, but for right now, just trust me and go with it.

Soft updates ensure disk integrity even when the system shuts down incorrectly. As discussed in acl(9), UFS access control lists allow you to grant very detailed permissions on files, which I won’t need on my virtual host. Whack!

UFS_DIRHASH enables directory hashing, making directories with thousands of files more efficient. Keep that. And I’m going to use soft updates journaling, not gjournaling, so UFS_GJOURNAL can go away.

options         MD_ROOT                 # MD is a potential root device

This option—and all other _ROOT options—lets the system use something other than a standard UFS or ZFS filesystem as a disk device for the root partition. The installer uses a memory device (MD) as a root partition. If you’re using a diskless system (see Chapter 23), you’ll need an NFS root partition. If you’re running FreeBSD on a standard computer system, with a hard drive and a keyboard and whatnot, your kernel doesn’t need any of these features.

options         NFSCL                   # Network Filesystem Client
options         NFSD                    # Network Filesystem Server
options         NFSLOCKD                # Network Lock Manager

These two options support the Network File System (see Chapter 13). The vital question here is, do you need NFS? If so, do you need to be a server or a client? I’ll include these.

options         MSDOSFS              # MSDOS filesystem
options         CD9660               # ISO 9660 filesystem
options         PROCFS               # Process filesystem (requires PSEUDOFS)
options         PSEUDOFS             # Pseudo-filesystem framework

These options support intermittently used filesystems, such as FAT, CDs, the process filesystem, and the pseudo-filesystem framework. We discuss many of these filesystems in Chapter 13, but they’re all available as kernel modules. Kill them.

options         COMPAT_FREEBSD32        # Compatible with i386 binaries
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         COMPAT_FREEBSD5         # Compatible with FreeBSD5
options         COMPAT_FREEBSD6         # Compatible with FreeBSD6
--snip--

These compatibility options let your system run software built for older versions of FreeBSD or software that makes assumptions about the kernel that were valid for older versions of FreeBSD but are no longer true. If you’re installing a system from scratch, you probably won’t need compatibility with FreeBSD 4, 5, or 6, but a surprising amount of software requires compatibility with 32-bit FreeBSD. Keep the COMPAT_FREEBSD32 option, or your system will break.

options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI

The SCSI_DELAY option specifies the number of milliseconds FreeBSD waits after finding your SCSI controllers before probing them, giving them a chance to spin up and identify themselves to the SCSI bus. If you have no SCSI hardware, you can remove this line.

options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores

These options enable System-V-style shared memory and interprocess communication. Many database programs use this feature.

Multiple Processors

The following entries enable symmetric multiprocessing (SMP) in i386 kernels:

options         SMP                     # Symmetric MultiProcessor Kernel
options         DEVICE_NUMA             # I/O Device Affinity
options         EARLY_AP_STARTUP

These probably don’t hurt, but if you know you’re running on a board with a single core, possibly a system that’s very old or using embedded hardware, you can remove them.

Device Drivers

After all the options, you’ll find device driver entries, which are grouped in fairly sensible ways. To shrink your kernel, you’ll want to get rid of everything that your host isn’t using—but what, exactly, is your host not using? Search for each device driver in dmesg.boot.

The first device entries are buses, such as device pci and device acpi. Keep these, unless you truly don’t have that sort of bus in your system.

Next, we reach what most people consider device drivers proper—entries for floppy drives, SCSI controllers, RAID controllers, and so on. If your goal is to reduce the size of your kernel, this is a good place to trim heavily; remove all device drivers for hardware your computer doesn’t have. You’ll also find a section of device drivers for such mundane things as keyboards, video cards, USB ports, and so on. You almost certainly don’t want to delete these.

The network card device driver section is quite long and looks much like the SCSI and IDE sections. If you’re not going to replace your network card any time soon, you can eliminate drivers for any network cards you aren’t using.

We won’t list all the device drivers here, as there’s very little to be learned from such a list other than the hardware FreeBSD supported at the time I wrote this section. Check the release notes for the version of FreeBSD you’re running to see what hardware it supports.

You’ll also find a big section of drivers for virtualization. The most commonly used virtual interfaces are based on VirtIO, but you’ll also see specific drivers for Xen, Hyper-V, and VMware. A kernel needs only the drivers for the virtualization platform it’s run on. Kernels for real hardware don’t need any of them, even if the host will have virtual machines running on it.

Pseudodevices

You’ll find a selection of pseudodevices near the bottom of the GENERIC kernel configuration. As the name suggests, these are created entirely out of software. Here are some of the more commonly used pseudodevices.

device          loop            # Network loopback

The loopback device allows the system to communicate with itself via network sockets and network protocols. We’ll discuss network connections in some detail in the next chapter. You might be surprised at just how many programs use the loopback device, so don’t remove it.

device          random                  # Entropy device
device          padlock_rng             # VIA Padlock RNG
device          rdrand_rng              # Intel Bull Mountain RNGdevice          

These devices provide pseudorandom numbers, required for cryptography operations and such mission-critical applications as games. Some of them require support in the underlying chipset. FreeBSD supports a variety of randomness sources, transparently aggregating them all into the random devices /dev/random and /dev/urandom.

device          ether           # Ethernet support

Ethernet has many device-like characteristics, and it’s simplest for FreeBSD to treat it as a device. Leave this, unless you’re looking for a learning opportunity.

device          vlan                    # 802.1Q VLAN support
device          tun                     # Packet tunnel
device          gif                     # IPv6 and IPv4 tunneling

These devices support networking features like VLANs and different sorts of tunnels.

device          md              # Memory "disks"

Memory disks allow you to store files in memory. This is useful for very fast, temporary data storage, as we’ll learn in Chapter 13. For most (but not all) internet servers, memory disks are a waste of RAM. You can also use memory disks to mount and access disk images. If you’re not using memory disks, you can remove them from your kernel.

Removable Hardware

The GENERIC kernel supports a few different sorts of removable hardware. If you have a laptop built in a year containing two consecutive nines or zeros, it might have Cardbus or even PCMCIA cards. Otherwise, you don’t need that support in your kernel. FreeBSD supports hot-pluggable PCI cards, but if you don’t have them? Throw those drivers out.

Including the Configuration File

Your kernel binary might be separated from the machine it’s built on. I recommend using the INCLUDE_CONFIG_FILE option to copy the kernel configuration into the compiled kernel. You’ll lose any comments, but at least you’ll have the options and devices in this kernel and can duplicate it if needed. The sysctl kern.conftxt contains the kernel.

Once you have your trimmed kernel, try to build it. Your first kernel configuration will invariably go wrong.

Troubleshooting Kernel Builds

If your kernel build fails, the first troubleshooting step is to look at the last lines of the output. Some of these errors are quite cryptic, but others will be self-explanatory. The important thing to remember is that errors that say, “Stop in some directory” aren’t useful; the useful error will be before these. We talked about how to solve these problems in “Asking for Help” on page 11: take the error message and toddle off to the search engine. Compile errors usually result from a configuration error.

Fortunately, FreeBSD insists upon compiling a complete kernel before installing anything. A busted build won’t damage your installed system. It will, however, give you an opportunity to test those troubleshooting skills we talked about way back in Chapter 1.

The most common sort of error is when the make buildkernel stage fails. It might look something like this:

--snip--
linking kernel.full
vesa.o: In function `vesa_unload':
/usr/src/sys/dev/fb/vesa.c:1952: undefined reference to ➊ `vesa_unload_ioctl'
vesa.o: In function `vesa_configure':
/usr/src/sys/dev/fb/vesa.c:1169: undefined reference to ➋ `vesa_load_ioctl'
*** Error code 1
--snip--

You’ll see a few pages of Error code 1 messages, but the actual error appears before them.

Some line in our kernel requires the functions vesa_unload_ioctl ➊ and vesa_load_ioctl ➋, but the device or option that provides that function isn’t in the kernel. Try an internet search for the errors. See whether there’s a man page for those functions. If all else fails, search the source code.

# cd /usr/src/sys
# grep -R vesa_unload_ioctl *
dev/fb/vesa.h:int vesa_unload_ioctl(void);
dev/fb/vesa.c:  if ((error = vesa_unload_ioctl()) == 0) {
dev/syscons/scvesactl.c:vesa_unload_ioctl(void)

Wait—wasn’t there a reference to a “syscons” driver in the GENERIC config file?

# syscons is the default console driver, resembling an SCO console
#device          sc
#options         SC_PIXEL_MODE           # add support for the raster text mode

I had commented out the sc(4) driver. Add it back in and try again.

There are more “proper” ways of figuring out what kernel devices require what devices. They all boil down to “read and comprehend the source code.” Trial, error, research, and more trial and error turn out to be quicker for most of us.

Inclusions, Exclusions, and Expanding the Kernel

Now that you can build a kernel, let’s get a little fancy and see how to use inclusions, the various no configurations, and the NOTES file.

NOTES

FreeBSD’s kernel includes all sorts of features that aren’t included in GENERIC. Many of these special features are intended for very specific systems or for weird corner cases of a special network. You can find a complete list of hardware-specific features in the file NOTES under each platform’s kernel configuration directory—for example, /sys/amd64/conf/NOTES. Hardware-independent kernel features—those that work on every platform FreeBSD supports—can be found in /sys/conf/NOTES. If you have hardware that doesn’t appear to be completely supported in the GENERIC kernel, take a look at NOTES. Some of these features are obscure, but if you have the hardware, you’ll appreciate them. Let’s take a look at a typical entry from NOTES:

# Direct Rendering modules for 3D acceleration.
device          drm             # DRM core module required by DRM drivers
device          mach64drm       # ATI Rage Pro, Rage Mobility P/M, Rage XL
device          mgadrm          # AGP Matrox G200, G400, G450, G550
device          r128drm         # ATI Rage 128
device          savagedrm       # S3 Savage3D, Savage4
device          sisdrm          # SiS 300/305, 540, 630
device          tdfxdrm         # 3dfx Voodoo 3/4/5 and Banshee
device          viadrm          # VIA
options         DRM_DEBUG       # Include debug printfs (slow)

Are you using any of these video cards on your desktop? Maybe you want a custom kernel that includes the appropriate device driver.

If the NOTES file lists all the features for every possible device, why not just use it as the basis for your kernel? First, such a kernel would use up far more memory than the GENERIC kernel. While even small modern machines have enough memory to run GENERIC without trouble, if the kernel becomes ten times larger without the corresponding increase in functionality, people would get annoyed. Also, many options are mutually exclusive. You’ll find options that let you dictate how the kernel schedules processes, for example. The kernel can use only one scheduler at a time, and each scheduler runs its tendrils throughout the kernel. Adding all of them to the kernel simultaneously would increase code complexity and decrease stability.

I make it a point to review NOTES every release or two, just to look for interesting new features.

Inclusions and Exclusions

FreeBSD’s kernel configuration has two interesting abilities that can make maintaining a kernel easier: the no options and the include feature.

The include feature lets you pull a separate file into the kernel configuration. For example, if you have a kernel configuration that can be described as “GENERIC with a couple extra tidbits,” you could include the GENERIC kernel configuration with an include statement:

include GENERIC

So, if you want to build a kernel that has all the functionality of GENERIC but also supports the DRM features of the VIA 3d chips, you could create a valid kernel configuration composed entirely of the following:

ident        VIADRM
include      GENERIC
options      drm
options      viadrm

You might think that this is actually more work than copying GENERIC to a new file and editing it, and you’d be correct. Why would you bother with this, then? The biggest reason is that as you upgrade FreeBSD, the GENERIC configuration can change. The GENERIC in FreeBSD 12.1 is slightly different from that in 12.0. Your new configuration is valid for both releases and in both cases can be legitimately described as “GENERIC plus my options.”

This works well for including items but isn’t very good for removing things from the kernel. Rather than manually recreating your kernel for every new FreeBSD version, you can use an include statement but exclude unneeded entries with the nodevice and nooptions keywords. Remove unwanted device drivers with nodevice, while nooptions disables unwanted options.

Take a look at the GENERIC-NODEBUG kernel configuration on a -current machine. It’s the same as the GENERIC configuration, but it has all of the debugging features disabled.

include GENERIC

ident   GENERIC-NODEBUG

nooptions       INVARIANTS
nooptions       INVARIANT_SUPPORT
nooptions       WITNESS
nooptions       WITNESS_SKIPSPIN
nooptions       BUF_TRACKING
nooptions       DEADLKRES
nooptions       FULL_BUF_TRACKING

We start by including the GENERIC kernel configuration. This kernel identifies itself as GENERIC-NODEBUG, though. The following seven nooptions statements turn off FreeBSD-current’s standard debugging options. Developers use the GENERIC-NODEBUG kernel to see whether the kernel debugger is causing problems. If a kernel with debugging panics while a kernel without debugging does not panic, the debugging code suddenly looks suspiciously dubious.

Skipping Modules

If you’ve gone to the trouble of building a custom kernel, you probably know exactly which kernel modules your host needs. Why build all these dozens of kernel modules if you’re never going to use them? You can turn off the building of modules with the MODULES_OVERRIDE option. Set MODULES_OVERRIDE to the list of modules you want to build and install.

# make MODULES_OVERRIDE='' kernel

Perhaps you want to build most of the modules, but you have reason to loathe a specific module. Exclude it from the build with WITHOUT_MODULES. Here, I exclude vmm from the build, because I don’t want even the temptation of running bhyve(8) on VirtualBox. It’s only a small step from there to running a dozen layers of virtualization and wondering why my laptop is slow.

# make KERNCONF=VBOX WITHOUT_MODULES=vmm kernel

Selectively building modules, combined with custom kernels, lets you lock yourself into really itty-bitty boxes. You’ll only understand how itty-bitty those boxes are when you find you’re missing a feature you never thought you’d need. If you must build a kernel, be generous in what you keep.

Now that your local machine is tuned precisely the way you want it, let’s consider the rest of the internet.