18
UPGRADING FREEBSD

Upgrading servers is perhaps the most annoying task in the system administrator’s routine. I can manage unexplained behavior on my desktop after an upgrade, but when my whole organization or hundreds of customers depend on one server, even thinking of touching that system makes my bowels churn. Any operating system upgrade can expand your burgeoning gray hair collection. Even very experienced sysadmins, faced with a choice between upgrading a critical system in-place and jabbing red-hot needles into their own eyes, frequently have to sit down and consider their choices. Virtualized and orchestrated cloud systems might seem less troublesome, but even with these, preparing for an upgrade can cause sleepless nights. Remember, despite all its benefits, automation is a wonderful way to go wrong at scale.

One of FreeBSD’s greatest strengths is its upgrade procedure. FreeBSD is designed as a monolithic operating system, not a collection of packages. (Even if FreeBSD migrates to providing the base system as packages, it will remain designed and built as a monolithic entity.) I’ve had hosts running through five different major releases of FreeBSD and innumerable patch levels in between without reinstalling the system. I decommission FreeBSD systems only when they are so old that the risk of hardware failure keeps me awake at night.¹ While I might worry about applications running on top of the operating system, even upgrading across major FreeBSD releases doesn’t worry me anymore.

FreeBSD Versions

Why is upgrading FreeBSD a relatively simple matter? The key is FreeBSD’s development method. FreeBSD is a continually evolving operating system. If you download the current version of FreeBSD in the afternoon, it’ll be slightly different from the morning version. Developers from around the world continually add changes and improvements, which makes the traditional strict release numbering used by less open software impractical. At any given moment, you can get several different versions of FreeBSD: releases, -stable, -current, and snapshots.

Releases

FreeBSD issues major and minor releases. A major release has a version number like 11.0, 12.0, 13.0, or so on. Each major release includes large features not found in earlier major releases. Sweeping changes appear only in major releases.

A minor release is an updated version of a major release. You’ll see minor releases like 12.1-RELEASE, 12.2-RELEASE, 12.3-RELEASE, and so on. (Most people drop the word release from these names.) These minor releases add small features and bug fixes to the major release. You might get new functions or programs, but only if they don’t interfere with the existing functions. Unexpected, disruptive changes are avoided.

You’ll also see patch levels. Thanks to freebsd-update(8), patching a release is quick and easy. Patch numbers are given as numbers after the release. This means you’ll see FreeBSD versions like 12.1-RELEASE-p20, 11.4-RELEASE-p9, 13.0-RELEASE-p31, and so on.

Users are expected to closely track their major release by upgrading through successive minor releases, much like other operating systems.

FreeBSD-current

FreeBSD-current, also called -current or HEAD, is the bleeding-edge, latest version of FreeBSD, which contains code that’s making its first public appearance. While the developers have test servers and post patches for review before applying, that’s still much less exposure than the wide userbase of FreeBSD-current. FreeBSD-current is where much initial peer review takes place; at times, current undergoes radical changes that give experienced sysadmins migraines.

FreeBSD-current is made available for developers, testers, and interested parties, but it’s not intended for general use. Support for user questions about -current is very slim because the developers simply don’t have time to help a user fix his web browser while thousands more critical problems demand attention. Users are expected to help fix these problems or to patiently endure them until someone else fixes them.

To make matters worse, -current’s default settings include assorted debugging code, special warnings, and related developer features. These make -current run slower than any other version of FreeBSD. You can disable all this debugging, but if you do so, you won’t be able to file a proper trouble report when you have a problem. This means that you’re even more out on your own. Check out the file /usr/src/UPDATING on a -current system for debugging details.

If you can’t read C and shell code, don’t feel like debugging your OS, don’t like computer functions failing arbitrarily, or just don’t like being left hanging until your problem annoys someone who can fix it, -current isn’t for you. The brave are certainly welcome to try -current, as is anyone willing to devote a large amount of time to learning and debugging FreeBSD or anyone who needs a lesson in humility. You’re not forbidden to use -current; you’re just on your own. FreeBSD-current isn’t always the bleeding edge, but sometimes it might be the why-are-my-fingers-suddenly-little-wiggling-stumps? edge. You’ve been warned.

To run -current, you really must read the [email protected] and [email protected] mailing lists. These are high-traffic lists with hundreds of warnings, alerts, and comments a day. If you’re reading this book, you probably shouldn’t post on these lists; just read and learn. If someone discovers that the newest filesystem patches transform hard drives into zombie minions of Cthulhu, this is where the information will be made available.

-current Code Freezes

Every 12 to 18 months, FreeBSD-current goes through a month of code freeze, during which no noncritical changes are permitted and all known critical problems are being fixed. The goal is to stabilize FreeBSD’s latest and greatest and to polish off the rough corners. At the end of the code freeze (or shortly after), -current becomes the .0 version of a new FreeBSD major release. For example, FreeBSD 12.0 was -current at one point, as was FreeBSD 13.0. When a new major release happens, the -current version number gets incremented. The release of FreeBSD 17.0 means that -current will be called FreeBSD 18.

Once the .0 major release escapes into the wild, development work branches into two lines: FreeBSD-current and FreeBSD-stable.

FreeBSD-stable

FreeBSD-stable (or just -stable) is the “bleeding edge for the average user,” containing some of the most recent peer-reviewed code. Sysadmins familiar with Linux know -stable as a “rolling release.” You’ll find a version of FreeBSD-stable for each major release.

Once a piece of code is thoroughly tested in -current, it might be merged back into -stable. The -stable version is the one that’s mostly safe to upgrade to at almost any time; you might think of it as FreeBSD-beta.

Three or four times a year, the Release Engineering team asks the developers to focus on resolving outstanding problems with -stable rather than making major changes. The Release Engineering team cuts several release candidates from this code and offers each for public testing. When the FreeBSD team is satisfied with the results of its own and the community’s testing, the result is given a release number. The development team then returns their attention to their regular projects.²

How does this work in practice? Consider FreeBSD 13. Once 13.0 comes out, developers will start merging bug fixes and additions to the 13-stable version. Users who want to help test the new FreeBSD release can run 13-stable. After a few months of merging features and some testing, 13-stable will become 13.1. After 13.1 comes out, that development track reverts to 13-stable. FreeBSD 13.1, 13.2, and 13.3 are just points on the continuum of FreeBSD 13-stable.

FreeBSD-stable is expected to be calm and reliable, requiring little user attention.

As -stable ages, the differences between -stable and -current become greater and greater, to the point where it becomes necessary to branch a new -stable off of -current. The older -stable is actively maintained for several months while the new -stable is beaten into shape. Some users upgrade to the new version of -stable immediately, others are more cautious. After a release or two of the new -stable, the older -stable is obsoleted and the developers encourage users to migrate to the new version. After some time, the older stable will receive only critical bug fixes, and finally, it’ll be abandoned entirely. You can see how this works in Figure 18-1.

Figure 18-1: FreeBSD development branches

Each release really should have a little dangling tail off to the side for patch levels, but that makes the diagram really confusing.

Users of FreeBSD-stable must read the [email protected] mailing list. While this mailing list has a moderate level of traffic and a fair amount of question-and-answer exchanges that really should be on -questions@, important messages from developers generally have a subject beginning with HEADS UP. Look for those messages; they generally mean that a change in the system can ruin your day if you don’t know about it.

Merging from -current

The phrase merged from -current (MFC) means that a function or subsystem has been backported from FreeBSD-current into FreeBSD-stable (or, rarely, into an errata branch). Not all features are MFC’d, however. The point of FreeBSD-current is that it’s where major changes take place, and many of those changes require months of testing and debugging. Those large changes can’t be backported, as they’d badly impact the -stable users who expect a stable codebase.

Sometimes, features that seem “obvious MFC candidates” can’t be merged. Sometimes the kernel infrastructure changes to support new drivers and features, and that infrastructure can’t be safely merged. New drivers that require such infrastructure can’t be MFC’d. This happens most often with video and wireless network drivers.

Select new drivers, bug fixes, and minor enhancements can be MFC’d—but that’s about it. The FreeBSD Project makes it a point not to MFC large changes that could break user applications.

Snapshots

Every month or so, the FreeBSD Release Engineering team releases snapshots of -current and -stable and makes them available on an FTP site. Snapshots are just points along the development branch; they undergo no special packaging or testing. Snapshots don’t receive the same attention to quality that releases do, but they’re intended as a good starting point for people interested in running -current or -stable. There’s only modest quality control, and many developers have no idea that a snapshot has come out until it appears on the FTP servers. You’ll find bugs. You’ll find errors. You’ll experience issues that will turn your mother’s hair white, assuming you haven’t done that to the poor woman already.

FreeBSD Support Model

With FreeBSD 11.0, the Project’s support model changed to more closely resemble that used by other commercial and noncommercial operating systems.

Each major release is supported with security and stability patches for five years after the first release. If FreeBSD 13 is released on January 1, 2021, support will end on January 1, 2026.

Each minor release is supported for three months after the release of the next minor release. Support for FreeBSD 12.3 will end three months after the release of FreeBSD 12.4. This gives you three months to schedule an upgrade.

A loss of official support doesn’t mean that you can’t continue to run unsupported versions. However, you’ll need to examine each security announcement, determine whether it affects your environment, and create your own patches. You’re better off running the upgrade.

The whole point of FreeBSD minor releases is that they’re minimally intrusive. Upgrading from FreeBSD 12.3 to 12.4 should have a similar impact to applying Windows updates or going from Centos 8.1 to 8.2. Applications should continue to run just fine.

The last FreeBSD minor release of a version gets supported and patched out to the five-year mark. If FreeBSD 12.5 is the last release of FreeBSD 12, it’ll get security patches out until five years after the release of FreeBSD 12.0.

Testing FreeBSD

Each version and release of FreeBSD is tested in a variety of ways. Individual developers check their work on their own hardware and ask each other to double-check their work. If the work is sufficiently complicated, they might use the official FreeBSD Phabricator system (https://reviews.FreeBSD.org/) or even a private source code repository to offer their work to a broader community before committing it to -current. Analysis companies have donated analysis software to the FreeBSD team so that the source code can be automatically audited, tested, and debugged on an ongoing basis, catching many errors before they have a chance to affect real-world users. Corporations such as Sentex, EMC, Netflix, and iX Systems provide testing for the Project. Several highly regarded FreeBSD developers have made testing a major issue within the Project. They now have an automated Kyua testing framework.

Ultimately, however, a volunteer project with a few hundred developers can’t purchase all computer hardware ever made, nor can they run that hardware under all possible loads. The FreeBSD Project as a whole relies on companies and people that use FreeBSD to test the software.

The most useful testing comes from users who have real-world equipment and real-world testbeds with real-world workloads. Sadly, most of these users perform testing when they put a release CD into the computer, run an install, and fire up the system. At that point, it’s too late to benefit the release. Any bugs you find might help the next release, but in the meantime, a patch might fix your problem.

The solution here is obvious—test FreeBSD on your real-world workloads before the release is cut. Requests for testing of new -stable releases appear on [email protected]. By testing a -stable or -current, you’ll get even better value from FreeBSD.

Which Version Should You Use?

-current, -stable, releases, snapshots—the head spins. Yes, this seems complicated, but it ensures specific quality levels. Users can rest assured that a release is as reliable as possible and has survived extensive testing and peer review. The same users know that the nifty new features in -stable and -current are available—if they’re willing to assume the risk inherent in each version. So, which version should you use? Here are my suggestions:

Production

If you’re using FreeBSD in a production setting, install the most recent minor release. When the next minor release comes out, upgrade to it.

Staging

If you need to know what’s coming in the next FreeBSD minor release and how it’ll affect your production environment, track -stable in your staging environment.

Test

The question here is what you want to test. The Project appreciates testing on both -current and -stable. If you’re in doubt, start by running -stable.

Development

Operating system developers, people with too much spare time and too little excitement, and utter fools should run -current. When -current destroys your MP3 collection, debug the problem and submit a patch to fix it.

Hobby

If you’re a hobbyist, run any version! Just keep in mind the limitations of the branch you choose. If you’re just learning Unix, I’d recommend -release. Once you have your feet under you, upgrade to -stable. While -current is far more steady than it was 20 years ago, if you think a chance of an adrenaline-boosting system failure makes your day more exciting, that’s where to go. The high-stakes gamblers running -current welcome like-minded company.

Upgrade Methods

FreeBSD provides two ways to upgrade: binary updates and building from source.

FreeBSD supports binary updates through freebsd-update(8). This is very similar to the binary update services offered for Windows, Firefox, and other commercial software. (The software author states that freebsd-update(8) was named after Windows Update.) You can use FreeBSD Update to upgrade across major releases, minor releases, and patch levels.

Upgrading from source code means downloading the FreeBSD source code, building the programs that make up FreeBSD, and installing them to your hard drive. For example, if you have the source code for FreeBSD 13.1, you can upgrade to that version. This requires more effort to set up and use, but it gives you much more flexibility. Upgrade from source when tracking -stable or -current.

Binary Updates

Many operating systems offer binary updates, where users can download new binaries for their operating system. FreeBSD provides a similar program through freebsd-update(8), allowing you to easily upgrade your system.³ You can’t track -stable or -current with freebsd-update(8), only releases. For example, if you installed FreeBSD 12.0, freebsd-update(8) can upgrade you to 12.0-p9, 12.1, or 13.0, but not 12-stable or 14-current.

If you have a custom kernel, you must build updates to your kernel from source instead of relying upon the update service. Similarly, if you’ve upgraded a host from source (discussed later this chapter), running freebsd-update(8) overwrites your custom binaries with default ones.

Configure updates in /etc/freebsd-update.conf.

/etc/freebsd-update.conf

Updating with freebsd-update(8) is designed to be seamless for the average user, and changing its configuration is rarely advisable. You might have unusual circumstances, however, so here are the most useful options you’ll find in this file:

KeyPrint 800...

KeyPrint lists a cryptographic signature for the update service. If the FreeBSD Update service suffered a security breach, the FreeBSD Project would need to repair the breach and issue new cryptographic keys. In this case, the breach would be announced on the security announcements mailing list (and would also be big news in the IT world). In other words, there’s no reason to change this in normal use. (Building your own customized FreeBSD and distributing it via freebsd-update(8), while both possible and practical, is considered abnormal use.)

ServerName update.freebsd.org

The ServerName tells freebsd-update(8) where to fetch its updates from. While the FreeBSD Project does provide the tools to build your own updates, there’s really no need to do so. If you have enough servers that you’d consider building your own update server, you probably also have a proxy server that can cache the official updates.

Components src world kernel

By default, FreeBSD Update provides the latest patches for the source code in /usr/src, the userland (world), and the GENERIC kernel. You might not need all of these components, however. While the userland is mandatory, you might have a custom kernel. Remove the kernel statement to make freebsd-update(8) ignore the kernel. Custom kernel users could also copy the GENERIC kernel to /boot/GENERIC. The update will update the GENERIC kernel but leave your custom kernel alone. Or, you can remove the kernel entry and save yourself the warning. If you don’t have the source code installed on your machine, freebsd-update realizes that and doesn’t try to patch it, but you could eliminate the src entry and save the software the trouble. You could also choose to receive only portions of the source code update, as described in freebsd-update.conf(5).

UpdateIfUnmodified /etc/ /var/ /root/ /.cshrc /.profile

The updates include changes to configuration files in /etc. If you have modified these files, however, you probably don’t want freebsd-update(8) to overwrite them. Similarly, /var is very fluid, designed for customization by the sysadmin; you don’t want FreeBSD Update to muck with your settings. FreeBSD Update applies patches to files in the directories listed in UpdateIfUnmodified only if they’re unchanged from the default.

MergeChanges /etc/ /boot/device.hints

Updating to a new release can change configuration files. The update process will give you a chance to merge changes into files that appear in the MergeChanges locations.

MailTo root

If you schedule a run of FreeBSD Update (as described later in this chapter), freebsd-update(8) sends an email of the results to the account listed in MailTo.

KeepModifiedMetadata yes

Perhaps you’ve modified the permissions or owner of a system file or command. You probably don’t want freebsd-update(8) to change those permissions back. With KeepModifiedMetadata set to yes, freebsd-update(8) leaves your custom permissions and ownership unchanged.

See freebsd-update.conf(5) for more possibilities.

Running freebsd-update(8)

Updating your system with binary updates has two stages: downloading the updates and applying them. The process looks slightly different if you’re applying patches versus if you’re crossing major releases.

If you’re using ZFS, always create a new boot environment before upgrading or patching!

Updating to the Latest Patch Level

To download the latest patches to your chosen release, run freebsd-update fetch. Here, I’m updating a FreeBSD 11.0 host to the latest patchlevel.

# freebsd-update fetch

You’ll see the program finding the download sources for the patches, comparing cryptographic keys for those download sources, and eventually downloading patches into /var/db/freebsd-update. Inspecting the system might take a couple minutes, depending on the speed of your storage.

Occasionally, you’ll see a message similar to this:

   The following files will be removed as part of updating to 11.0-RELEASE-p12:
➊ /boot/kernel/hv_ata_pci_disengage.ko
➋ /usr/share/zoneinfo/America/Santa_Isabel
➌ /usr/share/zoneinfo/Asia/Rangoon

We’re updating a .0 release, the first version of a major FreeBSD release, straight to 11.0-RELEASE-p12, so there’s a few accumulated patches. Why would such a patchset start by removing files?

The time zone files are pretty straightforward. Between the release of FreeBSD 11.0 and the present time, Santa Isabel ➋ and Rangoon ➌ changed their time zones. Sadly, nations don’t coordinate their time zones with FreeBSD releases. Removing those time zones from the system simplifies life for sysadmins in those countries and doesn’t hurt the rest of us.

But they’re also removing a kernel module ➊. Why would that happen? A little research on the FreeBSD mailing lists shows that this module should never have been shipped with 11.0, and you really shouldn’t be using it. This sort of change is rare but can happen right after a major release.

You’ll then see files added as part of this patchset, if any.

The following files will be added as part of updating to 11.0-RELEASE-p12:
/usr/share/zoneinfo/Asia/Barnaul
/usr/share/zoneinfo/Asia/Famagusta
/usr/share/zoneinfo/Asia/Tomsk
--snip--

It seems sysadmins in Rangoon are quite busy this summer.

Almost all patches alter existing files on the system. You’ll see those next.

The following files will be updated as part of updating to 11.0-RELEASE-p12:
/bin/freebsd-version
/boot/gptboot
/boot/gptzfsboot
/boot/kernel/cam.ko
/boot/kernel/hv_storvsc.ko
--snip--

If your release is nearing its End of Life, you’ll get a notice like this:

WARNING: FreeBSD 11.0-RELEASE-p1 is approaching its End-of-Life date.
It is strongly recommended that you upgrade to a newer
release within the next 1 month.

If the release has gone past End of Life, the notice gets more . . . emphatic.

To install the downloaded files, run freebsd-update install:

# freebsd-update install
Installing updates... done.

If the update requires any more steps, you’ll see them here. Reboot your system, and you’ll see that you’re running the newest patchlevel.

Updating Releases

This FreeBSD 11.0-p12 machine is within a month of End of Life? Let’s update it with freebsd-update upgrade. Specify the target release with the -r flag.

# freebsd-update -r 11.1-RELEASE upgrade

The hardest part of this is to remember that -RELEASE is part of the version name.

The upgrade will inspect your system and present its conclusions.

The following components of FreeBSD seem to be installed:
kernel/generic world/base world/lib32

The following components of FreeBSD do not seem to be installed:
kernel/generic-dbg world/base-dbg world/doc world/lib32-dbg

Does this look reasonable (y/n)? y

Remember the install process, when you selected FreeBSD components to set up on your new host? That’s what freebsd-update is checking for. You could have added or removed components, though, so take a look at the list. If it looks right, hit y to continue.

The update will more carefully scrutinize the system, comparing all existing files to the new release, and then start downloading.

Fetching 10697 patches.....10....20....30....40....50....60....70....80....90

Go make a cup of tea. Depending on your host’s bandwidth, you should return to see:

Applying patches...

You tea’s probably too hot. Let it cool a bit.

Fetching 236 files...

More downloading? Fine, enjoy your tea and watch the program work.

Attempting to automatically merge changes in files... done.

The following files will be removed as part of updating to 11.1-RELEASE-p1:
/usr/include/c++/v1/__undef___deallocate
/usr/include/c++/v1/tr1/__undef___deallocate
/usr/include/netinet/ip_ipsec.h
--snip--

You can search the mailing list archives and the FreeBSD source code tree to learn why each of these files was removed.

The minor release will include new features backported from -current. Those probably involve adding files to the system.

The following files will be added as part of updating to 11.1-RELEASE-p1:
/boot/kernel/amd_ecc_inject.ko
/boot/kernel/bytgpio.ko
/boot/kernel/cfiscsi.ko
/boot/kernel/cfumass.ko
--snip--

None of these new features should interfere with existing functions, but perusing the list might give you some interesting reading.

An upgrade should change just about every file on the system, as we’ll see next.

The following files will be updated as part of updating to 11.1-RELEASE-p1:
/.cshrc
/.profile
/COPYRIGHT
/bin/[
/bin/cat
--snip--

Eventually you’ll get to:

To install the downloaded upgrades, run "/usr/sbin/freebsd-update install".

Who are you to ignore instructions?

Up until this point, the update process has only downloaded files and done comparisons in temporary staging areas. The operating system hasn’t been touched. If you feel comfortable proceeding, run the installation.

# freebsd-update install
src component not installed, skipped
Installing updates...
Kernel updates have been installed.  Please reboot and run
"/usr/sbin/freebsd-update install" again to finish installing updates.

Why reboot between parts of the update? New userland programs might require new kernel features. Installing a nonfunctional version of the reboot command results in needing to power cycle the host, which would earn you an embarrassing punch on your geek card.

# reboot

Once the machine comes back up, complete the userland upgrade.

# freebsd-update install
src component not installed, skipped
Installing updates...
Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run "/usr/sbin/freebsd-update
install" again to finish installing updates.

What madness is this?

The update process works hard not to leave you with a damaged system or dysfunctional software. If freebsd-update removes older versions of shared libraries required by your add-on software, it won’t run. The update pauses so you have a chance to upgrade your software. We discuss upgrading packages and ports later this chapter. Upgrades along a -stable branch don’t normally need to remove old cruft.

This last run of freebsd-update removes old shared libraries and such.

# freebsd-update install

Your upgrade is now complete. As with any time you perform wide-ranging system maintenance, reboot one last time to verify everything starts cleanly.

Reverting Updates

You thought the upgrade would go easily. They always have before. But this time, you were wrong. Some subtle interaction between the new patches and your software has done brung you low. If you’re using boot environments, this is the time to revert to your previous install. If you’re not, remove the most recently installed upgrade with freebsd-update’s rollback command.

# freebsd-update rollback
Uninstalling updates... done.

A rollback is much faster than installing patches. There’s no need to inspect the system; freebsd-update just reads the log of its previous actions and undoes everything.

Scheduling Binary Updates

Best practice would say to download and apply updates at a consistent time on a regular schedule, such as on your monthly maintenance day. The freebsd-update program includes specific support for this to avoid flooding the download servers with requests every hour, on the hour. The freebsd-update cron command tells the system to download the updates at a random point in the next hour. Put this command in /etc/crontab to download updates during that one-hour window. This helps reduce the load on the download servers. You’ll get an email when the system has updates, so you can schedule a reboot at your convenience.

Optimizing and Customizing FreeBSD Update

Two common questions about FreeBSD Update concern the custom builds of FreeBSD and distributing updates locally.

Many people build their own versions of FreeBSD for internal use. Frequently, this is just a version of FreeBSD with various sections cut out, but some companies use extensive modifications. If you have deleted files from your FreeBSD install, freebsd-update(8) won’t attempt to patch them.

Similarly, many companies like to have internal update servers for patch management. The FreeBSD Update system is specifically designed to work with caching web proxies. While all the files are cryptographically signed and verified, they’re transmitted over vanilla HTTP so that your proxy can cache them.

Upgrading via Source

Another way to update your system is to build it from source code. FreeBSD is self-hosting, meaning that it includes all the tools needed to build FreeBSD. You don’t need to install any compilers or development toolkits. The only thing you need to build a new FreeBSD from source code is the newer source code.

When a developer releases improvements to FreeBSD, the changes are made available worldwide within minutes. The FreeBSD master source code server tracks the source code, all changes made to that code, and the author of those changes. Developers can check in new code, and users can check out the latest versions through Subversion (SVN). FreeBSD has a simple SVN client, svnlite(1), that suffices for all source code operations. It’s a standard Subversion client built without any of the complicated options svn(1) supports. Many people find svnlite(1) perfectly adequate for all their non-FreeBSD Subversion needs.

Upgrading from source requires console access. You won’t always need it, but recovering from a bad build might require intervention at the keyboard. Test your serial console, Java app, or IPMI console before installing your home-built operating system version. In my experience, the only upgrades that require console access are those where I don’t have console access.

Which Source Code?

Every FreeBSD release ships with the source code used to build the system you’re installing. If you didn’t choose to install the source when installing the system, you’ll find it on the install media in /usr/freebsd-dist/src.txz. If you did install the source code, you’ll find it in /usr/src.

Unfortunately, this version of the source code lacks the version control tags needed to build FreeBSD. It’s for reference only. You’ll need to use svnlite(1) to download a version of the code with those tags intact.

Is your copy of source code in /usr/src suitable for building FreeBSD? Ask svnlite(1).

# svnlite info /usr/src
svn: E155007: '/usr/src' is not a working copy

The “not a working copy” error means that any source code here can’t be used with Subversion.

The svnlite(1) command to grab source code looks like so.

# svnlite checkout ➊svn.freebsd.org/➋repository/➌branch ➍localdir

The mirror ➊ is a FreeBSD Subversion mirror. The mirror svn.FreeBSD.org is a geo-routed alias for the closest subversion mirror.

The repository ➋ is the group of code you’re working with. You can get a complete list of current repositories at https://svnweb.FreeBSD.org/. The main Project repositories include base, for the operating system; doc, for documentation; and ports, for the Ports Collection.

The branch ➌ is the version of FreeBSD you want. For the very latest stumpy-fingered -current, use head. To get a stable version, use the branch stable/ followed by the major release. FreeBSD 12-stable would be stable/12. To get a release plus all current patches, use releng/ and the release number. The fully patched FreeBSD 12.2 would be at releng/12.2.

If you have trouble figuring out which branch you need, wander through https://svnweb.freebsd.org/. The branch literally tells svnlite(1) which subdirectory to grab from this site.

Finally, the localdir ➍ is the local directory where I want to put the source code.

This host is running FreeBSD 11.1. I want to be adventuresome and move up to FreeBSD 11-stable. Here’s how I’d do that:

# svnlite checkout https://svn0.us-east.FreeBSD.org/base/stable/11 /usr/src/
Error validating server certificate for 'https://svn0.us-east.freebsd.org:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: svnmir.ysv.FreeBSD.org
 - Valid: from Jul 29 22:01:21 2013 GMT until Dec 13 22:01:21 2040 GMT
 - Issuer: svnmir.ysv.FreeBSD.org, clusteradm, FreeBSD.org, CA, US([email protected])
 - Fingerprint: 1C:BD:85:95:11:9F:EB:75:A5:4B:C8:A3:FE:08:E4:02:73:06:1E:61
(R)eject, accept (t)emporarily or accept (p)ermanently? p

What fresh madness is this? We try to get the FreeBSD source code and instead get a certificate error?

Compare the certificate fingerprint shown to the server’s fingerprint on the FreeBSD.org website. If it matches, permanently accept the certificate by entering p. Filenames of source code files will stream past your terminal.

Once svnlite(1) finishes, take a look in /usr/src.

# ls /usr/src/
COPYRIGHT               cddl                    sbin
LOCKS                   contrib                 secure
MAINTAINERS             crypto                  share
Makefile                etc                     sys
Makefile.inc1           gnu                     targets
Makefile.libcompat      include                 tests
ObsoleteFiles.inc       kerberos5               tools
README                  lib                     usr.bin
README.md               libexec                 usr.sbin
UPDATING                release
bin                     rescue

This is the top directory of the FreeBSD source tree, which contains all the code needed to build your chosen FreeBSD version.

Updating Source Code

So you built FreeBSD yesterday? Cool. But now you want to build today’s version.

The good news is, Subversion needs only to update the code you’ve downloaded, not download the whole source code tree. The better news is, the source code records where you got it from and what branch it’s supposed to be from in the .svn directory. This makes updating much simpler than the initial download.

FreeBSD has integrated the Subversion commands into the source code. All you’ll need to do is tell the system that it may call svnlite(1) for updates by setting SVN_UPDATE in /etc/src.conf.

# echo 'SVN_UPDATE=yes' >> /etc/src.conf

Now run make update to get the latest source code.

# cd /usr/src
# make update

You’ll see the same sorts of updates flow past. These updates will be much quicker than the original download, though.

Building FreeBSD from Source

Once you have the latest source code, look at /usr/src/UPDATING. The beginning of this file lists, in reverse chronological order, any warnings and special notices about changes to FreeBSD that are of special interest to people who build from source. These notes tell you whether you must take any particular actions before rebuilding your system or whether any major system functionality has changed. If you want your system to work after the upgrade, follow these instructions exactly.

The end of the UPDATING file gives the official instructions for building FreeBSD from source. The procedure described in this book has been used since FreeBSD 6-current, which changed only slightly from 5-current, but I still recommend double-checking the instructions herein against those in UPDATING.

If you use a custom kernel, also examine the new GENERIC or NOTES kernel configuration files for any new options or interesting kernel changes.

If you hang around the FreeBSD community for a while, you’ll hear all sorts of stories about special methods people use for building FreeBSD. You’ll hear anecdotal evidence that one method is faster, more efficient, or somehow mystically “better” than the standard. While you are certainly free to use any build method that strikes your fancy, the only method supported by the FreeBSD Project is that documented at the end of /usr/src/UPDATING. If you follow some other procedure and have trouble, you’ll be referred to the documented procedure.

Build the World

First, build the new userland:

# cd /usr/src
# make buildworld

The make buildworld command builds from source the basic tools needed to build the system compiler and then builds the compiler and associated libraries. Finally, it uses the new tools, compiler, and libraries to build all the software included in a core FreeBSD install. (This is much like building a car starting with the instruction, “Dig iron ore out of the ground.”) The buildworld places its output under /usr/obj. It can take anywhere from one to several hours, depending on your hardware. You can continue working normally as the buildworld runs, if your hardware is robust enough; while the build consumes system resources, it won’t take any of your attention.

When the buildworld finishes, confirm that it completed without errors. If the build ends with a bunch of messages like those you see during a failed kernel compile, do not proceed with the upgrade. If you can’t figure out why the build failed, go to Chapter 1 to see how you can get help. Never attempt to install a damaged or incomplete upgrade.

Build, Install, and Test a Kernel

The best way to test your upgrade is to build a new GENERIC kernel. This separates problems in your custom kernel from general FreeBSD issues. The impetuous are certainly welcome to upgrade straight to their custom kernel configuration, but if your kernel fails, you’ll need to try a GENERIC kernel. Be sure to compare your custom kernel to the new GENERIC configuration, however, to catch any alterations your custom setup needs. You can use the Subversion history at https://svnweb.FreeBSD.org/ to compare the kernel configurations of different releases.

By default, the kernel upgrade process builds a GENERIC kernel. If you want to upgrade straight to a custom kernel, use the variable KERNCONF to tell make(1) the kernel name. You can set KERNCONF on the command line, in /etc/make.conf, or in /etc/src.conf.

You can build a new kernel in one of two ways. The make buildkernel command builds a new kernel but doesn’t install it. Follow a make buildkernel with a make installkernel to install the kernel. The make kernel command runs these two commands right after each other. Use the one that best matches your schedule. For example, if I’m doing a system upgrade at work during my Sunday maintenance window, I might run make buildworld and make buildkernel during the preceding week to save a few hours of my precious weekend. I don’t want to install that kernel before the maintenance day, however—if the machine has a problem on Friday and needs a reboot, I want to boot the old production kernel and not the new, upgraded kernel. On Sunday morning, when I’m ready to actually upgrade, I run make installkernel. On the other hand, using make kernel makes sense when upgrading my desktop. So, to upgrade with my custom kernel, I’d run:

# make KERNCONF=THUD kernel

Again, do not attempt to install a kernel that didn’t successfully compile. If your make buildkernel errors out and dies, fix that problem before proceeding.

Once you have a new kernel installed, reboot your computer into single-user mode. Why single-user mode? The userland might expect different kernel interfaces than the new kernel provides. While /usr/src/UPDATING should list all of these, no document can cover all possible changes and their impact on third-party software. Such changes happen rarely on -stable but unpredictably on -current. If you update your host every week, your userland might have a problem on the new kernel. If you haven’t updated the host for a year, you get a year’s worth of changes dumped on you simultaneously. While many people get away with installing the upgrades in full multiuser mode, single-user mode is safest.

If your system runs correctly in single-user mode with the new kernel, proceed. Otherwise, fully document the issue and boot the old kernel to restore service while you solve the problem.

Prepare to Install the New World

Beware, grasshopper! This is the point of no return. You can easily back out a bad kernel—just boot the older, known good one. Once you install a freshly built world, you can’t revert it out without recovering from backup or using a ZFS boot environment. Confirm that you have a good backup before proceeding, or at least recognize that the first irrevocable step is happening right now.

If your new kernel works, proceed to installing your freshly built userland. First, confirm that your system can install the new binaries. Each new version of FreeBSD expects that the old system supports all the necessary users, groups, and privileges that the new version requires. If a program must be owned by a particular user and that user doesn’t exist on the system, the upgrade will fail. That’s where mergemaster(8) comes in.

mergemaster(8)

The mergemaster program compares the existing configuration files under /etc to the new files in /usr/src/etc, highlights the differences between them, and either installs them for you, sets them aside for evaluation, or even lets you merge two different configuration files into one. This is extremely useful during upgrades. You run mergemaster once before installing the new world to ensure that your system can install the new binaries, and you run it once after installing the new world to synchronize the rest of /etc with your new world.

Start with mergemaster(8)’s prebuildworld mode, using the -Fp flags. The -F flag automatically installs any files that differ only by version control information. The -p flag compares /etc/master.passwd and /etc/group and highlights any accounts or groups that must exist for an installworld to succeed.

  # mergemaster -Fp

➊  *** Creating the temporary root environment in /var/tmp/temproot
   *** /var/tmp/temproot ready for use
➋  *** Creating and populating directory structure in /var/tmp/temproot

   *** Beginning comparison

These initial messages, all preceded by three asterisks, are mergemaster explaining what it’s doing. We start with a temporary root directory ➊ in /var/tmp/temproot so mergemaster can install a pristine set of configuration files ➋ to compare with the installed files. After that, mergemaster shows its first comparison.

➊   *** Displaying differences between ./etc/group and installed version:

  --- /etc/group  2017-09-01 11:12:49.693484000 -0400
  +++ ./etc/group 2017-09-01 13:22:15.849816000 -0400
  @@ -1,6 +1,6 @@
➋  -# $FreeBSD: releng/11.1/etc/group 294896 2016-01-27 06:28:56Z araujo $
➌  +# $FreeBSD: stable/11/etc/group 294896 2016-01-27 06:28:56Z araujo $
    #
➍  -wheel:*:0:root,mwlucas
➎  +wheel:*:0:root
    daemon:*:1:
    kmem:*:2:
    sys:*:3:
  @@ -33,4 +33,3 @@
    hast:*:845:
    nogroup:*:65533:
    nobody:*:65534:
➏  -mwlucas:*:1001:

One vital piece of information is the file being compared, and mergemaster displays the filename ➊ up front. We’re examining /etc/group on the installed system and comparing it to a new /etc/group.

We then see the two different versions of the file being compared, the installed file first ➋ and the upgraded version of the file second ➌. Notice the minus and plus signs at the beginning of these lines. A minus sign indicates that a line is from the currently installed file, while a plus sign shows that a line is from the version in /usr/src.

This is nicely illustrated by the next two lines mergemaster shows. The first group listed, marked by a minus sign, is for the current wheel group ➍. The second line is the password entry ➎ for the out-of-the-box upgrade. This host’s wheel group has a user that’s not in the default install. I want to keep my account there.

At the end of the listing, there’s a group with a minus sign in front of it ➏. This group exists on the local system, but not in the source code. That’s perfectly expected.

None of the changes here are relevant, this time.

Once mergemaster displays all the changes in this file, it displays my options.

  Use 'd' to delete the temporary ./etc/group
  Use 'i' to install the temporary ./etc/group
  Use 'm' to merge the temporary and installed versions
  Use 'v' to view the diff results again

  Default is to leave the temporary file to deal with by hand

How should I deal with this? [Leave it for later] d

I have four choices. I can delete the temporary /etc/group with d. If I want to throw away my current configuration and install one straight from the source code, I can install it with i. If I need some of both the old and new versions, I can use m. And if I wasn’t paying attention, I can see the comparison again with v.

The new /etc/group has no changes I need. I delete it, letting mergemaster go to the next file, /etc/passwd.

The mergemaster display of the password file starts off much like the groups file. Yes, the root password has changed—I’d hope so! There’s an extra entry for my account. But in the middle of the display, there’s an entry like this:

   _pflogd:*:64:64::0:0:pflogd privsep user:/var/empty:/usr/sbin/nologin
➊ +_dhcp:*:65:65::0:0:dhcp programs:/var/empty:/usr/sbin/nologin
   uucp:*:66:66::0:0:UUCP pseudo-user

The line for the user _dhcp ➊ is preceded by a plus sign, and there’s no corresponding _dhcp entry with a minus sign. The user _dhcp exists in the downloaded source code, but not on the currently running system. If a new user appears in the default FreeBSD configuration, it’s because a program or files in the new system expect to be owned by that user.

Installing the new world will fail if this user isn’t present.

I can’t replace my current /etc/passwd, as it contains active user accounts. I can’t throw away the new /etc/passwd because it has users I need in it. I guess I have to merge the two files together.

How should I deal with this? [Leave it for later] m

When merging files, mergemaster splits your command window in half with sdiff(1). The left side displays the beginning of the currently installed file, while the right side shows the new version. Only the sections that differ are shown. Pick the side you want in your new master.passwd file.

# $FreeBSD: releng/11.1/etc/master.passwd 299 | # $FreeBSD: stable/11/etc/master.passwd 29936

This line displays the version control information from both copies of /etc/passwd. On the left, we have the version of this file from the releng/11.1 branch, or 11.1. On the right, we have the version from stable/11, or 11-stable. Future mergemaster runs will use the version information (among other tools) to determine whether a file needs updating, so our merged file needs the correct version number. Choose between the left (l) and right (r) column. I want the entry on the right, so I enter r.

Mergemaster displays the next difference.

root:$6$fD7a5caQtkZbG93E$wGfw5G2zNORLq8qxlT8z | root::0:0::0:0:Charlie &:/root:/bin/csh

I’ve changed my root password, and I want to keep it. Enter l to keep the left-hand version.

                                              > _dhcp:*:65:65::0:0:dhcp programs:/var/empty:/

In this next example, there’s no entry in the current file and the new user _dhcp is in the new file. We need the user _dhcp to complete the installworld, so enter r to choose the right-hand entry and get the next difference.

mwlucas:$1$zxU7ddkN$9GUEEVJH0r.owyAwU0NFX1:10 <

And here’s my account. If I want to log on as myself after the upgrade, I better enter l.

Once we walk through every difference in the file, mergemaster presents our next choices:

  Use 'i' to install merged file
  Use 'r' to re-do the merge
  Use 'v' to view the merged file
  Default is to leave the temporary file to deal with by hand

    *** How should I deal with the merged file? [Leave it for later]

Viewing the merged file is always a good idea, unless you already know you screwed up and want to do it over. Review your merged file with v, and if it looks correct to you, install it with i.

*** You installed a new master.passwd file, so make sure that you run
    '/usr/sbin/pwd_mkdb -p /etc/master.passwd'
     to rebuild your password files

    Would you like to run it now? y or n [n]y

You need to rebuild the password database only if you want your new user account to work. Enter y.

You can now install your new userland.

Installing the World

Still in single-user mode, you can install your brand new FreeBSD with make installworld. You’ll see numerous messages scroll down the screen, mostly including the word install.

# cd /usr/src
# make installworld

You now have a new userland to go with your shiny new kernel. Congratulations!

Obsolete Files

Installing all the new programs isn’t quite enough, though. An update can remove programs and files from the base system. To see what’s obsoleted, run make check-old.

# make check-old
>>> Checking for old files
/usr/include/sys/ksyms.h
/usr/lib/clang/4.0.0/include/sanitizer/allocator_interface.h
/usr/lib/clang/4.0.0/include/sanitizer/asan_interface.h
/usr/lib/clang/4.0.0/include/sanitizer/common_interface_defs.h
--snip--

This lists every part of the system that was once installed on your system but is no longer required. Confirm that you’re no longer using these programs; if you are, either preserve the existing unsupported program or find an alternative.

A little later in the output, you’ll see the shared libraries that are now obsolete:

>>> Checking for old libraries
/lib/libzfs.so.2
/usr/lib/debug/lib/libzfs.so.2.debug
/usr/lib/libarchive.so.6
/usr/lib/debug/usr/lib/libarchive.so.6.debug
/usr/lib/libmilter.so.5
--snip--

Finally, you might see a list of directories that are no longer required. Removing a directory is fairly rare, compared to removing a file.

If you’re not specifically using any of the old programs or directories, delete them with make delete-old. make(1) prompts you with the name of each file and asks you to confirm that you want to delete the file.

# make delete-old
>>> Removing old files (only deletes safe to delete libs)
remove /usr/include/sys/ksyms.h? y
remove /usr/lib/clang/4.0.0/include/sanitizer/allocator_interface.h? y
remove /usr/lib/clang/4.0.0/include/sanitizer/asan_interface.h?

This is stupid. There’s dozens of these files. And I’m going to delete every single one. Fortunately, every real Unix includes tools to automate stupidity.

# yes | make delete-old

All of these files go away. Or, if you want to use FreeBSD’s built-in options, use the BATCH_DELETE_OLD_FILES flag.

# make -DBATCH_DELETE_OLD_FILES delete-old

I find yes(1) easier, though.

Obsolete Shared Libraries

Obsolete shared libraries require more care. Many third-party programs link against shared libraries. If you delete the shared library, the program won’t run. This can be really, really annoying if you, say, delete the library required by your mission-critical application. The only way to restore service is to recompile the program or replace the shared library. We discuss shared libraries in Chapter 17. If none of your programs require the library, you can delete it. Identifying every program that requires a library is a royal pain, however.

For example, check the list of obsolete shared libraries above. One of the entries is libzfs.so.2. Looking in /lib, I see that we now have libzfs.so.3. Perhaps I shouldn’t need two different versions of the ZFS library. This host uses ZFS, though, and I have a whole bunch of ZFS utilities installed. If I remove the old version of libzfs, there’s a chance one of those programs won’t work anymore. The presence of these obsolete library versions doesn’t hurt anything in the short term; you can bring your system back on line with the old libraries in addition to the new ones and reinstall your add-on software in a more leisurely manner. We’ll update your ports later in this chapter.

If you believe that none of the libraries listed as old are important and you can safely delete them, back up each before removing it. By just copying the library to an old-libs directory somewhere, you’ll make recovery much simpler when you find out that your mission-critical software doesn’t work anymore.

You can also copy old libraries into /usr/lib/compat so that your programs will continue to run but the old libraries will be safely out of the way. The problem here is that we both know perfectly well that you’re never going to go clean up those libraries.

I prefer to back up the libraries and then remove them from the live system. When I find a program doesn’t work, I temporarily copy the missing library from the backup into a compat directory. The added annoyance ticks me off enough to solve the real problem, so I can delete the compat library.

# make check-old-libs | grep '^/' | tar zcv -T - -f /root/old-libs.tgz

Once you have the libraries backed up, delete them all. You can use the BATCH_DELETE_OLD_FILES option here, but once again, I find yes(1) easier to type.

# yes | make delete-old-libs

If by some chance removing these libraries breaks pkg(8), run pkg-static install -f pkg to fix pkg(8) itself or pkg-static upgrade -f to reinstall all packages.

Another option is to use the libchk package to identify programs linked against old libraries.

mergemaster Revisited

We’re almost there! While we already updated the passwords and group information in /etc, we must update the rest of the files. mergemaster has many special functions, all documented in its man page. I’m going to specifically recommend the two that I find notably useful.

When a file is added to the base FreeBSD install, there’s no need to compare it to anything. The -i option makes mergemaster automatically install new files in /etc. I’ll get a list of automatically installed files at the end of the mergemaster run.

Another set of files that I don’t really care about are files that I haven’t edited. For example, FreeBSD has dozens of startup scripts in /etc/rc.d. If I haven’t edited a startup script, I just want to install the newest version of the script. The -U flag tells mergemaster to automatically update any base system file that I haven’t edited.

# mergemaster -iU

The mergemaster program examines every file in /etc and compares it to that in the base distribution of FreeBSD. This works exactly the same way as in your preinstallation mergemaster run, so we’re not going to walk through it here. You should be familiar with the customizations you’ve made to your system, so nothing should surprise you. If anything looks unfamiliar, refer to Chapter 14.

Reboot, and your base system is fully upgraded!

Customizing Mergemaster

Once you’ve run mergemaster a few times you’ll realize that certain files always annoy you. Mergemaster will always complain about your customized /etc/motd and /etc/printcap. You’ll probably wind up typing -F or -U every single time. You can set your preferred options in $HOME/.mergemasterrc, as documented in mergemaster(8). While you should read the man page for the complete list, here are the options I use most often.

Ignoring Files

Certain files you don’t want mergemaster to even bother examining. Your organization’s /etc/motd will never match that in the FreeBSD distribution. Neither will your custom printer configuration, your snmpd.conf, or your tailored sshd_config. To have mergemaster skip these files, list them in IGNORE_FILES.

IGNORE_FILES='/etc/motd /etc/printcap'

I don’t list the password or group file here because sometimes FreeBSD introduces new users.

Auto Install New Files

If you want mergemaster to automatically install files present in the new version of FreeBSD but not on the system yet, set AUTO_INSTALL.

AUTO_INSTALL=yes

This is equivalent to the -i flag.

Autoupdate Unchanged Files

Different versions of FreeBSD have similar files. Some files are almost identical, differing only by the line containing version control information. You can tell mergemaster to automatically update files that differ only by the version control information by using the FREEBSD_ID option.

FREEBSD_ID=yes

This is the same as the -F flag.

Autoupdate Unedited Files

You can tell mergemaster to update files that haven’t been edited since they were installed. The FreeBSD team changes /etc/ files when it wants to change how the system behaves. While many of those changes might be irrelevant to you, a few might give you trouble. If you want to blindly update everything you haven’t touched, set AUTO_UPGRADE.

AUTO_UPGRADE=yes

This is equivalent to the -U option.

Update Databases

FreeBSD builds databases from /etc/master.passwd, /etc/services, and so on. If you update these files, you also need to update the corresponding databases. Mergemaster asks you at the end of each run if you want to run these updates. Tell mergemaster always to run the updates by setting RUN_UPDATES.

RUN_UPDATES=yes

You can find other options in mergemaster(8).

Upgrades and Single-User Mode

According to the instructions, several parts of the upgrade must be done in single-user mode. Many users consider this an annoyance or even a handicap. FreeBSD programs are just files on disk, aren’t they? Common sense says that you can just copy them to the disk, reboot, and be done with it.

Here’s yet another instance where your common sense is trying to ruin your month. On rare occasions, the FreeBSD team needs to make some low-level changes in the system that require running the install in single-user mode. You can have conflicts where vital programs won’t run when installed in multiuser mode. This is rare, but if it happens with /bin/sh, you’re in a world of hurt. You have a very straightforward recovery route if that happens: remove the hard drive from the server, mount it in another machine, boot the other machine, and copy your data off the destroyed system before formatting and reinstalling. Or, you can boot from the installation media and demonstrate your amazing sysadmin skills.⁵

Running in multiuser mode can cause other upgrade problems, such as subtle races, symbol issues, and innumerable other headaches. You can choose to upgrade in multiuser mode, but don’t complain if your system has a problem.

It’s perfectly safe to build your new world in multiuser mode. You can even build and install your new kernel in multiuser mode. Once you start installing the userland, however, you must be in single-user mode and running on your upgraded kernel.

Shrinking FreeBSD

What’s the point of having all this source code if you can’t customize your operating system? FreeBSD not only gives you the source code; it provides ready-to-turn knobs to easily customize your FreeBSD build.

These options can be set in either /etc/make.conf (see Chapter 16) or /etc/src.conf. Settings in src.conf apply only to building the FreeBSD source, while make.conf ’s settings apply to all source code building. The full list of src.conf options are documented in src.conf(5), but they all follow a standard pattern.

Each of these options starts with either WITHOUT_ or, in a few cases, WITH_ and then names a specific subsystem. For example, the WITHOUT_BHYVE option turns off building or installing the bhyve(8) hypervisor. The WITHOUT_INETD option turns off building the inetd(8) daemon (see Chapter 20). The WITHOUT_INET6 option turns off IPv6. If you want to shrink your FreeBSD install, chop out everything you don’t need.

The build system checks to see whether any of these variables are defined to any value at all. This means that even if you set one of these to NO, the mere presence of the option activates the option. Don’t go copying all of these to src.conf and setting them all to NO because you’ll disable building a great big bunch of the system.

In most cases, adding these WITHOUT_ options includes the removed systems in the make delete-old checks. If you decide that your system doesn’t need bhyve(8), for example, the upgrade not only doesn’t build a new bhyve binary but also offers to remove the existing one from the installed system. If you’re not building a piece of software, you’re better off removing it entirely as opposed to leaving the old version lingering on the system.

Packages and System Upgrades

Operating system upgrades are great, except for the part where nobody cares.⁶ Base operating system updates are necessary, but most people don’t really care about using the base system. They care about using software that runs on the base system. And that software is prone to the same flaws as every other program. You need to upgrade it. Chapter 15 discusses upgrading packages in general, but let’s talk about what happens when you upgrade the operating system underneath the packages.

The common issues with packages and system upgrades normally boil down to shared libraries. If you’re upgrading FreeBSD major releases—say, from FreeBSD 12 to FreeBSD 13—you’ll need to reinstall all of your packages.

Start by upgrading pkg(8) itself, using the -f flag to pkg upgrade. If the upgrade broke pkg(8) itself, you’ll need to use pkg-static(8). This contains key pkg(8) functions, but it’s a static binary.

# pkg-static upgrade -f pkg

This will bootstrap you into the current package tools. Now you can force a redownload and reinstall of all packages.

# pkg upgrade -f

Once you’ve upgraded everything you built from packages, rebuild anything you built from ports. I really hope you installed your ports via poudriere, though.

Updating Installed Ports

If you use portsnap to update your ports tree, anything you install from now on will be the latest version. But what about your previously installed applications? FreeBSD tracks all sorts of dependency information between add-on packages, and often updating one program will impact dozens of others. This is a royal pain to manage. Wouldn’t it be nice to just say, “Update my Apache install,” and have FreeBSD manage the dependencies for you? There’s a few ways to solve this issue.

The best way is not to use the ports tree on a production host. Build a private repository with poudriere instead (see Chapter 16), and have all of your hosts pull from that. This is a change from traditional FreeBSD sysadmin practice.

Maybe you have the ports tree installed locally and use one or two custom ports atop a bunch of packages. If you have a single port installed, rebuild it, uninstall the package, and install the new port. This is terrible for complicated ports with many dependencies but works fine for hosts with one or two ports.

Some of us have been around a long time, though, and feel like we live between those solutions. Our hosts feel too small to run poudriere, but we need custom ports. FreeBSD doesn’t include an official tool for updating a system managed largely by ports, but people have written add-on tools, such as portupgrade and portmaster. The problem with these tools is that they’re maintained outside of FreeBSD. If they can’t upgrade a port but the normal build process works, the tool users are responsible for fixing the problems. That’s true of all parts of FreeBSD, but the base system has a wider base of users than any add-on tool.

As I write this, though, FreeBSD’s ports infrastructure is changing to support multiple versions of a single package. These tools haven’t been updated to accommodate the new infrastructure. I expect one of them will be, but I can’t say which that will be. Chapter 16 recommended using only packages in production. This is why.

Now that you can update your system and installed software, let’s look at some of FreeBSD’s more interesting security features.