15
MAKING YOUR SYSTEM USEFUL

A basic FreeBSD install includes exactly enough to make the system run, plus a few extra bits that Unix systems traditionally include. You can decide whether to install additional programs or source code. While FreeBSD has grown over the years, a complete base install fills about a gigabyte—much less disk space than either a Windows or a commercial Linux install.

The advantage to this sparseness is that it includes only necessary system components. Debugging becomes much simpler when you know that no shared library you’ve never even heard of, and would never use, can be responsible for your problems. The downside is that you must decide what functions you do need and select software to provide those functions. FreeBSD simplifies add-on software installation through ports and packages.

Ports and Packages

FreeBSD supports two different ways to install add-on software. Everything starts with the Ports Collection, but most users will prefer preconfigured packages.

FreeBSD has a system for building add-on software called the Ports Collection, or just ports. Ports let you start with raw source code from the program vendor and build the software in exactly the way you need, enabling and disabling features as you need. Ports are fast and easy for the experienced user but require a certain amount of Unix expertise and can intimidate the new user.

Packages are the result of building ports, using the options the port maintainer thinks will be most useful to the widest variety of people, and bundling them up in a lump to make them easily installable. The FreeBSD Project has a whole farm of systems that do nothing but build all the ports, bundle them up, and make them available for users to download and install. Packages let you quickly install, uninstall, and upgrade add-on software.

FreeBSD’s highly flexible packaging system is called package, pkg(8), or just plain pkg. Package information gets stored in SQLite databases, which lets you perform arbitrary queries on package data. During development, pkg was called pkgNG. That name’s been gone for years now but lingers on in some old docs and third-party software. Don’t let the name confuse you.¹

We’ll start by discussing managing systems with pkg(8) and then proceed to customizing software with ports.

Packages

Packages are precompiled software from the Ports Collection, bundled up for a particular version of FreeBSD. The FreeBSD Project offers several sets of packages in a public repository, updated every few days. Packages are the simplest way to install add-on software. Any software without legal restrictions on its distribution is probably available as a package.

Legal restrictions? Software can have any license terms, including some really odd ones. The license of some software prohibits distribution in any form other than source code. FreeBSD can’t legally package that. Other software can legally be distributed only in precompiled form. FreeBSD usually packages such software, distributing it as the precompiled binary plus FreeBSD-specific packaging information.

Packages are built on the oldest supported release of each major FreeBSD version. Packages for all versions of FreeBSD 12 are built on the oldest supported release of FreeBSD 12, FreeBSD 13 packages are built on the oldest supported version of FreeBSD 13, and so on. This helps reduce, identify, and contain ABI incompatibilities.

If you need to build your own package repository, investigate the Ports Collection (see Chapter 16) and the add-on package poudriere.

Package Files

Ultimately, packages contain files. Those files might be binary programs, documentation, configuration files, or anything else the software might need. These files are considered part of the operating system. Don’t edit them manually.

The one odd case is when a package contains a sample configuration file. If a program needs a configuration file, the package should include a sample. You’ll need to edit the configuration to fit your needs—that’s what configuration files are for.

FreeBSD reconciles this by installing the package’s configuration files with the suffix .sample. Our web server configuration file appears as something like httpd.conf.sample.

If there’s no production configuration file, the package installation also copies the sample file into place. That file is yours to edit.

If you upgrade a package, pkg(8) compares the current production file to the old sample file. If the sample is identical to the production version, the upgrade replaces the production file. If the files differ in any way, pkg updates only the sample file. It’s your job to merge any desirable changes into your production configuration. Note that the package upgrade always replaces the sample configuration, so if an old sample is important, you need to make a point to hang on to it.

Introducing pkg(8)

Unlike the older packaging system, pkg(8) is a single program with a whole flock of subcommands. You’ll use the same program to install, uninstall, and investigate packages. All changes to installed packages must be run as root. Here’s how you’d install a vital program desired by all right-thinking sysadmins:

# pkg install emacs

Those of you clinging to irrational biases against superior text processors probably want to remove it.

# pkg delete emacs

All package operations use the pkg(8) command.

While the pkg(8) man page documents the base pkg functions, each subcommand has its own man page, named pkg- and the subcommand. Examples include pkg-install(8) and pkg-delete(8). You can also use the pkg help command and the name of the subcommand to get assistance—for example, pkg help install.

FreeBSD doesn’t ship with pkg(8) installed. You need to install it . . . as a package. No, wait, don’t scream—it’s much better than it sounds.

Installing pkg(8)

FreeBSD ships with a very simple package manager in /usr/sbin/pkg, pkg(7). It has barely enough brains to find FreeBSD’s current package manager. It installs that new package manager and surrenders all responsibility for package management to it. This gives FreeBSD the flexibility to update the package manager with the packages.

The first time you try to install a package, pkg(8) prompts you to install the package manager. I found I needed the dmidecode package on a new server, so I can get an RMA on a bad power supply from the manufacturer. (Don’t worry about how I found the dmidecode package—just go with me for the moment.)

# pkg install dmidecode

FreeBSD runs pkg(8) and finds that no package management is installed yet.

The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y

The default answer appears in capital letters. If I hit n and ENTER, pkg terminates. If I hit y and ENTER, FreeBSD bootstraps the system.

➊ Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
➋ Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
➌ Installing pkg-1.10.0_2...
   Extracting pkg-1.10.0_2: 100%
➍ Updating FreeBSD repository catalogue...
   meta.txz                                 : 100%  944 B     0.9kB/s    00:01
   packagesite.txz                          : 100%    6 MiB   2.0MB/s    00:03
   Processing entries: 100%
   FreeBSD repository update completed. 26059 packages processed.
   All repositories are up to date.
   Updating database digests format: 100%

The installation starts by downloading the current pkg tools from a FreeBSD mirror ➊. It then checks the digital signature on the downloaded file ➋. The tools are extracted and installed ➌. Pkg then downloads and installs the catalog of available packages ➍.

The packaging system is now installed. To smooth things out, the stub pkg(8) that ships with FreeBSD tells the newly installed packaging system to install the program you really wanted. In this case, our new pkg(8) installs dmidecode for you.

You can install the packaging system on its own, without adding other packages, by running pkg bootstrap—but seriously, nobody does that at the command line. Running pkg bootstrap does nothing when the packaging system is already installed, so it’s useful for setup scripts.

Common pkg Options

While each pkg subcommand has unique features, a few command options work across almost all of them.

In its default configuration, pkg prompts you for confirmation before doing anything. Tell pkg to take action without prompting you with the -y flag.

On the other hand, maybe you want pkg to show you what it would do if you ran a command, but not actually do anything. Perform a dry run by using the -n flag. For example, a package install using -n would show the names of every package to be installed, including dependencies. There’s no risk that the system would install the packages, however. Dry runs can help you prepare for changes during a maintenance window.

Many pkg operations produce a bunch of output. Reduce the amount of output with -q.

The -a flag usually applies a command to all installed packages.

Finally, pkg usually refuses to do pointless things or things that damage the system. The -f flag forces pkg to do what you said. Forcing package activities is usually, but not always, a bad idea. For example, you might need -f to forcibly reinstall a damaged package.

Configuring pkg(8)

The pkg(8) program is designed to be highly flexible. While each subcommand has a whole bunch of options, you can establish customized but consistent behavior for most programs with the configuration file, /usr/local/etc/pkg.conf.

The pkg.conf file contains commented-out defaults for pkg(8). It’s a great place to look to see how the system behaves when you haven’t mucked with it at all. The configuration is written in UCL (see Chapter 2). Variables can be set to an integer; a string, such as a file path; or a Boolean value, like YES or NO. YES, ON, and TRUE are synonyms, as are NO, OFF, and FALSE. All are case-insensitive.

#PKG_DBDIR = "/var/db/pkg";
#PKG_CACHEDIR = "/var/cache/pkg";
#PORTSDIR = "/usr/ports";
#INDEXDIR = "";
--snip--

FreeBSD runs perfectly well with an empty pkg.conf. The default configuration contains a whole bunch of commented-out entries and quite a few aliases. You might consider these example settings as you proceed.

Most pkg operations offer a yes/no dialog, showing the default as a capital letter. Being conservative, pkg normally defaults to NO. Change that default to YES with the DEFAULT_ALWAYS_YES option.

You can make a pkg command assume you’ll answer yes to everything by adding the -y flag. If you get tired of typing -y, make pkg assume you always answer yes by setting the ASSUME_ALWAYS_YES flag to YES.

As I’m lazy but not reckless, I prefer these pkg.conf settings:

DEFAULT_ALWAYS_YES = true;
ASSUME_ALWAYS_YES = false;

If installing a package runs amok, you might want debugging output. Setting DEBUG_LEVEL turns on debugging output. This variable accepts an integer from 0 (no debugging) to 4 (complete debugging).

Many packages include scripts as part of their installation procedure. Turn on debugging for each script by setting DEBUG_SCRIPTS to YES.

Any pkg.conf settings are also usable as environment variables. Environment variables override anything in the configuration file. You could install a package with debugging like this:

   # env DEBUG_LEVEL=4 pkg upgrade

All of the options are documented in pkg.conf(5). Not all of the options have a commented-out entry, though. If a sample of an option doesn’t exist but you want it, add it. We’ll examine many of them in the following sections.

Finding Packages

Now that you have a package manager installed, you can install packages. Sysadmins familiar with a variety of Unix-like operating systems know that different operating systems assign different names to packaged versions of the same software. A package for the Apache web server on FreeBSD will have a completely different name than the packaged Apache on illumos or even different Linux distributions. Before you can install anything, you’ll need to figure out what it’s called.

Suppose the client wants to run WordPress on Apache. Your job isn’t to question the client’s choice in web servers; your job is to build and support the web server. First, find Apache with the pkg search command. You’ll need to provide a text string for pkg to perform a case-insensitive search.

# pkg search apache
apache-ant-1.9.7    Java- and XML-based build tool
apache-forrest-0.9  Tool for rapid development of small sites
apache-mode.el-2.0  Emacs major mode for editing Apache configuration files
--snip--

I deliberately picked an annoying example; FreeBSD has 50-odd packages related to the Apache web server. Fortunately, each search result lists a one-line package description. It’s pretty easy to flip through the results until you find the actual web servers.

--snip--
apache22-2.2.31_1              Version 2.2.x of Apache web server with prefork MPM
apache22-event-mpm-2.2.31_1    Version 2.2.x of Apache web server with event MPM
apache22-itk-mpm-2.2.31_1      Version 2.2.x of Apache web server with itk MPM
apache22-peruser-mpm-2.2.31_1  Version 2.2.x of Apache web server with peruser MPM
apache22-worker-mpm-2.2.31_1   Version 2.2.x of Apache web server with worker MPM
apache24-2.4.25_1              Version 2.4.x of Apache web server
--snip--

Six different versions of Apache. First, look at the package names. When a piece of software comes in multiple versions, the major version number gets integrated into the package name. Apache 2.2 is a very different beast than Apache 2.4, so the packages are named apache22 and apache24. The actual version number follows. Our first Apache 2.2 package is actually for Apache 2.2.31. The trailing _1 is the package version number, which means that this an updated package. The included software hasn’t changed, but the package has been altered somehow. Package version numbers get bumped for two reasons. When the source port changes in a way that has a material impact on the package, the version number is increased. When an ABI change in a required shared library demands recompiling the package, that also merits a version bump.

Apache 2.2 comes in five different packages. People familiar with Apache probably remember that this version of Apache could use different Multi-Processing Modules (MPMs), but the MPM had to be selected at compile time. I have blissfully forgotten everything I ever knew about MPMs, so rather than fuss with them, I’ll choose to install the Apache 2.4 package, apache24.

Package Searching Options

Some searches can generate hundreds of results. Try searching for Perl, and you’ll get about 150 packages. Perl modules all begin with the string p5-; FreeBSD has packages for over 5,200 Perl modules! Use command line options to trim or adjust the search results. While pkg-search(8) lists many options, here are some of the most common.

• Make a search case-sensitive with -C.
• If you know exactly which package you want, and you only want to see whether it’s available for your system, use the -e flag to search for an exact match. Your search term must include the package version number, though.
• If you need to highly customize your searches and your search results, investigate the -L, -S, and -Q flags in pkg-search(8).

Examining Found Packages

Perhaps you’re not sure whether a package is what you really want. You might look up details on the package from a third-party site, like FreshPorts (https://www.freshports.org/), but that would require leaving your terminal, and I can’t countenance that. Use the -R flag to examine the repository catalog’s metadata for the package. This metadata is a subset of the full package manifest built into each package.

# pkg search -R apache24
name: "apache24"
origin: "www/apache24"
version: "2.4.25_1"
comment: "Version 2.4.x of Apache web server"
maintainer: "[email protected]"
www: "http://httpd.apache.org/"
--snip--

The package manifest includes fields for the package name, the port the package is built from, the software version, the package repository, dependencies, and more. It’s rarely enough used and subject to change, so we won’t discuss it in detail, but scrolling through this information provides more details about the software inside the package.

One important detail here is the www field, which gives the website the original software comes from. This is the Apache web server, not a fork or some other project using that name.

The default format for this raw manifest is YAML, or “YAML Ain’t Markup Language.” It’s yet another syntax for formatting configuration files, but it’s fairly human-readable. Use the --raw-format flag to choose an alternate format. Other supported formats include json and json-compact.

# pkg search -R --raw-format json-compact apache24

If you want to automatically parse package information, this is how you grab the raw data.

Installing Software

Use pkg’s install subcommand and the name of a package to install a package. You don’t need to give the complete package name.

# pkg install apache24

The first thing that happens is that pkg checks to see whether its local copy of the package database is the same as that on the package server. You’ll either get a message like “Updating FreeBSD repository catalogue” or be told that the “FreeBSD repository is up to date.”

The system then checks for any packages that your chosen package requires. Read the dependency list. Is there anything here you don’t want installed on this host? Does the list give you a reason not to install the package?

The following 8 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
        apache24: 2.4.25_1
        libxml2: 2.9.4
--snip--
Number of packages to be installed: 8

As a final warning, pkg tells you how much disk space and bandwidth the installation requires. You then get prompted to change your mind.

The process will require 139 MiB more space.
33 MiB to be downloaded.

Proceed with this action? [y/N]:

Enter y, and pkg fetches the package from the repository and installs it to your system.

The install subcommand assumes that you’re either giving the complete name of a package or the name of a package without the package version number. You can request the apache24 package and pkg will figure out that the current package is apache24-2.4.25_1. You can also use the name of the port the package was built from, as in pkg install www/apache24.

In the last section, our package search turned up five different Apache 2.2 packages, each a slightly different variant. If you ask pkg install to grab the apache22 package, it installs the version named apache22 plus a package version number. If you want a variant, such as apache22-event-mpm, specify the full package name in the install command.

Some packages include installation messages. These messages might be helpful instructions, warnings, caveats, or anything else relevant. If the package creator felt a chunk of information sufficiently important enough to spend her precious time composing a message about it, then the least you can do is read it. You might use script(1) to record this information or run pkg info --pkg-message and the package name to show it again.

Fetching Packages

FreeBSD installs packages by downloading them over the internet. You might want to download packages in one location to install them elsewhere or at another time. Use the pkg fetch command to download but not install packages. Fetching packages makes the most sense when combined with -d, which makes pkg fetch grab all the dependencies as well as the named package.

# pkg fetch -d apache24

You’ll see the normal repository update messages, followed by a notice of what pkg will download.

New packages to be FETCHED:
        apache24-2.4.25_1 (5 MiB: 14.25% of the 33 MiB to download)
        libxml2-2.9.4 (821 KiB: 2.43% of the 33 MiB to download)
--snip--

Verify that what pkg plans to download matches what you expect, and then hit y to proceed. The packages are downloaded to the package file cache.

To install a downloaded package, run pkg install normally. The installation process uses the cached files rather than the downloaded ones.

Those of you who read man pages might notice the -a flag, which downloads the entire package repository. Don’t use that. The -a option is intended for public repository mirrors. Average sysadmins who mirror the entire repository waste bandwidth and slow down the system for everyone. Generous people donate FreeBSD’s package server bandwidth. Don’t waste it. You might need a whole bunch of packages. With dependencies, you might need hundreds or even thousands of packages. You don’t need tens of thousands of packages. Download only what you need.

Download Timing

Any tool that accesses the internet needs to set a maximum length of time to try to download files. You can customize pkg’s download behavior with two pkg.conf settings.

If a download fails, pkg tries again. The FETCH_RETRY option controls how many times pkg retries a download. The default is three, which means that it tries to download one time and retries up to three more times.

Downloads happen fairly quickly on most modern internet connections. If your uplink isn’t quite so modern, you might need to increase the amount of time pkg will spend on a single download attempt. The FETCH_TIMEOUT setting controls how long pkg waits for any one file to download. The default, 30, limits downloads to 30 seconds. If you’re downloading LibreOffice over a 33.6 modem, you’ll want to increase this setting and consider having files shipped to you on a removable drive via the Pony Express.

The Package Cache

The ability to download packages and install them later implies that pkg(8) sticks those packages somewhere on the disk for later consumption. The package cache, /var/cache/pkg, contains the original package files downloaded from the internet. While you can administer FreeBSD hosts for years without futzing with the cache, here are a few things you should know.

Cleaning the Cache

What with upgrades, new packages, removed packages, and the gleeful randomness of system administration, the cache directory can fill up. My web server has only a few packages but somehow has accumulated 1.7GB of old package files. The pkg clean command removes any cached packages that have been replaced by newer versions, as well as any package files that are no longer in the repository. You’ll get a list of all the files that will get removed, plus a chance to proceed or quit.

# pkg clean
The following package files will be deleted:
        /var/cache/pkg/php56-mbstring-5.6.26.txz
        /var/cache/pkg/mod_php56-5.6.21-c80f5ce183.txz
--snip--

If you’ve never cleaned the package cache on a long-running system, the list will be pretty long. At the prompt, hit y to proceed.

If you want to remove all cached packages, use the -a flag.

Remember that pkg clean removes package files that are no longer available in the package repository. If you depend on a package that’s been removed from the repository, back up that file outside the cache before a thoughtless cleaning removes it forever. You could also try pkg-create(8) to rebuild a package from its installed components.

If you want to clean the package cache automatically after each package install or upgrade, set the pkg.conf option AUTOCLEAN to true. I find autocleaning too aggressive, as sometimes the new bugs in an upgraded package compel me to revert to the older version. We cover upgrading packages at the end of this chapter.

Moving the Cache

You might want the package cache located elsewhere on the filesystem. Use the pkg.conf option PKG_CACHEDIR to set a new package cache directory.

Why move the cache directory? Many server farms share a package cache across multiple machines. You can safely share a package cache between hosts running the same FreeBSD major release and hardware architecture. Verify that your NFS configuration uses locking, and set the pkg.conf option NFS_WITH_PROPER_LOCKING.

Package Information and Automatic Installs

After a while, you’ll forget which packages you’ve installed on a system. Get the complete list of installed software with pkg info.

# pkg info
gettext-runtime-0.19.8.1_1  GNU gettext runtime libraries and programs
indexinfo-0.2.6             Utility to regenerate the GNU info page index
--snip--

If you want more information about an installed package, use pkg info and the package name. This shows the package manifest and installation details in a human-friendly report.

# pkg info apache24
apache24-2.4.25_1
Name           : apache24
Version        : 2.4.25_1
Installed on   : Tue Mar 14 16:56:14 2017 EDT
Origin         : www/apache24
Architecture   : freebsd:12:x86:64
--snip--

When was the package installed? Was the package installed on this machine built from the Ports Tree with certain options enabled? What’s the license? What shared libraries does every program in the package require? Answer all these and more with pkg info and the package name.

The pkg info subcommand has many other features. We’ll see some of them, such as locking status, later this chapter. The pkg-info(8) man page has the complete details.

Automatic Packages

Look back at the sample pkg info output. I deliberately installed a few different programs on this system, but I’m pretty sure I never knowingly installed anything about GNU info pages or gettext.

I did install those programs. I merely didn’t pay much attention to what they were because I was more concerned about installing the package that required them. They’re dependencies.

FreeBSD records whether you requested a package be installed or it was brought along as a dependency. Packages installed as dependencies are called automatic packages. Packages you requested are just packages, although they’re sometimes called nonautomatic packages.

You might want to know which packages you requested to be installed and which were dragged along as dependencies. That’s when things get tricky.

Querying the Package Database

The pkg tools can’t cover every possible contingency a sysadmin might face. The simplest way to get some information is to interrogate the installed package database. While you could use raw SQLite, that would mean you’d need to become intimate with the database’s innards. Most sysadmins don’t have that kind of time, especially when that database might change any time. FreeBSD insulates from that with the pkg query subcommand. A complete survey of package queries would fill a chapter, but here’s a quick overview.

Anything you might want to get out of the package database has a convenient representation in pkg query. The catch is, everything than anyone might possibly want to extract from the package database is in pkg query, as a quick perusal of pkg-query(8) shows. The query and command structure is deliberately designed for use in scripts, but we’ll use it interactively now and then.

Run queries by using patterns. A pattern is a variable that has an assigned meaning, represented by a percent sign and a letter. For example, %n contains the package name, %o contains the port the package was built from, and %t contains the timestamp indicating when the package was installed.

Running pkg query and giving a pattern produces that value for every installed package. As %n represents the package name, here’s how you’d get a list of everything on the system:

# pkg query %n
apache24
apr
--snip--

We don’t get the extra information pkg info shows—but maybe that’s what you want.

You can request multiple items in a single query. The %v pattern represents the package version, while %c represents the comment. Here, I separate the package name and version with a dash but put a tab between the version and the comment. Using the shell tab character means I must quote the pkg query argument.

# pkg query "%n-%v %c"
apache24-2.4.25_1       Version 2.4.x of Apache web server
apr-1.5.2.1.5.4_2       Apache Portability Library
--snip--

You know, this looks an awful lot like the output of pkg info. When a pkg command queries or manipulates the package database, it uses these exact same patterns. You have the same visibility into the packaging system that the rest of the tools do.

If you want to get a pattern for a specific package, give the package name as a final argument. Here, I get the port the apache24 package came from:

# pkg query %o apache24
www/apache24

We do have a middle ground between asking all the packages and specific packages, however.

Evaluations in Queries

Here’s one last nifty package querying feature. Many—not all, but many—patterns are available as variables. A command can evaluate those variables and take action based on the results. Use the -e command line option to evaluate a variable with using a logical operator. A complete list of logical operators appears in pkg-query(8).

Evaluation breaks down into “if this is true, do that.” The test goes inside quotes. Here’s an example:

# pkg query -e '%a = 0' %n

This query goes down the whole list of installed packages. The -e shows we’re evaluating a variable for each package. The statement inside the quotes, %a = 0, means we’re testing the value of %a in that package. If %a equals 0, the query evaluates to true and pkg query prints out the contents of %n. If %a equals anything except 0, the statement is false and pkg query proceeds to the next package without doing anything.

We already know that %n contains the package name. The variable %a contains pkg’s record of whether or not the package was automatically installed. If you requested this particular package, it’s set to 0. If a package was originally installed as a dependency, it’s set to 1. So: if a package is not a dependency, print the name. This query prints nonautomatic packages.

# pkg query -e '%a = 0' %n
apache24
dmidecode
pkg
youtube_dl

A couple things stand out here. First, I didn’t deliberately ask pkg to install pkg(8). I requested dmidecode, and pkg bootstrapped itself. The pkg suite itself is always considered a nonautomatic package, though.

Second: who installed youtube_dl on this box?

To find out which packages were installed as dependencies, evaluate whether %a is set to 1.

Realistically, though, I’m not going to bother remembering how to run this query on all my hosts. I need a simple way to make pkg(8) remember it for me.

Pkg Command Aliases

You can define aliases for pkg subcommands in pkg.conf. This lets you, say, create aliases to show automatic and nonautomatic commands. I could do something similar in my shell, but it wouldn’t show up as pkg(8) subcommands and I’m easily confused.

At the bottom of pkg.conf, you’ll find a section labeled ALIAS.

ALIAS              : {
  all-depends: query %dn-%dv,
  annotations: info -A,
--snip--
  }

An alias is a single word for the alias name, either a colon or an equal sign, and then the pkg command to run. If you run pkg all-depends, pkg(8) looks in pkg.conf and runs pkg query %dn-%dv. Every alias ends in a colon to indicate that the aliases list continues on the next line.

Many of the aliases in the default configuration represent hangovers from the pkg_add aeon, created for us old timers. The existing aliases are a great place to find sample queries and searches, though. And searching through the aliases turns up this fine entry:

noauto = "query -e '%a == 0' '%n-%v'",

This alias, noauto, runs a pkg query command to evaluate %a and print the package’s name and version number if it’s 0. It prints packages that weren’t automatically installed. I added a very similar alias to print automatic packages.

auto = "query -e '%a == 1' '%n-%v'",

When you find yourself repeatedly running complex commands, add aliases.

Uninstalling Packages

We’ve all installed software only to rip it out in disgust. The only difference is what, exactly, disgusted us. Uninstall packages with the pkg delete subcommand. It’s also available as pkg remove. That extraneous youtube_dl package? Let’s remove it from the system.

# pkg delete youtube_dl
Checking integrity... done (0 conflicting)

The removal process makes sure that nothing terrible has happened to the package, that nobody else needs it, and that its removal won’t do terrible things that the package system can predict.²

You’ll then get a list of packages to be removed and how much space they’ll free up. At the end is a final chance to say no.

Proceed with deinstalling packages? [Y/n]: y

[1/1] Deinstalling youtube_dl-2017.02.11...
[1/1] Deleting files for youtube_dl-2017.02.11: 100%

The package is deleted from your system.

Removing Dependencies

If you remove a package that other packages depend on, pkg removes the depending packages as well.

# pkg delete trousers
--snip--
Installed packages to be REMOVED:
        trousers-0.3.14_1
        gnutls-3.5.9
        emacs-nox11-25.1,3
--snip--

The gnutls package needs trousers, and emacs-nox11 needs gnutls. Removing trousers breaks both of them, so pkg figures you clearly don’t want them on your system either.

If you really want to delete a package that other packages depend on, add the -f flag.

Read the warnings from pkg delete very carefully!

Autoremoval

Leaving unnecessary software installed on a host increases the security risks and sysadmin workload. On a long-running system, you don’t always know which software to remove. Removing software you chose to install is easy, but that software might have brought along dependencies that you never really paid attention to. Or maybe a new version of a package has fewer or different dependencies than the previous version.

I removed the youtube_dl package from my test system. That leaves me with other packages I deliberately installed and their dependencies. It also leaves the packages youtube_dl depended on but that nothing else needs. The pkg autoremove subcommand identifies packages that were installed as dependencies but are no longer required by any other package. It offers to remove these no longer needed. I strongly recommend performing a dry run before removing unneeded dependencies, simply to give your feeble human brain a chance to look at the list twice.

# pkg autoremove

Pkg runs a database query to identify unneeded dependencies and proposes them for removal.

Installed packages to be REMOVED:
        python27-2.7.13_1
        readline-6.3.8
        rtmpdump-2.4.20151223
        librtmp-2.4.20151223
--snip--

Study this list carefully. It’s not uncommon for a piece of nonpackaged software to need a package that was brought in elsewhere. You probably don’t need the video processing tools rtmpdump and librtmp without youtube_dl, but an awful lot of software needs a Python interpreter. Do you really want to blow that away?

If you really can remove all these packages, answer y and proceed. If one of those dependencies has become critical, though, change your database to tell it so.

Changing the Package Database

Thinking of changing the package database outside of pkg(8)? Don’t. You will only cause yourself pain, and your pleas for assistance will be met either with derisive laughter or suggestions to blow away all your packages and start over.

There are a couple circumstances where pkg(8) supports altering the package database, though. That’s when you can use pkg set. The pkg-set(8) subcommand lets you correctly adjust a few sensible values within the database without corrupting the data. The most common is when you want to make an automatic package no longer automatic.

The -A flag to pkg set lets you change a package’s automatic setting. Setting this flag to 1 means that the package was installed automatically, as a dependency, while a 0 means that the package was specifically requested by the user.

In the previous section, the list of four packages to be deleted by pkg autoremove included Python. I want to keep Python—not just this time, but any time I perform autoremovery in the future. The simple way to do that is to change Python from automatic to nonautomatic.

# pkg set -A 0 python27
Mark python27-2.7.13_1 as not automatically installed? [Y/n]: y

Python is now a nonautomatic package. The results of pkg autoremove now look different.

# pkg autoremove -n
--snip--
Installed packages to be REMOVED:
        rtmpdump-2.4.20151223
        librtmp-2.4.20151223
--snip--

Only two packages instead of four? Apparently Python needs readline. I’m glad that pkg figured that out for me because I can’t be bothered to remember it.

We’ll cover pkg set more as needed.

Locking Packages

Some software is like a subway’s electrified rail. Touching it causes suffering or death.

My favorite example is the remote file synchronization program rsync(8). Rsync has been around for decades, and its internal protocol has changed over time. Many embedded and legacy systems use rsync, but it’s never been upgraded. I’ve spent many painful hours debugging why a current rsync can’t communicate with that on a 20th-century embedded phone switch controller. It turned out that an rsync point release dropped support for the very old protocol supported by the phone switch. Upgrading the phone switch wasn’t possible, so I needed the rsync package on my host to never upgrade. Never.

That’s where locking packages comes in.

When you lock a package, pkg won’t upgrade, downgrade, uninstall, or reinstall it. It applies the same rules to the package’s dependencies and the programs it depends on. The host responsible for fetching the phone switch files needed to have its rsync package locked. Use pkg lock to lock a package.

# pkg lock rsync
rsync-3.1.2_6: lock this package? [Y/n]: y
Locking rsync-3.1.2_6

This package is now nailed in place.

To show all the locked packages on the system, use the -l flag. This shows only the packages you’ve deliberately locked, not the dependents or dependencies.

# pkg lock -l
Currently locked packages:
rsync-3.1.2_6

Use the pkg unlock command to remove the lock.

# pkg unlock rsync
rsync-3.1.2_6: unlock this package? [Y/n]: y
Unlocking rsync-3.1.2_6

To lock or unlock all packages on the system, use the -a flag. You’ll get a confirmation prompt for every package, so if you really want to affect all the packages, add the -y flag.

# pkg unlock -a
apache24-2.4.25_1: already unlocked
apr-1.5.2.1.5.4_2: already unlocked
--snip--
rsync-3.1.2_6: unlock this package? [Y/n]: y
Unlocking rsync-3.1.2_6
--snip--

Package locking doesn’t prevent someone with root access from mucking with the files contained in a package.

On a related note, Chapter 22 covers using jails to contain really old software.

Package Files

Files installed by a package are considered system files, and you shouldn’t manually edit them. Before you can edit those files, you must know what files came with the package. Use pkg info -l and the package name to see the complete list. (It’s also available as pkg list, thanks to a pkg.conf alias.)

# pkg info -l rsync
rsync-3.1.2_6:
        /usr/local/bin/rsync
        /usr/local/etc/rc.d/rsyncd
        /usr/local/etc/rsync/rsyncd.conf.sample
--snip--

Another possibility is that you want to know which package a file came from. Use the pkg which command. I normally use this when I’ve found a weird library and want to know where it came from.

# pkg which libp11-kit.so
/usr/local/lib/libp11-kit.so was installed by package p11-kit-0.23.5

My question is now, “What is p11-kit?” But that’s progress.

Package Integrity

While you shouldn’t alter package files, eventually, someone does. You can use pkg to discover those alterations and undo the damage.

The pkg-check(8) tool includes features for identifying damage to packages and package dependencies. Developers can also use pkg-check(8) to check the bundled packages built from ports and distributed to end users, but that’s a whole separate problem.

File Corruption

Verify that a package’s files are unaltered with pkg check -s and the package name. When my locked rsync package stops synchronizing files from the finicky remote server, one thing I verify is the package integrity.

# pkg check -s rsync
Checking rsync:   0%
rsync-3.1.2_6: checksum mismatch for /usr/local/bin/rsync
Checking rsync: 100%

Either the disk is failing or someone has mucked with my rsync(1) binary. As this system uses self-healing ZFS, there’s gonna be a paddling.

You could uninstall and reinstall the package, but that might trigger changes depending on which packages require the package you’re updating. Also, as discussed earlier, this particular package is special. I don’t want pkg to upgrade the package to the newest version. Instead, I want to force pkg to reinstall the current package from the package cache. Use the -f flag to pkg install. While it updates the repository database, it reinstalls the cached package. If the package is locked, you must unlock it first.

# pkg unlock -y rsync
Unlocking rsync-3.1.2_6
# pkg install -fy rsync
--snip--
[1/1] Reinstalling rsync-3.1.2_6...
[1/1] Extracting rsync-3.1.2_6: 100%
# pkg lock -y rsync

My precious rsync is restored.

Check the integrity of all your packages by running pkg check -saq. It produces no output unless something has changed, so you could schedule it via cron (see Chapter 20).

Dependency Problems

If someone really tries, they can delete packages that other packages depend on. Use the -d flag of pkg check to identify and fix missing dependencies.

# pkg check -d emacs-nox11
Checking emacs-nox11: 100%
emacs-nox11 has a missing dependency: gnutls
emacs-nox11 is missing a required shared library: ➊libgnutls.so.30
--snip--
>>> Try to fix the missing dependencies? [Y/n]: y

The first thing to note is that when pkg check identifies a missing dependency, it tries to correct it. Answer y at the prompt to reinstall the dependency.

Note that this pkg check run shows us a missing library, libgnutls.so.30 ➊. The dependency check doesn’t actually search for all the files in all of the packages. It knows that this library is missing only because the package that includes it is gone. If you manually remove the library, the dependency check won’t find it. You need to check package file integrity, as earlier.

If you want to check all package dependencies with pkg check -d, don’t give it a package name. You could add -a to explicitly check all packages, but that’s not necessary. If you add the -q flag, this command produces output only when it finds a problem. Adding -q also tells pkg check to attempt to resolve any dependency problems it finds, without user intervention.

The combination means that while I can run this check as a scheduled job, I’m less comfortable with my host reinstalling a missing dependency. Think about your system installing packages without your attention before automating dependency corrections.

The pkg check subcommand includes several other useful options, such as -B to rebuild shared library dependencies and -r to manually recompute the checksum of an installed package. Read pkg-check(8) for details.

Package Maintenance

The package system includes several maintenance scripts intended to be run from periodic(8). Enable these in /etc/periodic.conf, as discussed in Chapter 20. Each adds to the daily, weekly, or security status emails.

To have the daily maintenance check package checksums and replace damaged packages, as with pkg check -saq, set daily_status_security_pkg_checksum_enable to YES.

To determine whether installed packages have security vulnerabilities published in the FreeBSD package security system, as discussed in Chapter 19, set daily_status_security_pkgaudit_enable to YES.

If you want FreeBSD to back up the installed packages and the package database every day, set daily_backup_pkg_enable to YES.

To be notified of changes in the installed packages, set daily_status_pkg_changes_enable to YES.

Finally, you can check for obsolete packages each week by setting weekly_status_pkg_enable to YES.

Package Networking and Environment

FreeBSD’s package system is designed to work for a normal network attached to the internet. That’s something of a cruel joke because no network is normal. You can adjust pkg’s behavior to fit your network.

The most common change is the need for a proxy server. Pkg uses fetch(3) to download package files, which takes any special networking configuration through environment variables. Set environment variables in the PKG_ENV section of pkg.conf. Each variable needs the variable name, a colon, and the value. Here, I set the HTTP_PROXY environment variable to my network proxy:

pkg_env : {
  HTTP_PROXY: "http://proxy.mwl.io/"
}

See fetch(3) for the complete list of proxy environment settings.

Some networks have separate bandwidth for different network stacks. I’ve been on more than one network that has better IPv6 connectivity than IPv4, or the other way. Direct pkg to use one network protocol or the other with the IP_VERSION setting in pkg.conf. You can set this to 4, 6, or let the host autoselect with the default of 0.

Finally, the pkg.conf NAMESERVER setting lets you override the nameservers given in /etc/resolv.conf. Put an IPv4 or IPv6 address here. You can use a hostname here, but pkg will look up that hostname using the default system nameservers.

Package Repositories

You might want to use packages other than those provided by the FreeBSD Project. Maybe you build your own packages, as discussed in Chapter 16. Perhaps you have access to an experimental package repository. Or maybe you want to switch which set of official packages you’re using.

Pkg supports package repositories, or repos, which are named collections of packages. You can add, remove, enable, and disable repositories.

Normal repositories are very simple, but in rare cases, they can get quite complicated. We won’t go into the edge cases of configuring your own repositories, but the basics will take you quite far.

Repository Configuration

Configure each repository in its own file. Official FreeBSD repositories belong in /etc/pkg. Configure repositories in UCL format (see Chapter 2). FreeBSD ships with the repo FreeBSD enabled. You’ll find the configuration file in /etc/pkg/FreeBSD.conf.

   FreeBSD: {
➊ url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
➋ mirror_type: "srv",
➌ signature_type: "fingerprints",
➍ fingerprints: "/usr/share/keys/pkg",
➎ enabled: yes
   }

This repository, named FreeBSD, supports the FreeBSD repo. When you decide to set up your own repository, give it a meaningful name.

The mirror_type entry ➋ tells pkg whether this repository is hosted on a normal website. Setting mirror_type to NONE tells pkg to use fetch to get packages using the standard network methods, like HTTP, FTP, or even a file path.

Millions of machines run FreeBSD and need access to the package repository. A single web server can’t keep up. By setting mirror_type to srv, you tell pkg to check DNS for an SRV record. SRV records are used to direct high-availability services, like VoIP and Active Directory.

The url entry ➊ shows the internet site where this repository can be found. I’m sure you’ve seen http URLs before, but what about this pkg+http thing? It ties the request to the SRV record used to direct pkg requests, as set by mirror_type.

The package system can verify downloaded packages with public keys or cryptographic hash fingerprints ➌. You’ll need to tell pkg where to find the keys or hashes, though ➍.

Finally, you must explicitly enable or disable ➎ each repository.

Repository Customization

You can add and remove repositories as needed. As /etc/pkg is reserved for official FreeBSD repositories, you’ll need another directory. The traditional location is /usr/local/etc/pkg/repos. If you want to use a different directory, you’ll need to set a location in pkg.conf with the REPO_DIRS option. You’ll see commented-out examples for the defaults.

#REPOS_DIR [
#    "/etc/pkg/",
#    "/usr/local/etc/pkg/repos/",
#]

The local repository directory doesn’t exist by default, so you’ll need to create it.

# mkdir -p /usr/local/pkg/repos

Put your own repository configurations in that directory.

FreeBSD searches for packages in directory order, checking directories in the order given in REPOS_DIR. The obvious implication is that the default FreeBSD repo can’t be disabled or overridden. That’s not quite true, but the reason is a little tricky.

Repository Inheritance

You can split a repository’s configuration between multiple files. Entries in later files overwrite the entries in earlier files.

To see how this works, consider the default repository, called FreeBSD. If you have all of your custom repositories configured in /usr/local/etc/pkg/repos, pkg finds the FreeBSD repo first.

But now create a /usr/local/etc/pkg/repos/FreeBSD.conf file. Define the FreeBSD repo in there, but include only a single configuration statement.

FreeBSD: { enabled: no }

Pkg finds the repo named FreeBSD first in /etc/pkg/FreeBSD.conf. This configuration defines the defaults for this repo. It finds the second configuration later. The second configuration overrides only one option, but that option turns off the repository.

While disabling the FreeBSD repository is an extreme case for folks who don’t run their own repository, there’s good reason to make minor changes to the repo, as we’ll see next.

Package Branches

FreeBSD’s packages are built from the Ports Collection (see Chapter 16). The Ports Collection attempts to bring tens of thousands of different software suites to FreeBSD. These different programs all have their own release schedules, and the Ports Collection evolves continuously in an effort to keep up with them. As you can imagine, the Ports Collection has a whole bunch of churn. Most of us who run servers want stability. When most sysadmins consider “stability,” the word churn isn’t what comes to mind.

Most of us don’t need the very latest software on our servers. Most of the time, I’m fine if my database server is a minor point release or two behind; I care only that it keeps working. I’m certainly not going to upgrade my servers just because MySQL or nginx or PHP has a new software version. That way lies the madness of constant upgrades.

I do want security and stability updates, however. The database server being a little older doesn’t bother me. The database server occasionally losing its brain and sending all my data to the bit bucket, or offering everything to a Detroit hacking crew, bothers me a whole bunch.

The FreeBSD package system’s quarterly branches try to strike a middle ground between the world’s constant churning software and a sysadmin’s peace of mind. Every January, April, July, and October, the Ports crew forks the Ports Collection into a quarterly branch. The quarterly branch receives only security and stability updates, while the main Ports Collection charges merrily ahead.

The FreeBSD Project builds two sets of packages for each release. The quarterly packages are built from the quarterly Ports Collection. The latest packages are built from the bleeding-edge packages.

Some of you prefer the most current packages, despite the churn. That’s okay. Switching is simple. You need override only one entry in the FreeBSD repository. Create a new repository file, /usr/local/etc/pkg/repos/FreeBSD.conf, just as in the last section. Rather than disabling the default repository, though, we’re going to override the package source. Change the “quarterly” and the end of the URL to “latest.”

FreeBSD: { url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest" }

Welcome to the churn!

It’s strongly recommended to run pkg update -f after changing repositories in order to force pkg to download the latest repository catalogs.

Switching package collections doesn’t necessarily mean you need to reinstall everything. If your old packages work, use them. If weird problems appear, though, reinstall all of your packages with a command like pkg upgrade -fa. Even packages that have the same version as those in the other package collection might be subtly different.

Upgrading Packages

As much as we might wish it were otherwise, you can’t set up a new system and ignore it. Either stability bugs appear or some clever jerk figures out a security exploit. (Chapter 19 discusses auditing package security.) Sometimes you must upgrade your third-party software. With FreeBSD’s original packaging system, pkg_add, package upgrades risked a certain degree of heartache. With pkg(8), you still risk heartache—but it’s from the newer versions of the software, not the packaging system itself.

Before upgrading packages, back up your system. Then, use the upgrade subcommand to have pkg(8) upgrade all your packages. I recommend running a dry run first, with -n.

# pkg upgrade -n
--snip--
Checking for upgrades (2 candidates): 100%
Processing candidates (2 candidates): 100%
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        ca_root_nss: 3.29.1 -> 3.29.3

Number of packages to be upgraded: 1

335 KiB to be downloaded.

Carefully look at the list of packages to be upgraded. Are there any large jumps? Do you need to look at any release notes? How intrusive is this likely to be? Does the upgrade remove any packages that you want, like X.org or your text editor? Should you wait until Sunday at 3 AM and have your flunky do it?³ Studying the upgrade and considering the risks of each package upgrade might not reduce the amount of work you need to do, but it will reduce the amount of downtime and the amount of time people yell at you.

Once you’re comfortable with what will change, run the upgrade.

# pkg upgrade -y

You’ll see very similar messages about the packages to be upgraded and then notifications of the download and install process. Finally, pkg displays the installation message for every upgraded package.

Even the world’s most flexible packaging system won’t always meet your needs. FreeBSD makes customizing add-on software very easy through the Ports Collection, where we’ll go next.