16
CUSTOMIZING SOFTWARE WITH PORTS
Packages provide the most common configurations of the most desirable programs. If you’re building a generic web server, chances are that the official FreeBSD package of nginx or lighttpd or whatever your preferred web server is will suffice. If you have a special environment or less common needs, that’s where the Ports Collection comes in. The Ports Collection is a tool for easily building customized versions of many software packages. It combines dependency, licensing, maintainer, and all other software information in a standard machine- and human-readable format. Ports let you set system options like “forbid third-party GPL-licensed code” (useful for embedded vendors), “add LDAP to everything,” or “disable X11.”
In the long term, ports are best managed with the poudriere package-building system. Before you can use poudriere, though, you must understand how ports work. I’d encourage you to explore ports on a test system. Rather than deploying ports on your individual servers, though, use poudriere to build your own package repository. Manage your servers entirely with packages. Never use the ports tree on a production server other than your package builder.
Before we dive into ports, let’s talk about building software in general.
Making Software
Traditional software building is complicated because source code must be processed very specifically to create a workable, running program—let alone one that works well! It’s a completely different process than using, say, a JavaScript compiler. While programmers could include installation instructions with each program, full of lines like Now type ar cru .libs/lib20_zlib_plugin.a istream-zlib.o zlib-plugin.o, this would be downright sadistic. While Unix admins might seem to approve of sadism, they categorically disapprove of cruelty directed at themselves; if something can be automated, it will be.
The main tool for building software is make(1). When run, make looks in the current directory for a file called Makefile, which is full of instructions much like that horrid example in the previous paragraph. It reads the instructions and carries them out, automating the installation process no matter how complicated it might be. You don’t really have to know the internals of a Makefile, so we’re not going to dissect one.
Each Makefile includes one or more targets, or sets of instructions to carry out. For example, typing make install tells make(1) to check the Makefile for a target called install and, if found, execute it. A target’s name usually relates to the process to be carried out, so you can safely assume that make install installs the software. You’ll find targets to install, configure, and uninstall most software. make(1) handles a huge variety of functions, some of which far outstrip the creators’ original intents. But that’s part of the fun of Unix!
Source Code and Software
Source code is the human-readable instructions for building the actual machine code that makes up a runnable program. You’ve probably been exposed to source code in some form. If you’ve never seen it, take a look at a few files under /usr/src or at https://svnweb.freebsd.org/. Even a neophyte sysadmin needs to recognize source code two tries out of three.
Once you have source code for a program, you build (or compile) the program on the type of system you want to run it on. (Building software for a foreign platform via cross-compiling demands is more complicated when it’s possible.) If the program was written for an operating system that’s sufficiently similar to the platform you’re building it on, it works. If your platform is too different from the original, it fails. Once you’ve successfully built the software on your system, you can copy the resulting program (or binary) to other identical systems, and it should run.
Some programs are sufficiently well written that they can be compiled on many different platforms. A few programs specifically include support for widely divergent platforms; for example, the Apache web server can be compiled on both Windows and Unix-like systems. This represents heroic effort by the software authors, and even so, you must run a few scripts and configure your environment precisely by the directions before building on Windows.
Generally speaking, if you can build a program from source, it will probably run. It might not run correctly, it might not do anything you expected, but it runs. A sufficiently experienced sysadmin can use the source code and error messages to learn why a program won’t build or run. In many cases, the problem is simple and can be fixed with minimal effort. This is one reason why access to source code is important.
Back when every sysadmin was a programmer, debugging software absorbed a major part of the admin’s time. Every Unix-like system was slightly different, so every sysadmin had to understand his platform, the platform the software was designed for, and the differences between the two before he could hope to make a piece of code run. The duplication of effort was truly horrendous.
Over the years, programmers developed tools such as autoconf to help address these cross-platform issues. Not every program used these tools, and when they broke, the sysadmin was kicked back to square one. Sysadmins had to edit the source code and Makefiles just to have a chance of making the programs work. And working isn’t nearly the same as working well, let alone working correctly.
The FreeBSD Ports Collection was designed to simplify this process for FreeBSD users.
The Ports Collection
The Ports Collection, also called the ports tree or simply ports, contains an automated system for compiling software on FreeBSD.
The basic idea behind the ports system is that if source code must be modified to run on FreeBSD, the modifications should be automated. If you need other software to build this program from source code or to run the software, those dependencies should be documented and tracked. If you’re going to automate the changes, you might as well record what the program includes so you can easily install and uninstall it. And since you have a software-building process that produces exactly the same result each time, and you’ve recorded everything that the process creates, you can copy the binaries and install them on any similar system.
In addition to the information needed to create the packages, the Ports Collection contains legal restrictions on building the software, security information, licensing details, and more.
Ports interoperate with packages. The Ports Collection is used to create packages. You can install some software from ports and some from packages as you need, freely mixing where you install software from. You’ll need to use the same version of the Ports Collection used to build your packages, either a quarterly branch or the latest version. Most ports users want the latest software, so we’ll focus on that.
Ports
A port is a set of instructions on how to apply fixes to, or patch, a set of source code files and then build and install those files. A port contains a complete record of everything necessary to create the finished software. This frees sysadmins from struggling to install programs and lets them struggle to configure them.
Ports Tree Installation
If you followed the installation instructions in Chapter 3, you installed the ports tree in /usr/ports. In that directory, you should find several files and a couple dozen directories. If you don’t have anything in /usr/ports, you apparently can’t follow instructions. That’s okay—I can’t either—but you must install the ports tree to continue.
FreeBSD supports a couple different ways to get the ports tree. You can check it out using svn(1) or download a copy off the web. The recommended method for sysadmins is to use portsnap(8) to download the latest (nonquarterly) version of the ports tree.
# portsnap auto
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
Fetching snapshot tag from your-org.portsnap.freebsd.org... done.
Fetching snapshot metadata... done.
Updating from Mon Oct 17 15:59:41 EDT 2018 to Mon Mar 20 14:13:53 EDT 2019.
Fetching 5 metadata patches... done.
Applying metadata patches... done.
Fetching 5 metadata files... done.
Fetching 10202 patches.
(700/10202) 6.86% .........
Here, portsnap searches for a mirror of the portsnap files, cryptographically verifies the integrity of those files on the portsnap server, downloads the files, and verifies the integrity of the download itself.
You now have all the latest versions of all FreeBSD ports. To update an existing Ports Tree to the latest version, run portsnap auto again.
If you wish to schedule a regular portsnap update run via cron(1), use the portsnap cron update command instead of portsnap auto. This tells portsnap to update the ports tree at some random time within 60 minutes of the command running. This helps distribute the load on the FreeBSD portsnap server. Schedule a portsnap run at some point between 5 AM and 5:59:59 AM in root’s crontab with an entry like this:
0 5 * * * /usr/sbin/portsnap cron update
This kicks off the actual update at a random time between 5 AM and 6 AM, which is much more effective than 1 out of 24 portsnap users hitting the download server simultaneously at 5 AM.
Ports Tree Contents
Most of the directories you see here are software categories. Each category contains a further layer of directories, and each of those directories is a piece of software. FreeBSD has over 28,000 ports as I write this, so using the directory tree and categorizing software properly is vital. Of the files and directories in this category that aren’t software categories, the major ones are described here.
The CHANGES file lists changes made to the FreeBSD ports infrastructure. It’s primarily of use to the FreeBSD ports developers and people interested in the internals of the Ports Collection.
The CONTRIBUTING.md file exists for FreeBSD source code mirrors on GitHub. All FreeBSD source code is mirrored on GitHub for people’s convenience, but FreeBSD doesn’t use Git internally. GitHub users traditionally check CONTRIBUTING.md for information on how to contribute—which, in FreeBSD’s case, is “go to the FreeBSD website.” (Work on automatically feeding GitHub pull requests into the FreeBSD PR system is ongoing as I write this.)
COPYRIGHT contains the licensing information for the Ports Collection as a whole. While each individual piece of software supported by the Ports Collection has its own copyright and licensing information, the Ports Collection is licensed under the two-clause BSD license.
The GIDs file contains a list of all the group IDs used by software in the Ports Collection. Many pieces of software in the collection expect to run as an unprivileged user. If each port gets to create a random user, the usernames, user IDs, and group IDs will overlap. Instead, ports that need an unprivileged GID reserve one in this file. This file records GIDs assigned to Ports Collection. GIDs aren’t actually assigned in /etc/passwd until used.
Your /usr/ports has an INDEX file with a suffix named after the version of FreeBSD you’re running. This FreeBSD 12 system has /usr/ports/INDEX-12. The ports system’s search and description features use this index. The index is generated locally and not stored in Subversion.
The Keywords directory contains information for the Universal Configuration Language system, discussed in Chapter 23.
LEGAL describes the legal restrictions on any software in the Ports Collection. Some pieces of software have specific limitations on them—such as no commercial use, no redistribution, no monetary gain, and so on. Individual ports also list these restrictions; this is just a master list built from all the ports.
MOVED lists all the ports that have been renamed, moved from one category to another, or removed, along with the reason why. Automated management tools such as portmaster(8) use this list to find the new home of moved ports. Why move a port? When I started with FreeBSD, it had one category for X Windows software. The category grew ridiculously huge, so the ports team split it, and split it again, until we reached 2017’s nine categories.
The Makefile contains high-level instructions for the whole Ports Collection. You’ll only use this if you want to build every port in the entire Ports Collection. You’d be better off using poudriere as discussed in “Private Package Repositories” on page 381 than just running make here.
The Mk subdirectory contains the logic that drives make(1) in fetching source files from the internet, patching them, building them, and installing them. Many types of programs expect to integrate together, and these files ensure that different parts of the same tool are built and installed in a compatible manner. Some features, like LDAP and Emacs, can touch many ports. This directory contains Makefiles like bsd.ldap.mk and bsd.emacs.mk for exactly these functions.
Beneath the Mk subdirectory, you’ll find Uses. This directory contains broadly used Makefiles for other widely used functions or software suites. For example, the KDE and GNOME desktop suites include dozens or hundreds of smaller programs, and each must be built correctly to interoperate. If you look in Uses, you’ll see the files gnome.mk and kde.mk dedicated to configuration of these programs, as well as files for GSSAPI, Lua, Varnish, and many other software families.
If you really want to learn how the Ports Collection works, read everything in /usr/ports/Mk and /usr/ports/Mk/Uses. It’s highly educational, even though the nature of supporting all these different programs means the Makefiles are as tangled as a yarn basket attacked by a horde of crazed kittens.
The README file contains a high-level introduction to the Ports Collection.
The Templates directory contains skeleton files used by other portions of the Ports Collection.
The Tools directory contains programs, scripts, and other automation, mostly used by ports developers.
The UIDs file contains unprivileged user IDs used by ports in the system. Much like the GIDs file, this helps the ports developers avoid conflicts between unprivileged users required by ported software.
UPDATING contains notes for use when upgrading your software. Updates that require special intervention appear here in reverse date order. Before updating your software, check this file for important notes that affect you.
The distfiles directory contains the original source code for ported software. When a port downloads a chunk of source code, that source code is kept under /usr/ports/distfiles.
All the other directories are categories of ports. The following shows the contents of the ports/arabic directory, where software specific to the Arabic language is kept. Much software elsewhere in the Ports Collection supports Arabic, but this category is for software focused on Arabic—such as fonts, translations of certain types of documents, and so on. This category isn’t useful for most people, but it has the serious advantage of being small enough to fit in this book. Some ports categories have hundreds of entries.1
Makefile ae_fonts_ttf kacst_fonts libitl
Makefile.inc arabtex kde4-l10n libreoffice
ae_fonts_mono aspell khotot
This Makefile contains instructions for all the ports in the directory. They’re more specific than the global Makefile in /usr/ports, but not as specific as individual port Makefiles. The file Makefile.inc contains meta-instructions for the ports in this directory. All the other directories are individual software packages. We’ll dissect one of those directories in “Installing a Port” on page 371.
Individual ports are often called by their directory in the Ports Collection. The gnuplot graphing program might be called math/gnuplot, as its port can be found at /usr/ports/math/gnuplot.
The Ports Index
The ports index file contains a list of all ports that build on a particular FreeBSD release. On FreeBSD 13, this is /usr/ports/INDEX-13. The Ports Collection uses the index for several purposes, including searching the whole ports tree.
The index file describes each port on a single line, with fields separated by pipe symbols (|). While this is convenient for system tools, it’s not particularly human-readable. Run make print-index in /usr/ports to get a longer, much more intelligible index. This index is filled with entries like this:
Port: p5-Archive-Extract-0.80
Path: /usr/ports/archivers/p5-Archive-Extract
Info: Generic archive extracting mechanism
Maint: [email protected]
Index: archivers perl5
B-deps: perl5-5.24.1
R-deps: perl5-5.24.1
E-deps:
P-deps:
F-deps:
WWW: http://search.cpan.org/dist/Archive-Extract/
The index starts with the port’s name and the full path to the port directory. Info gives a very brief description of the port. The Maint heading lists the port’s maintainer, a person or team who has assumed responsibility for this software’s integration into the Ports Collection. The Index space lists every category where this port might be filed. The first category listed is the directory where it appears in the Ports Collection. In this case, the port appears in the archivers directory.
We then have dependencies. B-deps lists the build dependencies—that is, other software that must be installed to build this port. R-deps lists runtime dependencies, software needed for this to actually run. This is a Perl module, so it needs a Perl interpreter. Some software must be extracted or decompressed by particular tools, specified in E-deps. The P-deps field lists any dependencies for patching the software—rare pieces of software must be patched with a certain tool. The F-deps field is similar, specifying fetch dependencies—that is, any special software that must be used to download the software.
Finally, the WWW space gives the home page of the software.
Searching the Index
The Ports Collection includes tools to search the index. If you want a particular program, you might be better off finding the ports directory with pkg search or even locate -i. Reserve searching the Ports Collection to answer questions like “What ports use SNMP?”
If you know the name of a piece of software, search INDEX for it with make search. Here, I look for ports with names that include net-snmp:
# cd /usr/ports
# make search name=net-snmp
➊ Port: net-snmp-5.7.3_12
Path: /usr/ports/net-mgmt/net-snmp
Info: Extendable SNMP implementation
Maint: [email protected]
B-deps: perl5-5.24.1
R-deps: perl5-5.24.1
WWW: http://net-snmp.sourceforge.net/
Port: p5-Net-SNMP-6.0.1_1rt: p5-Net-SNMP-365-3.65
--snip--
As of this writing, FreeBSD has several ports with net-snmp in their name. The first is the current standard net-snmp software collection ➊. Others include Perl libraries that use SNMP over the network but otherwise have nothing to do with the net-snmp suite, old versions of net-snmp that are no longer supported, and Tcl/Tk interfaces to net-snmp. The fields in the description are taken straight from the INDEX file.
If you don’t need this much detail, try make quicksearch to get only the port, path, info, and (if applicable) notes on reasons why it’s not there anymore.
Key Searches
You can also search using any of the fields in the port description as a key. Remove any hyphens from the key name. You want all the ports that have a runtime dependency on Perl?
# make quicksearch rdeps=perl5
You can combine multiple search terms in one query. Suppose you want all the programs with Apache in the name but with a runtime dependency on Python.
# make quicksearch name=apache rdeps=python
Exclude a word from the search results by putting an x in front of the key. Here, we look for everything that has a runtime dependency on Python but doesn’t have Apache in the name:
# make quicksearch xname=apache rdeps=python
These by-field searches don’t work for all software, however. For example, if you’re looking for the Midnight Commander file manager, you might search for it by name.
# make search name=midnight
#
Well, that was less than helpful. Search all the fields for a match with the term key.
This scans more fields and returns more hits. If you’re searching for a common word, however, the key search can provide far too much information. Trim the output with quicksearch.
# make quicksearch key=midnight
This returns every port with the string midnight in its description, name, or dependencies. We’ll quickly learn that Midnight Commander can be found under /usr/ports/misc/mc.
Other Ways to Browse the Ports Collection
If you prefer using a web browser, build an HTML index. Just go to /usr/ports and, as root, type make readmes to generate a README.html file with the index of your ports tree and a HTML file in every port. You can click through various categories and even view detailed descriptions of every port.
If none of these options work, try the FreeBSD Ports Tree search at http://www.freebsd.org/cgi/ports.cgi. Also, the FreshPorts search engine at http://www.freshports.org/ provides a separate but very nice search function.
Between the web browser and the search engine, you should be able to find a piece of software to meet your needs. Finding the port you need might well be the most difficult part of working with ports.
Legal Restrictions
While most of the software in the Ports Collection is free for any use, some of it has a more restrictive license. The /usr/ports/LEGAL file lists legal restrictions on the contents of the Ports Collection. The most common restriction is a prohibition on redistribution; the FreeBSD Project doesn’t include such software on its FTP sites or on a CD image but provides instructions on how to build it.
Legal restrictions appear in places you might not expect. You can’t download a compiled, ready-to-go package for Oracle Java, and the FreeBSD Project can’t redistribute the Java source code. FreeBSD can and does distribute instructions on how to build the Oracle Java source code on FreeBSD, but the user must go to the Oracle site and download the code themselves. Fortunately, OpenJDK has supplanted Oracle Java for most software, and FreeBSD has a high-quality package for it.
Similarly, some pieces of software prohibit commercial use or embedding in commercial products. A few cannot be exported from the United States, thanks to Department of Commerce rules restricting the export of cryptography.2 If you’re building FreeBSD systems for redistribution, export, or commercial use, you need to check this file.
Fortunately, most of the software in the Ports Collection is free for either commercial or noncommercial use. These restricted packages are the exception, not the norm.
What’s In a Port?
Installing software from ports takes longer than using packages, and the Ports Collection requires a live internet connection. In exchange, the Ports Collection can produce more optimal results than packages. Let’s take a look at a port. Here’s the innards of dns/bind911, version 9.11 of the ISC BIND nameserver:
Makefile files pkg-help
distinfo pkg-descr pkg-plist
The Makefile contains the basic instructions for building the port. If you read this file, you’ll quickly find that it’s only a few hundred lines long. That’s not a huge amount of instructions for such a complicated piece of software, and most Makefiles are much shorter. Most of that file is dedicated to customizations that are only rarely used. There’s almost no information about BIND itself in here and not much about how to build software on FreeBSD. Most of the FreeBSD ports system’s Makefiles are in /usr/ports/Mk.
The distinfo file contains checksums for the various files the port downloads so that your system can be sure that the file transferred without error and that nobody tampered with the file before you got it.
The files directory contains all the add-on files and patches required to build this port on FreeBSD. BIND 9.11 takes a dozen patches. Most of these patches aren’t required for building, as the ISC supports their DNS servers on FreeBSD. They provide integration only into the FreeBSD package system.
The file pkg-descr contains a lengthy description of the software.
A few ports include a pkg-help file that offers additional details on how to use the port.
Some ports (not this one) have a pkg-message file that contains a template used to create the package’s installation message.
Finally, the pkg-plist file is a list of all the files installed (the “packing list”). The port installs only the files listed in the packing list. Some ports (such as Python-related ones) use an automatically generated packing list, so don’t be surprised if the packing list is missing.
Combined, these files comprise the tools and instructions needed to build the software.
Installing a Port
If you’re familiar with source code, you’ve probably already noticed that a port contains very little actual source code. Sure, there are patches to apply to the source code and scripts to run on the source code, but no source code for the software! You might rightly ask just how building software from source is supposed to work without source code?
When you activate a port, FreeBSD automatically downloads the appropriate source code from an included list of sites. The port then checks the downloaded code for integrity errors, extracts the code to a temporary working directory, patches it, builds it, installs everything, and records the installation in the package database. If the port has dependencies, and those dependencies are not installed, it interrupts the build of the current port to build the dependencies from source. To trigger all this, you just go into the port directory and type:
# make install
You’ll see lots of text scroll down your terminal as the port carries out its work, and you’ll get your command prompt back when it finishes.
As you grow more experienced in building from source, however, you’ll find that this all-in-one approach isn’t appropriate for every occasion. Not to worry; the Ports Collection provides the ability to take the port-building process exactly as far as you like because make install actually runs a whole series of subcommands. If you specify one of these subcommands, make(1) runs all previous commands as well as the one you specify. For example, make extract runs make config, make fetch, make checksum, make depends, and make extract. These subcommands are, in order:
make config
Many ports have optional components. Running make config lets you select which of those options you wish to support in this port. The options you select are saved in /var/db/ports for future builds of the port. These options affect how the port is built—for example, if you choose to build a program with net-snmp support, you’re adding a dependency on net-snmp. We discuss make config in more detail in “Port Customization Options” on page 373 later in this chapter.
make fetch
Once you’ve configured the port, the system searches a preconfigured list of internet sites for the program source code. The port’s Makefile might list the authoritative download site for the file, or it might use one of several authoritative lists provided by the Ports Collection. When the port finds the source code, it downloads it. The original, downloaded source code is called a distfile and is stored in /usr/ports/distfiles.
If the port requires a particular program to fetch a distfile, the port installs that program as part of make fetch.
make checksum
The make checksum step computes the distfile’s cryptographic hash and compares it to that recorded in the port’s distinfo file. Files can be corrupted in any number of ways: during download, by malicious intruders on a download site, or sheer random what-the-heck. Checksum verification detects file damage from any cause and stops the build if the files are corrupt.
This step makes no effort to determine why or how the file was corrupted. For the port’s purposes, it doesn’t matter whether the source code was corrupted during download or some malicious intruder put his backdoor code into the distfile before you downloaded it. Either way, don’t waste time building it, and certainly don’t install it!
make depends
A lot of software is built on top of other software. While FreeBSD includes make(1) and a compiler, some software can be compiled only with a particular compiler or demands a certain version of make. Maybe the distfile is distributed compressed with a rarely used algorithm. Perhaps it needs a third-party library that doesn’t come with FreeBSD. At the make depends stage, the port checks for missing dependencies and attempts to resolve them by building the ports.
Dependencies can have their own dependencies. The make depends recursively processes dependencies until the port has everything it needs to build, install, and run.
make extract
Once FreeBSD has the port distfiles, it must uncompress and extract them. Most source code is compressed with something like gzip(1), bzip(1), or xz(1), and collated with tar(1). This command creates a work subdirectory in the port and extracts the tarball there. If the port requires a particular program to extract the distfile, it will install it now.
make patch
This command applies any patches in the port to the extracted source code in the work subdirectory. If the port requires a special patch program instead of the base system’s patch(1), the port installs it now.
make configure
Next, FreeBSD checks to see whether the software has a configure script. This isn’t the same as the make config step performed by the port. If the software came with its own configure script, the port runs it. Some ports interrupt the build at this stage to prompt you for information, but most run silently.
make build
This step compiles the checked, extracted, patched, and configured software. Ports that don’t compile anything might have an empty step here. Some ports exist only to conveniently package a bunch of other ports.
make install
Finally, make install installs the software and tells the package system to record its presence.
Port Customization Options
Many software packages have extensive custom-build features. While enabling these features isn’t hard for any individual piece of software, there’s no universal method for defining them. With one piece of software, you might have to edit the original software’s Makefile; with another, you may have to offer flags to the configure script. Learning how to make these changes takes time and can be an annoyance. The FreeBSD Ports Collection offers two ways to consistently configure these options on your system.
The newer, prettier method is supported by make config. This brings up a dialog box much like those you saw when you first installed FreeBSD. For example, the popular access control system sudo (http://www.sudo.ws/) includes support for LDAP, auditing, and, most vitally, insulting the user when they enter their password incorrectly. If you go to /usr/ports/security/sudo and type make config, you’ll see a menu much like the one shown in Figure 16-1.
Use the spacebar to select options you like and the arrows and TAB key to move around. Hit ENTER over either OK or Cancel to finish. The port records your desired options in /var/db/ports/<category>_<portname>/options. When you have to rebuild or upgrade the port, the port reuses those same options unless you run make config to change them or make rmconfig to blow them away.
Figure 16-1: Port configuration
Customizing at the Command Line
Sometimes you don’t want a pretty arrow-select menu but a proper sysadmin interface of words on a command line. The Ports Collection lets you skip the menu and give all the configuration options in the make(1) command. Before you can do that, you’ll want to turn off the pretty menu. Set the environment variable BATCH=1 on the command line to turn off the menu. Here, we build the port with the default configuration, exactly as the FreeBSD package cluster does:
# make BATCH=1 install
Now that you’ve ditched the annoying menu, see what configure options the port supports. The make pretty-print-config command displays the current settings in an easily readable format. Let’s check out security/sudo.
# make pretty-print-config
+AUDIT -DISABLE_AUTH -DISABLE_ROOT_SUDO +DOCS -INSULTS -LDAP +NLS -NOARGS_
SHELL -OPIE -SSSD
Each of these represents a configuration option. Options marked with a plus are turned on, while those flagged with a minus are turned off. What do these options mean? Running make showconfig displays all the port’s options and what they do.
# make showconfig
===> The following configuration options are available for sudo-1.8.19p2:
AUDIT=on: Enable BSM audit support
DISABLE_AUTH=off: Do not require authentication by default
DISABLE_ROOT_SUDO=off: Do not allow root to run sudo
DOCS=on: Build and/or install documentation
INSULTS=off: Enable insults on failures
--snip--
While sudo supports LDAP and SSD and all sorts of complicated information sources, what I truly need is for sudo to insult the user any time he enters an incorrect password. I want the INSULTS option. Use the WITH environment variable on the command line to set the option. Option names are case-sensitive. Here, I set the option and check the configuration again:
# make WITH=INSULTS pretty-print-config
+AUDIT -DISABLE_AUTH -DISABLE_ROOT_SUDO +DOCS +INSULTS -LDAP +NLS -NOARGS_
SHELL -OPIE -SSSD
The INSULTS option is now set.
Use quotation marks to enable multiple options.
# make WITH="INSULTS LDAP" pretty-print-config
Similarly, use WITHOUT to turn off an option.
# make WITH=INSULTS WITHOUT="AUDIT NLS" pretty-print-config
If you leave the menu enabled when building the port, the make config graphical menu appears, but with your selected options set. Remember, turn the menu off with the BATCH variable.
Using Customizations Globally
You build ports to get specific features in your software. Often, you want that feature in all the ports that support it. Consider LDAP for a moment. If your enterprise uses LDAP, you probably want all of your software to use it. You’ll want LDAP to be the default.
FreeBSD stores settings used for every run of make in /etc/make.conf. Here’s where you’d enable LDAP or LibreSSL or other customizations that should appear across the system. Put any options you want applied globally in make.conf. Unlike the command line, make.conf uses the variables OPTIONS_SET and OPTIONS_UNSET.
Here, I want the options LDAP and INSULTS enabled on every port:
OPTIONS_SET=INSULTS LDAP
A make.conf setting has no effect on a port that doesn’t support the option. Many ports don’t know anything about LDAP. I don’t know whether any ports other than sudo include an optional feature to insult my users, but if the feature’s available, I need it.
Why use separate options in make.conf as opposed to the command line? Precedence. Options applied using WITH override options set using OPTIONS_SET. In this example, I’ve enabled insults globally. If for some unfathomable reason I needed a particular port not to insult the user,3 I could use WITHOUT=INSULTS on the command line when building the port to override the global default.
/etc/make.conf and Single Ports
Perhaps you want to build a particular port with a specific option, but you don’t want to specify it on the command line. Use the port category, an underscore, the port name, another underscore, and the SET variable in /etc/make.conf.
security_sudo_SET=INSULTS
While the port should cache the configuration, this would provide additional protection against fat-finger mistakes.
Setting Default Versions
FreeBSD supports dozens of port customization choices. Not all of them are sensible as port options, though. Some options must be used across the entire ports collection to be effective. The most common example is the SSL library. You can build all of your ports with the base system SSL library, and things will work fine. You can build all the ports with an external SSL library and, again, the software works. Building some ports with the base system SSL and some with a third-party SSL leads to catastrophe. The same applies to, say, different versions of the PostgreSQL database server and the Python interpreter. Different SSL libraries combined with different database server versions creates the sort of debacle I really enjoy handing off to a junior sysadmin who desperately needs an unforgettable lesson in how shared libraries work.
The Ports Collection uses the DEFAULT_VERSIONS variable to list critical software that should be used as the default. This replaces older variables like DEFAULT_MYSQL_VER and WITH_BDB_VER. The only way to get the complete list of variables is to trawl through /usr/ports/Mk/. The file bsd.default-versions.mk, bsd.apache.mk, and the files under Mk/Uses are notably useful.
Here, I’m telling the Ports Collection always to build ports with LibreSSL instead of the base system’s OpenSSL library and to use Python 3.7.
DEFAULT_VERSIONS += ssl=libressl
DEFAULT_VERSIONS += python=python3.7
I list each default version on a separate line and use the += syntax to tell the ports system to add this to the list.
I recommend setting default versions before building your first port. Otherwise, you’ll wind up rebuilding ports so that they link against your preferred libraries.
Don’t mix prebuilt packages with ports built using an alternate DEFAULT_VERSIONS. Programs built from packages will use the default libraries, while your ports will use your preferred libraries. If your system works afterword, it will be only by sheer accident.
Front-Loading Recursion
Sometimes the interactivity in building a port isn’t the problem. Recursion is the problem.
Suppose you’re building a big port, such as LibreOffice or GNOME. These ports had dozens or even hundreds of dependencies. Many of these ports require interactive configuration. Perhaps you decide to launch a KDE build before going to bed, thinking that you’ll wake up with the latest window manager or at least an amusing error message. Instead, you’ll rise to discover a dependency’s make config menu that’s been patiently awaiting your attention since 30 seconds after you walked away.
The point of building software from ports is that you can customize it. For these big builds, though, you really want to do all the customizations up front. That’s where make config-recursive comes in.
The make config-recursive walks through the tree of required ports and runs make config on each and every one of them. You’ll spend a few minutes selecting options in each port or just hitting OK on the ones you don’t care about. Once you finish the recursive config, though, you can safely run make install on the port you actually want and go off to do other things. You’ll return to an installed port or a build failure.
Changing a port’s build options can add or remove dependencies. If you decide to enable, say, SNMP support in LibreOffice,4 the port will need the proper SNMP library. The port for that library will need configuring. Re-run make config-recursive until none of your decisions change.
The ports system caches all your configuration choices. To remove that cache for a port and all its dependencies, run make rmconfig-recursive.
If bandwidth timing is the problem, you can download all the distfiles required for all the dependencies with make fetch-recursive. This is useful if you’re in a place like Antarctica, where build time and server cooling is unlimited but you have internet only a few hours a day.
Packaged Dependencies
Some software has hundreds of dependencies, and you probably don’t want to build all of them. While I might want a custom Emacs build, I probably don’t want to build gmake and the latest GNU C compiler from source. The make missing command displays missing dependencies. You can use that command to pick and choose what you want to build.
If you don’t want to build any dependencies from source but install them all from packages instead, you can feed make missing into a pkg command.
# pkg install -Ay $(make -DBATCH missing)
If a package is available, it’ll be installed. The only things you’ll need to install from ports are those available only from ports.
Port Flavors
Some ports have complicated dependencies. While you can build Ansible with Python 2 or Python 3, an Ansible package that works with Python 2 is very different than one for Python 3. Flavors is a mechanism for expressing these possibilities within a single port, and was very recently introduced into the Ports Collection. Flavors are not yet pervasive throughout the ports system, but at the time I write this, they’ve been implemented for Python, Perl, Qt, and Emacs. You can expect to see them more and more frequently.
To see if a port supports any flavors, go to the port directory and run make -V FLAVORS. Here, I see what flavors of the popular Python packaging toolkit Setuptools are available.
# cd /usr/ports/devel/py-setuptools
# make -V FLAVORS
py27 py36 py35 py34
My current ports tree supports Python 2.7, 3.6, 3.5, and 3.4.
To build Setuptools for a specific Python version, give the flavor on the command line.
# make FLAVOR=py34 install clean
If you don’t specify a flavor, the port gets built with the current default Python. To set the default Python for your system, set DEFAULT_VERSIONS in make.conf.
Building Packages
You can create a package from an installed port. You can then copy your customized port to other machines and install it.
Before creating the package, create the directory /usr/ports/packages. The ports system puts built packages in that directory. Without a packages directory, the package winds up in the port directory and you wind up with package files scattered all over your filesystem.
Use make package to create a package. If you want to package not only the current port but all its dependencies, run make package-recursive.
People who need a whole bunch of customized ports should consider setting up their own repositories with poudriere (discussed later this chapter), but one-off package builds are okay if you have special circumstances or you like saving trouble for later.
Uninstalling and Reinstalling Ports
While you can use pkg remove to uninstall a port, you can also uninstall a port from the port directory. Running make deinstall in the port directory removes the program from the system but leaves the port compiled and ready to reinstall.
After uninstalling a port, the compiled program and source files still live under the work subdirectory in the port. Running make reinstall reinstalls the compiled program. You can uninstall and reinstall as many times as you like.
Tracking Port Build Status
How does the Ports Collection keep track of what’s already been done? If you can run make extract and then make install, how does FreeBSD know what it has already finished? The Ports Collection uses hidden files (files with a name beginning with a dot), or cookies, to track completed steps. See those files by listing all the files in the port’s work directory:
# cd /usr/ports/security/sudo/work
# ls -a
ls -a
--snip--
.PLIST.flattened
.PLIST.mktmp
.PLIST.objdump
.PLIST.setuid
.PLIST.writable
.build_done.sudo._usr_local
.configure_done.sudo._usr_local
.extract_done.sudo._usr_local
.install_done.sudo._usr_local
.license-catalog.mk
--snip--
The file .configure_done.sudo._usr_local indicates that the make configure step is complete.
On more than one occasion, after multiple make install/deinstall cycles, I’ve had a port refuse to reinstall itself. That’s generally caused by the hidden file indicating that the install has finished. Remove that file, and the reinstall can proceed.
Cleaning Up Ports
Ports can take up a lot of disk space. Programs with many dependencies, like GNOME, KDE, and LibreOffice, can take dozens of gigabytes! Much of this resides in the port’s work directory, where the port puts the source code files and all the intermediate parts of the completed binaries. Once the port is installed, though, you no longer need those files.
Remove the port’s working files with make clean. This erases the work directory of the current port and all dependencies, so be sure you’re happy with your new program before doing this. You can also clean a port immediately upon install by running make install clean.
You might also want to remove the original distfiles, stored in /usr/ports/distfiles. The make distclean command removes the distfiles for the current port and all dependencies.
To clean the entire ports tree, run make clean -DNOCLEANDEPENDS directly under /usr/ports. The -DNOCLEANDEPENDS is optional, but it prevents the default recursive cleaning. Without it, you’ll clean some popular ports dozens or hundreds of times. While there are faster ways to remove every work directory in the ports tree, this one is directly supported by the FreeBSD Project.
Read-Only Ports Tree
Many people dislike having temporary files and even packages in /usr/ports. You can move the various working directories to other parts of the filesystem to keep your /usr/ports read-only except for updates.
Use the WRKDIRPREFIX option in make.conf to build ports in a separate directory. Many people will set this to a location like /usr/obj.
The PACKAGES option sets a new package directory other than /usr/ports/packages.
Finally, DISTDIR sets a location to store distfiles other than /usr/ports/distfiles.
On a related note, it’s possible to build ports and packages without being root, provided the permissions on these directories are set so that the builder can write to these directories. Only root can install software, however.
Changing the Install Path
Many environments have standards for how add-on software gets installed. I’ve been in organizations where /usr/local is reserved for files specific to that machine and software installs in that directory are forbidden. Instead, software installs must go in /opt or some other mandated location.
Set an alternate installation location with the LOCALBASE and PREFIX variables. You could do this on the command line, but if you’re complying with an organization standard, use make.conf instead. Whichever you use, start by building pkg(8) itself.
# cd /usr/ports/ports-mgmt/pkg
# make LOCALBASE=/opt PREFIX=/opt install
The port installs all of its files under this directory. For example, programs that normally go into /usr/local/bin end up in /opt/bin.
Not every port can handle changing LOCALBASE and PREFIX from /usr/local. Some software has hardcoded dependencies on /usr/local, while others have undiscovered bugs. If a port chokes on changing the install path, file a PR (see Chapter 24). Consider taking a look at the port to figure out why it choked. Submitting fixes like this is one of the easiest ways to get involved with FreeBSD.
Private Package Repositories
Packages are great, until you need customized versions; then you need ports. Similarly, ports are great until you have dozens of machines that all need customized ports. What’s easy to build on one host is difficult to maintain on several and impossible across a large server farm. When you outgrow ports, you need packages. Customized packages, that is.
The FreeBSD project uses poudriere (pronounced poo-DRE-er) for building packages. Why poudriere? It’s French for powderkeg. The successor to the “tinderbox” tool,5 poudriere is a collection of shell scripts that leverage existing FreeBSD infrastructure, such as jails and tmpfs and the Ports Collection.
Building packages that work across multiple systems is different than building software that works on the local host. Anything managed by human beings accumulates cruft. Once my desktop is more than a few months old, I’m pretty confident that some minor change I’ve made will make it subtly different than any newly installed system. Maybe I saved a shared library after an upgrade. Perhaps I installed something by hand and forgot about it. Gremlins could have tampered with the linker, I don’t know. The important thing is, my host isn’t pristinely identical to every other host running what’s supposed to be the same operating system. A port built and packaged on this host might include dependencies, libraries, or who knows what that will keep it from working on other hosts.
Poudriere evades this problem by building everything in jails it manages itself. A poudriere can build packages for any supported FreeBSD release older than the host it runs on. You can’t, say, build packages for 13.0-RELEASE on a 12.4-RELEASE host, because the kernel lacks the necessary interfaces.
With poudriere, you can build packages on one host and distribute them among all of your servers. While poudriere includes many advanced features, getting a basic repository running isn’t hard at all.
Poudriere Resources
Package building takes system resources. You can restrict how many processors poudriere uses during builds, which helps reduce its memory use. While poudriere itself is only a few megabytes, however, the jails and build environments can take up a whole lot of disk space. The official poudriere docs recommend allocating at least 4GB of disk for each jail and 3GB of disk space for the ports tree. I normally use about 1GB for each using ZFS, but I encourage you to err on the side of following the recommendations.
Poudriere leverages ZFS clones and snapshots to build jails, vastly reducing the needed disk space and, increasingly, performance. You can run poudriere on UFS, but it will use more space and run more slowly.
Of greater concern is the space needed to build the ports. My web servers run only a few dozen pieces of software, and many of these are tiny. Poudriere needs only a few gigabytes of disk to build them. If you’re building hundreds or thousands of packages, you need a whole bunch of disk. How much? Well, are you building GnuPG or are you building LibreOffice? To get an estimate, build but don’t clean all of your packages using ports, and then see how big /usr/ports gets.
Each host should use only one package repository. Yes, it’s technically possible to build your local packages and install them alongside packages from the official FreeBSD repository. The problem is that packages are interdependent. You could have your host check your repository first and then fall back to the official repository. The official repository updates every few days, however. The time between updates varies with the hardware available in the build cluster, but a few days is a good guess. Are the updates to your poudriere perfectly synchronized with the official repository’s slightly irregular updates? Is your ports tree exactly identical to the one used on the ports cluster? Packages are meant to work as an integrated collection, not a pile of stuff from two different collections. Ask any Linux administrator for their horror stories about packages installed from multiple repositories and then commit to building all your own packages. Plan your disk usage accordingly.
Finally, start by building your packages on a host of the same architecture that you intend to install them on. If you’re building packages for arm64 systems, use an arm64 host for poudriere. You can build i386 packages on amd64, but amd64 hardware is literally designed to run i386 code. Once you’re comfortable with poudriere, you can use the qemu-user-static package to cross-build packages for slow platforms.
Can you add poudriere to an existing production host? Maybe. A few poudriere runs on a test system will provide insight into the resources your environment needs.
Installing and Configuring Poudriere
Poudriere has no build options, so install it from packages.
# pkg install poudriere
Configure poudriere in /usr/local/etc. You’ll find a directory for configuring specific package builds, poudriere.d, but we’ll start with the generic configuration file, poudriere.conf. Here’s where you’ll tell poudriere how to behave. While you can customize directories and paths, we’ll stick with the defaults.
You must tell poudriere where to download FreeBSD install files from by setting the FREEBSD_HOST variable. If you don’t have a local install mirror, use the default of download.freebsd.org.
FREEBSD_HOST=https://download.FreeBSD.org
Poudriere includes ZFS-aware features. ZFS isn’t necessary for poudriere, of course, but if it’s run on ZFS, it will create, clone, and destroy datasets as needed. Running on UFS won’t hinder poudriere, but copying files is slower than cloning. If you’re using UFS, uncomment the NO_ZFS=yes configuration option. That’s it.
ZFS users need to specify the ZFS pool poudriere will use. My main operating install might be on the pool zroot, but that pool’s on a pair of flash SATADOMs that I don’t want to abuse too badly. I have a scratch pool specifically for churning data. Set ZPOOL in poudriere.conf.
ZPOOL=scratch
Before your first poudriere run, create a /usr/local/poudriere dataset. You’ll be happier.
All of poudriere’s work files get put under /usr/local/poudriere. If you’re using a separate ZFS pool, the mount points for the datasets on that pool get set to various locations under /usr/local/poudriere. On UFS, it’s a directory like any other.
My examples run on ZFS because I can. Poudriere’s output might look slightly different on UFS systems, but the commands you run are identical no matter the underlying filesystem.
We’ll look at a few poudriere customizations later, but this will get you started. Now create jails for your packages.
Poudriere Jail Creation
Poudriere can create jails from a whole bunch of different sources. You can download from a few different sources, build from a source tree, and more. Read poudriere(8) for a full list. Here, I’ll install three different jails from my three favorite methods: from the internet, from an install image, and from my custom-built /usr/src and /usr/obj. All the installation commands use the same general syntax. Some installation methods will add a new option, but everything starts with these.
# poudriere jail -c -j jailname -v version
The jail subcommand tells poudriere to work on a jail. The -c flag means create, and -j lets you assign a name to the jail. A jail can have any name that doesn’t include a period. I name my poudriere jails after the architecture and release, substituting a dash for any dots. This gives me jails like amd64-12-0, amd64-11-4, and so on. The -v flag takes one argument, the FreeBSD version from uname -r but without any patch level information. If your hosts are currently running 12.3-RELEASE-p20, just use 12.3-RELEASE. The patch level will change in subsequent poudriere runs—yes, poudriere applies security patches to jails.
Install Jail from Network
The default jail install grabs the FreeBSD software from the download site specified in poudriere.conf. FreeBSD’s main download site is geographically load balanced, so there’s no need to use any other site unless you have your own mirror. Here, I create a jail called amd64-11-1 for building 11.1 packages:
# poudriere jail -c -j amd64-11-1 -v 11.1-RELEASE
[00:00:00] ====>> Creating amd64-11-1 fs... done
[00:00:01] ====>> Using pre-distributed MANIFEST for FreeBSD 11.1-RELEASE amd64
[00:00:01] ====>> Fetching base.txz for FreeBSD 11.1-RELEASE amd64
--snip--
Poudriere goes to the website and starts downloading the distribution files. Once it has all the files locally, it copies /etc/resolv.conf into the jail and runs freebsd-update to get all the latest security patches. The poudriere run ends with:
[00:04:21] ====>> Recording filesystem state for clean... done
[00:04:21] ====>> Jail amd64-11-1 11.1-RELEASE-p1 amd64 is ready to be used
You can now configure this jail.
Install Jail from Media
Downloading from the internet is fine, but what if you have the install media locally? Why redownload what you already have sitting on an ISO or a memory stick image? Extract those distribution files onto your local hard drive and you can use them for as many jails as you need. For an ISO, use tar(1).
# tar -xf ../FreeBSD-11.0-RELEASE-amd64-disc1.iso usr/freebsd-dist
A memory stick image is slightly more complicated; sadly, libarchive can’t open disk images yet. You must attach the image to a memory device and mount it.
# mdconfig -at vnode -f FreeBSD-11.0-RELEASE-amd64-memstick.img
md0
If you try to mount /dev/md0, you’ll get an error. It’s not a filesystem; it a partitioned disk image. Identify the partitions on the disk.
# gpart show md0
=> 3 1433741 md0 GPT (700M)
3 1600 1 efi (800K)
1603 125 2 freebsd-boot (63K)
1728 1429968 3 freebsd-ufs (698M)
1431696 2048 4 freebsd-swap (1.0M)
Partition 3 is a UFS filesystem. That looks promising.6 Mount it.
# mount /dev/md0p3 /mnt
The distribution files are now available in /mnt/usr/freebsd-dist. I could copy them out or just install from their current location.
Here, I create a jail for building FreeBSD 11.0 packages. It’ll be called amd64-11-0 and use the files from the mounted memory stick. Use the -m flag to specify where poudriere should grab the files from.
# poudriere jail -c -j amd64-11-0 -v 11.0-RELEASE -m url=file:///mnt/usr/freebsd-dist/
Note that the argument to -m is a URL. I could specify a website here, but file:// is a perfectly valid type of URL. On a Unix host, a file:// URL has a third slash to indicate the filesystem root.
Install Jail from a Local Build
I run -current and regularly build from source. I want to build packages for my custom build, so the jail needs a version of FreeBSD that matches my host. The easy way to get that is to install from the same /usr/src you built the host from. (You could also use Subversion to download a fresh copy of the source code you used to build this system, but that requires understanding Subversion.) Use -m to give the location to a source directory.
# poudriere jail -c -j amd64-current -v 12.0-CURRENT -m src=/usr/src
[00:00:00] ====>> Copying /usr/src to /usr/local/poudriere/jails/amd64-current/usr/src...
--snip--
Poudriere runs make installworld on the prebuilt world in /usr/obj to create your jail. It won’t run freebsd-update because -current doesn’t support it.
We’ll use the amd64-current jail in all future examples.
Viewing Jails
To see all the jails poudriere has set up, run poudriere jail -l. The output is very wide, so I can’t reproduce it in this book, but you’ll see the jail’s name, the installed version of FreeBSD, the hardware architecture, the method used to install, the timestamp of the installation, and the path to the jail.
Install a Poudriere Ports Tree
Poudriere can use different ports trees for different builds. You might use a quarterly ports branch for one host, the current ports tree for another, and last year’s ports tree for a third. (You need to use Subversion to extract particular ports trees from the FreeBSD mirrors, so we won’t cover them.) The possibility of supporting multiple ports trees means you must assign a name to each ports tree you do install. Multiple jails can share a ports tree. The default is the current ports tree.
Use the poudriere ports subcommand for all ports-related actions. The -c flag tells poudriere to create a ports tree, and the -p flag lets you assign the name.
# poudriere ports -cp head
[00:00:00] ====>> Creating head fs... done
[00:00:00] ====>> Extracting portstree "head"...
Looking up portsnap.FreeBSD.org mirrors... 6 mirrors found.
--snip--
Poudriere leverages portsnap(8), which we discussed earlier this chapter.
If you install multiple ports trees, view them with poudriere ports -l.
Configuring Poudriere Ports
The whole point of building a port is to customize it. You don’t need to build the whole ports tree as packages, though—not unless you’re running the FreeBSD package building cluster or something analogous! You have to tell poudriere which ports to build. Once you have that list, you might need specific options for certain ports, but you might also need global options. You’d normally use /etc/make.conf to set those options, but you don’t want poudriere to use the system’s settings. Poudriere needs an isolated make.conf. Similarly, you might use make config to set up a port, but how can you do that in poudriere?
The Package List
Start by defining the list of packages you want poudriere to build. This list usually goes in the file /usr/local/etc/poudriere.d/pkglist, although you can put it anywhere you want. Specify each port by its category and directory. To build poudriere itself, use an entry like this:
ports-mgmt/poudriere
The difficult part here is establishing a base package set. You have to build all the packages the host needs. A host might need dozens or hundreds of packages. Do you really need all of those packages? How did all of those packages get on this system anyway?
Remember, you probably didn’t choose to install all of those packages. You installed an application like Emacs or Apache or LibreOffice, and that application dragged in all those dependencies. You care only about those dependencies as they affect the software you want. If LibreOffice loses a dependency, you don’t want poudriere to build that dependency anymore. Poudriere automatically builds and packages dependencies. All you need to specify is the application you want, and let poudriere do the rest.
Use pkg-query(8) to get a list of all the nonautomatically installed software on one of your production systems.
# pkg query -e '%a=0' %o
www/apache24
shells/bash
sysutils/beadm
--snip--
Use this as a base for your package list. Review it for unneeded stuff. Get a similar list from your other production hosts. Use them to assemble your repository’s package list.
Poudriere make.conf
Poudriere assembles a unique make.conf for each jail from files in /usr/local/etc/poudriere.d/. The file /usr/local/etc/poudriere.d/make.conf contains the global make.conf options that you want set for all of your jails. Other make.conf files can override those settings, as discussed in poudriere(8), but we’ll focus on per-jail make.conf files.
Suppose I want LDAP everywhere across my enterprise. Poudriere’s /usr/local/etc/poudriere.d/make.conf would contain:
OPTIONS_SET=LDAP
Hosts running my custom FreeBSD build all use LibreSSL, though. I would create a separate make.conf just for that jail, named amd64-current-make.conf. It would contain the LibreSSL configuration.
DEFAULT_VERSIONS += ssl=libressl
More specific files override general files. Settings in the per-jail files override poudriere’s global make.conf. I could turn off LDAP on this one jail even as I enable LibreSSL.
Running make config
Use poudriere options to run make config for your jail. Each combination of jail and ports tree can have its own unique port options, so you need to specify them on the command line. You must specify the jail with -j, the name of the ports tree with -p, and the package file with -f.
# poudriere options -j amd64-current -p head -f pkglist
Poudriere figures out which ports actually get built and all their dependencies. It runs you through make config for every one of them.
Take note of the options you select; should some of those go into the global or per-jail make.conf? Setting them as defaults can save you trouble in future poudriere runs.
You can now build your package repository.
Running Poudriere
The poudriere bulk subcommand builds packages in bulk. Use -j to specify the jail, -p to give the ports tree name, and -f to specify the package list file. (Yes, those are the same flags as configuring a port; it’s like the poudriere designers wanted to be consistent or something.)
# poudriere bulk -j amd64-current -p head -f pkglist
Poudriere fires up the jail, mounts all the ports, copies the various configuration files into the jail, decides what order to build stuff in, and starts building. You’ll see the name of each port as it starts building.
Some of those port builds might run quite a while. Hit CTRL-T to get the current status, or check the logs to see the current status.
At the end of the build, you’ll see the list of any ports that get built and a list of ports that failed to build. Here are the results from an itty-bitty pkglist:
[00:04:56] ====>> Built ports: ports-mgmt/pkg devel/pkgconf security/libressl
[00:04:56] ====>> Failed ports: www/obhttpd:build
The ports pkg, pkgconf, and libressl built fine. They might not run, but the ports collection could build and package them. The obhttpd package did not build, however. If this package is critical, I’ll want to fix this problem before letting my clients use this repository.
Let’s look at the problems first and then examine the repository.
Problem Ports
After the list of ports that gets built, you’ll see a message pointing out where to find the logs.
[00:04:56] ====>> Logs: /usr/local/poudriere/data/logs/bulk/amd64-current-head/2018-10-10_15h05m43s
The logs go in a directory named after the jail and the ports tree, with a subdirectory by date. If you don’t want to type out the date, there’s a convenient latest that takes you straight to the most recent log directory.
# cd /usr/local/poudriere/data/logs/bulk/amd64-current-head/latest
You won’t find only logs here; you’ll find a website. If you configure your web server to serve up /usr/local/poudriere/data, you can use a web browser to check poudriere builds (as well as to serve repositories to clients). The logs subdirectory here contains poudriere’s build logs for every port. If you don’t want to sort through those, the logs/errors subdirectory contains only the logs for the failed builds.
Now you need to do something terribly radical: read the error log. Perhaps poudriere couldn’t fetch the distfile. Maybe the host ran out of disk space. Perhaps something truly weird happened. Or, maybe, the port is actually broken with the build options you chose. Not all ports are built with all options all the time; it’s very easy for a port maintainer to miss that a rarely used function is busted. Remember, though, that poudriere is FreeBSD’s official port-building mechanism. If a port fails to build under poudriere, it’s busted and you should consider filing a bug (see Chapter 24).
Package Repository
Find your completed packages under /usr/local/poudriere/data/packages. Each combination of jail and ports tree gets its own subdirectory. I build this set of packages on the jail amd64-current using the ports tree head, so my new repository is in /usr/local/poudriere/data/packages/amd64-current-head. You’ll find the catalogs as the various .txz files and the Latest subdirectory for the most recent packages.
Congratulations. You have a private package repository. Now to get your clients to use it.
Using the Private Repository
The easiest way to use a private repository is on the poudriere host itself. Local repository configurations for pkg(8) go in /usr/local/etc/pkg/repos. That directory doesn’t exist by default, so create it.
# mkdir -p /usr/local/etc/pkg/repos
Create a FreeBSD.conf file therein. Local repository configurations augment or override the system defaults—that’s built into UCL. We need to add one setting to the default repository configuration in /etc/pkg/FreeBSD.conf.
FreeBSD: {
enabled: no
}
This leaves the file /etc/pkg/FreeBSD.conf untouched but sets enabled to no for the repository named FreeBSD. The default repository is no more.
Now create a separate configuration file for our custom repository. I’m naming this repository amd64-current, after the jail.
amd64-current: {
url: "file:///usr/local/poudriere/data/packages/amd64-current-head",
enabled: yes,
}
Your host is now ready to use those packages. You’ll want to forcibly reinstall all the current packages to stop using the FreeBSD repository’s versions and use your local versions.
# pkg install -fy
The pkg(8) program will download the repository catalog, but the download will look a little different than usual.
--snip--
Updating amd64-current repository catalogue...
Fetching meta.txz: 100% 260 B 0.3kB/s 00:01
Fetching packagesite.txz: 100% 17 KiB 17.4kB/s 00:01
Processing entries: 100%
amd64-current repository update completed. 62 packages processed.
--snip--
Compared to the official repository catalog, this catalog is pretty tiny. It extracts the catalog and metadata in one second. The last line shows that this repository has only 62 packages. You’re using the new repository. Install your custom packages!
Remote Custom Repositories
The whole point of a package repository is that you build packages once and deploy them everywhere. You could use a read-only NFS export to provide packages to your local machines, but the internet loves to abuse publicly accessible NFS servers. The pkg.conf file defines the repository location with a URL. While I used a file for the URL, there’s no reason this repository can’t use a website instead. Install a web server on your package builder, and have it offer the contents of /usr/local/poudriere/data/packages to your other servers. Then give the other hosts that should use that repo their own repository configuration.
amd64-current: {
url: "https://pkg.mwl.io/amd64-current-head",
enabled: yes,
}
All our machines now get an identical set of customized ports. This change gets my flunky Bert out of building ports on a dozen machines and into polishing my car.
All Poudrieres, Large and Small
Poudriere performs pretty well by default but has a couple options that can help on small and large systems.
Small Systems
If you have a resource-constrained host, you don’t want to let poudriere run amok. Here’s a couple poudriere.conf options to restrain it.
Generally speaking, if you can build a port on a host, poudriere can build that port. What you don’t want is for multiple simultaneous poudriere runs to overwhelm the host. Poudriere normally runs the same number of simultaneous processes as the number of processors in the host. Use the PARALLEL_JOBS option to limit the number of parallel builds.
PARALLEL_JOBS=1
Other restrictions, like reducing the amount of memory a poudriere build can use, are less useful than you might think. A piece of software takes as much memory to build as it requires. Building LibreOffice with only 1GB of RAM will not end well.
Remember that you can also globally deprioritize poudriere runs with nice(1), as discussed in Chapter 21.
Large Systems
Poudriere can take advantage of beefy systems to accelerate builds. You can’t speed up the disk, but you can take advantage of memory to use tmpfs(5) for critical parts of the build. Set the USE_TMPFS option to use memory for the working directory.
USE_TMPFS=yes
You can use tmpfs(5) for parts of the build beyond the working directory, but few of us have that much memory. Read the poudriere.conf.sample for details.
If you build many package repositories, investigate poudriere’s cache (https://ccache.samba.org/) support. You’ll use about 5GB of disk space per jail but save a whole bunch of time rebuilding packages.
Updating Poudriere
New ports get added all the time, with new options. Other software projects continually release new versions, and the FreeBSD port is correspondingly updated. You’ll want those new versions on your servers. If you build your ports with poudriere, updating is pretty simple. You’ll need to update your jail and your ports tree. Before updating either, though, make sure poudriere.conf is set up to handle updates.
Poudriere has two options for handling dependency changes. You’ll want to enable both. CHECK_CHANGED_DEPS tells poudriere not to trust earlier dependency calculations and perform those checks again. This catches changes in underlying Perl, Python, and so on. Similarly, CHECK_CHANGED_OPTIONS tells poudriere to verify each port’s options. Setting this to verbose tells poudriere to show you any changes.
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
Now you can update your jails and the ports tree. Use the -u flag to update the jail. Give the jail name with -j. Here, I update poudriere’s amd-11-1 jail.
# poudriere jail -j amd64-11-1 -u
For jails installed from official media, poudriere runs freebsd-update(8) and applies any missing security patches. If you installed from source, poudriere repeats the install process.
Similarly, update the ports tree with -u. Specify the name of the ports tree with -p.
# poudriere ports -p head -u
You’ll see poudriere use portsnap(8) to grab the latest updates. Now you can build the new version of the package repository, exactly as you did the first time.
# poudriere bulk -j amd64-current -p head -f pkglist
Poudriere determines what needs updating and what must be rebuilt and proceeds accordingly. Once the build is complete, your clients can upgrade their packages from the repository.
More Poudriere
Poudriere has many more features than what I cover here. You can cryptographically sign your packages with the PKG_REPO_SIGNING_KEY variable. Package sets let you define different build options for different repositories. You want to build an experimental package run with the latest Python? Look at package sets. You can blacklist ports so that they’re never built, even if called as a dependency. See poudriere(8) for all kinds of nifty stuff.
Between ports and poudriere, you can now customize software any way you need. If you really want to get into the nitty-gritty of the Ports Collection, check out the FreeBSD Porter’s Handbook on https://www.freebsd.org/. The rest of us will move on to some of FreeBSD’s advanced software features.