5
READ THIS BEFORE YOU BREAK SOMETHING ELSE! (BACKUP AND RECOVERY)

The most common cause of system failure is those pesky humans, but hardware and operating systems also fail. Hackers learn new ways to disrupt networks and penetrate applications, and you’ll inevitably need to upgrade and patch your system on a regular basis. (Whether or not you will upgrade and patch is an entirely separate question.) Any time you touch a system, there’s a chance you’ll make a mistake, misconfigure a vital service, or otherwise totally ruin your system. Just think of how many times you’ve patched a computer running any OS and found something behaving oddly afterward! Even small system changes can damage data. You should, therefore, always assume that the worst is about to happen. In our case, this means that if either the hardware or a human being destroys the data on your hard drive, you must be able to restore that data.

We’ll start with system backups and managing tape drives using tar(1) and then review recording system behavior with script(1). Finally, should you suffer a partial or near-total disaster, we’ll consider recovering and rebuilding with single-user mode and the install media.

System Backups

You need a system backup only if you care about your data. That isn’t as inane as it sounds. The real question is, “How much would it cost to replace my data?” A low-end tape backup system can run a few hundred dollars. How much is your time worth, and how long will it take to restore your system from the install media? If the most important data on your hard disk is your web browser’s bookmarks file, a backup system probably isn’t worth the investment. But if your server is your company’s backbone, you’ll want to take this investment very seriously.

Online backups can easily be damaged or destroyed by whatever ruins the live server. Proper backups are stored safely offline. Tools like rsync(1), and even ZFS replication, don’t create actual backups; they create convenient online copies.

A complete backup and restore operation requires a tape drive and media. You can also back up to files, across the network, or to removable media, such as CDs or DVDs. Many people use removable multiterabyte hard drives connected via USB 3 for backups. Despite our best efforts, tape is still an important medium for many environments.

Backup Tapes

FreeBSD supports SCSI and USB tape drives. SCSI drives are the fastest and most reliable. USB tape drives are not always standards-compliant and hence not always compatible with FreeBSD. Definitely check the release notes or the FreeBSD mailing list archives to confirm that your tape drive is compatible with FreeBSD.

Once you’ve physically installed your tape drive, you need to confirm that FreeBSD recognizes it. The simplest way is to check the /var/run/dmesg.boot file for sa devices (see Chapter 4). For example, the following three lines from dmesg.boot describe the SCSI tape device in this machine:

➊sa0 at mps0 bus 0 ➋scbus0 ➌target 3 lun 0
sa0: ➍<QUANTUM ULTRIUM 5 3210> Removable Sequential Access SPC-4 SCSI device
sa0: Serial Number HU1313V6JA
sa0: ➎600.000MB/s transfers
sa0: Command Queueing enabled

Of all the information we have on this tape drive, the most important is that your FreeBSD system knows this device as sa0 ➊. We also see that it’s attached to the SCSI card mps0 ➋ at SCSI ID 3 ➌, and we see the drive’s model number ➍ as well as the fact that it can run at 600MB per second ➎.

Tape Drive Device Nodes, Rewinding, and Ejecting

Tape is a linear storage medium. Each section of tape holds a particular piece of data. If you back up multiple chunks of data to tape, avoid rewinding after each backup operation. Imagine that you wrote a backup of one system to tape, rewound the tape, and backed up another system. The second backup would overwrite the first because it used the same chunk of tape. When you run multiple backups on a single tape, use the appropriate device node to ensure you don’t rewind the tape between tasks.

As with many Unix devices with decades worth of history, the way you access a tape drive controls how it behaves. Tape drives have several different device nodes, and each one makes the tape drive behave differently. The most basic tape-control mechanism is the device node used to access it. Normal tape drives have three nodes: /dev/esa0, /dev/nsa0, and /dev/sa0.

Tapes are sequential access devices, and to access certain data on a particular section of tape, you must roll the tape back to expose that section. To rewind or not to rewind is an important question.

If you use the node name that matches the device name, the tape drive will automatically rewind when your command finishes. Our sample SCSI tape drive has a device name of sa0, so if you run a command using /dev/sa0 as the device node, the tape will rewind when the command finishes.

If you don’t want the tape to automatically rewind when the command completes, stop it from rewinding by using the node name that starts with n. Perhaps you need to append a second backup from a different machine onto the tape or you want to catalog the tape before rewinding and ejecting. In our example, use /dev/nsa0 to run your command without rewinding.

To automatically eject a tape when a command finishes, use the node that begins with e. For example, if you’re running a full system backup, you probably want the tape to eject when the command finishes so the operator can put the tape in a case to ship offsite or place in storage. Our example uses the /dev/esa0 device name to eject the tape when the command finishes. Some tape drives might not support automatic ejection; they’ll require you to push the physical button to work the lever that winches the tape out of the drive. The easiest way to identify such a drive is to try to eject it via the device node and see what happens.

The $TAPE Variable

Many programs assume that your tape drive is /dev/sa0, but that isn’t always correct. Even if you have only one tape drive, you might want to eject it automatically (/dev/esa0) or not to rewind it upon completion (/dev/nsa0).

Many (but not all) backup-related programs use the environment variable $TAPE to control which device node they use by default. You can always override $TAPE on the command line, but setting it to your most commonly used choice can save you some annoyances later.

Tape Status with mt(1)

Now that you know how to find your tape drive, you can perform basic actions on it—such as rewinding, retensioning, erasing, and so on—with mt(1). One basic thing mt(1) does is check a tape drive’s status, as follows:

# mt status
Mode      Density              Blocksize      bpi      Compression
Current: ➊0x25:DDS-3          variable       97000   ➋DCLZ
---------available modes---------
0:        0x25:DDS-3           variable       97000    DCLZ
1:        0x25:DDS-3           variable       97000    DCLZ
2:        0x25:DDS-3           variable       97000    DCLZ
3:        0x25:DDS-3           variable       97000    DCLZ
---------------------------------
             ➌ Current Driver State: at rest.
---------------------------------
File Number: 0  Record Number: 0          Residual Count 0

You don’t have to worry about most of the information here, but if you want to go through it line by line, the mt(1) man page contains a good description of all the features. At the very least, if the command returns anything useful, this means mt(1) can find your tape drive.

One of the first things we see is the drive density ➊. Older drives can have tapes of different densities for different purposes, but modern tape drives pack data as tightly as possible. This particular tape drive is a DDS-3 model; while you could choose to use another density, all the choices it offers are DDS-3. We also see that this tape drive offers hardware compression with the DCLZ algorithm ➋. Near the bottom, we see what the tape drive is doing right now ➌.

The status command might give you different sorts of messages. The most problematic is the one that tells you that your tape drive is not configured:

#mt status
mt: /dev/nsa0: Device not configured

This means that you don’t actually have a tape at the device node that your $TAPE variable points at. You can experiment with device nodes and mt(1) by using the -f flag to specify a device node (for example, mt -f /dev/nsa1 status), although you should get correct information from dmesg.boot. If you’re sure that your device node is correct, perhaps you don’t have a tape inserted into the drive or the tape drive needs cleaning.

Another response you might get from mt status is mt: /dev/nsa0: Device busy. You asked for the status of your tape, and it replied, “I can’t talk now. I’m busy.” Try again later, or check ps -ax to see what commands are using the tape drive. When you’re working with actual tape, only one program instance can access it at a time. You can’t list the contents of a tape while you’re extracting a file from that tape.

Other Tape Drive Commands

You can do more with a tape drive than just check to see whether it’s alive. The mt(1) subcommands I use most frequently are retension, erase, rewind, and offline.

Tapes tend to stretch, especially after they’re used the first time. (I know perfectly well that modern tape vendors all claim that they prestretch their tapes or that their tapes can’t be stretched, but that claim and two slices of bread will get you a bologna sandwich.) Retensioning a tape is simply running the tape completely through, both forward and back, with the command mt retension. Retensioning takes all the slack out of the tape and makes backups more reliable.

Erasing removes all data from a tape. This isn’t a solidly reliable erasure, which you’d need to conceal data from a data recovery firm or the IRS; mt erase simply rolls through the tape and overwrites everything once. This can take a very long time. If you want to erase the tape quickly, you can use mt erase 0 to simply mark the tape as blank.

The mt rewind command rolls a tape back to the beginning, same as accessing the device through its default device node.

When you offline a tape, you rewind and eject it so that you can put a new tape in. The command is, oddly enough, mt offline.

Now let’s get some data on that tape.

BSD tar(1)

The most popular tool for backing up systems to tape is tar(1). Tar is short for “tape archiver”—it’s literally written for backups. FreeBSD also includes dump(8), but that’s intended only for UFS filesystems that don’t use soft updates journaling. You’ll certainly encounter other backup tools too, such as pax and cpio, as well as network-based backup tools, like Amanda, Bacula, and Tarsnap. These tools are well suited for certain environments but aren’t as universal as tar. Tar is a common standard recognized by almost every operating system vendor; you can find tar for Windows, Linux, Unix, BSD, macOS, AS/400, VMS, Atari, Commodore 64, QNX, and just about everything else you might encounter.

You can use tar(1) to back up to tape or to a file. A backup file containing tarred files is called a tarball. It’s very fast and easy to restore just one file or a subset of files from a tarball. It’s also easy to restore a portion of your backup from tape, but it’s not nearly as fast.

FreeBSD uses a version of tar called bsdtar. Bsdtar can behave completely consistently with GNU tar and can also behave in strict accordance with POSIX tar. If you’re at all concerned about the differences between GNU tar, POSIX tar, and bsdtar, read tar(1) for all the gory details. Bsdtar is built on libarchive(3), a library specifically for creating and extracting backup archives. Thanks to libarchive, bsdtar can extract files from anything from a traditional tape backup to an ISO image, all with the same interface. If you need to open an RPM, a zip file, or almost any other archive, bsdtar is your friend.

Bsdtar, like any other tar(1), can be dumb. If your filesystem is corrupt in any way, bsdtar will back up what it thinks you asked for. It will then happily restore files that were damaged during the original backup, overwriting working-but-incorrect files with not-working-and-still-incorrect versions. These sorts of problems rarely happen, but tend to be unforgettable when they do.

tar Modes

Tar can perform several different actions, controlled by the command line flags. These different actions are called modes. You’ll need to read the man page for a complete description of all tar modes, but the following list describes the most commonly used ones.

Create an Archive

Use create mode (-c) to create a new archive. Unless you specify otherwise, this flag backs up everything to your tape drive ($TAPE, or /dev/sa0 if you haven’t set $TAPE). To back up your entire system, you’d tell tar to archive everything from the root directory down:

# tar -c /

In response, your tape drive should light up and, if your tape is big enough, eventually present you with a complete system backup. Many modern hard drives are bigger than tape drives can hold, however, so it makes sense to back up only the vital portions of your system. For example, if the only files on your computer that you need are in the directories /home and /var, you could specify those directories on the command line:

# tar -c /home /var

List Archive Contents

List mode (-t) lists all the files in an archive. Once you’ve created an archive, you can use this mode to list the tape’s contents.

# tar -t
.
.snap
dev
tmp
--snip--

This list includes all the files in your backup and might take a while to run. Note that the initial slashes are missing from filenames; for example, /tmp shows up as tmp. This becomes important during restores.

Extract Files from Backup

In extract mode, tar retrieves files from the archive and copies them to the disk. (This is also called untarring.) Tar extracts files in your current location; if you want to overwrite the existing /etc directory of your system with files from your backup, go to the root directory first. On the other hand, to restore a copy of /etc in my home directory, I’d go to my home directory first.

# cd /home/mwlucas
# tar -x etc

Remember when I said that the missing initial slash would be important? Here’s why. If the backup included that initial slash, tar would always extract files relative to the root directory. The restored backup of /etc/rc.conf would always be written to /etc/rc.conf. Without the leading /, you can recover the file anywhere you want; the restored /etc/rc.conf can be resurrected as /home/mwlucas/etc/rc.conf. If I’m restoring files from a machine that’s been decommissioned, I don’t want them to overwrite files on the current machine; I want them placed elsewhere so they won’t interfere with my system.

Verify Backups

Once you have a backup, you probably want to confirm that it matches your system. Diff mode (-d) compares the files on tape to the files on disk. If everything on the tape matches the system, tar -d runs silently. A perfect match between tape and system is not normal, however. Log files usually grow during the backup process, so the log files on tape shouldn’t match the files on disk. Similarly, if you have a database server running, the database files might not match. If you truly want a perfect backup (also called a cold backup), you’ll need to shut down to single-user mode before taking the backup. You must decide which errors you can live with and which require correction.

Other tar Features

Tar has several other features that can make it more friendly or useful. These include verbose behavior, different types of compression, permissions restore, and the most popular option—alternate storage.

Use Non-default Storage

Tar feeds everything to your tape drive by default, but the -f flag allows you to specify another device or file as the destination. In all of the preceding examples, either I’m using the default tape drive, /dev/sa0, or I’ve set $TAPE. If I have neither of these, I’d need to specify a tape drive with -f:

# tar -c -f /dev/east0 /

You can also back up to a file (or tarball) instead of using a tape. Source code distributed via the internet is frequently distributed as tarballs. Use the same -f flag to specify a filename. For example, to back up the chapters of this book as they were written, I ran the following every so often to create the tarball bookbackup.tar:

#tar -cf bookbackup.tar /home/mwlucas/af3e/

This file can easily be backed up on machines elsewhere—so even if my house burns down, the book would be safe. I could then run phone and power lines to the neighbor’s house, borrow a laptop, find an open wireless access point, run tar -xf bookbackup.tar, and work amidst the charred timbers while waiting for the insurance company. (I couldn’t do much else at the time, anyway.)

Verbose

Tar normally runs silently unless it encounters an error. This is good most of the time (who wants to read the complete list of files on the server every time a backup runs?), but sometimes you like to have the warm fuzzy feeling of watching a program do its work. Adding the -v flag makes tar print the name of each file it processes. You can use the verbose flag to create a complete list of all the files being backed up or restored. In a routine backup or restore, this verbosity makes errors difficult to see.

Compression

Bsdtar inherits support for every compression algorithm libarchive(3) understands. We’ll cover a few you might use to create archives, in order from the most to least desirable. Bsdtar supports many more compression algorithms, but you wouldn’t normally use them to create an archive.

XZ Compression

The XZ compression algorithm is the new hotness. Enable it with -J. Non-FreeBSD hosts might need to pipe restores through xz(1) to read them. Tarballs compressed with XZ usually end in .txz.

bzip Compression

FreeBSD’s tar supports bzip compression, which shrinks files even more tightly than gzip, with the -j flag. Bzip uses more CPU time than gzip, but these days, CPU time is not nearly as limited as when gzip came out. Not all versions of tar support bzip compression, either. If you’ll only be reading your files on a FreeBSD machine or you’re comfortable installing bzip on other platforms, use the -j flag. Most tarballs compressed with bzip(1) end in .tbz.

gzip Compression

The gzip flag (-z) runs the files through the gzip(1) compression program on their way to or from the archive. Compressed tarballs usually have the extension .tar.gz, .tgz, or, on rare occasion, .taz. Compression can greatly reduce the size of an archive; many backups shrink by 50 percent or more with compression. While all modern versions of tar support gzip, older versions don’t, so if you want absolutely everybody to be able to read your backup, don’t use -z.

Primordial Unix Compression

In contrast, all Unix versions of tar can use the -Z flag to compress files with compress(1). The compress program isn’t as efficient as gzip, but it does reduce file size. Every implementation of tar you’re likely to encounter supports compress(1). Tarballs compressed with -Z have the extension .tar.Z.

Permissions Restore

The -p flag restores the original permissions on extracted files. By default, tar sets the owner of an extracted file to the username that’s extracting the file. This is fine for source code, but for system restores, you really want to restore the file’s original permissions. (Try to restore these permissions by hand some time; you’ll learn quite a bit about why you should have done it right the first time.)

And More, More, More . . .

Tar has many, many more functions to accommodate decades of changes in backups, files, filesystems, and disks. For a complete list of functions, read man tar(1).

Recording What Happened

You can now back up your entire system as well as track changes in a single file. All that remains is to track what’s happening on the screen in front of you. One of those rarely mentioned but quite useful tools every sysadmin should know is script(1). It logs everything you type and everything that appears on the screen. You can record errors and log output for later dissection and analysis. For example, if you’re running a program that fails in the same spot every time, you can use script to copy your keystrokes and the program’s response. This is notably useful when upgrading your system or building software from source code; the last 30 lines or so of the log file make a nice addition to a help request.

To start script(1), just type script. You’ll get your command prompt back and can continue working normally. When you want the recording to stop, just type exit or press CTRL-D. Your activity will appear in a file named typescript. If you want the file to have a particular name or be in a particular location, just give that name as an argument to script:

# script /home/mwlucas/debug.txt

This tool is extremely useful for recording exactly what you typed and exactly how the system responded. Any time you need to ask for help, consider script(1).

Repairing a Broken System

The best way to learn an operating system is to play with it, and the harder you play, the more you learn. If you play hard enough, you’ll certainly break something, which is a good thing—having to fix a badly broken system is arguably the fastest way to learn. If you’ve just rendered your system unbootable or plan to learn quickly enough to risk doing that, this section is for you. If your system is deeply hosed, you’ll learn a lot and quickly.

Single-user mode (discussed in Chapter 4) gives you access to many different commands and tools. What if you’ve destroyed those tools, however? Perhaps you’ve even damaged the statically linked programs in /rescue. That’s where the install media comes in.

The FreeBSD installation images have an option to activate a live system. This live system includes all the programs that come by default with FreeBSD. When you boot off the install media, you can choose to enter the live CD instead of installing.

You must have some familiarity with system administration to use the live CD. Essentially, the live CD gives you a command prompt and a variety of Unix utilities. You get to use the boot-time error messages and that ballast you keep between your ears to fix the problem. It’s you against the computer. Of the first half-dozen times I’ve resorted to a live CD or its predecessors, the computer won three. After that, though, my success rate was much improved. Reading this book, as well as other Unix administration manuals, will improve your odds of success.

It’s impossible to outline a step-by-step process for generic problem situations; the exact steps you must follow depend on the exact damage you’ve inflicted on your poor, innocent operating system. If you’re really desperate, however, the live CD gives you a shot at recovery without reinstalling. I’ve had problems where I’ve accidentally destroyed my /etc directory or fried the getty(1) program that displays a login prompt. Careful use of the live CD can repair these problems in a fraction of the time a reinstall would take. If nothing else, perhaps you can back up any data that survived being exposed to you and then reinstall.

Always use install media that’s roughly equivalent to the FreeBSD version you’re running. You can probably use a 12.2 install CD to repair a 12.1 system, but a 14-current install disk might cause a whole new set of problems.

Now that you can recover from almost any mistake you might make, let’s dive into the heart of FreeBSD: the kernel.