24
PROBLEM REPORTS AND PANICS

FreeBSD is produced by human beings. Human beings make mistakes. Some of these mistakes are pretty trivial, while others can crash the whole system. FreeBSD explicitly has no warranty, but the community takes problems fairly seriously. Developers can’t fix these problems without a proper bug report, however—and that’s where you come in. Learning how to file a usable bug report will help you interact with not only the FreeBSD Project but also every other entity that produces software.

Bug reports need program output that demonstrates the bug, but what if the problem crashes the whole system? A system-halting panic is perhaps the most vexing of problems, but with proper preparations, you can deal with a panic as routinely as any less intrusive bug. The FreeBSD folks definitely want debugging output from your panic reports, and you can easily provide it in your problem report.

But first, useful bug reports.

Bug Reports

A bug report is a detailed description of a problem that causes the system to behave in an unexpected manner. That’s kind of vague, yes. What’s unexpected? What’s a problem? There’s a whole spectrum of valid problem reports, ranging from “I’d expect a man page for this thing” to “when I mount an SMB filesystem, the operating system crashes.” Trivial problems like a missing man page reference might not seem worth your time to report, but every reference in every man page is there because someone thought it was worth including. So what’s a bug report, and what isn’t?

A bug report isn’t where you say that you have a problem. A bug report is where you prove that FreeBSD has a problem. Yes, I said prove. It’s not that FreeBSD is innocent until proven guilty, but for you to submit a bug report, you must substantiate your claim. Bugs filed without evidence will be closed with terse replies, like “not a bug” or “useless report.” The proper venue to cry for help is a search engine, followed by a mailing list or the FreeBSD forums.

Any variation on “I don’t know what I’m doing” doesn’t belong in a bug report. This includes “FreeBSD doesn’t work the way I think it should” or “something bad happens when I do something dumb.” If you start your new hobby of free climbing with a master-rated sheer cliff face, fall, and break your fool neck, the hospital kind of has to take you in. If you yank out a new hard drive halfway through running newfs(8) and complain about filesystem corruption, the FreeBSD folks will dismiss your bug.

Bugs can be about system inconsistencies. Every network interface, API, and system call has a man page. If you try to call up the man page for a system call and don’t get a match, that’s a bug. If you’re reading source code and find a place where the documentation doesn’t match the code, that’s a bug. If you can make a program (or the whole system) reproducibly crash, that’s a bug.

You can also file bugs to submit improvements to the FreeBSD Project. The key word here is improvement, not wish. An improvement needs actual code attached to the bug, along with how you’ve tested your code and any related information, such as specifications and standards. If you file enough of these, you might be invited to become a committer.

Bugs are collaborative. By filing a bug, you’re indicating a willingness to work with the FreeBSD developers to resolve your issue. This might mean applying a patch, trying a different approach, or running debugging commands and sending the output to the developers. Filing a bug and expecting an answer like, “Fixed! Go do this,” is unrealistic. Including everything in your initial bug report helps resolve the issue much more quickly. Err on the side of providing too much data.

Repeat after me: “Free software. Donated support time.” A server’s RAID card making your hard drives spin to a conga beat feels critical to you, but the people on the other end of the bug are giving up their personal time to help you. Remember that.

It’s best if you’re running a recent version of FreeBSD when filing a bug. If you file a bug on FreeBSD 12.0-RELEASE when 12.4-p15 is the current version, someone will ask you to update and try again. Nobody will look at a problem report for a FreeBSD release past End of Life.

Before Filing a Bug

Ideally, you won’t ever have to file a bug. Not only is a proper report for a serious bug a lot of work for you; it’s a lot of work for the FreeBSD developers. The FreeBSD Project has an internal mailing list dedicated to assessing the bug database and guiding reports to their most likely owners. While sending an email to a FreeBSD mailing list announces your woes to thousands of people, opening a bug announces your woes to thousands of highly skilled people and demands that they handle virtual paperwork for you. Before filing a bug, be absolutely certain that both you and the FreeBSD Project need it.

Does the problem happen on all of your hosts or only on one? Problems restricted to a single host might result from failing hardware. Consistent, reproducible behavior is much more likely to be a bug.

Treat your issue as a general problem and search the usual FreeBSD resources. Review the FAQ and the Handbook. Check the FreeBSD bug database at https://bugs.FreeBSD.org/ for an existing bug. Search the mailing list archives, the forums, and the wider internet for people who’ve already had this problem. Ask on the forums or the FreeBSD-questions mailing list whether anyone else has seen this behavior. Is this expected, or should you open a bug? The questions people ask will be invaluable in troubleshooting your problem and in creating your bug report.

Before starting a bug report, gather every scrap of information that might possibly be helpful. This includes:

• Verbose boot output
• System version
• Custom kernel configuration, if any
• Program debugging output
• What do you expect to happen?
• What actually happens?

Can you reproduce this problem? A developer investigating a bug needs a reproducible test case. If your server starts singing show tunes at 3 AM, that is a problem. If it happened only once and you can’t reproduce it, you’re best served by keeping your mouth shut so people don’t think you’re loony. If it happens whenever you run a particular combination of commands on certain hardware, though, the matter can be verified and investigated so that either the problem is resolved or someone offers your server a recording contract.

FreeBSD tracks bugs with Bugzilla at https://bugs.FreeBSD.org/. Before submitting a bug, search the existing bug database to see whether there’s something similar or related. Does your problem resemble any existing bugs? If your server sings Disney tunes, but another bug shows someone’s identical hardware does Broadway hits, you should probably mention that bug in your report. Are there any illuminating comments on those bugs? The comments might tell you how to cope with or work around your issue without filing another identical bug. If you want updates, add yourself to the bug’s cc field. You’ll get an email every time the bug is updated.

While you’re searching Bugzilla, create an account there. Even if you don’t need to file this bug, one day you will . . . and you’ll probably be pretty annoyed about it. Having your Bugzilla account ready will make that submission just a little bit easier.

If you get this far and still have a problem, you might actually need to file a bug report. Let’s see what not to put in it.

Bad Bug Reports

The easiest way to understand a good bug report is to read some bad ones and identify what makes them bad. Digging through the closed bugs uncovers bunches of bad reports, but here’s an archetype:

When I boot the FreeBSD 12.1 ISO image, I can’t get past the “Welcome to FreeBSD” options screen. The boot menu is stuck, and each time the screen refreshes it stays at 10. It doesn’t matter what I press, the system never boots. If I press a whole bunch of buttons, I eventually get a kernel panic. The same ISO image launches in VirtualBox and I can install it to the disk.

The bug report includes the model number of a dead standard SuperMicro motherboard, keyboard, and mouse. None of the hardware is exotic. The reporter suggests reproducing the problem by booting the ISO with similar hardware.

First off, the reporter obviously has a problem installing FreeBSD. It might even be that FreeBSD has a problem. I’ve no doubt that this system fails at boot exactly as advertised. But there’s no evidence and no diagnostic information. The reproduction process isn’t very useful; if every 12.1 installation image behaved this way on common hardware, the release engineers would have never signed off on the release.

Including the hardware make and model isn’t as useful as you might hope. Vendors occasionally change chipsets without changing the model number. The verbose boot information identifies the hardware in the machine in a way that the model number never can. This reporter can’t get a 12.1 verbose boot, however.

If I experienced this behavior, I’d first try a second CD. Perhaps the first burned disk was bad. If the behavior persisted, I’d download a slightly older version of FreeBSD to see whether the problem exists there. If 12.0 fails, how about 11.0? I’d include the verbose boot information from the earlier version in my bug report. If the older version failed, I’d ask the FreeBSD-questions mailing list for further advice before filing a bug.

As you might guess, nobody follows up on bugs like this.

Many developers like fixing bugs. They enjoy digging through code and identifying subtle problems. What they don’t enjoy is sorting through people’s erratic bug reports; they expect to be paid for dealing with difficult people. Your goal is to file a bug so complete and compelling that a developer that’s looking for a bug to work on will think you’re easy to work with—and then, you need to actually be easy to work with.

The FreeBSD FAQ includes a joke by Dag-Erling Smørgrav: “How many -current users does it take to change a light bulb?” The answer is 1,169 and includes “three to submit (bugs) about it, one of which is misfiled under doc and consists only of ‘it’s dark.’” If your bug amounts to “it’s dark,” it’s a bad problem report.¹

The Fix

The most important part of any bug report is the fix. How do you remedy the problem? Perhaps all you have is a workaround. “The program crashes if I do this, but I can run it and pipe the output through such-and-such and do well enough.” That comment helps the next person to hit your bug.

When you hit a bug, take a look at the source code. Fixing typos in man pages or on the website isn’t hard. If you’re a programmer, a couple minutes of perusing the source might uncover the problem. If it doesn’t, well, figuring out why the system behaves in this way will make you a better programmer and debugger.

Maybe you can’t fix this bug. Letting people know the bug exists is still helpful. But by including a fix, your bug transcends a report and becomes a contribution to the community.

Filing Bugs

All bugs get filed and handled at https://bugs.FreeBSD.org/. FreeBSD has three categories of bug: ports, base system, and documentation. Use a ports bug for anything with add-on software. Use the base system for anything that gets installed with a basic FreeBSD install. Use the documentation category for problems with man pages, the FAQ, the Handbook, and the website. Each brings up a slightly different web form. The fields needed for documentation and ports bugs are mostly subsets of the base system bugs, though, so we’ll walk through filing a base system bug.

The web form includes several drop-down fields that let you steer your bug toward the right people. Your bug might get reassigned right after you file it, but that’s okay; initially, you’re looking for someone who understands what the heck you’re talking about. You don’t want a doc committer triaging system call issues or a source committer figuring out a port-packaging problem.

The Component field lets you select a part of the system the bug affects. The list of components varies over time, but selecting a component brings up a description. While there’s always a catch-all field, such as Bin for base system bugs, making a good choice will accelerate handling of your bug.

In the Version field, select the FreeBSD version this bug applies to.

The Severity field is both somewhat misleading and requires a little detachment from your own emotions. The choices are “Affects Only Me,” “Affects Some People,” and “Affects Many People.” A bug terrible enough that you consider gnawing off your own foot to escape might affect only you. It’s critical to you, but not to the FreeBSD project. Resist the urge to declare that it affects everyone. Similarly, a typo on the website might be visible to everyone, but if nobody’s noticed until now, it’s probably not worth “Affects Many People.” Reserve the more important severity levels for bugs that negatively impact all users of a particular device driver or anyone who uses a certain filesystem. If you get a reputation for filing trivial reports as critical, you’ll quickly find yourself being ignored. The FreeBSD Project works on the honor system, and reputation counts for more than you might think.

In the Hardware field, select the platform you found the bug on. Even if that seems irrelevant, it might be critical.

The OS field is vestigial from Bugzilla. Ignore it.

Below these drop-downs, Bugzilla offers text fields. These need a little more thought.

The Summary takes a brief description of the problem. A good summary should provide unique information to make your bug stand out from other bugs. “Panic when unmounting an SMB filesystem” is decent. “Can’t install,” “system broken,” and “problem” are terrible. A developer perusing the bug database will see your summary first. A bad summary will encourage him to gloss right over it.

The Description area is where you get to describe the problem. Don’t rant and rave about how awful everything is. Say what happened and what you expected to happen. Include debugging output, if it’s short enough to fit reasonably; otherwise, add the debugging output as an attachment. Include advice on how to replicate the problem. If you have a fix, give it. Add in anything you’ve discovered about the problem. Sometimes, the most unusual detail provides the vital clue.

Add Attachments beneath the description. This is where you can upload your custom kernel configuration, verbose boot messages, kernel panic messages, and lengthy diatribes.

Use the Preview button to verify you included everything you thought you did. Once the bug looks correct, hit Submit.

After Submitting

You’ll shortly receive an email stating that you’re now the proud owner of bug number such-and-such. Any response you make to that email gets attached to that bug, so long as you don’t change the subject.

A high percentage of bugs that include the proper information get closed quickly. Complex or elusive bugs might take longer, but if you provide enough detail, you’ll see updates.

If it seems that your bug report has been forgotten, drop a note to the appropriate mailing list with your bug number, a brief explanation of the issue, and a sentence or two on why it’s important. FreeBSD is a volunteer effort, and it’s possible that something happened to the person who would normally handle that bug report. While many FreeBSD developers are professional programmers, for many of them, this is still a hobby that must take a backseat to sick kids or the big work deadline. If nothing else, hire a developer on a contract basis to address your particular issue.

If you file a notably vexing bug, a FreeBSD developer will probably ask you for more information. Provide it as quickly and thoroughly as possible. If you don’t understand what they’re asking for, spend some time researching and then ask. Most developers are happy to provide pointers to a willing and basically competent partner, especially if you can help them improve their code.

I’ve lost count of how many FreeBSD bugs I’ve filed. I tend to file either trivial or serious bugs, such as documentation errors and kernel panics, but very little in between. Most were solved and/or committed and then closed. The odd ones were mostly trivial goofs on documentation that lives under /usr/src/contrib, an area where the FreeBSD Project specifically disavows responsibility for minor fixes. If a doofus like me can get over 90 percent of his bugs successfully closed, anyone can. Be warned, however: if you submit enough correct patches, you’ll find that the committers you work with will start to talk about you behind your back. Eventually, they’ll grow tired of acting as the secretary for your high-quality work and offer you commit access. If you refuse, they’ll offer more insistently. Don’t worry; becoming a committer isn’t that painful. The rumors that the FreeBSD Project initiation ritual involves a bunch of Danes with axes behind a bike shed are completely untrue. Mostly.

Keep filing good bug reports anyway; that’s the only way FreeBSD improves!

The worst sort of bug to deal with is a full-on system crash. Let’s talk about how to get information from one.

System Panics

A panic is when the operating system completely stops working. All systems, from the network stack to the disk drive, stop working. A system chooses to panic, or completely stop working, when the kernel faces an unresolvable conflict. If the system achieves a condition that it doesn’t know how to handle, or if it fails its own internal consistency checks, it throws up its hands and says, “I don’t know what to do!” A panic is the kernel’s version of malicious obedience.² Production versions of FreeBSD are increasingly difficult to panic, but it can still happen. The easiest way to panic a system is to do something daft, like yank out a non–hot swappable hard drive while it’s in use. Panics aren’t uncommon when running -current; they’re not frequent, mind you, but they’re not exotic rarities.

FreeBSD is very complex, and neither its royal blood lineage nor the open source development process can protect it from all bugs. Fortunately, that heritage and the development process do give you the tools you need to provide the information for other people to debug your problem. You might begin with a cryptic error code, but you’ll quickly learn that your string of garbage characters means something to someone.

A panicking kernel can copy critical information into a crash dump. The crash dump contains enough information about the panic that hopefully a developer can identify and fix the underlying problem. Configure every system to capture crash dumps before you allow them to enter production. FreeBSD can capture crash dumps with an install-time setting, but if you reconfigure your server or have unique disk partitioning, you’ll want to confirm that crash dumps still work. This precaution will be wasted on most of your servers but pays off when something explodes.

Recognizing Panics

When a system panics, it stops running all programs, writing to the disk, and listening to the network. On any version of FreeBSD except -current, a panicking system automatically reboots. Not all unexplained reboots are panics—bad hardware, such as a failing power supply or cruddy memory, can cause a reboot without any sort of log or console message. If you’re running -current, though, a panic will cause a console message much like this:

panic: Assertion cp->co_locker == curthread failed at /usr/src/sys/modules/smbfs/../..
/netsmb/smb_conn.c:363
cpuid = 5
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe085d0db630
vpanic() at vpanic+0x182/frame 0xfffffe085d0db6b0
kassert_panic() at kassert_panic+0x126/frame 0xfffffe085d0db720
smb_co_unlock() at smb_co_unlock+0x9c/frame 0xfffffe085d0db740
smb_co_put() at smb_co_put+0x68/frame 0xfffffe085d0db770
nsmb_dev_ioctl() at nsmb_dev_ioctl+0x484/frame 0xfffffe085d0db800
devfs_ioctl_f() at devfs_ioctl_f+0x15d/frame 0xfffffe085d0db860
kern_ioctl() at kern_ioctl+0x230/frame 0xfffffe085d0db8c0
sys_ioctl() at sys_ioctl+0x17e/frame 0xfffffe085d0db9a0
amd64_syscall() at amd64_syscall+0x2de/frame 0xfffffe085d0dbab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe085d0dbab0
--- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x800b8016a, rsp = 0x7fffffffe4f8, rbp = 0x7fffffffe530 ---
KDB: enter: panic
db>

The only part of this message that seems even vaguely sensible to me is the first line. I unmounted an SMB filesystem and got this panic message. The first line mentions smbfs, netsmb, and smb_conn, which seem pretty SMB-ish to me.

The db> at the bottom is a debugger command prompt. Hit ENTER a couple of times and you’ll see the debugger respond; you can enter commands. Debugger instructions aren’t Unix commands, but they help you extract more information out of the system.

Responding to a Panic

If you get a panic, the first thing to do is get a copy of the panic message. Since FreeBSD is no longer running, the standard methods for copying data from your machine won’t work—you can’t SSH in, and script(1) is no longer viable. The console might even be completely locked up and unresponsive instead of being in the debugger. In any event, you must have that error message.

Back in the bad old days of the 1990s, FreeBSD didn’t automatically reboot after a panic; originally, it just sat there displaying the panic message. The first time I saw a panic, I scrambled for paper and pen. Eventually, I found an old envelope and a broken stub of pencil that made marks if you held it at just the right angle and crawled between the server rack and the rough brick wall. I balanced the six-inch black-and-white monitor in one hand, and with my other hand, I held the envelope against the wall. Apparently, I grow a third hand under duress because I recorded the panic message on the envelope somehow. Finally, scraped and cramped, I slithered back out of the rack and victoriously typed the whole mess into an email. Surely the FreeBSD Project’s Panic Emergency Response Team would be able to look at this garbage and tell me exactly what had happened.

I quickly learned that FreeBSD has no elite PERT standing by to take my problem report. Instead, I got a lonesome email: “Can you send a backtrace?” When I asked how, I was directed to a man page. (Drag yourself all the way back to Chapter 1.) Fortunately, the panic was easily reproducible—the only thing that had to happen to recreate the issue was for a customer to log in to the system. I spent the rest of the day struggling to master serial consoles and core dumps.

The problem with the panic message on my envelope was that it gave only a tiny scrap of the story. It was so vague, in fact, that it was like describing a stolen car as “red, with a scratch on the fender.” If you don’t give the car’s make, model, VIN, and license plate number, you can’t expect the police to make much headway. Similarly, without much more information from your crashing kernel, the FreeBSD developers can’t catch the criminal code.

The good news is, panic handling has vastly improved since those days. FreeBSD can automatically record crash dumps and capture everything about a panic. There’s even a toggle in the installer to enable it. I highly recommend testing the panic capture before putting a machine into production, however. This way, if you get a panic, you’re ready and you’ll be able to easily file a complete problem report.

Preparations

Configuring crash dumps requires telling FreeBSD which swap device to save the dump on, through the dumpdev /etc/rc.conf variable. If you set dumpdev to AUTO, the kernel will automatically save the dump to the first swap device. You can specify a different swap device if needed, but the whole dump must fit in a single swap device. If your regular swap space doesn’t have enough space to hold the dump, add a disk to get sufficient swap space and set dumpdev to that partition.

The Crash Dump in Action

When a system configured to capture panics crashes, it saves a copy of the kernel memory. The copy is called a dump. The system can’t save the dump straight to a file. The crashed kernel doesn’t know anything about filesystems, for one thing, and the filesystem might be corrupt or a write could corrupt it. A crashed kernel understands partitions, however, so it can write the dump to a partition. Most FreeBSD hosts have readily available scratch space—the swap partition. FreeBSD defaults to dumping on the first swap partition on the system, placing the dump as close to the end of the partition as possible. After the dump, the computer reboots.

After a panic, a host’s filesystems will almost certainly be dirty. Perhaps they’re ZFS or they use soft updates journaling, but the system still must recover from the journal or roll back to the last successful ZFS transaction group. Cleaning a filesystem with fsck(8) can use a lot of memory, so FreeBSD must enable swap before running fsck(8). Hopefully, you have enough memory for fsck(8) to not require swapping, and if swapping is necessary, hopefully you have enough swap space to avoid overwriting the dump file lurking at the end of the swap partition. Worst case, you could boot into single-user mode, enable swapping to partitions that don’t have the dump, clean up a filesystem to save the dump to, and then manually run savecore(8).

Once FreeBSD has a useful filesystem where it can save a core dump, it checks the swap partition for a dump. If it finds a core dump, FreeBSD runs savecore(8) to copy the dump out of swap and into a proper filesystem file, runs crashinfo(8) to gather information from the dump, clears the dump from swap space, and continues rebooting. You now have a kernel core file usable for debugging.

The savecore(8) automatically places kernel dumps in /var/crash. Each crash is in a file called vmcore with a trailing period and number. The first panic is vmcore.0, the second vmcore.1, and so on. FreeBSD defaults to keeping the most recent 10 crash dumps. The file vmcore.last always points to the most recent crash dump.

If your /var partition is not large enough to contain the dump, choose a different directory with the dumpdir variable in rc.conf :

dumpdir="/usr/crash"

While savecore(8) also supports a few other options, such as compression, they aren’t usually necessary on modern systems.

FreeBSD defaults to running crashinfo(8) on each recovered crash dump. The crashinfo(8) program runs a series of debugger scripts to gather information from the panic, storing it in a convenient text file, core.txt.0. This information includes a panic backtrace, process list, and a whole bunch of virtual memory statistics.

Testing Crash Dumps

You’ve set the savecore rc.conf option, so everything should work. Any time you hear the phrase “should work,” immediately ask, “How can I verify that it does work?” Force FreeBSD to panic by setting the sysctl debug.kdb.panic to any integer above 0. While this is the ugliest way to reboot a machine ever, it does make the host run through the panic and core-preservation process. Shut down any active processes that might corrupt data if interrupted, such as databases, and deliberately trigger a panic.

# sysctl debug.kdb.panic=1

You’ll see the panic message flash on the console, followed by the host’s progress in dumping core to swap space. If you watch the reboot messages, you should see a quick mention of saving core files. When you can log into the machine again, take a look in /var/crash. You’ll find three files: info.0, vmcore.0, and core.txt.0.

The info.0 text file describes the dump recovery process. It includes the hostname, the architecture, the panic message, and more. The most important detail is the last line, however.

  Dump Status: good

This dump is usable. You can proceed with debugging.

The file vmcore.0 contains the memory dump in binary form. It should be anywhere from a couple hundred megabytes to gigabytes, depending on what your host was doing when it panicked.

The file core.txt.0 contains the panic information from vmcore.0. When you file a problem report, include the core.txt file for your panic.

Congratulations—you have a core dump! When your host actually panics, you can get information from the dump. Sometimes, though, the panic gets a little more complicated.

Crash Dump Types

FreeBSD supports three different sorts of crash dumps: minidumps, full dumps, and textdumps. All get written to swap space at a panic and copied to files at boot.

A minidump, the current default dump format, contains the memory used by the kernel. While the kernel itself isn’t that big, you’ll also get the UFS buffer cache. The swap space needed should be only a fraction of your system memory, but it’s possible that it could be almost as large as your system memory, depending on what your system was doing at the time. The dump excludes both memory not used by the kernel and the ZFS ARC.

A full dump contains every scrap of memory the system has. If it’s in RAM, it gets dumped. The whole kernel memory? Yep. Your web server’s buffer? Passwords? It’s all in there. A full dump takes up as much swap space as your host has memory. Enable full dumps by setting the sysctl debug.minidump to 0. Enable full dumps only if a FreeBSD developer asks you to do so to help debug a particularly intractable panic.

A textdump is an advanced type of dump that contains only the information captured by the ddb(8) debugger and associated scripts. It’s available only on hosts with the DDB option in their kernel—as in, not the GENERIC kernel of any release. It’s in the -current GENERIC kernel, though, so you brave souls running -current can take advantage of textdumps.

Textdumps

A textdump takes advantage of the ddb(8) debugger to run scripts on a panicked kernel. The default scripts in /etc/ddb.conf pull the most commonly needed information out of the kernel and then dump that information to disk. While crashinfo(8) runs on a captured memory image, though, ddb(8) runs on the panicked kernel. An experienced developer can take advantage of this. You might not be an experienced developer, but if you’ve made it this far into this book, you can follow directions and edit /etc/ddb.conf, and that’s close enough.

Enable textdumps with the ddb_enable rc.conf option.

# sysrc ddb_enable=YES

At system boot, the kernel debugger ddb(8) reads the debugging scripts from /etc/ddb.conf and loads them into the kernel. The debugger runs those scripts at a panic. The scripts switch the kernel to textdump mode, call several commands to gather useful information, write that data to swap space, and reboot the host. Textdumps aren’t as useful as minidumps, but they fit in much less space.

A complete textdump shows up in /var/crash as a tar file, textdump.tar.0. The textdump number gets incremented at each panic, and the file textdump.tar.last always points to the most recent textdump. The textdump(4) man page describes each of the files found inside the tarball, but as a user, what you really need to know is that you attach the whole thing to your bug report.

Dumps and Security

The vmcore file contains everything in your kernel memory at the time of the panic, which might include sensitive security information. Someone could conceivably use this information to break into your system. A FreeBSD developer might request a copy of the vmcore file and the bad kernel for many legitimate reasons; it makes debugging easier and can save countless rounds of email. Still, carefully consider the potential consequences of someone having this information. If you don’t recognize the person who asks or if you don’t trust him, don’t send the file and don’t feel bad about it. Take the time to research any developer who wants your vmcore. Even if they seem reputable and respected, it’s perfectly acceptable if you decide to work via emails back and forth rather than offer up the vmcore. Anyone qualified to work on your crash understands why you hesitate to send a vmcore, and anyone who tries to shame you into sending it probably shouldn’t have it.

Posting a link to your vmcore on the public internet offers the guts of your server to the whole world. Don’t do that.

If the panic is reproducible, however, you can cold-boot the system to single-user mode and trigger the panic immediately. If the system never starts any programs containing confidential information and nobody types any passwords into the system, the dump can’t contain that information. Reproducing a panic in single-user mode generates a security information–free, sanitized dump. Boot into single-user mode and then run:

# mount -ar
# /etc/rc.d/dumpon start
# command_that_panics_the_system

The first command mounts the filesystems as read-only so that you won’t have to fsck(8) yet again after a panic. The second command tells FreeBSD where to put a dump. Finally, run the command that triggers the panic. Triggering the panic might require more than one command, but this should get a clean dump for you in most cases.

If your panic requires that you load confidential information into memory, that information will be present in the dump. If you can file a useful bug report, you’re among the FreeBSD elite. Congratulations!