Chapter 12: A Few More Kernel Debugging Approaches

At the outset, back in Chapter 2, Approaches to Kernel and Driver Debugging, we covered various approaches to kernel debugging. (In that chapter, Table 2.5 provides a quick summary of kernel debug tools and techniques versus the various types of kernel defects; take another look.) The previous chapters of this book have covered many (if not most) of the tools and techniques mentioned, but certainly not all of them.

Here, we intend to merely introduce ideas and frameworks not covered so far (or just briefly mentioned) that you might find useful when debugging Linux at the level of the OS/drivers. We have neither the intent nor the bandwidth (space/pages) to go into these topics in depth, but feel free to use the links in the Further reading section to go deeper! Nevertheless, there are important topics covered in this chapter.

In this chapter, we're going to cover the following main topics:

  • An introduction to the kdump/crash framework
  • A mention on performing static analysis on kernel code
  • An introduction to kernel code coverage tools and testing frameworks
  • Miscellaneous – using journalctl, assertions, and warnings

An introduction to the kdump/crash framework

When a userspace application (a process) crashes, it's often feasible to enable the kernel core dump feature; this allows the kernel to capture relevant segments (mappings) of the process virtual address space (VAS), and write them to a file that is traditionally named core. On Linux, the name – and indeed various features – are now settable (look up the man page on core(5) for details). How does this help? You can later examine and analyze the core dump using the GNU debugger (GDB) (the syntax is gdb -c core-dump-file original-binary-executable); it can help to find the root cause of the issue! This is called post-mortem analysis, as it's done upon the dead body of the process, which is the core dump image file.

That's great, but wouldn't it be useful to be able to do the same with the kernel? This is precisely what the kernel dump (kdump) infrastructure provides – the ability to collect and capture the entire kernel memory segment (the kernel VAS) when the kernel crashes! Furthermore, a powerful userspace open source app (tool), crash, allows you to perform post-mortem analysis upon the kdump image, helping to find the root cause of the issue!

Why use kdump/crash?

Why use kdump/crash when we know how to analyze an Oops and kernel panic, use KGDB, KASAN, KCSAN, and so on? There are several reasons:

  • Tooling such as debug instrumentation (printk), KASAN, UBSAN, KCSAN, and KGDB are typically effective and enabled on a debug kernel. When your software is running in production and fails with a kernel-level issue, they are usually disabled and so don't help much.
  • Even having the Oops/panic diagnostic (the complete kernel log when the Oops occurred) might not be sufficient to find the root cause of a deep kernel bug. For example, you might require all frames of the kernel-mode stack(s) in question, not just the one where the crash occurred, as well as the content of kernel memory – in effect, the state of all kernel data.
  • Only kdump enables capturing all of this, in production. And crash lets you analyze it.

There's a downside to using kdump: it implies reserving some fairly significant amounts of system RAM and even possibly flash/disk memory space; this can be impractical, especially on some types of embedded systems.

Understanding the kdump/crash basic framework

Still interested? There are essentially two parts to using kdump/crash:

  1. Setting up the kernel to capture the kernel memory image if it does crash (Oops or panic); this involves configuring the primary kernel to enable kdump, and the setup to launch a so-called dump-capture kernel via a special kexec mechanism if a crash/Oops/panic does occur.
  2. Installing the crash utility on the dev/debug/host system; it takes the kdump image as one of its parameters. Learn how to use it to help debug the kernel/module issue(s).

Setting up and using kdump to capture the kernel image on crash

We won't attempt to go into details here as we lack the space to do so and as it's well-documented in the official kernel documentation, Documentation for Kdump - The kexec-based Crash Dumping Solution, here: https://www.kernel.org/doc/html/latest/admin-guide/kdump/kdump.html#documentation-for-kdump-the-kexec-based-crash-dumping-solution. If you are setting up kdump, I'd urge you to check this document out in detail. Do note, though, that many Linux distributions – especially the enterprise-class ones such as Red Hat, CentOS, SUSE, and Ubuntu – have their own wrappers around setting up kdump (special config files, packages, and modes, for example); look up the documentation for your distribution as required.

Distilled down, the kdump setup process goes like this:

  • Install the kexec-tools (via source or distribution package).
  • Configure one or two kernels:
    • The primary kernel (configured for kdump; runs in the usual manner)
    • A dump-capture kernel

In architectures that support relocatable kernels (i386, x86_64, arm, arm64, ppc64, and ia64, as of this writing), the primary kernel can also work as the dump-capture kernel (yay!). Look up the kernel configuration details here: https://www.kernel.org/doc/html/latest/admin-guide/kdump/kdump.html#system-kernel-config-options.

Right, continuing with the kdump activation process, follow these steps:

That's it! The primary kernel will quite literally warm-boot (preserving RAM content) into the dump-capture kernel when a trigger point is hit; as of writing, these include the following:

  • panic(): Setting the kernel.panic_on_oops sysctl to 1 ensures that the dump-capture kernel is booted into when the kernel Oops'es (recommended in production).
  • die() and die_nmi()
  • Magic SysRq's c command (when enabled, of course): This allows you to test the kdump feature by forcing a NULL pointer dereference and therefore, a kernel Oops by doing this: echo c > /proc/sysrq-trigger, as root.

Do note, though, that kdump is essentially useless if the reboot into the dump-capture kernel can't happen, perhaps due to hardware issues.

Good, kdump is now set up. But, how does this help? Upon a kernel crash or panic, how exactly do you capture the kernel memory image? It's like this: the kernel makes the dump image available via the /proc/vmcore pseudofile; so, if the dump-capture kernel drops you to a shell, simply use the cp command to write it out to disk, for example, [sudo] cp /proc/vmcore </path/to/dump-file>. Alternately, you can scp it to a remote server, or use a utility named makedumpfile to write out the content to disk/flash. Of course, you can always have a script do this non-interactively...

(This might be useful: Slide 14 of the Linux Kernel Crashdump presentation has a simple diagram showing the previous; this can be found at https://www.slideshare.net/azilian/linux-kernel-crashdump.)

Great. So, let's say the kdump-enabled kernel crashed and we now have the kernel dump image. Now what? The developer now learns to use the pretty powerful crash utility to interpret the dump file, helping to find the root cause of the actual issue! (I've provided several links to tutorials on using crash in the Further reading section; do look them up.)

kdump/crash is quite often used in industry; this is because, ultimately, kernel debugging is a difficult and painstaking job, put mildly. We can certainly appreciate having the entire kernel memory image available at the time of the crash, along with a tool to analyze it!

A mention on performing static analysis on kernel code

Broadly, there are two kinds of analysis tools – static and dynamic. Dynamic analysis tools are those that operate at runtime while the code executes. We've covered (most) of them in previous chapters – they include kernel memory checkers (KASAN, SLUB debug, kmemleak, and KFENCE), undefined behavior checkers (UBSAN), and locking-related dynamic analysis tools (lockdep and KCSAN).

Static analysis tools are those that operate upon the source code itself. Static analyzers (for C) uncover common bugs such as Uninitialized Memory Reads (UMRs), Use-After-Return (UAR), also known as use-after-scope), bad array accesses, and simply code smells.

For the Linux kernel, static analysis tools include Coccinelle, checkpatch.pl, sparse, and smatch. There are other, more general but still useful static analyzers as well; among them are cppcheck, flawfinder, and even the compilers (GCC and clang; FYI, GCC 10 onward has a new –fanalyzer option switch). There are many more static analyzers; several are high-quality commercial tools that require a license, such as Coverity, Klocwork, SonarQube, and plenty more.

Besides finding potential bugs, static analyzers are often used to expose security vulnerabilities (if you think about it, most code-related security vulnerabilities are nothing but bugs at heart).

The sparse and smatch static analyzers are specific to Linux. Coccinelle (French for ladybug) used to be in this bracket but is now quite generic (not just for the Linux kernel); it's a really powerful framework for code transformation as well as static analysis (with a bit of a learning curve). Coccinelle has four modes in which you can run it; to mention two of them, the report mode is of course to report potential code issues, whereas the patch mode can be used to propose a fix (by it generating a patch in the usual unified diff format). The official Linux kernel documentation provides the details for Coccinelle and sparse. It's important to read through them and try them out, so check them out here:

Examples using cppcheck and checkpatch.pl for static analysis

Due to space constraints, I can't show many examples of static analysis here but will show a couple. First, I'd like to remind you of something we saw clearly back in Chapter 5, Debugging Kernel Memory Issues – Part 1, in the Catching memory defects in the kernel – comparisons and notes (Part 1) section, when bug hunting using the Linux kernel's powerful memory checkers:

"Neither KASAN nor UBSAN catch the first three testcases – UMR, UAR and leakage bugs, but the compiler(s) generate warnings and static analyzers (cppcheck) can catch some of them."

The source file in question is this one: ch5/kmembugs_test/kmembugs_test.c. The first three test cases within it are the UMR, UAR, and the memory leakage bugs! Let's run cppcheck via our so-called better Makefile's sa_cppcheck target (look up ch5/kmembugs_test/Makefile to see the exact way in which we invoke cppcheck):

cd <lkd_src>/ch5/kmembugs_test
make sa_cppcheck
[...]
kmembugs_test.c:113:9: error: Returning pointer to local variable 'name' that will be invalid when returning. [returnDanglingLifetime]
return (void *)name;
        ^
kmembugs_test.c:113:17: note: Array decayed to pointer here.
return (void *)name;
                ^
kmembugs_test.c:105:16: note: Variable created here.
volatile char name[NUM_ALLOC];
[...]

Bang on target (do reread the code to see for yourself)!

As another example of how static analyzers can help, the kernel's checkpatch.pl Perl script is, in many ways, very specific to the Linux kernel and attempts to enforce the Linux kernel code style guidelines, which is very important to follow when submitting a patch (the guidelines are here: https://www.kernel.org/doc/html/latest/process/coding-style.html). A couple of quick examples to show you the value of running checkpatch.pl on your module's source code; here, I run it on our ch5/kmembugs_test/kmembugs_test.c source file, by leveraging our Makefile, invoking the appropriate target via make:

make checkpatch
[...]
WARNING: Using vsprintf specifier '%px' potentially exposes the kernel memory layout, if you don't really need the address please consider using '%p'.
#134: FILE: kmembugs_test.c:134:
+#ifndef CONFIG_MODULES
+    pr_info("kmem_cache_alloc(task_struct) = 0x%px
",
+        kmem_cache_alloc(task_struct, GFP_KERNEL));
[...]
WARNING: unnecessary cast may hide bugs, see http://c-faq.com/malloc/mallocnocast.html
#312: FILE: kmembugs_test.c:312:
+    kptr = (char *)kmalloc(sz, GFP_KERNEL);

These warnings are valuable (the first one, on a security aspect, while the second is of the usual type); do pay attention to them!

A significant issue with many static analysis tools is the problem of false positives – issues raised by the tool that turn out to be non-issues for the developer; it is a thorn in the side. Nevertheless, using static analysis as part of the development workflow is critical and must be incorporated by the team.

An introduction to kernel code coverage tools and testing frameworks

Code coverage is tooling that can identify which lines of code get executed during a run and which lines of code don't. Tools such as GNU coverage (gcov), and kcov and frontend tools such as lcov can be very valuable in gleaning this key information.

Why is code coverage important?

Here are a few typical reasons why you should (I'd go so far as to say must) perform code coverage:

  • Debugging: To help identify code paths that are never executed (error paths are pretty typical), thereby making it clear that you need test cases for them (to then catch bugs that lurk in such regions).
  • Testing/QA: Identify test cases that work and, more to the point, ones that need to be written in order to cover lines of code that never get executed, as, after all, 100% code coverage is the objective!
  • They can help with (minimal) kernel configuration. Seeing that certain code paths are never taken perhaps implies you don't require the configuration that uses them (this can be off the mark; take care to ensure it's really not required before disabling it).

Let's dig deeper into the area of interest here – the first point, debugging. To illustrate the point, we take a simple pseudocode example of an error code path within regular code:

p = kzalloc(n, GFP_KERNEL);
if (unlikely(!p)) {  [...] } // let's assume this alloc is fine
foo();  // assume it all goes well here
q = kzalloc(m, GFP_KERNEL);
if (unlikely(!q)) {  // if this allocation fails ...
   ret = do_cleanup_one();
   if (!ret) /* ... and if this is true, then we end up with a memory leak!!! */
      return -ENOSPC;
   kfree(p);
   return -ENOMEM;
}

If you don't have a (negative) test case where the value ret is NULL, then that code path – the one where we return an error value but fail to free the previously allocated memory buffer first – never gets run; therefore, it never gets tested. Then, even powerful dynamic analysis tools, such as KASAN, SLUB debug, kmemleak, and so on, cannot catch the leakage bug, as they never run the code path! This illustrates why 100% code coverage is key to a successful product or project.

Tip – Fault Injection

So, how exactly do we create a (negative) test case to test error paths (such as in the previous simple example)? Also, kernel-level allocations via the slab cache (kmalloc(), kzalloc(), and similar), pretty much never fail, yet we're taught to always check and write code for the failure case (there are corner cases where they can fail; please, always check for the failure case!); but how do we test that code? The kernel has a fault-injection framework to help with precisely this! It's important as only when you run the code can you catch potential bugs (except for static analyzers). The official kernel documentation covers the kernel fault-injection framework in detail (Fault injection capabilities infrastructure: https://www.kernel.org/doc/html/latest/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure); do check it out, and look in the Further reading section for more on this topic.

Although gcov is a userspace tool, it's also used for Linux kernel (and module) coverage analysis. When used in the context of the Linux kernel, the gcov coverage data is read off debugfs pseudofiles (under /sys/kernel/debug/gcov). The mechanics of using gcov to perform kernel-level code coverage are definitively covered in the official kernel documentation here: https://www.kernel.org/doc/html/latest/dev-tools/gcov.html#using-gcov-with-the-linux-kernel. Tools such as lcov are frontends to gcov; they provide useful features such as generating HTML-based code coverage reports (they work in the usual manner, whether used for user or kernel-space reporting).

As experienced folk in the industry know, many customers' service level agreements (SLAs) or contracts will mandate 100% (or close to it) code coverage being documented and signed off.

A brief note on kernel testing

Testing/QA is a key part of the software process. Although the aphorism testing can reveal the presence of bugs but not their absence is, unfortunately, true, giving testing its due, by using state-of-the-art Linux kernel testing tools and frameworks, you can indeed root out (and thus let you subsequently fix) many, if not most, OS- and driver-level bugs. It's a key thing to do; ignore testing at your peril!

As explained in the official kernel documentation (Kernel Testing Guide: https://www.kernel.org/doc/html/latest/dev-tools/testing-overview.html#kernel-testing-guide), there are two major types of test infrastructure within the Linux kernel, which differ in how they're used. Besides them, a technique called fuzzing turns out to be a key and powerful means to catch those difficult-to-tease-out bugs; read on!

Linux kernel selftests (kselftest)

This is a collection of user-mode scripts and programs (with a few modules thrown in as well); you'll find them within the kernel source tree under tools/testing/selftests. The approach here is more of a black box one; kselftest is appropriate when testing or verifying large-ish features of the kernel, using well-defined user-to-kernel interfaces (system calls, device nodes, pseudofiles, and such). To see how to use kselftest and run it, refer to the official kernel documentation here: https://www.kernel.org/doc/html/latest/dev-tools/kselftest.html#linux-kernel-selftests.

Linux kernel unit testing (KUnit)

These tend to be smaller self-contained test cases that are part of the kernel code and so understand internal kernel data structures and functions (therefore, more in line with unit testing). We've already covered using KUnit test cases to test the powerful KASAN memory checker; refer back to Chapter 5, Debugging Kernel Memory Issues – Part 1, and the Using the kernel's KUnit test infrastructure to run KASAN test cases section. KUnit is covered in depth (including how to write your own test cases) in the official kernel documentation here: https://www.kernel.org/doc/html/latest/dev-tools/kunit/index.html#kunit-linux-kernel-unit-testing.

Test results are often generated in a well-known form – the Test Anything Protocol (TAP) format is used by apps and the kernel. There are going to be cases, though, where the original protocol doesn't align well with kernel requirements; so, the kernel community has evolved a kernel TAP (KTAP) format for reporting. The official kernel documentation has the details: https://docs.kernel.org/dev-tools/ktap.html#the-kernel-test-anything-protocol-ktap-version-1.

What is fuzzing?

There are other means by which both apps and kernel code can be (very effectively!) tested – a powerful one is called fuzzing. Essentially, fuzzing is a test technique, a framework, where the program under test (PUT) is fed (semi) randomized input (the monkey-on-the-keyboard technique!); this often leads to it failing and/or triggering bugs in subtle ways, not commonly caught by more traditional testing techniques. Fuzzing can be especially helpful in catching security vulnerabilities, which tend to be the typical memory bugs. (We covered these in some detail in Chapter 5, Debugging Kernel Memory Issues – Part 1, and Chapter 6, Debugging Kernel Memory Issues – Part 2).

There are many well-known fuzzers; among them are American Fuzzy Lop (AFL), Trinity, and syzkaller. For the Linux kernel, syzkaller (also known as syzbot or syzkaller robot) is perhaps the best-known de facto continuously running (unsupervised) fuzzer on the kernel codebase; it has already found and reported hundreds of bugs (https://github.com/google/syzkaller#documentation). The syzkaller web dashboard, showing reported bugs of the upstream kernel and other interesting statistics, is available here: https://syzkaller.appspot.com/upstream. Do check it out.

Where Does kcov Fit In?

Fuzzers internally mutate interesting test cases into many more test cases. To do their job well, they require good code coverage tooling so that they can prioritize which mutated test cases will likely yield the most interesting results. For the Linux kernel, this is where kcov comes in – it's a code coverage tool that "exposes kernel code coverage information in a form suitable for coverage-guided fuzzing (randomized testing)."

Would you like to learn more and even try some hands-on kernel fuzzing? Check out the following (quite non-trivial) exercise.

Exercise

Try out fuzzing a portion of the Linux kernel with AFL! To do so, read the excellent A gentle introduction to Linux Kernel fuzzing tutorial and follow along: https://blog.cloudflare.com/a-gentle-introduction-to-linux-kernel-fuzzing/. (Also see https://github.com/cloudflare/cloudflare-blog/blob/master/2019-07-kernel-fuzzing/README.md.)

Well, almost there! Let's round off this chapter with a few miscellaneous areas.

Miscellaneous – using journalctl, assertions, and warnings

The modern framework for system initialization on Linux is considered to be systemd (although, by now, it's been in use on Linux for over a decade). It's a very powerful framework, although it does have its share of detractors as well. One thing you'll notice regarding systemd is that it's a pretty invasive system! On many (if not most) Linux distributions, besides providing a robust initialization framework (via service units, targets, and such), systemd takes over many activities, replacing their original counterparts, such as system logging, the udevd userspace daemon service, network services startup/shutdown, core dump management, watchdog, and so on. Also, with systemd, apps can be carefully tuned to operate within specified system resource limits by leveraging the powerful kernel control groups (cgroups) framework.

Looking up system logs with journalctl

As our central topic is debugging, we'll briefly look at only the logging aspect of systemd. Logging is often a straightforward way to get insight into what exactly was happening on the system before a bug occurred (and, if lucky, even during and after).

A feature of systemd logging is that it maintains the logs of both userspace app (and system daemon) processes as well as the kernel log. By using its frontend to view and filter log messages – journalctl – we can pretty intuitively gain insight into what's going on at any moment. This is largely because journalctl automatically, and by default, displays all logs in chronological order – those of user-mode processes as well as those of all kernel components (the core kernel itself and drivers/modules), in short, all printk-type messages.

A quick, simple example of using journalctl is seen in Figure 12.1; this is on a BeagleBone Black running a custom Yocto-based (Poky) Linux:

Figure 12.1 – Truncated screenshot showing journalctl executing on a BeagleBone Black embedded Linux system (in a minicom Terminal window)

Figure 12.1 – Truncated screenshot showing journalctl executing on a BeagleBone Black embedded Linux system (in a minicom Terminal window)

By default, journalctl shows the entire log it has saved so far; therefore, the kernel messages right from when it was installed (on this particular system, you can see from the first line in Figure 12.1 that it's November 19, 2021).

Okay, so how do we see messages from this, the current boot? Easy:

$ journalctl –b

[...]

-- Journal begins at Fri 2021-11-19 17:19:31 UTC, ends at Mon 2022-05-09 05:00:09 UTC. --

Nov 19 17:19:31 mybbb kernel: Booting Linux on physical CPU 0x0

Nov 19 17:19:31 mybbb kernel: Linux version 5.14.6-yocto-standard (oe-user@oe-host) (arm-poky-linux-gnueabi-gcc (GCC) 11.2.0, GNU >

Nov 19 17:19:31 mybbb kernel: CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c5387d

[...]

In this example, the dates happen to be the same; try this on your x86_64 Linux system; it'll probably differ! The log message format is intuitive:

<timestamp> <hostname> <logger_id>:  <... log message ...>

Here's another truncated, partial screenshot (the messages from journalctl showing the transition from kernel to userspace during bootup):

Figure 12.2 – Partial screenshot showing handover from the kernel to the systemd process PID 1

Figure 12.2 – Partial screenshot showing handover from the kernel to the systemd process PID 1

Due to space constraints, we will skim over the details; I urge you to look up the man page on journalctl for all possible option switches and even some examples. It's very useful: https://man7.org/linux/man-pages/man1/journalctl.1.html.

As one more interesting example, how do we know how often this system has been rebooted or shutdown/restarted? The journalctl frontend makes it easy with the --list-boots option; here's some sample truncated output from our x86_64 Ubuntu virtual machine (VM):

$ journalctl --list-boots |head -n2

-82 cd5<...>37a Tue [...] IST—Tue 2022-01-25 ... IST

-81 6ea<...>0a1 Tue [...] IST—Tue 2022-01-25 ... IST

$ journalctl --list-boots |tail -n2

-1 093<...>cb3 Fri [...] IST—Fri 2022-05-06 ... IST

0 d72<...>a8b Fri [...] IST—Mon 2022-05-09 ... IST

This output informs us that this particular system has been booted 83 times. The integer value in the left-most column is how many boots ago, so the last boot (the current session) is the left column value 0 (the negative numbers imply earlier boots, in chronological order; so, -1 implies the boot before this one). Nice.

journalctl – a few useful aliases

There are just too many option switches to journalctl to discuss here. To keep it short but still useful, here are a few aliases to journalctl that you might find useful. I typically put these into a startup script and source them at login:

# jlog: current boot only, everything
alias jlog='journalctl -b --all --catalog --no-pager'
# jlogr: current boot only, everything, *reverse* chronological order
alias jlogr='journalctl -b --all --catalog --no-pager --reverse'
# jlogall: *everything*, all time; --merge => _all_ logs merged
alias jlogall='journalctl --all --catalog --merge --no-pager'
# jlogf: *watch* log, 'tail -f' mode
alias jlogf='journalctl -f'
# jlogk: only kernel messages, this boot
alias jlogk='journalctl -b -k --no-pager'

Using the journalctl -f variant can be particularly useful to literally watch logs as they appear in real time. Also, simply use the -k option switch to show kernel printks.

You can do more with journalctl; filtering logs based on flexibly stated since and/or until-type keywords. For example, let's say you want to see all logs since 11 a.m. today but only until an hour ago (let's say it's 1 p.m. now, lunch right?). You could do so like this:

journalctl –b --since 11:00 --until "1 hour ago"

There are several variations too; powerful stuff, indeed!

Assertions, warnings, and BUG() macros

Assertions are a way to test assumptions. In userspace, the assert() macro serves the purpose. The parameter to assert() is the Boolean expression to test – if true, execution continues as usual (within the calling process or thread); if false, the assertion fails. This makes it invoke the abort() function, causing the process to die accompanied by a noisy printf message conveying the fact that the assertion failed (it will display the filename and line number as well as the failing assertion's expression).

Assertions are in effect a code-level debug tool, helping us achieve something very important (that I have tried to emphasize throughout the book): do not make assumptions; be empirical. Assertions allow us to test those assumptions. As a silly example, let's say a signal handler within a process sets an integer x to the value 3; in another function, foo(), we're assuming it's set to 3. Hey, that can be dangerous! Instead, we test our assumption with an assertion and then proceed on our merry way:

static int foo(void) {
     assert(x == 3);
     bar(); [...]
}

Now, you can see that an assertion is a way to say what you expect; if what's expected doesn't actually happen at runtime, you'll be notified! That's very useful.

So, why don't we use the same idea within the kernel? Wouldn't that be useful? It would, but there's a problem: we can't realistically have the kernel abort if the assertion fails, can we? Well, actually, we can: it's what macros such as BUG_ON() (and friends) do. So, some kernel/driver authors write their own version, in effect, a custom assert macro; here's an example (from a block driver named sx8):

// drivers/block/sx8.c
#define assert(expr) 
        if(unlikely(!(expr))) {                                   
        printk(KERN_ERR "Assertion failed! %s,%s,%s,line=%d
", 
    #expr, __FILE__, __func__, __LINE__);          
        }

Nice and simple, and an effective way to check assumptions! This driver invokes its custom assert macro a few times; here's one example:

assert(host->state == HST_PORT_SCAN);

Exercise

Look up the kernel code for the definition of BUG_ON(). You'll see it's a macro that invokes the BUG() macro when the condition comes true. Guess what? The (arch-specific) BUG() macro typically invokes a printk specifying the location of the code and then calls panic("BUG!").

Don't lightly invoke any of the BUG*() macros; you only call them when you have an unrecoverable situation, when there's no way out, when you must panic. A better alternative, perhaps, is using one of the many WARN*() type macros found within the kernel; they cause a warning-level printk to be emitted to the kernel log when the condition (passed as a parameter) is true! Thus, the WARN*() macros are perhaps the closest built-in kernel equivalent to the user-mode assert() macro. Do realize, though, that even the WARN*() macros spell out that a significant situation exists within the kernel – again, don't invoke them unnecessarily!

Summary

How awesome! Congratulations on completing this, the final chapter, and this book!

Here, you got an introduction to some remaining kernel debugging approaches – things we perhaps mentioned but hadn't covered elsewhere. We began by mentioning the powerful kdump/crash framework. Kdump allows capturing the complete kernel image (the trigger typically being a kernel crash/Oops /panic), and the crash userspace utility helps you (post-mortem) analyze it.

Static analyzers can play a really useful role in discovering potential bugs and security vulnerabilities. Don't ignore them; learn to leverage them!

The importance of code coverage was then delved into for a bit (along with a brief mention of how the kernel's fault-injection framework helps in setting up negative test cases, having control actually going to those pesky and possibly buggy error code paths). We briefly examined the kernel testing framework landscape; you saw that the kernel selftests and KUnit frameworks are the typical ones used to cover a lot of ground. Don't forget the powerful fuzzing technique though – Google's syzbot (syzkaller robot) uses it to its advantage to automatically and continually fuzz the Linux kernel, teasing out many bugs!

We finished the chapter with a quick mention of how system (and app) logs can be examined and filtered using the powerful journalctl frontend. Testing your assumptions by employing a custom kernel-space assert macro, and a mention of using the WARN*() and BUG*() macros, completed the discussion here.

A key point I'd like to (re)emphasize upon completion of this book is one of Fred Brook's well-known aphorisms: there is no silver bullet. In effect, what's meant is that one tool or debugging technique or analysis type cannot and will not catch all possible bugs; use several. These include compiler warnings (-Wall and -Wextra), static and dynamic analyzers (KASAN and others), dynamic debug printks, kprobes, lockdep, KCSAN, ftrace and trace-cmd, KGDB/kdb, and custom panic handlers. Our so-called better Makefile (for example, https://github.com/PacktPublishing/Linux-Kernel-Debugging/blob/main/ch3/printk_loglevels/Makefile) tries to enforce exactly this discipline by having several targets. Take the trouble to use them!

So, you're at the end? No, it's really more like the beginning, but armed with precious, useful, and practical tools, techniques, tips, and knowledge (this is our sincere hope!). Go forth, my friend!

Further reading

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset