To refresh our memory, here is the help screen from our membugs program:

$ ./membugs_dbg_asan 
Usage: ./membugs_dbg_asan option [ -h | --help]
 option =  1 : uninitialized var test case
 option =  2 : out-of-bounds : write overflow [on compile-time memory]
 option =  3 : out-of-bounds : write overflow [on dynamic memory]
 option =  4 : out-of-bounds : write underflow
 option =  5 : out-of-bounds : read overflow [on compile-time memory]
 option =  6 : out-of-bounds : read overflow [on dynamic memory]
 option =  7 : out-of-bounds : read underflow
 option =  8 : UAF (use-after-free) test case
 option =  9 : UAR (use-after-return) test case
 option = 10 : double-free test case
 option = 11 : memory leak test case 1: simple leak
 option = 12 : memory leak test case 2: leak more (in a loop)
 option = 13 : memory leak test case 3: "lib" API leak
-h | --help : show this help screen
$ 

Test case #1: UMR

Let's try the very first one—the uninitialized variable read test case:

$ ./membugs_dbg_asan 1
false case
$ 

It did not catch the bug! Yes, we have hit upon ASan's limitation: AddressSanitizer cannot catch UMR on statically (compile-time) allocated memory. Valgrind did.

Well, that's taken care of by the MSan tool; its specific job is to catch UMR bugs. The documentation states that MSan can only catch UMR on dynamically allocated memory. We found that it even caught a UMR bug on statically allocated memory, which our simple test case uses:

$ ./membugs_dbg_msan 1
==3095==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x496eb8 (<...>/ch5/membugs_dbg_msan+0x496eb8)
    #1 0x494425 (<...>/ch5/membugs_dbg_msan+0x494425)
    #2 0x493f2b (<...>/ch5/membugs_dbg_msan+0x493f2b)
    #3 0x7fc32f17ab96 (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #4 0x41a8c9 (<...>/ch5/membugs_dbg_msan+0x41a8c9)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (<...>/ch5/membugs_dbg_msan+0x496eb8) 
Exiting
$  

It has caught the bug; however, this time, though we have used a debug binary executable, built with the -g -ggdb flags, the usual filename:line_number information is missing in the stack trace. Actually, a method to obtain this is demonstrated in the next test case.

For now, no matter: this gives us a chance to learn another useful debug technique: objdump(1) is one of the toolchain utilities that can greatly help here (we can achieve similar results with tools such as readelf(1) or gdb(1)). We'll disassemble the binary executable with objdump(1) (-d switch, and, with source via the -S switch), and look within its output for the address where the UMR occurs:

SUMMARY: MemorySanitizer: use-of-uninitialized-value (<...>/ch5/membugs_dbg_msan+0x496eb8) 

As the output of objdump is quite large, we truncate it showing only the relevant portion:

$ objdump -d -S ./membugs_dbg_msan > tmp 

<< Now examine the tmp file >>

$ cat tmp

./membugs_dbg_msan: file format elf64-x86-64

Disassembly of section .init:

000000000041a5b0 <_init>:
  41a5b0: 48 83 ec 08 sub $0x8,%rsp
  41a5b4: 48 8b 05 ad a9 2a 00 mov 0x2aa9ad(%rip),%rax # 6c4f68 <__gmon_start__>
  41a5bb: 48 85 c0 test %rax,%rax
  41a5be: 74 02 je 41a5c2 <_init+0x12>

[...]

0000000000496e60 <uninit_var>:
{
  496e60: 55 push %rbp
  496e61: 48 89 e5 mov %rsp,%rbp
  int x; /* static mem */
  496e64: 48 83 ec 10 sub $0x10,%rsp
 [...]
  if (x)
  496e7f: 8b 55 fc mov -0x4(%rbp),%edx
  496e82: 8b 31 mov (%rcx),%esi
  496e84: 89 f7 mov %esi,%edi
  [...]
  496eaf: e9 00 00 00 00 jmpq 496eb4 <uninit_var+0x54>
  496eb4: e8 a7 56 f8 ff callq 41c560 <__msan_warning_noreturn>
  496eb9: 8a 45 fb mov -0x5(%rbp),%al
  496ebc: a8 01 test $0x1,%al
[...]

The closest match in the objdump output to the address provided by MSan as the 0x496eb8 error point is 0x496eb4. That's fine: just look at the preceding for the first source line of code; it's the following line:

   if (x)

Perfect. That's exactly where the UMR occurred!

Test Case #2: write overflow [on compile-time memory]

We run the membugs program, both under Valgrind and ASan, only invoking the write_overflow_compilemem() function to test the out-of-bounds write overflow memory errors on a compile-time allocated piece of memory.

Case 1: Using Valgrind
Notice how Valgrind does not catch the out-of-bounds memory bug:

$ valgrind ./membugs_dbg 2
==8959== Memcheck, a memory error detector
==8959== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==8959== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==8959== Command: ./membugs_dbg 2
==8959== 
==8959== 
==8959== HEAP SUMMARY:
==8959==     in use at exit: 0 bytes in 0 blocks
==8959==   total heap usage: 0 allocs, 0 frees, 0 bytes allocated
==8959== 
==8959== All heap blocks were freed -- no leaks are possible
==8959== 
==8959== For counts of detected and suppressed errors, rerun with: -v
==8959== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
$ 

This is because Valgrind is limited to working with only dynamically allocated memory; it cannot instrument and work with compile time allocated memory.

Case 2: Address Sanitizer

ASan does catch the bug:

A similar textual version is shown as follows:

$ ./membugs_dbg_asan 2
=================================================================
==25662==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff17e789f4 at pc 0x00000051271d bp 0x7fff17e789b0 sp 0x7fff17e789a8
WRITE of size 4 at 0x7fff17e789f4 thread T0
    #0 0x51271c (<...>/membugs_dbg_asan+0x51271c)
    #1 0x51244e (<...>/membugs_dbg_asan+0x51244e)
    #2 0x512291 (<...>/membugs_dbg_asan+0x512291)
    #3 0x7f7e19b2db96 (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #4 0x419ea9 (<...>/membugs_dbg_asan+0x419ea9)

Address 0x7fff17e789f4 is located in stack of thread T0 at offset 52 in frame
    #0 0x5125ef (/home/seawolf/0tmp/membugs_dbg_asan+0x5125ef)
[...]
SUMMARY: AddressSanitizer: stack-buffer-overflow (/home/seawolf/0tmp/membugs_dbg_asan+0x51271c) 
[...]
==25662==ABORTING
$ 

Notice, however, that within the stack backtrace, there is no filename:line# information. That's disappointing. Can we obtain it?

Yes indeed—the trick lies in ensuring a few things:

• Compile the application with the -g switch (to include debug symbolic info; we do this for all the *_dbg versions).
• Besides the Clang compiler, a tool called llvm-symbolizer must be installed as well. Once installed, you must figure out its exact location on the disk and add that directory to the path.
• At runtime, the ASAN_OPTIONS environment variable must be set to the symbolize=1 value.

Here, we rerun the buggy case with llvm-symbolizer in play:

$ export PATH=$PATH:/usr/lib/llvm-6.0/bin/
$ ASAN_OPTIONS=symbolize=1 ./membugs_dbg_asan 2
=================================================================
==25807==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffd63e80cf4 at pc 0x00000051271d bp 0x7ffd63e80cb0 sp 0x7ffd63e80ca8
WRITE of size 4 at 0x7ffd63e80cf4 thread T0
    #0 0x51271c in write_overflow_compilemem <...>/ch5/membugs.c:268:10
    #1 0x51244e in process_args <...>/ch5/membugs.c:325:4
    #2 0x512291 in main <...>/ch5/membugs.c:375:2
    #3 0x7f9823642b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #4 0x419ea9 in _start (<...>/membugs_dbg_asan+0x419ea9)
[...]
$ 

Now the filename:line# information shows up!

Clearly, ASan can and does instrument compile-time allocated as well as dynamically allocated memory regions, and thus catches both memory-type bugs.

Also, as we saw, it displays a call stack (read it from bottom to top of course). We can see the call chain is:

_start --> __libc_start_main --> main --> process_args --> 
             write_overflow_compilemem

The AddressSanitizer also displays, "Shadow bytes around the buggy address:"; here, we do not attempt to explain the memory-shadowing technique used to catch such bugs; if interested, please see the Further reading section on the GitHub repository.

Test case #3: write overflow (on dynamic memory)

As expected, ASan catches the bug:

$ ./membugs_dbg_asan 3
=================================================================
==25848==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000018 at pc 0x0000004aaedc bp 0x7ffe64dd2cd0 sp 0x7ffe64dd2480
WRITE of size 10 at 0x602000000018 thread T0
    #0 0x4aaedb in __interceptor_strcpy.part.245 (<...>/membugs_dbg_asan+0x4aaedb)
    #1 0x5128fd in write_overflow_dynmem <...>/ch5/membugs.c:258:2
    #2 0x512458 in process_args <...>/ch5/membugs.c:328:4
    #3 0x512291 in main <...>/ch5/membugs.c:375:2
    #4 0x7f93abb88b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #5 0x419ea9 in _start (<...>/membugs_dbg_asan+0x419ea9)

0x602000000018 is located 0 bytes to the right of 8-byte region [0x602000000010,0x602000000018) allocated by thread T0 here:
    #0 0x4d9d60 in malloc (<...>/membugs_dbg_asan+0x4d9d60)
    #1 0x512896 in write_overflow_dynmem <...>/ch5/membugs.c:254:9
    #2 0x512458 in process_args <...>/ch5/membugs.c:328:4
    #3 0x512291 in main <...>/ch5/membugs.c:375:2
    #4 0x7f93abb88b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[...]

With llvm-symbolizer in the path, the filename:line# information again shows up.

Attempting to compile for sanitizer instrumentation (via the -fsanitize= GCC switches) and trying to run the binary executable over Valgrind is not supported; when we try this, Valgrind reports the following:

$ valgrind ./membugs_dbg 3
==8917== Memcheck, a memory error detector
==8917== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==8917== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==8917== Command: ./membugs_dbg 3
==8917== 
==8917==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
[...]

Test Case #8: UAF (use-after-free). Take a look at the following code:

$ ./membugs_dbg_asan 8
uaf():162: arr = 0x615000000080:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
=================================================================
==25883==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000000080 at pc 0x000000444b14 bp 0x7ffde4315390 sp 0x7ffde4314b40
WRITE of size 22 at 0x615000000080 thread T0
    #0 0x444b13 in strncpy (<...>/membugs_dbg_asan+0x444b13)
    #1 0x513529 in uaf <...>/ch5/membugs.c:172:2
    #2 0x512496 in process_args <...>/ch5/membugs.c:344:4
    #3 0x512291 in main <...>/ch5/membugs.c:375:2
    #4 0x7f4ceea9fb96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #5 0x419ea9 in _start (<...>/membugs_dbg_asan+0x419ea9)

0x615000000080 is located 0 bytes inside of 512-byte region [0x615000000080,0x615000000280)
freed by thread T0 here:
    #0 0x4d9b90 in __interceptor_free.localalias.0 (<...>/membugs_dbg_asan+0x4d9b90)
    #1 0x513502 in uaf <...>/ch5/membugs.c:171:2
    #2 0x512496 in process_args <...>/ch5/membugs.c:344:4
    #3 0x512291 in main <...>/ch5/membugs.c:375:2
    #4 0x7f4ceea9fb96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

previously allocated by thread T0 here:
    #0 0x4d9d60 in malloc (<...>/membugs_dbg_asan+0x4d9d60)
    #1 0x513336 in uaf <...>/ch5/membugs.c:157:8
    #2 0x512496 in process_args <...>/ch5/membugs.c:344:4
    #3 0x512291 in main <...>/ch5/membugs.c:375:2
    #4 0x7f4ceea9fb96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

SUMMARY: AddressSanitizer: heap-use-after-free (<...>/membugs_dbg_asan+0x444b13) in strncpy
[...]

Super. ASan not only reports the UAF bug, it even reports exactly where the buffer was allocated and freed! Powerful stuff.

Test case #9: UAR

For the purpose of this example, let's say we compile the membugs program in the usual manner, using gcc. Run the test case:

$ ./membugs_dbg 2>&1 | grep -w 9
 option =  9 : UAR (use-after-return) test case
$ ./membugs_dbg_asan 9
res: (null)
$ 

ASan, as such, does not catch this dangerous UAR bug! As we saw earlier, neither does Valgrind. But, the compiler does emit a warning!

Hang on, though: the Sanitizers documentation mentions that ASan can indeed catch this UAR bug, if:

• clang (ver r191186 onward) is used to compile the code (not gcc)
• A special flag, detect_stack_use_after_return is set to 1

So, we recompile the executable via Clang (again, we assume the Clang package is installed). In reality, our Makefile does make use of clang for all the membugs_dbg_* builds. So, ensure we rebuild with Clang as the compiler and retry:

$ ASAN_OPTIONS=detect_stack_use_after_return=1 ./membugs_dbg_asan 9
=================================================================
==25925==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f7721a00020 at pc 0x000000445b17 bp 0x7ffdb7c3ba10 sp 0x7ffdb7c3b1c0
READ of size 23 at 0x7f7721a00020 thread T0
    #0 0x445b16 in printf_common(void*, char const*, __va_list_tag*) (<...>/membugs_dbg_asan+0x445b16)
    #1 0x4465db in vprintf (<...>/membugs_dbg_asan+0x4465db)
    #2 0x4466ae in __interceptor_printf (<...>/membugs_dbg_asan+0x4466ae)
    #3 0x5124b9 in process_args <...>/ch5/membugs.c:348:4
    #4 0x512291 in main <...>/ch5/membugs.c:375:2
    #5 0x7f7724e80b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #6 0x419ea9 in _start (/home/seawolf/0tmp/membugs_dbg_asan+0x419ea9)

Address 0x7f7721a00020 is located in stack of thread T0 at offset 32 in frame
    #0 0x5135ef in uar <...>/ch5/membugs.c:141

  This frame has 1 object(s):
    [32, 64) 'name' (line 142) <== Memory access at offset 32 is inside this variable
[...]

It does work. As we showed in Test case #1: UMR, one can further make use of objdump(1) to tease out the exact place where the bug hits. We leave this as an exercise for the reader.

More information on how ASan detects stack UAR can be found at https://github.com/google/sanitizers/wiki/AddressSanitizerUseAfterReturn.

Test case #10: double free

The test case for this bug is kind of interesting (refer to the membugs.c source); we perform malloc, free the pointer, then perform another malloc with such a large value (-1UL, which becomes unsigned and thus too big) that it's guaranteed to fail. In the error-handling code, we (deliberately) free the pointer we already freed earlier, thus generating the double free test case. In simpler pseudocode:

ptr = malloc(n);
strncpy(...);
free(ptr);

bogus = malloc(-1UL); /* will fail */
if (!bogus) {
     free(ptr);  /* the Bug! */
     exit(1);
}

Importantly, this kind of coding reveals another really crucial lesson: developers often do not pay sufficient attention to error-handling code paths; they may or may not write negative test cases to test them thoroughly. This could result in serious bugs!

Running this via ASan instrumentation does not, at first, have the desired effect: you will see that because of the glaringly enormous malloc failure, ASan actually aborts process-execution; hence, it does not detect the real bug we're after—the double free:

$ ./membugs_dbg_asan 10
doublefree(): cond 0
doublefree(): cond 1
==25959==WARNING: AddressSanitizer failed to allocate 0xffffffffffffffff bytes
==25959==AddressSanitizer's allocator is terminating the process instead of returning 0
==25959==If you don't like this behavior set allocator_may_return_null=1
==25959==AddressSanitizer CHECK failed: /build/llvm-toolchain-6.0-QjOn7h/llvm-toolchain-6.0-6.0/projects/compiler-rt/lib/sanitizer_common/sanitizer_allocator.cc:225 "((0)) != (0)" (0x0, 0x0)
    #0 0x4e2eb5 in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (<...>/membugs_dbg_asan+0x4e2eb5)
    #1 0x500765 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (<...>/membugs_dbg_asan+0x500765)
    #2 0x4e92a6 in __sanitizer::ReportAllocatorCannotReturnNull() (<...>/membugs_dbg_asan+0x4e92a6)
    #3 0x4e92e6 in __sanitizer::ReturnNullOrDieOnFailure::OnBadRequest() (<...>/membugs_dbg_asan+0x4e92e6)
    #4 0x424e66 in __asan::asan_malloc(unsigned long, __sanitizer::BufferedStackTrace*) (<...>/membugs_dbg_asan+0x424e66)
    #5 0x4d9d3b in malloc (<...>/membugs_dbg_asan+0x4d9d3b)
    #6 0x513938 in doublefree <...>/ch5/membugs.c:129:11
    #7 0x5124d2 in process_args <...>/ch5/membugs.c:352:4
    #8 0x512291 in main <...>/ch5/membugs.c:375:2
    #9 0x7f8a7deccb96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #10 0x419ea9 in _start (/home/seawolf/0tmp/membugs_dbg_asan+0x419ea9)

$  

Yes, but, notice the preceding line of output that says:

[...] If you don't like this behavior set allocator_may_return_null=1 [...]

How do we tell ASan this? An environment variable, ASAN_OPTIONS, makes it possible to pass runtime options; looking them up (recall we have provided the documentation links to the sanitizer toolset), we use it like so (one can pass more than one option simultaneously, separating options with a :; for fun, we also turn on the verbosity option, but trim the output):

$ ASAN_OPTIONS=verbosity=1:allocator_may_return_null=1 ./membugs_dbg_asan 10
==26026==AddressSanitizer: libc interceptors initialized
[...]
SHADOW_OFFSET: 0x7fff8000
==26026==Installed the sigaction for signal 11
==26026==Installed the sigaction for signal 7
==26026==Installed the sigaction for signal 8
==26026==T0: stack [0x7fffdf206000,0x7fffdfa06000) size 0x800000; local=0x7fffdfa039a8
==26026==AddressSanitizer Init done
doublefree(): cond 0
doublefree(): cond 1
==26026==WARNING: AddressSanitizer failed to allocate 0xffffffffffffffff bytes
membugs.c:doublefree:132: malloc failed
=================================================================
==26026==ERROR: AddressSanitizer: attempting double-free on 0x615000000300 in thread T0:
    #0 0x4d9b90 in __interceptor_free.localalias.0 (<...>/membugs_dbg_asan+0x4d9b90)
    #1 0x5139b0 in doublefree <...>/membugs.c:133:4
    #2 0x5124d2 in process_args <...>/ch5/membugs.c:352:4
    #3 0x512291 in main <...>/ch5/membugs.c:375:2
    #4 0x7fd41e565b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #5 0x419ea9 in _start (/home/seawolf/0tmp/membugs_dbg_asan+0x419ea9)

0x615000000300 is located 0 bytes inside of 512-byte region [0x615000000300,0x615000000500) freed by thread T0 here:
    #0 0x4d9b90 in __interceptor_free.localalias.0 (<...>/membugs_dbg_asan+0x4d9b90)
    #1 0x51391f in doublefree <...>/ch5/membugs.c:126:2
    #2 0x5124d2 in process_args <...>/ch5/membugs.c:352:4
    #3 0x512291 in main <...>/ch5/membugs.c:375:2
    #4 0x7fd41e565b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

previously allocated by thread T0 here:
    #0 0x4d9d60 in malloc (<...>/membugs_dbg_asan+0x4d9d60)
    #1 0x51389d in doublefree <...>/ch5/membugs.c:122:8
    #2 0x5124d2 in process_args <...>/ch5/membugs.c:352:4
    #3 0x512291 in main <...>/ch5/membugs.c:375:2
    #4 0x7fd41e565b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

SUMMARY: AddressSanitizer: double-free (<...>/membugs_dbg_asan+0x4d9b90) in __interceptor_free.localalias.0
==26026==ABORTING
$ 

This time, ASan continues even though it hits an allocation failure, and thus finds the real bug—the double free.

Test case #11: memory leak test case 1—simple leak. Refer to the following code:

$ ./membugs_dbg_asan 11
leakage_case1(): will now leak 32 bytes (0 MB)
leakage_case1(): will now leak 1048576 bytes (1 MB)

=================================================================
==26054==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 1048576 byte(s) in 1 object(s) allocated from:
    #0 0x4d9d60 in malloc (<...>/membugs_dbg_asan+0x4d9d60)
    #1 0x513e34 in amleaky <...>/ch5/membugs.c:66:8
    #2 0x513a79 in leakage_case1 <...>/ch5/membugs.c:111:2
    #3 0x5124ef in process_args <...>/ch5/membugs.c:356:4
    #4 0x512291 in main <...>/ch5/membugs.c:375:2
    #5 0x7f2dd5884b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

Direct leak of 32 byte(s) in 1 object(s) allocated from:
    #0 0x4d9d60 in malloc (<...>/membugs_dbg_asan+0x4d9d60)
    #1 0x513e34 in amleaky <...>/ch5/membugs.c:66:8
    #2 0x513a79 in leakage_case1 <...>/ch5/membugs.c:111:2
    #3 0x5124e3 in process_args <...>/ch5/membugs.c:355:4
    #4 0x512291 in main <...>/ch5/membugs.c:375:2
    #5 0x7f2dd5884b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

SUMMARY: AddressSanitizer: 1048608 byte(s) leaked in 2 allocation(s).
$ 

It does find the leak, and pinpoints it. Also notice that LeakSanitizer (LSan) is effectively a subset of ASan.

Test case #13: memory leak test case 3—libAPI leak

Here is a screenshot showcasing the action when ASan (under the hood, LSan) catches the leak:

Well caught!