Nat Pryce

Whether using test-driven development or not, programmers writing automated tests suffer from positive test bias:^{1, 2} they are more likely to test that the software behaves correctly when given valid input than that the software is robust when given invalid input. As a result, our test suites fail to detect entire classes of defects. Fuzz testing³ is an unreasonably effective technique for negative testing that is easy to include in existing automated test suites. Including fuzz tests in your test-driven development process will help you build more robust systems.

For example, we were extending the software of a widely used consumer product to fetch data from web services. Although we were careful to write robust networking code and test-drove negative as well as positive cases, fuzzing immediately uncovered a surprising number of inputs that would make the software throw unexpected exceptions. Many of the standard Java APIs that parse data throw unchecked exceptions, so the type checker hadn’t been able to ensure that the application handled all possible parsing errors. These unexpected exceptions could leave the device in an unknown state. In a consumer device, even one that can be updated remotely, that can mean an expensive increase in customer support calls or engineer callouts.

A fuzz test generates many random inputs, feeds them into the software under test, and checks that the software continues to exhibit acceptable behavior. To provide useful coverage, a fuzzer must generate inputs that are valid enough not to be rejected by the software immediately, but invalid enough to uncover corner cases that are not covered or defects in error-handling logic.

There are two ways to approach this:

• Mutation-based fuzzers mutate examples of good input to create possibly invalid test inputs.

• Generation-based fuzzers generate inputs from a formal model, such as a grammar, that defines the structure of valid inputs.

Mutation-based fuzzers are considered impractical for black box testing because it is difficult to obtain enough samples of valid input.⁴ However, when we test-drive our code, the positive test cases provide a ready-made collection of valid inputs that exercise many of the control paths in the software. Mutation-based fuzzing becomes not just practical, but easy to apply.

Running thousands of random inputs through the entire system can take a long time. Again, if we fuzz during development, we can fuzz test particular functions of our system and design them so they can be tested in isolation. We then use fuzzing to check the correct behavior of those units and type checking to ensure that they compose correctly with the rest of the system.

Here’s an example fuzz test that, along with the type checker, ensures a JSON message parser will throw only the checked exceptions declared in its signature:

@Test public void
only_throws_declared_exceptions_on_unexpected_json() {
  JsonMutator mutator = new JsonMutator();
  mutator.mutate(validJsonMessages(), 1000)
    .forEach(possiblyInvalidJsonMessage -> {
      try {
        // we don't care about the parsed result in this test
        parseJsonMessage(possiblyInvalidJsonMessage);
      }
      catch (FormatException e) {
        // allowed
      }
      catch (RuntimeException t) {
        fail("unexpected exception: " + t +
             " for input: " + possiblyInvalidJsonMessage);
      }
    });
}

Fuzz testing is now an essential part of my test-driven development toolbox. It helps eliminate defects and guides the design of the system to be more compositional.

A simple library for doing mutation-based fuzz testing in Java and Kotlin projects is available on GitHub.