Chapter 10. Testing Strategies

FAQ 10.01 What is the purpose of this chapter?

This chapter describes a systematic technique to pinpoint the root cause of a certain category of bugs.

The chapter is concerned with finding the root cause of problems, not just the symptoms. This is in contrast with most testing efforts, which focus on exposing symptoms but don't provide any formal help in locating the root cause of the problems.

This chapter also focuses on systematic techniques as opposed to ad hoc or luck-based debugging.

The basic idea is to bury various checks inside the objects so that the objects end up checking their own work; thus the notion of self-testing objects.

FAQ 10.02 What are the advantages of self-testing objects?

Testing starts earlier, continues longer, requires almost no human intervention, focuses on the most commonly used paths, and adapts as the system evolves.

There is very little human effort required for objects that test themselves other than writing the behavioral self-tests and the testInvariant() member function. The runtime system works a lot harder because it must continually reverify the object's state and its transitions, but there is very little human (payroll-intensive) intervention.

By integrating an object's test harnesses with the object, the self-testing strategy reduces reliance on big-bang testing. In practice, self-testing detects defects earlier than they otherwise would be with traditional, big-bang testing. This is because every use of the class becomes an impromptu test harness. This reduces the cost of finding and repairing defects, and it improves the business efficiency during system integration. The effect is to change the concept of testing from an event to a life style.

By building self-testing objects, developers ensure that their objects continually adapt as the system evolves. Thus, when the objects are used in new and unforeseen ways, the objects continue to verify themselves using the same test code and against the same standards without anyone having to reverse engineer them and build new test harnesses.

Since every use of an object becomes a miniature test harness, the overall effect is to exhaustively test the most commonly used paths through an object. This is quite a different result than that provided by most unit testing approaches, since most unit testing approaches require explicit and often elaborate test harnesses to be built, and these test harnesses typically provide spotty testing at best.

Finally, since self-testing objects check their own results, traditional test harnesses for unit testing are significantly simpler to develop. For example, the test harnesses don't need to check the result of an operation, since the object already checks its own results. This means that the test harnesses don't need to start out with some specific set of tests to be run but instead can generate test scenarios on the fly by passing randomly generated values into randomly selected member functions. This can both reduce the cost of building test harnesses for unit testing and improve their coverage.

The self-testing technique is similar to the quality mandate in manufacturing. The whole is correct because every part is correct and because every combination of parts is tested.

FAQ 10.03 What are some common excuses people use for not building self-testing into their objects?

Excuse: “The self-testing code is too trivial to worry about.” Reality: If it's that simple, then it will be easy to write.

Excuse: “The self-testing code is too complex.” Reality: If it's complex, then it's worth the trouble no matter how long it takes.

Excuse: “I can't afford the time to write the self-testing code.” Reality: You haven't done your job until your class's internals are documented.

Excuse: “Calling the self-testing code will consume too much CPU.” Reality: That's bogus. You can remove 100% of the runtime overhead by using #ifdef. Besides, if it's going to consume a lot of CPU, then either the self-testing checks are sophisticated or the class is very important and commonly used, both of which are arguments in favor of this approach, not against it.

Excuse: “The self-testing code might contain bugs, so the technique would be worthless.” Reality: Also bogus. If you're not sure that you can express the promise (postcondition) and invariant of your own class correctly, then those promises and invariants should be tested. The self-testing approach provides this for free since it simultaneously tests the class's documentation (its promises and invariants) at the same time as it tests the class's implementation.

FAQ 10.04 What will happen if techniques like those presented here are not used?

The maintenance programmers are doomed to crawl for miles over broken glass.

Here are some facts. First, in large systems the “distance,” as measured in statements executed, from where an error occurs to where it is detected by the program crashing or producing incorrect results is usually very large, sometimes millions of instructions. Second, one of the most effective techniques for reducing the time and pain associated with debugging is to minimize this distance, that is, discover the error as soon as possible after it has occurred. Third, there are usually thousands of small invariants that programs must maintain to be correct. Fourth, when programs are built using classes and objects, most of these little invariants are associated with the classes and objects.

Therefore, by collecting all these small and seemingly inconsequential invariants and attaching them to the right objects in a systematic fashion, it is possible to build a very robust system that tests itself. This, in turn, minimizes the amount of broken glass that has to be crawled through in trying to find that one fault that occurred a few million instructions ago and just caused the system to crash.

Some programmers build self-testing objects for the good of the team, others out of enlightened self-interest. Either way, self-testing objects should be built. After all, no one should have to crawl over broken glass when such a simple and obvious technique is available.

By the way, if you happen to be the developer who finds everyone else's bugs because you are methodical and they are cowboys, then you should be lobbying hard for your team to use these techniques. Either that or maybe you should just institute a policy that the cowboys on the team eat the glass you have to crawl through because they refuse to build self-testing objects.

FAQ 10.05 When is a class correct?

When it meets or exceeds its external agreements and abides by its internal constraints.

A class's external agreements include requirements imposed on users of the class and promises made to the users. This behavior is observable in the sense that it is expressed in terms of the class's public: member functions. This behavior can and should be tested. For example, just before a member function returns, the member function can check that it actually fulfilled its promises (a.k.a. postconditions; see FAQ 6.04). This is called behavioral self-testing, and it often involves adding checks at the bottom of the member function that check the member function's promises. These checks are normally put in an assert() statement or perhaps in an #ifdef; that way the checks can be removed if they cause too much performance degradation (see FAQ 10.06).

A class's internal constraints define the allowed states of data structures associated with objects of the class. Every object of the class must abide by these restrictions at all times. The class can and should test these class invariants (see FAQ 10.07).

FAQ 10.06 What is behavioral self-testing?

When an object checks its work before letting others see what happened.

The promises made by a member function can be encoded as a test that is executed at the end of the member function. For example, if the member function List::removeFirst() promises that List::size() will return one less than it did before, an explicit test to this effect can be made at the end of the List::removeFirst() member function. The code associated with behavioral self-tests can be wrapped in an #ifdef so that it can be easily removed or reinstalled as desired:

Naturally the assert(...) statements can be replaced by other assertion-checking techniques, if desired. The point is that the object checks its own results.

FAQ 10.07 What is a class invariant?

Stuff that's true whenever anyone else is looking.

The class invariant is the collection of boolean expressions that are always true for objects of the class. It is normal for a member or friend function (see FAQ 19.05) to temporarily violate the invariant, but the member or friend function must restore the invariant before returning to the user.

Here is the invariant for a Date class, encoded in the protected: testInvariant() member function.

The statement assert(month_ >= 1 && month_ <= 12); evaluates the conditional as a boolean. If month_ is out of range, the assertion fails and the assert() statement causes an error message to be printed and the program to be killed. During development, the debugger often opens at this point so that the programmer can determine exactly what went wrong and why. Compiling with the symbol NDEBUG defined (for example, via the -DNDEBUG option on many command-line driven compilers) causes the assert(...) code to vanish completely.

FAQ 10.08 Why should the invariant be captured explicitly?

As maintenance documentation and to catch bugs.

The class invariant should be recorded, if for no other reason than as documentation for future maintainers. Since a developer's job isn't done until the internal constraints of the data structure have been properly documented, there is really no choice. Someone somewhere has to write down these internal constraints.

If the internal constraints of the class's data structure are to be documented, the most natural and accessible place is along with the class's code. And source code is the most unambiguous way to express this documentation since expressing it in a natural language is relatively imprecise.

Plus, if the invariant is captured in a member function, the member function can be called at strategic moments during an object's life cycle, which effectively tests the documentation as well as testing the class's member functions. In other words, this technique makes sure that the invariant is correct and makes sure that the other member functions don't violate the invariant. If desired, these calls to the invariant can be placed in an #ifdef or an assert() so they can easily be removed before the software is shipped.

In cases where a class has a nontrivial invariant, practical experience has shown that this can catch a sizeable percentage of a class's bugs.

FAQ 10.09 When should the testInvariant() member function be called?

At the end of constructors to make sure the invariant was established and at the end of mutator member functions to make sure the invariant was maintained.

Rule 1:Every public: constructor must establish the invariant. Every public: constructor must initialize its object so that it passes the invariant test (this means avoiding any technique that allows the object to be initialized to garbage and requiring the user to call an init() member function). Thus every public: constructor should call testInvariant() as the last thing it does. Normally this call should be in an #ifdef or an assert() so that it can be easily removed or reinstalled as desired.

Rule 2:Every public: member function must maintain the invariant. Every public: member function may assume that its object passes the invariant test at the beginning and must restore its object's invariant by the time it returns. Thus every public: member function that mutates the object should call testInvariant() as the last thing it does. Normally this call should also be in an #ifdef or an assert() so that it can be easily removed or reinstalled as desired.

FAQ 10.10 What can be done to ensure that an object doesn't get blown away by a wild pointer?

Empower the object to test its invariant at the beginning of every public: member function and every friend function (see FAQ 19.05).

Wild pointers can corrupt a sleeping object. Wild pointers are the SCUD missiles of the software world—they are undirected terrorist instruments that wreak havoc in chaotic ways. Once a wild pointer has scribbled on an object, the object also exhibits chaotic behavior, often developing wild pointers of its own. The chain reaction spreads like a virus—each wild pointer infects a few more objects. Eventually one of the wild pointers attempts to scribble on something protected by the hardware and then the system crashes.

Once this chain reaction has occurred, programmers must rely on intuition and blind luck when looking for the root cause of the problem. We call this voodoo debugging, since it is about as effective as a fortune teller reading chicken entrails—indeed, the technology of reading entrails is remarkably similar to that of reading a core dump after corruption by wild pointers.

An object can help detect wild pointers by beginning all its public: member functions with a call to testInvariant(). This ensures that the object is still in a consistent state.

Since the assert(...) statements within the testInvariant() member function vanish when the symbol NDEBUG is defined, and since the testInvariant() member function is defined using the inline keyword, all the calls to testInvariant() will vanish when NDEBUG is defined. Normally it is not necessary to define the symbol NDEBUG until fairly late in the software development cycle, and some projects (particularly business applications that are not CPU bound) leave it on even after the software is deployed to its users.