Photo by Russell Lee. Library of Congress, Prints & Photographs Division, FSA-OWI Collection, Reproduction Number: LC-USF33-011632-M3.
Fighting fire of rice straw stack in rice field near Crowley, Louisiana. September 1938.
An Integration Build (9) or a Private System Build (8) are useful for verifying build-time integration issues. But even if the code builds, you still need to check for runtime issues that can cause you grief later. This verification is essential if you want to maintain an Active Development Line (5). This pattern addresses the decisions you need to make to validate a build.
You hope that you tested the code adequately before checking it in. The best way to do this is to run tests after every build and before you check something in to source control, but you need to decide which tests to run.
You can write tests that target the most critical or failure-prone parts of the code, but it is hard to develop complete tests.
You can play it safe and test everything you can think of, but it is time-consuming to run exhaustive tests, and your progress can be very slow. Also, if the test takes too long, you may lose focus and have a harder time correlating your recent changes with any problems. Long-running tests encourage larger-grained changes between testing. Unstructured and impromptu testing can help you discover new problems, but it may not have much of an effective yield.
Running detailed tests is time-consuming, but if you check in a change that breaks the system, you waste everyone's time. Rapid development and small-grained check-ins mean that you want the cost of precheck-in verification to be small.
Note
Subject each build to a smoke test that verifies that the application has not broken in an obvious way.
A smoke test should be good enough to catch “showstopper” defects but disregard trivial defects (McConnell 1996). The definition of “trivial” is up to the individual project, but you should realize that the goal of a smoke test is not the same as the goal of the overall quality assurance process.
The scope of the test need not be exhaustive. It should test basic functions and simple integration issues. Ideally, it should be automated so that there is little cost to do it. The smoke test should not replace deeper integration testing. A suite of unit tests can form the basis for the smoke test if nothing else is immediately available. Most important, these tests should be self-scoring. They should return a test status and not require manual intervention to see whether the test passed. An error may well involve some effort to discover the source, but discovering that there is an error should be automatic.
Running a smoke test with each build does not remove the responsibility for developers to test their changes before submitting them to the repository. Developers should run the smoke test manually before committing a change. A smoke test is most useful for bug fixes and for looking for inadvertent interactions between existing and new functionality. All code should be unit tested by the developer, and where reasonable, run through some scenarios in a system environment. A smoke test can also be run as part of the build process in concert with more thorough tests when the build is to be a release candidate.
When you add new basic functionality to a system, extend the smoke test to test this functionality as well. But do not put exhaustive tests that better belong in Unit Tests or Regression tests.
Daily Build and Smoke Test (20) (Coplien 1995) describes the role of the smoke test in maintaining quality. Having a smoke test as part of a daily build is key to establishing Named Stable Bases (20), which form the basis for workspaces.
A smoke test should
Be quick to run, where “quick” depends on your specific situation
Be self-scoring, as any automated test should be
Provide broad coverage across the system that you care about
Be runnable by developers as well as part of the quality assurance process
The hardest part about a self-scoring test is to determine input/output relationships among elements of a complex system. You don't want the testing and scoring infrastructure to be buggy. You want the test to work with realistic data exchanged between parts of the system.
To get meaningful results from a smoke test, you need to work from a consistent build. A Private System Build (8) lets you build the system in a way that will give meaningful test results.
Canned inputs are fine as long as they are realistic enough. If your testing infrastructure is too complicated, you add risks around testing the test.
If the quality goals are such that you need to do exhaustive testing, consider using Task Branches, or have a different codeline policy. Also consider branching release lines.
A smoke test is an end-to-end test, more black box than white box.
A smoke test does leave gaps that should be filled by a more thorough QA procedure and Regression Test (15) suite to do more exhaustive testing to identify degradation in behavior. Developers should also develop a Unit Test (14) for every module they need. Use a Unit Test (14) to verify that the module you are changing still works adequately before you check in the change.
The trade-off we need to make involves the speed of check-in versus the thoroughness of the test. The longer the precheck-in test, the longer the check-in. Longer check-ins may encourage developers to have larger-granularity commits. This goes against an important goal of using version control.
Rapid Development (McConnell 1996) has some good advice on various testing strategies, including the trade-offs between completeness and speed.
The Mythical Man-Month (Brooks 1995) also has advice on making these trade-offs, and it is a classic book that every software developer should read.