Photo by Alfred T. Palmer. Library of Congress, Prints & Photographs Division, FSA-OWI Collection, Reproduction Number: LC-USE6-D-007389.
An experimental scale model of the B-25 plane is prepared for wind tunnel tests in the Inglewood, California, plant of North American Aviation, Inc. October 1942.
A Smoke Test (13) is quick but not exhaustive. For it to be effective, you need to do the hard work of exhaustive testing as well. If you want to establish release candidates, you need to be sure that the code base is robust. This pattern explains how to generate builds that are no worse than the last build.
Software systems are complex, and with each change or enhancement to a system comes the possibility of breaking something seemingly unrelated to your changes. Fixing a defect has a substantial chance of introducing another (Brooks 1995). Without change, you can't make progress, but the impact of a change is hard to measure, especially in terms of how a unit of code interacts with the rest of the system.
You can exhaustively test your system after each build. Exhaustive testing takes time, but if you don't do this testing, you waste developer and perhaps customer time. If everyone runs exhaustive tests all the time, they will not be able to spend much time coding.
Even if you decide to do exhaustive system-level testing periodically on the code base, you are left with the problem of how to structure the tests. You can write tests from first principles by analyzing the inputs and outputs, but the payoff for writing tests like this on a system level may be small relative to the effort they take to write. You can also make intelligent guesses about what to test, but a software system is always full of surprises, especially in the ways it can fail.
When you solve the problem of which system tests to execute, you are still left with the problem of when to run them. Some integration tests may need resources that are not on every development machine. What is a good development environment may not always be the best test of the system. Reproducing some problems may require large data sets or multiple clients in a client-server system. You may not have the resources to run these sorts of tests all the time.
When the system does break, you want to identify the point in time when something broke. If you run the tests on every build or check-in to source control, you will be able to identify when something failed. But your testing may not keep up with your check-in and build process if the tests take long enough.
If your exhaustive tests find a problem and you fix it, you want to be sure that you can identify when this problem happens again because you don't want to waste time on known issues. A problem that happens once can happen again (this is what we mean by regression, after all). This means that we should accumulate a set of test cases as we discover problems, especially because we may not be able to guess all the problems ahead of time. These test cases can add up over time.
We need to be able to check for these recurring failure modes.
Note
Run regression tests on the system whenever you want to ensure stability of the codeline, such as before you release a build or before a particularly risky change. Create the regression tests from test cases the system has failed in the past.
Regression tests are end-to-end black-box tests that cover actual past or anticipated failure modes. A regression test can identify a system-level failure in the code base but may not necessarily identify what broke. When a regression test fails, debugging and unit tests may be necessary to determine what low-level component or interface broke.
Regression tests test changes in integration behavior. They are large-grained and test for unexpected consequences of integrating software components. Unit tests can be thought through fairly easily. As you add component interactions, it is harder to write tests based on first principles.
Build regression test cases from
Problems that you find in the prerelease QA process
Customer- and user-reported problems
System-level tests based on requirements
As you discover problems, write a test that reproduces the problem and add that scenario to the test. Over time you will end up with a large suite of tests that covers your most likely problem areas. Each problem may involve more than one test case. You can include running all unit tests in your regression testing, but it is better if the tests involve system input.
Regression testing is designed to make sure that the software has not taken a step backward (or regressed). Always run the same tests for each regression cycle. Add tests as you find more conditions or problematic items to test. Always add test cases to the regression test suite; if a problem happened once, it is likely to happen again, so remove test cases only for very well thought-through reasons.
Because regression tests can take a long time to run, you don't want to run them before every check-in or even after every build (unless resources permit). There are advantages, however, to having an automated procedure to run the regression test after each change so that you can identify the point at which the system regressed. Run the regression tests as part of the nightly build. Developers should also run a regression test before any significant sweeping change. If something breaks, you can always run the unit tests to localize the change. You also have to investigate if the unit test inputs no longer match the system. Institute a policy of automated regression testing tied to each release (Booch 1996).
Steve McConnell has a lot of information about testing of all kinds in Code Complete (McConnell 1993).
The Art of Software Testing, by Glen Myers (Myers 1979), is a classic.