When reading about software testing, we often encounter types of testing that include such samples as black box testing, white box testing, unit testing, system testing, regression testing, mutation testing, stress testing, and so on. The trouble with this list is that the qualifiers that we put before testing refer to different attributes of the testing activity: depending on the case, they may refer to a test data selection criterion, or to the scale of the asset under test, or to the assumptions of the test activity, or to the product attribute being tested, and so on. The purpose of this chapter is to classify software testing activities in a systematic manner, using orthogonal dimensions; like all classifications schemes, ours aims to capture in a simple abstraction a potentially complex set of related attributes. A software testing activity can be characterized by a number of interdependent attributes; in order to build an orthogonal classification scheme, we must select a set of independent attributes that is sufficiently small so that its elements are indeed independent, yet sufficiently large to cover all classes of interest. In Section 7.2, we introduce this classification scheme, by identifying the set of attributes that we use for the purpose of classification, along with the secondary attributes that depend on these. In Section 7.3, we consider a number of important testing activities, analyze how they can be projected on our classification scheme, and discuss what inferred attributes they have by virtue of their classification.
While all testing activities consist in executing a software product on selected input data, observing its outputs and analyzing the outcome of the experiment, there is a wide variance in how testing is conducted, depending, broadly, on the goal of the test, the asset under test, the circumstances of the test, and the assumptions being made about the product and its environment. We identify four independent attributes of a software testing activity, and seven dependent attributes. We refer to the first set as primary attributes and refer to the second set as secondary attributes.
In Sections 7.2.1 and 7.2.2, we review in turn the primary then the secondary attributes, by discussing what values each attribute may take, and any dependencies that each value entails.
We consider four primary attributes: the scale, the goal, the property, and the method. We review these in turn in the following:
We get one instance or another, depending on the oracle that we use for the test: To estimate reliability, we use the functional specifications of the software product; to estimate safety, we use the safety-critical requirements of the product; to estimate security, we use the security requirements of the product.
We get one instance or another, depending on the oracle that we use for the test.
Test data generation methods are usually divided into three broad families:
It is possible to map the goal of the test to the test data generation method in a nearly deterministic manner, as shown in the following table:
Method | Structural | Functional | Random |
Goal | |||
Finding and removing faults | √ | ||
Proving the absence of faults | √ | ||
Estimating the frequency of failure | √ | ||
Ensuring the infrequency of failure | √ | ||
Estimating fault density | √ | ||
Target Attribute: Most typically, one tests a software product to affect or estimate some functional quality of the product, such as correctness, reliability, safety, security, and so on; but as the following list indicates, one may also be interested in testing the product for a broad range of attributes.
In Section 7.2.2, we review the secondary attributes and discuss how they are affected by the choices made for the primary attributes reviewed in this section.
We consider the secondary attributes listed earlier and review the set of values that are available for each attribute, as well as how these values are impacted by the primary attributes.
The Oracle: If the target attribute of the test is an operational attribute, such as the response time of the product under normal workloads, or under exceptional workloads, then the oracle takes the form of an operational condition (a response time, or a function plotting the response time as a function of the workload). If the target attribute of the test is functional, then the oracle depends on whether the goal of the test is to find faults or to certify failure freedom. The following table highlights these dependencies.
The Test Life Cycle: Whereas in Chapter 3 we have presented a generic test lifecycle, we can imagine three variations thereof, which we present in the following text:
{testDataGeneration(D); // D: test data set;
T=empty; // T: reportwhile (not empty(D))
{d=removeFrom(D);
d’=P(d);
if (not(oracle(d,d’)) {add(d,T;}}analyze(T);}
In this cycle, the phases of test data generation, test execution, and test analysis take place sequentially.
{testDataGeneration(D); // D: test data set;
while (not empty(D))
{repeat {d=removeFrom(D); d’=P(d);}
until not(oracle(d,d’));
offLineAnalysis(d); // fault diagnosis and removal
}
}
{while (not completeTest())
{d=generateTestData();
d’=P(d);
if (oracle(d,d’)) {successfulTest(d);}
else {unsuccessfulTest(d);}
}
}
The following table shows how the value of this attribute may depend on the primary attributes of goal and method.
Test Assumptions: A test can be characterized by the assumptions it makes about the product under test and/or about the environment in which it runs. As such, this attribute can take three values, depending on the scale of the product being tested, as shown in the following table.
Assumptions | Scale | |||
Unit | Subsystem | System | ||
Test assumption | The oracle/specification of the unit is not in question. Only the unit’s correctness is. | Only the targeted subsystem is in question, not the remainder of the system. | The test environment mimics the product’s operating environment. | |
Test Completion: Test completion is the condition under which the test activity is deemed to achieve its goal. Such conditions are as follows:
The following table illustrates how this attribute depends on the goal of the test and the test data generation method.
Required Artifacts: Many artifacts may be needed to conduct a test, including any combination of the following artifacts:
This attribute depends on virtually all four primary attributes; for the sake of parsimony, we only show its dependence on the goal of testing and on the test data generation method.
Artifacts | Goals of testing | ||||
Fault removal | Proving absence of faults | Estimating frequency of failures | Ensuring infrequency of failures | ||
Test data generation method | Structural | Source + Executable + Function |
|||
Functional | Executable + Specification + Function |
Executable + Specification |
Source + Function + Specification + Signature + Usage pattern |
||
Random | Executable + Function + Signature + Usage pattern |
Executable + Signature + Usage pattern |
Source + Specification + Signature + Usage pattern |
||
Stakeholders: A stakeholder in a test is a party that has a role in the execution of the test, or has a role in the production of the software asset being tested, or has a stake in the outcome of the test. Possible stakeholders include the product developer, the product specifier, the product user, the quality assurance team, the verification and validation team, the configuration management team, and so on. The following table shows how this attribute depends on the goal of the test and the scale of the asset.
Stakeholders | Goals of testing | ||||||
Fault removal | Proving absence of faults | Estimating frequency of failures | Ensuring infrequency of failures | ||||
Scale | Unit | Unit developer | Unit developer, CM/QA team | ||||
Subsystem (maintenance) | Subsystem developer, maintenance engineer | Subsystem developer, maintenance engineer, CM/QA team | |||||
System | Verification and validation team Design team | Specifier team, design team, and end users | |||||
Test Environment: The environment of a test is the set of interfaces that the product under test interacts with as it executes. The following table shows the different values that this attribute may take, depending on the goal of the test and the scale of the software product under test.
Test Environment | Goals of testing | ||||||||
Fault removal | Proving absence of faults | Estimating frequency of failures | Ensuring infrequency of failures | ||||||
Scale | Unit | Development environment | Project configuration | ||||||
Subsystem (maintenance) | Software system | ||||||||
System | Development environment | Operating environment | Simulated operating environment | ||||||
Position in the Life Cycle: As we have seen in Chapter 3, several phases of the software life cycle include a testing activity. The software activity at each phase can be characterized by primary attributes; the following table shows how the goal of testing and the scale of the product under test determine the phase at which each test activity takes place.
Position in the Lifecycle | Goals of testing | ||||||
Fault removal | Proving absence of faults | Estimating frequency of failures | Ensuring infrequency of failures | ||||
Scale | Unit | Unit testing | Adding the asset into the project configuration | ||||
Subsystem (maintenance) | Maintenance | Regression testing | |||||
System | Integration testing | Acceptance testing | Reliability estimation | Reliability growth | |||
In this section, we consider a number of different test activities, analyze them, and discuss to what extent the classification scheme presented in this chapter enables us to characterize them in a meaningful manner.
We distinguish between two types of unit-level testing:
The following table illustrates how these two tests differ from each other, by comparing and contrasting their attributes.
Attributes | Unit-level fault removal | Unit-level certification | |
Primary attributes | Scale | Unit (module, routine, function) | Unit (module, routine, function) |
Goal | Finding and removing faults | Certifying compliance with project-wide quality standards | |
Property | Functionality | Functionality | |
Method | Structural (attempting to sensitize and expose as many faults as possible) | Functional (attempting to exercise as many functional aspects as possible) | |
Secondary attributes | Oracle | The function that the unit is designed to compute | The specification that the unit is designed to satisfy |
Test life cycle | Semisequential | Sequential (generate test data, run the unit on the test data, and record outcomes, rule on certification) | |
Test assumptions | The intended function is not in question (the correctness of the unit is) | The unit specification is not in question (the correctness of the unit is) | |
Completion criterion | Confidence that most egregious faults have been removed | Confidence that the unit has passed/ or has failed the certification standard | |
Required artifacts | Executable code + Source code + test environment + Intended function |
Executable code + test environment + Unit specification |
|
Test stakeholders | Unit developer | Unit developer + Configuration management/quality Assurance team |
|
Test environment | Simulated environment | Existing (evolving) system+ Simulated environment | |
Position in the SW life cycle | During the programming phase | Concludes the programming phase for each individual unit | |
We consider three system-level tests:
Even though these tests are all at the same scale (system-wide), they differ from each other in significant ways, as we see in the table below.
Attributes | Integration test | Reliability test | Acceptance test | |
Primary attributes | Scale | System | System | System |
Goal | Find and remove design faults (dealing with inter-component coordination) | Assess the reliability of the product | Check whether the system meets its requirements to the satisfaction of the user | |
Property | Design | Functionality | Functionality | |
Method | Structural | Functional (compatible with usage pattern) | Functional (as per user requirements) | |
Secondary attributes | Oracle | System function | System Specification (or subspecification with respect to which we want to estimate reliability) | System specification |
Test life cycle | Semisequential | Iterative | Sequential | |
Test assumptions | Units are not in question; only system design is | Test environment mimics operating environment | Test environment mimics operating environment | |
Completion criterion | All relevant interactions exercised, all possible faults removed | Reliability adequately estimated/ or reliability requirement met | Contractual obligation met | |
Required artifacts | Executable code + source code + design documentation + expected function |
Executable code + usage pattern +Relevant specification |
Executable code + contractual requirements |
|
Test stakeholders | Product designers | Product designers + verification and validation |
Requirements engineers + user representative + managers |
|
Test environment | Development environment | Operating environment (or simulation thereof) | Operating environment (or simulation thereof) | |
Position in the SW life cycle | Integration | Prior to delivery | At delivery | |
Alternative classification schemes for software testing can be found in the following references: Culbertson et al. (2002); Mathur (2008); Naik and Tripathy (2008); Perry (2002).