Chapter 7
Static Testing the Requirements
The testing process should begin early in the application development life cycle, not just at the traditional testing phase at the end of coding. Testing should be integrated with the application development phases.
During the requirements phase of the software development life cycle, the business requirements are defined on a high level and are the basis of the subsequent phases and the final implementation. Testing in its broadest sense commences during the requirements phase (see Figure 7.1), which increases the probability of developing a quality system based on the user’s expectations. The result is that the requirements are verified to be correct and complete. Unfortunately, more often than not, poor requirements are produced at the expense of the application. Poor requirements ripple down the waterfall and result in a product that does not meet the user’s expectations. Some characteristics of poor requirements include the following:
■ Partial set of functions defined
■ Performance not considered
■ Ambiguous requirements
■ Security not defined
■ Interfaces not documented
■ Erroneous and redundant requirements
■ Requirements too restrictive
■ Contradictory requirements
Functionality is the most important part of the specification and should include a hierarchic decomposition of the functions. The reason for this is that it provides a description that is described in levels to enable all the reviewers to read as much detail as needed. Specifically, this will make the task of translating the specification to test requirements much easier.
Another important element of the requirements specification is the data description (see Appendix C, “Requirements Specification,” for more details). It should contain details such as whether the database is relational or hierarchical. If it is hierarchical, a good representation is a data model or entity relationship diagram in terms of entities, attributes, and relationships.
Another section in the requirements should be a description of the interfaces between the system and external entities that interact with the system, such as users, external software, or external hardware. A description of how users will interact with the system should be included. This would include the form of the interface and the technical capabilities of the users.
During the requirements phase, the testing organization needs to perform two functions simultaneously. It needs to build the system/acceptance test plan and also verify the requirements. The requirements verification entails ensuring the correctness and completeness of the documentation prepared by the development team.
Testing the Requirements with Ambiguity Reviews
An Ambiguity Review, developed by Richard Bender from Bender RBT, Inc., is a very powerful testing technique that eliminates defects in the requirements phase of the software life cycle, thereby avoiding those defects from propagating to the remaining phases of the software development life cycle. A QA Engineer trained in the technique performs the Ambiguity Review. The Engineer is not a domain expert (SME), and is not reading the requirements for content, but only to identify ambiguities in the logic and structure of the wording. The Ambiguity Review takes place after the requirements, or section of the requirements, reach first draft, and prior to them being reviewed for content, i.e. correctness and completeness by domain experts. The Engineer identifies all ambiguous words and phrases on a copy of the requirements. A summary of the findings is presented to the Business Analyst.
The Ambiguity Review Checklist identifies 15 common problems that occur in writing requirements.
Testing the Requirements with Technical Reviews
A software technical review is a form of peer review in which a team of qualified personnel examines the suitability of the software product for its intended use and identifies descrepancies from specifications and standards. Technical reviews may also provide recommendations of alternatives and examiniation of various alternatives. Technical reviews differ from software walkthroughs in its specific focus is on the technical quality of the product reviews. It differs from a software inspection in its ability to suggest direct alterations to the product reviewed, and its lack of a direct focus on training and process improvements (Source: Std. 1028-1997, IEEE Standard for Software Reviews, clause 3.7).
Inspections and Walkthroughs
These are formal techniques to evaluate the documentation form, interface requirements, and solution constraints as described in the previous section.
Checklists
These are oriented toward quality control and include questions to ensure the completeness of the requirements.
Methodology Checklist
This provides the methodology steps and tasks to ensure that the methodology is followed.
If the review is totally successful with no outstanding issues or defects discovered, the requirements specification is frozen, and any further refinements are monitored rigorously. If the review is not totally successful and there are minor issues during the review, the author corrects them. The corrections are reviewed by the moderator and signed off. On the other hand, if major issues and defects are discovered during the requirements review process, the defects are corrected; a new review then occurs with the same review members at a later time.
Table 7.1 Requirements Phase Defect Recording
Each defect uncovered during the requirements phase review should be documented. Requirement defect trouble reports are designed to assist in the proper recording of these defects. It includes the defect category and defect type. The description of each defect is recorded under the missing, wrong, or extra columns. At the conclusion of the requirements review, the defects are summarized and totaled. Table 7.1 shows a partial requirements phase defect recording form (see Appendix F1, “Requirements Phase Defect Checklist,” for more details).
Requirements Traceability Matrix
A requirements traceability matrix is a document that traces user requirements from analysis through implementation. It can be used as a completeness check to verify that all requirements are present or that there are no unnecessary/extra features, and as a maintenance guide for new personnel. At each step in the development cycle, the requirements, code, and associated test cases are recorded to ensure that the user requirement is addressed in the final system. Both the user and developer have the ability to easily cross-reference the requirements to the design specifications, programming, and test cases. (See Appendix E3, “Requirements Traceability Matrix,” for more details.)
Building the System/Acceptance Test Plan
Acceptance testing verifies that a system satisfies the user’s acceptance criteria. The acceptance test plan is based on the requirement specifications and is required in a formal test environment. This test uses black-box techniques to test the system against its specifications and is generally performed by the end user. During acceptance testing, it is important for the project team to coordinate the testing process and update the acceptance criteria, as needed. Acceptance testing is often combined with the system-level test plan, which is the case in this discussion.
The requirements phase is the first development phase that is completed before proceeding to the logical design, physical design, program unit design, and coding phases. During the requirements phase, it is not expected that all sections in the test plan will be completed, because not enough information is available.
In the Introduction section of the test plan (see Appendix E2, “System/Acceptance Test Plan”), the documentation of “first-cut” test activities begins. Included are the system description, the overall system description, acceptance test objectives, assumptions, risks, contingencies, and constraints. At this point, some thought about the appropriate authorities for the approval signatures begins.
The key parts in the Test Approach and Strategy section include: (1) the scope of testing, (2) test approach, (3) types of tests, (4) logistics, and (5) the regression policy. The scope of testing defines the magnitude of the testing effort, for example, whether to test the whole system or part of it. The testing approach documents the basis of the test design approach, for example, black-box, white-box, gray-box testing, incremental integration, and so on. The types of tests identify the test types, such as unit, integration, system, or acceptance, that will be performed within the testing scope. Details of the types of system-level tests may not be available at this point because of the lack of details, but will be available during the next phase. Logistics documents the working relationship between the development and testing organizations and other interested parties. It defines such issues as how and when the testing group will receive the software, and how defects will be recorded, corrected, and verified. The regression policy determines whether previously tested system functions perform properly after changes are introduced.
A major difficulty in testing the requirements document is that testers have to determine whether the problem definition has been translated properly to the requirements document. This requires envisioning the final product and coming up with what should be tested to determine that the requirement solves the problem.
A useful technique to help analyze, review, and document the initial cut at the functional decomposition of the system in the Test Specifications section is the requirement/test matrix (see Figure 7.2). This matrix defines the scope of the testing for the project and ensures that tests are specified for each requirement as documented in the requirements specification. It also helps identify the functions to be tested as well as those not to be tested.
Some benefits of the requirements/test matrix are that it:
Correlates the tests and scripts with the requirements
Facilitates status of reviews
Acts as a traceability mechanism throughout the development cycle, ex. requirement, test case(s), defect(s) linkage
The requirement/test matrix in Figure 7.2 documents each requirement and correlates it with the test cases and scripts to verify it. The requirements listed on the left side of the matrix can also aid in defining the types of system tests in the Test Approach and Strategy section.
It is unusual to come up with a unique test case for each requirement and, therefore, it takes several test cases to test a requirement thoroughly. This enables reusability of some test cases to other requirements. Once the requirement/test matrix has been built, it can be reviewed, and test case design and script building can commence.
The status column is used to track the status of each test case as it relates to a requirement. For example, “Q” in the status column can indicate that the requirement has been reviewed by QA, “U” can indicate that the users had reviewed the requirement, and “T” can indicate that the test case specification has been reviewed and is ready.
In the Test Specifications section of the test plan, information about the acceptance tests is available and can be documented. These tests must be passed for the user to accept the system. A procedure is a series of related actions carried out using an operational mode, that is, one that tells how to accomplish something. The following information can be documented in the Test Procedures section: test case, script, data development, test execution, correction, version control, maintaining test libraries, automated test tool usage, project management, monitoring, and status reporting.
It is not too early to start thinking about the testing personnel resources that will be needed. This includes the required testing skills, roles and responsibilities, the numbers and time required, and the personnel training needs.