Task 1: Finalize the Acceptance Test Types

In this task, the initial acceptance testing type list is refined, and the actual tests to be performed are selected.

Acceptance testing is an optional user-run test that demonstrates the ability of the application to meet the user’s requirements. The motivation for this test is to demonstrate rather than be destructive, that is, to show that the system works. Less emphasis is placed on the technical issues and more on the question of whether the system is a good business fit for the end user. Users usually perform the test. However, the users sometimes define “special tests,” such as intensive stress or volume tests, to stretch the limits of the system even beyond what was tested during the system test.

Task 2: Finalize the Acceptance Test Schedule

In this task, the acceptance test schedule should be finalized. It includes the testing steps (and perhaps tasks), target begin dates and target end dates, and responsibilities. It should also describe how it will be reviewed, tracked, and approved. For acceptance testing, the test team usually consists of user representatives. However, the team test environment and test tool are probably the same as those used during system testing. A sample acceptance test schedule is shown in Table 21.1.

Task 3: Organize the Acceptance Test Team

The acceptance test team is responsible for designing and executing the tests, evaluating the test results, and reporting any defects to development, using the defect-tracking system. When development corrects defects, the test team retests the defects to validate the correction. The acceptance test team typically has representation from the user community, because this is their final opportunity to accept the system.

The acceptance test team is led by a test manager whose responsibilities include the following:

1. ■ Organizing the test team

2. ■ Establishing the test environment

3. ■ Organizing the testing policies, procedures, and standards

4. ■ Ensuring test readiness

5. ■ Working the test plan and controlling the project

6. ■ Tracking test costs

7. ■ Ensuring test documentation is accurate and timely

8. ■ Managing the team members

Task 4: Establish the Acceptance Test Environment

During this task, the acceptance test environment is finalized. Typically, the test environment for acceptance testing is the same as that for system testing. The purpose of the test environment is to provide the physical framework necessary for the testing activity. For this task, the test environment needs are established and reviewed before implementation.

The Business usually performs the user acceptance tests. Thus, it is important that the details of the acceptance test environment be communicated to them.

Task 5: Install Acceptance Test Tools

During this task, the acceptance test tools are installed and verified for readiness. A trial run of sample tool test cases and scripts should be performed to verify that the test tools are ready for the actual acceptance test. Typically, the acceptance testing tools are the same as the system level testing tools, but this needs to be confirmed between the Business and the QA department. Some other tool readiness considerations include the following:

1. ■ Test team tool training

2. ■ Tool compatibility with operating environment

3. ■ Ample disk space for the tools

4. ■ Maximizing the tool potentials

5. ■ Vendor tool help hotline

6. ■ Test procedures modified to accommodate tools

7. ■ Installing the latest tool changes

8. ■ Verifying the vendor contractual provisions

Task 1: Identify the System-Level Test Cases

Acceptance test cases are typically (but not always) developed by the end user and are not normally considered the responsibility of the development organization, because acceptance testing compares the system to its original requirements and the needs of the users. It is the final test for the end users to accept or reject the system. The end users supply the test resources and perform their own tests. They may or may not use the same test environment that was used during system testing. This depends on whether the test will be performed in the end user’s environment. The latter is the recommended approach.

Typically, the acceptance test consists of a subset of system tests that have already been designed during system testing. Therefore, the current task consists of identifying those system-level tests that will be used during acceptance testing.

Task 2: Design/Script Additional Acceptance Tests

In addition to the system-level tests to be rerun during acceptance testing, they may be “tweaked” with special conditions to maximize the acceptability of the system. For example, the acceptance test might require that a certain throughput be sustained for a period of time with acceptable response time tolerance limits; for example, 10,000 transactions per hour are processed with a mean response time of 3 seconds, with 90 percent less than or equal to 2 seconds. Another example might be that an independent user “off the street” sits down with the system and the document to verify that he can use the system effectively.

The user might also envision other tests not designed during system testing. These may become more apparent to the user than they would have been to the developer because the user knows the business requirements and is intimately familiar with the business operations. He or she might uncover defects that only a user would see. This also helps the user to get ready for the real installation and production.

The acceptance test design might even include the use of live data, because the acceptance of test results will probably occur more readily if it looks real to the user. There are also unusual conditions that might not be detected unless live data is used.

Task 1: Schedule/Conduct the Review

The acceptance test plan review should be scheduled well in advance of the actual review, and the participants should have the latest copy of the test plan.

As with any interview or review, it should contain certain elements. The first defines what will be discussed; the second discusses the details; the third summarizes; and the final element is timeliness. The reviewer should state up front the estimated duration of the review and set the ground rule that if the allotted time expires before completing all items on the agenda, a follow-on review will be scheduled.

The purpose of this task is for development and the project sponsor to agree and accept the system test plan. If there are any suggested changes to the test plan during the review, they should be incorporated into the test plan.

Task 2: Obtain Approvals

Approval is critical in a testing effort because it helps provide the necessary agreements among testing, development, and the sponsor. The best approach is with a formal sign-off procedure of an acceptance test plan. If this is the case, use the management approval sign-off forms. However, if a formal agreement procedure is not in place, send a memo to each key participant, including at least the project manager, development manager, and sponsor. Attach to the document the latest test plan, and point out that all feedback comments have been incorporated and that if you do not hear from them, it is assumed they agree with the plan. Finally, indicate that in a spiral development environment, the system test plan will evolve with each iteration but that you will include them in any modification.

Task 1: Regression Test the Acceptance Fixes

The purpose of this task is to retest the tests that discovered defects in the previous acceptance test cycle for this build. The technique used is regression testing. Regression testing detects spurious errors caused by software modifications or corrections.

A set of test cases must be maintained and made available throughout the entire life of the software. The test cases should be complete enough so that all the software’s functional capabilities are thoroughly tested. The question arises as to how to locate those test cases to test defects discovered during the previous test spiral. An excellent mechanism is the retest matrix.

As described earlier, a retest matrix relates test cases to functions (or program units). A check entry in the matrix indicates that the test case is to be retested when the function (or program unit) has been modified due to enhancements or corrections. The absence of an entry indicates that the test does not need to be retested. The retest matrix can be built before the first testing spiral, but needs to be maintained during subsequent spirals. As functions (or program units) are modified during a development spiral, existing or new test cases need to be created and checked in the retest matrix in preparation for the next test spiral. Over time with subsequent spirals, some functions (or program units) may be stable with no recent modifications. Selective removal of their check entries should be considered, and undertaken between testing spirals.

Task 2: Execute the New Acceptance Tests

The purpose of this task is to execute new tests that were created at the end of the previous acceptance test cycle. In the previous spiral, the testing team updated the function/GUI, system fragment, and acceptance tests in preparation for the current testing spiral. During this task, those tests are executed.

Task 3: Document the Acceptance Defects

During acceptance test execution, the results of the testing must be reported in the defect-tracking database. These defects are typically related to individual tests that have been conducted. However, variations to the formal test cases often uncover other defects. The objective of this task is to produce a complete record of the defects. If the execution step has been recorded properly, the defects have already been recorded on the defect-tracking database. If the defects are already recorded, the objective of this step becomes to collect and consolidate the defect information.

Tools can be used to consolidate and record defects, depending on the test execution methods. If the defects are recorded on paper, the consolidation involves collecting and organizing the papers. If the defects are recorded electronically, search features can easily locate duplicate defects.