8      IMPLEMENTING THE TESTS

So far we have planned our UAT exercise and created an overall strategy for the testing, which we then followed up by designing all the tests and preparing test scripts. We are now ready to implement the plan and do the testing.

In this chapter we will explain how we schedule all the tests to achieve our test strategy and assess the system against the acceptance criteria. To do this we will need to log all the testing, so that we can determine when testing is complete, and collect the data we need to make a judgement about the system’s status.

THE TESTING SCHEDULE

Test scheduling is just like any other scheduling; it puts activities in the right order and, ideally, estimates when each of them should happen.

Test scheduling is actually a very important step because:

•  It ensures that all the tests required by our test strategy are included.

•  It sequences the tests for the most efficient use of time and resources.

•  It sequences the tests to follow business processes.

•  It allocates testing resources to testing activities.

•  It enables us to log testing and keep track of progress.

•  It enables us to reschedule if we hit problems.

Test scheduling is to test execution what the test plan is to our overall strategy; it turns the principles into hard reality that we can manipulate, track and modify. We know from experience that no plan is ever complete and nothing ever goes completely to plan. Test scheduling is our opportunity to maintain control of the testing project in the face of change and to ensure that we are always making the best use of the time and resources available to us.

The high-level test schedule

The purpose of the high-level schedule is to fit together the large ‘building blocks’ required by the test strategy into a sequence that works for our strategy and makes best use of the time and resources available.

The high-level test schedule is based directly on the UAT strategy. For example if the strategy is risk-based, the high-level schedule will schedule all the high-risk test items early. If it is based on relative importance, the most important features will be tested first.

Test scheduling has to take account of a number of different factors that will affect what can be tested when. These factors include:

•  priority of tests;

•  availability of a test environment;

•  availability of testers.

Priority of tests

The priorities set by the UAT strategy determine the main sequence of testing but this may be amended by clashes in priority or by the unavailability of essential resources such as test environments and testers to run the tests.

Availability of a test environment

We need test environments to replace those parts of the real world or of the system to which we do not have access for some reason.

In some cases access to the real world is not feasible, for example a system that will operate interactively in real time across multiple time zones will require a complex, reliable and probably secure communications network. The network may already be in use or may not have been completed. In either case a test environment that duplicates the network would be impractical. A test environment may create a simulation of some aspect of the real world to enable testing or it may create a simplified version of the real hardware or software for testing. In the example either of these options may be too complex and costly, and some other alternative way of testing the system may have to be identified.

Where test environments are used throughout UAT there may be competition for the use of a given environment and testing will need to be scheduled to avoid or minimise the competition.

Availability of testers

Clearly testing requires testers to be available. It may be that specific skills or user background may be required for particular areas of testing and this will need to be taken into account when scheduling the tests.

The detailed test schedule

The detailed test schedule takes the building blocks from the high-level schedule and breaks them down into individual tests. The scheduled flow of individual tests may, however, be interrupted by the status of the software under test (SUT) at each stage. If any test failure identifies a defect in the SUT, the sequence will break down. Figure 8.1 shows the life cycle of a test

Figure 8.1 The life cycle of a test

Status of the software under test (SUT)

The SUT is the specific area of the system being tested at any particular time or in any particular test. As testing progresses the focus will shift from one area of the system to another.

At any given stage there may be outstanding incident reports, or defects may be under investigation or correction. The availability of the SUT scheduled for a given time is therefore never certain. As far as possible, scheduling should take account of the expected availability of the specific area of the SUT for which tests are being scheduled.

Anything that knocks a test out of the queue to be tested affects the schedule in one way or another. Figure 8.1 demonstrates the ways in which a scheduled test may generate a reschedule:

•  the need to reschedule if tests are blocked;

•  the need to retest after defects have been corrected;

•  the need for regression testing.

The need to reschedule if tests are blocked

If a serious defect is found there may be a need to prioritise the defect correction, and that may block tests that depended on having the defective module in place. It will also displace tests when the correction is scheduled for priority testing, necessitating a rejig of the schedule. There are many ways that a test can be blocked. When we have a block in our scheduled tests we have to reschedule to keep testing moving and fit the blocked test(s) in later.

The need to retest after defects have been corrected

Clearly any defects found are likely to lead to defect correction (though defect correction may be deferred if the defect is not too serious and the defect correction will not have a significant impact on completion of UAT). If defects are corrected, the correction will need to be tested; this will be testing that was not originally scheduled and so will necessitate some changes to the test schedules.

The need for regression testing

We can assume that some regression testing will be required during UAT but we cannot easily predict when. Two options are open to us: one is to plan a regression testing activity on a regular basis; the other is to do regression testing when it is required and reschedule around it.

Planned regression has the advantage that it does not cause rescheduling and, if no regression testing is required, it creates some ‘slack’ in the schedule that can absorb other reasons for change. The downside is that scheduled regression reduces the amount of time available for other testing. If we plan regression in as it is needed, we have to accept rescheduling of other tests, which may be inconvenient and may lead to delays. If we take this approach, we should assume that we will not complete all of the UAT initially planned (unless the timescale is extended) so we need to ensure the later tests are of relatively low priority or importance.

Streamlining

The detailed test schedule takes advantage of opportunities to streamline the testing by taking advantage of opportunities to sequence things in a natural order.

Here are some examples of streamlining:

•  If the precondition of one test matches the post-condition of another, it may be advantageous to run these tests in sequence (even if the high-level schedule does not require that sequence). One good example is where test data are processed by test 1 into a form that test 2 needs as input.

•  Where tests are time-sequenced it makes sense to keep them together.

•  Where data input are required it makes sense to test the data input function at the same time. This could include error handling, even if error handling is notionally part of a block of testing that is scheduled separately.

•  Business processes should not be separated into different parts of the schedule, even if one part of the process is a higher priority than another. The advantage of processing test data through the entire process and checking the output generally outweighs any disadvantages of adjusting the schedule to make the process fit.

The detailed test schedule will be a test-by-test sequence that provides the most efficient use of resources and provides the best visibility of the system behaviour within the overall constraints imposed by the UAT strategy. Table 8.1 is an example of a test schedule.

Exercise 8.1

In Chapter 7 we used an example of a test script (reproduced below for your convenience) and we know that test scripts can be created quickly when they cover the same process in different parts of the system, for instance logging on to each Excelsior module.

1.2.1 Test scenario

1.  Check an unverified user cannot log on to Contracts.

2.  Check that a verified user can log on to Contracts.

Table 8.1 Example of a test schedule
Excelsior UAT Test Schedule V1.0

Test script #1 – Check an unverified user cannot log on to Contracts.

Test script #2 – Check a verified user can log on to Contracts.

Using the same example but applying it to the task of scheduling, do you think:

1.  These tests should be scheduled in sequence, one following the other?

2.  They could be run as part of another scenario or scenarios and, if so, which ones?

Our answers can be found in Appendix B.

IMPLEMENTING THE TEST SCHEDULE

Implementation is about assigning activities from the detailed test schedule to individual testers and ensuring that they have the necessary test scripts and test environments in place. Testers then set up and run their tests according to the test script, identifying any difficulties by annotating the test script, noting down the exact results and raising an incident report if the test does not produce the expected output.

Allocating tasks

Testing tasks may be allocated on a ‘first come, first served’ basis where any available tester takes on the next scheduled test script, or test scripts may be annotated for execution by testers from a particular specialism, for example end-users, managers or business analysts.

Once tasks are allocated the tester is expected to locate the appropriate test script(s) and the workstation at which the test is to be executed.

Executing test scripts

Testers set up the test as defined in the test script and run the script with any test data specified for the test.

If a test passes, the tester completes the input and output data and any other output events then adds the test result to the test script. If a test fails, the tester will raise an incident report and let the UAT team leader know what has happened.

All testing activity is entered by the UAT team leader onto the test log.

Logging the tests

The test log is initially a copy of the detailed test schedule onto which information is added to indicate who ran each test, when it was run, what the result was and so on. Table 8.2 provides a simple example of a test log.

The test log is a ‘living’ document. Information is added to it continually as tests are completed, and new activities are created as necessary to accommodate the following:

•  Incidents are reported, defects are corrected and retesting is required.

•  Tests are rescheduled because of delay or lack of resources.

•  Tests are rescheduled around retesting.

•  Regression testing is added.

•  Feedback from testing identifies that additional or reduced testing is required.

The schedule identifies what testing has still to be done at any stage and the test log identifies what testing has been done. From the two documents we can determine where we are with respect to the schedule and with respect to the acceptance criteria. We can also derive other valuable test metrics such as the numbers of incidents raised and closed, and average times for testing. These will help us to refine our estimates of when we will complete UAT.

Raising issues from UAT

If a test cannot be completed satisfactorily for any reason, the UAT team leader should be consulted to resolve the problem.

Table 8.2 Example of a test log Issue

Figure 8.2 Example of an incident report

If the test executes correctly but produces an output that is different from the expected output on the test script, an incident report should be raised. Where possible it is always prudent to attempt to rerun the test to confirm that the test was correctly set up and executed, but this is not always possible (for example if data have been changed and cannot be reset). If the problem relates to an error in the test case or the test script, the documentation will need to be amended and the test rescheduled. Figure 8.2 provides an example of a simple incident report.

Completed incident reports should be handed to the UAT team leader for evaluation and onward transmission to the development team.

Evaluating the importance of issues

The relative importance of incidents will depend on the severity of the incident and the anticipated impact on testing. The severity of an incident is usually based on guidelines set by the project manager in conjunction with users. Severity relates to the impact of the defect on the system and therefore the impact on the business when the system is in operation. Therefore a high-severity incident is typically one that threatens the viability of a key business process being supported by the system. The impact on testing will depend to some extent on severity (because a high-severity incident may halt testing while an investigation is carried out) but also on the other testing that is going on in related areas of the system. The UAT team leader needs to communicate the relative priority of the incident as far as UAT is concerned so that defect corrections can be prioritised to minimise the impact on UAT progress as well as correcting the most serious defects as early as possible.

IDENTIFYING PROGRESS

The test log identifies what testing has been done and what the outcomes were. The test schedule summarises what still remains to be done for UAT – we cannot identify exactly what remains to be completed because we do not know what defects might still be present in the software. Trend information from the test schedule and test log can be very helpful in this estimation.

Progress against schedule

Progress against schedule is a measure of how much of the scheduled testing has been completed to date. If we had expected to complete half of the tests on a certain date and the test log indicates we have completed less than half then we are behind schedule. How far we are behind will depend on why we have been delayed. A delay caused by unavailability of a test environment or of testers may not have any knock-on effect, provided the reason for unavailability is not likely to cause further delays. A delay caused by defect corrections and retesting is likely to have a greater effect. Metrics related to the number and severity of defects found, the average times taken to correct and retest, and the subsequent average impact on the test schedule will provide a basis for estimating the future impact of defects, though we will have to either assume that the defect rate will stay the same as it was in the first half of testing or estimate how it might change in the second half.

Accurate estimation of delay in a situation where defect counts are relatively high is difficult, but this is not a situation we would normally expect to apply to UAT. It is more likely that the start of UAT will have been delayed because of defect corrections during system testing or even earlier in the development life cycle. Wherever delay originates, though, it needs to be assessed and projected onto the remaining part of the schedule so that the project manager is aware of the potential delay to project completion.

Progress against acceptance criteria

Progress against acceptance criteria will depend on what the criteria are. For illustration we assume that the main criteria were related to test coverage and defects outstanding on completion of UAT.

Test coverage is directly measurable and the tests would have been designed to achieve the required level of coverage, so completion of the tests for which coverage is being tracked will provide a measure of progress against this criterion. If coverage is not being tracked directly, we should be able to see what tests remain to be completed to achieve the required coverage.

Progress against defect counts is harder to assess but the history will inform us. If defect counts have been high so far, we should expect them to remain high and estimate completion accordingly. If defect counts have so far been very low, we can reasonably assume they will stay low and estimate little or no delay to the scheduled completion date.

Adding up the potential delays associated with each of the acceptance criteria will provide us with an overall estimate of completion that can be combined with the estimate of completion of scheduled tests. The estimates cannot be expected to be accurate to within hours or even a day or so, but we should be able to determine whether or not we expect to be delayed in completing UAT.

THE STATUS REPORT

The UAT status report is a summary of all our progress information, our estimates of when we expect to complete UAT and our recommendations with data to support them. Figure 8.3 suggests some headings for a brief test summary report. In a small UAT exercise this may be as simple as a spreadsheet with commentary. In a large UAT project lasting weeks, it may be a formal report delivered on a regular, perhaps weekly, basis.

The status report lays out for the project manager our measurements and the basis of our estimates so that they can determine whether any anticipated delay is acceptable. If not, some form of compromise will need to be sought.

At this stage we cannot expect to be able to remedy any fundamental weaknesses in the system before release so the choice comes down to relaxing acceptance criteria, postponing the delivery date or both.

THE POST-TESTING SUMMARY

On completion of UAT, or when we are directed to stop testing, we will need to produce a report that contains all of the summary information up to the end of UAT and provides an accurate assessment of the status of the system. That evaluation will be the subject of Chapter 9.

Figure 8.3 Example status report contents

What have I learned?

Test your knowledge of Chapter 8 by answering the following questions. The correct answers can be found in Appendix B.

1. What should be done by the UAT leader immediately after a tester discovers a defect?

A.  Check the test script is not the cause of the fault

B.  Attempt to recreate the defect

C.  Ensure the defect has been logged in enough detail and the severity has been established

D.  All of the above

2. A test failed because of a mistake in the script. What do you do?

A.  Raise an incident report

B.  Do not raise an incident report; amend the script and run it again

C.  Do not raise an incident report; amend the script and schedule it for future testing

D.  Raise an incident report, amend the script and run it again

3. What does the test log record?

A.  How much time is needed to complete the remaining testing

B.  What testing has been completed and the outcomes

C.  Incidents and their severity

D.  What testing has been completed

A question to consider (our response is in Appendix B)

1.  Halfway through testing you are asked to provide a fixed date for finishing UAT. What would your answer to this request be?