Purpose

The purpose of the service validation and testing process is to ensure that a new or changed IT service matches its design specification and will meet the needs of the business.

Objective

The objectives of service validation and testing are to ensure that a release will deliver the expected outcomes and value within the projected constraints and to provide quality assurance by validating that a service is “fit for purpose” and “fit for use.” Another objective is to confirm that the requirements are correctly defined, remedying any errors or variances early in the service lifecycle. The process aims to provide objective evidence of the release’s ability to fulfill its requirements. The final objective is to identify, assess, and address issues, errors, and risks throughout service transition.

Scope

The service provider has a commitment to deliver the required levels of warranty as defined within the service agreement. Throughout the service lifecycle, service validation and testing can be applied to provide assurance that the required capabilities are being delivered and the business needs are met.

The testing activity of service validation and testing directly supports release and deployment by ensuring appropriate testing during the release, build, and deployment activities. It ensures that the service models are fit for its purpose and use before being authorized as live through the service catalog. The output from testing is used by the change evaluation process to judge whether the service is delivering the service performance with an acceptable risk profile.

Value to the Business

The key value to the business and customers from service testing and validation is in terms of the established degree of confidence it delivers that a new or changed service will deliver the value and outcomes required of it and the understanding it provides of the risks.

Successful testing provides a measured degree of confidence rather than guarantees. Service failures can harm the service provider’s business and the customer’s assets and result in outcomes such as loss of reputation, loss of money, loss of time, injury, and death.

Policies, Principles, and Basic Concepts

Now we’ll look at the policies and principles of service validation and testing and the basic concepts behind it. The policies for this process reflect strategy and design requirements. The following list includes typical policy statements:

• All tests must be designed and carried out by people not involved in other design or development activities for the service.
• Test pass/fail criteria must be documented in a service design package before the start of any testing.
• Establish test measurements and monitoring systems to improve the efficiency and effectiveness of service validation and testing.
• Integrate testing into the project lifecycle to help detect and remove defects as soon as possible.
• Test library and reuse policy. Since many of the tests can be repeated, it is valuable to keep a library of test scripts, test models, test cases, and test data that can be reused.
• Engage with customers, stakeholders, users, and service teams to enhance test skills.
• Automate testing wherever possible.

Service validation and testing is affected by policies from other areas of service management. Policies that drive and support service validation and testing include the service quality policy, the risk policy, the security policy, and the service transition, release management, and change management policies.

Service quality will be defined by the senior management, based on customer and stakeholder input. This will drive the adoption and measurement of the basic quality perspectives as listed in the service strategy—level of excellence, value for money, conformance to specifications, and meeting or exceeding expectations. The organization will help to prioritize these, and this will influence the approach taken to service validation and testing.

Risk policies have a direct link to testing. The testing must measure risk according to the appetite for risk in the organization. The testing should include any requirements of risk from the security policy.

The service transition policy sets the working practices for the entire service lifecycle stage. This will include the approach to managing transitions of all scales, and defining the levels of control required during the lifecycle stage. It will cover areas such as governance, use of frameworks, reuse approaches, business alignment and relationship management, knowledge transfer, managing course corrections, early life support, and resource management.

The release policy will define the frequency and type of releases, which in turn will influence the testing approach. The more frequent the releases, the more likely it is to automate and look for reusable test models.

The use of change windows will also influence the testing, so the change management policy will be important in defining the testing approach.

Test Models and Testing Perspectives

Because testing is directly related to the building of the service assets and products that make up services, each one of the assets or products should have an associated acceptance test. This will ensure that the individual components will work effectively prior to use in the new or changed service. Each service model should be supported by a reusable test model that can be used for both release and regression testing in the future. Testing models should be introduced early in the lifecycle to ensure that there is a lifecycle approach to the management of testing and validation.

Levels of Testing

Testing is directly related to the building of the service assets and products, and it is necessary to ensure that each one has an associated acceptance test and activity to verify that it meets requirements.

The diagram in Figure 13.1, sometimes called the service V-model, maps the types of tests to each stage of development. Using the V-model ensures that testing covers business and service requirements, as well as technical ones, so that the delivered service will meet customer expectations for utility and warranty. The left-hand side shows service requirements down to the detailed service design. The right-hand side focuses on the validation activities that are performed against these specifications. At each stage on the left-hand side, there is direct involvement by the equivalent party on the right-hand side. It shows that service validation and acceptance test planning should start with the definition of the service requirements. For example, customers who sign off on the agreed service requirements will also sign off on the service acceptance criteria and test plan.

Process Activities, Methods, and Techniques

There are seven phases to service validation and testing, shown in Figure 13.2. The basic activities are as follows:

• Validation and test management
• Plan and design tests
• Verify test plan and test design
• Prepare test environment
• Perform tests
• Evaluate exit criteria and report
• Test cleanup and closure

The test activities are not undertaken in a sequence; several may be done in parallel; for example, test execution begins before all the test design is complete. Figure 13.2 shows an example of a validation and testing process; it is described in detail in the following list:

• The first activity is validation and test management, which includes the planning, control, and reporting of activities through the test stages of service transition. Also included are managing issues, mitigating risks, and implementing changes identified from the testing activities because they can impose delays.
• The next step, plan and design tests, starts early in the service lifecycle and covers many of the practical aspects of running tests, such as the resources required; any supporting services (including access, security, catering, and communications services); agreement on the schedule of milestone, handover, and delivery dates; agreement on the time for consideration of reports and other deliverables; specification of the point and time of delivery and acceptance; and any financial requirements.
• The third step is to verify the test plan and test design, ensuring that the testing included in the test model is sufficient and appropriate for the service and covers the key integration points and interfaces. The test scripts should also be checked for accuracy and completeness.
• The next step is to prepare the test environment using the services of the build and test environment staff; use the release and deployment management processes to prepare the test environment where possible. This step also includes capturing a configuration baseline of the initial test environment.
• Next comes the perform tests step. During this stage, the tester carries out the tests using manual or automated techniques and records findings during the tests. In the case of failed tests, the reasons for failures must be fully documented, and testing should continue if at all possible. Should part of a test fail, the incident should be resolved or documented (e.g., as a known error) and the appropriate retests should be performed by the same tester.
• The next step is to evaluate the exit criteria and the report, which has been produced from the test metrics. In this stage, the actual results are compared to what was expected. The service may be considered as having passed or failed, or it may be that the service will work but with higher risk or costs than planned. A decision is made as to whether the exit criteria have been met. The final action of this step is to capture the configuration baselines into the CMS.
• The final step is test cleanup and closure. During this step, the test environments are initialized. The testing process is reviewed, and any possible improvements are passed to CSI.

Next we’ll look at the trigger, inputs and outputs, and process interfaces for service validation and testing.

Trigger

This process has only one trigger, a scheduled activity. The scheduled activity could be on a release plan, test plan, or quality assurance plan.

Inputs

A key input to this process is the service design package. This defines the agreed requirements of the service, expressed in terms of the service model and service operation plan. The SDP, as we have discussed previously, contains the service charter, including warranty and utility requirements, definitions of the interface between different service providers, acceptance criteria, and other information. The operation and financial models, capacity plans, and expected test results are further inputs.

The other main input consists of the RFCs that request the required changes to the environment within which the service functions or will function.

Outputs

The direct output from service validation and testing is the report delivered to change evaluation. This sets out the configuration baseline of the testing environment, identifies what testing was carried out, and presents the results. It also includes an analysis of the results (for example, a comparison of actual results with expected results) and any risks identified during testing activities.

Other outputs are the updated data and information and knowledge gained from the testing along with test incidents, problems, and known errors.

Interfaces

Service validation and testing supports all of the release and deployment management steps within service transition. It is important to remember that although release and deployment management is responsible for ensuring that appropriate testing takes place, the actual testing is carried out as part of the service validation and testing. The output from service validation and testing is then a key input to change evaluation. The testing strategy ensures that the process works well with the rest of the service lifecycle—for example, with service design, ensuring that designs are testable, and with CSI, managing improvements identified in testing. Service operation will use maintenance tests to ensure the continued efficacy of services, whereas service strategy provides funding and resources for testing.

Service Validation and Testing Process Roles

In Chapter 1, “Introduction to Operational Support and Analysis,” we explored the generic roles applicable to all processes throughout the service lifecycle. These are relevant to the service validation and testing process, but there are specific additional requirements that also apply. Remember that these are not “job titles”; they are guidance on the roles that may be needed to successfully run the process.

Information Management

As previously mentioned, the nature of IT service management is repetitive and benefits greatly from reuse of data, scripts, and models during transition. Service management suggests that a test library is maintained. This will also include the use of automated testing tools (computer-aided software testing), which are becoming more and more a part of the service validation and testing process.

Critical Success Factors and Key Performance Indicators

As with all processes, the performance of service validation and testing should be monitored and reported, and action should be taken to identify and implement improvements to the process. Each critical success factor (CSF) should have a small number of key performance indicators (KPIs) that will measure its success, and each organization may choose its own KPIs.

The following are two examples of CSFs for service validation and testing and the related KPIs for each.

The success of the CSF “Achieving a balance between cost of testing and effectiveness of testing” can be measured using KPIs that measure

• The reduction in budget variances and in the cost of fixing errors
• The reduced impact on the business due to fewer testing delays and more accurate estimates of customer time required to support testing

The success of the CSF “Providing evidence that the service assets and configurations have been built and implemented correctly in addition to the service delivering what the customer needs” can be measured using KPIs that measure both the improvement in the percentage of service acceptance criteria that have been tested for new and changed services and the improvement in the percentage of services for which build and implementation have been tested separately from any tests of utility or warranty.

Challenges

The most frequent challenges to effective testing are based on other staff’s lack of respect and understanding for the role of testing. Traditionally, testing has been starved of funding, and this results in an inability to maintain an adequate test environment and test data that matches the live environment, with not enough staff, skills, and testing tools to deliver adequate testing coverage. Testing is often squeezed due to overruns in other parts of the project so the go-live date can still be met. This impacts the level and quality of testing that can be done. Delays by suppliers in delivering equipment can reduce the time available for testing.

All of these factors can result in inadequate testing, which, once again, feeds the commonly held feeling that it has little real value.

Risks

The most common risks to the success of this process are as follows:

• A lack of clarity regarding expectations or objectives
• A lack of understanding of the risks, resulting in testing that is not targeted at critical elements
• Resource shortages (e.g., users, support staff), which introduce delays and have an impact on other service transitions

Purpose

The purpose of the change evaluation process is to understand the likely performance of a service change and how it might impact the business, the IT infrastructure, and other IT services. The process provides a consistent and standardized means of assessing this impact by assessing the actual performance of a change against its predicted performance. Risks and issues related to the change are identified and managed.

Objectives

The objectives of change evaluation include setting stakeholder expectations correctly and providing accurate information to change management to prevent changes with an adverse impact and changes that introduce risk being transitioned unchecked. Another objective is to evaluate the intended and, as much as possible, the unintended effects of a service change and provide good-quality outputs to enable change management to decide quickly whether a service change is to be authorized.

Scope

Effective change management means that every change must be authorized by a suitable change authority at various points in its lifecycle. Typical authorization points include before build and test starts, before being checked into the DML (if software related), and before deployment to the live environment. The decision on whether to authorize the next step is made based on the evaluation of the change resulting from this process. The evaluation report provides the change authority with advice and guidance. The process describes a formal evaluation suitable for use for significant changes; each organization will decide which changes need formal evaluation and which will be evaluated as part of change management.

Value to the Business

Change evaluation is concerned with value. Effective change evaluation will judge whether the resources used to deliver the benefit that results from the change represent good value. This information will encourage a focus on value in future service development and change management. CSI can benefit enormously from change evaluation regarding possible areas for improvement within the change process itself and the predictions and measurement of service change performance.

Policies, Principles, and Basic Concepts

The following reviews some of the key policies that apply to the change evaluation process. The first is that service designs or service changes will be evaluated before being transitioned. Second, although every change must be evaluated, the formal change evaluation process will be used only on significant changes; this requires, in turn, that criteria be defined to identify which changes are “significant.”

Change evaluation will identify risks and issues related to the new or changed service and to any other services or shared infrastructure. Deviation from predicted to actual performance will be managed by the customer accepting the change with the deviation, rejecting the change, or introducing a new change to correct the deviation. These three are the only outcomes of change evaluation allowed.

The principles behind change evaluation include committing to identifying and understanding the consequences of both the unintended and intended effects of a change, as far as possible. Other principles include ensuring that each service change will be fairly, consistently, openly, and, wherever possible, objectively evaluated and ensuring that an evaluation report is provided to change management to facilitate decision making at each authorization point.

The change evaluation process uses the Plan-Do-Check-Act (PDCA) model to ensure consistency across all evaluations. Using this approach, each evaluation is planned and then carried out in multiple stages, the results of the evaluation are checked, and actions are taken to resolve any issues found.

Change Evaluation Process

Figure 13.3 shows the change evaluation process and its key inputs and outputs. You can see the inputs that trigger the evaluation activity and the interim and final evaluation reports that are outputs of the process. Where performance does not meet the requirement, change management is responsible for deciding what to do next.

Trigger

Let’s look first at the trigger for this process. The trigger for change evaluation is receipt of a request for evaluation from change management.

Inputs

Inputs to change evaluation are the service design package (which includes the service charter and service acceptance criteria), a change proposal, an RFC, a change record, and detailed change documentation. Other possible inputs are discussions with stakeholders and the test results and report.

Outputs

The outputs from change evaluation are interim and final evaluation report(s) for change management.

Interfaces

Change evaluation interfaces with a number of other processes. The main interfaces are as follows:

Transition Planning and Support Change evaluation works with transition planning and support to ensure that appropriate resources are available when needed and that each service transition is well managed.

Change Management There is a critical interface with change management. The processes of change evaluation and change management must be tightly integrated, with clear agreement on which types of change will be subject to formal evaluation. The time required for this evaluation must be included when planning the change. In addition, it is change management that triggers change evaluation, and change management is dependent on receiving the evaluation report in time for the CAB (or other change authority) to use it to assist in their decision making.

Service Design Coordination This process provides the information about the service that change evaluation requires in the form of a service design package.

Service Level Management or Business Relationship Management Change evaluation may need to work with these processes to understand the impact of any issues that arise and to agree on the use of customer resources to perform the evaluation.

Service Validation and Testing This process provides change evaluation with information; the two processes must coordinate activities to ensure that required inputs are available in sufficient time.

Information Management

Information from testing should be available from the service knowledge management system. Interim and final evaluation reports should be checked into the configuration management system.

Change Evaluation Process Roles

This section explores the process roles relating to the change evaluation process.

Critical Success Factors and Key Performance Indicators

As with the other processes, the performance of change evaluation should be monitored and reported, and action should be taken to improve it. Here are two examples of CSFs for change evaluation and the related KPIs for each.

• Critical success factor: “Stakeholders have a good understanding of the expected performance of new and changed services.”
  • KPI: The reduction in incidents reported when a new or changed service fails to deliver the required utility or warranty
  • KPI: Increased customer satisfaction with new or changed services
• Critical success factor: “Change management has good-quality evaluations to help them make correct decisions.”
  • KPI: Increased percentage of evaluations delivered within the agreed timeframe
  • KPI: Reduced number of changes that fail or need to be backed out
  • KPI: Increased change management staff satisfaction when surveyed

Challenges

Challenges to change evaluation include developing standard performance measures and measurement methods across projects and suppliers, understanding the different stakeholder perspectives that underpin effective risk management for the change evaluation activities, and understanding (and being able to assess) the balance between managing risk and taking risks because this affects the overall strategy of the organization and service delivery. A further challenge is to measure and demonstrate less variation in predictions during and after transition.

The remaining challenges to change evaluation include taking a pragmatic and measured approach to risk and communicating the organization’s attitude toward risk and approach to risk management effectively during risk evaluation. Finally, change evaluation needs to meet the challenges of building a thorough understanding of risks that have impacted or may impact successful service transition of services and releases and encouraging a risk management culture where people share information.

Risks

The most common risks to the success of this process include a lack of clear criteria for when change evaluation should be used and unrealistic expectations of the time required. Another risk is that staff members carrying out the change evaluation have insufficient experience or organizational authority to be able to influence change authorities. Finally, projects and suppliers who fail to deliver on the promised date cause delays in scheduling change evaluation activities.