Timescales

Time is of the essence in incident management because every incident represents some loss or deterioration of service. Every aspect of the process needs to be optimized to produce the fastest end result. Service level agreements, operational level agreements, and underpinning contracts with measurable targets will define how long a support group or third party has to complete each step. When an incident is passed to a support group for investigation, a clock starts ticking—that group has a certain amount of time, defined in the OLA, to complete its work. If the incident is passed to a third party, the timescales set in the underpinning contracts would apply. These OLA and contract target timescales support the achievement of the SLA target.

Service management toolsets should be configured to capture how long it takes to log and escalate an incident, how many incidents are resolved within the first few minutes without requiring escalation, and how long support teams take to respond to and to fix incidents. These times should be monitored, and steps should be taken to identify bottlenecks or underperforming teams so that improvement actions can be taken. The tools can be used to automate timescales and escalate an incident as required based on predefined rules. The tools can also use alarms to warn when incidents are nearing a breach of the SLA. Care needs to be taken in organizations that cross time zones to make sure the clock is monitoring the appropriate time zone as well as the targeted time.

Incident Models

Many incidents have happened before and may well happen again. For this reason, many organizations will find it helpful to predefine standard incident models and apply them to appropriate incidents when they occur. A model is a predefined way of carrying out a commonly required task. ITIL recommends models for a number of processes, such as incident, request, and change processes. An incident model describes the steps needed to investigate and resolve a particular type of incident. The benefit of these models is that they speed up the resolution of recurring incidents as well as ensure consistency in handling them. Most service management tools have the capability to store multiple models; when one of these incidents occurs, it is logged using the appropriate prestored model. The incident can then be handled using the model. The tool can also automate many of the model steps, such as automatic assignment to the correct support group, escalations, and so on.

Incidents that would require specialized handling can be treated in this way (for example, security-related incidents can be routed to information security management, and capacity- or performance-related incidents would be routed to capacity management). Using incident models will help ensure consistency of approach and will speed up resolution.

A typical incident model should include the following contents:

• The steps required to handle the incident, including the timescales, chronological order, and any dependencies they might have
• Details of who is responsible for each step, and the escalation contacts
• Precautions to be taken, such as backing up data, or steps required to comply with health- and safety-related guidelines, such as isolating equipment from the power supply
• In the case of security- and capacity-related incidents, any steps to be taken to preserve evidence

Major Incidents

All incidents should get resolved as quickly as possible, but some incidents are so serious, with such an impact on the business, that they require extra attention. The first step is to agree on exactly what is a major incident. Some organizations will define all priority one incidents as major; others may restrict priority one incidents to those whose impact will be felt by the external customers. In this definition, an incident with a major impact within the organization would not normally be classed as major. An incident that (for example) prevents customers from ordering goods from the organization’s website and that is therefore affecting both revenue and reputation would be included. The definition must align with the priority scheme to avoid confusion.

The purpose of defining an incident as a major incident is so that it can receive special focus. Specific actions to be undertaken are defined in advance so that when the major incident occurs, everyone knows what they are expected to do. The following typical actions might be defined:

• Notification of key contacts within the service provider organization and the business as soon as the major incident is declared.
• Regular updates posted through agreed channels. This would include who should be updated (which could be all users and IT or business management contacts) and the content and frequency of the updates. The agreed channels might include, for example, an intranet announcement, email, and telephone calls.
• Recorded greeting on the service desk phones to inform callers that the incident has occurred and is being dealt with to reduce the number of calls being handled by the desk.
• Appointment of a major incident manager (this may be the service desk manager) and a separate team to focus on resolving the incident.

As with any incident, some major incidents can be resolved without understanding the cause (perhaps by restarting a server); some require the underlying cause to be understood. In the second case, problem management would become involved. It is essential, however, that the focus of incident management remains on restoring service as quickly as possible.

As we discuss in Chapter 39, a major responsibility of the service desk is communicating with the users; this is particularly true in the case of major incidents. Regular updates should be provided. The service desk staff members are also accountable for ensuring that the incident record is kept up-to-date throughout the incident, although it may be the technicians in other teams who actually enter the information. An accurate record is essential during an incident so that there is no confusion; it will also be used after the incident is resolved as part of the major incident review. Regular updates showing the steps taken and whether they were successful will allow improvements to be identified for future events.

Incident Status

Incident management tracks incidents through their lifecycle, using status codes and moving from when the incident is identified through diagnosis and resolution and finally closure. Incident management will remind resolving groups of the associated target times, making sure no incident is forgotten or ignored.

Most service management toolsets will allow a number of statuses to be defined for each incident to facilitate progress tracking. The following statuses are typical:

Open The incident has been identified and logged. A service desk analyst may be working on it, or the service desk may be considering which second-line team it should be escalated to. Incidents resolved by the first-line team may move directly from Open to Closed because the service desk analyst obtains the user confirmation that the incident has been satisfactorily resolved.

Assigned This may mean the incident has been sent to a support team but not allocated to a particular individual.

Allocated or In Progress This is usually defined as when a support technician has been allocated the call.

On Hold This status is sometimes used when the user is not available or doesn’t have the time to test the resolution. It is used to “stop the target clock,” because the service provider cannot do anything further to resolve the incident without the user.

Resolved This status indicates that the technician has completed their work but it has not been confirmed by the customer that it was successful. It is common to use the service management’s automated email facility to automatically email the user when an incident is resolved, asking for a response within a certain timescale if the user is still not happy. If no reply is received, the incident is automatically closed.

• If the user is unhappy, the call is put back into In Progress, and further work is carried out to resolve it.
• The service desk should attempt to contact users to obtain permission to close calls before the automated closure, especially for high-impact incidents where the user may not be aware of the resolution.

Closed This status confirms that the incident is over to the user’s satisfaction. The incident management process has no further involvement, although problem management may now investigate the underlying cause.

Expanded Incident Lifecycle

The expanded incident lifecycle is used by the service design availability management process and within CSI. It breaks down and examines each step of the process to understand the reasons for the failed targets. For example, the diagnosis of the incident may ascertain very quickly that the resolution requires the restoration of data, which takes three hours; this information would be used to pinpoint where improvements should be made. Delays in any step of the lifecycle can be analyzed, and improvements can be implemented to speed up resolution; implementing a knowledge base or storing spare parts on site are two typical measures that are taken to shorten the diagnosis and repair steps.

Step 1: Incident Identification and Logging

First, the incident is identified and then logged. Many incidents will be reported to the service desk, which will log them. But not all calls to the service desk are related to incidents; some will be service requests, which are handled by the request fulfilment process. As Figure 34.1 shows, these are rerouted.

It is essential that incidents are resolved in the shortest possible time because each represents business disruption. Whenever possible, therefore, we should be trying to realize that an incident has occurred before the user notices or, failing that, before they have reported it to the service desk. Chapter 36 shows how monitoring tools can be used to identify failures. The event management process should link directly to incident management so that any incidents spotted are worked on immediately and resolved quickly. Where event management is not in place, incidents will be identified by users contacting the service desk.

The incident record contains all the information concerning a particular incident; details of when it was logged, assigned, resolved, and closed are often required for service level management reporting. Details of symptoms and the affected equipment may be used by problem management. Steps taken to resolve the incident can be used to populate a knowledge base. It is essential, therefore, that all relevant information is added to the record as it progresses through its lifecycle.

A good integrated service management tool makes good recordkeeping much easier because it can automatically populate the record with user details (from Active Directory or a similar tool) and equipment and warranty details (based on the CI number). Automatic date and time stamping of each update and identification of who made the update will both improve the completeness of the information in the record.

Step 2: Incident Categorization

Incidents are categorized during the logging stage. This can be helpful in guiding the service desk agent to the correct known error entry or the appropriate support team for escalation. A simple category structure should be used, however; a scheme that’s too complex leads to incidents all being logged as “other” or “miscellaneous” because the agent does not want to spend the time considering which category is correct. This makes later analysis very difficult. A multilevel scheme, as shown in Figure 34.2, achieves granularity without facing the service desk agent with a long list to choose from. Incidents should be recategorized during investigation and on resolution if the original choice was incorrect.

Step 3: Incident Prioritization

Incidents need to be prioritized to ensure that the most critical incidents are dealt with first. It is often said that all users believe that their own incident is the highest priority, so it is important to agree during service level negotiations about what criteria should be used to decide priority. It is also imperative that incidents are prioritized consistently.

The ITIL framework recommends that two factors should be considered: business impact (effect the incident is having or will have on the business) and urgency (how quickly the business needs a resolution). Business impact can be assessed by considering a number of factors: the number of people affected, the criticality of the service, the financial loss being incurred, damage to reputation, and so on. Depending on the type of organization, other factors such as health and safety (for a hospital or a railway company or similar) and potential breach of regulations (financial institutions and so on) might be considered.

During the life of an incident, it may be necessary to adjust the priority of an incident if the assessment of the impact changes or a resolution becomes more urgent.

Deciding the priority must be a simple process because the incident has to be logged quickly. Employing service desk staff with good business knowledge and ensuring that they are trained to be aware of business impact will help to make a realistic assessment of business impact. Table 34.1 shows a simple but very effective way to determine priority.

Table 34.2 shows how the determination of priority made using the matrix in Table 34.1 can in turn be employed to set a target resolution time for the incident.

Step 4: Initial Diagnosis

The initial diagnosis step refers to the actions taken at the service desk to diagnose the fault and, where possible, to resolve it at this stage. The service desk agent will use the known error database provided by problem management, incident models (covered earlier), and any other diagnostic tools to assist in the diagnosis and possible resolution. Where the service desk is unable to resolve the incident, the initial diagnosis will identify the appropriate support team for escalation.

Part of this stage is the gathering of information to assist the second-line technician in resolving the incident quickly. Again, sufficient time is required for this step; “saving” time by passing the incident to a second-line technician quickly but with sparse details is not helpful. The second-line technician will need to contact the customer to obtain the information, adding delay and frustration. Support teams should provide guidance to the service desk about the type and level of information they should be gathering.

Step 5: Incident Escalation

The ITIL framework describes two forms of escalation that may take place during incident management: functional escalation and hierarchic escalation.

Functional escalation takes place when the service desk is unable to resolve the incident; this may be realized immediately because of the type of incident, such as a server failure, or because the service desk agent may have spent the maximum time allowed under the organization’s guidelines attempting to resolve the incident without success. It then needs to be passed to another group with a greater level of knowledge. The second-line support group that receives this escalated incident will also have a time limit for resolution, after which the incident gets escalated again to the next support level. Sometimes, as with the service desk, it is obvious that the incident will require a high level of technical knowledge, and in such a case the incident would be immediately escalated, without any attempt by second-line support staff to resolve it.

The service desk must know to whom the incident should be escalated to avoid unnecessary delays, so the service desk staff needs to have sufficient technical knowledge to be able to identify which incident goes to which team. Operational level agreements will specify the responsibilities of each group. There may be occasions when cooperation between support groups is required or the incident needs to be referred to third parties such as hardware maintenance companies. The OLAs and UCs should specify what should happen in these situations.

The second type of escalation is hierarchic escalation, which takes place for high-priority or major incidents. Hierarchic escalation consists of informing the appropriate level of management about the incident so that they are aware of it. This ensures that the management is able to make decisions that are required regarding prioritization of work, suppliers, and so on. In the case of a major incident, the IT director may be expected to brief the business directors about the progress of the incident; even if this is not the case, business managers may go directly to senior IT managers when a serious incident has occurred, so it is essential that the IT managers have been thoroughly briefed themselves.

It is also sometimes necessary to use hierarchic escalation when the incident is not progressing as quickly as it should or if there is disagreement among the support groups regarding assignment.

A good service management tool will be able to automatically escalate incidents, based on the SLA targets, and update the record with details. For example, a tool could be set to notify a team leader when 90 percent of the SLA target time had passed and to inform the team leader’s line manager when the incident breached the target.

Step 6: Investigation and Diagnosis

The major activity that takes place for every incident is investigation and diagnosis. The incident will have undergone the initial diagnosis step (step 4, covered earlier); that step identifies whether the service desk can resolve the incident. The investigation and diagnosis stage here is different; it involves trying to ascertain what has happened and how the incident can be resolved.

The incident record should be updated to record what actions have been taken, and an accurate description of the symptoms, and the various actions taken, is required to prevent duplication of effort; the record will also be useful when the incident is reviewed, perhaps as part of problem management. Typical investigation and diagnosis actions would include gathering a full description of the issue and its impact and urgency, creating a timeline of events, identifying possible causes (such as recent changes), interrogating knowledge sources such as the known error database, and so on.

Step 7: Resolution and Recovery

Potential incident resolutions should be tested to ensure that they actually resolve the issue completely with no unintended consequences. This testing may involve the user. Other resolution actions might include the service desk agent or technician remotely taking over the user’s equipment to implement a resolution or to show the user what they need to do in the future. Once the incident is resolved, it returns to the service desk for closure.

Step 8: Incident Closure

When the incident has been resolved and the service restored, the service desk will contact the user to verify that the incident can be closed. This is an important step because the fault may appear to be resolved to the IT department, but the user may still be having difficulties, especially if there were actually two incidents, with the symptoms of one being hidden by the other. The second incident would become apparent only after the first was resolved. The service desk may contact the user directly, or an email could be sent with a time limit, stating when the incident will be closed, as described earlier.

If the underlying cause of the incident is still unknown, despite the fact it has been resolved, a problem record may be raised to investigate the underlying cause and to prevent a recurrence. Finally, a user satisfaction survey is carried out.

Service Design

Several of the service design processes interface directly with the incident management process. These processes are among those we discussed previously; many of the process activities take place in the service operation lifecycle stage. Service level management interfaces with incident management because SLAs will contain incident targets; the other service design processes may result in incidents if the processes fail to prevent a security breach, a lack of capacity, or unplanned downtime.

Service Level Management As you have seen, there are numerous links between incident and service level management (SLM). Incidents have target response and resolution times; these targets are set in the SLAs. Incident management in return provides the management information from the service management toolset to enable SLM to report on the success achieved in meeting these targets. Incident reporting enables SLM to identify failing services and to implement service improvement plans for them (in conjunction with continual service improvement).

Information Security Management, Capacity Management, and Availability Management Incident management collects data on the number of security-related incidents and capacity issues. It provides the data on downtime that availability uses to calculate availability reports, and analysis of incident records helps availability management understand the weak points in the infrastructure that need attention. An efficient incident lifecycle also improves overall availability.

Service Transition

The service transition processes of service asset and configuration management (SACM) and change management interface with incident management; SACM provides useful information to the incident process, and changes may be the cause of incidents or the means by which incidents are resolved.

Service Asset and Configuration Management Incident management uses SACM data to understand the impact of an incident because it shows the dependencies on each CI. It provides useful information regarding who supports particular categories of CI. By logging each incident against a CI and checking that the user of the CI is as recorded in the CMS, incident management helps keep the CMS accurate.

Change Management Changes are often implemented to overcome incidents. Incidents may often be caused by changes. An important input into incident investigation is the change schedule; asking the question, What changed just before this incident occurred? can often highlight the cause of incidents. In the case of a major incident caused by a change, the decision may be made to back out of the change. Information identifying how many incidents were caused by changes should be fed back to change management to improve future changes.

Also, in some cases a fix to an incident may require an emergency change to be raised, to ensure that the CMS is kept up-to-date.

Service Operation

There is a strong interface between incident management and problem management. Access management issues may also cause incidents.

Problem Management As you will learn when we discuss problem management later in this chapter, incident management and problem management have many links. Incident management provides the data on repeat incidents that problem management uses to identify underlying problems. The permanent resolution of these problems helps incident management by reducing the number of incidents that occur. The incident impact and urgency information helps problem management prioritize between problems.

Problem management provides known error information, which enables incident management to restore service.

Access Management Incident management raises incidents following security breaches or unauthorized access attempts. This information can be used by access management to investigate access breaches. Failure to ensure that users are granted the access they require to do their job may result in incidents being reported because the user is greeted with an error message when attempting to carry out a task.

Reactive and Proactive Aspects

Unlike incident management, which is entirely reactive (you cannot resolve an incident until it has occurred), problem management has both reactive and proactive features:

• Problem management will react to incidents and attempt to identify a workaround and a permanent resolution.
• Problem management will also proactively try to identify potential incidents and take action to prevent them from ever happening. This might include analysis of incident trends, such as intermittent but increasingly frequent complaints about poor response times, to identify a potential capacity issue. By working with capacity management, problem management can take proactive measures to provide sufficient capacity and avoid any major breaks in service. Event management reports can also be analyzed to the same end, in this case to prevent an incident before the user is aware of any issue.
• Problem management may assist in a major incident review, trying to identify how to prevent a recurrence.

The process steps for managing problems that are raised in reaction to incidents and for managing problems that are proactively identified are broadly similar. The main difference is the trigger for the process. Reactive activities take place as a result of an incident report and help prevent the incident from recurring or provide a workaround if avoidance is impossible; these activities complement the incident management process.

Proactive problem management analyzes incident records to identify underlying causes of incidents. It may be that analysis of previous incidents reveals a trend or pattern that was not apparent when each incident occurred. For example, users may complain of poor response periodically; it is only when all these complaints are analyzed that it becomes apparent that the poor response is always reported against the same module or from the same location. This would trigger a problem record to be raised to identify the common cause linking all these incidents.

Reactive and proactive problem management activities normally take place as part of service operation, but problem management is also closely related to continual service improvement. Where improvement opportunities are identified as a result of problem management, they should be entered into the CSI register.

Problem Models

Most problems are unique and can’t be investigated in a predetermined way. But sometimes there are underlying issues that cannot be resolved—they are too difficult, the resolution is too costly, or they are outside our scope of action. For example, a badly designed application may, over time, experience intermittent incidents, each of which has a different specific cause, a different problem. A problem model might guide the investigation of problems in that application. It might describe how to collect and interpret relevant diagnostic information, for example.

Incidents vs. Problems

As we have said, problem management and incident management are closely related. Although incident management is not concerned with determining the underlying causes of the incidents it investigates, sometimes it is necessary to do that in order to resolve the incident. When that is the case, problem management should get involved. Determining the root cause might provide the workaround needed to close the incident. Potentially, problem management could get involved whenever an incident occurred that had no corresponding problem or known error record, but that would probably be overkill. Every organization should define criteria for determining when it’s appropriate to involve problem management. It is very common, usual even, for problem management to be involved in the handling of major incidents.

Known Errors

When you examine the problem management process in detail, you will see that the process attempts to identify and document known errors, usually when the cause of the problem is known and a workaround is available but the permanent fix has yet to be applied. Known error records are used to document root causes and workarounds, allowing quicker diagnosis and resolution if further incidents do occur. Problem management is not the only source of known errors. Some will come from application development. It’s often the case that a new application will go live in spite of known bugs in the software because they are not considered serious enough to delay the rollout. Where this is the case, the bugs should be documented in the known error database so that the information is available should there be in incident during live operation. These bugs might be discovered in development or during testing in service transition. A final source of known errors is the notifications received from suppliers when they discover issues in their products.

Workaround

A workaround is defined as a means of reducing or eliminating the impact of an incident or problem for which a full resolution is not yet available. Restarting a failed configuration item is an example of a workaround. Workarounds for problems are documented in known error records.

When Is a Problem Raised?

Sometimes it is helpful to raise a problem record while the incident is still open. Each organization will decide on its own criteria for when a problem should be raised. For example, a problem might be raised when the support teams are sure the incident has been caused by a new problem because the incident appears to be part of a trend or because there is no match with existing known errors. The service desk or support teams may have resolved the incident without knowing the cause, and so there is a risk that the fault may recur. This is particularly true in the case of a major incident; the underlying cause needs to be identified as soon as possible to prevent future disruption to the business. (The problem diagnosis activity may take place in parallel with the incident resolution and may continue after the successful resolution until the underlying cause is identified and removed.) As already explained, it is also possible that suppliers may inform their customers of problems that they have identified.

As stated earlier, problem management is not an optional activity; it is fundamental to providing a consistent service in line with SLA commitments. By providing workarounds to enable resolution of incidents by the first-line staff, the organization makes better use of the more skilled and therefore more expensive second- and third-line staff, who are freed up to use their skills in problem investigation.

Step 1: Detecting Problems

The first step in the process is to identify that a problem exists. As we discussed previously, problems may be raised either reactively (in reaction to incidents) or proactively. In addition to the triggers identified earlier, a problem may also be identified as a result of alerts received as part of event management. The event monitoring tools may identify a fault before it becomes apparent to users and may automatically raise an incident in response.

As the diagram shows, problems can be identified in a number of ways, but they are all handled by a single problem management process and logged in a single database. A major source is the service desk acting for the incident management process. Any incident whose cause is unknown could trigger a problem investigation. In practice, few organizations will do that because of the very large number of problems that result. Most organizations are quite selective about the problems they log and investigate. This is supplemented by regular analyses of incident data, searching for recurring incidents that might have been missed. Potential problems can also be identified by event management and suppliers, and of course proactive problem management itself identifies problems.

Step 2: Logging Problems

Having identified that a problem exists, a problem record containing all the relevant information should be logged and time-stamped to provide a complete picture. Typical details would include who reported it and when, the service and equipment used, and a description of the incident or incidents and actions taken. The incident record number(s) and the priority and category would also be required.

Where possible, use the service management tool to link problem records with the associated incident records. Remember that the incident has not “become” a problem; the incident must continue to be managed to resolution whether the problem is resolved or not.

Step 3: Categorizing Problems

The problem is then categorized using the same system used by incident management, which allows incidents and problems to be more easily matched. For example, it might be possible to relate a number of types of incidents to a single common problem, which would bring a lot of extra diagnostic information.

The problem manager should emphasize the importance of accurate categorization to the service desk. The use of incident models can be very helpful here because they standardize the way common incidents are recorded. Enforcing categorization on incident resolution, as mentioned earlier, will also help ensure that incident categories are accurate.

Step 4: Prioritizing Problems

As with incidents, the priority of a problem should be based on the business impact of the incidents that it is causing and the urgency with which it needs to be resolved. The problem manager should also consider how frequently the incidents are occurring. It is possible that a “frozen screen” that can be resolved with a reboot is not a high-priority incident; if it is occurring 100 times a day, the combined impact to the business may be severe, so the problem needs to be allocated a high priority. The impact to the business must always be considered, so factors such as the cost of resolving the incident and the time this is likely to take will be relevant when assessing priority. It is likely that some problems are not considered important enough to spend resources on, especially if a reasonable and effective workaround has been identified.

Analyzing factors such as number affected, the duration of the downtime, and the business cost is known as pain value analysis. This technique can be useful in determining the business impact of a problem. Of course, circumstances change, new information is learned, and further incidents may occur, so the priority of a problem should be kept under review and changed if appropriate.

Step 5: Investigating and Diagnosing Problems

The next stage in the process is to investigate and diagnose the problem. The purpose of this stage is first to discover the root cause and then to determine a workaround and permanent fix. There may not be the resources to investigate every problem, so the priority level assigned to each will govern which ones get the necessary attention. It is important to allocate resources to problem investigation because until the problem is resolved, the incident will recur and resources will be spent on incident resolution.

Usually the task of investigating and diagnosing the problem is passed to the appropriate technical or application team; if it is unclear where the problem originates, it might be necessary to establish a team of technical specialists to investigate it. Sometimes it will be necessary to escalate a problem to a supplier.

The ITIL framework suggests a number of different problem-solving techniques, which are helpful in approaching the diagnosis logically. The CMS can be very helpful in providing CI information to help identify the underlying cause. It will also help in identifying the point of failure when several incidents are reported; the CMS may identify that all the affected CIs are linked to one “parent” CI. The fault affecting the “parent” configuration item is causing all those CIs attached to it to experience a loss of service. The KEDB may also provide information about previous, similar problems and their causes. Where a test environment exists, this can be used to recreate the fault and to try possible solutions.

Step 6: Identifying a Workaround

Although the aim of problem management is to find and remove the underlying cause of incidents, this may take some time; meanwhile, the incident or incidents continue, and the service is affected. When a user suffers an incident, the first priority is to restore the service so that the user can continue working. A priority of the process, therefore, is to provide a workaround to be used until the problem is resolved. A workaround is a means of reducing the impact of an error. This can be in the form of a circumvention, which is a way of working that avoids triggering incidents, or it may be a way of resolving incidents should they occur. At this point a known error should be raised and stored in the known error database.

The workaround does not fix the underlying problem, but it allows the user to continue working by providing an alternative means of achieving the same result. The workaround can be provided to the service desk to enable them to resolve the incidents while work on a permanent solution continues. The problem record remains open because the fault still exists and is continuing to cause incidents. The details of the workaround are documented within the problem record, and a reassessment of its priority may be carried out.

It is possible that IT or business management may decide to continue to use the workaround and suspend work on a permanent solution if one is not justified. A problem affecting a service that is due to be replaced, for example, may not be worth the effort and risk involved in implementing a permanent solution.

Of course it is not always possible to devise a workaround, and sometimes a workaround is not acceptable to the business—the fault must be fixed.

Step 7: Raising a Known Error Record

When problem management has identified and documented the root cause and workaround, this information is made available to support staff as a known error. Information about all known errors, including which problem record it relates to, is kept in the known error database (KEDB). When repeat incidents occur, the support staff can refer to the KEDB for the workaround.

Although ITIL defines a known error as a problem with a cause that is known and a workaround that has been provided, in the real world, there may be times when a workaround is available even though the root cause is not yet known (for example, a reboot restores the service, but it’s not known what causes the error). On other occasions, we may know the cause but not have a workaround because a change has to be implemented to fix the fault.

Sometimes a known error is raised before a workaround is available and sometimes even before the root cause has been fully identified. This may be just for information purposes; a workaround may be available that has not been fully proven. Rather than have a rigid rule about when a known error should be raised, a more pragmatic approach is advisable; a known error should be raised as soon as it becomes useful to do so.

Step 8: Problem Resolution

When problem management has identified a solution to the problem, it should be implemented to resolve the underlying fault and thus prevent any further incidents from disrupting the service. Implementing the resolution may involve a degree of risk, however, so the change management process will ensure that the risk and impact assessment of the RFC is satisfactory before allowing the change. The error might be in an application that is scheduled to be phased out in the near future, so the business may choose to accept the likely disruption temporarily rather than accept the cost and risk of a fix. Ultimately, the decision whether to go ahead with the resolution despite the risk is a business decision; the business damage being done by the problem may mean the business is prepared to accept the risk in order to have the fix implemented.

Problem Closure

When a permanent solution to the problem has been identified, tested, and implemented through the change management process, the problem record can be updated and closed. Any open incidents caused by the problem can be closed too. The KEDB should be updated to show that the problem is resolved so any future incidents will not have been caused by it; however, the information contained within the problem record may prove useful in addressing a future, similar problem.

Major Problem Review

Each organization must determine what constitutes a major problem, based on its own business priorities. Once a major problem is resolved, it is advisable to hold a formal review to learn from the experience. The review should be carried out in the immediate aftermath of events, when memories are fresh. It is led by the problem manager but should include the participation of everyone who played a role in the events under review.

The review is not only concerned with the failure that first triggered the major problem, but also with everything that happened and was done afterward. It takes a holistic view, in other words.

Specifically, the review asks the following questions:

• What did we do that was right? What did we do that made things worse?
• What could we do better in future?
• What can we do to prevent it from happening again?
• What proposals can we make for improvement?

These could involve making changes to processes or technology, requiring specific training for support or other staff, and requiring action by suppliers or changes to contracts.

A major problem review also has a significant role to play in rebuilding customer confidence, which will probably have suffered due to the major problem. The review should be conducted with this in mind, and it should be reflected in the resulting report. It might be useful to include a representative from the business in the review.

Figure 34.4 illustrates the way incident, problem, and change management activities are linked. It is largely self-explanatory, but note that an incident is closed when service is restored but a problem is closed only when a fix has been applied and is confirmed to be effective. Sometimes problem management cannot supply a workaround, so the incident stays open until the permanent fix is implemented.

Inputs

The incident database is a vital source of information for problem management because it contains information about current incidents and problems and is the basis of proactive problem management.

The configuration management system (CMS) provides information about the hardware and software components that underpin the IT services; this is essential for problem investigation and diagnosis. It also helps the process to prioritize problems appropriately by showing the links between components and services.

Problem management must assess whether a business case exists for the change request, which is required to implement a permanent fix to a problem; financial information is needed for this. Input from change management will provide feedback on progress and whether the changes are successful.

Finally in this list comes feedback from customers. If customers are unhappy about repeated recurrences of incidents interrupting their work, this would be both a trigger for and an input into problem management. Building and maintaining customer satisfaction is an essential objective of all service management activity, including problem management.

Outputs

Outputs from problem management include the obvious outputs such as updated problem records, requests for change, workarounds, and known error records. In addition, the process will output various reports. For example, a problem report might be discussed at a service review meeting. Reports of issues found should be referred to design teams and other processes. If reports include recommendations for improvement, they may be logged in the CSI register.

Service Strategy

Problem management has a link with financial management for IT services to assess the financial impact of possible solutions or workarounds. This information can be used to decide whether a permanent resolution is financially justified.

Service Design

Problem management interfaces with several of the service design processes:

Availability Management This process and problem management have a similar goal: to prevent downtime. The proactive activities undertaken by availability management are directly related to proactive problem management; availability attempts to proactively identify risks that could result in a loss of service and to take preventative action. Problem management can supply information to availability management about the success of any measures taken.

Capacity Management Some performance problems can be caused by capacity issues. Capacity management will be involved in resolving these issues and also taking proactive measures to prevent capacity issues. Again, problem management can supply information about the success of any measures taken.

IT Service Continuity Management If a significant problem is causing or will cause major disruption to the business, it may be necessary to invoke the IT service continuity management (ITSCM) plan until the issue is resolved. ITSCM also attempts to proactively identify risks that could result in a major loss of service and to take preventative action.

Service Level Management The service level management process interfaces with problems in a different way. SLM is dependent on problem management to identify the root cause of incidents and resolve them in order to prevent downtime that could cause a service level target to be breached.

Service Transition

Problem management interfaces with several of the service transition processes:

Change Management As discussed, changes may be the cause or solution to problems.

Service Asset and Configuration Management Service asset and configuration management provides invaluable information to enable common factors to be identified across multiple incidents.

Release and Deployment Management Release and deployment is involved in contributing to problem management’s known error database, which is also related to knowledge management.

Knowledge Management The last interface with service transition is with knowledge management. The workarounds developed by problem management are examples of the service knowledge that is the core concern of knowledge management. The service knowledge management system can be the basis of the known error database.

Service Operation

As we discussed in the section about incident management, the major relationship that problem and incident management have is with each other.

Continual Service Improvement

Problem management and continual service improvement have similar objectives; problem management activities can also be seen as CSI activities. The aims of CSI and problem management are very closely aligned. Both seek to drive out errors and improve service quality. As stated earlier, actions identified to resolve or prevent problems may be entered into the CSI register.

The seven-step improvement process can be used by CSI or problem management to identify and resolve underlying problems.