Monitoring and Event Management

We need to monitor events, but monitoring covers more than events. Monitoring can be used, for example, to make sure devices are operating correctly, even without any events being generated. Monitoring actually looks for conditions that do not generate events.

Event management is about having useful notifications about the status of the IT infrastructure and services. Event management sets up rules to ensure that events are generated so they can be monitored, captured, and acted upon if necessary. Action is the key to event management.

The particular notifications themselves may be vendor specific. However, they are likely to use Simple Network Management Protocol (SNMP), which is an Internet standard protocol for managing devices on IP networks. Devices that typically support SNMP include routers, switches, servers, workstations, printers, modem racks, and more. Because SNMP is an open standard, it makes interaction between different products simpler. Events must generate useful notifications. The time taken to create meaningful descriptions, with suggested actions, will save enormous effort later.

Event management can be enormously useful in managing large and complex infrastructures. It is often the case, however, that the full value of these tools is not realized. This is usually because there has been insufficient time spent making sure they are configured correctly to only notify staff of events for which they actually need notification. Failing to specify the correct thresholds, for example, will mean that far too many breaches are reported, causing staff to ignore them because they are seldom significant. Of course, this means that significant events are missed. If events are not filtered properly, the service management tool would be flooded with multiple spurious events, which would make it difficult to use its ability to automatically raise incidents.

Another important definition is that of an alert: An alert is a warning that a threshold has been reached, something has changed, or a failure has occurred. Alerts are often created and managed by system management tools and the event management process. Creating an alert when a disk or mailbox is nearly full is one such example.

Informational, Warning, and Exception Events

Some events indicate a failure that must be fixed, while others simply flag that something has happened and should be recorded. These are two types of event: the first is an exception and the second informational. There is a third type—a warning event. A warning event signifies unusual but not necessarily exceptional behavior. Warning events require further analysis to determine whether any action is required. Events will be handled according to their type.

Here are some examples of each type of event:

• Informational
  • A scheduled workload has completed.
  • A user has logged in to use an application.
  • An email has reached its intended recipient.
• Warning
  • A server’s memory utilization reaches within 5 percent of its highest acceptable performance level.
  • The completion time of a transaction is 10 percent longer than normal.
• Exception
  • A user attempts to log on to an application with an incorrect password.

Notice that not all of these examples relate to a failure. (Failures would be alerts.) Some simply contain information, but information that for some reason it is important to record. For example, the business might want to maintain a record of who is using an application for audit purposes.

There are no definitive criteria for determining the type of an event; it depends very much on the specific situation of the organization. For example, an event might be that a previously unknown device has been detected on the network. Some organizations allow their staff to attach their own laptops to the corporate network, in which case the event would be informational. In a highly secure organization, it would almost certainly be treated as an exception.

Filtering

The next topic we’ll examine is event filtering. We don’t have complete control over the notifications that are generated by the configuration items. The manufacturers of hardware will have decided what notifications will be generated, and they may not have provided their customers with the ability to switch them off. A common experience when first beginning to monitor networks is that the monitoring tool is swamped by unrecognized and therefore unneeded notifications.

Filtering prevents the event management system from being overwhelmed by discarding notifications of events that have no significance to the organization.

There are four possible approaches to the problem:

• The first approach is to integrate event management into each service management process. What this means is that each process will identify the events that it is interested in.
• The next approach is to include event management requirements into the design of new services.
• A third approach is to use trial and error—evaluate notifications on a case-by-case basis and adjust the filtering accordingly.
• The final approach is to plan the introduction of event management within a formal project.

These approaches are not mutually exclusive; many organizations will adopt some hybrid of them.

Designing for Event Management

Successful event management in service operation requires analyzing and planning for what will be required. This should happen in the service design phase, although it will continue to be adjusted in service operation. Many organizations attempt and abandon event management, or they fail to achieve real value from it because this crucial design phase has been neglected.

The following questions should be asked when designing a service or planning the introduction of new technology:

• What needs to be monitored?
• What type of monitoring is needed?
• When should events be generated?
• What information should be communicated?
• Who are messages intended for?
• Who will respond to the event?

When event management is first established, these questions should be asked about the existing services and infrastructure. Stakeholders who must be consulted include the business, process owners, and operations management staff. Each of these groups will have monitoring requirements, and each could be involved in handling events when they occur.

Instrumentation

Instrumentation refers to specific ways to monitor and control the infrastructure and services. A number of practical issues need to be addressed when designing an event management system:

• How will events be generated? In the case of bought-in components, of course, this question is really, How are events generated?
• How will they be classified? This is not straightforward, and classifications can change over time.
• How will they be communicated and escalated? How exactly will the events get to the appropriate function? For example, how will an exception event trigger the incident management process? Ideally, this will be automated by integrating the event and incident management tools so that an incident will be logged automatically. Of course, this can only happen if the two tools have the necessary functionality.
• What data must be included in the event notification? What data will be needed for the event management system itself to interpret and make sense of the event, and what data will be needed by the function that will respond to the event? If the event relates to an error, then it should include necessary diagnostic information such as error messages and codes. Again, for bought-in software, this question is really, What data is included?
• Another question to be asked relates to the type of monitoring to use. Should it be active or passive?
• Where will event data be stored? There is likely to be a significant amount of event data, so the question of storage is important. Associated with this is the question of how long the data should be retained. This decision must be made on a case-by-case basis in consultation with the relevant stakeholders.
• How will supplementary data be gathered? In some cases, the event data alone will not be sufficient to evaluate the event. For example, it might be necessary to integrate the event management system with a CMDB.

Correlation Engine

As events are detected, the event management system must interpret and make decisions about how to handle them. This is done by software known as a correlation engine. The correlation engine allows the creation of rule sets that it will use to process events.

Using a correlation engine will enable the system to determine the significance of each event and also to determine whether there is any predefined response to an event. Patterns of events are defined and programmed into correlation tools for future recognition. The correlation engine can translate component-level events into service impacts and business impacts, as shown in Figure 36.1.

Event Notification

Let’s look at the initial process activities in a little more detail. Event notification refers to the communication of information about an event. You’ve already seen that some components will generate notifications independently, while others have to be prompted by polling.

Event Detection and Logging

Some events will be detected directly by the event management tool. Other events will be detected by a software agent running on the device being monitored. This agent then generates a notification that can be detected by the event management tool. All events are logged.

Correlation and Filtering

In first-level correlation, a decision about whether any further action is required is made, including whether the event has any significance to the organization. Correlation will determine whether an event is informational, a warning, or an exception. We discussed filtering earlier; this is necessary to stop staff being overwhelmed with events that do not require any action, or multiple reports of the same fault.

Informational events are closed at this point. Exception events will trigger one of the other service management processes. Warning events will go forward to second-level correlation.

Next, we consider the criteria that might be used by the second-level correlation engine to interpret an event:

• The number of similar events might be significant. For example, an event might notify us of an unsuccessful attempt to access our network. If this is just a single instance, we might treat it as informational, but if there have been 500 such attempts in the last 5 minutes, we might judge that we are the target of an organized attempt to break in.
• The number of devices generating similar events might be significant. It might indicate a widespread virus infection, for example.
• The data supplied with the event might indicate its significance.
• Events related to device utilization might be compared with a defined threshold.

The correlation engine determines whether the event requires some action or whether it can be treated as informational and closed.

Actions Taken

Some events indicate conditions that can be resolved automatically without human intervention. For example, if a file server is detected to be nearly full, then a script could be run that would free up space by archiving old data.

Some events will require human intervention—an alert from a smoke detector, for example. It’s important that the alert is directed to the right person and that they know what to do.

Exception events will normally trigger the incident management process. Ideally, the event and incident management systems will be integrated so that an incident record can be raised automatically. A word of warning, however: This should be implemented only when you are happy that the filtering of events is working correctly. If this is not the case, your incident management system will be flooded with thousands of spurious incidents!

The problem management process might be triggered if the organization has a policy of always investigating the root causes of incidents that impact key services. Event management can support such a policy by automatically raising a problem record when it detects such an incident.

Change management can be triggered in two circumstances:

• First, if a previously unknown device is detected, an RFC should be raised, which can be progressed as appropriate by change management to have the device either added to the CMS or removed if required.
• The second circumstance is if a change is needed. For example, a server might need to be allocated more storage from a SAN.

Remember, sometimes it will be necessary to trigger a combination of these responses.

Review and Closure

There could be thousands of events each day, so it’s unlikely that every one of them could be reviewed. It’s sensible to review only what the service provider considers to be significant events. It is probably unnecessary to review events that have triggered other service management processes except to ensure that the triggers were effective.

Most events are neither opened nor closed but just logged in management systems or system logs. Many others can be closed automatically. For example, when a script is triggered to respond to an issue, the script itself could check that the corrective action has worked and generate an event to that effect.

Triggers

Any type of change in state can trigger event management, and an organization should define which of these state changes need to be acted upon. Some examples are shown here:

• Exceptions to any level of CI performance defined in the design specifications or standard operating procedures.
• A breach of a threshold in an OLA could also generate an event.
• Any exceptions to a process, such as a failure to complete a process within the target time. An exception could also be a routine change that has been assigned to a build team or a business process that is being monitored by event management.
• The completion of an automated task or job could trigger the issuing of an event, as could a status change in a server or database CI.
• A user accessing a particular application or database could also cause an event to be issued if this is information that the business or the IT department wanted to know about. For example, IT may wish to track how often a service is being used to decide on the number of licenses required, or the business may wish to know how often a customer service representative needs to refer to the knowledge base to answer a query, as this might indicate a training requirement.
• The more usual trigger for an event, such as a situation in which a device, database, or application has reached a predefined threshold of performance.

Inputs

Inputs to event management usually come from service design and service transition. They include the examples listed here:

• Operational and service level requirements associated with events and their actions.
• Alarms, alerts, and thresholds for recognizing events.
• Event correlation tables, rules, event codes, and automated response solutions that will support event management activities.
• Roles and responsibilities for recognizing events and communicating them to those who need to handle them.
• Operational procedures for recognizing, logging, escalating, and communicating events.
• SLAs, which can be used by the correlation engine to determine the significance of an event or to identify a performance threshold.
• Rule sets that are provided by technical or application management staff based on the monitoring and management requirements. For example, the capacity management process would define the capacity thresholds that should generate an event.
• The roles and responsibilities of all those involved.
• Procedures for logging and escalating events as required.

Outputs

Outputs from event management are usually passed to other service management processes, such as incident management, change management, and request fulfilment. They include the examples listed here:

• Events that have been communicated and escalated to those responsible for further action
• Incident, problem, change, or request records required as a result of an event
• Event logs describing what events took place and any escalation and communication activities taken to support forensic, diagnosis, or further CSI activities
• Events that indicate an incident has occurred
• Events that indicate the potential breach of an SLA or OLA objective
• Events and alerts that indicate completion status of deployment, operational, or other support activities
• Populated service knowledge management system (SKMS) with event information and history

The most obvious output of the process is the events themselves. These should have been communicated and escalated to the appropriate people. Another output is a chronological event log describing what events took place and any escalation and communication activities taken. This may be useful information if further investigation is required or to spot possible improvement opportunities.

Some events output by event management will indicate that an incident has occurred, and others will warn of the potential breach of an SLA or OLA objective. Of course, as we have said, not all events show that something is wrong, and many events will just indicate successful completion of deployment or operational activities. The data output from event management can be used to populate the SKMS with the event information and history.

Interfaces between Event Management and the Lifecycle Stages

Finally, let’s consider the interfaces event management has with the other lifecycle stages and their associated processes. Event management can interface with any process that requires monitoring and control, especially those that don’t require real-time monitoring but do require some form of intervention following an event or group of events. First we’ll consider how the process can even help the business directly.