Understanding the costs of an outage aids in understanding the return on investment (ROI) of proactive performance engineering. Remember, operational costs “hide” the true cost of system development. Costs of downtime in production (post-deployment) include the following:

Consider the example of a company that spends millions of dollars on application support instead of new application development. In this case, each 15-second timeout in the enterprise application integration (EAI) infrastructure could result in a $5 call to an outsourced contact center. Over the course of six months, this could result in unanticipated support costs of almost $3 million—funds that could otherwise have been used for new development efforts.

In addition to the costs of an outage, it is important to understand the scope of the impact—specifically, who is impacted. For example, an outage that affects the top customers, responsible for the majority of revenue leveraged by the system, carries a much higher weight than one that affects only the smallest customers. When defining service levels for availability and transactions, consider which customers are impacted and when they’re impacted, especially in the context of business “events”(dates, time frames) where access to systems is more crucial.

Service-level agreements help organizations meet business objectives. By clearly defining and measuring against goals, organizations can monitor progress internally and in relation to competitors.

SLAs are critical because they provide business metrics and key performance indicators for organizations to manage against. SLAs specific to response time (e.g., a web page must render within three seconds) can be effectively measured with application performance management (APM) technologies. The primary goal of IT is to service the business, and well-defined SLAs provide a clear set of objectives, identifying the activities that are most appropriate to monitor, report, and build incentives around. Few organizations clearly define SLAs.

Service-level agreements should be designed with both organizational costs and benefits in mind. When set too low, business value is negatively affected. When set too high, additional and unnecessary costs may be incurred. Establishing and agreeing on the appropriate service levels requires IT and the business groups to work together to set realistic, achievable SLAs.

All systems must strive to avoid the culture of it’s not a problem until users complain. The non-functional goals for system performance must be part of the overall business requirements. Business requirements are the output of the inception (requirements and analysis) phases of any system development initiative.

Many business initiatives do not effectively define and track the non-functional requirements (NFRs) of response time, throughput, and scalability. Non-functional requirements are not limited specifically to performance, though all have an effect on a system’s ability to scale and perform. For reference, common NFRs include:

Key performance objectives and internal incentives should ideally support defining and reporting against service level compliance and regulatory compliance. This can be best accomplished by ensuring there are clearly defined regulatory compliance requirements and well-defined service-level agreements. Managing to these requirements and SLAs can then be enabled by identifying and executing activities to monitor, report, and build incentives around. Keep in mind that any undocumented requirements will likely be missed, and managing these requirements will not be possible.

A critical first step toward defining and implementing SLAs is the identification of the key business transactions, key performance indicators (KPIs), and system transaction volumetrics. Development and PE teams should begin the discussion of service-level agreements and deliver a draft at the end of each analysis phase within each iteration. For example, these may include transaction response times, batch processing requirements, and data retention requirements.

Regulatory requirements including access control, confidentiality, and logging should also be addressed at this time. The requirements will help determine if a performance test and proof-of-concept design validation test is required in order to verify that specific service levels are achievable while meeting these requirements.

Large organizations frequently don’t define service-level objectives, and find it difficult to meet these objectives when they’re added later, during analysis phases; enforcing them is therefore a challenge.

Performance goals and non-functional requirements must be defined to ensure that a system can be effectively managed.