Chapter 5

Criteria for Assessing Safety Performance Measurement Systems: Insights from Resilience Engineering

Tarcisio Abreu Saurin, Carlos Torres Formoso and Camila Campos Famá

Although the use of safety performance measurement systems (SPMS) is an important practice, the literature does not offer clear recommendations on how they might be assessed. In this chapter, six criteria for assessing SPMS are proposed, which are founded on the resilience engineering paradigm. The use of those criteria is illustrated by two case studies, in which the SPMS of two construction companies were evaluated. The insights obtained from using those criteria are unlikely to result from general criteria for assessing performance measurement systems.

Introduction

Performance measurement is closely related to the four abilities of resilient systems proposed by Hollnagel (2009): responding, monitoring, anticipating and learning. Indeed, it can support the identification of disturbances the system should respond to as well as producing data that enable learning to take place systematically. The connections between performance measurement and the monitoring and anticipating abilities are straightforward, since the former may be interpreted as equivalent to data collection and the latter should be a consequence of analysing the data produced.

A number of studies have discussed performance measurement, considering ideas from the RE perspective (for example, Wreathall, 2011), including the proposal of methods for developing resilience-based early warning indicators (Oien et al., 2011). The aim of this chapter is to propose a set of criteria for assessing safety performance measurement systems (SPMS) from the RE perspective, which are intended to complement general criteria that any performance measurement system should meet (Neely et al., 1997). This set of criteria was applied in two case studies, carried out in the construction industry. Although a number of studies on RE place an emphasis on complex socio-technical systems that involve intensive use of automation, such as aviation and power plants, previous studies have shown the benefits of applying RE to the construction industry (Saurin et al., 2008). Indeed, construction is recognized for having typical characteristics of complex systems, such as uncertainty and a large number of dynamically interactive elements. RE principles and methods are particularly suitable for this type of environment, in which human performance variability is frequent and necessary for successful performance (Hollnagel, 2012).

Criteria for Assessing SPMS from the RE Perspective

As the RE abilities are generic for any aspect of safety management, they need to be translated into other constructs, which can support the operationalization of RE for the purpose of assessing SPMS. The proposed set of criteria for assessing SPMS is focused on the contents and efficiency of the SPMS. Of course, organizational learning, as well as the implementation of actions arising from the insights provided by the criteria, are required if they are to make a real contribution to safety performance. The proposed criteria are:

•  (a) The SPMS should monitor normal work: this criterion arises from a distinctive premise of RE, namely the acknowledgment that accidents are not intrinsically different from normal performance. ‘Normal’ should be interpreted in the sense of everyday work (Hollnagel, 2012), with all shortcuts that gradually become incorporated in the routine and are accepted as normal, at least until a serious accident happens. Monitoring of normal work should shed light on the reasons for performance variability, since this supports its management (Macchi, 2010).

•  (b) The SPMS must be resilient: this criterion arises from the dynamics of a complex system, which implies that the SPMS should have the ability to adapt in order to continue capturing relevant information (Oien et al., 2011). In order to check if and how the SPMS is adapting, it should also be monitored, such as, for instance, by means of external audits and metrics to assess its efficiency and effectiveness. Indeed, a non-resilient SPMS may deteriorate (for example, when people stop collecting and analysing data) and become obsolete.

•  (c) The SPMS should monitor hazards throughout the socio-technical system: a source of hazard might be fairly distant, in time and space, from the situations it affects. RE recognizes this fact, once it proposes that the combination and propagation of the normal variability of multiple functions, over time and space, can lead to unexpected outcomes (Hollnagel, 2009). Thus, a broad scope of hazard identification and monitoring should be adopted, comprising the technical, social, work organization, and external environment dimensions of a socio-technical system.

•  (d) The SPMS should get close to real-time monitoring: due to the dynamics of a complex system, the information provided by SPMS might become out-of-step in relation to the system’s status (Hollnagel, 2009). Thus, when performance information is delivered to its users, the system might no longer be as it was at the time of data collection. This criterion implies that there should be reduced the time lag between the events and data analysis. A possible alternative to shorten this lag is the decentralization of the tasks of collecting, analysing and disseminating information generated by the SPMS. In fact, distributed control is typical of complex systems, and the design of SPMS should take advantage of this fact.

•  (e) The SPMS should take into account performance in other dimensions of the organization: this criterion is derived both from the RE premise that safety cannot be separated from business and from criterion (c). As a result, the SPMS should permeate all areas and activities, not only those normally associated with safety (Hopkins, 2009). Based on this criterion, it can be assumed that the other performance measurement systems, such as quality and environment control, may indirectly provide important information for SPMS.

•  (f) The SPMS should balance the trade-off between completeness and ease of use: due to constraints of resources and time, human performance cannot maximize efficiency and thoroughness at the same time (Hollnagel, 2012). A SPMS is not immune to the efficiency-thoroughness tradeoff. From the perspective of completeness (or thoroughness), the evaluation of safety in complex systems cannot focus on few indicators and subjects, otherwise it will not capture the nuances that comprise the situation. From the perspective of ease of use, which is an aspect of efficiency, any performance measurement system must be cost-effective. The use of an underlying safety paradigm, such as RE, helps to manage this trade-off, to the extent that there will be guidelines on what is important to measure and what is not.

Four out of the six criteria presented above are innovative, in comparison with either general criteria for assessing performance measurement systems or safety management approaches limited to compliance with regulations. They are: SPMS should monitor normal work; SPMS should monitor hazards throughout the socio-technical system; SPMS should get close to real-time monitoring; and SPMS should take into account performance in other business dimensions. Moreover, these four criteria are fairly safety-specific, as it is not obvious how they could be relevant for other business dimensions.

The other two criteria might be portrayed as reinterpretations of existing criteria from the RE perspective. The criterion stating that the SPMS should be resilient resembles the well-known recommendation, for performance measurement systems in general, that they should be themselves an object of continuous improvement (Neely et al., 1997). However, the RE perspective provides guidance on which grounds improvement should take place, such as by learning from normal work and taking into account performance on business dimensions that are not directly associated with safety.

The criterion stressing the need for the SPMS balancing the trade-off between completeness and ease of use resembles the advice that any performance measurement system should be cost-effective (Neely et al., 1997). However, the proposed criterion translates the cost-effectiveness trade-off in a more meaningful language for an SPMS. Indeed, while ease of use has an impact on cost, completeness has an impact on the SPMS effectiveness. Moreover, the RE perspective supports the identification of what is worth measuring by an SPMS, which is an issue neglected by generic recommendations without an underlying safety paradigm.

Also, while general criteria state that the metrics should be aligned with strategies (Neely et al., 1997), it is not straightforward to analyse this alignment when the focus is on safety. In this study, it is argued that both a company strategy and a safety management paradigm share an important commonality, as they provide a vision of what a company should be. In this respect, the proposed criteria provide a basis to assess the extent to which an SPMS is aligned with the RE view, which may be useful for companies that pursue the implementation of RE principles.

Research Method

Two construction companies (A and B) in Brazil were selected for the case studies. Company A’s main activities were the development and construction of buildings projects for middle-and higher-middle-class customers or residents, and it had about 1,200 employees. Company B was mostly focused on complex and fast hospital and industrial building projects. It had about 200 employees. The safety management systems of both companies were quite similar, as they had a number of noteworthy practices in common, such as a full-time safety specialist in all building sites and weekly construction planning meetings, involving both production and safety staff. Both companies had safety indicators that were not limited to those required by regulations, and had been implemented in a fairly standardized way in most of their building sites.

As the criteria for assessing an SPMS from the RE perspective were fairly abstract, it was necessary to break them down into sub-criteria which could guide data collection (Table 5.1). Two sites of each company, which were considered to be typical for their SPMS, were chosen for collecting data. In each site, a member of the research team carried out four visits over a period of two months, each of them lasting approximately two hours. In the first visit, the aim was to obtain an overall understanding of the SPMS, emphasizing the identification of both the metrics adopted and of the procedures of data collection, analysis and dissemination. In the three remaining visits, doubts arising from the first visit were clarified and the investigation moved towards more specific topics, as the researchers looked for the sources of evidence listed in Table 5.1. After concluding data collection, the researchers prepared a report on the evaluation of the SPMS, which, in both companies, was discussed with production and safety staff.

Table 5.1    Criteria and sub-criteria for assessing SPMS from the RE perspective

Results

Main Characteristics of the SPMS of the Companies Investigated

Table 5.2 presents the main characteristics of each company SPMS. These characteristics indicate a concentration of tasks on the safety specialists, and a very low involvement by top managers and production managers, especially in company A.

Table 5.2    Main characteristics of the SPMS of companies A and B

Safety Indicators Used by the Companies

Five indicators were used in both companies. While three of them are fairly straightforward (accident frequency rate, near misses frequency rate and index of training, which measured the amount of training provided to workers), clarification is necessary for two other indicators. The Percentage of Safety Activities Concluded (PSAc) indicator was inspired by another indicator adopted in both companies, called Percentage of Production Activities Completed (PPAC). The PSAc formula measures the ratio between the number of safety activities concluded and the number of safety activities planned. Although there was no formal definition of what was regarded as a safety activity, the observations indicated it was concerned with implementing physical protections (for example, guardrails) and access equipment to work stations, such as ladders. The PSAc was monitored weekly, and the causes that led to the non-completion of safety activities were discussed. The aim of the NR-18 index (INR-18) was to evaluate the compliance with the main Brazilian regulation concerned with construction safety, called NR-18. The INR-18 was calculated from a checklist that had 213 items, corresponding to the ratio between the total of items marked with yes (meeting the regulation) and the total of items marked with yes or no.

Three indicators were used only in company A: estimate of fines due to non-compliance with NR-18; the number of production stoppages due to lack of safety; and the index of subcontractors’ performance, which took into account a number of requirements applicable to the subcontractors. Two indicators were used only in company B: frequency rate of first-aid accidents; and index of compliance and commitment, which assessed the number of safety notifications solved within the deadline established by the safety specialist, who carried out a daily inspection of site activities.

Evaluation of the SPMS Based on the Proposed Criteria

The SPMS Should Monitor Normal Work

Five out of the ten indicators existing in both companies (accident frequency, near miss frequency, estimate of fines, frequency of first aid accidents, number of stoppages due to lack of safety) are focused on measuring adverse events. Rather than monitoring normal work (sub-criterion 1.1), they monitor more or less unfrequent events that reflect lack of safety, instead of its presence. Nevertheless, those indicators could give insights into normal work, provided the descriptions of adverse events were compared with the work prescribed. This analysis could reveal adaptations that have been incorporated into the day-today routine.

The other five indicators (index of training, index of compliance with NR-18, index of subcontractors performance, percentage of safety activities completed and index of compliance and commitment) are focused on analysing normal work and they monitor either the presence of safety or actions that have been adopted to create safety, such as training and planning. However, the index of compliance and commitment is the only indicator that involves observing people working, which is necessary for identifying human performance variability, a key concern from the RE perspective. Indeed, the other indicators might be limited to the observation of the technical system. Although information on the sources and reasons of variability (sub-criterion 1.2) could be extracted from the ten indicators existing in both companies, such extraction was limited to the sources and reasons of variability leading to unsuccessful outcomes. Indeed, when analysing data arising from all indicators, the staff of both companies focused on identifying what went wrong and why, rather than identifying what went right and why. Data collected by the researchers showed that learning opportunities were missed due to this approach, since successful outcomes were not necessarily due to adherence to formal system design.

The SPMS Should Be Resilient

The evaluation according to this criterion was hindered by the duration of the case study. In fact, two months were not enough to detect substantial changes in the procedures for collecting, analysing and disseminating the metrics (sub-criterion 2.1) as well as to detect major changes in the metrics, such as inclusions, exclusions, or adaptations (sub-criterion 2.2).

The resilience of the SPMS could also be enhanced based on insights arising from formal assessments of their effectiveness and efficiency (sub-criterion 2.3). However, none of the companies had procedures for evaluating their SPMS, relying mostly on gut feelings of the safety staff. Nevertheless, the SPMS provided a large amount of information, which, if properly interpreted, could be used for such assessments, in order to reduce the dependence on safety staffs’ individual insights. For example, the pieces of information used to calculate five indicators (frequency rate of near misses, number of stoppages, accident frequency rate, index of compliance and commitment and first-aid) could potentially point out hazards of any nature. Therefore, those indicators could be interpreted as meta-monitoring mechanisms, since they provided data for monitoring the SPMS itself. In companies A and B, for example, the need for the indicator called index of training could be questioned, as the lack of training was not identified as a major contributing factor to accidents, near misses and non-execution of safety activities.

The SPMS Should Monitor Hazards Throughout the Socio-technical System

Even though eight indicators existed in company A, and seven in company B, the implicit definition of what counted as a hazard (sub-criterion 3.1), and should therefore be monitored by the SPMS, was limited. Some indicators had a narrow focus on certain elements and hazards of the socio-technical system, such as the index of compliance with NR-18, which detected failures related to the technical system, that is, whether physical protections were installed and kept in good conditions. By contrast, other indicators, as mentioned in the previous section, could potentially monitor a broader range of hazards. Of course, a thorough analysis of the data used by these indicators would be necessary to check the extent to which this really happens. For example, it might be the case that workers have not reported hazards that have been incorporated into their routine, and thus certain types of hazards would not be monitored by the frequency rate of near misses.

The analysis according to the sub-criterion 3.1 also provided insights about process safety monitoring. This task was performed as part of quality control, such as conducting tests on the performance of materials, and visual inspections for checking the maximum loads stored on a floor. Those procedures were part of a certified quality management system that was independent from the safety management system, which was focused on personal safety. Thus, safety staff and workers were not involved in the monitoring of process safety hazards, as they were not aware of the safety implications of the quality management procedures.

Concerning process safety, it is also worth noting that neither of the companies had an indicator to monitor accidents with material damages. Nevertheless, small accidents of this nature seemed to be frequent in both companies. For example, in one of the visits to a site of company B, the researchers noticed that a wall had collapsed, as a result of strong winds on the previous night. However, the safety specialist reported that he was not concerned with documenting and investigating that particular accident. The specialist took it for granted that the investigation of those types of accidents required technical knowledge in civil engineering. However, even the civil engineers who were legally responsible for the construction site were reluctant to take responsibility for process safety issues, since they, in turn, were relying on the knowledge of outsourced experts, such as those responsible for designing scaffolds, trenches and excavations. This context means that no one who worked full time on the construction sites was fully aware of all process safety hazards and on how they should be monitored. The possibility of monitoring safety performance during the whole product life-cycle was also neglected by both SPMS (sub-criterion 3.2). This could be done, for example, by monitoring safety during the product design stage, assessing the extent to which each design discipline (for example, architecture, utilities and so on) is complying with good practices of safety in design.

The SPMS Should Get Close to Real-time Monitoring

The data processing and analysis cycles of the SPMS were relatively long. Only two out of the ten indicators adopted by the two companies were monitored on a daily basis: the index of training, and index of compliance and commitment. Overall, the evaluation based on sub-criterion 4.1 pointed out that feedback lagged substantially behind the moment at which events of interest took place. This feedback lag was due to: (a) the collection of data from past events (for example, accidents); (b) the collection of data concerned with unsafe conditions that could be in place for quite a while (for example, the lack of guardrails); (c) the fact that only one employee in each construction site, the safety specialist, centralized data collection, analysis and feedback, implying an overload (this was in conflict with sub-criterion 4.2); and (d) the delay involved in preparing reports and submitting them to the interested parties.

The SPMS Should Take into Account Performance in Other Business Dimensions

The SPMS of both companies did not have mechanisms for assessing how safety was performing in comparison with other areas (sub-criterion 5.1). Nevertheless, the existing SPMS had information which allowed some insights in this respect. For example, the SPMS could give visibility to the trade-off between safety and production, by calculating the ratio between the indicators PSAc (related to completion of safety activities) and PPAC (related to the completion of production activities). Concerning sub-criterion 5.2, companies A and B did not interpret the indicators from other areas from a safety perspective. Of course, some indicators did not have relevant links with safety (for example, number of complaints from clients) and interpretations from this perspective would be counter-productive. By contrast, other indicators were clearly relevant for safety. For example, both companies had indicators that monitored deviations from time and cost targets. If project time and cost are higher than expected, this should work as a warning that competition for the available resources (for example, money, time and labour) is increasing, and that resources that otherwise would be allocated to safety may be allocated elsewhere.

The SPMS Should Balance the Trade-off Between Completeness and Ease of Use

The trade-off between completeness and ease of use was managed intuitively by the safety staff of both companies, who were responsible for designing the SPMS. Since the safety staff were, at the same time, the designers and the main users of the SPMS, their goals and knowledge guided them in this regard. Therefore, they chose indicators that were fairly easy to collect and that provided meaningful information, from their viewpoint. This can be interpreted as an adaptive strategy adopted by the safety staff, in order to keep the SPMS compatible with the existing human, technical and financial resources (sub-criterion 6.1). Nevertheless, the centralization of tasks on the safety staff contributed to disseminate the idea that safety management was a subject of interest just for safety experts.

The only indicator whose objectives were misunderstood by managers (sub-criterion 6.2) was the PSAc. Instead of assessing whether the production activities were being carried out safely, as believed by management, the PSAc assessed if the activities of installing physical protections had been completed. While the ease of use was as a natural concern for the designers of the SPMS, the same concern did not exist with completeness. Indeed, assessing the completeness of a SPMS is much more difficult than assessing its ease of use, since it is impossible to know, and as a result to monitor, all hazards a complex system is exposed to. Nevertheless, the evaluation of sub-criteria 3.1 and 3.2 (the SPMS should monitor hazards throughout the socio-technical system) pointed out gaps concerning the completeness of the SPMS of both companies. The incompleteness of the SPMS was also a result of their over-reliance on the quantitative data produced by the indicators (sub-criterion 6.3). The frequency rate of near misses was an example of an indicator that provided quantitative data that was of little relevance in comparison with the qualitative data which was necessary for calculating the indicator (that is, the descriptions of near misses). Thus, the available pieces of evidence suggest that, in both companies, the trade-off between completeness and ease of use was pending in favour of ease of use.

Conclusions

This chapter presented a set of criteria for assessing SPMS that is founded on the RE paradigm. Based on the RE perspective, it was possible to obtain insights that would not have been obtained based on criteria normally used to assess performance measurement systems in general. Due to the abstract character of the criteria, their application requires personnel who are familiar with RE and are domain experts. Moreover, the criteria, sub-criteria and sources of evidence need to be refined by extending its use to a large number of projects. As an indication of the usefulness of the criteria, a number of improvement opportunities were detected in the SPMS of two construction companies in which they were applied, such as: (a) the reports on safety performance should include key indicators from other areas, such as those related to project time and cost, as they can be proxy measures of the intensity of production pressures; (b) specific indicators could be designed to assess the trade-off between safety and production, such as the ratio between PSAc and PPAC; and (c) the SPMS should take a broader view on what counts as a hazard. Process safety and organizational hazards are neglected in favour of monitoring the more visible hazards emphasized by regulations (for example, falls, shocks and so on).

Commentary

Performance measurements and performance indicators are the necessary foundation for management – of processes, of production, of safety and of resilience. A performance measurement system must furthermore be cost-effective. The notions of cost and effectiveness require a practical foundation. This chapter argues that the concepts of resilience engineering can be used for that purpose and illustrates that by looking closer at how two construction companies improved their safety performance measuring systems.