Chapter 11

Measuring Resilience in the Planning of Rail Engineering Work

P. Ferreira, J.R. Wilson, B. Ryan and S. Sharples

The significant pressures under which UK rail infrastructure currently operates provide ample research grounds for the field of resilience engineering. One of the areas on which these pressures mostly impact is the planning and delivery of engineering work. Resilience engineering was proposed as a framework for research aiming to improve the ability of the organisational system responsible for the planning of all engineering work to respond to these pressures. Within this scope, an approach to measuring resilience was developed by means of a questionnaire. A factor analysis method was used to identify underlying trends from the questionnaire data, which could then potentially be used as measurable aspects of resilience in rail engineering planning.

Introduction

The demand for increased capacity of the UK rail network has generated growing pressure to improve the planning and delivery of engineering work. As the owner of the UK rail infrastructure, Network Rail faces the challenge of delivering increasing volumes of work (maintenance, enhancements and renewals) within more diverse and shorter opportunities for access to the infrastructure, while maintaining the safety performance standards imposed by the regulatory bodies. A balance between productivity pressures and the assurance of the required safety standards has become critical for the sustainability of the rail organisation.

Resilience engineering was proposed as a framework for research aiming to understand and improve the planning system for rail engineering work delivery and its protection. The purpose is to assess the preparedness of the system, not only to respond to unforeseen (and unforeseeable) events, but also to manage known threats and pressures. This is to be achieved by looking at what aspects provide the planning system with potential for resilience (Woods, 2006a) as well as those that may erode this potential. More precisely, this research contemplates the following four major steps:

•  The identification of key aspects of system operation by means of an interview process (Ferreira et al., 2008).

•  The identification and assessment of parameters which describe planning performance based on analysis of the industry’s historic data records.

•  The identification and assessment of resilience factors applicable to the context of rail engineering planning.

•  The comparison between different geographical areas of the railways in terms of system operation, planning performance parameters and resilience factors.

Given that the context of this book is addressing practical aspects of resilience, only the third work stream concerning the assessment of resilience factors will be discussed in the chapter.

Measuring Resilience Factors

A questionnaire was developed according to key concepts that characterise a resilient and a non-resilient system. After implementing the questionnaire at national level, a factor analysis was applied to the data, aiming to extract underlying trends as indicators for the level (and type) of resilience maintained by the system.

Questionnaire Design

The questionnaire has three sections. The first section aims at the assessment of resilience factors and will be the object of this discussion. The remaining two sections are dedicated to the assessment of the planning aspects identified and discussed throughout the interview process. These two sections will not be addressed here.

Woods (2006a) and Wreathall (2006) provide a broad range of concepts as indicators for the presence or absence of resilience in systems. Similar to the approach followed by Mendonça (2008), this was considered an obvious starting point for any attempt towards measuring resilience. Table 11.1 shows the concepts extracted from the literature sources and provide a brief description for each.

Table 11.1  Resilience concepts (from various chapters in Hollnagel et al., 2006)

While maintaining the definitions given in Table 11.1, the earlier work developed within this research, in particular the interview process (Ferreira et al., 2008), provided grounds for outlining a set of questions aimed at the context of rail engineering planning. This initial group of questions was peer reviewed by the members of the Ergonomics National Team at Network Rail in order to test their comprehensiveness as well as their meaningfulness concerning the intended concepts. This gave rise to an iterative process of revision and piloting that was concluded when the format of each question was believed to be strongly related to the underlying resilience concepts. The initial set of questions was brought down to the 22 statements shown in Table 11.2.

Questionnaire Implementation

As mentioned above, within the larger frame of this research the purpose was to compare the outcome of this questionnaire against the other work streams using a common geographical basis. To comply with this, the questionnaire was implemented at a national level, aiming to obtain a sample not only with organisational relevancy, but also with similar geographical representation as the other work streams.

Planners were asked to give their rating on a scale of 6 (1: Strongly disagree, 2: disagree, 3: Slightly disagree, 4: Slightly agree, 5: Agree, 6: Strongly agree). A sample of 105 planners was obtained from an estimated universe of 210 (due to ongoing reorganisation processes no exact numbers were available). The estimate is based on an average of 10 people at each of the 21 planning units existing at the time at national level. Of the initial 105 cases 7 were excluded from the analysis process on the basis of missing data.

Principal Components Analysis

Before undertaking any factor analysis process, basic statistics were developed in order to verify the reliability of the data as well as their suitability for factoring. Skewness and kurtosis tests were run to verify the distribution of each variable, as well as reliability tests for internal consistency of the data. In addition, the inter-item correlation matrix showed a substantial number of significant correlations and ratios for partial correlations indicated good levels for factorability of data. On the basis of this preliminary analysis, all variables were taken forward for the factor extraction.

Several factor extraction solutions and methods were explored using SPSS (Statistical Package for the Social Sciences). The selection of the most appropriate solution took into consideration the concept of ‘simple structure’ described by Kline (1994). The best fit to the selection criteria was a five component solution one with orthogonal rotation (Tabachnick and Fidell, 2007). Table 11.2 shows the loading factors for each variable.

Table 11.2  Matrix for extracted components

Only loading factors above 0.400 were considered (shown in bold) and, where this led to multiple loadings, a minimum difference of 0.200 was imposed (Tabachnick and Fidell, 2007). According to these criteria the solution in Table 11.2 shows one non-loading (no loading factor above 0.400) and one cross-loading variable (more than one loading factor above 0.400 with difference between them below 0.200). The italicised characters indicate the items rejected on the basis of non-loading or cross-loading (‘I am encouraged to reflect on my planning’ and ‘Because something has always gone well before …’). Overall, loading coefficients are significantly high, which demonstrates a strong correlation between items and their loading components.

Interpretation of the Extracted Components

The possibility of matching the initial set of questions to the ‘four main resilience factors’ discussed in the course of this book was considered as a starting point. The four resilience factors were defined as follows.

•  Knowing what to do – The ability to respond to regular and irregular disruptions by adjusting normal functioning.

•  Knowing what to look for – The ability to monitor aspects of system performance and its operating environment which are, or could become a threat in the near term.

•  Knowing what to expect – The ability to anticipate developments and shifts in the operating environment on a long term basis, such as potential threats and pressures.

•  Knowing what has happened – The ability to learn from experience.

The fact that the most appropriate solution was the one extracting five components, counts against a direct match to the four main resilience factors. Thus, other possible relations were investigated using literature support. Ferguson and Cox (1993) suggest two methods for naming the extracted components. Both methods resort to a sample of judges as a way to develop an independent interpretation, which makes their use time consuming and requiring a rather large number of participants.

For this research, an approach was developed on the basis of the Delphi method (Turoff and Linstone, 1975). The 25 members of the Ergonomics National Team at Network Rail were used as the ‘discussion group’. Team members were asked to name each of the five groups of statements (variables loaded into each of the five components) according to what concept or idea they felt most accurately would describe that group, using as few words as possible. A total of 16 people responded with interpretations for each component. Table 11.3 summarises the expressions and concepts that were used by the majority of the respondents for each of the components.

Table 11.3  Interpretation of the extracted components

Based on the previous feedback, the names shown in Table 11.4 were proposed by the investigator. Beyond considering the key words that were most frequently used by the respondents, whenever considered adequate, interpretations made use of concepts found in the resilience engineering literature.

Following the Delphi approach, team members were then given the opportunity to confirm or dispute the proposed names in the light of their initial interpretations. Each respondent was given a new table showing their own interpretations against the proposed names and asked whether they agree with the given name or still prefer their initial interpretation. 12 out of the initial 16 people responded to this second inquiry. Table 11.4 indicates the percentage (of the 12 respondents) of confirmations obtained for each of the proposed names.

Table 11.4  Names proposed for each component and confirmation level

The interpretation for all five components was considered valid by the majority of respondents. Nevertheless, to improve confidence in the outcome of this process, a clarification was sought whenever challenges were made.

The initial interpretations tended to favour one particular sub-group of the questions in each component. This was most evident for component 3. Comments made by respondents pointed towards the high number of questions contained in this component and the fact that these (apparently) bring together a more diverse set of issues. This accounted for a lower number of confirmations obtained for this component.

While components 1 and 2 seem to have a higher focus on personal capabilities, components 3, 4 and 5 could be seen as shifting towards a more organisational nature. The fact that the cross-loading item (‘I am encouraged to reflect on my planning’) refers to an organisational cultural aspect and that it loads onto components 3, 4 and 5 supports this assumption. Within this frame of mind, having the organisational support to reflect on ones planning could be an important underlying condition to allow for an adequate performance with regard to the aspects comprised in components 3, 4 and 5.

The non-loading variable (‘Because something has always gone well before, I feel confident that it will continue to go well in the future’) was aimed at complacency issues. As shown in Table 11.3, none of the interpretations for the extracted components allude to these issues, which would account for the non-loading of the variable.

The definitions shown in Table 11.5 aim at placing the resilience concepts found in Hollnagel et al. (2006) within the context of rail engineering planning. The earlier work streams, namely the interview process (Ferreira et al., 2008), was also used as a background for this process.

Table 11.5  Definition of components

Extracted Factors and the Potential for Resilience

The extracted components underline a relation between the questionnaire and resilience engineering constructs, hence showing a potential use as measurable factors. The integration of these factors into the original data set as new variables (using SPSS) provides grounds for a quantified assessment. Although no reference can be given as to how much of each factor indicates a positive or negative contribute to resilience, the values obtained can be judged in the context of a geographical comparison. The outcome of the work stream focusing on planning performance will also provide ample grounds for comparison, not only on a geographical basis, but also by investigating variations in planning performance against the scores obtained on each of the extracted components.

The cross-system nature of the four resilience factors suggests that other parts of the organisation should be investigated, even if the focus is on one particular function. In other words, although this survey addresses engineering planning, it is likely that aspects such as the ability to learn (‘knowing what has happened’) will depend a great deal on other parts of the organisation. This can be put forward as one of the reasons why the outcome of the principal component analysis shows no direct relation to the four main resilience factors. It is likely that some of the aspects comprised within the four factors were beyond the scope of the planning system and therefore would not likely be captured by the respondents to the questionnaire.

On the other hand, the need to design questions in such a way that planners could relate them to their normal activities may be the cause of some distancing from a larger system perspective. In view of the questions format, the extracted components can only be interpreted from planners’ personal perspectives, even if indicating behaviours contributing to or eroding resilience. Although requiring further investigation, the five extracted components can be viewed as aspects of planners’ performance that would potentially contribute to the development of the system level abilities described by the four main factors.

Self-reporting methods, such as questionnaires, may be insufficient to provide a robust measure for resilience. However, such methods can be an efficient way of monitoring the system’s behaviour in terms of resilience, particularly on a longer term basis through periodical applications. The progress of this research will determine how significant the obtained parameters are for the overall potential for resilience in rail engineering planning and delivery. This is expected to set the path for fine-tuning the questionnaire towards a more accurate measurement of resilience, not only within rail engineering but also considering its transfer to potential applications in other organisational sectors.

Acknowledgements

This project was funded and facilitated by Network Rail. The authors wish to recognise the fundamental contribution of all those who very openly cooperated with the surveys here discussed.