Written work procedures: Identifying and understanding their risks and a proposed framework for modeling procedure risk

Safety Science 82 (2016) 382–392

Contents lists available at ScienceDirect

Safety Science journal homepage: www.elsevier.com/locate/ssci

Written work procedures: Identifying and understanding their risks and a proposed framework for modeling procedure risk Gregory Praino, Joseph Sharit ⇑ University of Miami, Department of Industrial Engineering, P.O. Box 248294, Coral Gables, FL 33124, USA

a r t i c l e

i n f o

Article history: Received 1 July 2015 Received in revised form 30 September 2015 Accepted 13 October 2015 Available online 11 November 2015 Keywords: Work procedures and rules Hazardous processes Procedural controls Risk Procedure failure likelihood

a b s t r a c t Organizations often direct considerable attention toward the identification and assessment of the various risks associated with hazardous process operations, and as part of their risk control system they typically rely on written procedures for guiding workers in carrying out the necessary task activities. However, these procedures can, in and of themselves, serve as sources of risk, which strongly suggests the need for methods that could enable organizations to efficiently assess the risks potentially intrinsic to their written procedures. This paper focuses on the identification, understanding, and modeling of the risks potentially associated with written work procedures. The idea of controls within procedures and a taxonomy of procedures based on the nature of a procedure’s controls are first presented. This is followed by a systematic reappraisal of the risks resident in written procedures that are incurred through the processes of development and management of procedures by organizations. The focus then shifts to the implications for an organization’s risk control system of behavioral variability associated with carrying out procedures. This leads to the presentation of a proposed modeling framework intended for translation or adaptation by organizations as a practical means for assessing what is referred to as ‘‘procedure risk”—the risk resident in procedures. Key concepts that are emphasized in this framework are the value of a procedural control and the likelihood of failure of a procedural control. Guidance is provided concerning a possible way for instantiating the modeling framework through a case study involving space shuttle ground processing operations. Ó 2015 Elsevier Ltd. All rights reserved.

1. Introduction Most organizations rely on procedures or rules, usually conveyed in some type of written format, as the means by which relevant knowledge and actions governing the performance of potentially hazardous work activities are communicated to workers (Hale and Swuste, 1998). These procedures, which are often considered to be ‘‘cornerstones of the risk control system,” encompass ‘‘controls” suggested from an organization’s risk analysis studies that are intended for meeting the organization’s commitments to its safety and mission goals (Hale and Borys, 2013a). However, despite these intentions, it has been well documented that procedures can, in and of themselves, contribute to the causation of incidents or accidents (e.g., Reason, 1997; Sharit, 1998; Dekker, 2005; Alper and Karsh, 2009; Hollnagel, 2009). Generally, procedures that are ambiguous, poorly understood, or not rationalized; are cumbersome in their content (due in part to incremental aggregation of content without reevaluation of the rule); are effortful to ⇑ Corresponding author. Tel.: +1 305 284 6472. E-mail addresses: [email protected] (G. Praino), [email protected] (J. Sharit). http://dx.doi.org/10.1016/j.ssci.2015.10.002 0925-7535/Ó 2015 Elsevier Ltd. All rights reserved.

carry out; provide little guidance concerning appropriate actions to take when conditions that are novel or unanticipated are encountered; offer little room for improvisation (especially by skilled personnel) that could potentially improve system performance; are perceived by workers as imposing on them unacceptably high risks; or are resistant to changes due to the inability for workers to communicate to management or designers insights obtained from performance of the procedures, will lead to personnel either failing to perform the procedure, performing it incorrectly, or violating the procedure, and more generally to less resilient organizations (Woods, 2006). Two relatively distinct models of procedures can be contrasted (Dekker, 2005; Hale and Borys, 2013a, 2013b): model 1, which views procedures or rules as a set of relatively rigid prescriptive norms imposed by management on its workers; and model 2, whose rules are more accurately described as ‘‘routines” that emerge from adaptive responses to highly variable and often complex situations, and which often require deviation from any prescribed rules in order to meet performance goals. In this conceptualization of procedures, model 1 rules can be viewed as top-down and more static in nature, devised by experts who are

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

knowledgeable with regard to the process activities, the tasks governed by these activities, and the risks inherent to the process and associated tasks. These individuals may feel compelled to clearly specify behaviors that need to be performed, through documentation, communication, and training, in order to counteract potential fallible human tendencies that might arise from the limited competence or experience levels of the workforce. In contrast, model 2 rules are bottom-up and more dynamic, intentionally underspecified in their written content in order to allow presumably higher skilled personnel to determine how quantitative or qualitative performance goals should be achieved. Both model 1 and model 2 rules have strengths and weaknesses that are largely determined by the correspondence between the nature of the task activities underlying system processes, the skill level of the workers responsible for these task activities, and the degree of standardization or flexibility implicit to the procedures governing these activities (Schulman, 2013). On one extreme, when the knowledge base concerning the process is relatively complete (implying low input variance) and the task activities supporting the process are performed repetitively and with few surprises (implying low system performance and safety variance), highly specified procedures can be prescribed for workers who lack deeper process knowledge in order to ensure standardization and uniformity of the process. On the other extreme are situations for which the existing organizational knowledge base is incomplete or informal and task conditions are unexpected and changing (implying high input and high output variance). These conditions would benefit from skilled workers who would be undermined by highly detailed and inflexible procedures. Instead, such workers would likely need to resort to pattern-recognition and intuitive skills shaped by the linking of highly contextualized past process situations to system outcomes in order to transform fluctuating inputs, which may be signifying conflicting goals, into low-variability high-quality outputs. Responses or ‘‘routines” deemed successful by these skilled workers would then have the opportunity, assuming a resilient, ‘‘learning” organization (Wreathall, 2006), to become dynamically embedded into the largely informal knowledge base of the organization’s work culture. Between these two extreme cases are various situations which may require compromises incorporating elements of both model 1 and model 2 rules (Grote et al., 2009), with the optimal instantiation of any type of rule model ultimately depending on factors related to the organization’s rule-management process (Hale and Borys, 2013b). As implied above, procedures generally encompass rules that can be thought of as ‘‘controls.” ISO 31000 (2009) defines controls as any process, policy, device, practice, or other actions which modify risk (risk will be defined more explicitly below), and importantly, which may not always exert the intended or assumed modifying effect. Within procedures, controls typically define concrete actions that need to be taken under particular conditions (e.g., if the radiation levels exceed 20 rad, immediately back out from the vessel and ensure that it is sealed), or that require specific system states be established (e.g., do not initiate operations until inspection of the pump has determined that its insulation is not compromised). Such controls derive from the considerable attention that organizations often direct toward the identification of the various risks associated with hazardous (i.e., potentially harmful) work operations and to the quantitative or qualitative assessment of those risks (Center for Chemical Process Safety, 1992; Kumamoto and Henley, 1996; Sharit, 2012). However, although procedures are invariably in place for carrying out these activities as part of an organization’s risk control system, organizations

383

currently lack guidance or methods for enabling them to efficiently assess, ideally early on in the design of their procedures, the risks potentially intrinsic to these written procedures themselves. These risks can arise in part from the fact, as ISO 31000 emphasizes, that the controls in place within these procedures ‘‘may not always exert the intended or assumed modifying effect.” The primary purpose of this paper is to elaborate on a modeling framework that an organization’s management could use or adapt as a tool for estimating the relative risks implicit to their written work procedures. To put this problem in perspective, issues related to the generation, identification, and understanding of what will be referred to as ‘‘procedure risk”—the risks resident within procedures—will first be presented. Also, although the modeling framework is presumed to address mainly model 1 rules, which are often referred to as standardized work procedures, the boundaries which define this class of rules are often not very sharp. Thus, the framework is considered to be applicable as well to ‘‘process rules” (Grote et al., 2009). Such rules, though they might specify the process by which task activities should be undertaken, still allow some leeway with regard to how these activities can be accomplished. To clarify the term ‘‘procedure risk” and other references to terminology related to risk used throughout this paper, we follow the conventions offered in ISO 31000 (2009) in which risk is defined broadly as an ‘‘effect of uncertainty on objectives.” The effect is some deviation from the expected; uncertainty refers to a state of deficiency related to the understanding or knowledge of an event, its consequence, or likelihood; an event can be an occurrence of a particular set of circumstances (which could include incidents or accidents); and an objective can encompass different aspects (e.g., financial cost, health and safety, and environmental goals) and can apply at different levels (e.g., at the product, process, or organization-wide level). In addition, ISO 31000 defines a risk source as an element which has the intrinsic potential to give rise to risk; risk analysis as a process for comprehending the nature of risk and for determining the level of risk (which is often expressed in terms of the combination of the consequences and corresponding likelihoods associated with an event); and risk management as the coordinated activities needed to direct and control an organization with regard to the risks that it may encounter. Notably, while the consideration of the combinations of consequences and likelihoods associated with an (adverse) event is a common way of expressing risk, the ISO 31000 definitions clearly imply a broader context in which risk can be considered, and one that is consistent with the central concept of ‘‘procedure risk.” As will be argued, the problem of risk analysis as applied directly to written work procedures, which are regarded here as risk sources, benefits from this broader context as conventional approaches for expressing risk that rely on explicit identification of consequences and corresponding likelihoods of an event—which in this case represents the circumstances stemming from instantiation of the procedure—would not be practical from the standpoint of an organization’s risk management process. This paper is structured as follows. First, the central idea of controls within procedures is examined, followed by a proposed taxonomy of procedures and the corresponding role of controls within this taxonomy. Next, we consider the potential risks resident in written procedures that are incurred through the processes of development and maintenance of procedures by organizations and the implications of behavioral variability associated with carrying out these procedures. We conclude with a proposed modeling framework for assessing procedure risk, which we believe can be used to guide risk management strategies directed at the

384

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

process of managing an organization’s procedures. To illustrate the modeling framework, a case study related to NASA’s space shuttle ground processing operations is presented. This application area formed the basis for many of the ideas related to the proposed methodology. 2. The concept of controls in written procedures As noted, a control is a type of essential instruction purposely designed within a procedure, often in the form of a rule, which reflects an organization’s safety (and mission) goals (Johnson and Gill, 1993), although the detailed actions on how to accomplish such goals do not necessarily have to be provided. Controls can be further differentiated in terms of whether they apply to the process during normal operating modes or are intended to protect a process from the effects of an undesired state. Because operating controls are intended to successfully transition the process from desired state to desired state, protecting controls are needed when the process fails to move into the next desired state, either to mitigate the consequences or to return the process to a desired state. Controls can also be differentiated on the basis of how they work on the process. Those that are directing steer the process to achieve a specific state, whereas controls that are limiting establish the state conditions that are to be avoided. There is a larger context, however, within which such controls exist and must be evaluated in terms of their prospects for being successful. In particular, Perrow’s (1999) distinctions between linear and complex systems and especially between tightly and loosely coupled systems, which form the basis to his system theory of accidents, help to define these larger contexts in which procedures are carried out and enable a clearer perspective on the risks associated with the various controls. For example, if processes within a system are tightly coupled, then a system parameter exceeding its limit may result in an immediate rise in another system variable that can potentially place the system at risk. Controls that are in place within the procedure thus may not be sufficiently effective in deterring the consequences of such tight coupling. 2.1. Characteristics of procedures and the role of controls A proposed taxonomy of procedures based on seven independent dimensions or characteristics that describe how the procedural control works, as well as the effect the control is intended to elicit from the system, is shown in Table 1. Two of these dimensions (structure and level of detail) comprise a continuum of possible values, while the remaining five dimensions are defined in terms of discrete categories. The purpose characteristic describes the way in which the activity controlled by the procedure influences the system. Operating

Table 1 Seven proposed dimensions characterizing procedures and associated control attributes.

tasks provide workers with information about activities that comprise the direct path to the desired output, whereas protecting and restoring tasks are the activities needed for assuring continued ability to provide the system output in the face of inherent system uncertainty. Examples of protective controls include physical barriers, preventive maintenance, quality inspection, and even risk analysis; restoring controls include corrective maintenance and emergency activities to secure an out-of-control system. Nature is a characteristic of the task that reflects whether a procedure is fixed by the operation of the system or is established at the discretion of the process designer, the procedure author, or the workers themselves. An inherent control is one that is a feature of the system configuration and system states as, for example, in the specified order of operations in a production line, whereas an imposed control might be the communication of a decision to the workers to resolve an ambiguity in the process flow or system states. Structure concerns how thoroughly the procedure addresses the associated tasks and the possible variations. A limited procedure may only have controls related to a sub-process or a portion of the time a system will be running, whereas a comprehensive procedure will contain controls that, for example, might relay task information for all possible settings on a machine. The target characteristic relates to whether the control is aimed at controlling the processes of the system or the output of the system. A processtargeted control assumes that an effectively managed process is necessary to sustain acceptable outputs; output-targeted controls make no such assumptions and aim to directly control the resulting product or service. The level of detail characteristic refers to the degree of specificity in the description of an individual control. Whereas goaloriented controls typically allow for flexibility in how goals will be achieved, rule-oriented controls will provide the step-by-step details necessary to accomplish the task under the anticipated conditions. Method refers to the differentiation noted earlier between directing and limiting controls, where the former intend to create or maintain a desired state while the latter seek to avoid an undesired one. Finally, the duration characteristic of a procedure relates to the timeframe over which the worker exerts effort to comply with the procedure. Momentary procedural tasks have inherent endpoints such as flipping a switch or entering an input and may be unique or periodic. Sustained procedural tasks, in contrast, are executed until a procedural cue terminates the task as when monitoring temperature to ensure a system remains within its defined temperature limits. 3. Procedure risk Although a written procedure exists to communicate a formalized process to a worker, there are numerous ways in which

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

procedures can fail to accomplish this objective and thus serve as sources of risk. Below, we consider procedures as sources of risks deriving from their violation and from the management processes responsible for their development and maintenance. 3.1. Risks related to violations of procedures Violations of procedures, which are a critical concern in organizations responsible for managing potentially hazardous process operations, are typically classified as routine, situational, exceptional, or optimizing (Reason, 1990; Hale and Borys, 2013a). The factors contributing to violations are examined more closely in Section 4.1 from the standpoint of human behavioral variability in the face of various issues related to procedures. Routine procedural violations have been identified as contributing causes in a wide variety of accidents, including the crash of ValuJet flight 592 (NTSB, 1996) and the loss of the space shuttle Columbia (Columbia Accident Investigation Board, 2003). These typically arise when short-cuts in time or effort are taken in place of the specified rules, and the perpetrator is reasonably certain that such deviations will not lead to an accident, disapproval by coworkers, or disciplinary action, thereby reinforcing the violation and enmeshing it into the work culture. Situational violations occur when the available rule does not appear to be adequate or appropriate for a particular scenario that arises, resulting in improvisation by the worker. In a resilient organization, improvisations that lead to positive outcomes can, in turn, lead to rule refinement and greater specificity regarding the boundaries within which the rule applies and actions to undertake if the situation falls outside of these boundaries (Weick and Sutcliffe, 2007).

385

Exceptional violations are much less common and typically occur when confronting an unexpected situation under extreme time pressure that may induce a worker to perform, for instance, an unadvisable system shutdown. Optimizing violations are more likely to be performed by workers who are highly-skilled or with strong creative mindsets who must solve relatively difficult problems related to trading off safety objectives with production goals or feel compelled to explore the boundaries of system operation. 3.2. Risks related to development and maintenance of procedures The ways in which an organization creates and maintains procedures can dramatically impact the degree to which procedures can serve as sources of risk. Fig. 1 portrays a framework of a rule management system that can be used for identification and understanding of such procedural risks. 3.2.1. Designer responsibility As the architect of the process, the designer’s role in developing procedures naturally extends from the need to identify how the system ideally should operate and how it can deviate from the original design intent. Whether the deviation is the result of differences between the as-designed and as-implemented system or the result of an unintended system state, the designer is responsible for identifying when procedures are necessary and what they should address. Failure by the designer to completely and accurately describe the process increases the likelihood of workers being unable to find the necessary instructions at the right time. Similarly, designers’ detailed understanding of the possible system

Fig. 1. The management of procedures (adapted from Hale et al. (2003) and Hale and Borys (2013b)).

386

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

failure modes is essential to the procedure author for deciding which instructions to provide to workers. Designers also need to be responsive to information from the workers involved with the implementation of a design as they may unsuspectingly induce scenarios whereby workers are forced to work inefficiently or unsafely, which can lead to violations of the designer’s intent in order for workers to more effectively achieve the goals of the process as they interpret them or to protect themselves from perceived hazards. Designers must also be responsive to changes in processes, and particularly to whether the impacts of those changes are localized to that system or are more global in their effect. When designing an alteration to a procedure in response to such process changes, the lack of consideration of the effects of changes in the primary process on loosely coupled dependent processes can result in procedures that mask the recognition of possible interactions with other dependent processes, which can result in adverse consequences. 3.2.2. Author responsibility The role of the procedure author is to capture the detailed instructions necessary to perform the appropriate tasks, and to structure those instructions so that they can be found and followed under the circumstances in which workers will be operating. The author must also assure that the procedure is appropriately structured for any training that might be necessary for performing the specified tasks, with the caveat that complex tasks may require extensive training that would preclude the need for making explicit any extensive details within the procedure. The procedure author needs to be particularly sensitive to the provision of insufficient detail, which provides the basis for ‘‘cognitive underspecification” (Reason, 1990; Sharit, 1998; Hollnagel, 2004, 2009). This factor could promote improvisation by workers, potentially through their substitution of the well-designed but poorly-communicated rule with one containing sufficient detail to appear appropriate without satisfying the intent of the rule. Conversely, the inclusion of excessive detail increases the likelihood that important details will not be located or followed correctly, or that entire sections of a procedure will be ignored altogether. A critical task for authors of procedures is to ensure that the procedure has a clearly defined scope; the inclusion of instructions that are very unlikely to be required or even impossible to execute can result in procedures being delayed in their execution or improvised. An unclear scope can also contribute to the bloating that occurs in procedures when incidents occur. The shortsighted fix is to add additional rules to address the proximate causes of the incident associated with a procedure, which can diminish the procedure’s effectiveness to communicate instructions and increase its propensity to be violated (Reason, 1997). As with designers, authors need to be receptive to feedback from workers who may find the instructions to be confusing due to factors such as the use of incongruent instructions, in which the order of the instructions does not correspond with the order in which they are to be carried out, or the use of excessive negation, which imposes added cognitive load on the worker’s information processing capacity (Wickens et al., 2004). As the writers of procedures, authors also need to be aware of the level of training on the procedures that workers possess. Otherwise, skilled workers may receive detailed instructions for tasks they need no instructions to perform, which may encourage them to ignore the procedure and thus prevent them from noticing details that differ significantly from their normal modes of performing the task. Conversely, unskilled workers may be assigned to tasks that require additional instructional details, which can lead to improvisation or ignoring procedural elements.

3.2.3. Management responsibility As indicated in Fig. 1, the roles of designers and authors, as well as the expectations of workers, are concerns that intersect and are within the boundaries of management responsibility. Management is also the entity most responsible for shaping the cultural work environment within which procedures are conveyed to and executed by workers (Glendon et al., 2006; Nahrgang et al., 2011), and thus may contribute to issues that workers have with procedures that may arise from a host of primarily psychosocial factors (e.g., autonomy and a supportive environment). Though these issues may be peripheral to the design of the procedures themselves, they could potentially influence their implementation by shaping work environments into ones that are more conducive toward positive engagement. More specifically, it is management’s responsibility to clearly identify and coordinate among designers, authors, and workers what the goals and expectations are with regard to work procedures, as part of its commitment toward ensuring that workers receive appropriate training on the interpretation and application of procedures (Fig. 1). In this regard, management is tasked with ensuring default interpretations in procedures that mix inclusive controls (activities that are permitted) with exclusive controls (activities that are prohibited) in order to reduce or eliminate uncertainty regarding whether behaviors that have not been addressed in the procedure are allowed or prohibited. In addition, management is expected to clarify to its workers the functions of the procedures. Although procedures are generally genuine sources of critical information for workers, they can also serve pro-forma attempts to meet documentation requirements imposed by regulators, certifiers, or customers (Grote and Weichbrodt, 2013). The belief that management views procedures as a means for satisfying auditors could then prompt workers into believing that following procedures is unnecessary for satisfying their managers. Finally, management must govern the process of updating changes to procedures. Specifically, a balance must be struck between two extremes: (1) sluggishness in addressing changes in tools, methods, and knowledge associated with operations, which risks making the procedure an ineffective method for delivering information to the worker and thus increases the likelihood for improvisation; and (2) updating procedures on an arbitrary schedule or incorporating any change, no matter how minor, which could force the worker to repeatedly reorient to the new material and thus increase the likelihood of confusion and error. 4. Input, process, and behavioral variability To this point, the consideration of possible risks inherent to procedures was directed at issues related to the development and maintenance aspects of procedures. This section focuses on the influence of system performance variability, which comprises input variability, process variability, and behavioral (human performance) variability. Input variability is the result of variation in the quality of raw materials supplied, changing environmental factors, or new or altered system requirements. These factors can be stabilized in a variety of different ways, such as implementing a procurement quality process or isolating operations from the environment. Process variability is the sum total of variability caused by the limitations of the components of the process; it is inherent to the system and independent of the inputs to the process. Examples of process variability include measurement error, machining tolerances, or human reaction times, which are usually managed through process redesign.

387

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

Agree with procedure

Worker’s Intent Positive Unintentional Compliance violation

No procedure exists

Improvisation

Time-out

Disagree with procedure

Intentional violation

Time-out

Negative Sabotage Sabotage

Malicious compliance

Sabotage

Fig. 2. Behavioral outcomes deriving from decision intent and procedure content.

Although both input and process variability can impact an organization’s total risk, these risks do not result from the procedures in use by the organization. In contrast, the behavioral variability component of system performance variability can stem from issues associated with procedures, and is thus considered in more detail below. 4.1. Behavioral variability and procedure risk The components of human performance variability attributed to interacting with procedures are summarized in Fig. 2, in terms of the confluence of the person’s underlying decision intent with regard to the procedures and the perception of the actions identified in the procedure. In an ideal scenario, not only do workers agree that the existing procedure represents the most effective way to accomplish the goals of the organization, but additionally the intentions of the workers are commensurate with the organization’s goals, resulting in compliance with the procedure. It should be noted, however, that even in compliance there are various negative risks (in terms of mission, safety, and financial cost objectives) to the organization that could occur when, for example, the procedure is inadequate but the worker lacks the experience and knowledge to recognize the deficiency and thus the ability to feed this information back to management. Unintentional violations represent situations where workers believe a good rule exists to address the conditions but the rule is incorrectly recalled, leading them to act differently than the rule would have them do. Procedures that are too complex to readily consult or thoroughly understand within the operational context are factors that can contribute to such behaviors. Situations where the intent of the worker is negative are generally classified as sabotage. Actions intended to result in a negative outcome are independent of the correct system operating modes; thus, no distinction is made regarding the saboteurs’ knowledge of the content of the procedures (Fig. 2). Although processes and controls can be implemented to prevent or mitigate sabotage, this action itself is not the result of a procedure deficiency. Improvisation in Fig. 2 concerns workers acting in accordance with the perceived intent of the process. However, either no instructions exist in the form of procedures for identifying appropriate actions or decisions, or the worker is aware that such instructions exist but cannot find them. Complex processes can benefit from improvisation if workers have a deep understanding of the process and the customer needs. As a result, they may be able to formulate a solution tailored to the situation that may be more optimal than a prescribed procedure (Dekker, 2005), although process variability can undermine the actions of such skilled workers as well (Hollnagel, 2004). Improvisation by unskilled workers, however, is more likely to lead to greater, and an increased variety of system-related risks. Workers may be reluctant to choose to improvise as their judgment can be questioned afterward by management and their peers, particularly if

the decisions result in negative outcomes. If the consequences for a worker are severe when a negative outcome results, future decisions will be more likely to result in time-outs, where the worker stops work on the task and contacts management for advice, or in malicious compliance (discussed below) if the worker wants to avoid responsibility. A time-out can also result if the worker considers the procedure that is in place to be inconsistent with the process goals, in which case the worker may opt to stop work on the task and contact the designer or author to properly approve alternate actions. Whether a procedure does not exist or the worker disagrees with the existing procedure, by waiting for the possibility of advice from management or a change in the documented procedure the worker ensures that the task is done in accordance with the goal tradeoffs made by the organization rather than the locally important goal. The risks that occur as a result of a time-out are thus to the mission and financial cost (delays could cause schedules to be missed or costs to increase), but quality or safety would generally not be compromised. If a worker disagrees with an existing procedure, the worker may be reluctant to choose the time-out option if there are official or even informal consequences due to causing a delay. This situation may provoke an intentional future procedural violation if the worker feels comfortable with breaking the rules in order to, for example, keep operations on schedule. However, when a worker disagrees with an existing procedure and, in addition, does not accurately understand the details and goals of the process, then the risks of intentional violations become potentially consequential. This situation was exemplified in the nuclear accident in Tokaimura, Japan in 1999 (IAEA, 1999) when workers involved with the accident altered the process and the system configuration to make operations more time and cost efficient. Unfortunately, the system’s safety was compromised when a critical mass of radioactive material was allowed to accumulate—a threat that the workers did not have the expertise to recognize. Malicious compliance comprises situations where the worker believes that a negative outcome will occur if the procedure is followed but still complies without attempting to mitigate the effects. This behavior is what allows work-to-rule slowdowns to be so effective—a negative outcome develops but there is no personal consequence because the worker can invoke the excuse that the procedural instructions were being followed and a negative outcome was unexpected. Such behavior is more likely to be evident in organizations that lack resilience by virtue of not fostering feedback from workers regarding operations, thereby compromising the ability for the organization to learn. Unlike well-intended compliance with a deficient procedure, there is little safety risk to the maliciously compliant worker because the worker would have the knowledge to recognize a personal threat. Cost goals, however, are likely to be impacted most seriously because of the inefficiency caused by performing in ways the worker knows could be improved or due to delayed delivery of outputs.

388

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

5. Assessing procedure risk Integral to the use of written work procedures by organizations is that these procedures will prevent or reduce the severity of any potential losses that might stem from their work operations, and certainly not increase the risk faced by workers or the organization. However, as emphasized in the preceding analysis of risks associated with procedures, deficient design and poorly implemented procedures can lead to undesired behaviors by workers, increasing risks to themselves or to the goals of their organizations. The logical question then is, how can organizations somehow gauge the risk that may be inherent to their work procedures? 5.1. Limitations of applying conventional risk analysis to procedure risk Risk analysis and risk management, which were defined earlier based on ISO 31000, are areas with rich histories that cover enormous ground. Nonetheless, and not surprisingly given the various perspectives to risk that can be adopted (Aven, 2010), a universal definition of risk still remains an elusive goal (Haimes, 2009). Fundamentally, risk addresses the likelihood of some type of loss. This implies that risk assessment should capture the impact of various consequences deriving from hazardous situations and the corresponding likelihood of the occurrence of those consequences (Kumamoto and Henley, 1996). Thus, it would seem to be possible as well, at least in principle, to be able to apply a conventional approach to risk analysis for assessing the risk associated with procedures—what has been referred to as ‘‘procedure risk.” In a landmark paper that has served as the foundation for many approaches to risk assessment, Kaplan and Garrick (1981) proposed that risk could be described as the answers to three questions: ‘‘What can go wrong, what are the consequences, and what is the likelihood.” In the context of procedure risk, this risk triplet implies that the total risk can be evaluated as the sum of the products of each potential consequence by the likelihood of that consequence occurring (Fig. 3), under the assumption that the consequences can be transformed to a common metric (e.g., monetary cost). In a later refinement to this conceptualization of risk (Kaplan et al., 2001) it was noted that although the set of possible scenarios is generally non-enumerable, in practice this set can be partitioned into a finite number of disjoint scenarios for which each hazard/consequence pair has an associated likelihood. Still, even with these refinements, problems in assessing risk remain. One problem relates to the concept of likelihood, which can be considered in different ways: for example, does it refer to the likelihood of the consequence itself, the likelihood of the event causing the loss (Haimes, 2009), or the likelihood that a particular probability distribution will describe the distribution of the consequences (Kaplan, 1993)? This ambiguity has even led to the

Fig. 3. Conventional risk assessment based on the framework of Kaplan and Garrick (1981).

opinion that a fourth question be amended to the risk triplet (Haimes, 2009): over what timeframe? Much more problematic, however, with regard to the assessment of procedure risk is the practicality of identifying all of the hazardous scenarios and their associated consequences, let alone the likelihoods corresponding to these consequences (which, even if theoretically feasible, may require long observation periods or costly data collection procedures to obtain). Organizations may have numerous procedures in place designed to cover various potentially hazardous operations, and which are often subjected to ongoing modifications, making such an intensive process unrealistic. Instead, what would be useful for organizations is a tool that can, in a relatively efficient way, provide an indicator or estimate of the risk inherent to their procedures. This could then be used to support risk management efforts directed at identifying procedures that are potential threats to the organization’s goals, and ultimately to guide needed solutions.

5.2. A proposed modeling framework for assessing procedure risk The proposed framework for assessing procedure risk seeks to develop a less resource intensive and more intuitive approach to quantifying such risks that relies on critical attributes of a procedure—most notably, on the role of controls in procedure risk that was elaborated on earlier. Although this approach was motivated by the analysis techniques used in the wake of the Columbia accident investigation to filter through the numerous procedures that governed NASA’s space shuttle ground processing operations, it is considered to be sufficiently robust to be applicable across many work domains in which procedures are applied to modify risks associated with hazardous work operations. Fundamental to this approach is the underlying assumption that each action or requirement placed in a procedure by the process designer or the procedure author represents a control, in the sense that it is intended to modify the effect of uncertainty on system objectives, and that these controls can, as was noted earlier in the formal definition of the term control provided by ISO (2009) in the context of risk management, sometimes fail to achieve the predicted effect. Unplanned behavior resulting from procedures that do not achieve the intended effect cannot be expected to be an improvement in risk exposure. Even in cases where uncertainty is not increased, the anticipated benefit is not achieved and the control fosters new undesired outcomes by increasing the cost or duration of the task, and by creating additional interactions that

Fig. 4. The set of modified questions underlying procedure risk.

389

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

can lead to increased opportunities for human error (Reason, 1997). Whereas Fig. 3 emphasizes the total of all risks (based on Kaplan and Garrick’s three questions), the proposed approach to procedure risk assessment is concerned only with those risks that have had procedural controls enacted to reduce, mitigate, or prevent the hazard. Fig. 4 conceptualizes procedure risk in terms of a modified set of questions that address how effectively the controls respond to the hazard. How well a control modifies the risk is based on two separate elements: that the control provides a valuable function (the third question in Fig. 4), and that it is successful at accomplishing its function (the fourth question in Fig. 4). Neither a control that perfectly performs a trivial role (low control value) nor a control that fails to fulfill its intended role (low likelihood of achieving the intended effect) can be effective at modifying risks. Thus, in this view of procedure risk, the ‘‘consequence” component of risk is replaced with the ‘‘control value,” while likelihood now addresses how likely it is for the control to fail rather than how likely it would be for a particular loss to occur. The rationale for using control value (CV) in place of consequence is that procedural controls intended to prevent the greatest consequences would be performing the most critical functions and therefore be the most valuable controls. Using the likelihood of failure of a control (CFL) in place of the more conventional likelihood of a loss event derives from the necessity, when procedure risk is being considered, for providing information on whether the procedures in place to address a hazard are successful. As in many other risk assessments, screening criteria can be used to deselect some hazards from the analysis, making the process responsive to the time available for personnel to perform the assessment tasks as well as for the organization to determine and implement corrective actions. In principle, an organization might have available failure rate data associated with procedural controls. However, as a proactive method of establishing resiliency within an organization (Wreathall, 2006), it would be much more efficient, practical, and even feasible to employ subjective methods (Seaver and Stillwell, 1983) for assessing the value of a control, as well as the likelihood that the control achieves its objective, using subject matter experts (SMEs). Although the use of expert judgment may appear to diminish the method’s utility by potentially introducing variability into the assessment process, the use of data derived from SMEs remains critical to many types of risk-related assessments. For example, the use of expert judgment underlies

almost all methods of human reliability analysis (Kirwan, 1994; Sharit, 2012), whether it be to provide lower or upper bound estimates of human failure probabilities, identify work context factors that could influence operator performance in given scenarios, or generate importance weights and quality ratings associated with such factors. 5.2.1. Control value (CV) assessment In assessing the risks of procedural controls, a CV score would represent how effective an individual control or a specific group of procedural controls would be at reducing the consequences of the associated hazard. A suggested approach to CV score assessment is to consider it as comprised of the following two attribute scores: the opportunity for intervention and inevitability of consequence. These two attributes are derived from the two dimensions of a system that are fundamental to Perrow’s (1999) system theory of accidents, notably coupling and interactive complexity. Uncertainty scenarios that are less tightly coupled provide a greater opportunity for workers to intervene. These potentially hazardous scenarios would thus tend to have lower CV scores because the intervention can negate the consequences arising from the initiating events. Similarly, scenarios where the critical consequences won’t be realized unless a string of other contributing events occur—that is, hazardous scenarios with lower degrees of inevitability of consequence—would also have lower CV scores. Less opportunity to intervene and greater inevitability of adverse consequences in response to hazardous scenarios would, in contrast, make the procedural controls in place less trivial and consequently higher in value. 5.2.2. Control failure likelihood (CFL) assessment The proposed approach to assessing procedure risk focuses on the likelihood that the procedural control will fail, rather than on the likelihood of occurrence of each consequence associated with ‘‘what could go wrong” as in the more conventional approach to risk assessment. This is more than just a subtle distinction as a failed procedural control does not necessarily result in a negative consequence. It should be noted, however, that situations where workers perform the correct action, despite the existence of a failed control within the procedure, are undesired. In the absence of a successful control, a correct action could not be reliably expected to occur again due to various changing conditions in future executions of the process.

Table 2 Process failures corresponding to the four CFL attributes. Work process attributes

Procedural control failure modes Worker is unaware of the actions to be performed

Define

– Multiple actions are possible to accomplish task – Actions necessary for task completion are not apparent – Responsibility for task performance has not been specifically given to the expected worker

Assign

Train

Organize

a b

b

(see footnote)

– Appropriate cues and instructions are unavailable so the worker is unable to recall the necessary actions

Worker is unable to perform expected actions a

Worker chooses to perform a different action than expected

(see footnote)

– Worker lacks the physical or mental capacity to perform the task – Worker lacks the experience or knowledge necessary for task performance – Necessary tools or resources are not provided – Action sequence is confusing – Difficulty tracking progress allows for distractions to impact task completion

A failure to define the task for the worker does not necessarily affect the worker’s ability to perform the actions. A failure in training does not necessarily result in the worker being unaware of the actions to be performed.

– A better means of meeting the intent of the task has been identified by the worker – A more appropriate worker for the task may exist, so the assigned worker avoids the task until it is reassigned – Worker confidence in the necessary skills is low and an alternate technique can meet the known goals – Worker chooses an easier way to perform the task

390

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

In the proposed modeling framework, the assessment of control failure likelihood (CFL) appeals to human factors and ergonomics considerations related to the design of the work procedure. Specifically, within the context of the procedure that is to be utilized, the subject matter expert (SME) should consider the following four attributes: how well the process that the procedure addresses is defined, to whom the control or execution of the process is being assigned to, the nature of the training that has been provided, and how the process is organized in presenting work content. In addition, a fifth attribute, what monitoring is performed (not to be confused with process quality inspection activities) in order to ensure that the organization’s expectations are met, should also be considered. Monitoring is not, however, included as a model component as it is typically not an explicit part of the procedural control but rather a feature of the work cultural environment. The inclusion of monitoring is based on the assumption that in the absence of some form of check, the process will continue to remain vulnerable to changes in the inputs, the environment, or interpretations of the wording of the documented rules. The first four attributes can serve as a substitute for failure likelihood analogous to the role that the CV score serves as a substitute for the consequence magnitude in traditional risk assessment. As indicated in Table 2, these process attributes can be used to identify a number of ways in which procedural controls can fail, although not all attributes are relevant for each failure mode.

Assignment is unclear because hazardous operations require the involvement of at least two technicians, but without clear responsibility, it is possible that the first would connect the lines and leave them for the other to torque, while the second may reasonably assume that the work was completed by the first. While training is provided in critical skills, like torqueing of threaded connectors and fasteners, the procedure fails to capture that this training is applicable. Finally, the organization is poor not only because determining torque requires referring to two separate documents—the installation drawing and the referenced specification—but also because the instructions are written in passive voice. Option 2 is more capable of obtaining the necessary behavior for the opposite reasons. The task is clearly defined as installation of two hoses with unambiguous from/to instructions. The assignment is made to the propellants technician #1 through the use of the PROP1 radio call sign at the start of the step. Training of the personnel involved is ensured by the callout for the torque certification number above where the technician will stamp the completion of the work. Task organization is also improved by the tabulation because the desired torque is clearly presented to the technician and the two installations can be performed independently, in the event of an interruption, such as a stand-down due to inclement weather. 5.3. Instantiation of the modeling framework

5.2.3. Case study: nitrogen tetroxide tanking operations An example follows that is intended to illustrate the extremes of the CFL attributes and to demonstrate that CV is often dominated by the environment in which the procedural controls are implemented. For this case, the scenario is the refilling of a nitrogen tetroxide storage tank from a tanker truck, which involves the connection of a pair of flex-lines to allow vapor-rich air to flow into the tanker truck as a closed system while liquid flows into the storage tank. If the hoses are plumbed incorrectly, the air will not flow into the tanker truck and the pressure will rise until relieved into the atmosphere through the vapor scrubber, incurring unnecessary expense of activating the scrubber. If the lines are not torqued properly, they can loosen during the transfer, allowing vapor to be vented directly to the atmosphere, creating a hazardous situation. These two consequences will be referred to as the ‘‘scrubber” consequence and the ‘‘release” consequence. The value of a procedural control in this scenario would be based on the effects it is intended to prevent if the control fails. With respect to use of the scrubber (the scrubber consequence), the inevitability is high if the hoses are installed incorrectly and the intervention is moderate because a second technician participates and may detect the error before the start of oxidizer flow. For leaking flex line joints (the release consequence), the inevitability is low because a leak will not always result from a snug but under-torqued connection, but the intervention is extremely weak because the low torque is not easily detectable. The CV score would account for both of these consequences, resulting in an overall CV score as will be illustrated in Section 5.3.1. For this case study, two procedural control options are considered for mitigating these consequences, denoted in Fig. 5 as step 5.0 of the procedure. Each of these control options relate to the same task activities and thus will have the same CV score; the options, however, would be presented differently to the worker. In option 1, the factors governing the CFL, namely the definition, assignment, training, and organization attributes, are weakly executed. The task is poorly defined because the desired condition cannot be definitively established from the text—it is unclear if the first hose is intended to connect port K01 to K02 or K01 to K98.

Ultimately, the goal is to derive some type of index of risk associated with an organization’s procedures. A relatively straightforward approach for deriving such an index would first require that an important work process hazard be identified for which a procedural control is in place. The next step would be to generate questions to be evaluated by an organization’s SME(s) using a scoring rubric to establish consistent criteria across reviewers. The scoring would be performed using identical rating scales for addressing how effectively the procedural control would reduce the consequences of the associated hazard. In the case of mapping the two constructs underlying the CV score toward this end, one such question would relate to the opportunity for intervention and the other question would be directed at the inevitability of consequence. Since a decreasing opportunity for intervention is associated with a higher control value, to appropriately capture the inverse relationship the opportunity for intervention would be subtracted from the inevitability of consequence. The resulting difference of these ratings could then serve as the CV score. Higher scoring controls would represent valuable controls, where the effort associated with carrying out the control directly corresponds to threats to the organization. A low CV is associated with wasteful controls, where the resources are expended to implement the control without benefit to the organization. A categorical example of wasteful controls is the generation of procedures to avoid liability, which exist primarily to be referred to in a legal suit (Pélegrin, 2013). Similarly, one or more questions (using identical rating scales) could be constructed with associated criteria corresponding to each of the four process attributes underlying the CFL. The sum of the ratings associated with the CFL attributes could then serve as an estimate of the CFL score, where high scores represent strong controls and controls receiving lower CFL scores being the weaker controls. This relatively simplistic method is not expected to capture all the nuances of a specific type of rule in a particular industry, but serves as a first approximation that can be improved through enhancement of the scoring rubric, the use of weighting factors adjusted in response to empirical data, and more generally, through further research. Similar considerations would apply to

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

391

Fig. 5. Two presentation options for the procedural control (T refers to a check-off by the technician that the procedure was performed; 2ea is a call-out for ‘‘two each”).

Fig. 6. Example scores for the case study.

the attributes of the CV score; for example, if warranted, the opportunity for intervention and inevitability of consequence attributes could be weighted differently. 5.3.1. Deriving CV and CFL scores for the case study If the characteristic components of the CV and the CFL are scored on a 1–10 scale (from least to greatest magnitude), Fig. 6 demonstrates how they could be scored and combined to obtain an overall CV score for the scenario and an overall CFL score for each of the two procedural control options being considered. The consequence CV score shows the value of the control in modifying the risk associated with that consequence scenario, which is obtained by subtracting the intervention score from the inevitability score. The overall CV score is the mean value of the two scenarios—this example arbitrarily uses equal weighting, which the organization may adjust by balancing the direct costs of operating the scrubber against the cleanup and indirect costs of venting nitrogen tetroxide into the atmosphere. The CFL scores for each option are calculated by summing the four component scores. 5.3.2. The value of the model An organization making use of this model could use the resulting information both in developing new procedures and in improving existing procedures. In either case, the application of this modeling approach is straightforward in the simple implementation, though additional complexity can be applied as experience is acquired. The application of the model to new procedure development makes use of the scoring to determine if prospective controls perform a useful function or if they have little value. The scoring can similarly highlight which aspects of candidate controls are weak, and can then exploit that information to suggest improvements. A control with a weak definition could suggest that the specific task was not fully considered or that a systematic gap exists in

Fig. 7. Priority improvement quadrants.

how the roles of workers are perceived. Low organization scores could suggest that a physical or informational tool would be useful for reducing the ergonomic stress or cognitive workload on the assigned worker. As alternatives are explored, the model can be used to compare options, first informally, and eventually more objectively for organizations where weights are applied to the scoring. The method being suggested for improving an organization’s existing procedures begins by computing the CV and CFL scores for all the controls in a procedure suspected of being flawed and then prioritizing the controls for improvement. Fig. 7 provides a means for roughly categorizing the controls by their CV and CFL. Controls in Quadrant I are strong controls in circumstances where it is valuable to have a control. There may be benefits in strengthening these controls by making changes that would increase the CFL, but they are not necessarily cost effective. Quadrant II controls

392

G. Praino, J. Sharit / Safety Science 82 (2016) 382–392

are situations where controls are valuable if successful, but the CFL scores indicate that they would not necessarily succeed. These controls are the ones of highest priority for improvement, which can be accomplished through their strengthening. Quadrant III is the opposite situation, where the high CFL signifies that the controls are likely to perform their role, but the low CV implies that a successful control provides little benefit. These controls are strong candidates for being removed from a procedure because labor spent on these efforts is likely to not be worth the small impact on the organization’s overall risk posture. The low CV in Quadrant IV indicates the similarly low impact of a successful control, so it should be considered for removal, but the weakness indicated by the low CFL suggests that significant resources are not spent on the control, so the correspondingly low cost makes it a lower priority for removal than a control in Quadrant III. A numerical aggregation of multiple controls within a procedure is possible, in much the same way that traditional risk assessments multiply consequence and likelihood to obtain an expected value. However, a single number representation is no more useful for this model than expected value is for a conventional risk assessment because of the multiple types of scenarios that can result in identical scores with significantly different practical implications (Department of Defense, 2014). Especially troublesome is the zero value in the middle of the CV scale, which effectively precludes multiplicative combinations of CV and CFL scores. Should an organization choose to offset and scale the scores to avoid a singularity it would be possible to multiply CV by the CFL, but such a number would not differentiate between controls in Quadrant II and Quadrant III; thus an individual assessment would still be needed. Similarly, dividing CFL by CV to obtain a notional ratio of strength to benefit wouldn’t differentiate between controls falling in Quadrants I and IV. 6. Conclusions The primary goals of this paper were to provide the basis for identifying and understanding the risks inherent to written work procedures, particularly in organizations that rely on such procedures for managing safety, financial cost, and mission risks, and a modeling framework for assessing the risk inherent to such procedures that relies on a perspective considered to be more suitable and efficient than conventional perspectives to risk assessment. Through a case study, some guidance was given on translating the modeling framework into a method that could be used by organizations to capture the relative risks associated with the procedural controls that they have in place for limiting or preventing hazards from propagating into adverse consequences. In no way, however, is it implied that there is only one way to instantiate this modeling framework or, for that matter, that this is the only valid framework for assessing procedure risk. In fact, given the lack of work in this critical area, the expectation is that the ideas in this paper will direct more attention to this important organizational safety issue. Acknowledgment The authors would like to thank the many experts associated with NASA’s space shuttle program who helped contribute to some of the ideas in this paper. References Alper, S., Karsh, B.-T., 2009. A systematic review of safety violations in industry. Accid. Anal. Prev. 41, 739–754. Aven, T., 2010. On how to define, understand and describe risk. Reliab. Eng. Syst. Saf. 95, 623–631.

Center for Chemical Process Safety, 1992. Guidelines for Hazard Evaluation Procedures, second ed. American Institute of Chemical Engineers, New York. Columbia Accident Investigation Board, 2003. Report Volume I, CAIB Final Report. National Aeronautics and Space Administration, Washington, DC. Dekker, S.W.A., 2005. Ten Questions about Human Error: A New View of Human Factors and System Safety. Lawrence Erlbaum Associates, Mahwah, NJ. Department of Defense, 2014. Risk Management Guide for Defense Acquisition Programs. . Glendon, A.I., Clarke, S.G., McKenna, E.F., 2006. Human Safety and Risk Management, second ed. Taylor & Francis, Boca Raton, FL. Grote, G., Weichbrodt, J., Gunter, H., Zala-Mezö, E., Künzle, B., 2009. Coordination in high-risk organizations: the need for flexible routines. Cogn. Technol. Work 11, 17–27. Grote, G., Weichbrodt, J., 2013. Why regulators should stay away from safety culture and stick to rules instead. In: Bieder, C., Bourrier, M. (Eds.), Trapping Safety into Rules. Ashgate, Aldershot, England, pp. 225–240. Haimes, Y.Y., 2009. On the complex definition of risk: a systems-based approach. Risk Anal. 29, 1647–1654. Hale, A.R., Heijer, T., Koornneef, F., 2003. Management of safety rules: the case of railways. Syst. Saf. Monit. 1, 1–11. Article III-2. Hale, A.R., Swuste, P., 1998. Safety rules: procedural freedom or action constraint? Saf. Sci. 29, 163–177. Hale, A., Borys, D., 2013a. Working to rule, or working safely. In: Bieder, C., Bourrier, M. (Eds.), Trapping Safety into Rules. Ashgate, Aldershot, England, pp. 43–68. Hale, A., Borys, D., 2013b. Working to rule or working safely? Part 2: the management of safety rules and procedures. Saf. Sci. 55, 232–233. Hollnagel, E., 2004. Barriers and Accident Prevention. Ashgate, Aldershot, England. Hollnagel, E., 2009. The ETTO Principle: Efficiency-Thoroughness Trade-Off. Ashgate, Aldershot, England. IAEA, 1999. Report on the Preliminary Fact Finding Mission following the Accident at the Nuclear Fuel Processing Facility in Tokaimura, Japan, August, 1999. International Atomic Energy Agency, Vienna, Austria. ISO 31000, 2009. ISO 31000: Risk Management: Principles and Guidelines = Management Du Risque: Principles Et Lignes Directrices. ISO, Geneva. Johnson, P., Gill, J., 1993. Management Control and Organisational Behavior. Paul Chapman Publishing Ltd, London. Kaplan, S., 1993. Formalisms for handling phenomenological uncertainties: the concepts of probability, frequency, variability and probability of frequency. Nucl. Technol. 102, 137–142. Kaplan, S., Garrick, B.J., 1981. On the quantitative definition of risk. Risk Anal. 1, 11– 27. Kaplan, S., Haimes, Y.Y., Garrick, B.J., 2001. Fitting hierarchical holographic modeling into the theory of scenario structuring and a resulting refinement to the quantitative definition of risk. Risk Anal. 21, 807–819. Kirwan, B., 1994. A Guide to Practical Human Reliability Assessment. Taylor & Francis, London. Kumamoto, H., Henley, E.J., 1996. Probabilistic Risk Assessment and Management for Engineers and Scientists, second ed. IEEE Press, Piscataway, New Jersey. Nahrgang, J.D., Morgeson, F.P., Hofmann, D.A., 2011. Safety at work: a meta-analytic investigation of the link between job demands, job resources, burnout, engagement, and safety outcomes. J. Appl. Psychol. 96, 71–94. NTSB, 1997. Aircraft Accident Report: In-flight Fire and Impact with Terrain, Valujet Airlines, Flight 592, DC-9-32, N904VJ, Everglades, near Miami, Florida, May 11, 1996. National Transportation Safety Board, Washington, DC. Pélegrin, C., 2013. The never-ending story of proceduralization in aviation. In: Bieder, C., Bourrier, M. (Eds.), Trapping Safety into Rules. Ashgate, Aldershot, England, pp. 13–26. Perrow, C., 1999. Normal Accidents: Living with High-Risk Technologies. Princeton University Press, Princeton, New Jersey. Reason, J., 1990. Human Error. Cambridge University Press, New York. Reason, J., 1997. Managing the Risks of Organizational Accidents. Ashgate, Aldershot, England. Schulman, P., 2013. Procedural paradoxes and the management of safety. In: Bieder, C., Bourrier, M. (Eds.), Trapping Safety into Rules. Ashgate, Aldershot, England, pp. 244–255. Seaver, D.A., Stillwell, W.G., 1983. Procedures for using expert judgment to estimate human error probabilities in nuclear power plant operations. NUREG/CR-2743. USNRC, Washington, DC. Sharit, J., 1998. Applying human and system reliability analysis to the design and analysis of written procedures in high-risk industries. Hum. Fact. Ergon. Manuf. 8, 265–281. Sharit, J., 2012. Human error and human reliability analysis. In: Salvendy, G. (Ed.), Handbook of Human Factors and Ergonomics, fourth ed. J. Wiley, New York, pp. 734–800. Weick, K.E., Sutcliffe, K.M., 2007. Managing the Unexpected: Resilient Performance in an Age of Uncertainty. Jossey-Bass, San Francisco, CA. Woods, D.D., 2006. Essential characteristics of resilience. In: Hollnagel, E., Woods, D. D., Leveson, N. (Eds.), Resilience Engineering: Concepts and Precepts. Ashgate, Aldershot, England, pp. 21–34. Wreathall, J., 2006. Properties of resilient organizations: an initial view. In: Hollnagel, E., Woods, D.D., Leveson, N. (Eds.), Resilience Engineering: Concepts and Precepts. Ashgate, Aldershot, England, pp. 275–285. Wickens, C.D., Gordon Becker, S.E., Liu, Y., Lee, J.D., 2004. An Introduction to Human Factors Engineering, second ed. Prentice-Hall, New York.