- Email: [email protected]
The Threat-Strategy Interview
Applied Ergonomics 47 (2015) 336e344
Contents lists available at ScienceDirect
Applied Ergonomics journal homepage: www.elsevier.com/locate/apergo
The Threat-Strategy Interview Francis T. Durso*, Sadaf Kazi, Ashley N. Ferguson School of Psychology, Georgia Institute of Technology, 654 Cherry Street, Atlanta, GA 30332, USA
a r t i c l e i n f o
a b s t r a c t
Article history: Available online 23 September 2014
Operators in dynamic work environments use strategies to manage threats in order to achieve task goals. We introduce a structured interview method, the Threat-Strategy Interview (TSI), and an accompanying qualitative analysis to induce operator-level threats, strategies, and the cues that give rise to them. The TSI can be used to elicit knowledge from operators who are on the front line of managing threats to provide an understanding of strategic thinking, which in turn can be applied toward a variety of problems. © 2014 Elsevier Ltd and The Ergonomics Society. All rights reserved.
Keywords: Strategies Knowledge elicitation Threat and error management
1. Introduction Safety-critical dynamic environments such as aviation or healthcare experience numerous challenges to system safety. In one view, Threat and Error Management (TEM; Helmreich et al., 1999; Merritt and Klinect, 2006), these challenges have been referred to as threats. Threats refer to characteristics of operational environments which impede safe completion of tasks; this may include factors such as non-optimal staffing policies, poor facility design, strictly enforced role hierarchies, or poor communication within teams. Errors are likely to occur in the presence of threats such as fatigue and cognitive overload. Similar to TEM's use of the term “threat” to denote any condition which impedes safe task completion, healthcare research uses terms such as “failure modes” (Faye et al., 2010), “adverse events” (Kohn et al., 2000), “performance obstacles” (Gurses et al., 2009), “operational failures” (Tucker, 2004), “problems” (Holden et al., 2012), or “variances” (Kleiner, 2006) to describe conditions that challenge performance. 1.1. Strategies The TEM model suggests that threats and errors are managed by the operator. This management of threats and their cognates e failures, operational failures, problems, and performance obstacleseoccurs by the operators' selection and implementation of strategies. Strategies are important in maintaining performance and safety in dynamic domains. Strategies can be seen at all levels
* Corresponding author. Tel.: þ1 404 894 6771. E-mail address: [email protected] (F.T. Durso). http://dx.doi.org/10.1016/j.apergo.2014.08.001 0003-6870/© 2014 Elsevier Ltd and The Ergonomics Society. All rights reserved.
of socio-technical systems (Appelbaum, 1997) and socio-natural systems (Durso and Drews, 2010). For example, at a macro-level, (Reason 2000; Amalberti and Hourlier, 2011) speaks of organization level strategies that can be adopted to improve patient safety. At a meso-level, crew resource management strategies affect how teams in a cockpit or in operating rooms coordinate activities. Finally, at an individual, micro-level, operators at work are believed to use strategies to perform “on par” (Helson, 1949) keeping performance high and workload low (Durso and Alexander, 2010). In this article, we focus on strategies at the level of the operator, that is, strategies that an air traffic controller might use to move traffic or that a nurse might use to deliver care to a patient. From its origins in the military, strategies are now recognized and used as an explanatory concept in virtually every modern endeavor. There have also been notable studies in particular domains designed to determine the strategies that are used and why they are selected (e.g., Siegler and Shrager, 1984). Luwel, et al. (2009) remark on the 25 years of research on strategies that demonstrate that individuals show “remarkable variability in their strategies for accomplishing various cognitive tasks.” (p. 753). Luwel et al. (2009) note the use of strategies in a variety of academic domains such as arithmetic, scientific reasoning, spelling, decision-making, time telling, serial recall, and currency conversion. Researchers have also looked at strategy use in operational, work domains (e.g., Loft et al., 2007; Sperandio, 1971). Strategies can be thought of as plans or methods to achieve a goal (Merriam-Webster.com, n.d.). Siegler and Jenkins (1989) add that strategies are nonobligatory. Shifting from first gear to second by moving the gear shift is a procedure, not a strategy, because moving the gear shift is obligatory. Strategies give rise to and are implemented by actions. Actions can be thought of as any behavior
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
carried out by the operator to achieve a goal. Thus, strategies are more abstract than the actions by which they are implemented. Successful strategies take into account constraints from the environment and from the organization. Strategies can be implemented quickly, as when a nurse uses the ABCs, airway-breathingcirculation prioritization strategy, or they can be selected more deliberately as when deciding whether or not to seek help rather than administer treatment to a patient alone. Once operators understand a situation, including the threats to performing their task, they can select a strategy to deal with those threats. Much of our strategic thinking is conscious (see Evans, 2000, for the argument that all strategies are conscious). However, the cues from the situation that give rise to a strategy may be difficult to articulate. With sufficient expertise, such strategy selection can be relatively automatic and relatively inscrutable to not only the researcher, but to the operator as well. Thus, it can be difficult to accumulate a knowledge base of strategies that are being used or gain an appreciation of the underlying processes that guide their use. Although it is sometimes difficult for an operator to verbalize the cues that initiated a strategy or the reason a particular strategy was selected, the strategies and reasons behind their selection can be made accessible (Siegler and Stern, 1998). 1.1.1. Strategies in operational work domains The interest in strategies is also valuable beyond academic domains. Over 40 years ago, Sperandio (1971) argued that strategies were key to understanding how operators control complex industrial environments. He argued that the operative strategy used by the air traffic controllers depended on the operator, the task, and the level of workload. Following Sperandio (1971), Durso and Alexander (2010) have argued that operators attempt to understand a situation, that is, they attempt to have sufficient situation awareness, so as to know when to change strategies in a continuous effort to keep performance high and workload manageable. Their perspective echoes Helson's (1949) Par Hypothesis: Performance that is less than a standard calls for greater effort whereas performance that exceeds that standard allows for less effort. As an illustration, Sperandio (1971) demonstrated that professional air traffic controllers under time pressure reduced the duration of important messages and eliminated secondary messages entirely. They also became more proactive, initiating more messages. Controllers increased the task bandwidth by reducing noise and redundancy. Since Sperandio, others interested in air traffic control (ATC, e.g., Averty et al., 2008; Koros, et al., 2006; Loft et al., 2007) have elicited a corpus of strategies from controllers. Some have argued that strategies are important to understanding fundamentals of how work is conducted. For example, Delaney et al. (1998) showed that the power law of practice fit individual data better when conditionalized based on strategy. Durso and Alexander (2010) point to a number of examples where superficially straightforward relationships such as between workload and performance, situation awareness and workload, or even task load and performance, disassociate. To Durso and Alexander the reason for the dissociations were changes in strategy. That is, rather than see a straightforward increase between task load and workload, operators change strategies as task load increases allowing them to restrict workload to a manageable range. In a review of the workload literature in ATC, Loft et al. (2007) argue that “Controller workload will not be effectively modeled until controllers' strategies for regulating the cognitive impact of task demand have been modeled.” (p. 376). The study of strategies also has credibility in a number of human factors research perspectives. For example, in Cognitive Work Analysis (CWA, e.g., Lintern, 2009; Vicente, 1999), strategies are used to manage changing task constraints. In TEM, team-based
337
strategies are used proactively to manage threats. In cockpit task management, strategies are used to prevent errors, or to manage the consequences of errors (Helmreich et al., 1999). This view is consistent with the development of resilience engineering (Hollnagel et al., 2010) and adaptation to dynamic industrial tasks. 1.2. Methods of strategy elicitation Researchers have tried to understand strategy use in work domains both analytically and empirically. Analytical methods produce a formative characterization. A formative characterization would in principle detail all possible strategies that could be produced in the work domain. This type of analysis characterizes disciplines like linguistics, in which the understanding of the structure of even extinct languages can be understood even though no one currently speaks the language. Similarly, a formative analysis produces strategies even if no one ever uses the strategy, or perhaps could ever use the strategy. Roth (2008) draws the connection to the concept of competence. Several researchers have taken a formative approach to strategies (e.g., Cornelisson et al., 2014; Hassall and Sanderson, 2012; Naikar and Pearce, 2003). Roth (2008) noted that formative analysis is especially valuable early in design and can be used to understand futuristic systems (e.g., Nehme et al., 2006). On the other hand, empirical methods produce a characterization of the strategies that operators actually perform in the work domain. These methods are descriptive, rather than formative. Unlike linguistics, psycholinguistics is, indeed, concerned about the language that is actually used by people. Thus, empirical methods relate to the concept of performance as contrasted with competence. Descriptive methods are essential when researchers want to know about psychology and behavior that actually characterizes the work domain. There are several methods available to elicit knowledge from subject matter experts and less experienced operators. Crandall and Getchell-Reiter (1993), for example, used the critical decision method, to understand how expert nurses assessed the presence of sepsis in neonates. The critical incident method is another knowledge elicitation technique to understand cognitive processes of operators during non-normal situations (Flanagan, 1954; Gurses and Carayon, 2009). The Staged World method uses simulations to observe how the use of artefacts shapes work and how practitioners adapt to “probes” or challenges to work (Woods, 2003). The methodology presented in the current paper is related to cognitive task analysis (CTA) and to a variety of structured interview procedures. However, these methods often focus on the states in which the operators find themselves, and less often on the strategies that operators use to manage these states. For instance, the CDM focuses on identifying specific cues that alert the operator to changes in critical safety parameters; the critical incident technique focuses on identifying whether certain states of the situation pose a threat to system safety. These methods do not focus specifically on how operators effectively manage threats to system safety. One interesting exception was Roth, et al. (1999) study of rail operators; they used a strategies analysis (see also Roth, 2008) that combined observation and interviews. Initial observations suggested strategies to the researchers which were in turn presented to the operators for their reactions. It is difficult to elicit strategies from experts. In fact, Roth (2008) notes that none of the strategies they explored in the rail study were explicitly mentioned by the operators in the interviews, although when proffered the strategies to other railroad workers “they readily confirmed that these were routine practices.” (p. 142). In healthcare, much of these inscrutable strategies are considered “nurse's intuition” (Benner, 1982).
338
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
We attempted to create an interview procedure that focused on strategies and that was easily understood by domain experts. We wanted a procedure that presented the world to the operators in a way concordant with the way the operators naturally segmented their work: Responding with strategies to threats they face seems to be such a natural segmentation. In addition, we wanted a procedure that was effective at encouraging operators to compare and contrast strategies along with the constraints and cues that served as boundary conditions of their strategies. This, we believed, was important to helping operators articulate heretofore implicit strategies. Finally, we wanted to do this in an interview procedure, rather than for example relying on our interpretation of observations to create the initial strategies. Observations are often difficult and in some cases impossible. In addition, we wanted operators to offer their own strategies rather than ones we induced from observations. The purpose of the current article is to describe a structured interview procedure designed to elicit strategies from skilled operators. The interview procedure asks operators first to generate threats to performing a task and then to provide strategies they would use when faced with those threats. We then use the result of this Threat-Strategy Interview (TSI) to map strategies to the cues in the situation that give rise to the strategy selected. In this way we hope to offer a procedure to illuminate the “some cue that something was not right” to which Henneman's nurses referred (Henneman et al., 2010). We agree with Roth (2008) that descriptive, empirical methods “offer an attractive alternative and/or complement to analytic approaches” (p. 132). Observations are time consuming and not always possible. Structured interviews have been successful at eliciting insights from the interviewed operators about the reasons they made particular decisions even when that decision was not readily apparent even to the operator. For example, work by Klein (1989) presented a variant of the critical incident technique (Flanagan, 1954) to elicit knowledge about managing dynamic environments that are characterized by time pressure. Crandall and Getchell-Reiter (1993) used the critical decision method (CDM) to interview neonatal intensive care unit nurses. They found that CDM interviews were able to elicit more information than interviews without CDM probes. 2. The TSI method Our structured interview procedure differs from previous ones in its focus on strategies. Leveraging the TEM perspective (Merritt and Klinect, 2006) we developed a structured interview that begins by eliciting from the operator threats to performing a task. This interview can be used for any dynamic environment to elicit threats and operator's strategies. Generating threats to their job came naturally to the operators. Once in the mindset of a specific threat, operators easily generated strategies to deal with that threat. Operators were then able to generate the cues that weighted the selection of one strategy over the other strategies they had available. Anecdotally, articulating the cues often came with an introspective “aha” that suggested the nurse had brought to consciousness a connection previously tacit and part of her or his operator's intuition (Kahneman and Klein, 2009; Klein, 1998; Thompson, 2014). The TSI was conducted in a structured format in the following four stages. Table 1 summarizes the stages. Stage I: Introduction of the critical task and elicitation of past experiences with the critical task. In this stage, the interviewer describes a critical work-related task and requests the interviewee to reflect upon relevant past experiences. The task should be selected based on discussion with an experienced confederate in the area to ensure the task is a common and important task in this
Table 1 Sketch of the Threat Strategy Interview (TSI). Stage
Goal
Description/example questions
I
Introduction of the Critical Task & Elicitation of interviewee's past experiences with the critical task. Threat elicitation
Describe the critical work-related task and discuss formal and informal experience with the threat as well as a recollection of a recent or memorable experience. Tell me something that might happen to threaten your effective management of the situation. How do you become aware of the threat? {multiple cues-to-threat are elicited} Tell me a strategy you use to keep the threat from interfering with this task. A strategy is a plan or a method to achieve a goal. A strategy is not usually one action, but you may think of it as an action. {multiple strategies are elicited} Thinking of Strategy A, can you tell me when you would choose to try that strategy? In other words, in what context or situation would that strategy work best? What would the situation be like that leads to picking that strategy? What would need to be true for you to pick that strategy? This might be something in the environment, something in the way you’re thinking, or something about the situation as a whole. So, when would you choose to use Strategy A? {multiple cuesto-strategies are elicited} Could you think about a specific piece of information or a cue that would encourage you to choose Strategy A over Strategy B? How can you tell if Strategy A is working? Or how can you tell when it's not working and you should switch? If that strategy didn't work, what would you do?
II
Cues-to-threat elicitation III
Strategies elicitation
Elicit cues-to-strategies
Strategy evaluation
IV
Additional judgments (optional)
field. As with other methods that ask participants to reflect (e.g., critical incident analysis and cognitive interview, Flanagan, 1954; Geiselman and Fisher, 1988; Klein, 1989), eliciting the interviewee's past experiences with the critical task serves three purposes. First, it helps the interviewer confirm the interviewee's familiarity with the task. Second, it provides the interviewee with a concrete and familiar context from which to consider the critical task. Third, interviewees gain familiarity with verbalizing their thoughts about the task. Stage II: Elicitation of threats and cues-to-threats. The TSI then attempts to elicit multiple “threats” to performing the task. “Tell me something that might happen that would threaten your effective management of the situation.” In practice, most participants are comfortable generating an initial collection of threats, sometimes adding threats as the interview continues. After a particular threat or set of threats is identified, one threat is chosen based on the participant's familiarity with the threat and the researcher's goals. The TSI then seeks to elicit diagnostic indicators the interviewee used to establish and confirm the particular threat situation: “How do you become aware of the threat?” or a similar question can be used to elicit these cues-to-threat. If the participant has generated multiple threats, the interviewer could return to each in order to ascertain the cues to that particular threat. Establishing the threat along with the cues-to-threat is believed to help situate the participant within the developing, imagined scenario. Stage III: Elicitation of strategies and cues-to-strategies. In this stage, the interviewer elicits strategies and actions used to manage
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
the chosen threat: “Tell me a strategy you use to keep the threat from interfering with this task. A strategy is a plan or a method to achieve a goal. A strategy is not usually one action, but we sometimes think of it as an action.” At this point, we have found it helpful to illustrate strategies and the relationship of strategies to threats by giving the participant an unrelated example. We have had the most success with a scenario in which plans to get to the airport are thwarted. “… your friend who was going to drive you to the airport has gotten sick. What strategies might you use to make your flight?” This is an easy example that helps the participant see relationships among actions (e.g., I’d call Fred, I’d take the first MARTA) and more general strategies (e.g., I’d call a friend, I’d take public transportation). Once the participant feels comfortable with what a strategy is, the TSI returns to the task and threat at hand. This phase of the TSI is interactive and iterative. The interviewer should continue to request additional strategies until the participant can no longer think of a strategy. The TSI is designed to impose increasing constraints on strategy selection by asking the interviewee what they would do if the given strategy were unsuccessful. This technique helps reveal less common strategies. Also, it shows that some strategies are only implemented after multiple other strategies prove unsuccessful. The participant can continue to generate strategies in response to the hypothetical failure of a strategy or until they can no longer generate strategies or until they generate a predetermined “stopping” strategy. An example of the former might be when a pilot says “Well, I guess I’d crash”; an example of the latter might be when a nurse generates “I’d call a code” or “I’d call the physician” moving responsibility from him or her to others. After identifying strategies to manage a particular threat, the questions in this stage focus on encouraging the interviewee to verbalize knowledge, often implicit knowledge, about the conditions underlying the choice of each strategy. The TSI uses a variety of techniques such as contextualization and contrasts between choices (e.g., why would you do X rather than the Y strategy?) to help make explicit otherwise implicit information that cues one strategy over another. Cues-to-strategies are properties of the situation that help the operator nominate or select a strategy. These cues-to-strategy can be deduced from answers to questions such as: “When would you choose to try that strategy? In what context or situation would you use this strategy? What would have to be true for you to use this strategy in the acute situation? This might be something in the environment, something in the way you’re thinking, or something about the situation as a whole.” A cue-tostrategy may only link to a single strategy, or it may suggest multiple strategies. The interviewer should be alert to cues that are apparent to the expert being interviewed, but that are not otherwise clear. Occasionally, the participant will use a “cue” that actually is a complex of cues or that presupposes processing by the participant that has not been articulated. For example, “If the child looks like he's uncomfortable” could be an opportunity to discover more specific cues about what it means “to look uncomfortable.” The strategy phase can benefit from summarizing and organizing. As part of this, the researcher may find it useful to ask about which strategies are used often, or have the higher priority. Optionally, these types of questions can be expanded into a separate data collection stage for additional judgments. Stage IV: Additional judgments. The TSI progresses in an iterative fashion, eliciting strategies and cues-to-strategies. Following these iterations, it is possible to gather additional information about the threats, strategies, or cues elicited. We have, for example, asked operators to make judgments on factors (e.g., workload, situation awareness, effectiveness) affecting strategy choice.
339
Ratings can also be used to elaborate on impediments to job performance or to educate new operators. 3. Analyzing the TSI Interviews are transcribed verbatim. The nonredundant utterances in the transcripts are then coded by a pair of judges into cuesto-threats, cues-to-strategies, and actions for each threat. For illustrative purposes, we used the TSI to analyze the task of a professor revising a paper with the threat of a graduate student from the original paper being unavailable to help on the revision. 3.1. Threats Threats can be classified based on human factors principles in general or based on any of a number of existing taxonomies. Threats can also be prioritized based on the importance of tasks. For example, in aviation, threats to the task of aviating would be prioritized higher than threats to the task of navigation. Threats can also be mapped on to strategies used to manage them (see Durso et al. 2015). Once threats within a domain are identified, system modifications could be implemented in order to mitigate the effects of the threat or eliminate the threat altogether. Thus, the TSI could be used as a method to uncover system problems and suggest or motivate a particular redesign. 3.2. Strategies Although we ask interviewees for both strategies and actions indicative of strategies, they typically give actions that characterize strategies. These actions are not always organized into obvious strategies. Therefore, two judges independently sort the actions into “preferably 10 or fewer” piles mindful of the ultimate goals of the operator in managing the threat. The judges read the transcript prior to sorting and are given access to it while sorting. Each sort is done for each threat for each operator. The sort helps organize the actions into meaningful units and uncovers the latent strategies. The non-redundant actions mentioned in the graduate student unavailable example can be seen in Table 2. Final groupings of actions are determined by identifying agreements between the two judges. Fig. 1 shows an illustrative agreement matrix. The 28 actions given by the interviewee define both the columns and rows of the square matrix. Imagine judge 1 grouped actions 2 through 7 together and left action 8 isolated and judge 2 grouped all the actions 2 through 8 together. We can represent this in the two-dimensional matrix depicted in Fig. 1 by putting a forward slash, “/” for judge 1's groupings and a backward slash, “y” for judge 2. A positive agreement between the two judges is represented as the combination of a forward slash for judge 1 and a backward slash for judge 2 creating an “X”. Of course, each individual item must be sorted with itself, thus the main diagonal will always contain X's. In addition, both judges 1 and 2 grouped actions 2 through 7 together. Action 8 is part of the larger grouping for judge 2 but isolated for judge 1. A negative agreement is represented by an empty cell (e.g., both judges 1 and 2 agreed that action 1 did not go with action 2 in Fig. 1). This agreement matrix can be used to calculate reliability between the sorters. Strategies and sub-strategies from the agreement matrix were derived through a series of steps. Influenced by Guttman's (1947; Trochim, 2006) procedures for unidimensional scaling, our procedure orders the rows and columns based on relatedness. Although the unidimensional solution is algorithmically tractable, the two dimensional solution remains computationally prohibitive but can be expedited with researcher intervention. The first set of strategy
340
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
Table 2 Actions for the task of “Revising a paper” under the threat of “Graduate student on paper is unavailable.” Action # Actions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Negotiate amount of time that's available [with original student] Get somebody else Go find somebody else with expertise Worry about replacing them [original student] [New grad] would help you Talk to [new] grad about possibility of him taking over paper Get another grad student on board Get a faculty member involved to replace [original] grad student Touch base with them [original student] Try to contact [original] student Finish it myself Decide if you want to do it yourself Do it yourself without [original] student Forget paper Contact editor about the revisions and see what's necessary Focus on fewer comments from the reviews, rather than all of them Leave the [original] student as 1st author Reevaluate authorship order May have to replace authorship May take [original student] off paper and move someone else up and let them help with paper I would move them [original student] down [in the authorship order] Contact editor to get more time Ignore the editor's deadline and get it to them when I can get it to them Try to make the deadline with a poorer version of the chapter Go back to them [your help] and say we have to deal with the time we have Reevaluate expected value of the paper, what's the probability of publishing, value of that publication, and how much work is involved Reevaluate how much work it would take to do it yourself Reevaluate
clusters (solid boxes) capture the largest number of consecutive agreed upon groupings (an “X” in the matrix). Second, the dotted boxes represent a mini-max regrouping solution in which the fewest number of boxes cover the most number of groupings between actions while excluding the least amount of groupings. Groupings with sub-groupings (e.g., the sub-grouping “Get another student” within the grouping “Get Help” in Fig. 2) represent superordinate strategies containing sub-strategies. The twodimensional matrix lets one distinguish between actions that
form a strategy and those that form a sub-strategy. An empty cell (i.e., when judges agreed two actions did not belong to the same strategy) is never included in a strategy box. In our experience with the TSI, most of the disagreements between judges have proven not to be true disagreements. Rather judges typically disagree about the granularity of the strategies, that is whether the utterances reflect one broad strategy or two narrower, but related, strategies. This procedure resolves these disagreements by creating an overall strategy that may entail substrategies. Any judgments that were not accounted for by the strategy groupings (i.e., the two unboxed single slashes) were considered true disagreements. Additional details of the process are presented in Ferguson et al. (2012). The strategies obtained from this method can be vetted by a subject matter expert (SME). Additionally, looking at strategies across operators for the same threat or even different threats reveals strategies that are more common and utilized by multiple operators and strategies that are more specific to an individual operator. SMEs and even the operators themselves are also able to identify the strategies that are standard operating procedures and those strategies that may have been created by the operator. When interviewing multiple operators within the same domain standard procedures are often brought to light. Illustratively, our participant offered nine strategies to handle the threat of “graduate student unavailable” (see Fig. 2), two of which contained sub-strategies resulting in a total of 11 strategy clusters. The strategy of “Get Help” contained the sub-strategies “Get help from another graduate student” and “Get help from faculty member”. The strategy of “Evaluate Authorship” had two sub-strategies, “Leave student as first author” and “Change authorship”. The remaining strategies were “Contact Original Student”, “Negotiate Time with Original Student”, “Finish it Yourself”, “Time Management”, “See What Revisions are Necessary”, “Reevaluate”, and “Forget Paper”. 3.3. Graphs In addition to the omnibus comparisons discussed above, we were able to develop detailed depictions of the relationships among threats and strategies through the cues-to-threats and cues-
Fig. 1. Agreement Matrix for the task of “Revising a paper” under the threat of “Graduate student on paper is unavailable”. Each action is represented by a number for both the rows and columns. A forward slash (/) indicates that one action (row) and another action (column) were sorted together by judge 1, a backward slash (y) by judge 2, and X by both judges. Boxes represent strategies and sub-strategies (see text for explanation).
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
341
Fig. 2. Example Chart for the task of “Revising a paper” under the threat of “Graduate student on paper is unavailable.” The left column has the task, threat, and cues-to-threat, the central column contains cues-to-strategies, and the right column has actions grouped into strategies and sub-strategies. The entries in the chart are utterances or close paraphrases from the interview.
342
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
Fig. 2. (continued).
to-strategies. These graphical relationships of the strategies, threats, and cues should be vetted by a SME. Cues-to-threats are similar to Crandall and Getchell-Reiter’s (1993) indicators. However, cues-to-strategies are a unique product of the TSI. Cues-tostrategies are not necessarily mapped to strategies as are stimuli to responses, but rather reflect the indicators operators use to supply evidence in favor of a strategy. One strategy may be suggested by multiple cues and one cue can supply evidence for multiple strategies. Such cue-strategy mapping is similar to the cues-goals connection in the lens model (Tolman and Brunswik, 1935). Our analysis assumes that evidence from cues accumulates until, roughly, evidence passes a threshold and the strategy is nominated. Thus, the model should be viewed as an evidenceaccumulation model (Newell and Lee, 2011). We focus here on cues-to-strategies. Fig. 2 shows an example of a graph depicting the interrelationships among the threat (e.g., graduate student unavailable) on the left side of the graph and their cues, cues-to-threats, beneath the threat (e.g., “it's a paper that we’ve worked on and sent out and between the time it gets started and time reviewed, grad student has graduated”), to strategies on the right side of the figure (e.g., Negotiate Time) and their cues, cues-to-strategies, in the middle of the figure (e.g., “If I could get a hold of the [original] student”). Occasionally, cues-to-threats may also act as cues-to-strategies. For example, the cue “graduate student not available for enough hours” not only leads to awareness of the threat of graduate student unavailable to help on paper, but also suggests the strategy of “negotiate time with the graduate student.” We have found that many of the cues-to-strategies also suggest more abstract categories. For example, cues may “block” or weigh against consideration of a strategy, making it less likely that a strategy would apply. In Fig. 2, blocker cues are depicted by dashed lines. In the example depicted, the cue, “If [the original graduate student] didn't want to participate anymore”, blocks the strategy of Negotiate Time and suggests the strategy of Get Help. Some cues also indicated the operators were using coordinated action sequences, in other words, they used the failure of one strategy to cue another. Finally, some cues were such that they were not really evident in the situation directly but rather suggested a complex evaluation of the environment that the operator did not detail in
the interview (e.g., “if not worth your time”). These cues imply the use of epistemic strategies that give rise to additional information (Martinez-Moyano, Rich, Conrad, Stewart, and Andersen, n.d.; Sorum et al., 2002; Tolman and Brunswik, 1935). 4. Discussion Eliciting strategies from skilled professionals can be an important contribution to a variety of human factors approaches. Asking operators first to identify threats and then to state the strategies used to manage those threats, seemed to have facilitated the operators' ability to reflect on how they manage their work. We were able to glean substantial data at both an omnibus and micro level. The microanalysis captured in the charts of threats, cues, and strategies is also of value. Thus, the TSI seemed to be effective in eliciting information that might otherwise lay dormant as inscrutable operators' intuition. The strategies, and the cues that experts use to launch those strategies, can also be useful in the development of training programs. For example, Staszewski and Davison (2000) were able to develop an effective training program for mine detection in the military by eliciting the cues used by an expert mine sweeper and developing a training program based on those insights. It is possible that focusing on the strategies and cues used by operators can lead to a training program for operators and can give cognitive ergonomics researchers insights into the strategic thinking used by operators to keep performance high and workload within tolerance. This in turn can be used along with methodologies like CWA in redesigning the work system or as a tool for proactive risk management. 4.1. Limitations The TSI has the limitations of other interview methods that elicit subjective reports from experts. It is possible that strategies elicited through the TSI may reflect only those which were accessible in memory by the interviewees. Further, some strategies are not verbalized by interviewees because of social desirability. The TSI may be most useful in domains that allow for operator variability in
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
decision making rather than domains with rigid standard operating procedures. The TSI is explicitly designed to elicit knowledge about strategies at the level of the individual; it is less able to capture in a direct way threats and strategies at other levels. Researchers should keep in mind that operators are likely involved in threats and strategies that only touch the individual level. Thus, for example, a threat emerging from the macro level may be understood and perceived at the individual level in a limited way and only from a particular perspective. Although interviewees adapt quickly to the TSI, the interview often takes a substantial time for the initial threat (perhaps 30 min); certainly, longer than subsequent interviewed threats with the same operator. The TSI also benefits from a trained interviewer who is facile in moving between threats and strategies and guiding the participant in comparing and contrasting the reasons for using one strategy rather than another. Because the TSI seeds the interview with a particular task, and focuses on a particular threat, it takes multiple interviews with multiple operators to cover multiple tasks or threats. Finally, currently the TSI produces networks that are well suited for representing an individual operator, however methods for combining the results across operators are proving viable. Gregg et al. (2014) aggregated strategies across four nurses for a common threat experienced in the intensive care unit by applying the agreement matrix to each of the four nurse's strategy clusters. This aggregate matrix allows for common strategies shared among operators and individual strategies that are unique to a single operator revealing the commonalities and differences between the strategies operators possess. 5. Future work Future research could focus on expanding the TSI analysis to include multiple operators, multiple threats, and multiple tasks into a single expert system. The work could focus on developing a compendium of strategies within a task. We are currently working on combining strategies across operators for managing a single threat to a task. Different threats to the same task could also be combined to better understand how implementing strategies to combat a particular threat may affect management of other threats to that task. Certainly threats interact in a way that is not yet apparent from the TSI analyses. Relatedly, it is also possible that some strategies may be helpful in mitigating multiple threats. The components of the TSI can be used to model strategic behavior of operators. For example, we can investigate how the presence of certain cues influence threat recognition and strategy selection. Such a model can be used as a resource to train expert behavior to novices. Finally, future work is needed to compare knowledge elicitation through the TSI with other methods aimed at eliciting strategies from expert operators. Behavioral observations can be used to confirm the data elicited from the TSI. Ultimately, the TSI could prove useful in creating an integrated database across operators, tasks, threats, and strategies that could be used to predict behavior of expert operators or to facilitate training of apprentice operators. Acknowledgments We would like to thank James Roberts for discussing the twodimensional Guttman problem with us. Thanks also to Kate Bleckley, Pat DeLucia, and three anonymous reviewers for comments on an earlier draft of this paper.
343
References Amalberti, R., Hourlier, S., 2011. Human error reduction strategies in health care. In: Carayon, P. (Ed.), Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. Lawrence Erlbaum Associates Hillsdale, NJ, pp. 385e399. Appelbaum, S.H., 1997. Socio-technical systems theory: an intervention strategy for organizational development. Manag. Decis. 35, 452e463. http://dx.doi.org/ 10.1108/00251749710173823. Averty, P., Guittet, K., Lezaud, P., 2008. An ordered logit model of air traffic controllers' conflict risk judgment. Air Traffic Control Q. 16, 101e125. Benner, P., 1982. From novice to expert. Am. J. Nurs. 82, 402e407. Cornelissen, M., McClure, R., Salmon, P.M., Stanton, N.A., 2014. Validating the strategies analysis diagram: assessing the reliability and validity of a formative method. Appl. Ergon. 45 (6), 1484e1494. http://dx.doi.org/10.1016/j.apergo. 2014.04.010. Crandall, B., Getchell-Reiter, K., 1993. Critical decision method: a technique for eliciting concrete assessment indicators from the “intuition” of NICU nurses. Adv. Nurs. Sci. 16, 42e51. Delaney, P.F., Reder, L.M., Staszewski, J.J., Ritter, F.E., 1998. The strategy-specific nature of improvement: the power law applies by strategy within task. Psychol. Sci. 9, 1e7. http://dx.doi.org/10.1111/1467-9280.00001. Durso, F.T., Alexander, A.L., 2010. Managing workload, performance, and situation awareness in aviation systems. In: Salas, E., Maurino, D. (Eds.), Human Factors in Aviation, second ed. Academic Press, Inc, pp. 217e247. Durso, F.T., Ferguson, A.N., Kazi, S., Cunningham, C., Ryan, C., 2015. Strategic threat management: An exploration of nursing strategies in the pediatric intensive care unit. Appl. Ergon. 47, 345e354. Durso, F.T., Drews, F.A., 2010. Health care, aviation, and ecosystems: a socio-natural systems perspective. Curr. Dir. Psychol. Sci. 19, 71e75. http://dx.doi.org/10.1177/ 0963721410364728. Evans, J., St, B.T., 2000. What could and could not be a strategy in reasoning. In: Schaeken, W., Gino, D.V., D'Ydewalle, G. (Eds.), Deductive Reasoning and Strategies. Lawrence Erlbaum Associates, Mahwah, N.J, pp. 1e22. Faye, H., Rivera-Rodriguez, A.J., Karsh, B.T., Hundt, A.S., Baker, C., Carayon, P., 2010. Assessment of the medication management process. Jt. Comm. J. Qual. Patient Saf. 36, 376e384. Ferguson, A.N., Kazi, S., Durso, F.T., 2012. Revealing latent strategy structures from expert critical care nurses. In: Proceedings of the Human Factors and Ergonomics Society 56th Annual Meeting, vol. 56, pp. 801e805. http://dx.doi.org/ 10.1177/1071181312561167. Flanagan, J.C., 1954. The critical incident technique. Psychol. Bull. 51, 327e358. Retrieved from. http://www.ncbi.nlm.nih.gov/pubmed/13177800. Geiselman, R.E., Fisher, R.P., 1988. The cognitive interview: an innovative technique for questioning witness of crime. J. Police Crim. Psychol. 2 (2), 2e5. Gregg, S., Ferguson, A.N., Durso, F.T., 2014. A method for aggregating strategies across individuals: managing a lack of IV access. Proc. Hum. Factors Ergon. Soc. (in press). Gurses, A.P., Carayon, P., 2009. Exploring performance obstacles of intensive care nurses. Appl. Ergon. 40 (3), 509e518. http://dx.doi.org/10.1016/ j.apergo.2008.09.003. Gurses, A.P., Carayon, P., Wall, M., 2009. Impact of performance obstacles on intensive care nurses' workload, perceived quality and safety of care, and quality of working life. Health Serv. Res. 44 (2 Pt 1), 422e443. http://dx.doi.org/ 10.1111/j.1475-6773.2008.00934.x. Guttman, L., 1947. The Cornell technique for scale and intensity analysis. Educ. Psychol. Meas. 7, 247e279. Hassall, M.E., Sanderson, P.M., 2012. A formative approach to the strategies analysis phase of cognitive work analysis. Theor. Issues Ergon. Sci., 1e47. http:// dx.doi.org/10.1080/1463922X.2012.725781. Helmreich, R.L., Klinect, J.R., Wilhelm, J.A., 1999. Models of threat, error, and CRM in flight operations. In: Proceedings of the Tenth International Symposium on Aviation Psychology, pp. 677e682. Columbus: OH. Helson, H., 1949. Design of equipment and optimal human operation. Am. J. Psychol. 62, 473e497. Henneman, E., Gawlinski, A., Blank, F.S., Henneman, P.L., Jordan, D., McKenzie, J.B., 2010. Strategies used by critical care nurses to identify, interrupt, and correct medical errors. Am. J. Crit. Care 19 (6), 500e509. http://dx.doi.org/10.4037/ ajcc2010167. Holden, R.J., Rivera-Rodriguez, A.J., Faye, H., Scanlon, M.C., Karsh, B.-T., 2012. Automation and adaptation: nurses' problem-solving behavior following the implementation of bar-coded medication administration technology. Cogn. Technol. Work 15, 283e296. http://dx.doi.org/10.1007/s10111-012-0229-4. Hollnagel, E., Paries, J., Woods, D., Wreathall, J., 2010. Resilience Engineering in Practice: a Guidebook. Ashgate, England. Kahneman, D., Klein, G., 2009. Conditions for intuitive expertise: a failure to disagree. Am. Psychol. 64, 515e526. http://dx.doi.org/10.1037/a0016755. Klein, G. (Ed.), 1998. Sources of Power: How People Make Decisions. MIT Press, Cambridge, MA. Klein, G.A., 1989. Recognition-primed decisions. In: Rouse, W.B. (Ed.), Advances in Man-machine Systems Research, fifth ed. JAI Press, Greenwich, CT, pp. 47e92. Kleiner, B.M., 2006. Macroergonomics: analysis and design of work systems. Appl. Ergon. 37, 81e89. http://dx.doi.org/10.1016/j.apergo.2005.07.006. Kohn, L.T., Corrigan, J.M., Donaldson, M.S. (Eds.), 2000. To Err Is Human. National Academy Press, Washington, DC.
344
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
Koros, A., Della Rocco, P.S., Panjwani, G., Ingurgio, V., D'Arcy, J.-F., 2006. Complexity in Airport Traffic Control Towers: a Field Study. Part 2. Controller Strategies and Information Requirements. Atlantic City International Airport, NJ (p. DOT/FAA/ TCe06/22). Lintern, G., 2009. The Foundations and Pragmatics of Cognitive Work Analysis a Systematic Approach to Design of Large-scale Information Systems, first ed. Loft, S., Sanderson, P., Neal, A., Mooij, M., 2007. Modeling and predicting mental workload in en route air traffic control: critical review and broader implications. Hum. Factors 49, 376e399. http://dx.doi.org/10.1518/001872007X197017. Luwel, K., Schillemans, V., Onghena, P., Verschaffel, L., 2009. Does switching between strategies within the same task involve a cost? Br. J. Psychol. 100, 753e771. http://dx.doi.org/10.1348/000712609X402801. Martinez-Moyano, I. J., Rich, E. H., Conrad, S. H., Stewart, T. R., & Andersen, D. F. (n.d.). Integrating Judgment and Outcome Decomposition: Exploring Outcomebased Learning Dynamics (pp. 1e12). Merriam-Webster com. (n.d.). Strategy. Retrieved October 11, 2013, from http:// www.merriam-webster.com/dictionary/strategy. Merritt, A., Klinect, J., 2006. Management Origin and Development of TEM, pp. 1e20. Retrieved from. http://homepage.psy.utexas.edu/homepage/group/ helmreichlab/publications/pubfiles/TEM.Paper.12.6.06.pdf. Naikar, N., Pearce, B., 2003. Analysing activity for future systems. In: Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting, vol. 47, pp. 1928e1932. http://dx.doi.org/10.1177/154193120304701706. Nehme, C.E., Scott, S.D., Cummings, M.L., Furusho, C.Y., 2006. Generating requirements for futuristic heterogenous unmanned systems. In: Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, vol. 50, pp. 235e239. http://dx.doi.org/10.1177/154193120605000306. Newell, B.R., Lee, M.D., 2011. The right tool for the job? Comparing an evidence accumulation model and a naïve strategy selection model of decision making. J. Behav. Decis. Mak. 24, 456e481. Reason, J., 2000. Human error: models and management. BMJ: Br. Med. J. 320, 768e770. Roth, E.M., Malsch, N., Multer, J., Coplen, M., 1999. Understanding how train dispatchers manage and control trains: a cognitive task analysis of a distributed team planning task. In: Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting, vol. 43, pp. 218e222. http://dx.doi.org/10.1177/ 154193129904300319.
Roth, E.M., 2008. Understanding cognitive strategies for shared situation awareness across a distributed system: an example of strategies analysis. In: Bisantz, A.M., Burns, C.M. (Eds.), Applications of Cognitive Work Analysis. CRC, Boca Raton, FL, pp. 129e147. Siegler, R.S., Jenkins, E., 1989. How Children Discover New Strategies. Lawrence Erlbaum, Mahwah, NJ. Siegler, R.S., Stern, E., 1998. Conscious and unconscious strategy discoveries: a microgenetic analysis. J. Exp. Psychol. General. 127, 377e397. Retrieved from. http://www.ncbi.nlm.nih.gov/pubmed/9857493. Siegler, R.S., Shrager, J., 1984. Strategy choices in addition and subtraction: how do children know what to do? In: Sophian, C. (Ed.), The Origins of Cognitive Skills. Erlbaum, Hillsdale, NJ, pp. 229e293. Sorum, P.C., Stewart, T.R., Mullet, E., Gonzalez-Vallejo, C., Shim, J., Chasseigne, G., Grenier, B., 2002. Does choosing a treatment depend on making a diagnosis? US and French physicians' decision making about acute Otitis media. Med. Decis. Mak. 22 (5), 394e402. http://dx.doi.org/10.1177/027298902236941. Sperandio, J.C., 1971. Variation of operator strategies and regulating effects on workload. Ergonomics 14, 571e577. Staszewski, J.J., Davison, A., 2000. Mine detection training based on expert skill. In: Dubey, A.C., Harvey, J.F., Broach, J.T., Dugan, R.E. (Eds.), Proceeding of Society of Photo-optical Instrumentation Engineers 13th Annual Meeting, pp. 766e777. Thompson, V.A., 2014. What intuitions are... and are not. Psychol. Learn. Motiv. 60, 35e75. Tolman, E.C., Brunswik, E., 1935. The organism and the causal texture of the environment. Psychol. Rev. 42, 43e77. http://dx.doi.org/10.1037/h0062156. Trochim, W.M.K., 2006. Guttman Scaling. Retrieved March 10, 2012, from. http:// www.socialresearchmethods.net/kb/scalgutt.php/. Tucker, A.L., 2004. The impact of operational failures on hospital nurses and their patients. J. Operat. Manag. 22, 151e169. http://dx.doi.org/10.1016/ j.jom.2003.12.006. Vicente, K.J., 1999. Cognitive Work Analysis: toward Safe, Productive, and Healthy Computer-based Work. Lawrence Erlbaum Associates, Mahwah, N.J. Woods, D.D., 2003. Discovering how distributed cognitive systems work. In: Hollnagel, E. (Ed.), Handbook of Cognitive Task Design. Lawrence Erlbaum, Mahwah, NJ, pp. 37e53.
Contents lists available at ScienceDirect
Applied Ergonomics journal homepage: www.elsevier.com/locate/apergo
The Threat-Strategy Interview Francis T. Durso*, Sadaf Kazi, Ashley N. Ferguson School of Psychology, Georgia Institute of Technology, 654 Cherry Street, Atlanta, GA 30332, USA
a r t i c l e i n f o
a b s t r a c t
Article history: Available online 23 September 2014
Operators in dynamic work environments use strategies to manage threats in order to achieve task goals. We introduce a structured interview method, the Threat-Strategy Interview (TSI), and an accompanying qualitative analysis to induce operator-level threats, strategies, and the cues that give rise to them. The TSI can be used to elicit knowledge from operators who are on the front line of managing threats to provide an understanding of strategic thinking, which in turn can be applied toward a variety of problems. © 2014 Elsevier Ltd and The Ergonomics Society. All rights reserved.
Keywords: Strategies Knowledge elicitation Threat and error management
1. Introduction Safety-critical dynamic environments such as aviation or healthcare experience numerous challenges to system safety. In one view, Threat and Error Management (TEM; Helmreich et al., 1999; Merritt and Klinect, 2006), these challenges have been referred to as threats. Threats refer to characteristics of operational environments which impede safe completion of tasks; this may include factors such as non-optimal staffing policies, poor facility design, strictly enforced role hierarchies, or poor communication within teams. Errors are likely to occur in the presence of threats such as fatigue and cognitive overload. Similar to TEM's use of the term “threat” to denote any condition which impedes safe task completion, healthcare research uses terms such as “failure modes” (Faye et al., 2010), “adverse events” (Kohn et al., 2000), “performance obstacles” (Gurses et al., 2009), “operational failures” (Tucker, 2004), “problems” (Holden et al., 2012), or “variances” (Kleiner, 2006) to describe conditions that challenge performance. 1.1. Strategies The TEM model suggests that threats and errors are managed by the operator. This management of threats and their cognates e failures, operational failures, problems, and performance obstacleseoccurs by the operators' selection and implementation of strategies. Strategies are important in maintaining performance and safety in dynamic domains. Strategies can be seen at all levels
* Corresponding author. Tel.: þ1 404 894 6771. E-mail address: [email protected] (F.T. Durso). http://dx.doi.org/10.1016/j.apergo.2014.08.001 0003-6870/© 2014 Elsevier Ltd and The Ergonomics Society. All rights reserved.
of socio-technical systems (Appelbaum, 1997) and socio-natural systems (Durso and Drews, 2010). For example, at a macro-level, (Reason 2000; Amalberti and Hourlier, 2011) speaks of organization level strategies that can be adopted to improve patient safety. At a meso-level, crew resource management strategies affect how teams in a cockpit or in operating rooms coordinate activities. Finally, at an individual, micro-level, operators at work are believed to use strategies to perform “on par” (Helson, 1949) keeping performance high and workload low (Durso and Alexander, 2010). In this article, we focus on strategies at the level of the operator, that is, strategies that an air traffic controller might use to move traffic or that a nurse might use to deliver care to a patient. From its origins in the military, strategies are now recognized and used as an explanatory concept in virtually every modern endeavor. There have also been notable studies in particular domains designed to determine the strategies that are used and why they are selected (e.g., Siegler and Shrager, 1984). Luwel, et al. (2009) remark on the 25 years of research on strategies that demonstrate that individuals show “remarkable variability in their strategies for accomplishing various cognitive tasks.” (p. 753). Luwel et al. (2009) note the use of strategies in a variety of academic domains such as arithmetic, scientific reasoning, spelling, decision-making, time telling, serial recall, and currency conversion. Researchers have also looked at strategy use in operational, work domains (e.g., Loft et al., 2007; Sperandio, 1971). Strategies can be thought of as plans or methods to achieve a goal (Merriam-Webster.com, n.d.). Siegler and Jenkins (1989) add that strategies are nonobligatory. Shifting from first gear to second by moving the gear shift is a procedure, not a strategy, because moving the gear shift is obligatory. Strategies give rise to and are implemented by actions. Actions can be thought of as any behavior
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
carried out by the operator to achieve a goal. Thus, strategies are more abstract than the actions by which they are implemented. Successful strategies take into account constraints from the environment and from the organization. Strategies can be implemented quickly, as when a nurse uses the ABCs, airway-breathingcirculation prioritization strategy, or they can be selected more deliberately as when deciding whether or not to seek help rather than administer treatment to a patient alone. Once operators understand a situation, including the threats to performing their task, they can select a strategy to deal with those threats. Much of our strategic thinking is conscious (see Evans, 2000, for the argument that all strategies are conscious). However, the cues from the situation that give rise to a strategy may be difficult to articulate. With sufficient expertise, such strategy selection can be relatively automatic and relatively inscrutable to not only the researcher, but to the operator as well. Thus, it can be difficult to accumulate a knowledge base of strategies that are being used or gain an appreciation of the underlying processes that guide their use. Although it is sometimes difficult for an operator to verbalize the cues that initiated a strategy or the reason a particular strategy was selected, the strategies and reasons behind their selection can be made accessible (Siegler and Stern, 1998). 1.1.1. Strategies in operational work domains The interest in strategies is also valuable beyond academic domains. Over 40 years ago, Sperandio (1971) argued that strategies were key to understanding how operators control complex industrial environments. He argued that the operative strategy used by the air traffic controllers depended on the operator, the task, and the level of workload. Following Sperandio (1971), Durso and Alexander (2010) have argued that operators attempt to understand a situation, that is, they attempt to have sufficient situation awareness, so as to know when to change strategies in a continuous effort to keep performance high and workload manageable. Their perspective echoes Helson's (1949) Par Hypothesis: Performance that is less than a standard calls for greater effort whereas performance that exceeds that standard allows for less effort. As an illustration, Sperandio (1971) demonstrated that professional air traffic controllers under time pressure reduced the duration of important messages and eliminated secondary messages entirely. They also became more proactive, initiating more messages. Controllers increased the task bandwidth by reducing noise and redundancy. Since Sperandio, others interested in air traffic control (ATC, e.g., Averty et al., 2008; Koros, et al., 2006; Loft et al., 2007) have elicited a corpus of strategies from controllers. Some have argued that strategies are important to understanding fundamentals of how work is conducted. For example, Delaney et al. (1998) showed that the power law of practice fit individual data better when conditionalized based on strategy. Durso and Alexander (2010) point to a number of examples where superficially straightforward relationships such as between workload and performance, situation awareness and workload, or even task load and performance, disassociate. To Durso and Alexander the reason for the dissociations were changes in strategy. That is, rather than see a straightforward increase between task load and workload, operators change strategies as task load increases allowing them to restrict workload to a manageable range. In a review of the workload literature in ATC, Loft et al. (2007) argue that “Controller workload will not be effectively modeled until controllers' strategies for regulating the cognitive impact of task demand have been modeled.” (p. 376). The study of strategies also has credibility in a number of human factors research perspectives. For example, in Cognitive Work Analysis (CWA, e.g., Lintern, 2009; Vicente, 1999), strategies are used to manage changing task constraints. In TEM, team-based
337
strategies are used proactively to manage threats. In cockpit task management, strategies are used to prevent errors, or to manage the consequences of errors (Helmreich et al., 1999). This view is consistent with the development of resilience engineering (Hollnagel et al., 2010) and adaptation to dynamic industrial tasks. 1.2. Methods of strategy elicitation Researchers have tried to understand strategy use in work domains both analytically and empirically. Analytical methods produce a formative characterization. A formative characterization would in principle detail all possible strategies that could be produced in the work domain. This type of analysis characterizes disciplines like linguistics, in which the understanding of the structure of even extinct languages can be understood even though no one currently speaks the language. Similarly, a formative analysis produces strategies even if no one ever uses the strategy, or perhaps could ever use the strategy. Roth (2008) draws the connection to the concept of competence. Several researchers have taken a formative approach to strategies (e.g., Cornelisson et al., 2014; Hassall and Sanderson, 2012; Naikar and Pearce, 2003). Roth (2008) noted that formative analysis is especially valuable early in design and can be used to understand futuristic systems (e.g., Nehme et al., 2006). On the other hand, empirical methods produce a characterization of the strategies that operators actually perform in the work domain. These methods are descriptive, rather than formative. Unlike linguistics, psycholinguistics is, indeed, concerned about the language that is actually used by people. Thus, empirical methods relate to the concept of performance as contrasted with competence. Descriptive methods are essential when researchers want to know about psychology and behavior that actually characterizes the work domain. There are several methods available to elicit knowledge from subject matter experts and less experienced operators. Crandall and Getchell-Reiter (1993), for example, used the critical decision method, to understand how expert nurses assessed the presence of sepsis in neonates. The critical incident method is another knowledge elicitation technique to understand cognitive processes of operators during non-normal situations (Flanagan, 1954; Gurses and Carayon, 2009). The Staged World method uses simulations to observe how the use of artefacts shapes work and how practitioners adapt to “probes” or challenges to work (Woods, 2003). The methodology presented in the current paper is related to cognitive task analysis (CTA) and to a variety of structured interview procedures. However, these methods often focus on the states in which the operators find themselves, and less often on the strategies that operators use to manage these states. For instance, the CDM focuses on identifying specific cues that alert the operator to changes in critical safety parameters; the critical incident technique focuses on identifying whether certain states of the situation pose a threat to system safety. These methods do not focus specifically on how operators effectively manage threats to system safety. One interesting exception was Roth, et al. (1999) study of rail operators; they used a strategies analysis (see also Roth, 2008) that combined observation and interviews. Initial observations suggested strategies to the researchers which were in turn presented to the operators for their reactions. It is difficult to elicit strategies from experts. In fact, Roth (2008) notes that none of the strategies they explored in the rail study were explicitly mentioned by the operators in the interviews, although when proffered the strategies to other railroad workers “they readily confirmed that these were routine practices.” (p. 142). In healthcare, much of these inscrutable strategies are considered “nurse's intuition” (Benner, 1982).
338
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
We attempted to create an interview procedure that focused on strategies and that was easily understood by domain experts. We wanted a procedure that presented the world to the operators in a way concordant with the way the operators naturally segmented their work: Responding with strategies to threats they face seems to be such a natural segmentation. In addition, we wanted a procedure that was effective at encouraging operators to compare and contrast strategies along with the constraints and cues that served as boundary conditions of their strategies. This, we believed, was important to helping operators articulate heretofore implicit strategies. Finally, we wanted to do this in an interview procedure, rather than for example relying on our interpretation of observations to create the initial strategies. Observations are often difficult and in some cases impossible. In addition, we wanted operators to offer their own strategies rather than ones we induced from observations. The purpose of the current article is to describe a structured interview procedure designed to elicit strategies from skilled operators. The interview procedure asks operators first to generate threats to performing a task and then to provide strategies they would use when faced with those threats. We then use the result of this Threat-Strategy Interview (TSI) to map strategies to the cues in the situation that give rise to the strategy selected. In this way we hope to offer a procedure to illuminate the “some cue that something was not right” to which Henneman's nurses referred (Henneman et al., 2010). We agree with Roth (2008) that descriptive, empirical methods “offer an attractive alternative and/or complement to analytic approaches” (p. 132). Observations are time consuming and not always possible. Structured interviews have been successful at eliciting insights from the interviewed operators about the reasons they made particular decisions even when that decision was not readily apparent even to the operator. For example, work by Klein (1989) presented a variant of the critical incident technique (Flanagan, 1954) to elicit knowledge about managing dynamic environments that are characterized by time pressure. Crandall and Getchell-Reiter (1993) used the critical decision method (CDM) to interview neonatal intensive care unit nurses. They found that CDM interviews were able to elicit more information than interviews without CDM probes. 2. The TSI method Our structured interview procedure differs from previous ones in its focus on strategies. Leveraging the TEM perspective (Merritt and Klinect, 2006) we developed a structured interview that begins by eliciting from the operator threats to performing a task. This interview can be used for any dynamic environment to elicit threats and operator's strategies. Generating threats to their job came naturally to the operators. Once in the mindset of a specific threat, operators easily generated strategies to deal with that threat. Operators were then able to generate the cues that weighted the selection of one strategy over the other strategies they had available. Anecdotally, articulating the cues often came with an introspective “aha” that suggested the nurse had brought to consciousness a connection previously tacit and part of her or his operator's intuition (Kahneman and Klein, 2009; Klein, 1998; Thompson, 2014). The TSI was conducted in a structured format in the following four stages. Table 1 summarizes the stages. Stage I: Introduction of the critical task and elicitation of past experiences with the critical task. In this stage, the interviewer describes a critical work-related task and requests the interviewee to reflect upon relevant past experiences. The task should be selected based on discussion with an experienced confederate in the area to ensure the task is a common and important task in this
Table 1 Sketch of the Threat Strategy Interview (TSI). Stage
Goal
Description/example questions
I
Introduction of the Critical Task & Elicitation of interviewee's past experiences with the critical task. Threat elicitation
Describe the critical work-related task and discuss formal and informal experience with the threat as well as a recollection of a recent or memorable experience. Tell me something that might happen to threaten your effective management of the situation. How do you become aware of the threat? {multiple cues-to-threat are elicited} Tell me a strategy you use to keep the threat from interfering with this task. A strategy is a plan or a method to achieve a goal. A strategy is not usually one action, but you may think of it as an action. {multiple strategies are elicited} Thinking of Strategy A, can you tell me when you would choose to try that strategy? In other words, in what context or situation would that strategy work best? What would the situation be like that leads to picking that strategy? What would need to be true for you to pick that strategy? This might be something in the environment, something in the way you’re thinking, or something about the situation as a whole. So, when would you choose to use Strategy A? {multiple cuesto-strategies are elicited} Could you think about a specific piece of information or a cue that would encourage you to choose Strategy A over Strategy B? How can you tell if Strategy A is working? Or how can you tell when it's not working and you should switch? If that strategy didn't work, what would you do?
II
Cues-to-threat elicitation III
Strategies elicitation
Elicit cues-to-strategies
Strategy evaluation
IV
Additional judgments (optional)
field. As with other methods that ask participants to reflect (e.g., critical incident analysis and cognitive interview, Flanagan, 1954; Geiselman and Fisher, 1988; Klein, 1989), eliciting the interviewee's past experiences with the critical task serves three purposes. First, it helps the interviewer confirm the interviewee's familiarity with the task. Second, it provides the interviewee with a concrete and familiar context from which to consider the critical task. Third, interviewees gain familiarity with verbalizing their thoughts about the task. Stage II: Elicitation of threats and cues-to-threats. The TSI then attempts to elicit multiple “threats” to performing the task. “Tell me something that might happen that would threaten your effective management of the situation.” In practice, most participants are comfortable generating an initial collection of threats, sometimes adding threats as the interview continues. After a particular threat or set of threats is identified, one threat is chosen based on the participant's familiarity with the threat and the researcher's goals. The TSI then seeks to elicit diagnostic indicators the interviewee used to establish and confirm the particular threat situation: “How do you become aware of the threat?” or a similar question can be used to elicit these cues-to-threat. If the participant has generated multiple threats, the interviewer could return to each in order to ascertain the cues to that particular threat. Establishing the threat along with the cues-to-threat is believed to help situate the participant within the developing, imagined scenario. Stage III: Elicitation of strategies and cues-to-strategies. In this stage, the interviewer elicits strategies and actions used to manage
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
the chosen threat: “Tell me a strategy you use to keep the threat from interfering with this task. A strategy is a plan or a method to achieve a goal. A strategy is not usually one action, but we sometimes think of it as an action.” At this point, we have found it helpful to illustrate strategies and the relationship of strategies to threats by giving the participant an unrelated example. We have had the most success with a scenario in which plans to get to the airport are thwarted. “… your friend who was going to drive you to the airport has gotten sick. What strategies might you use to make your flight?” This is an easy example that helps the participant see relationships among actions (e.g., I’d call Fred, I’d take the first MARTA) and more general strategies (e.g., I’d call a friend, I’d take public transportation). Once the participant feels comfortable with what a strategy is, the TSI returns to the task and threat at hand. This phase of the TSI is interactive and iterative. The interviewer should continue to request additional strategies until the participant can no longer think of a strategy. The TSI is designed to impose increasing constraints on strategy selection by asking the interviewee what they would do if the given strategy were unsuccessful. This technique helps reveal less common strategies. Also, it shows that some strategies are only implemented after multiple other strategies prove unsuccessful. The participant can continue to generate strategies in response to the hypothetical failure of a strategy or until they can no longer generate strategies or until they generate a predetermined “stopping” strategy. An example of the former might be when a pilot says “Well, I guess I’d crash”; an example of the latter might be when a nurse generates “I’d call a code” or “I’d call the physician” moving responsibility from him or her to others. After identifying strategies to manage a particular threat, the questions in this stage focus on encouraging the interviewee to verbalize knowledge, often implicit knowledge, about the conditions underlying the choice of each strategy. The TSI uses a variety of techniques such as contextualization and contrasts between choices (e.g., why would you do X rather than the Y strategy?) to help make explicit otherwise implicit information that cues one strategy over another. Cues-to-strategies are properties of the situation that help the operator nominate or select a strategy. These cues-to-strategy can be deduced from answers to questions such as: “When would you choose to try that strategy? In what context or situation would you use this strategy? What would have to be true for you to use this strategy in the acute situation? This might be something in the environment, something in the way you’re thinking, or something about the situation as a whole.” A cue-tostrategy may only link to a single strategy, or it may suggest multiple strategies. The interviewer should be alert to cues that are apparent to the expert being interviewed, but that are not otherwise clear. Occasionally, the participant will use a “cue” that actually is a complex of cues or that presupposes processing by the participant that has not been articulated. For example, “If the child looks like he's uncomfortable” could be an opportunity to discover more specific cues about what it means “to look uncomfortable.” The strategy phase can benefit from summarizing and organizing. As part of this, the researcher may find it useful to ask about which strategies are used often, or have the higher priority. Optionally, these types of questions can be expanded into a separate data collection stage for additional judgments. Stage IV: Additional judgments. The TSI progresses in an iterative fashion, eliciting strategies and cues-to-strategies. Following these iterations, it is possible to gather additional information about the threats, strategies, or cues elicited. We have, for example, asked operators to make judgments on factors (e.g., workload, situation awareness, effectiveness) affecting strategy choice.
339
Ratings can also be used to elaborate on impediments to job performance or to educate new operators. 3. Analyzing the TSI Interviews are transcribed verbatim. The nonredundant utterances in the transcripts are then coded by a pair of judges into cuesto-threats, cues-to-strategies, and actions for each threat. For illustrative purposes, we used the TSI to analyze the task of a professor revising a paper with the threat of a graduate student from the original paper being unavailable to help on the revision. 3.1. Threats Threats can be classified based on human factors principles in general or based on any of a number of existing taxonomies. Threats can also be prioritized based on the importance of tasks. For example, in aviation, threats to the task of aviating would be prioritized higher than threats to the task of navigation. Threats can also be mapped on to strategies used to manage them (see Durso et al. 2015). Once threats within a domain are identified, system modifications could be implemented in order to mitigate the effects of the threat or eliminate the threat altogether. Thus, the TSI could be used as a method to uncover system problems and suggest or motivate a particular redesign. 3.2. Strategies Although we ask interviewees for both strategies and actions indicative of strategies, they typically give actions that characterize strategies. These actions are not always organized into obvious strategies. Therefore, two judges independently sort the actions into “preferably 10 or fewer” piles mindful of the ultimate goals of the operator in managing the threat. The judges read the transcript prior to sorting and are given access to it while sorting. Each sort is done for each threat for each operator. The sort helps organize the actions into meaningful units and uncovers the latent strategies. The non-redundant actions mentioned in the graduate student unavailable example can be seen in Table 2. Final groupings of actions are determined by identifying agreements between the two judges. Fig. 1 shows an illustrative agreement matrix. The 28 actions given by the interviewee define both the columns and rows of the square matrix. Imagine judge 1 grouped actions 2 through 7 together and left action 8 isolated and judge 2 grouped all the actions 2 through 8 together. We can represent this in the two-dimensional matrix depicted in Fig. 1 by putting a forward slash, “/” for judge 1's groupings and a backward slash, “y” for judge 2. A positive agreement between the two judges is represented as the combination of a forward slash for judge 1 and a backward slash for judge 2 creating an “X”. Of course, each individual item must be sorted with itself, thus the main diagonal will always contain X's. In addition, both judges 1 and 2 grouped actions 2 through 7 together. Action 8 is part of the larger grouping for judge 2 but isolated for judge 1. A negative agreement is represented by an empty cell (e.g., both judges 1 and 2 agreed that action 1 did not go with action 2 in Fig. 1). This agreement matrix can be used to calculate reliability between the sorters. Strategies and sub-strategies from the agreement matrix were derived through a series of steps. Influenced by Guttman's (1947; Trochim, 2006) procedures for unidimensional scaling, our procedure orders the rows and columns based on relatedness. Although the unidimensional solution is algorithmically tractable, the two dimensional solution remains computationally prohibitive but can be expedited with researcher intervention. The first set of strategy
340
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
Table 2 Actions for the task of “Revising a paper” under the threat of “Graduate student on paper is unavailable.” Action # Actions 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
Negotiate amount of time that's available [with original student] Get somebody else Go find somebody else with expertise Worry about replacing them [original student] [New grad] would help you Talk to [new] grad about possibility of him taking over paper Get another grad student on board Get a faculty member involved to replace [original] grad student Touch base with them [original student] Try to contact [original] student Finish it myself Decide if you want to do it yourself Do it yourself without [original] student Forget paper Contact editor about the revisions and see what's necessary Focus on fewer comments from the reviews, rather than all of them Leave the [original] student as 1st author Reevaluate authorship order May have to replace authorship May take [original student] off paper and move someone else up and let them help with paper I would move them [original student] down [in the authorship order] Contact editor to get more time Ignore the editor's deadline and get it to them when I can get it to them Try to make the deadline with a poorer version of the chapter Go back to them [your help] and say we have to deal with the time we have Reevaluate expected value of the paper, what's the probability of publishing, value of that publication, and how much work is involved Reevaluate how much work it would take to do it yourself Reevaluate
clusters (solid boxes) capture the largest number of consecutive agreed upon groupings (an “X” in the matrix). Second, the dotted boxes represent a mini-max regrouping solution in which the fewest number of boxes cover the most number of groupings between actions while excluding the least amount of groupings. Groupings with sub-groupings (e.g., the sub-grouping “Get another student” within the grouping “Get Help” in Fig. 2) represent superordinate strategies containing sub-strategies. The twodimensional matrix lets one distinguish between actions that
form a strategy and those that form a sub-strategy. An empty cell (i.e., when judges agreed two actions did not belong to the same strategy) is never included in a strategy box. In our experience with the TSI, most of the disagreements between judges have proven not to be true disagreements. Rather judges typically disagree about the granularity of the strategies, that is whether the utterances reflect one broad strategy or two narrower, but related, strategies. This procedure resolves these disagreements by creating an overall strategy that may entail substrategies. Any judgments that were not accounted for by the strategy groupings (i.e., the two unboxed single slashes) were considered true disagreements. Additional details of the process are presented in Ferguson et al. (2012). The strategies obtained from this method can be vetted by a subject matter expert (SME). Additionally, looking at strategies across operators for the same threat or even different threats reveals strategies that are more common and utilized by multiple operators and strategies that are more specific to an individual operator. SMEs and even the operators themselves are also able to identify the strategies that are standard operating procedures and those strategies that may have been created by the operator. When interviewing multiple operators within the same domain standard procedures are often brought to light. Illustratively, our participant offered nine strategies to handle the threat of “graduate student unavailable” (see Fig. 2), two of which contained sub-strategies resulting in a total of 11 strategy clusters. The strategy of “Get Help” contained the sub-strategies “Get help from another graduate student” and “Get help from faculty member”. The strategy of “Evaluate Authorship” had two sub-strategies, “Leave student as first author” and “Change authorship”. The remaining strategies were “Contact Original Student”, “Negotiate Time with Original Student”, “Finish it Yourself”, “Time Management”, “See What Revisions are Necessary”, “Reevaluate”, and “Forget Paper”. 3.3. Graphs In addition to the omnibus comparisons discussed above, we were able to develop detailed depictions of the relationships among threats and strategies through the cues-to-threats and cues-
Fig. 1. Agreement Matrix for the task of “Revising a paper” under the threat of “Graduate student on paper is unavailable”. Each action is represented by a number for both the rows and columns. A forward slash (/) indicates that one action (row) and another action (column) were sorted together by judge 1, a backward slash (y) by judge 2, and X by both judges. Boxes represent strategies and sub-strategies (see text for explanation).
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
341
Fig. 2. Example Chart for the task of “Revising a paper” under the threat of “Graduate student on paper is unavailable.” The left column has the task, threat, and cues-to-threat, the central column contains cues-to-strategies, and the right column has actions grouped into strategies and sub-strategies. The entries in the chart are utterances or close paraphrases from the interview.
342
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
Fig. 2. (continued).
to-strategies. These graphical relationships of the strategies, threats, and cues should be vetted by a SME. Cues-to-threats are similar to Crandall and Getchell-Reiter’s (1993) indicators. However, cues-to-strategies are a unique product of the TSI. Cues-tostrategies are not necessarily mapped to strategies as are stimuli to responses, but rather reflect the indicators operators use to supply evidence in favor of a strategy. One strategy may be suggested by multiple cues and one cue can supply evidence for multiple strategies. Such cue-strategy mapping is similar to the cues-goals connection in the lens model (Tolman and Brunswik, 1935). Our analysis assumes that evidence from cues accumulates until, roughly, evidence passes a threshold and the strategy is nominated. Thus, the model should be viewed as an evidenceaccumulation model (Newell and Lee, 2011). We focus here on cues-to-strategies. Fig. 2 shows an example of a graph depicting the interrelationships among the threat (e.g., graduate student unavailable) on the left side of the graph and their cues, cues-to-threats, beneath the threat (e.g., “it's a paper that we’ve worked on and sent out and between the time it gets started and time reviewed, grad student has graduated”), to strategies on the right side of the figure (e.g., Negotiate Time) and their cues, cues-to-strategies, in the middle of the figure (e.g., “If I could get a hold of the [original] student”). Occasionally, cues-to-threats may also act as cues-to-strategies. For example, the cue “graduate student not available for enough hours” not only leads to awareness of the threat of graduate student unavailable to help on paper, but also suggests the strategy of “negotiate time with the graduate student.” We have found that many of the cues-to-strategies also suggest more abstract categories. For example, cues may “block” or weigh against consideration of a strategy, making it less likely that a strategy would apply. In Fig. 2, blocker cues are depicted by dashed lines. In the example depicted, the cue, “If [the original graduate student] didn't want to participate anymore”, blocks the strategy of Negotiate Time and suggests the strategy of Get Help. Some cues also indicated the operators were using coordinated action sequences, in other words, they used the failure of one strategy to cue another. Finally, some cues were such that they were not really evident in the situation directly but rather suggested a complex evaluation of the environment that the operator did not detail in
the interview (e.g., “if not worth your time”). These cues imply the use of epistemic strategies that give rise to additional information (Martinez-Moyano, Rich, Conrad, Stewart, and Andersen, n.d.; Sorum et al., 2002; Tolman and Brunswik, 1935). 4. Discussion Eliciting strategies from skilled professionals can be an important contribution to a variety of human factors approaches. Asking operators first to identify threats and then to state the strategies used to manage those threats, seemed to have facilitated the operators' ability to reflect on how they manage their work. We were able to glean substantial data at both an omnibus and micro level. The microanalysis captured in the charts of threats, cues, and strategies is also of value. Thus, the TSI seemed to be effective in eliciting information that might otherwise lay dormant as inscrutable operators' intuition. The strategies, and the cues that experts use to launch those strategies, can also be useful in the development of training programs. For example, Staszewski and Davison (2000) were able to develop an effective training program for mine detection in the military by eliciting the cues used by an expert mine sweeper and developing a training program based on those insights. It is possible that focusing on the strategies and cues used by operators can lead to a training program for operators and can give cognitive ergonomics researchers insights into the strategic thinking used by operators to keep performance high and workload within tolerance. This in turn can be used along with methodologies like CWA in redesigning the work system or as a tool for proactive risk management. 4.1. Limitations The TSI has the limitations of other interview methods that elicit subjective reports from experts. It is possible that strategies elicited through the TSI may reflect only those which were accessible in memory by the interviewees. Further, some strategies are not verbalized by interviewees because of social desirability. The TSI may be most useful in domains that allow for operator variability in
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
decision making rather than domains with rigid standard operating procedures. The TSI is explicitly designed to elicit knowledge about strategies at the level of the individual; it is less able to capture in a direct way threats and strategies at other levels. Researchers should keep in mind that operators are likely involved in threats and strategies that only touch the individual level. Thus, for example, a threat emerging from the macro level may be understood and perceived at the individual level in a limited way and only from a particular perspective. Although interviewees adapt quickly to the TSI, the interview often takes a substantial time for the initial threat (perhaps 30 min); certainly, longer than subsequent interviewed threats with the same operator. The TSI also benefits from a trained interviewer who is facile in moving between threats and strategies and guiding the participant in comparing and contrasting the reasons for using one strategy rather than another. Because the TSI seeds the interview with a particular task, and focuses on a particular threat, it takes multiple interviews with multiple operators to cover multiple tasks or threats. Finally, currently the TSI produces networks that are well suited for representing an individual operator, however methods for combining the results across operators are proving viable. Gregg et al. (2014) aggregated strategies across four nurses for a common threat experienced in the intensive care unit by applying the agreement matrix to each of the four nurse's strategy clusters. This aggregate matrix allows for common strategies shared among operators and individual strategies that are unique to a single operator revealing the commonalities and differences between the strategies operators possess. 5. Future work Future research could focus on expanding the TSI analysis to include multiple operators, multiple threats, and multiple tasks into a single expert system. The work could focus on developing a compendium of strategies within a task. We are currently working on combining strategies across operators for managing a single threat to a task. Different threats to the same task could also be combined to better understand how implementing strategies to combat a particular threat may affect management of other threats to that task. Certainly threats interact in a way that is not yet apparent from the TSI analyses. Relatedly, it is also possible that some strategies may be helpful in mitigating multiple threats. The components of the TSI can be used to model strategic behavior of operators. For example, we can investigate how the presence of certain cues influence threat recognition and strategy selection. Such a model can be used as a resource to train expert behavior to novices. Finally, future work is needed to compare knowledge elicitation through the TSI with other methods aimed at eliciting strategies from expert operators. Behavioral observations can be used to confirm the data elicited from the TSI. Ultimately, the TSI could prove useful in creating an integrated database across operators, tasks, threats, and strategies that could be used to predict behavior of expert operators or to facilitate training of apprentice operators. Acknowledgments We would like to thank James Roberts for discussing the twodimensional Guttman problem with us. Thanks also to Kate Bleckley, Pat DeLucia, and three anonymous reviewers for comments on an earlier draft of this paper.
343
References Amalberti, R., Hourlier, S., 2011. Human error reduction strategies in health care. In: Carayon, P. (Ed.), Handbook of Human Factors and Ergonomics in Health Care and Patient Safety. Lawrence Erlbaum Associates Hillsdale, NJ, pp. 385e399. Appelbaum, S.H., 1997. Socio-technical systems theory: an intervention strategy for organizational development. Manag. Decis. 35, 452e463. http://dx.doi.org/ 10.1108/00251749710173823. Averty, P., Guittet, K., Lezaud, P., 2008. An ordered logit model of air traffic controllers' conflict risk judgment. Air Traffic Control Q. 16, 101e125. Benner, P., 1982. From novice to expert. Am. J. Nurs. 82, 402e407. Cornelissen, M., McClure, R., Salmon, P.M., Stanton, N.A., 2014. Validating the strategies analysis diagram: assessing the reliability and validity of a formative method. Appl. Ergon. 45 (6), 1484e1494. http://dx.doi.org/10.1016/j.apergo. 2014.04.010. Crandall, B., Getchell-Reiter, K., 1993. Critical decision method: a technique for eliciting concrete assessment indicators from the “intuition” of NICU nurses. Adv. Nurs. Sci. 16, 42e51. Delaney, P.F., Reder, L.M., Staszewski, J.J., Ritter, F.E., 1998. The strategy-specific nature of improvement: the power law applies by strategy within task. Psychol. Sci. 9, 1e7. http://dx.doi.org/10.1111/1467-9280.00001. Durso, F.T., Alexander, A.L., 2010. Managing workload, performance, and situation awareness in aviation systems. In: Salas, E., Maurino, D. (Eds.), Human Factors in Aviation, second ed. Academic Press, Inc, pp. 217e247. Durso, F.T., Ferguson, A.N., Kazi, S., Cunningham, C., Ryan, C., 2015. Strategic threat management: An exploration of nursing strategies in the pediatric intensive care unit. Appl. Ergon. 47, 345e354. Durso, F.T., Drews, F.A., 2010. Health care, aviation, and ecosystems: a socio-natural systems perspective. Curr. Dir. Psychol. Sci. 19, 71e75. http://dx.doi.org/10.1177/ 0963721410364728. Evans, J., St, B.T., 2000. What could and could not be a strategy in reasoning. In: Schaeken, W., Gino, D.V., D'Ydewalle, G. (Eds.), Deductive Reasoning and Strategies. Lawrence Erlbaum Associates, Mahwah, N.J, pp. 1e22. Faye, H., Rivera-Rodriguez, A.J., Karsh, B.T., Hundt, A.S., Baker, C., Carayon, P., 2010. Assessment of the medication management process. Jt. Comm. J. Qual. Patient Saf. 36, 376e384. Ferguson, A.N., Kazi, S., Durso, F.T., 2012. Revealing latent strategy structures from expert critical care nurses. In: Proceedings of the Human Factors and Ergonomics Society 56th Annual Meeting, vol. 56, pp. 801e805. http://dx.doi.org/ 10.1177/1071181312561167. Flanagan, J.C., 1954. The critical incident technique. Psychol. Bull. 51, 327e358. Retrieved from. http://www.ncbi.nlm.nih.gov/pubmed/13177800. Geiselman, R.E., Fisher, R.P., 1988. The cognitive interview: an innovative technique for questioning witness of crime. J. Police Crim. Psychol. 2 (2), 2e5. Gregg, S., Ferguson, A.N., Durso, F.T., 2014. A method for aggregating strategies across individuals: managing a lack of IV access. Proc. Hum. Factors Ergon. Soc. (in press). Gurses, A.P., Carayon, P., 2009. Exploring performance obstacles of intensive care nurses. Appl. Ergon. 40 (3), 509e518. http://dx.doi.org/10.1016/ j.apergo.2008.09.003. Gurses, A.P., Carayon, P., Wall, M., 2009. Impact of performance obstacles on intensive care nurses' workload, perceived quality and safety of care, and quality of working life. Health Serv. Res. 44 (2 Pt 1), 422e443. http://dx.doi.org/ 10.1111/j.1475-6773.2008.00934.x. Guttman, L., 1947. The Cornell technique for scale and intensity analysis. Educ. Psychol. Meas. 7, 247e279. Hassall, M.E., Sanderson, P.M., 2012. A formative approach to the strategies analysis phase of cognitive work analysis. Theor. Issues Ergon. Sci., 1e47. http:// dx.doi.org/10.1080/1463922X.2012.725781. Helmreich, R.L., Klinect, J.R., Wilhelm, J.A., 1999. Models of threat, error, and CRM in flight operations. In: Proceedings of the Tenth International Symposium on Aviation Psychology, pp. 677e682. Columbus: OH. Helson, H., 1949. Design of equipment and optimal human operation. Am. J. Psychol. 62, 473e497. Henneman, E., Gawlinski, A., Blank, F.S., Henneman, P.L., Jordan, D., McKenzie, J.B., 2010. Strategies used by critical care nurses to identify, interrupt, and correct medical errors. Am. J. Crit. Care 19 (6), 500e509. http://dx.doi.org/10.4037/ ajcc2010167. Holden, R.J., Rivera-Rodriguez, A.J., Faye, H., Scanlon, M.C., Karsh, B.-T., 2012. Automation and adaptation: nurses' problem-solving behavior following the implementation of bar-coded medication administration technology. Cogn. Technol. Work 15, 283e296. http://dx.doi.org/10.1007/s10111-012-0229-4. Hollnagel, E., Paries, J., Woods, D., Wreathall, J., 2010. Resilience Engineering in Practice: a Guidebook. Ashgate, England. Kahneman, D., Klein, G., 2009. Conditions for intuitive expertise: a failure to disagree. Am. Psychol. 64, 515e526. http://dx.doi.org/10.1037/a0016755. Klein, G. (Ed.), 1998. Sources of Power: How People Make Decisions. MIT Press, Cambridge, MA. Klein, G.A., 1989. Recognition-primed decisions. In: Rouse, W.B. (Ed.), Advances in Man-machine Systems Research, fifth ed. JAI Press, Greenwich, CT, pp. 47e92. Kleiner, B.M., 2006. Macroergonomics: analysis and design of work systems. Appl. Ergon. 37, 81e89. http://dx.doi.org/10.1016/j.apergo.2005.07.006. Kohn, L.T., Corrigan, J.M., Donaldson, M.S. (Eds.), 2000. To Err Is Human. National Academy Press, Washington, DC.
344
F.T. Durso et al. / Applied Ergonomics 47 (2015) 336e344
Koros, A., Della Rocco, P.S., Panjwani, G., Ingurgio, V., D'Arcy, J.-F., 2006. Complexity in Airport Traffic Control Towers: a Field Study. Part 2. Controller Strategies and Information Requirements. Atlantic City International Airport, NJ (p. DOT/FAA/ TCe06/22). Lintern, G., 2009. The Foundations and Pragmatics of Cognitive Work Analysis a Systematic Approach to Design of Large-scale Information Systems, first ed. Loft, S., Sanderson, P., Neal, A., Mooij, M., 2007. Modeling and predicting mental workload in en route air traffic control: critical review and broader implications. Hum. Factors 49, 376e399. http://dx.doi.org/10.1518/001872007X197017. Luwel, K., Schillemans, V., Onghena, P., Verschaffel, L., 2009. Does switching between strategies within the same task involve a cost? Br. J. Psychol. 100, 753e771. http://dx.doi.org/10.1348/000712609X402801. Martinez-Moyano, I. J., Rich, E. H., Conrad, S. H., Stewart, T. R., & Andersen, D. F. (n.d.). Integrating Judgment and Outcome Decomposition: Exploring Outcomebased Learning Dynamics (pp. 1e12). Merriam-Webster com. (n.d.). Strategy. Retrieved October 11, 2013, from http:// www.merriam-webster.com/dictionary/strategy. Merritt, A., Klinect, J., 2006. Management Origin and Development of TEM, pp. 1e20. Retrieved from. http://homepage.psy.utexas.edu/homepage/group/ helmreichlab/publications/pubfiles/TEM.Paper.12.6.06.pdf. Naikar, N., Pearce, B., 2003. Analysing activity for future systems. In: Proceedings of the Human Factors and Ergonomics Society 47th Annual Meeting, vol. 47, pp. 1928e1932. http://dx.doi.org/10.1177/154193120304701706. Nehme, C.E., Scott, S.D., Cummings, M.L., Furusho, C.Y., 2006. Generating requirements for futuristic heterogenous unmanned systems. In: Proceedings of the Human Factors and Ergonomics Society 50th Annual Meeting, vol. 50, pp. 235e239. http://dx.doi.org/10.1177/154193120605000306. Newell, B.R., Lee, M.D., 2011. The right tool for the job? Comparing an evidence accumulation model and a naïve strategy selection model of decision making. J. Behav. Decis. Mak. 24, 456e481. Reason, J., 2000. Human error: models and management. BMJ: Br. Med. J. 320, 768e770. Roth, E.M., Malsch, N., Multer, J., Coplen, M., 1999. Understanding how train dispatchers manage and control trains: a cognitive task analysis of a distributed team planning task. In: Proceedings of the Human Factors and Ergonomics Society 43rd Annual Meeting, vol. 43, pp. 218e222. http://dx.doi.org/10.1177/ 154193129904300319.
Roth, E.M., 2008. Understanding cognitive strategies for shared situation awareness across a distributed system: an example of strategies analysis. In: Bisantz, A.M., Burns, C.M. (Eds.), Applications of Cognitive Work Analysis. CRC, Boca Raton, FL, pp. 129e147. Siegler, R.S., Jenkins, E., 1989. How Children Discover New Strategies. Lawrence Erlbaum, Mahwah, NJ. Siegler, R.S., Stern, E., 1998. Conscious and unconscious strategy discoveries: a microgenetic analysis. J. Exp. Psychol. General. 127, 377e397. Retrieved from. http://www.ncbi.nlm.nih.gov/pubmed/9857493. Siegler, R.S., Shrager, J., 1984. Strategy choices in addition and subtraction: how do children know what to do? In: Sophian, C. (Ed.), The Origins of Cognitive Skills. Erlbaum, Hillsdale, NJ, pp. 229e293. Sorum, P.C., Stewart, T.R., Mullet, E., Gonzalez-Vallejo, C., Shim, J., Chasseigne, G., Grenier, B., 2002. Does choosing a treatment depend on making a diagnosis? US and French physicians' decision making about acute Otitis media. Med. Decis. Mak. 22 (5), 394e402. http://dx.doi.org/10.1177/027298902236941. Sperandio, J.C., 1971. Variation of operator strategies and regulating effects on workload. Ergonomics 14, 571e577. Staszewski, J.J., Davison, A., 2000. Mine detection training based on expert skill. In: Dubey, A.C., Harvey, J.F., Broach, J.T., Dugan, R.E. (Eds.), Proceeding of Society of Photo-optical Instrumentation Engineers 13th Annual Meeting, pp. 766e777. Thompson, V.A., 2014. What intuitions are... and are not. Psychol. Learn. Motiv. 60, 35e75. Tolman, E.C., Brunswik, E., 1935. The organism and the causal texture of the environment. Psychol. Rev. 42, 43e77. http://dx.doi.org/10.1037/h0062156. Trochim, W.M.K., 2006. Guttman Scaling. Retrieved March 10, 2012, from. http:// www.socialresearchmethods.net/kb/scalgutt.php/. Tucker, A.L., 2004. The impact of operational failures on hospital nurses and their patients. J. Operat. Manag. 22, 151e169. http://dx.doi.org/10.1016/ j.jom.2003.12.006. Vicente, K.J., 1999. Cognitive Work Analysis: toward Safe, Productive, and Healthy Computer-based Work. Lawrence Erlbaum Associates, Mahwah, N.J. Woods, D.D., 2003. Discovering how distributed cognitive systems work. In: Hollnagel, E. (Ed.), Handbook of Cognitive Task Design. Lawrence Erlbaum, Mahwah, NJ, pp. 37e53.