Cognitive Work Analysis practicality and limitations in complex control systems: a case of alarm handling

Cognitive Work Analysis practicality and limitations in complex control systems: a case of alarm handling Nastaran Dadashi* John R Wilson *&** Sarah Sharples* and David Golightly* 

*Human Factors Research Group, University Park, University of Nottingham, NG7 2RD UK (Tel: 115-9514040; e-mail: epxnd2@ Nottingham.ac.uk). **Network Rail, No 40 Melton Street, London, NW1 2EE Abstract: Cognitive Work Analysis is an approach to understand work in complex socio-technical environments. Despite the promising advantages of framing cognitive tasks within this model most of the published research is limited to first stages of CWA and developed an abstract understanding rather than tangible design recommendations. In this paper alarm handling in rail electrical control room was chosen as the case study which conducts CWA in order to understand its potential and limitations and to determine the challenges in doing so. Keywords: Cognitive Work Analysis, Cognitive System, Human –Machine interface, Alarm System, Railways 1. INTRODUCTION The aim of the work presented here is to understand utility of the overall Cognitive Work Analysis (CWA) approach in the design of control room displays, especially for rail Electrical Control Rooms (ECR). Cognitive work analysis is a conceptual and system-based approach for analysing human information interaction in highly dynamic socio-technical work places (Sanderson et al., 1999; Fidel and Pejtersen, 2004). It was initially proposed by Rasmussen et al, (1994) and subsequently modified by Vincente (1999) and Lintern (2009). Table 1 below lists the CWA stages in each of the three approaches. Table 1: Cognitive Work Analysis stages Rasmussen et al. (1994) 1- Work domain analysis 2- Activity analysis in work domain terms 3- Activity analysis in decision making terms 4- Activity analysis in terms of mental strategies 5- Analysis of work organisation 6- Analysis of system users

Vicente (1999)

Lintern (2009)

1- Work domain analysis 2- Control task analysis 3- Strategies analysis 4- Social organisation and cooperation analysis 5- Worker competencies analysis

1- Work domain analysis 2- Work organisation analysis 3- Cognitive transformation analysis 4- Strategies analysis 5- Cognitive processing analysis 6- Social transaction analysis

The stages of CWA (Rasmussen et al., 1994; Vincente 1999; Lintern, 2009) are mainly to analyse the structure of work domain and work tasks, intitally through the form of meansend hierarchy. This leads to an analysis of cognitive processes (Reising and Sanderson, 2002) and strategies.

Sanderson et al. (1999) reviewed the potential of CWA to inform all stages of system life-cycle including: requirements, specification, design, simulation, evaluation, implementation, operator training, and maintenance. CWA is used to structure common observational methods which provide useful information about human information behaviour, often in textual narratives. Nirula and Woodruff (2006) considered CWA as an “integral precursor to any design iteration”. CWA have been applied in systems designs but most of the works are focused on initial stages of CWA framework such as Work Domain Analysis (WDA) and Abstraction Hierarchy (AH) (Groppe et al., 2009; Reising and Sanderson 2002, Janzen and Vincente 1998; Golightly et al., 2010). For example, Groppe et al., (2009) assessed pilots’ operational information requirements during airport collaborative decision making using work domain analysis. Reising and Sanderson (2002) formalised an ecological user interface design approach with an abstraction hierarchy. Janzen and Vincente (1998) applied abstraction hierarchy to quantify human attention allocation within various levels of the hierarchy in a thermal-hydraulic process simulation. Nirula and Woodruff (2006) conducted a full CWA to understand design implications of ubiquitous computing in schools. In order to fulfil the requirements of the stages they started with an observational study followed by interviews and focused field observations which gave an indication of the activities in the environment. Talk-aloud with end users after they used the interfaces together with the results obtained from observational study were analysed to capture the strategies adapted by users. In this paper, we present alarm handling in rail electrical control room as a case study to explore the CWA approach and the extent of its benefit in guiding design of such

complex systems. This is part of a larger programme of work, funded by Network Rail, to understand methods and guidance relevant to the design of new display technologies for controlling rail infrastructure. Rail electrical control room alarms are events configured in the system requiring operator’s attention due to any form of abnormality in the rail network’s electrical supply system (e.g. through overhead wires or third rail). It is announced by sounding an audible alarm, updating any related symbols on an alarm banner as well as live indications on the SCADA (Supervisory Control and Data Acquisition) which is responsible for remotely controlling the electrical supplies through remote monitoring equipments and visual displays (Dadashi et al, 2009). Previous research in alarm handling has studied response times (Stanton, 2006), direction of attention (Woods, 1995;Gilson, et al., 2001), modelling operators’ diagnostic procedures (Woods, 1995; Stanton and Baber, 1997), information load (Woods, et al., 2002); and informativeness of alarms (Seagull, et al., 2000). The dynamic nature of alarm handling emphasises the need to understand the cognitive and collaborative work involved. Therefore CWA seems to provide an in-depth investigation of cognitive processes performed while handling alarms in such socio-technical environment. 2. METHODS OF DATA COLLECTION The rail electrical control room in Lewisham near London was chosen for the purpose of this study. Two rail electrical control room operators were responsible for providing electricity supplies to rail tracks and ensure railway safety for trains and maintenance. During the 18 hours of observation total of 22 alarms occurred and had been analysed. Six electrical control room operators were participated in this study. Field study techniques were adopted to explore the workplace and work systems. Pre-study briefing visits were conducted in order to familiarise the researcher (first author ND) with the control room and tasks as well as to brief the electrical control room operators about the study. This was followed by four visits of 4.5 hours each. These visits focused on alarm handling tasks only and were designed to explore the work domain and the functions associated with it (WDA).

Critical Decision Method (CDM) technique (O’Hare et al., 1998) were used to associate each of the tasks with strategies. 3. ANALYSIS OF ALARM HANDLING STRATEGIES Five stages of CWA as adapted by Lintern (2009) (Table 1) were used for the purpose of this study: work domain analysis (abstraction hierarchy), organisation of work (contextual-activity matrix), and cognitive transformation analysis (decision ladder) were developed to provide an initial understanding of the domain. Cognitive strategies and processing modes were also identified to develop an understanding of information processing and some insights on information presentation and formats. The sixth stage of CWA, social transaction analysis has not been considered in this paper. 3.1 Work domain analysis Artefacts utilised by operators during alarm handling are: SCADA, log book, phone, map, regulations and isolation requests. Moreover the operator might consult with his/her colleague within the control room or call someone on-site to obtain more information. Use of these artefacts enables the operator to achieve an understanding of system status as well as the optimum plan he/she ought to develop and follow. Alarm handling can be broken down into two main tasks: alarm recognition and alarm clearance and the ultimate purpose of to ensure safety and efficiency for the work and optimise the use of resources within railway while providing rail electrical supply. Figure 1 shows an AH of alarm handling in rail ECR. It is assumed that human reasoning is based on navigation through the levels of the abstraction hierarchy (Lintern 2009). Each node in the diagram points to the information required and hence has to be presented within the environment. It also feeds into the second stage of cognitive work analysis: Work Organisation Analysis.

The second stage - work organisation analysis (contextual activity matrix) - required more detailed observation in order to achieve the level of detail required. Videos of episodes of alarm handling cases were recorded. These videos captured operators’ interactions with either the interface or other people, the sequence of these interactions has been recorded and was confirmed through retrospective talk aloud. A checklist was developed to organise task allocations (Dadashi et al., 2009), alarm handling durations and information flow which informed the third stage of CWA (states and processes). Knowing the states and processes associated with alarm handling is not sufficient to identify the strategies applied by operators in the real life work environment. Talk aloud and retrospective analysis of video recordings of all of the 22 alarm handling episodes within categories derived from the

Figure 1: Abstraction hierarchy

Figure 2: Alarm handling contextual activity matrix

Figure 4: Alarm handling activities and strategies

3.2 Work organisation analysis Work functions identified in AH: alarm recognition and alarm clearance were explored in more detail in this stage. The product of this stage is a contextual activity matrix, which associates the activity-independent domain functions from the Abstraction-Hierarchy with the activity-dependent work tasks, and relates work tasks with work situations. The work tasks related to each of these functions are defined through their corresponding activities: Identifying alarm priority; locating the alarm; information requirement assessment; information collection; assessment of the alarm; making appropriate changes and finalising the alarm. Work situations in this study are defined as the state in which operators’ conceptual level of work changes. The work situation in which these tasks might be conducted can be categorised as different areas of the work setting to which the operator is attending including overview display, main display, alarm banner and functions on the menu on the visual display and while he is talking with someone over the phone or going through hard copy sheets as well as talking to their colleagues within the ECR (Figure 2). 3.3 Cognitive transformation analysis Cognitive transformation analysis aims to identify the cognitive activity associated with alarm handling work tasks. The cognitive states and cognitive processes of these work tasks are identified and presented in state process diagrams (Figure 3). Ovals present cognitive states and arrows shows cognitive processes which guide the transitions between cognitive states. The presented example demonstrates some of the cognitive states and processes associated with the functional task of ‘information requirement assessment’. The type of alarm is being recognised (cognitive state) earlier and the search is being performed (cognitive process) to evaluate (cognitive process) the level of information available (Figure 3).

attention, interpretation, evalution and decision-making, planning and action. For example, the states and processes shown in Figure 3 conform to the evaluation stages of the Decision Ladder. The Decision Ladder also represents shortcuts through cognitive processing, therefore supporting the expression of kind of cognitive activity that is typical of expert performance. 3.4 Strategies analysis Strategies were identified through retrospective analysis of video recordings and structured using an adaptation of Critical Decision Method and observational checklists. Every one of the 22 alarm handling cases was broken down into a set of immediate goals, cues which led into selection of those goals, level of expectancy in receiving those cues and the uncertainty associated with them .The same pattern of goals and cues observed led into realisation of key alarm handling activities: Notification, acceptance, analysis and clearance. Moreover other information classified within the framework enables a basic understanding of the type of strategies operators use to achieve those goals .Figure 4 shows the alarm initiated activities together with the strategies found to support those activities (marked in bold).. Notification: Several information sources notify operators of the existence of an alarm. These include the flashing alarm banner, colour codes, acronyms of alarm type and location, sirens, phone calls, flashing circle around the location on the overview display, etc. Operators have to categorise and filter these sources to achieve a basic understanding of the alarm. Acceptance: The situation awareness process that we assume is built in the previous stage (notification) is the basis on which the operators almost immediately accepts the alarm and silences the siren without informing an authorised person. In the rare cases of an alarm where immediate on-site action is required, operators use their local knowledge (of the track, the electrical equipment, the work which might be taking place out there and the train service running) and their experience of previous similar cases in order to assess the criticality. The strategy in this stage is mostly similarity matching which is highly related to operators’ experience. Usually this stage is tightly coupled with the analysis and assessment of the alarm. Analysis:

Figure 3: state-process diagram for the work task of information requirement assessment Ultimately, state-process diagrams of all possible work tasks within the domain should conform to the Decision Ladder (Rasmussen et al, 1994; Lintern, 2009). The Decision Ladder is a template that frames potential cognitive state and process within a standardised model of cognition comprising

Information presented to the operator is being used by them for the purpose of assessing and evaluating the underlying meaning and causes of alarms. Operators usually analyse alarms by stretching the existing evidence to match them with similar cases (extrapolation). Unlike similarity matching where all of the evidence is matched with a similar previous alarm, here the operators have to use his/her imagination to fill the gaps until they become similar to one.

Clearance: The operator identifies possible courses of actions, evaluates them and executes the optimum action to clear the alarm. The operator remembers similar cases and tries to match the stretched evidence to other potential causes and trials the corrective actions of those cases. Hence while clearing an alarm; similarity matching is being used. 3.5 Cognitive competency analysis Cognitive competency analysis aims to identify the operator’s processing mode while attending to work tasks. This stage provides designers with the information that operators will mostly require based on the cognitive processing modes they are in. Cognitive processing mode is categorised into rulebased, skill-based and knowledge-based behaviour (Rasmussen et al., 1994). Each of these modes can benefit from certain formats of information (Lintern, 2009). In skillbased mode, operators require information in the form of space-time patterns in order to optimise the direct manipulation of data. In Rule-based mode, information should be in consistent words and symbols easily conceived by operators, and in knowledge based mode information should be in the form of semantics which supports meaningful assembly of information. The finding of this stage of CWA can provide design implications in order to assist alarm handling. In other words, by having the contextual activity matrix (Figure 2) we can determine what area of interface is mainly being used while conducting any of the alarm initiated activities. Knowing the optimum information format for each of those activities will provide the designer with the optimum information format for each of the areas on the interface (Figure 5). For example, in order to facilitate optimum alarm recognition information presented on the alarm banner and operational display should be in words and symbols. Along with the information format to support categorising, it is necessary to identify the type information and the hierarchy of its presentation on the display

Figure 5: Design implications of processing modes

Categorising, similarity matching and filtering are rule-based processing modes since the operator has to apply a set of instructions. In this level the operator has to see the pattern but that does not necessarily lead into a recognised and clear set of actions as, after identifying the pattern the operator has to set an action plan. Queuing is skill-based processing since there is no conscious sequence of actions and it depends mainly on operator’s expertise where they need to recognise the pattern automatically without considering other options. Extrapolation, which refers to stretching the evidence, is knowledge-based processing since the operator has to infer the need for such strategy. Similar to the skill-based processing mode the pattern is recognised explicitly, but as the action plan is not an automatic procedure, the operator has to reason and infer the best action plan. 4. DISCUSSION This paper is part of series of studies conducted in rail ECR. Dadashi et al., (2009) explored alarm handling artefacts and the effect of information load on the alarm handling durations; other current work underway to understand strategies in more detail. The focus of this paper was to analyse applying five stages of CWA as an approach. While some of the products of this analysis have been presented in this paper, a number of considerations have come to light that may influence the success of applying the CWA approach. 4.1 Identify the aims of analysis Different stages of CWA aimed to explore complexity of socio-technical environments in various aspects. It is important to use the right stage for the right purpose and, as discussed in the introduction, it is not always necessary to perform all stages of CWA. If the requirement of the case presented here was to only explore the domain and provide an initial understanding of ECR alarms, then the first few stages would have sufficed and given additional value in comparison to traditional task analysis methods (Golightly et al., 2010). Work organisational analysis provides information about the most critical interactional aspects in ECR while handling alarms and these areas can be targeted for alterations in order to optimise the alarm handling. Cognitive transformation analysis indentifies the cognitive process and operator’s state of awareness corresponding to that process. While it does not explain how the operator reaches those cognitive states in depth without considerable effort on the part of the investigator, even a basic application of the approach can highlight the cognitive nature of work which can be used for developing appropriate cognitive processing methods. For example alarm handling is more a problem solving and decision making, hence CDM technique was utilised to elicit the knowledge regarding operator’s decision making. Strategies analysis identifies the ‘how’ for operational processes which lead into the ‘what’ and guide information presentation on the visual display. Cognitive processing analysis explore the level of competency required, and this level can guide some of the design characteristics of the display (Lintern, 2009) as well as contributing to operator training.

4.2 Deploy rigorous methods to frame the stages CWA is a framework rather than a method; therefore few guidelines about what methods should be deployed. The choice of appropriate methods to capture data is therefore crucial, and the analysis involved can be time-intensive. Within our ECR case a total of 20 hours of observation was conducted together with SME interviews to provide an appropriate level of understanding. It is also important to choose the appropriate methods to elicit data from observations. In this case video recording analysis, checklists, and CMD framework were used to structure the findings. Finally, and similar to all research in live, socio-technical environments, the success of the analysis is very much dependent on the experience of the researcher both in the domain he/she is investigating as well as good understanding generally of the principles of human information processing. 4.3 Break the complex control system into smaller domains CWA can potentially explore features of complex control system. It seems, however, unrealistic to conduct all of the stages for every function possible. This paper has tackled only one of the domain functions of ECR operation (alarm handling) as it might have been impossible or of limited use to apply CWA to elaborate every ECR functions. This might explain the lack of full implementation of CWA in literature. The risk, however, is that by focussing on only one aspect of the system, critical interdependencies between people, technology and functions within the overall system will be lost. Therefore, it may be valuable to conduct a WDA of the control system as a whole, in order to determine these interdependencies and also to prioritise the functions which require more analysis. 5. CONCLUSION This paper conducted five stages of CWA as described by Lintern(2009). Various methods such as field study, video recording analysis, CDM technique, observational checklist, talk aloud and semi-structured interviews were applied to structure the cognitive behaviour of operators while alarm handling in rail ECR. The findings suggest that CWA has value in this case in supporting an understanding of those domain features, and subsequent cognitive implications, which can inform design. However, certain methodological factors need to be taken into account to apply CWA in an effective manner. REFERENCES Dadashi, N., Wilson, J. R., Sharples, S.(2009). Cognitive system engineering in rail: a case study in electronic control rooms. ECCE2009. Espo-Finland. Fidel, R., & Pejtersen, A. M. (2004). From infromation behaviour research to the design of infromation systems: the Cognitive Work Analysis framework. IR infrormation research , 10. Gilson, R. D., Mouloua, M., Graft, A. S., & McDonald, D. P. (2001). Behavioral Influences of Proximal Alarms. Human Factors , 43 (4), 595-610.

Golightly, D., Ryan, B.& Sharples,S. (2010). Cognitive work analysis of signalling protection for rail engineering. Contemporary Ergonomics(in press). Groppe, M., Pagliari, R., & Harris, D. (2009). Applying Cognitive Work Analysis to Study Airport Collaborative Decision Making Design . ENRI International Workshop on ATM/CNS(EIWAC2009). Janzen, M. E., & Vincente, K. J. (1998). Attention allocation within the abstraction heirarchy. International Journal of Human-Computer Studies , 521-545. Lintern, G. (2009). The foundations & pragmatics of Cognitive Work Analysis: A Systematic Approach to Design of Large-Scale Information Systems. Onlinewww.cognitivesystemsdesign.net . Naikar, N. (2006). Beyond interface design:Further applications of cognitive work analysis. International Journal of Industrial Ergonomics , 423-438. Nirula, L., & Woodruf, E. (2006). Cognitive work analysis and design research: designing for mobile humantechnology interaction within elementary classrooms. Fourth IEEE International workshop on wireless, mobile and ubiquitous technology in education (ICHIT'06). O'Hare, D., Wiggins, M., Williams, A., & Wong, W. (1998). Cognitive task analyses for decision centred design and. Ergonomics , 1698-1718. Rasmussen, J., Pejtersen, A. M., & Goodstein, L. P. (1994). Cognitive systems engineering . New York: John Wiley & Sons, Inc. Reising, D. V., & Sanderson, P. E. (2002). Work domain analysis and sensors I:principles and simple example. International journal of Human-Computer Studies , 569596. Sanderson, P., Naikar, N., Lintern, G., & Goss, S. (1999). Use of cognitive work analysis across the system life cycle: from requirements to decommissioning.43rd Annual Meeting of the Human Factors and Ergonomics Society -1999. Stanton, N. A., & Baber, C. (1997). Comparing speech versus text displays for alarm handling. Ergonomics , 12401254. Stanton, N. A., & Baber, C. (2006). Editorial, The ergonomics of command and control. Ergonomics , 49 (12&13), 1131-1138. Vicente, K. J. (1999). Cognitive work analysis. NJ: Lawrence Erlbaum Associates Mahwah. Woods, D. D. (1995). The Alarm Problem and Directed Attention in Dynamic Fault Management. Ergonomics , 2371-2393. Woods, D. D., Patterson, E. S., & Roth, E. M. (2002). Can We Ever Escape from Data Overload? A Cognitive Systems Diagnosis. Cognition, Technology & Work , 22-36.