A sensemaking perspective on framing the mental picture of air traffic controllers

Applied Ergonomics 44 (2013) 327e339

Contents lists available at SciVerse ScienceDirect

Applied Ergonomics journal homepage: www.elsevier.com/locate/apergo

A sensemaking perspective on framing the mental picture of air traffic controllers Stathis Malakis a, *, Tom Kontogiannis b a b

Hellenic Civil Aviation Authority, Rhodes/Diagoras International Airport ATC Unit, Rhodes, Greece Technical University of Crete, Department of Production Engineering & Management Chania, Greece

a r t i c l e i n f o

a b s t r a c t

Article history: Received 6 January 2011 Accepted 4 September 2012

It has long been recognized that controller strategies are based on a ‘mental picture’ or representation of traffic situations. Earlier studies indicated that controllers tend to maintain a selective representation of traffic flows based on a few salient traffic features that point out to interesting events (e.g., potential conflicts). A field study is presented in this paper that examines salient features or ‘knowledge variables’ that constitute the building blocks of controller mental pictures. Verbal reports from participants, a field experiment and observations of real-life scenarios provided insights into the cognitive processes that shape and reframe the mental pictures of controllers. Several cognitive processes (i.e., problem detection, elaboration, reframing and replanning) have been explored within a particular framework of sensemaking stemming from the data/frame theory (Klein et al., 2007). Cognitive maps, representing standard and non-standard air traffic flows, emerged as an explanatory framework for making sense of traffic patterns and for reframing mental pictures. The data/frame theory proved to be a useful theoretical tool for investigating complex cognitive phenomena. The findings of the study have implications for the design of training curricula and decision support systems in air traffic control systems. Ó 2012 Elsevier Ltd and The Ergonomics Society. All rights reserved.

Keywords: Mental pictures Cognitive maps Sensemaking Data/frame model Air Traffic Control

1. Introduction As information technology has increasingly expanded in organizations, practitioners found themselves at the receiving end of an endless flow of information and data streams. On the other hand, the ability of practitioners to attend to and interpret data streams, extract meaningful relationships and finally place the data into the right context has increased much more slowly than was anticipated (Woods et al., 2002). The rapid expansion of information technology has increased the amount of information presented to practitioners without any assistance on how to make sense of the situation or how to anticipate future trends of the situation. In the case of Air Traffic Control (ATC), we are dealing with a complex and dynamic environment that requires practitioners to attend to multiple events, anticipate aircraft conflicts and comprehend or make sense of evolving scenarios. It has long been recognized that effective controller strategies should be based on a ‘mental picture’ or representation of the current and future traffic situations (Falzon, 1982; Whitfield and Jackson, 1982). Several studies in ATC have explored the content and structure of controller mental pictures. Using verbal reports and memory recall techniques, it was shown that controllers recalled aircraft

* Corresponding author. E-mail address: [email protected] (S. Malakis).

position and flight direction most reliably (Bisseret, 1971; Mogfort, 1997; Niessen et al., 1997). The mental picture is dependant upon earlier crew-controller interactions so that aircraft perceived as a factor, or being in radio-contact, may figure more prominently and get recalled with more parameters (Niessen et al., 1997; RoskeHofstrand and Murphy, 1998). To cope with high workload, controllers also tend to structure their mental picture by grouping aircraft into meaningful units. In this way, they manage to reduce the number of aircraft within working memory (Bainbridge, 1975). Several ‘knowledge variables’ have been utilised by controllers (e.g., proximity, vertical movement, weather information) to group aircraft into (a) those requiring continuous monitoring to avoid conflicts and (b) those safely separated at a particular moment (Amaldi and Leroux, 1995; Niessen et al., 1997). In summary, earlier studies have indicated that controllers tend to maintain a selective mental representation of traffic flow based on a few salient features that point out to situations related to a risk of conflict or situations requiring constant attention in the near future. The concept of ‘situation awareness’ (Endsley, 1995) provides a framework for studying mental representations of complex and time-constrained situations. Situation awareness characterizes a state of knowledge about a situation and defines a measure for the capacity of practitioners to cope with the situation. However, situation awareness does not model how the mental picture of the situation changes as a result of changing events in the traffic environment. The current study addresses the cognitive processes

0003-6870/$ e see front matter Ó 2012 Elsevier Ltd and The Ergonomics Society. All rights reserved. http://dx.doi.org/10.1016/j.apergo.2012.09.003

328

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

and their interactions with the environment that lead to the development (i.e. framing) and reframing of the mental picture of a situation. Other studies in modelling the processes that affect a mental picture have taken an information-processing approach (Niessen et al., 1997; Oprins et al., 2006) that focused on how controllers generate inferences on data elements. These studies have not been able to address how controllers match data with a mental picture. For instance, a common confusion may arise when data appear relevant but they do not fit with a well-framed relationship with other data. The present study proposes to conceptualize mental pictures on the basis of ‘cognitive maps’, a concept developed within the cyclic theory of perception (Neisser, 1976) and sensemaking (Weick, 1995; Klein et al., 2007). Mental pictures or cognitive maps are information-seeking structures within a perceptual cognition cycle of exploration, sampling and modification (Neisser, 1976). In this thought cycle, mental pictures guide an active search for what may be considered as important data as well as direct exploration of possible controller actions that change the environment. This enactment process may also contribute to a better understanding of the original assessment of the situation (e.g., elaboration or reframing of the situation) and result in further modifications of earlier actions. In this conceptualization, mental pictures can be described as cognitive maps of reality as perceived by controllers. A similar approach has been taken in the field of management science where ‘network pictures’ have been seen as cognitive maps depicting the business world so that managers understand where they are and where they can go (Fiol and Huff, 1992). Like mental pictures, network pictures are not based on complete information but on data samples which are obtained by the practitioner interaction with the environment. They guide the assimilation of new information and allow inferences to be drawn by inductive reasoning (Fiske and Taylor, 1984). They enable practitioners to build imaginary connections between events, objects and situations in their environments so that these become meaningful to them and direct future action (Henneberg et al., 2006; Colville and Pye, 2010). It appears that the framework of the perceptione cognition cycle and sensemaking provides a basis for converging research findings from different domains including air traffic control, command and control, and management science. The process of sensemaking was introduced by Weick (1995) as one of the most important cognitive functions in natural settings, especially in safety critical organizations. Sensemaking begins with the realization of an inadequacy in fitting an ongoing stream of events into a meaningful context. Sensemaking is triggered as a response to a situational surprise and a failure of expectations. Prior understanding is put in doubt and further attempts are made to integrate data into a better understanding of the situation. Sensemaking allows practitioners to understand how current accounts of the situation came about and to anticipate future evolutions through a process of fitting data into a frame or cognitive map (Crandall et al., 2006). The purpose of this study is to investigate aspects of sensemaking in the ATC environment. The data/frame theory (Sieck et al., 2004; Klein et al., 2006, 2007) provides a theoretical underpinning for looking into sensemaking in the domain of terminal approach radar control. The data/frame theory claims that situational elements can be explained when they are adequately fitted into an explanatory structure or frame (see Fig. 1). Quite similarly to a closed feedback loop, the data are used to identify the frame whilst the frame determines what data must be attended to. The explanatory frame defines relationships between data that may be spatial (e.g., maps), causal (e.g., stories), temporal (e.g., plan) and functional (e.g. scripts). In the context of ATC, a cognitive map or mental picture may include both spatial relationships (e.g., aircraft

trajectories) and abstract data referring to weather conditions, aircraft performance, types of flights and so forth. However, the data/frame model is an abstraction and requires further field applications to provide a basis for pragmatic interventions (Klein et al., 2010). The present study is undertaken in order to address two issues regarding the content of mental pictures and the cognitive processes underlying the framing of mental pictures. Several cognitive processes are examined from the perspective of sensemaking such as sizing-up situations, questioning understanding, discovering inconsistencies, reframing problems, and finally constructing new actions. These aspects of sensemaking have their own dynamics and are influenced by the broader organizational environment, the demands and affordances of the situation, and finally the stance of practitioners. Sensemaking has implications for the design of training curricula and decision support systems that are very important in the context of major system interventions (e.g., the Single European Sky Air Traffic Management Research (SESAR) and Next Generation Air Transportation System (NextGen). These projects will make significant changes to the delegation of authority between pilots and controllers, or between humans and automated agents, which requires further research on how practitioners can make sense of complex situations in a distributed environment. This paper is structured as follows. Section 2 presents the task roles and responsibilities of the approach controllers in our research setting. Section 3 describes the methods used to collect data in a field experiment and the experience of the participating controllers. The results are presented in Section 4.1 that shows the ranking of a set of ‘knowledge variables’ in their contribution to the construction of the mental picture as well as Section 4.2 that focuses on the performance criteria that control the flow and transition of sensemaking processes (Fig. 1). Finally, an exploration of the cognitive processes and the criteria of framing/reframing of mental pictures is presented in Section 5 by making reference to the scenarios used in the experiment. 2. Research setting Air traffic controllers are responsible for the safe, expeditious and orderly flow of the air traffic as defined by the overarching regulatory authority, International Civil Aviation Organization (ICAO, 2007). Three levels of ATC operations can be distinguished that correspond to the major phases of flight, that is, (i) aerodrome control of air traffic in the vicinity of an aerodrome and ground movement of aircraft, (ii) approach control of arriving and departing aircraft, and (iii) area control for handling the en-route phase from a central ATC unit. Approach controllers manage traffic in aerodromes by issuing clearances, instructions and information to aircraft under their jurisdiction. Their Controllers Working Positions (CWPs) comprise a range of standard voice and data Input/Output devices (i.e. keyboards, radar displays, mouse, microphones, telephones) and special software that enable then to perform conflict detection and resolution (i.e. Automation Safety Nets). Our research setting was a medium level European airport with seasonal traffic. In low-tempo operations, work-shifts comprised two controllers in the Tower and the Approach units. In medium traffic, shifts comprised one controller in the Tower unit and another two in the Approach unit (i.e., the executive and planning controllers). The executive controller was responsible for direct control of aircraft in the terminal area and for the implementation of the overall plan whilst the planner was responsible for establishing the overall plan for managing the traffic and assisting the executive. In high tempo operations, shifts comprised four to five

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

329

Frame

Data

Manage attention,

Recognize / Construct a frame

define, connect and filter the data

Elaboration

Reframing

Cycle

Elaborate a frame

Cycle

Questioning a frame

Re -Frame

Comparing Frames

Creating new frame Add & Fill Slots

Track Anomalies

Seek & Infer Data

Detect Inconsistencies

Discover new data/new

Judge Plausibility

relationships

Gauge data quality

Discard data

Preserve Fig. 1. Graphical depiction of the data-frame model (Klein et al., 2006).

controllers split in the Tower and Approach units. Controllers were shifting between the Tower and the Approach units on different days to retain operational competency in both areas. The ATC department had obtained and utilized an ATC simulator for training purposes. Our research focused on the Approach unit that was responsible for terminal control around the airport and for the spacing of traffic on the final approach for landing. Arriving aircraft are transferred from the Area Control Centre (ACC) via Approach to the Tower control and departing aircraft are transferred from Tower control via Approach to the ACC. The primary task of an approach radar controller is to detect and resolve potential separation losses while guiding inbound aircraft to be positioned for landing and outbound aircraft to join the airways of their flight plan. 3. Methods Initially, a documentation analysis was performed in order to become familiar with the procedures, control position responsibilities and systems used by the controllers. Documentation analysis is the process of coming up to speed with the research context and becomes part of the preparation phase although in reality

occurs throughout the entire field research as any input supplements the procedure (Crandall et al., 2006). The aim of the present study is twofold. a Explore the ‘knowledge variables’ that controllers use to build their mental pictures. b Address several cognitive processes and their interactions with the environment in the re-framing of mental pictures; to this end, the data/frame theory of sensemaking has been used as a research framework. The main work concerns a field experiment designed to explore the extent that cognitive maps or mental pictures provide an explanatory structure in the data/frame theory as they may represent several spatial, temporal and functional relationships in a frame. Cognitive maps can be seen as sketch maps of air traffic flows superimposed on the Radar Vectoring Area (RVA) depicting not only lines and curves that delineate the lateral projections of traffic planning but also critical annotations about heading and level changes, approach sequence indicators and other annotation relevant to traffic handling. For example, an air traffic controller

330

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

may sketch the lateral path of an arriving aircraft until its landing, annotate the points where he/she anticipates to provide turning (i.e. vectoring) instructions, write down those headings and the associated level changes and finally mark the approach sequence of the aircraft. Its equivalent in the command and control domain may be seen as depicting the movements of combat elements on a map and updating critical annotations regarding direction and other pertinent information. For this purpose, ten variables have been selected to examine whether they could function as ‘knowledge variables’ for the construction of mental pictures (see Section 4.1 for the listing of variables). The selection of variables was based on a pilot study that considered a large number of variables of interest in the construction of frames. However, some variables were difficult to define in meaningful operational terms and were not included in the final set. For example, some familiar traffic patterns were based on similar flight plans, airways, cruising levels, scheduled arrival times and aircraft types that were difficult to define in meaningful terms. In general, the role of the standardised traffic patterns in the sensemaking process was documented from observations of real traffic handling. To assess how controllers use the ‘knowledge variables’ to construct their cognitive maps, an experiment was carried out which presented controllers with ten traffic scenarios resembling real operational situations. After the experiment, the controllers were asked to fill in a questionnaire listing the ten ‘knowledge variables’ and rate their importance in resolving the traffic conflicts. The second objective of the study was to examine the cognitive maps of controllers and their criteria for making sense of the situation. For this purpose, controllers were asked to draw cognitive maps and freely annotate what they thought to be significant in handling the traffic scenarios. In the drawings of cognitive maps, the planning of traffic was depicted as heading changes (in the form of degrees of direction) and sequences of approach of inbound aircraft (see examples in Section 4.2). Following the experiment, controllers were asked to rank the importance of ten variables in the construction of their cognitive maps for each scenario. Controllers were requested to do the ranking of the variables using a Likert type 5-point scale (i.e., 1 - insignificant, 3 - medium significance and 5 - highly significant). Verbal reports from the participants were used to gain insights into the cognitive processes of building and reframing mental pictures supplemented by observations of real life scenarios. Observational data during shifts were also combined with qualitative data from handover procedures, focused interviews with Onthe-Job-Training (OJT) instructors and finally, analysis of training curricula. From a Cognitive Systems Engineering (CSE) perspective, these research techniques belong to the ‘natural history’ family of methods, which are based on a diverse collection of observations in situ (Hoffman and Militello, 2009). In this case, the shaping of conditions is minimal and a diverse amount of situations is observed and documented (Woods and Hollnagel, 2006). Controllers use several criteria in order to build a mental picture of the traffic situation, question their picture and reframe it to achieve an efficient traffic pattern. Our interviews with the OJT instructors indicated that criteria for framing e reframing mental pictures could be generally assigned to eight categories as shown in Table 1. Our motivation was to evaluate the significance of these criteria by correlating them to an overall measure of scenario efficiency that rated controller performance in a scale from 1 to 5. Eleven (N ¼ 11) operational controllers holding Terminal Approach Radar Control ratings participated in the study. Three controllers had more than 10 years operational experience and eight had less than 10 years (Mean ¼ 9.09 years, SD ¼ 6.12 years). All controllers were holding Tower and Approach Procedural (non-

Table 1 Criteria used to frame and reframe mental pictures. #

Performance criteria

Brief description

1

Avoids potential conflicts

2

Creates open & inspectable patterns

3

Provides more options to crews

4

Minimizes chances for go-around

5

Makes subtle changes to flight paths

6

Takes into account terrain features

7

Takes into account crews preferences

8

Avoids stormy/turbulent zones

Creates patterns that does not give rise to potential conflicts; this is the basic conflict resolution task. Creates an open traffic flow pattern that minimizes the need to monitor horizontal separation distances; this creates a slack that allows controllers to engage with additional tasks. Provides more options to crews to adapt to unexpected situations (changes in weather, turbulence, etc). In this way, crews are given more opportunities to detect problems and recover from them. Minimizes chances of go-around by placing the aircraft at the correct altitude, speed and distance on the final approach for landing. But should the need arise; the go-around procedure does not upset the overall planning too much. Performs small-scale changes so that aircraft are not far from the standard instrument approaches and departures (IAPs & SIDs) or standard routings; this makes it easier to bring flights back to original paths should any need arise (e.g. radar failure, communication failure). Takes into account terrain factors and obstacles (i.e., mountains, obstacles on airport, etc that may reduce acceptability in terms of safety and quality of flight). Takes into account the preferences of crews (i.e. near continuous descent profiles, optimal speed profiles, preferred rates of climb /descent, direct routings). Avoids routing aircraft through or very near to stormy zones (e.g., significant weather) or turbulence zones that can make pilots and passengers feel uncomfortable.

radar) ratings and three of them were OJT instructors and examiners. The scenarios were designed with the aid of the training simulator and comprised graphical depictions of traffic situations, weather data, and lists of departing/arriving aircraft, unit related coordination data and information about the operational status of Communication Navigation and Surveillance (CNS) systems that were supporting the airport. The ten scenarios were divided equally between those using the precision approach runway of the airport which was served by an ILS (Instrument Landing System) and those using a second runway with no ILS services which was more demanding in managing traffic. The utilization of the instrument approach runway was approximately 95% per year and it was considered as the preferred runway for operations. The precision runway was used for scenarios 1 to 5 while the second one for scenarios 6 to 10. For each runway, the traffic load (arriving and departing aircraft) was progressively increased from 1 to 5 arrivals while the departing aircraft were held constant at 2 departures (Table 2). With an ATC declared capacity of 22 flights per hour (10 arrival and 12 departures), the scenarios were representative of approximately 15 min of traffic management. For all scenarios, the paths of departing aircraft paths were conflicting with those of arriving aircraft which required the intervention of controllers. In addition, all inbound aircraft were in the form of a wave arriving at the same time at the airport which required controllers to establish a valid approach sequence; an exception was the first scenario for each

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

331

Table 2 Essential characteristics of the scenarios in the experiment. Runway scenario

Precision runway

Non - precision runway

1 2 3 4 5 6 7 8 9 10

Arriving aircrafts

Departing aircrafts

Total number of aircrafts

Approach sequence permutations

Weather data significance

CNS systems functionality status

1 2 3 4 5 1 2 3 4 5

2 2 2 2 2 2 2 2 2 2

3 4 5 6 7 3 4 5 6 7

1 2 6 24 120 1 2 6 24 120

Insignificant Insignificant Insignificant Significant Highly Significant Insignificant Insignificant Insignificant Significant Highly Significant

All systems operational

runway where there was only one arriving aircraft. According to ICAO (2007) and local operating manuals, the approach sequence shall be established in a manner that facilitates maximum arrival of aircraft with the least possible delay. Scenarios where the approach sequence was already established by the previous unit were not examined as they were treated as variations of the elementary scenario of only one arrival. Approach sequence permutations can be defined as the set of all possible approach sequences of a given number of inbound aircraft when they all arrive at the same time at the airport; for instance, 4 arrivals generate 24, (4! ¼ 24) possible approach sequence permutations. The weather was not a factor in planning and management of the air traffic for the first three scenarios of each runway but it was a factor for the fourth and the fifth scenarios. 4. Results Results are divided into two subsections. The first one provides an analysis of the knowledge variable while the second one attempts to operationalize the sensemaking processes in terms of a set of performance criteria.

All systems operational

as a minimum and preferably higher whilst a rough criterion for communalities would be .70 (Field, 2005). In the second step, the pattern correlation tables were calculated to examine whether the number of factors and the loadings of variables on them would conform to what was theoretically expected. In the third step, the reliability of the results was tested using Cronbach’s alphas for each factor; a criterion of .60 is common in exploratory research regarding cognitive tests although a value of .70 or higher is more reliable (Kline, 1999). The oblique rotation e which is considered more appropriate for naturalistic data (Field, 2005) e revealed a four factor structure (Table 4). Using the KMO criterion, the fourfactor solution appeared statistically stable and accounted for the 83.33% of the total variance (Factor-1: 53.48%, Factor-2: 13.08%, Factor-3: 10.76%, Factor-4: 7.01%). Finally, separate reliability analyses for all factors were completed and all Cronbach’s alphas were above .70. Factor 1 - (RVA Structural elements & standardized operations): Factor-1 corresponds to the “static” features of controller mental picture that are stable in time and represent well-documented and easily retrievable knowledge. For example, IAPs data are incorporated into the radar map of the CWPs whilst RVA data are displayed as radar map layers in the CWPs. Coordinated FLs are defined by the

4.1. Knowledge variables The first section of results presents the rankings of the ‘knowledge variables’ (i.e., Likert scale 1e5) that controllers used to build their mental picture of traffic situations (Table 3). From the sample of eleven controllers, one controller did not draw any cognitive maps whilst two controllers did not draw any maps for the first and last scenarios. As revealed by subsequent discussions, the first controller said that he did not fully understand that he had to draw a map whilst the other two controllers reported that the first scenario was elementary and the last one was too complicated for a proper drawing. It is noted that it was not in the scope of this study to make extensive comparisons between novices and experts controllers because the sample was rather small. A Principal Component Analysis (PCA) was also undertaken to explore clusters of related variables in the list of ‘knowledge variables’ and identify any ‘structures’ or constructs underlying their representation. In our case, we used the data/frame model as a foundation for the targeted factor solution. Kim and Mueller (1978) argued that the minimum requirement for the researcher is to hypothesize a priori the number of factors in the PCA model. In our study, we hypothesised that we were looking for three or four factors that would represent the anchors or salient data for the explanatory frames or cognitive maps. The Kaiser-Meyer-Olkin (KMO) measure for sampling adequacy was calculated followed by the KMO statistics for the individual variables and the communalities. The value of the KMO statistics should be above .50

Table 3 Importance ratings for knowledge variables. Knowledge variables

Mean

Position and flight level of the arriving aircraft when transferred from the Area Control Centre (ACC) Aircraft performance data Radar Vectoring Area (RVA) data e e.g., location of navigation aids, terrain features and minimum sector altitudes Coordinated Flight Levels (i.e. pre agreed levels of transferring aircraft between the ATC units) Departure Sequence Instrument Approach Procedures (IAPs) e e.g., standard flows of departing aircraft ICAO Weight Categories (i.e. classification based on weight for separation purposes) ICAO Approach Categories (i.e. classification based on approach speeds for instrument approaches criteria design) Weather Data (i.e. cloud base and type, visibility, precipitation and any significant phenomena) Wind Data (i.e. wind direction and speed)

4.37

SD .82

4.02 3.51

.76 1.41

3.46

1.26

3.28 3.13

1.15 1.06

2.97

1.19

2.79

1.22

2.76

1.32

2.67

1.26

332

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

Table 4 Four clusters of knowledge variables explaining 83.33% of total variance. Knowledge variables

Factors 1

Factor 1

Factor 2 Factor 3 Factor 4

Instrument Approach Procedures (IAPs) Radar Vectoring Area (RVA) Data Coordinated Flight Levels Departure Sequence Weather Data Wind Data Position & Flight Levels Aircrafts’ Performance ICAO Approach Categories ICAO Weight Categories

2

3

4

.80 .79 .57 .40 .91 .89 .85 .84 .94 .84

Extraction Method: Principal Component Analysis. Rotation Method: Oblimin with Kaiser Normalization. Rotation converged in 13 iterations. Factor loadings with absolute values less than .40 were suppressed.

Letters of Agreements between Approach and Area Control units. The sequence of departing aircraft is coordinated with the Tower controllers and is known in advance (e.g., voice coordination or standard procedures). Histon and Hansman (2002) argued that controllers classify aircraft into standard or non-standard classes according to a number of traffic characteristics (e.g., ingress and egress points, coordination requirements, and crossing routes/ altitude profiles). In our case, Factor 1 provides the data for this type of classification. It is evident that this factor has a static character to operational controllers and represents readily available knowledge. Factor 2 - (weather): this factor represents dynamic information as weather updates are provided every half an hour by the meteorological office and wind data at the runway are available in the CWP constantly. Weather is a primary trigger for controller-pilot interactions because planned flight trajectories may become unacceptable in safety or quality terms (Hansman and Davison, 2000). In approach control, weather factors play an important role in selecting appropriate traffic routes for both arriving and departing aircraft. Pilots and controllers both strive for departure and arrival flows that will route aircraft clear of convective weather, precipitation and air turbulence. Factor 3 - (Aircraft position & performance): This factor refers to the flight levels, lateral positions and aircraft performance (e.g., ground speed, and rate of climb/descend). All items are displayed on the labels of the aircraft tracks on the radar except from the rate of climb/descend that can be either inferred or directly requested from flight crews. Factor 3 represents the “dynamic” data of a mental representation. For instance, the position and the flight level of an aircraft during transfer might deviate from those previously coordinated or prescribed in Letters of Agreements (e.g., flight crews may delay to call the Approach unit until after they passed beyond the transfer point requested by the Area Control unit). In addition, ground speed and rate of climb may vary even for similar aircraft (e.g., two identical jets may have different rates due to their weight). Controllers appear to take into account the dynamic information of this factor in the formation of cognitive maps. Factor 4 - (ICAO Aircraft classification categories): This factor refers to static data that are well known to controllers for each type of aircraft. These items are taken into account for separation purposes and for assigning IAPs, because different aircraft may follow variants of the same procedure. Overall, aircraft position and performance data (Factor 3) appeared to have the highest significance. This factor represents

information displayed on the aircraft track (i.e. position symbol, velocity leader, Mode C readout altitude indicator and the aircraft performance). A candidate explanation is that this factor includes ‘knowledge variables’ with the highest ‘rate of change’; hence, these items are very useful for handling traffic. Controllers monitor constantly these items to notice and extrapolate meaningful trends in the trajectory of aircraft. For example, a change in the actual Mode C readout indicates a flight level change while a change in the direction of the velocity leader indicates the initiation of a turn. Controllers need to be constantly aware of these changes and attend to a small number of updates to discriminate trends. When a controller instructs an aircraft to turn, he/she needs one or two updates of the velocity leader to discriminate the initiation of the turn. RVA structural elements & standard operations (Factor 1) is the next significant factor. This factor represents information that is static in nature but forms the context on which Factor-3 acquires its meaning. The turn of an aircraft can only be interpreted in relation to where this change is happening. Controllers need to be constantly aware of the position of the aircraft in relation to the static elements of RVA in order to manage traffic. Factor-4 (ICAO classification categories) is ranking third, very close to Factor-2. Controllers are well aware of these classifications and consider them accordingly in handling traffic. Weather and wind information (Factor 2) was ranked last in significance by the controllers. An explanation could be that controllers need Factors 1 and 3 items to put weather information into perspective. Controllers first need to be aware of the aircraft performance and position and then visualize where the aircraft is positioned in relation to the static features of RVA elements. Subsequently, knowledge of ICAO classifications enables them to put weather data into perspective. Furthermore, the fact that weather was a significant factor only in 40% of all scenarios may also explain its low ranking in significance. If weather had been a significant factor in all scenarios then it would have narrowed the scope of our study. 4.2. Operationalizing sensemaking in terms of criteria for reframing mental pictures It was anticipated that certain performance criteria (see Table 1) would influence the ‘shaping’ of mental models and hence, the overall efficiency in traffic planning. To this end, the correlations of criteria to scenario efficiency were assessed on the basis of the ratings of On-the-Job-Training (OJT) instructors. In specific, OJT instructors assessed the competence of controllers individually in each performance criterion and totally for the whole scenario on a Likert scale 1e5. As it can been from Table 5, most criteria were correlated with overall scenario efficiency. Only two criteria (i.e., C5: avoid stormy/turbulent zones, C6: consider terrain features) did not play an important role in the shaping of mental models and did not correlate to scenario efficiency. All other six criteria seemed to be taken very seriously by controllers and affected their efficiency. Apart from creating traffic patterns with minimal conflicts (criterion 1, Table 5), controllers appeared to create patterns that minimized the need to monitor the horizontal separation distance and gave more options to crews to adapt to unexpected situations (criteria 2 & 3). A related criterion 4 was creating patterns that would minimize the chances of crews aborting landing on the final approach segment and following a goaround procedure. This is an important criterion as it signifies efficient handling of altitude, position and speed in order to place aircraft on the final approach for landing. Positioning an aircraft high and fast on the final approach is the prime cause for an ATC induced go around. Finally, controllers appeared to take into

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

333

Table 5 Correlation of criteria to scenario efficiency and comparison of expert/novices along eight criteria. Pearson correlation of criteria with scenario efficiency

Novice criteria

Expert criteria

Differences

Criteria for selecting mental models

Pearson

(P)

Mean

SD

Mean

SD

t test

(P)

1. 2. 3. 4. 5. 6. 7. 8.

.67 .70 .71 .57 .23 .20 .65 .61

*** *** *** ***

2.73 2.53 2.43 2.57 4.10 4.03 2.70 3.10

.83 .82 .77 1.04 .92 .93 .79 .99

3.46 3.23 3.20 3.26 4.10 4.10 3.33 3.60

.73 .73 .76 .87 .88 .88 .76 .67

3.6 3.5 3.8 2.8

*** *** *** **

3.2 2.3

** *

Avoids potential conflicts Creates open & inspectable patterns Provides more options to crews Minimizes chances for go-around Avoids stormy/turbulent zones Takes into account terrain features Takes into account crews preferences Makes subtle changes to flight paths

** **

***Correlations are significant at .001 level. **Correlations are significant at .01 level. *Correlations are significant at .05 level.

account the preferences of crews and performed subtle changes in aircraft direction that would not divert them from their original standardized path (criteria 7 & 8). From Table 5 it may be seen that differences between experts and novices were mainly reflected in the six criteria that were used to reframe mental models. Criteria C1eC4 are all related to creating traffic patterns that are de-conflicted, more open and easier to inspect, and more flexible for crews to follow and adapt to unexpected situations. Expert controllers also appeared to pay attention to more subtle criteria for reframing their mental models (C7, C8) such as, taking into account the preferences of the flight crews and making gentle changes to flight paths that would respect economical issues in flight management and provide them with the least possible replanning in case of a CNS system failure (e.g. radar failure). Furthermore, an attempt was made to document the significance of the criteria used to frame and reframe the mental models in a qualitative form for the most representative cases of expert and novice controllers. Table 6 displays the ten scenarios, the number of aircrafts on each scenario, the weather significance and finally, the importance of performance criteria given by expert and novice controllers. A mark X indicates that controllers perceived a criterion to be very important for a specific scenario. From this table, it appears that expert controllers were able to employ more criteria for framing-reframing their mental models as the scenario difficulty increased in comparison to novices. Actually, experts were

able to hold all relevant criteria for each scenario in most of the cases. For example, expert controllers were able to utilize all eight criteria in the most difficult scenarios (i.e., scenarios 4 & 5 for the precision runway and scenarios 9 & 10 for the non-precision runway). On the other hand, novices were holding only the criteria relating to the basic tasks (i.e. avoiding potential conflicts) and hence, sacrificing nearly all other criteria in the demanding scenarios. Novices were mostly employing the context free-rules to avoid traffic conflicts. Experts were observed to “maneuver through” the complexity of the scenarios and accommodate all the relevant criteria on the basis of a larger repertoire of accumulated traffic and weather patterns. They were remaining sensitive to the operational context and were receptive to flight crews preferences. Experts employed recognition strategies that implied some sort of classification of traffic patterns and well-known traffic routines; in this sense, they were able to ‘read’ the traffic situation and employ well crafted solutions. Domain expertise enables a more nuanced appreciation of the traffic context and alternative ways of dealing with it. In other words, traffic planning was primed by the recognition of familiar problems (Klein, 1998). Novices appeared to decrease their performance as the scenario demands increased because they were focussing on how to minimize conflicts and were trying to ‘sort out’ traffic patterns to reach some sort of acceptable traffic arrangements; hence they seemed to ignore the significance of the weather particularly in the last two scenarios.

Table 6 Differences between experts and novices in terms of criteria for framing & reframing (A mark X indicates that a criterion was perceived to be significant for a particular scenario). Scenario

Precision runway

Non - precision runway

1 2 3 4 5 6 7 8 X 10

Number of aircraft

Weather

3 4 5 6 7 3 4 5 6 7

Insignificant Insignificant Insignificant Significant Highly Sign. Insignificant Insignificant Insignificant Significant Highly Sign.

Criteria. 1. Avoids potential conflicts. 2. Creates open & inspectable patterns. 3. Provides more options to crews. 4. Minimizes chances for go-around. 5. Takes into account terrain features. 6. Considers crew preferences. 7. Makes subtle changes to flight paths. 8. Avoids stormy/turbulent zones.

Criteria for framing & reframing (Experts)

Criteria for framing & reframing (Novices)

1

2

3

4

5

6

7

X X X X X X X X X X

X X X X X X X X X X

X X X X X X X X X X

X X X X X X X X X X

X X X X X X X X X X

X X X X X X X X X X

X X X X X X X X X X

8

X X

X X

1

2

3

4

5

6

7

X X X X X X X X X X

X X X X

X X X

X X X

X X X X

X

X

X X X

X X X

X X X X

X X X

X X

X X X

8

X

334

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

Differences between experts and novices are more clearly illustrated in the drawings of concept maps for handling traffic in several scenarios. Figs. 2 and 3 show drawings of concept maps for scenario 10 that seemed to be the most difficult one. It may be seen that expert and novice controllers assigned the same approach sequence on the five arriving aircraft and both made an attempt to de-conflict the traffic patterns. However, the planning of the novice was tighter than the expert and more fragile to a possible request for deviation of a flight crew due to weather. A tight planning also entails that deviations cannot be easily accommodated as they may disrupt overall planning. In the novice’s traffic plan, for instance, if the third aircraft cannot reach its position on the final as it was planned, it may produce a disruption and a possible re-sequencing for the following aircraft. In contrast, the expert provided the third aircraft with more time and space slack to accommodate a possible deviation in future. On the one hand, the expert made a considerable effort to avoid stormy/turbulence area, create open patterns, minimize chances for go around and optimize planning in terms of the eight criteria. On the other hand, the novice did not seem to take into account the terrain features and the crews’ preferences; in addition, the choices offered to flight crews were more restrictive as traffic planning was rather tight. It is important to note that novices used the same approach sequences with experts in most of the cases but differed in how they were proceeding in the overall planning of traffic. It was the structuring of the aircraft routes that differentiated between meeting the criteria or not. It is finally noted that these remarks were drawn mostly from qualitative statements made by OJT instructors.

3). The data/frame theory of sensemaking (Klein et al., 2007) has been adopted to examine how controllers identify a cognitive map, how they make up their minds to preserve or reframe it and finally, how they elaborate on their preferred choices. In this respect, sensemaking is not a single concept but incorporates multiple processes such as detecting problems, forming plausible explanations and coming up with actions that offer leverage points. In this section, a further attempt is made to operationalize sensemaking by presenting a loose coupling between the six processes of sensemaking and the eight performance criteria (Table 7). Table 7 essentially displays a roadmap on what criteria controllers use in order to transition between sensemaking processes (or adapt sensemaking). Usually, sensemaking starts with the identification of a frame (S1) that challenges the criterion of conflict avoidance (C1). Questioning a frame (S2) relies on several criteria of evaluation, three of the most frequent may include consideration of terrain features, crews preferences and stormy/turbulent areas (i.e., criteria C2, C3 and C4). A decision whether to reframe or create a new frame (S3 and S4) is likely to compare multiple frames in terms of how open a traffic pattern remains, how many options the flight crew are given, and whether chances of going around are minimized (i.e., criteria C5, C6 & C7). Finally, elaborating and preserving a frame (S5 and S6) involve making subtle changes to flight paths (C8). Overall, Table 7 provides a framework for structuring several data from the experiment and real life observations in order to understand the criteria that influence the flow and adaptation of sensemaking processes in a specific scenario. The following subsections provide several examples drawn from the analysis of scenarios, verbal reports and real traffic observations.

5. Discussion 5.1. Identifying a frame In this section, we use drawings of cognitive maps in combination with verbal reports from the experiment and from real traffic scenarios in order to gain an insight of the cognitive processes that operate and shape the ‘mental pictures’ or cognitive maps of controllers. Cognitive maps are annotated with headings (i.e. vectors) and level changes issued to aircraft as well as the proposed approach sequence of the inbound traffic (see Figs. 2 and

In many cases, identifying a frame is an effortless process that does not require deliberate sensemaking. When controllers were handling familiar scenarios, they were observed to resort to simple pattern matching e i.e., familiar traffic flows were assigned to standard traffic routings. As it was expected, the process of pattern identification was coupled with the first criterion of avoiding

Fig. 2. Drawing of a cognitive map of an expert controller for scenario 10.

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

335

Fig. 3. Drawing of a cognitive map of a novice controller for scenario 10.

potential conflicts. For the most familiar scenarios (i.e., scenarios 1, 2, 3 and 6, 7, 8), traffic planning was well-practised and aircraft instructions were kept to a minimum. In many typical operations, pattern recognition would lead to the selection of appropriate cognitive maps for handling traffic (see recognition-primed decisions in Klein, 1998). Controllers were also able to monitor traffic outside their sectors and would intervene to coordinate with the ACC unit to optimize the transfer conditions. Knowledge factor-3, referring to aircraft position and performance, received the highest attention in judging traffic familiarity and identifying a frame. Apart from this knowledge, controllers also considered other characteristics such as airways in use, type of flights and number of aircraft. For example, a familiar mix of four familiar aircraft arriving at the same time from two over-utilized airways could be easier identified than a mix of four non-scheduled aircraft arriving from four under e utilized airways, particularly when aircraft types are unfamiliar. In the first case, a quick glance of traffic on the radar screen, even outside their sectors of responsibility, was adequate for a controller to identify the frame. In the second case, controllers had to monitor traffic more often to identify a frame and build their cognitive maps, particularly for traffic outside their sector. 5.2. Questioning a frame Frames provide controllers with modicums for expectations. When they encounter data that violate their expectations they Table 7 Linking performance criteria to sensemaking processes. Sensemaking processes

Performance criteria

S1 - Identifying a frame S2 - Questioning a frame

C1 C2 C3 C4 C5 C6 C7

S3 - Reframing: Comparing multiple frames S4 - Reframing: Creating a new frame S5 - Elaborating a frame S6 - Preserving a frame

-

Avoids potential conflicts Takes into account terrain features Takes into account crews preferences Avoids stormy/turbulent zones Creates open & inspectable patterns Provides more options to crews Minimizes chances for go-around

C8 - Makes subtle changes to flight paths

initiate a process of challenging the data they receive. In most cases, questioning a frame would take into account several criteria such as taking into account terrain features, crews preferences and areas of stormy/turbulent weather. In questioning a frame, controllers would not know immediately if their frame was incorrect or the situation had taken a sudden turn. When controllers identified an unanticipated trend in the track of the aircraft, they responded either by directly questioning the flight crew or, if the situation was critical, by issuing conflict-avoidance instructions. Sometimes the sensor data could be inaccurate or even misleading e e.g., the radar may display ghost aircraft tracks due to clutter (unwanted radar returns) that may “fool” the detection algorithms and display false aircraft tracks. Although the radar trackers usually terminate them after a few radar updates, such events may trigger a temporary questioning of the frame. In general, controllers were largely familiar with possible locations of “ghost tracks” in the radar displays and were able to reject them before the detection algorithm did so. Questioning a frame was related with how well the proposed aircraft paths were taking into account crews preferences and terrain features (i.e. criteria C2 & C3). A delayed compliance with an instruction or a request for an amendment in a clearance by the crew may be indicative of an attempt to follow an optimum flight path profile that is different from the one planned by the controller. For example, an aircraft instructed to descent to a lower level may still be at the initial flight level or start a very slow descent. This may be explained as an attempt of the flight crew to optimize the flight path by flying a near continuous descent approach where intermediate level offs are avoided. However this may not be the intent of the controller as other aircraft that follow may have to descend and the preceding aircraft does not vacate the levels as planned which could jeopardize traffic planning. A demanding situation included making sense of radar depictions of weather phenomena (i.e., areas of precipitation and convective weather) although ATC radars are not optimized for this role. For instance, radars may give false alarms or fail to display adverse weather due technical or topographical limitations (e.g. weather masked by elevated terrain). In one case, experienced controllers relied on their accumulated knowledge of local weather

336

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

and anticipated an evolution of the weather pattern northwest (light horizontal pattern in Fig. 4) while the radar was displaying significant weather southeast (grid pattern in Fig. 4). As a result, they would choose the more efficient circumnavigation route at the bottom area. Driven by a different interpretation of the context novices perceived the top circumnavigation route to be equally plausible, hence neglecting the evolution of the weather northwest. In this case, the expert controllers were questioning their frame using the criterion of avoiding stormy and turbulent areas (i.e. criterion C4) capitalizing on their accumulated weather-related experience. Nevertheless, most controllers were in agreement on how to question their frames (e.g., requesting weather information from pilots) and re-arranged their traffic plans according to the weather circumnavigation actions requested by pilots. It is worth noting that pilots are usually provided with superior weather information as displayed on the flight deck (e.g. dedicated weather radars), perceived by direct visual observation and felt by the change of aircraft aerodynamic characteristics. It is noted that the transitions of the sensemaking processes for the weather circumnavigation scenario can be visualized in an abstract form in Table 8. 5.3. Reframing - comparing multiple frames Practitioners may track up to three frames simultaneously with the usual case involving two frames (Klein et al., 2007). Having two or even three explanatory frames requires a mechanism for ultimately settling on only one. Comparison of multiple frames can be initiated by the detection of an anomaly that resembles the function of a bifurcation point. In chaos theory (Gleick, 1987), a bifurcation point represents an unstable, temporary state that can evolve into one of several stable states. However, the direction of change is not clear and extensive expertise is required to track down the possible states. In our study, when confronted with uncertainty, controllers were comparing multiple frames in terms of possible actions that could be taken by the Tower and Area

Control units. The process of reframing was coupled with criteria C5, C6 & C7 e that is, creating open and inspectable patterns, providing more options to crews and minimizing chances of goaround. In a sense, these criteria refer to the provision of slack in the traffic planning in order to accommodate disruptions. Field observations documented a case where these criteria were taken into account in an interleaved way. During the sequencing of a wave of arriving aircraft, the Approach controller was not sure whether the Tower controller was able to cast a departure slot between two successive arrivals. In that scenario, the Approach controller was tracking two frames simultaneously with different implications for traffic planning (see Fig. 5). One hypothesis was that the Tower controller was able to do that whilst the other anticipated that the Tower controller would request an increase in the spacing between the next two arrivals. Thus, an effort to provide more slack (i.e. create an open arrival pattern, provide more options to crews and minimize the chances of a go around) would modify the initial planning. This was a general process in medium and high tempo operations, as Approach controllers would build a “micro-slack” in their plans increasing the distance between aircraft or the size of space in order to accommodate possible difficulties in the Tower unit in regulating departures or preparing for a possible go-around (Fig. 5). In situations with deteriorating weather, controllers were able to track two or even three frames in order to accommodate aircraft deviations. Experienced controllers were able to provide two or three accounts of areas where the aircraft would encounter bad weather as well as to propose alternative route deviations to accommodate requests by the flight crews. Novice controllers were not able to construct cognitive maps that would accommodate route deviations without altering significantly their overall traffic plan. In essence, novices reframed their mental pictures by creating a new frame which caused them to induce delays and extensions to aircraft routes. The last two scenarios for each runway (i.e. 4,5 & 9,10) required controllers to engage in a process of reframing either by comparing frames or by creating a new frame.

Novice controller unable to choose the northwest or southeast route

Weather not displayed by radar but perceived by expert

Intended path of the aircraft

Weather displayed byy radar Expert controller stt favors the southea southeast circumnavigation route Fig. 4. Novice controller perceives both circumnavigation routes as equally plausible.

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

337

Table 8 An example of the transitions of sensemaking processes according to performance criteria. Sensemaking process

Criteria

Sensemaking process

Criteria

Sensemaking process

Criteria

Sensemaking process

S1: Identify a frame

C1: Avoiding potential Conflicts

S2: Questioning a frame

C4: Avoiding stormy/turbulent areas

S3: Reframing: Comparing Frames

C5: Creating open & inspectable patterns

S6: Preserving a frame

5.4. Reframing - creating a new frame This is not an easy option as it implies aborting a current account and constructing a new one that was not an option in the first place. This process is quite similar to re-planning where a whole network of tasks has to change in a restricted time window. Kontogiannis (2010) stressed that replanning involves modifications to a plan during execution which presents many challenges to teams working in situations of high uncertainty. Similarly, the creation of a new frame may imposes excessive demands that may render the process undesired. The degree of coupling of the original account to the available resources may influence the process of creating a new account; in some cases, it was better to hold on with the original account rather than replace it. The process of creating a new frame is coupled with criteria C5, C6 & C7 e that is, creating open and inspectable patterns, providing more options to crews and minimizing chances of going-around. As discussed in the previous paragraph, these criteria refer to the provision of slack in the traffic planning both in normal and abnormal operations. The creation of new frame may be triggered by novel situations that signify the onset of an emergency. For example, the declaration by the crew of an emergency may call for a prompt reframing e i.e., constructing a new frame to accommodate the demands of the emergency. The creation of a new account is focused on accommodating the demand for unrestricted descend of the affected aircraft by creating open and inspectable patterns and providing more options to the crews to deal with emergency or avoid the area of the emergency. Controllers strive to keep other traffic well clear from the affected aircraft by employing a set of cognitive strategies (see for example a study of enroute controllers in Malakis et al., 2010a, 2010b). However, not all situations are so obvious, and controllers may have to consider a new frame based on the assumption of an impending emergency that has not been declared by the flight crew yet. The creation of a new frame is a process of replanning that can be accomplished easier in scenarios where the micro-slack has already built into the current frame. Verbal reports from the field study showed that experienced controllers were able to change

a range of features in their plans that supported their revision with minimal disruption by creating open and easily inspectable patterns that minimized the need for constant monitoring. 5.5. Preserving the frame When controllers preserve a frame by explaining away inconsistent evidence there is a risk of fixation errors (De Keyser and Woods, 1993). In our study, several controller errors went unnoticed because the evidence suggested that a revision was very weak or was explained away. In safety critical systems, misleading cues, absent indicators, and unusual cue patterns may create an environment that impedes error detection (Kontogiannis and Malakis, 2009). Sometimes preserving a frame may be the result of fixation errors and explanations may be completely out of focus. Preserving a frame is coupled with making subtle changes to flight paths (criterion C8). Field observations documented a case where the Approach controller had to deal with two departing aircraft in close succession. The first one was a scheduled flight that normally utilized a fast climbing jet aircraft while the second one was a nonscheduled airline that utilized a slower aircraft. Both aircraft were bound to the same direction in order to join the same airway for their cruising phase. This was a safe procedure since the fast climbing aircraft could outperform the climb of the succeeding aircraft. A few seconds after the two departures, the controller noticed that the ‘fast’ aircraft was climbing slower than what was usually the case; he had the impression that the radar scanning cycle was lagging behind and expected that the indicated low rate of climbing would be quickly replaced by the usual high rate of climbing. However, this never happened because the airline had replaced the usual aircraft with a slower propeller aircraft for maintenance reasons. This was the typical case of preserving his initial frame by making subtle changes on the flight paths and explaining away the data displayed on the radar map. An imminent separation loss was merely avoided when the controller was forced to turn the two aircraft on diverging courses and stop the second

Initial plan of

Modified plan

arrivals without

with slack

slack

Fig. 5. Increasing slack in a landing pattern by means of aircraft distance and space size.

338

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339

aircraft from climbing. Although relevant information on aircraft types was available on paper strips, the controller simply thought that he was dealing with the usual mix of aircraft types and failed to make a cross check. 5.6. Elaborating a frame This process involves preserving the current frame but adding more details and filling in the slots. The chances of surprises or inconsistencies are minimized as more detail is added in due time. Normally, elaboration is one of the final steps in the sensemaking process and is coupled with making subtle changes to flight paths (C8). In essence, the practitioner makes minor calibrations in the account as new data fit the frame conveniently (e.g., subtle changes of 5 in the form of headings of the aircraft). The drive for new data is smooth performance and observed patterns are progressively familiar. In these scenarios, data synthesis is accomplished accurately and the observer gets a feeling of smoothness in the management of traffic while controllers demonstrate ‘anticipatory thinking’ in searching for new data. In parallel, transient signals are rejected as insignificant while new data serve primarily to verify the frame. For example, the controllers were able to fine tune a wave of aircraft by making minor changes in headings and by controlling their speeds. 6. Conclusion In the present paper, cognitive maps, representing standard and non-standard air traffic flows, emerged as an explanatory framework for making sense of traffic patterns and for reframing mental pictures. In designing our research approach we tried to “attack” this concept from different angles but always within paradigm of Cognitive Systems Engineering. By interacting with controllers, we tried to measure some anchor or salient data for building a mental picture or cognitive map as presented in the data/frame model (Klein et al., 2007). Our endeavour was to address several cognitive processes of sensemaking as building blocks that can be collated into different patterns and shape the mental picture of controllers. Additionally, we elicited eight performance criteria which seem to trigger the processes of framing and reframing of mental pictures. In an effort to operationalize the notion of sensemaking, the eight criteria were connected to the six processes of the sensemaking and prominent examples were provided. Our belief is that sensemaking can be useful in describing patterns of success and failure at the sharp-end which can result in a database with some predictive value. In this study, sensemaking appeared to overlap with other concepts of macrocognition such as, problem detection and replanning. Klein et al. (2005) proposed that detecting a problem is equivalent to questioning a frame and becoming suspicious about the interpretation given to unfolding events. The fact that problem detection has not been separated from other aspects of sensemaking may give rise to an intersecting question - Why use an amorphous concept (i.e., sensemaking) to describe problem detection and not work on problem detection per se? A consideration of this issue should take into account the context of work and the flow of controller activities within it. Focussing only on problem detection and discarding the context of work may reduce the informative power of an incident. Sensemaking is a continuous and interactive process (Crandall et al., 2006) while attempts to study problem detection in isolation may significantly reduce the validity and predictive power of results. Incorporating problem detection as an aspect of sensemaking, allows us to elicit informative patterns that take into account the context of work, the cognitive flow and the practitioner’s stance. A similar argument can be made for the interaction between reframing and replanning. For example, the anomaly that triggers

replanning may present weak signals or the anomaly may arrive when resources are committed to a different plan. Adapting plans in progress is tightly linked to the process of reframing a mental picture with regard to potential threats. However, only when replanning fits into the larger context of sensemaking do informative patterns emerge. In other words, simply trying to describe an incident as a successful case of replanning may severely limit the inherent value of the story. The findings of this study are pending further validation and generalization due to the limited data derived from a small sample of controllers. Any associations and inferences drawn from this study are expected to remain relatively stable when additional studies are carried out. The main contribution of this study has been on exploring how the data/frame theory of sensemaking can be applied in the ATC field at the individual level and the attempt to operationalyze the sensemaking processes by linking them to the performance criteria. Cognitive maps emerged as an explanatory framework for sensemaking in most real-traffic scenarios. We provided supporting evidence that cognitive maps were constructed using a small number of knowledge variables. We also elicited a set of 8 criteria that form the basis for transitioning between the processes of sensemaking and provided a loose connection between processes and criteria (Table 7). Practical benefits can be expected especially in the areas of decision support systems and training. Design attempts to increase the amount of data are unlikely to be as effective as decision support for evaluating and interpreting data. Data filtering algorithms may reduce information overload but also pose challenges in sensemaking if the logic basis of the algorithms are underspecified. The design of training scenarios may also include practice in several processes of sensemaking. In their annual refresher training, controllers may be required to practice the same scenario in different conditions (e.g., multi-tasking, high time pressure and presence of interruptions) in order to learn how to develop skills in questioning frames, comparing frames and creating new frames. An improvement in sensemaking can also be achieved by designing tactical scenarios that promote the range and richness of frames. Clearly, there is a need for developing appropriate forms of decision aiding and training that would enhance sensemaking skills especially in view of the ‘data overload’ problem that may be created by new developments in the SESAR and NextGen programmes. References Amaldi, P., Leroux, M., 1995. Selecting relevant information in a complex environment: the case of air traffic control. In: Norros, L. (Ed.), 5th European Conference on Cognitive Science Approaches to Process Control. VTT Automation, Finland, pp. 89e98. Bainbridge, L., 1975. Working Memory in Air Traffic Control. Unpublished manuscript, University College, Department of Psychology, London. Bisseret, A., 1971. Analysis of mental processes involved in air traffic control. Ergonomics 14, 565e570. Colville, I., Pye, A., 2010. A sensemaking perspective on network pictures. Industrial Market Management 39, 372e380. Crandall, B., Klein, G.A., Hoffman, R.R., 2006. Working Minds: a Practitioner’s Guide to Cognitive Task Analysis. MIT Press. Massachusetts Institute of Technology, Cambridge, Massachusetts. De Keyser, V., Woods, D.D., 1993. Fixation errors: failures to revise situation assessment in dynamic risky systems. In: Colombo, A.G., Saiz de Bustamente Hoffman, A. (Eds.), Advanced Systems in Reliability Modelling. Kluwer Academic, Norwell, MA. Endsley, M.R., 1995. Toward a theory of situation awareness in dynamic systems. Human Factors 37, 32e64. Falzon, P., 1982. Display structures: compatibility with the operator’s mental representation and reasoning process. In: Proceedings of the 2nd European Annual Conference on Human Decision Making and Manual Control, pp. 297e305. Field, A., 2005. Discovering Statistics Using SPSS, second ed. SAGE Publications, London. Fiol, C.M., Huff, A.S., 1992. Maps for managers: where are we? Where do we go from Here? Journal of Management Studies 29, 267e285.

S. Malakis, T. Kontogiannis / Applied Ergonomics 44 (2013) 327e339 Fiske, S.T., Taylor, S.E., 1984. Social Cognition. Random House, New York. Gleick, J., 1987. Chaos: Making a New Science. Viking Penguin, New York. Hansman, R.J., Davison, H.J., 2000. The effect of shared information on pilot/ controller and controller/controller interactions. In: Third USA/Europe Air Traffic Management R&D Seminar. Napoli, Italy. Henneberg, S.C., Mouzas, S., Naudé, P., 2006. Network pictures e concepts and representations. European Journal of Marketing 40, 408e429. Histon, J.M., Hansman, R.J., 2002. The Impact of Structure on Cognitive Complexity in Air Traffic Control (Rep. No. ICAT-2002-4). MIT International Center for Air Transportation, Cambridge, MA. Hoffman, R.R., Militello, L.G., 2009. Perspectives on Cognitive Task Analysis: Historical Origins and Modern Communities of Practice. Psychology Press. Taylor & Francis Group, New York. ICAO, 2007. Air Traffic Management. Procedures for Air Navigation Services, Document 4444 ATM/501, fifteenth ed. International Civil Aviation Organization. Kim, J.O., Mueller, C.W., 1978. Factor Analysis: Statistical Methods and Practical Issues. SAGE Publications, Thousand Oaks CA. Quantitative Applications in the Social Sciences Series, No 14. Klein, G.A., 1998. Sources of Power: How People Make Decisions. MIT Press, Cambridge, MA. Klein, G., Moon, B., Hoffman, R.R., 2006. Making sense of sensemaking 2: a macrocognitive model. IEEE Intelligent Systems 21 (5). September/October. Klein, G., Philips, J.K., Rall, E.L., Peluso, D.A., 2007. A data/frame theory of sensemaking. In: Hoffman, R.R. (Ed.), Expertise Out of Context, Proceedings of the 6th International Conference on Naturalistic Decision Making. Laurence Erlbaum & Associates, Mahwah, New Jersey. Klein, G., Pliske, R., Crandall, B., Woods, D.D., 2005. Problem detection. Cognition, Technology, and Work 7, 14e28. Klein, G., Wiggins, S., Dominguez, C.O., 2010. Team sensemaking. Theoretical Issues in Ergonomics Science 11, 304e320. Kline, P., 1999. The Handbook of Psychological Testing, second ed. Routledge, London. Kontogiannis, T., 2010. Adapting plans in progress in distributed supervisory work: aspects of complexity, coupling, and control. Cognition, Technology, and Work 12, 103e118.

339

Kontogiannis, T., Malakis, S., 2009. A proactive approach to human error detection and identification in the aviation and air traffic control. Safety Science 47, 693e706. Malakis, S., Kontogiannis, T., Kirwan, B., 2010a. Managing emergencies and abnormal situations in air traffic control (Part I): taskwork strategies. Applied Ergonomics 41, 620e627. Malakis, S., Kontogiannis, T., Kirwan, B., 2010b. Managing emergencies and abnormal situations in air traffic control (Part II): teamwork strategies. Applied Ergonomics 41, 628e635. Mogford, R.H., 1997. Mental models and situation awareness in air traffic control. International Journal of Aviation Psychology 7, 331e341. Neisser, U., 1976. Cognition and Reality: Principles and Implications of Cognitive Psychology. Freeman, San Francisco. Niessen, C., Eyferth, K., Bierwagen, T., 1997. Modeling cognitive processes of experienced air traffic controller. In: Bagnara, S., Hollnagel, E., Mariani, M., Norros, L. (Eds.), 6th European Conference on Cognitive Science Approaches to Process Control, Time and Space in Process Control, Baveno, Italy, 23e26 September 1997. Oprins, E., Burggraff, E., Weerdenburg, H., 2006. Design of a computer-based assessment system for air traffic control training. International Journal of Aviation Psychology 16, 297e320. Roske-Hofstrand, R.J., Murphy, E.D., 1998. Human information processing in air traffic control. In: Smolensky, M., Stein, E. (Eds.), Human Factors in Air Traffic Control. Academic Press, San Diego, CA, pp. 65e113. Sieck, W.R., Klein, G., Peluso, D.A., Smith, J.L., Harris-Thompson, D., 2004. FOCUS: a Model of Sensemaking. Klein Associates, Fairborn, Ohio. Weick, K.E., 1995. Sensemaking in Organizations. Sage, Thousand Oaks, CA. Whitfeld, D., Jackson, A., 1982. The air traffic controller’s picture as an example of mental model. In: Johannsen, G., Rijnsdorp, J.E. (Eds.), Proceedings of the IFAC Conference on Analysis, Design and, Evaluation of Man-Machine Systems. Pergamon Press, London, pp. 45e52. Woods, D., Hollnagel, E., 2006. Joint Cognitive Systems: Patterns in Cognitive Systems Engineering. CRC Press. Taylor & Francis Group, Boca Raton FL. Woods, D.D., Patterson, E.S., Roth, E.M., 2002. Can we ever escape from data overload? A cognitive systems diagnosis. Cognition, Technology, and Work 4, 22e36.