- Email: [email protected]
Adaptive Level of Automation for risk management
13th IFAC/IFIP/IFORS/IEA Symposium on 13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems 13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Available onlineSystems at www.sciencedirect.com 13th IFAC/IFIP/IFORS/IEA Symposium on Aug. 30 Sept. 2, 2016. Kyoto, Japan Analysis, Design, and Evaluation of Human-Machine Systems Aug. 30 Sept. 2, 2016. Kyoto, Japan Analysis, Design, and Evaluation of Human-Machine Systems Aug. 30 - Sept. 2, 2016. Kyoto, Japan Aug. 30 - Sept. 2, 2016. Kyoto, Japan
ScienceDirect
IFAC-PapersOnLine 49-19 (2016) 048–053
Adaptive Level of Automation for risk management Adaptive Adaptive Level Level of of Automation Automation for for risk risk management management Adaptive Level of Automation for risk management
Marie-Pierre Pacaux-Lemoine & Patrick Millot Marie-Pierre Pacaux-Lemoine & Patrick Millot Marie-Pierre Pacaux-Lemoine & Patrick Millot Marie-Pierre Patrick Millot LAMIH,CNRS UMR Pacaux-Lemoine 8201, University of&Valenciennes, France LAMIH,CNRS UMR 8201, University of Valenciennes, France (marie-pierre.lemoine, [email protected]) LAMIH,CNRS UMR 8201, University of Valenciennes, France (marie-pierre.lemoine, [email protected]) LAMIH,CNRS UMR 8201, University of Valenciennes, France (marie-pierre.lemoine, [email protected]) (marie-pierre.lemoine, [email protected]) Abstract: Robots will know a huge deployment in the near future. Several research works are Abstract: Robots will know a huge deployment in the near future. Several research works are related withRobots human-robot interaction with cooperation between studies aimare to Abstract: will know a huge and deployment in the near future.robots. SeveralOther research works related withRobots human-robot interaction with cooperation between robots. Other studies aimare to Abstract: will of know a huge and deployment in thethem near future.and Several research works bring self-organization which allow quick appropriated autonomous relatedthem withcapabilities human-robot interaction and with cooperation between robots. Other studies aim to bring them capabilities of self-organization which allow them quick and appropriated autonomous related with human-robot interaction and with cooperation between robots. Other studies aim to answers to quite simpleofevents. That endows the swarm of robots capabilities of bring them capabilities self-organization which allow them quickwith and interesting appropriated autonomous answers to quite simpleofevents. That endows the swarm of robots with interesting capabilities of bring them capabilities self-organization which allow them quick and appropriated autonomous resilience adaptation. But these willswarm be limited especially when the individual answers toand quite simple events. Thatobjectives endows the of robots with interesting capabilitiesand of resilience and adaptation. But these willswarm be limited especially when the individual and answers quite simple will events. Thatobjectives endows the of robots with interesting capabilitiesThe of collective abilities insufficient to cope complex demands of thethe environment. resiliencetorobot and adaptation. Butbethese objectives willwith be limited especially when individual and collective robot abilities will bethese insufficient to cope with complex demandswhen of thethe environment. The resilience and adaptation. But objectives will be limited especially individual and only solution therefore tobe ask the human Nevertheless the behavior a self-organized collective robotisabilities will insufficient to help. cope with complex demands of theofenvironment. The only solution therefore tobe ask the human help. Nevertheless the behavior ofenvironment. a self-organized collective robotis will insufficient cope with oftothe The swarm of robots is difficult to ask predict thetohuman, due complex to his/herdemands difficulty build and especially only solution isabilities therefore to the by human help. Nevertheless the behavior of a self-organized swarm of robots is difficultto to ask predict by the human, due to his/herthe difficulty to build and especially only solution is therefore the human help. Nevertheless behavior of a self-organized to update on line a precise representation of the situation. A first way to cope with this problem is to swarm of robots is difficult to predict by the human, due to his/her difficulty to build and especially to update on line is a precise representation of the situation. A his/her first way to cope to with thisand problem is to swarm ofthe robots difficult to predict the human, due to difficulty build especially analyze robots’ tasks according to by several levels of activity: strategic level inthis which the global to update on line a precise representation of the situation. A firstaway to cope with problem is to analyze robots’ tasks according to several of activity: strategic level which the global to updatethe on line a precise representation of thelevels situation. A tactical firstaaway to cope within problem is to objectives are decided and the whole mission, the level forlevel applying andthe updating analyze the robots’ tasks according to planned several levels of activity: strategic inthis which global objectives are decided and the whole planned mission, the tactical level forlevel applying andthe updating analyze the robots’ tasks according to several levels of activity: a strategic in which global the missionareregarding new the events or planned new objectives operational for implementing objectives decided and whole mission,and the the tactical level forlevel applying and updating the mission regarding new the events or planned new objectives and operational level for implementing objectives decided and whole mission, the the tactical level forof applying and updating decisions. Aregarding complementary way is increase cooperative capabilities robots and to adapt the missionare new events or to new objectives and the operational level for implementing decisions. Aregarding complementary way is to increase cooperative capabilities of robots and to adapt the mission new events or new objectives and the operational level for implementing robots’ tasks order to find the between robots and humans. Our proposal is to decisions. A in complementary waybest is organization to increase cooperative capabilities of robots and to adapt robots’ tasks in order to find the between robots and humans. Our proposal is to decisions. complementary waybest is organization toAdaptive increase Level cooperative capabilities robots and of to adapt answer thisA problematic by the studying of robots Automation inofthe domain crisis robots’ tasks in order to find best organization between and humans. Our proposal is to answer this problematic by studying Adaptive Level of Automation in the domain of crisis robots’ in order to find best organization between and humans. proposal is to management firefighting. answer tasks this and problematic by the studying Adaptive Level of robots Automation in the Our domain of crisis management and firefighting. answer this problematic by studying Adaptive Level of Automation in the domain of crisis managementhuman/machine and firefighting.systems, Human decision support systems, robot control, Adaptive Keywords: © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier All rights reserved. management and firefighting. human/machine systems, Human decision support systems, robotLtd. control, Adaptive Keywords: Level of Automation, Firefighting. human/machine systems, Human decision support systems, robot control, Adaptive Keywords: Level of Automation, Firefighting. human/machine systems, Human decision support systems, robot control, Adaptive Keywords: Level of Automation, Firefighting. Level of Automation, Firefighting. 1. INTRODUCTION 1. INTRODUCTION 1. INTRODUCTION Risk management 1. deals with prevention, decision-making, INTRODUCTION Risk management deals with prevention, decision-making, actionmanagement taking, crisisdeals management and recovery, taking into Risk with prevention, decision-making, action taking, crisis management and recovery, taking into Risk management withunexpected prevention, decision-making, account consequences of events. Holding action taking, crisisdeals management and recovery, taking into account consequences of unexpected events. Holding action taking, crisis management and recovery, taking into “human(s) in the loop” takes of the human ability account consequences of advantage unexpected events. Holding “human(s) in the loop” takes advantage of the humanHolding ability account consequences of unexpected events. to cope withinunexpected even advantage dangerous of events on oneability hand, “human(s) the loop” takes the human to cope withinunexpected even advantage dangerous of events on oneability hand, “human(s) the hand loop”attempts takes the human and on the to counterbalance to cope withother unexpected even dangerous events onthe onehuman hand, and on the other hand attempts to counterbalance the human to cope with unexpected even dangerous events on one hand, trends make errors. and ontothe other hand attempts to counterbalance the human trends tothe make errors. and on other hand attempts to counterbalance the human trends to make errors. Risk management consists in three complementary steps: — trends to make errors. Risk management consists in three complementary steps: — prevention, in suchconsists a way the event be stopped or Risk management in unexpected three complementary steps: — prevention, in suchconsists a way the event be stopped or Risk management in unexpected three— complementary steps: — managed recovery, case the prevention,before in suchitsa propagation, way the unexpected event beinstopped or managed before itsa propagation, — recovery, instopped case the prevention, in such way the unexpected event be or event results in its an propagation, accident, making protectionin measures managed before — recovery, case the event results in its an propagation, accident, making protectionin measures managed before — recovery, case the necessary for avoiding damages,making and despite these two steps, event results in an accident, protection measures necessary for avoiding damages,making and despite these two steps, event results inhappens, an accident, protection measures if the accident it is mandatory to these — manage the necessary for avoiding damages, and despite two steps, if the accident happens, it is mandatory to these — manage the necessary for avoiding damages, and despite two steps, consequences order toit minimize the to most ones if the accidentinhappens, is mandatory — severe manage the consequences in order to minimize the most severe ones if the accident is mandatory — severe manageones the (Millot, 2014a).inhappens, consequences order toit minimize the to most (Millot, 2014a).in order to minimize the most severe ones consequences (Millot, 2014a). Prevention can be achieved by enhancing the system and the (Millot, 2014a). Prevention can be achieved by enhancing the system and the human capabilities to perform the processthe in system the right Prevention can be achieved by enhancing andway: the human capabilities to perform the processthe in system the right Prevention can be procedures, achieved by enhancing andway: the — by capabilities defining system monitoring devices, human to perform the process in the right way: — by capabilities defining procedures, system monitoring devices, human to perform the process in the right way: control management methods; — by taking caredevices, of the — by and defining procedures, system monitoring control and management methods; — by taking caredevices, of the — defining procedures, system monitoring sociobyorganizational context of the activities control and management methods; — human by taking care of and the socio organizational context of the human activities control and the management methods; — human by taking care of and the particularly adequacy between the system demands socio organizational context of the activities and particularly the adequacy between the human system activities demands and socio organizational context of the the human resources. In case of lack of adequacy, assistance particularly the adequacy between the system demands and the human resources. In case of lackthe of system adequacy, assistance particularly theintroduced adequacy between demands and toolshuman must resources. be using the present development of the In case of lack of adequacy, assistance tools must resources. be introduced using the present development of the human In case of lack of adequacy, assistance Information and the Engineering Sciences. The tools must beTechnologies introduced using present development of Information Technologies and the Engineering Sciences. The tools must be introduced using present development of Human Centered Designandapproaches Information Technologies Engineeringcombine Sciences. these The Human Centered Designandapproaches combine these Information Technologies Engineering Sciences. The technologies with cognitive sciences competencies for Human Centered Design approaches combine these technologies with cognitive sciences competencies for Human Centered Design approaches combine these technologies with cognitive sciences competencies for technologies with cognitive sciences competencies for
holding human in the loop. The related research topics are holding human in the loop. The related research topics are mainly: human impact inofthenew improve human holding loop.technology The relatedtoresearch topics are mainly: impact inofthenew toresearch improve human holding loop.technology The2015; related topics are situation awareness (Millot, Platt et al., 2014), mainly: human impact of new technology to improve human situation awareness (Millot, 2015; Platt et al., human 2014), mainly: impact of new technology to improve cooperativeawareness work including human-machine and situation (Millot, 2015; Platt cooperation et al., 2014), cooperativeawareness work including human-machine cooperation and situation (Millot, 2015; Platt et and al.,function 2014), CSCW, responsibility and accountability (task cooperative work including human-machine cooperation and CSCW, responsibility and accountability (task and function cooperative work including human-machine cooperation and allocation, authority sharing); CSCW, responsibility and accountability (task and function allocation, authority sharing); CSCW, responsibility and accountability (task and function allocation, authority sharing); Recovery can be sharing); enhanced: — in providing technical allocation, Recovery authority can be enhanced: — in providing technical protection can measures like barriers which make Recovery be enhanced: — forin instance providing technical protection measures like barriers instance which make Recovery can be impossible enhanced: — for providing technical erroneous actions (Vanderhaegen, 2014), — in protection measures like barriers forin instance which make erroneous actions impossible (Vanderhaegen, 2014), — in protection measures like barriers for instance which make developingactions reliability assessment methods for detecting erroneous impossible (Vanderhaegen, 2014), — in developing reliability assessment methods for detecting erroneous actions impossible (Vanderhaegen, 2014), — in human errorsreliability as well asassessment system failures (Cacciabue, 2014); developing methods for detecting human errorsreliability as well asassessment system failures (Cacciabue, 2014); developing methods for detecting — and errors better in humansfailures to detect and correct2014); their human as allowing well as system (Cacciabue, — and errors better in humansfailures to detect and correct2014); their human as allowing well as system (Cacciabue, own the so called system resilience; — anderrors, better enhancing in allowing humans to detect and correct their own errors, enhancing the so called system resilience; — anderrors, better enhancing in allowing humans tocooperation detectsystem and is correct their human-machine or human-human a way to own the so called resilience; human-machine or human-human cooperation is resilience; a way to own errors, enhancing the so called system enhance resilience (Millot, Boy, 2012). human-machine or human-human cooperation is a way to enhance resilienceor(Millot, Boy, 2012). human-machine human-human cooperation is a way to enhance resilience (Millot, Boy, 2012). The last resilience step copes(Millot, with socio-organizational issues of crisis enhance Boy, 2012). The last step copes with socio-organizational issues of crisis occurrence and management. The last step copes with socio-organizational issues of crisis occurrence and management. The last step copes with socio-organizational issues of crisis occurrence and management. This paper and mainly focuses on recovery. Our research takes occurrence management. This paper mainly focuses on recovery. Our research takes advantage the increase of technology to research provide crisis This paper of mainly focuses on recovery. Our takes advantage of the increase of technology to research provide crisis This paperwith mainly focuses on recovery. Our takes managers thattechnology can prevent humans crisis from advantage of thetechnology increase of to provide managers with technology thattechnology can prevent humans crisis from advantage of the increase of to provide being exposed to dangers that suchcanas prevent fire, radiological or managers with technology humans from being exposed to dangers such as fire, radiological or managers with For technology that humans from chemical risks. purpose a can particular must be being exposed to that dangers such as prevent fire, attention radiological or chemical risks. For that purpose a particular attention must be being exposed to dangers such as fire, radiological or payed to risks. the interaction between humans and systems, chemical For that purpose a particular attention mustand be payed to risks. the interaction between humans and systems, and chemical For that purpose a particular attention must be more to the task allocation between humans and payed precisely to the interaction between humans and systems, more to the task allocation between humans and payed precisely toand thethe interaction between humans and systems, systems management of their cooperation. “Systems” more precisely to the task allocation between humans and systems and the to management of their cooperation. “Systems” more precisely the task allocation between humans are seenand here devices to support the choice of and the systems the as management of their cooperation. “Systems” are seenand here as devices to support the choice of the systems the management of their cooperation. organization between agents to (either humans and “Systems” robots) in are seen here as devices support the choice of the organization between agents to(either humans and robots) in are seen here as devices support the choice of the organization between agents (either humans and robots) in organization between agents (either humans and robots) in
Copyright © 2016, 2016 IFAC 48 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright 2016 responsibility IFAC 48 Control. Peer review© of International Federation of Automatic Copyright ©under 2016 IFAC 48 10.1016/j.ifacol.2016.10.460 Copyright © 2016 IFAC 48
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
the field, as well as the organization between humans at higher decisional levels like in a crisis unit.
to the SKR Rasmussen’s ladder. It is linked to expertise, experience and practices of the agent. Internal KH gathers information analysis as well as diagnosis/prognosis and decision making. Information perception and actions are parts of the external KH. It deals with ergonomic aspects, regarding data visibility, readability and comprehensibility. Agents have to be aware of their abilities to collect information from the process and to act on the process in order to control the situation.
The first section of this paper presents the cooperation principles and how cooperation provides the necessary tools to identify and design the organization between humans and all types of systems. Moreover an important issue is how to support the cooperation by providing means to exchange information with the other agents in order to get a good representation of the global situation, but also a good representation of the involvement of the others.
Several researches have already studied task allocation and authority management between humans and systems. These studies deal with levels of automation (LoA). In the second section we recall their different definitions and compare each other and finally propose new tools called Levels of Cooperation (LoC). These LoCs may set up a basis to identify criteria and cooperative organizations in order to adapt LoAs.
This framework is implemented and tested in a research project, called SUCRé (French acronym for Dependability and Resilience for Management and Cooperative control of sociotechnical systems). It deals with crisis management, with a focus on human-human and human-robots cooperation in hostile environment. The application domain is firefighting. The last sections of the paper describe briefly the part of firefighting activities we plan to support. Systems, new organizations, LoAs and experimental protocols are then detailed. 2.
49
Know-How (KH) Internal ability to solve problems (regarding the process) capabilities: knowledge, rules, skills / experience, expertise processing abilities: inferences, workload, fatigue… External ability to: get information (from the process and the environment) act (on the process) Know-How-to-Coope rate (KHC) Internal ability to: build up a model of other agents (KH and KHC) deduce the other agents’ intentions analyze the task and identify the cooperative organization produce a common plan regarding tasks and coordination External ability to communicate: understanding other agents providing information to other agents
Fig. 1. Model of a cooperative agent. To sum up, the internal KH allows agents to build up a representation of the process current situation using their competences and capacities to analyse the situation and to make a decision. Agents are able to perform those functions because they interact with the process through their external KH. Nevertheless, most of the time, other agents take part in the solving process. They must therefore be taken into account through their KHC, otherwise they are only considered as complementary resource providers.
HUMAN-MACHINE COOPERATION
The principles of Human-Machine Cooperation (HMC) have been laid down first with the aim of a Dynamic Task Allocation (Millot, Mandiau, 1995; Millot, Hoc, 1997). They have been applied in several domains: air traffic control (Vanderhaegen et al. 1994; Hoc & Pacaux-Lemoine, 1998), fighter aircraft (Pacaux-Lemoine & Loiselet, 2002), car driving (Pacaux-Lemoine et al., 2004) and robotics (PacauxLemoine et al., 2011). The objective of these studies over more than 20 years was a generic approach of cooperation which could be used for the design and the evaluation of Human-Machine Systems. They led to model the cooperative agents (either human or machine) according to two dimensions: — the agent’s ability to control the process, also called know-how (KH), and — the agent’s ability to cooperate with other agents concerned by the process, also called know-how-to-cooperate (KHC) (Fig.1).
The KHC is also split up into two parts, an external and an internal one. The external KHC is the ability of an agent to get information about other agents and to provide information to them. Three main ways are identified in order to reach those goals: — they do direct observations of other agents (movements, mimics, emotions…), — they have verbal exchanges or they communicate through mediated supports, — they analyse the activity of others through the effect of their actions on the process. The support of the external KHC is called Common Work Space (CWS), (Pacaux-Lemoine & Debernard, 2002). It allows the human Situation Awareness (SA) related to the process state and the environment. When several agents compose a team, they can share SA and build and enrich a team-SA related to the current and future activity of the other agents (Millot & Pacaux-Lemoine, 2013). The way to share SA among the team members and to design team-SA is described in Millot (2014b).
2.1. Cooperative agent model The KH of an agent only concerns the control of the process without taking into account potential communication with other agents. It is split up into two parts, one called internal KH, the other one external KH.
The internal KHC allows an agent to build up a model of other agents in order to make easier the cooperation with them. It is built up and updated by learning, training and through exchanges with other agents. Agents gather and analyse information about others in order to infer their KH and KHC. This model can be used to identify and/or to specify cooperation between two agents that have same local
The internal KH consists of agents’ competences and their capacity to control the process taking for instance into account the agents’ workload or their Situation Awareness (SA). The competence of a human agent mainly consists of knowledge, rules and skills to control the process according 49
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
50
objectives, but we can extend its use to the cooperation between several levels of activity (0).
Human h1
Goals
Agent a 1 Planning level Definition and allocation of functions/tasks
So, a lack of KH of one agent can be balanced by another agent KH, and a balance is achieved through the KHC of at least one agent. Adaptive LoAs are based on the definition of the LoCs, but a dynamic adaptation of the current level is done regarding modification of the current KH and KHC of each agent due to the modification of the environment or of the evolution of the controlled process.
Information exchanges (visual, audio, haptic) regarding decision making about environment (Obstacles, Risky areas…), other agents (system abilities, human state…) and new objectives
Mission Orders, strategies
Human h2 Agent a 2 T actical level Mission completion and update
Adaptation of the mission Orders, tactics Human h3
These different constructs will be applied and evaluated in the project “SUCRé” presented in the next section. Actions in the field
4.
4.1. Project SUCRé
Agent a 3
The project objective is to design methodological and technological tools for the management and control of crisis situation. More concretely we deal with sociotechnical systems such as military organization or civil security, involving human operators and technical resources cooperating in command structures as well as on the spot in order to manage crisis. Among these resources, in hostile environment, mobile robots can be used to prevent human operators being exposed to a danger. These robots can be provided with communication and cooperative abilities in order to increase the efficiency of the sociotechnical system.
Operational level motion control, tracking
Fig. 2. Cooperation inside and between levels. A cooperation can exist inside the strategic level (planning), inside the tactical and operational levels, and moreover between these levels. All agents have the same global objective but local objectives are different according to each point of view. 3.
APPLICATION TO CRISIS MANAGEMENT
Cooperation between human operators on one hand and between human operators and robot(s) on the other hand will be studied and modeled in this dangerous environment. One objective is to draw rules for designing Adaptive LoAs and moreover to insert these rules in assistance systems placed at each level and able to support cooperative, decision and planning activities. Furthermore, robot-robot cooperation at the operational level is based on self-organization swarm and has to reach objectives ordered by the crisis unit. The swarm is remotely supervised by a human operator who must be assisted to manage the task allocation between humans and robots in case of unexpected events.
ADAPTATIVE LEVELS OF AUTOMATION
Several definitions of LoA do exist (Sheridan, 1992; Parasuraman et al., 2000; Kaber & Ensdley, 2003). Current definitions take into account interactions on decision making and action implementation, but also on information perception and information analysis (Inagaki & Sheridan, 2008). But exchanges of information between human operator and machine are generally omitted. Then we have extended the construct of LoA in order to propose the so called Levels of Cooperation (LoC), using the definition of a cooperative agent (Pacaux-Lemoine & Vanderhaegen, 2013). LoCs aim to harmonize approaches by combining different levels of completion of KH and KHC with a scale each. This scale gives progressive steps in the increase of levels. LoCs are the results of the combination of both scales (KH respectively KHC) explained bellow.
Therefore SUCRé involves several scientific challenges from the different domains of automation, computer sciences and cognitive psychology as well as technological design methodologies such as fast-prototyping of human-robot interaction (Fig. 3).
The KH scale uses rules mainly based on the availability of each agent’s KH and the overlap (intersection) between them (either empty or nonempty). According to the overlap, LoCs lead to different types of agents’ organization based on the 3 forms of cooperation (Schmidt, 1991). Agents can: — debate the best solution (debative form), — join their competencies to perform a task impossible to do alone, e.g. a workload too high (augmentative form), — or integrate the solution found by the other agent when this one has the adequate competence to solve a problem (integrative form). The KHC scale uses rules based on the availability of each agent’s KHC, external as well as internal part. Fig. 3. Crisis management and multi-level cooperation
50
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
The project is divided into four main tasks:
51
4.2.3 Robot-Robot Cooperation and Self-organization This cooperation is necessary in risky areas. Robots can be self-organized in order to analyze the environment to identify map, data and to reach a common goal like carrying a heavy or wise object. For instance the swarm role could be to find a source of smoke, to isolate it and to carry it outside. The swarm is homogenous or becomes heterogeneous if robots are gathering to compensate sensor or actuator problem, or to reach a goal that they cannot reach alone (cf. Fig. 3 – Operational level: Part on the left).
Task 1: “information and resources management” (Common Work Space between crisis unit and rescue staff and robots). Task 2: “measurements and diagnosis of human operators’ state” (through intelligent clothes). Task 3: analysis methods: fuzzy classification, resilience, safety analysis, Human Centered Design; analysis criteria; knowledge elicitation. Task 4: human(s)-robots(s) sharing control: planning, task allocation between humans, between human and intelligent clothes (visual and tactile information), between humans and robots and between robots.
4.2.4
Human-Robot Cooperation between tactical and operational levels This type of cooperation is defined in order to bring solution when a self-organized swarm (see previous section) has a problem or when fire-fighters want to modify the swarm objectives. The fire-fighter is outside of the risky area at the tactical level. He/she has new objective like gathering more information from a specific zone. He/she can provide the information about this zone to the swarm, to a leader of the swarm, to one robot or he/she can control remotely one robot to reach his/her goal. In this case the robot is removed from the swarm that has to be reorganized (Fig. 3 – Operational level: Part on the left and one operator/control operator on the tactical level). A representation of the four types of cooperation is given Fig. 4. The representation highlights cooperation at the operational level, at the tactical level and between operational and tactical levels. Cooperation (LoC) is detailed according to the KH and the KHC of each agent. A cooperation between one fire-fighter working in the field and a swarm is shown at the operational level. At the tactical level, we foresee a cooperation between a fire-fighter in the tactical command vehicle and a decision assistance system. Cooperation deals with resources management and task allocation at the operational level according to strategic orders.
Different types of experiments will be conducted in simulated environment first (serious games) and in real environment with ground robots. This research is led in collaboration with the fire brigade of North of France (SDIS 59). Scenarios result of fruitful exchanges with its commander. 4.2 The different types of cooperation in the project Several types of cooperation, designed at the strategic level are being studied at the tactical and operational levels. 4.2.1 Human-Human Cooperation inside level of activity and between levels of activity The tactical level is closed to the crisis zone but not implanted in the risky zone. It takes place in the transmission and decision room located in a tactical command vehicle. Several commanders with different roles can operate in the decision room and gather and analyze information from the strategic level (local administrative government, ministry…) and from the operational level (fire-fighters). Today, they exchange data and information with white boards and digital display. Therefore a secondary objective of the project is to update these communication devices by providing a Common Work Space (CWS). CWS will be common to the three levels but the information and the size of the area will be adapted to the objectives of each level (Fig. 3 – right part). We expect that upper levels will get a better representation of lower levels activities and states, especially from the operational levels by the means of smart clothes’ and robots’ sensors.
4.3 Future experiments Experiments will be conducted in 3 steps: - the first step consists in a technical validation of the platform – at the second step the scenarios will be played with students and university staff – and at the final step with real fire-fighter team. Robots are Turtlebots and Lego Mindstorm NXT. Experiments will take place in three rooms of the lab. The strategic level will be simulated (and not played) in order to manage experiments. Distress calls will be sent from the strategic room, fire stations will be contacted and unexpected events (injured people, toxic zone…) will be triggered. Tactical and operational levels will be played each in a specific room.
4.2.2
Human-Robot(s) Cooperation through Smart clothes at the operational level The objective of this cooperation is to help a fire-fighter to reach a specific area in order to perform a task that a robot cannot do. But the fire-fighter often feels difficulties to see inside this area because of smoke or darkness and the impossibility to use a device to enlighten the area. Therefore a swarm of robots can indicate safe trajectory to fire-fighter. Robots gather and analyze information from the environment in order to advice the human a free way to reach the target. But they are also able to take into account fire-fighter position by the mean of the smart clothes, with safety purposes (Fig. 3 – Operational level: middle).
The experimental protocol will first present the experiment to participants who will then complete questionnaires about their habits in the use of new technology. Participants will then be trained with crisis management, with the decision making support system of the tactical level and with robot remote control.
51
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
52
Strategic Goals: M ission definition
LoC/KHC at the tactical level: • Explicit mode inside the level LoC/KH at the tactical level: • Augmentative form on the supervision task to control different areas • Debative form for task allocation between humans and robots at the operational level • Integrative form to supervise environment, robots and human state (by machine), but decision is made by human
Human in the tactical command vehicle Information gathering
Information analysis
Action implementation
Decision making
HUMAN KNOW-HOW-TO-COOPERATE Tactical Goals Display or/and joystick
M ission update ASSISTANCE KNOW-HOW-TO-COOPERATE Information gathering
Information analysis
Decision making
Decision making assistance
Action implementation
Orders (Geographical points or areas, actions)
Update of the model they have of each other
Operational Goals
Information about individual and cooperative activity and state
LoC/KHC between tactical and operational levels: • Assisted explicit mode (first allocation by tactical level, reallocation asked by operational level) • Control of the operational agent’s state, providing new robots organization LoC/KH between tactical and operational levels: • Integrative form: tactical level defines new orders according to strategic level goals, operational level provides precise information and local analysis; teleoperation from tactical level • Debative form: order does not square with situation perceived by operational level
Human in the field
LoC/KHC at the operational level: • M utual control between robots • M utual control between humans • Implicit mode by the upper level (humans have no time no manage allocation)
Information gathering
Information analysis
Decision making
Action implementation
HUMAN KNOW-HOW-TO-COOPERATE
LoC/KH at the operational level: • Augmentative form on a task, no distinction regarding functions (different geographical areas; robots join together to carry heavy or bulky things) • Integrative form on functions inside a common task for trajectory identification (Human follows robots; robots supply other robot damage; robots provide map to other robots)
Robot display or/and smart clothes
M ission achievement
ROBOT KNOW-HOW-TO-COOPERATE Information gathering
Information analysis
Decision making
Action implementation
Robot(s)
Hostile environment
Fig. 4. Multi-level cooperation. Several experiments will be conducted in order to compare different types of cooperation. After each experiment, participants will be asked for comments during selfconfrontation and answer to questionnaires about how they understand and feel the evaluated cooperative tools.
the involved agents are human fire-fighters and robots which operate in the field especially in areas too hostile for the humans. Some of the humans intervene directly in the field in cooperation with the robots; other firemen organize and control the mission of the human-robots teams in the field. The task is decomposed into 3 hierarchical levels: - strategic which defines the organization, ie: the function allocation between the agents (humans and/or robots) of the lower levels, - tactical which deals with the completion of the mission by the agents and with a possible adaptation of the organization in response to unexpected events; - operational which is concerned with movements, actions and information gathering in the field.
Analysis will be performed regarding the objective achievement at each level and between levels. Required time necessary to achieve objectives will also be evaluated. A global workload assessment with TLX will be asked after each experiment in addition to the questionnaire. All experiments will be video recorded for self-confrontation and for analysis afterwards. A coding of the individual and of the cooperative activity of each agent will be performed with the aim to analyze the quality of cooperation in each situation. Coding of robots and assistance system is generated during experiments; but coding of human activity will be done thanks to video records by one or two coders. 5.
Adaptive Levels of Automation have been defined in this context. They will be implemented through several cooperative activities between the agents present inside each level but also from one level to another. This conceptual framework is going to be implemented in the lab, its feasibility will be studied experimentally and experimental protocols have been proposed. The oral presentation will give preliminary results.
CONCLUSION AND PERSPECTIVES
This paper has presented a model of multilevel cooperation between humans and between humans and robots. The environment deals with risky situations such as fire-fighting, 52
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
6.
ACKNOWLEGEMENT
53
Millot P. (2014b) Cooperative organization for Enhancing Situation Awareness, in P. Millot (Ed) Risk Management in Life critical Systems, pp 279-300, ISTE-Wiley, London, November, ISBN: 978-1-84821-480-4 Millot P. (2015) Situation Awareness: is the glass half empty or half full? Cognition Technology & Work, Springer, London, May, volume 17, Issue 2, pp 169-177, DOI: 10.1007/s10111-015-0322-6 Pacaux-Lemoine M.-P. & Loiselet A. (2002). A common work space to support cooperation in the cockpit of a twoseater fighter aircraft. M. Blay-Fornarino, A.M. Pinna Dery, K. Schmidt, & P. Zaraté, Cooperative systems design: a challenge of mobility age, IOS Press, Amsterdam, NorthHolland, pp. 157-172, January. Pacaux-Lemoine, M.-P., & Debernard, S. (2002). Common work space for human – machine cooperation in air traffic control. Control Engineering Practice, 10, 571–576. Pacaux-Lemoine M.-P., Ordioni J., Popieul J.-C., Debernard S., Millot P. (2004). Conception and evaluation of an advanced cooperative driving assistance tool. Proceedings of the IEEE International Conference on Vehicle Power and Propulsion, Paris, France, October. Pacaux-lemoine M., Debernard S., Godin A., Rajaonah B., Anceaux F., Vanderhaegen F. (2011). Levels of automation and human-machine cooperation: Application to humanrobot interaction.18th IFAC World Congress, Milano, Italy, August. Pacaux-lemoine M., Vanderhaegen F. (2013). Towards Levels of Cooperation. IEEE SMC Conference, Manchester, UK, October. Parasuraman, R., Sheridan, T., Wickens, C. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics-Part A, may, 30(3), 286-297. Platt D.,Millot P., Boy G. (2014) Participatory Design of a Cooperative Exploration Mediation Tool for Human Deep Space Risk Mitigation. , In Don Harris (Ed.) Engineering Psychology and Cognitive Ergonomics, Lecture Notes in Computer Science, Springer, Volume 8532, 2014, pp 363374. Schmidt K., “Cooperative Work: a conceptual framework”, In Rasmussen J., Brehmer B., and Leplat J. (Eds), Distributed decision making: Cognitive models for cooperative work, pp 75-110, John Willey and Sons, Chichester, 1991. Sheridan T.B. (1992). Telerobotics, automation and human supervisory control, The MIT Press Vanderhaegen F., Crévits I., Debernard S., Millot P.,(1994), Man-Machine Cooperation : Toward an activity regulation assistance for different air Traffic Control levels. International Journal on Human Computer Interaction - Vol 6, pp 65-104 - n° 1. Vanderhaegen F. (2014) Dissonance Engineering for Risk Analysis: a Theoretical Framework, in P. Millot (Ed) Risk Management in Life critical Systems, pp 157-182, ISTEWiley, London, October, ISBN: 978-1-84821-480-4
This research is partly funded by the French-American PUF program “Risk Management in Life Critical Systems” and the Nord-Pas de Calais regional council. We thank sincerely the fire brigade of North of France (SDIS 59) for their fruitful advices to define the scenarios. REFERENCES Cacciabue P.C. (2014) Designing Advanced Driver Assistance Systems in a Risk Based Process, in P. Millot (Ed) Risk Management in Life critical Systems, pp 117-156, ISTE-Wiley, London, October, ISBN: 978-1-84821-480-4 Hoc, J-M., Pacaux-Lemoine, M.-P. (1998). Cognitive evaluation of human-human and human machine cooperation modes in air traffic control. International Journal of Aviation Psychology, 8, 1-32. Inagaki, T., Sheridan, T. (2008). Authority and responsibility in Human-machine systems: is machine-initiated trading of authority permissible in the human-centered automation framework, Proceeding of Applied Human Factors and Ergonomics. Kaber D., Endsley M. (2003). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task, Theoretical Issues in Ergonomics Science ISSN 1463–922X print/ISSN 1464–536X online, 1-40 DOI 10.1080/1463922021000054335 Millot P., Mandiau R. (1995). Men-Machine Cooperative Organizations: Formal and Pragmatic implementation methods. In J.M. Hoc, P.C. Cacciabue, E. Hollnagel Editors, Expertise and Technology : Cognition Computer Cooperation, Lawrence Erlbraum Associates, New Jersey , chap. 13, pp. 213-228, 199. Millot P., Hoc JM., (1997). Human-Machine Cooperation : Metaphor or possible reality ? Proceedings of the 2nd European Conference on Cognitive Science ECCS’97, pp165-174. Manchester, U.K. Millot P., Lemoine MP., (1998). An attempt for generic concepts toward human-machine cooperation. IEEE SMC'98 Conference. San Diego, USA, October. Millot, P., Boy, G. (2012) Human-machine cooperation: a solution for life-critical Systems?, Work: A Journal of Prevention, Assessment and Rehabilitation, IOS Press, volume 41, pages 4552-4559 DOI: 10.3233/WOR-20120033-4552 Millot P. & Pacaux-lemoine M. (2013). A common Work Space for a mutual enrichment of Human-machine Cooperation and Team-Situation Awareness. 12th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, Las Vegas, Nevada, USA, août . Millot P. (2014a). Risk management in Life Critical Systems, ISTE-Wiley, London. October 2014, 420 pages, ISBN: 978-1-84821-480-4
53
ScienceDirect
IFAC-PapersOnLine 49-19 (2016) 048–053
Adaptive Level of Automation for risk management Adaptive Adaptive Level Level of of Automation Automation for for risk risk management management Adaptive Level of Automation for risk management
Marie-Pierre Pacaux-Lemoine & Patrick Millot Marie-Pierre Pacaux-Lemoine & Patrick Millot Marie-Pierre Pacaux-Lemoine & Patrick Millot Marie-Pierre Patrick Millot LAMIH,CNRS UMR Pacaux-Lemoine 8201, University of&Valenciennes, France LAMIH,CNRS UMR 8201, University of Valenciennes, France (marie-pierre.lemoine, [email protected]) LAMIH,CNRS UMR 8201, University of Valenciennes, France (marie-pierre.lemoine, [email protected]) LAMIH,CNRS UMR 8201, University of Valenciennes, France (marie-pierre.lemoine, [email protected]) (marie-pierre.lemoine, [email protected]) Abstract: Robots will know a huge deployment in the near future. Several research works are Abstract: Robots will know a huge deployment in the near future. Several research works are related withRobots human-robot interaction with cooperation between studies aimare to Abstract: will know a huge and deployment in the near future.robots. SeveralOther research works related withRobots human-robot interaction with cooperation between robots. Other studies aimare to Abstract: will of know a huge and deployment in thethem near future.and Several research works bring self-organization which allow quick appropriated autonomous relatedthem withcapabilities human-robot interaction and with cooperation between robots. Other studies aim to bring them capabilities of self-organization which allow them quick and appropriated autonomous related with human-robot interaction and with cooperation between robots. Other studies aim to answers to quite simpleofevents. That endows the swarm of robots capabilities of bring them capabilities self-organization which allow them quickwith and interesting appropriated autonomous answers to quite simpleofevents. That endows the swarm of robots with interesting capabilities of bring them capabilities self-organization which allow them quick and appropriated autonomous resilience adaptation. But these willswarm be limited especially when the individual answers toand quite simple events. Thatobjectives endows the of robots with interesting capabilitiesand of resilience and adaptation. But these willswarm be limited especially when the individual and answers quite simple will events. Thatobjectives endows the of robots with interesting capabilitiesThe of collective abilities insufficient to cope complex demands of thethe environment. resiliencetorobot and adaptation. Butbethese objectives willwith be limited especially when individual and collective robot abilities will bethese insufficient to cope with complex demandswhen of thethe environment. The resilience and adaptation. But objectives will be limited especially individual and only solution therefore tobe ask the human Nevertheless the behavior a self-organized collective robotisabilities will insufficient to help. cope with complex demands of theofenvironment. The only solution therefore tobe ask the human help. Nevertheless the behavior ofenvironment. a self-organized collective robotis will insufficient cope with oftothe The swarm of robots is difficult to ask predict thetohuman, due complex to his/herdemands difficulty build and especially only solution isabilities therefore to the by human help. Nevertheless the behavior of a self-organized swarm of robots is difficultto to ask predict by the human, due to his/herthe difficulty to build and especially only solution is therefore the human help. Nevertheless behavior of a self-organized to update on line a precise representation of the situation. A first way to cope with this problem is to swarm of robots is difficult to predict by the human, due to his/her difficulty to build and especially to update on line is a precise representation of the situation. A his/her first way to cope to with thisand problem is to swarm ofthe robots difficult to predict the human, due to difficulty build especially analyze robots’ tasks according to by several levels of activity: strategic level inthis which the global to update on line a precise representation of the situation. A firstaway to cope with problem is to analyze robots’ tasks according to several of activity: strategic level which the global to updatethe on line a precise representation of thelevels situation. A tactical firstaaway to cope within problem is to objectives are decided and the whole mission, the level forlevel applying andthe updating analyze the robots’ tasks according to planned several levels of activity: strategic inthis which global objectives are decided and the whole planned mission, the tactical level forlevel applying andthe updating analyze the robots’ tasks according to several levels of activity: a strategic in which global the missionareregarding new the events or planned new objectives operational for implementing objectives decided and whole mission,and the the tactical level forlevel applying and updating the mission regarding new the events or planned new objectives and operational level for implementing objectives decided and whole mission, the the tactical level forof applying and updating decisions. Aregarding complementary way is increase cooperative capabilities robots and to adapt the missionare new events or to new objectives and the operational level for implementing decisions. Aregarding complementary way is to increase cooperative capabilities of robots and to adapt the mission new events or new objectives and the operational level for implementing robots’ tasks order to find the between robots and humans. Our proposal is to decisions. A in complementary waybest is organization to increase cooperative capabilities of robots and to adapt robots’ tasks in order to find the between robots and humans. Our proposal is to decisions. complementary waybest is organization toAdaptive increase Level cooperative capabilities robots and of to adapt answer thisA problematic by the studying of robots Automation inofthe domain crisis robots’ tasks in order to find best organization between and humans. Our proposal is to answer this problematic by studying Adaptive Level of Automation in the domain of crisis robots’ in order to find best organization between and humans. proposal is to management firefighting. answer tasks this and problematic by the studying Adaptive Level of robots Automation in the Our domain of crisis management and firefighting. answer this problematic by studying Adaptive Level of Automation in the domain of crisis managementhuman/machine and firefighting.systems, Human decision support systems, robot control, Adaptive Keywords: © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier All rights reserved. management and firefighting. human/machine systems, Human decision support systems, robotLtd. control, Adaptive Keywords: Level of Automation, Firefighting. human/machine systems, Human decision support systems, robot control, Adaptive Keywords: Level of Automation, Firefighting. human/machine systems, Human decision support systems, robot control, Adaptive Keywords: Level of Automation, Firefighting. Level of Automation, Firefighting. 1. INTRODUCTION 1. INTRODUCTION 1. INTRODUCTION Risk management 1. deals with prevention, decision-making, INTRODUCTION Risk management deals with prevention, decision-making, actionmanagement taking, crisisdeals management and recovery, taking into Risk with prevention, decision-making, action taking, crisis management and recovery, taking into Risk management withunexpected prevention, decision-making, account consequences of events. Holding action taking, crisisdeals management and recovery, taking into account consequences of unexpected events. Holding action taking, crisis management and recovery, taking into “human(s) in the loop” takes of the human ability account consequences of advantage unexpected events. Holding “human(s) in the loop” takes advantage of the humanHolding ability account consequences of unexpected events. to cope withinunexpected even advantage dangerous of events on oneability hand, “human(s) the loop” takes the human to cope withinunexpected even advantage dangerous of events on oneability hand, “human(s) the hand loop”attempts takes the human and on the to counterbalance to cope withother unexpected even dangerous events onthe onehuman hand, and on the other hand attempts to counterbalance the human to cope with unexpected even dangerous events on one hand, trends make errors. and ontothe other hand attempts to counterbalance the human trends tothe make errors. and on other hand attempts to counterbalance the human trends to make errors. Risk management consists in three complementary steps: — trends to make errors. Risk management consists in three complementary steps: — prevention, in suchconsists a way the event be stopped or Risk management in unexpected three complementary steps: — prevention, in suchconsists a way the event be stopped or Risk management in unexpected three— complementary steps: — managed recovery, case the prevention,before in suchitsa propagation, way the unexpected event beinstopped or managed before itsa propagation, — recovery, instopped case the prevention, in such way the unexpected event be or event results in its an propagation, accident, making protectionin measures managed before — recovery, case the event results in its an propagation, accident, making protectionin measures managed before — recovery, case the necessary for avoiding damages,making and despite these two steps, event results in an accident, protection measures necessary for avoiding damages,making and despite these two steps, event results inhappens, an accident, protection measures if the accident it is mandatory to these — manage the necessary for avoiding damages, and despite two steps, if the accident happens, it is mandatory to these — manage the necessary for avoiding damages, and despite two steps, consequences order toit minimize the to most ones if the accidentinhappens, is mandatory — severe manage the consequences in order to minimize the most severe ones if the accident is mandatory — severe manageones the (Millot, 2014a).inhappens, consequences order toit minimize the to most (Millot, 2014a).in order to minimize the most severe ones consequences (Millot, 2014a). Prevention can be achieved by enhancing the system and the (Millot, 2014a). Prevention can be achieved by enhancing the system and the human capabilities to perform the processthe in system the right Prevention can be achieved by enhancing andway: the human capabilities to perform the processthe in system the right Prevention can be procedures, achieved by enhancing andway: the — by capabilities defining system monitoring devices, human to perform the process in the right way: — by capabilities defining procedures, system monitoring devices, human to perform the process in the right way: control management methods; — by taking caredevices, of the — by and defining procedures, system monitoring control and management methods; — by taking caredevices, of the — defining procedures, system monitoring sociobyorganizational context of the activities control and management methods; — human by taking care of and the socio organizational context of the human activities control and the management methods; — human by taking care of and the particularly adequacy between the system demands socio organizational context of the activities and particularly the adequacy between the human system activities demands and socio organizational context of the the human resources. In case of lack of adequacy, assistance particularly the adequacy between the system demands and the human resources. In case of lackthe of system adequacy, assistance particularly theintroduced adequacy between demands and toolshuman must resources. be using the present development of the In case of lack of adequacy, assistance tools must resources. be introduced using the present development of the human In case of lack of adequacy, assistance Information and the Engineering Sciences. The tools must beTechnologies introduced using present development of Information Technologies and the Engineering Sciences. The tools must be introduced using present development of Human Centered Designandapproaches Information Technologies Engineeringcombine Sciences. these The Human Centered Designandapproaches combine these Information Technologies Engineering Sciences. The technologies with cognitive sciences competencies for Human Centered Design approaches combine these technologies with cognitive sciences competencies for Human Centered Design approaches combine these technologies with cognitive sciences competencies for technologies with cognitive sciences competencies for
holding human in the loop. The related research topics are holding human in the loop. The related research topics are mainly: human impact inofthenew improve human holding loop.technology The relatedtoresearch topics are mainly: impact inofthenew toresearch improve human holding loop.technology The2015; related topics are situation awareness (Millot, Platt et al., 2014), mainly: human impact of new technology to improve human situation awareness (Millot, 2015; Platt et al., human 2014), mainly: impact of new technology to improve cooperativeawareness work including human-machine and situation (Millot, 2015; Platt cooperation et al., 2014), cooperativeawareness work including human-machine cooperation and situation (Millot, 2015; Platt et and al.,function 2014), CSCW, responsibility and accountability (task cooperative work including human-machine cooperation and CSCW, responsibility and accountability (task and function cooperative work including human-machine cooperation and allocation, authority sharing); CSCW, responsibility and accountability (task and function allocation, authority sharing); CSCW, responsibility and accountability (task and function allocation, authority sharing); Recovery can be sharing); enhanced: — in providing technical allocation, Recovery authority can be enhanced: — in providing technical protection can measures like barriers which make Recovery be enhanced: — forin instance providing technical protection measures like barriers instance which make Recovery can be impossible enhanced: — for providing technical erroneous actions (Vanderhaegen, 2014), — in protection measures like barriers forin instance which make erroneous actions impossible (Vanderhaegen, 2014), — in protection measures like barriers for instance which make developingactions reliability assessment methods for detecting erroneous impossible (Vanderhaegen, 2014), — in developing reliability assessment methods for detecting erroneous actions impossible (Vanderhaegen, 2014), — in human errorsreliability as well asassessment system failures (Cacciabue, 2014); developing methods for detecting human errorsreliability as well asassessment system failures (Cacciabue, 2014); developing methods for detecting — and errors better in humansfailures to detect and correct2014); their human as allowing well as system (Cacciabue, — and errors better in humansfailures to detect and correct2014); their human as allowing well as system (Cacciabue, own the so called system resilience; — anderrors, better enhancing in allowing humans to detect and correct their own errors, enhancing the so called system resilience; — anderrors, better enhancing in allowing humans tocooperation detectsystem and is correct their human-machine or human-human a way to own the so called resilience; human-machine or human-human cooperation is resilience; a way to own errors, enhancing the so called system enhance resilience (Millot, Boy, 2012). human-machine or human-human cooperation is a way to enhance resilienceor(Millot, Boy, 2012). human-machine human-human cooperation is a way to enhance resilience (Millot, Boy, 2012). The last resilience step copes(Millot, with socio-organizational issues of crisis enhance Boy, 2012). The last step copes with socio-organizational issues of crisis occurrence and management. The last step copes with socio-organizational issues of crisis occurrence and management. The last step copes with socio-organizational issues of crisis occurrence and management. This paper and mainly focuses on recovery. Our research takes occurrence management. This paper mainly focuses on recovery. Our research takes advantage the increase of technology to research provide crisis This paper of mainly focuses on recovery. Our takes advantage of the increase of technology to research provide crisis This paperwith mainly focuses on recovery. Our takes managers thattechnology can prevent humans crisis from advantage of thetechnology increase of to provide managers with technology thattechnology can prevent humans crisis from advantage of the increase of to provide being exposed to dangers that suchcanas prevent fire, radiological or managers with technology humans from being exposed to dangers such as fire, radiological or managers with For technology that humans from chemical risks. purpose a can particular must be being exposed to that dangers such as prevent fire, attention radiological or chemical risks. For that purpose a particular attention must be being exposed to dangers such as fire, radiological or payed to risks. the interaction between humans and systems, chemical For that purpose a particular attention mustand be payed to risks. the interaction between humans and systems, and chemical For that purpose a particular attention must be more to the task allocation between humans and payed precisely to the interaction between humans and systems, more to the task allocation between humans and payed precisely toand thethe interaction between humans and systems, systems management of their cooperation. “Systems” more precisely to the task allocation between humans and systems and the to management of their cooperation. “Systems” more precisely the task allocation between humans are seenand here devices to support the choice of and the systems the as management of their cooperation. “Systems” are seenand here as devices to support the choice of the systems the management of their cooperation. organization between agents to (either humans and “Systems” robots) in are seen here as devices support the choice of the organization between agents to(either humans and robots) in are seen here as devices support the choice of the organization between agents (either humans and robots) in organization between agents (either humans and robots) in
Copyright © 2016, 2016 IFAC 48 Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright 2016 responsibility IFAC 48 Control. Peer review© of International Federation of Automatic Copyright ©under 2016 IFAC 48 10.1016/j.ifacol.2016.10.460 Copyright © 2016 IFAC 48
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
the field, as well as the organization between humans at higher decisional levels like in a crisis unit.
to the SKR Rasmussen’s ladder. It is linked to expertise, experience and practices of the agent. Internal KH gathers information analysis as well as diagnosis/prognosis and decision making. Information perception and actions are parts of the external KH. It deals with ergonomic aspects, regarding data visibility, readability and comprehensibility. Agents have to be aware of their abilities to collect information from the process and to act on the process in order to control the situation.
The first section of this paper presents the cooperation principles and how cooperation provides the necessary tools to identify and design the organization between humans and all types of systems. Moreover an important issue is how to support the cooperation by providing means to exchange information with the other agents in order to get a good representation of the global situation, but also a good representation of the involvement of the others.
Several researches have already studied task allocation and authority management between humans and systems. These studies deal with levels of automation (LoA). In the second section we recall their different definitions and compare each other and finally propose new tools called Levels of Cooperation (LoC). These LoCs may set up a basis to identify criteria and cooperative organizations in order to adapt LoAs.
This framework is implemented and tested in a research project, called SUCRé (French acronym for Dependability and Resilience for Management and Cooperative control of sociotechnical systems). It deals with crisis management, with a focus on human-human and human-robots cooperation in hostile environment. The application domain is firefighting. The last sections of the paper describe briefly the part of firefighting activities we plan to support. Systems, new organizations, LoAs and experimental protocols are then detailed. 2.
49
Know-How (KH) Internal ability to solve problems (regarding the process) capabilities: knowledge, rules, skills / experience, expertise processing abilities: inferences, workload, fatigue… External ability to: get information (from the process and the environment) act (on the process) Know-How-to-Coope rate (KHC) Internal ability to: build up a model of other agents (KH and KHC) deduce the other agents’ intentions analyze the task and identify the cooperative organization produce a common plan regarding tasks and coordination External ability to communicate: understanding other agents providing information to other agents
Fig. 1. Model of a cooperative agent. To sum up, the internal KH allows agents to build up a representation of the process current situation using their competences and capacities to analyse the situation and to make a decision. Agents are able to perform those functions because they interact with the process through their external KH. Nevertheless, most of the time, other agents take part in the solving process. They must therefore be taken into account through their KHC, otherwise they are only considered as complementary resource providers.
HUMAN-MACHINE COOPERATION
The principles of Human-Machine Cooperation (HMC) have been laid down first with the aim of a Dynamic Task Allocation (Millot, Mandiau, 1995; Millot, Hoc, 1997). They have been applied in several domains: air traffic control (Vanderhaegen et al. 1994; Hoc & Pacaux-Lemoine, 1998), fighter aircraft (Pacaux-Lemoine & Loiselet, 2002), car driving (Pacaux-Lemoine et al., 2004) and robotics (PacauxLemoine et al., 2011). The objective of these studies over more than 20 years was a generic approach of cooperation which could be used for the design and the evaluation of Human-Machine Systems. They led to model the cooperative agents (either human or machine) according to two dimensions: — the agent’s ability to control the process, also called know-how (KH), and — the agent’s ability to cooperate with other agents concerned by the process, also called know-how-to-cooperate (KHC) (Fig.1).
The KHC is also split up into two parts, an external and an internal one. The external KHC is the ability of an agent to get information about other agents and to provide information to them. Three main ways are identified in order to reach those goals: — they do direct observations of other agents (movements, mimics, emotions…), — they have verbal exchanges or they communicate through mediated supports, — they analyse the activity of others through the effect of their actions on the process. The support of the external KHC is called Common Work Space (CWS), (Pacaux-Lemoine & Debernard, 2002). It allows the human Situation Awareness (SA) related to the process state and the environment. When several agents compose a team, they can share SA and build and enrich a team-SA related to the current and future activity of the other agents (Millot & Pacaux-Lemoine, 2013). The way to share SA among the team members and to design team-SA is described in Millot (2014b).
2.1. Cooperative agent model The KH of an agent only concerns the control of the process without taking into account potential communication with other agents. It is split up into two parts, one called internal KH, the other one external KH.
The internal KHC allows an agent to build up a model of other agents in order to make easier the cooperation with them. It is built up and updated by learning, training and through exchanges with other agents. Agents gather and analyse information about others in order to infer their KH and KHC. This model can be used to identify and/or to specify cooperation between two agents that have same local
The internal KH consists of agents’ competences and their capacity to control the process taking for instance into account the agents’ workload or their Situation Awareness (SA). The competence of a human agent mainly consists of knowledge, rules and skills to control the process according 49
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
50
objectives, but we can extend its use to the cooperation between several levels of activity (0).
Human h1
Goals
Agent a 1 Planning level Definition and allocation of functions/tasks
So, a lack of KH of one agent can be balanced by another agent KH, and a balance is achieved through the KHC of at least one agent. Adaptive LoAs are based on the definition of the LoCs, but a dynamic adaptation of the current level is done regarding modification of the current KH and KHC of each agent due to the modification of the environment or of the evolution of the controlled process.
Information exchanges (visual, audio, haptic) regarding decision making about environment (Obstacles, Risky areas…), other agents (system abilities, human state…) and new objectives
Mission Orders, strategies
Human h2 Agent a 2 T actical level Mission completion and update
Adaptation of the mission Orders, tactics Human h3
These different constructs will be applied and evaluated in the project “SUCRé” presented in the next section. Actions in the field
4.
4.1. Project SUCRé
Agent a 3
The project objective is to design methodological and technological tools for the management and control of crisis situation. More concretely we deal with sociotechnical systems such as military organization or civil security, involving human operators and technical resources cooperating in command structures as well as on the spot in order to manage crisis. Among these resources, in hostile environment, mobile robots can be used to prevent human operators being exposed to a danger. These robots can be provided with communication and cooperative abilities in order to increase the efficiency of the sociotechnical system.
Operational level motion control, tracking
Fig. 2. Cooperation inside and between levels. A cooperation can exist inside the strategic level (planning), inside the tactical and operational levels, and moreover between these levels. All agents have the same global objective but local objectives are different according to each point of view. 3.
APPLICATION TO CRISIS MANAGEMENT
Cooperation between human operators on one hand and between human operators and robot(s) on the other hand will be studied and modeled in this dangerous environment. One objective is to draw rules for designing Adaptive LoAs and moreover to insert these rules in assistance systems placed at each level and able to support cooperative, decision and planning activities. Furthermore, robot-robot cooperation at the operational level is based on self-organization swarm and has to reach objectives ordered by the crisis unit. The swarm is remotely supervised by a human operator who must be assisted to manage the task allocation between humans and robots in case of unexpected events.
ADAPTATIVE LEVELS OF AUTOMATION
Several definitions of LoA do exist (Sheridan, 1992; Parasuraman et al., 2000; Kaber & Ensdley, 2003). Current definitions take into account interactions on decision making and action implementation, but also on information perception and information analysis (Inagaki & Sheridan, 2008). But exchanges of information between human operator and machine are generally omitted. Then we have extended the construct of LoA in order to propose the so called Levels of Cooperation (LoC), using the definition of a cooperative agent (Pacaux-Lemoine & Vanderhaegen, 2013). LoCs aim to harmonize approaches by combining different levels of completion of KH and KHC with a scale each. This scale gives progressive steps in the increase of levels. LoCs are the results of the combination of both scales (KH respectively KHC) explained bellow.
Therefore SUCRé involves several scientific challenges from the different domains of automation, computer sciences and cognitive psychology as well as technological design methodologies such as fast-prototyping of human-robot interaction (Fig. 3).
The KH scale uses rules mainly based on the availability of each agent’s KH and the overlap (intersection) between them (either empty or nonempty). According to the overlap, LoCs lead to different types of agents’ organization based on the 3 forms of cooperation (Schmidt, 1991). Agents can: — debate the best solution (debative form), — join their competencies to perform a task impossible to do alone, e.g. a workload too high (augmentative form), — or integrate the solution found by the other agent when this one has the adequate competence to solve a problem (integrative form). The KHC scale uses rules based on the availability of each agent’s KHC, external as well as internal part. Fig. 3. Crisis management and multi-level cooperation
50
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
The project is divided into four main tasks:
51
4.2.3 Robot-Robot Cooperation and Self-organization This cooperation is necessary in risky areas. Robots can be self-organized in order to analyze the environment to identify map, data and to reach a common goal like carrying a heavy or wise object. For instance the swarm role could be to find a source of smoke, to isolate it and to carry it outside. The swarm is homogenous or becomes heterogeneous if robots are gathering to compensate sensor or actuator problem, or to reach a goal that they cannot reach alone (cf. Fig. 3 – Operational level: Part on the left).
Task 1: “information and resources management” (Common Work Space between crisis unit and rescue staff and robots). Task 2: “measurements and diagnosis of human operators’ state” (through intelligent clothes). Task 3: analysis methods: fuzzy classification, resilience, safety analysis, Human Centered Design; analysis criteria; knowledge elicitation. Task 4: human(s)-robots(s) sharing control: planning, task allocation between humans, between human and intelligent clothes (visual and tactile information), between humans and robots and between robots.
4.2.4
Human-Robot Cooperation between tactical and operational levels This type of cooperation is defined in order to bring solution when a self-organized swarm (see previous section) has a problem or when fire-fighters want to modify the swarm objectives. The fire-fighter is outside of the risky area at the tactical level. He/she has new objective like gathering more information from a specific zone. He/she can provide the information about this zone to the swarm, to a leader of the swarm, to one robot or he/she can control remotely one robot to reach his/her goal. In this case the robot is removed from the swarm that has to be reorganized (Fig. 3 – Operational level: Part on the left and one operator/control operator on the tactical level). A representation of the four types of cooperation is given Fig. 4. The representation highlights cooperation at the operational level, at the tactical level and between operational and tactical levels. Cooperation (LoC) is detailed according to the KH and the KHC of each agent. A cooperation between one fire-fighter working in the field and a swarm is shown at the operational level. At the tactical level, we foresee a cooperation between a fire-fighter in the tactical command vehicle and a decision assistance system. Cooperation deals with resources management and task allocation at the operational level according to strategic orders.
Different types of experiments will be conducted in simulated environment first (serious games) and in real environment with ground robots. This research is led in collaboration with the fire brigade of North of France (SDIS 59). Scenarios result of fruitful exchanges with its commander. 4.2 The different types of cooperation in the project Several types of cooperation, designed at the strategic level are being studied at the tactical and operational levels. 4.2.1 Human-Human Cooperation inside level of activity and between levels of activity The tactical level is closed to the crisis zone but not implanted in the risky zone. It takes place in the transmission and decision room located in a tactical command vehicle. Several commanders with different roles can operate in the decision room and gather and analyze information from the strategic level (local administrative government, ministry…) and from the operational level (fire-fighters). Today, they exchange data and information with white boards and digital display. Therefore a secondary objective of the project is to update these communication devices by providing a Common Work Space (CWS). CWS will be common to the three levels but the information and the size of the area will be adapted to the objectives of each level (Fig. 3 – right part). We expect that upper levels will get a better representation of lower levels activities and states, especially from the operational levels by the means of smart clothes’ and robots’ sensors.
4.3 Future experiments Experiments will be conducted in 3 steps: - the first step consists in a technical validation of the platform – at the second step the scenarios will be played with students and university staff – and at the final step with real fire-fighter team. Robots are Turtlebots and Lego Mindstorm NXT. Experiments will take place in three rooms of the lab. The strategic level will be simulated (and not played) in order to manage experiments. Distress calls will be sent from the strategic room, fire stations will be contacted and unexpected events (injured people, toxic zone…) will be triggered. Tactical and operational levels will be played each in a specific room.
4.2.2
Human-Robot(s) Cooperation through Smart clothes at the operational level The objective of this cooperation is to help a fire-fighter to reach a specific area in order to perform a task that a robot cannot do. But the fire-fighter often feels difficulties to see inside this area because of smoke or darkness and the impossibility to use a device to enlighten the area. Therefore a swarm of robots can indicate safe trajectory to fire-fighter. Robots gather and analyze information from the environment in order to advice the human a free way to reach the target. But they are also able to take into account fire-fighter position by the mean of the smart clothes, with safety purposes (Fig. 3 – Operational level: middle).
The experimental protocol will first present the experiment to participants who will then complete questionnaires about their habits in the use of new technology. Participants will then be trained with crisis management, with the decision making support system of the tactical level and with robot remote control.
51
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
52
Strategic Goals: M ission definition
LoC/KHC at the tactical level: • Explicit mode inside the level LoC/KH at the tactical level: • Augmentative form on the supervision task to control different areas • Debative form for task allocation between humans and robots at the operational level • Integrative form to supervise environment, robots and human state (by machine), but decision is made by human
Human in the tactical command vehicle Information gathering
Information analysis
Action implementation
Decision making
HUMAN KNOW-HOW-TO-COOPERATE Tactical Goals Display or/and joystick
M ission update ASSISTANCE KNOW-HOW-TO-COOPERATE Information gathering
Information analysis
Decision making
Decision making assistance
Action implementation
Orders (Geographical points or areas, actions)
Update of the model they have of each other
Operational Goals
Information about individual and cooperative activity and state
LoC/KHC between tactical and operational levels: • Assisted explicit mode (first allocation by tactical level, reallocation asked by operational level) • Control of the operational agent’s state, providing new robots organization LoC/KH between tactical and operational levels: • Integrative form: tactical level defines new orders according to strategic level goals, operational level provides precise information and local analysis; teleoperation from tactical level • Debative form: order does not square with situation perceived by operational level
Human in the field
LoC/KHC at the operational level: • M utual control between robots • M utual control between humans • Implicit mode by the upper level (humans have no time no manage allocation)
Information gathering
Information analysis
Decision making
Action implementation
HUMAN KNOW-HOW-TO-COOPERATE
LoC/KH at the operational level: • Augmentative form on a task, no distinction regarding functions (different geographical areas; robots join together to carry heavy or bulky things) • Integrative form on functions inside a common task for trajectory identification (Human follows robots; robots supply other robot damage; robots provide map to other robots)
Robot display or/and smart clothes
M ission achievement
ROBOT KNOW-HOW-TO-COOPERATE Information gathering
Information analysis
Decision making
Action implementation
Robot(s)
Hostile environment
Fig. 4. Multi-level cooperation. Several experiments will be conducted in order to compare different types of cooperation. After each experiment, participants will be asked for comments during selfconfrontation and answer to questionnaires about how they understand and feel the evaluated cooperative tools.
the involved agents are human fire-fighters and robots which operate in the field especially in areas too hostile for the humans. Some of the humans intervene directly in the field in cooperation with the robots; other firemen organize and control the mission of the human-robots teams in the field. The task is decomposed into 3 hierarchical levels: - strategic which defines the organization, ie: the function allocation between the agents (humans and/or robots) of the lower levels, - tactical which deals with the completion of the mission by the agents and with a possible adaptation of the organization in response to unexpected events; - operational which is concerned with movements, actions and information gathering in the field.
Analysis will be performed regarding the objective achievement at each level and between levels. Required time necessary to achieve objectives will also be evaluated. A global workload assessment with TLX will be asked after each experiment in addition to the questionnaire. All experiments will be video recorded for self-confrontation and for analysis afterwards. A coding of the individual and of the cooperative activity of each agent will be performed with the aim to analyze the quality of cooperation in each situation. Coding of robots and assistance system is generated during experiments; but coding of human activity will be done thanks to video records by one or two coders. 5.
Adaptive Levels of Automation have been defined in this context. They will be implemented through several cooperative activities between the agents present inside each level but also from one level to another. This conceptual framework is going to be implemented in the lab, its feasibility will be studied experimentally and experimental protocols have been proposed. The oral presentation will give preliminary results.
CONCLUSION AND PERSPECTIVES
This paper has presented a model of multilevel cooperation between humans and between humans and robots. The environment deals with risky situations such as fire-fighting, 52
2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan
Marie-Pierre Pacaux-Lemoine et al. / IFAC-PapersOnLine 49-19 (2016) 048–053
6.
ACKNOWLEGEMENT
53
Millot P. (2014b) Cooperative organization for Enhancing Situation Awareness, in P. Millot (Ed) Risk Management in Life critical Systems, pp 279-300, ISTE-Wiley, London, November, ISBN: 978-1-84821-480-4 Millot P. (2015) Situation Awareness: is the glass half empty or half full? Cognition Technology & Work, Springer, London, May, volume 17, Issue 2, pp 169-177, DOI: 10.1007/s10111-015-0322-6 Pacaux-Lemoine M.-P. & Loiselet A. (2002). A common work space to support cooperation in the cockpit of a twoseater fighter aircraft. M. Blay-Fornarino, A.M. Pinna Dery, K. Schmidt, & P. Zaraté, Cooperative systems design: a challenge of mobility age, IOS Press, Amsterdam, NorthHolland, pp. 157-172, January. Pacaux-Lemoine, M.-P., & Debernard, S. (2002). Common work space for human – machine cooperation in air traffic control. Control Engineering Practice, 10, 571–576. Pacaux-Lemoine M.-P., Ordioni J., Popieul J.-C., Debernard S., Millot P. (2004). Conception and evaluation of an advanced cooperative driving assistance tool. Proceedings of the IEEE International Conference on Vehicle Power and Propulsion, Paris, France, October. Pacaux-lemoine M., Debernard S., Godin A., Rajaonah B., Anceaux F., Vanderhaegen F. (2011). Levels of automation and human-machine cooperation: Application to humanrobot interaction.18th IFAC World Congress, Milano, Italy, August. Pacaux-lemoine M., Vanderhaegen F. (2013). Towards Levels of Cooperation. IEEE SMC Conference, Manchester, UK, October. Parasuraman, R., Sheridan, T., Wickens, C. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics-Part A, may, 30(3), 286-297. Platt D.,Millot P., Boy G. (2014) Participatory Design of a Cooperative Exploration Mediation Tool for Human Deep Space Risk Mitigation. , In Don Harris (Ed.) Engineering Psychology and Cognitive Ergonomics, Lecture Notes in Computer Science, Springer, Volume 8532, 2014, pp 363374. Schmidt K., “Cooperative Work: a conceptual framework”, In Rasmussen J., Brehmer B., and Leplat J. (Eds), Distributed decision making: Cognitive models for cooperative work, pp 75-110, John Willey and Sons, Chichester, 1991. Sheridan T.B. (1992). Telerobotics, automation and human supervisory control, The MIT Press Vanderhaegen F., Crévits I., Debernard S., Millot P.,(1994), Man-Machine Cooperation : Toward an activity regulation assistance for different air Traffic Control levels. International Journal on Human Computer Interaction - Vol 6, pp 65-104 - n° 1. Vanderhaegen F. (2014) Dissonance Engineering for Risk Analysis: a Theoretical Framework, in P. Millot (Ed) Risk Management in Life critical Systems, pp 157-182, ISTEWiley, London, October, ISBN: 978-1-84821-480-4
This research is partly funded by the French-American PUF program “Risk Management in Life Critical Systems” and the Nord-Pas de Calais regional council. We thank sincerely the fire brigade of North of France (SDIS 59) for their fruitful advices to define the scenarios. REFERENCES Cacciabue P.C. (2014) Designing Advanced Driver Assistance Systems in a Risk Based Process, in P. Millot (Ed) Risk Management in Life critical Systems, pp 117-156, ISTE-Wiley, London, October, ISBN: 978-1-84821-480-4 Hoc, J-M., Pacaux-Lemoine, M.-P. (1998). Cognitive evaluation of human-human and human machine cooperation modes in air traffic control. International Journal of Aviation Psychology, 8, 1-32. Inagaki, T., Sheridan, T. (2008). Authority and responsibility in Human-machine systems: is machine-initiated trading of authority permissible in the human-centered automation framework, Proceeding of Applied Human Factors and Ergonomics. Kaber D., Endsley M. (2003). The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task, Theoretical Issues in Ergonomics Science ISSN 1463–922X print/ISSN 1464–536X online, 1-40 DOI 10.1080/1463922021000054335 Millot P., Mandiau R. (1995). Men-Machine Cooperative Organizations: Formal and Pragmatic implementation methods. In J.M. Hoc, P.C. Cacciabue, E. Hollnagel Editors, Expertise and Technology : Cognition Computer Cooperation, Lawrence Erlbraum Associates, New Jersey , chap. 13, pp. 213-228, 199. Millot P., Hoc JM., (1997). Human-Machine Cooperation : Metaphor or possible reality ? Proceedings of the 2nd European Conference on Cognitive Science ECCS’97, pp165-174. Manchester, U.K. Millot P., Lemoine MP., (1998). An attempt for generic concepts toward human-machine cooperation. IEEE SMC'98 Conference. San Diego, USA, October. Millot, P., Boy, G. (2012) Human-machine cooperation: a solution for life-critical Systems?, Work: A Journal of Prevention, Assessment and Rehabilitation, IOS Press, volume 41, pages 4552-4559 DOI: 10.3233/WOR-20120033-4552 Millot P. & Pacaux-lemoine M. (2013). A common Work Space for a mutual enrichment of Human-machine Cooperation and Team-Situation Awareness. 12th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, Las Vegas, Nevada, USA, août . Millot P. (2014a). Risk management in Life Critical Systems, ISTE-Wiley, London. October 2014, 420 pages, ISBN: 978-1-84821-480-4
53