- Email: [email protected]
Adaptable Joystick Control System for Underwater Remotely Operated Vehicles*
10th IFAC Conference on Control Applications in Marine Systems 10th Control in September 13-16, 2016.on Trondheim, Norway 10th IFAC IFAC Conference Conference on Control Applications Applications in Marine Marine Systems Systems 10th IFAC Conference on Control Applications in Marine Systems September September 13-16, 13-16, 2016. 2016. Trondheim, Trondheim, Norway Norway Available online at www.sciencedirect.com September 13-16, 2016. Trondheim, Norway
ScienceDirect IFAC-PapersOnLine 49-23 (2016) 167–172
Adaptable Joystick Control System for Adaptable Joystick Control System for Adaptable Remotely Joystick Control System for Underwater Operated Vehicles Underwater Remotely Operated Vehicles Underwater Remotely ∗,∗∗ Operated Vehicles ∗ Eirik Eirik Eirik Eirik
Hexeberg Henriksen ∗,∗∗ Ingrid Schjølberg ∗ ∗,∗∗ Ingrid ∗ Hexeberg Henriksen Schjølberg ∗∗ Hexeberg Henriksen Schjølberg ∗,∗∗ Ingrid Tor Berge Gjersvik Hexeberg Henriksen Ingrid Schjølberg ∗ ∗∗ ∗∗ Tor Berge Gjersvik Tor Tor Berge Berge Gjersvik Gjersvik ∗∗ ∗ Centre for Autonomous Marine Operations and Systems ∗ for Autonomous Marine Operations and Systems ∗∗ ∗ Centre for Autonomous Marine and ∗ Centre Department of Petroleum Engineering, and Applied Geophysics Centre forofAutonomous Marine Operations Operations and Systems Systems ∗∗ ∗∗ Department Petroleum Engineering, and Applied Geophysics Department of Petroleum Engineering, and Applied Geophysics ∗∗ Norwegian University of Science and Technology, NO-7491, Department of Petroleum Engineering, and Applied Geophysics Norwegian University of Science and Technology, NO-7491, Norwegian University of Science and Technology, NO-7491, Trondheim, Norway Norwegian University of Science and Technology, NO-7491, Trondheim, Norway Trondheim, (e-mail:[email protected], [email protected], Trondheim, Norway Norway (e-mail:[email protected], [email protected], (e-mail:[email protected], [email protected], [email protected]) (e-mail:[email protected], [email protected], [email protected]) [email protected]) [email protected]) Abstract: Current commercial remotely operated vehicles (ROVs) used for inspection, mainteAbstract: Current commercial (ROVs) used for inspection, mainteAbstract: Current commercial remotely operated vehicles (ROVs) used for inspection, maintenance and repair tasks of subsea remotely petroleumoperated facilitiesvehicles are operated with a low of automation. Abstract: Current commercial remotely operated vehicles (ROVs) used for level inspection, maintenance and repair tasks of subsea petroleum facilities are operated with aaoperators low level of automation. nance and repair tasks of subsea petroleum facilities are operated with low level of automation. Precise and efficient operation of the vehicles is hard, and the vehicle need extensive nance and repair tasks of subsea petroleum facilities are operated with a low level of automation. Precise and efficient operation of theAvehicles is hard, and the vehicle system operators Precise and efficient operation of is and the operators need extensive training to operate efficiently. properly designed automation hasneed the extensive potential Precise and efficientthese operation of the theAvehicles vehicles is hard, hard, andautomation the vehicle vehicle system operators need extensive training to operate these efficiently. properly designed has the potential training to operate these efficiently. A properly designed automation system has the potential to lower the required skill and experience level for the operator, increase operation efficiency training to operate these efficiently. A properly designed automation system has the potential to lower the required skill and experience the operator, increase operation to lower the skill and level for the increase efficiency and counteract operator fatigue. This paperlevel usesfor theory from development of human efficiency centered to lower the required required skill and experience experience level for the operator, operator, increase operation operation efficiency and counteract operator fatigue. This paper uses theory from development of human centered and counteract operator fatigue. This paper uses theory from development of human centered automation (HCA) in the aviation industry to propose a new human centered control system and counteract operator fatigue. This paper uses theory from development of human centered automation (HCA) in the aviation industry to propose a new human centered control system automation (HCA) in the aviation industry to propose a new human centered control enabling shared control of ROVs. automation (HCA) in the aviation industry to propose a new human centered control system system enabling shared control of ROVs. enabling shared control of ROVs. The control system is implemented in a simulator and evaluated qualitatively. The human enabling shared control of ROVs. The control system is implemented aa simulator and evaluated qualitatively. human The control system is in and qualitatively. The human centered control system includes fourin of operation; position control, objectThe of interest The control system is implemented implemented inmodes a simulator simulator and evaluated evaluated qualitatively. The human centered control system includes four modes of operation; position control, object of interest centered control system includes four modes of operation; position control, object of interest orbit control, autopilot mode, and waypoint guidance mode. The main contributions of this centered control system includes four modes of operation; position control, object of interest orbit control, autopilot mode, and waypoint guidance mode. The main contributions of this orbit control, autopilot mode, and waypoint guidance mode. The main contributions of work are as follows: a human centered approach in ROV control system design, development orbit control, autopilot mode, centered and waypoint guidance mode. Thesystem main contributions of this this work are as follows: a human approach in ROV control design, development work are as follows: a human approach in ROV control design, development of a reference velocity scaling centered for predictable position control, an system adaptive joystick deadband work are as follows: a human centered approach in ROV control system design, development of a reference velocity scaling for predictable position control, adaptive joystick deadband of a velocity scaling for predictable position an adaptive joystick deadband function, an orbit control mode a super-ellipse as basean shape. Finally, guidelines for of a reference reference velocity scaling for using predictable position control, control, an adaptive joystick deadband function, an orbit control mode using aa super-ellipse as base shape. Finally, guidelines for function, an orbit control mode using super-ellipse as base shape. Finally, guidelines for predictable control system behavior are suggested. function, an orbit control mode using a super-ellipse as base shape. Finally, guidelines for predictable control system behavior are suggested. predictable control system behavior are suggested. predictable control system behavior are suggested. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Human Centered Design, Human Machine Interface, Remote Control, Unmanned Keywords: Human Human Centered Design, Human Machine Machine Interface, Remote Remote Control, Unmanned Unmanned Keywords: Underwater Vehicle,Centered Subsea, Design, IMR Human Keywords: Human Centered Design, Human Machine Interface, Interface, Remote Control, Control, Unmanned Underwater Vehicle, Subsea, IMR Underwater Underwater Vehicle, Vehicle, Subsea, Subsea, IMR IMR 1. INTRODUCTION 1. INTRODUCTION INTRODUCTION 1. 1. INTRODUCTION Most inspection, maintenance and repair (IMR) operaMost on inspection, maintenance and repair repair (IMR)areoperaoperaMost inspection, maintenance and (IMR) tions subsea petroleum extraction facilities perMost inspection, maintenance and repair (IMR)areoperations on subsea petroleum extraction facilities pertions on subsea petroleum extraction facilities are performed using remotely operated vehicles (ROV). Curtions onusing subsea petroleum extraction facilities are Curperformed remotely operated vehicles (ROV). formed using remotely operated vehicles (ROV). Currently there over 5000 subsea vehicles Xmas Trees installed formed usingare remotely operated (ROV). Currently there are over 5000 subsea Xmas Trees installed rently are over subsea Xmas on the there seafloor the 5000 world’s oceans. AllTrees these installed need to rently areof subsea Xmas Trees installed on inspected the there seafloor ofover the 5000 world’s oceans. All theseThese needare to on the seafloor of the world’s oceans. All these need to be and maintained yearly by ROVs. on the seafloor ofmaintained the world’syearly oceans. All theseThese needare to be inspected and by ROVs. be inspected and maintained yearly by ROVs. These are unmanned underwater vehicles controlled from These a surface be inspected and maintained yearly by ROVs. are unmanned underwater vehicles controlled from surface unmanned underwater vehicles aaa surface vessel through an umbilical cable.controlled This classfrom of vehicles has unmanned underwater vehicles controlled from surface vessel through an umbilical umbilical cable. This class of vehicles vehicles has vessel through an cable. This class of has had a rapid development curve from the 1970s, when they vessel through an umbilical cable. This class of vehicles has had a rapid development curve from the 1970s, when they had aafirst rapid development curve from 1970s, they were introduced for work in the the oil and gaswhen industry. had rapid development curve from the 1970s, when they were first first introduced introduced ROVs for work work in the the oil have and gas gas industry. were for in oil and However used today not industry. followed were firstcommercial introduced ROVs for work in the oil have and gas However commercial used today not industry. followed However commercial ROVs used today have not followed the development of other industries when it comes to auHowever commercial ROVs used today have not followed the development of other industries when it comes to auauthe development of other industries when it comes to tomation (Offshore Engineer (2015)). ROVs are currently the development of Engineer other industries when it are comes to automation (Offshore (2015)). ROVs currently tomation (Offshore ROVs are operated in a directEngineer control (2015)). mode. The operator uses a tomation Engineer (2015)). ROVs are currently currently operatedto(Offshore incontrol direct control mode.byThe The operator uses aa operated in aa direct control mode. operator uses joystick forces produced the ROV-propellers, operated in a direct control mode. The operator uses a joystick to control control forces forces produced by the the ROV-propellers, ROV-propellers, joystick to produced by and a master-arm is used for controlling the position of joystick to control forces produced by the ROV-propellers, and manipulator a master-arm master-armarms is used used for controlling controlling the position position of and a is for the of the (slave-arm). IMR operations often and a master-armarms is used for controlling the position of the manipulator (slave-arm). IMR operations often the manipulator arms (slave-arm). operations often require at least two operators (ROVIMR pilots). These pilots the manipulator arms (slave-arm). IMR operations often require at least least two operators (ROV pilots). These pilots require at operators pilots). These need extensible training to be(ROV able to control the pilots ROV require at least two two operators pilots). These pilots need extensible extensible training tosituational be(ROV able to to control the ROV need training to be able control the ROV efficiently, and a persistent awareness is critical. need extensible training to be able to control the ROV efficiently, and a persistent situational awareness is critical. efficiently, and aa persistent situational awareness is critical. efficiently, and persistent situational awareness is critical. This work is supported by the Research Council of Norway,
Introducing more automation in the ROV control system Introducing more automation the ROV control system Introducing more automation in the control system has the potential relieve thein from the tedious Introducing more to automation inoperator the ROV ROV control system has the potential to relieve the operator from the tedious has the potential to relieve the operator from the tedious task of manual control during long operations. This will has the potential to relieve the operator from the tedious task of manual manual control during long operations. Thiswork will task of control during long operations. This will counteract operator fatigue and make the operator task of manual control during long operations. Thiswork will counteract operator fatigue and make the operator counteract operator fatigue and make the operator work faster with less training (Schjølberg and Utne, 2015). counteract operator fatigue and make the operator work faster with less training (Schjølberg and Utne, 2015). faster with training (Schjølberg and Utne, 2015). faster withofless less andshared Utne, control 2015). for The goal thetraining work is(Schjølberg to introduce The goal of the work is to introduce shared control for The goal of the work is to introduce shared control for ROVs performing IMR operations, making the operations The goal of the work isoperations, to introduce shared control for ROVs performing IMR making the operations ROVs performing IMR operations, making the operations faster, safer and easier. ROVs performing IMR operations, making the operations faster, safer and faster, and easier. easier. faster, safer safer easier. Yoerger et al.and (1986) introduce the idea of relating joystick Yoerger et al. (1986) introduce the idea of and relating Yoerger et al. (1986) introduce the idea relating joystick command to different reference use joystick this to Yoerger et al. (1986) introduce theframes idea of of and relating joystick command to different reference frames use thisthat to command to different reference frames and use to propose different control modes. The work showed command to different reference frames and showed use this thisthat to propose different control modes. The work propose different control modes. The work showed that performance is improved with a closed loop control system propose different control modes. The work showed that performance is improved with a closed loop control system performance improved with aa closed loop control using shared is supervisory was performance isor improved withcontrol. closedThis loop evaluation control system system using shared or supervisory control. This evaluation was using shared or supervisory control. This evaluation was based on simulation with a pilot in the loop. This very using shared or supervisory control. This evaluation was based on simulation with a pilot in the loop. This very based on simulation with a pilot in the loop. This very early work did not consider human performance, and the based on simulation with a pilot in the loop. This very early work did not consider performance, and the early work did not consider human performance, and the conclusion was based on the human track following capabilities of early work did not consider human performance, and the conclusion was based on the track following capabilities of conclusion was based on the track following capabilities of the control system. Dukan and Sørensen (2012) focused on conclusion was based on the track following capabilities of the control system. Dukan and Sørensen (2012) focused on the control system. Dukan and Sørensen (2012) focused on methods for relating joystick commands to ROV motions the control system. Dukan and Sørensen (2012) focused on methods for relating to ROV motions methods for joystick commands to motions and references. Three joystick differentcommands schemes were proposed and methods for relating relating joystick commands to ROV ROV motions and references. Three different schemes were proposed and and references. Three different schemes were proposed and experimentally tested.different Two of schemes the methods use a filtered and references. Three were proposed and experimentally tested. Two of the methods use a filtered experimentally tested. Two of the methods use a filtered joystick command as a velocity reference, one is integrated experimentally tested. Two of the methods use a filtered joystick command as a velocity one is integrated joystick command as reference, one to a position reference and one reference, is used directly. The third joystick command as aa velocity velocity reference, one is is integrated integrated to a position reference and one is used directly. The third to a position reference and one is used directly. The method relates the filtered joystick command directly to to a position reference and one is used directly.directly The third third method relates the filtered joystick command to method relates the filtered joystick command directly to thrust forces onthe thefiltered ROV. joystick The control system switches method relates command directly to thrust forces on the ROV. The control system switches thrust forces on the ROV. The control system switches to position control when the velocity or thrust reference thrust forces on the ROV. The control system switches to position control when the velocity or thrust reference to to position position control control when when the the velocity velocity or or thrust thrust reference reference
This and work is by the Council Norway, Statoil FMC Technologies the project Next of Generation work is supported supported bythrough the Research Research Council of Norway, This This and work is supported bythrough the Research Council of Norway, Statoil FMC Technologies the project Next Generation Subsea Inspection, Maintenancethrough and Repair, 234108/E30. The work Statoil and FMC Technologies the project Next Generation Statoil and FMC Technologies through the project Next Generation Subsea Inspection, Maintenance and Repair, 234108/E30. The is associated with CoE AMOS, 223254. Subsea Inspection, Maintenance and Repair, 234108/E30. The work work Subsea Inspection, Maintenance and Repair, 234108/E30. The work is associated with CoE AMOS, 223254. is associated with CoE AMOS, 223254. is associated with CoE AMOS, 223254. Copyright © 2016, 2016 IFAC 167Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 IFAC 167 Copyright ©under 2016 responsibility IFAC 167Control. Peer review of International Federation of Automatic Copyright © 2016 IFAC 167 10.1016/j.ifacol.2016.10.338
2016 IFAC CAMS 168 Sept 13-16, 2016. Trondheim, NorwayEirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172
is zero. This is to keep the vehicle in a fixed position. Candeloro et al. (2015) takes an alternative approach to ROV-control by using a head mounted display with internal motion sensors to control an ROV. This work develops a human in the loop (HITL) control system to be used for subsea IMR tasks. An ROV control system architecture for IMR operations is proposed. The architecture is developed using theory from HCA originating in the aviation industry. The different modes in the architecture include concepts from previous research. To investigate the usability, the control system was implemented in a simulator and tested with an operator in the loop. This testing was done in a qualitative manner with focus on user friendly operation of the ROV, and revealed weaknesses in the system. The presented work analyses these weaknesses, and proposes solutions for making the system more user friendly and robust. The contributions in the presented work are to bring in theory from HCA from the aviation industry to ROV control system design. This is used as a basis for proposing design guidelines for the ROV control system. The theory and guidelines form the fundament for a new ROV control architecture with a higher level of automation than what exists in current commercial ROVs. The control architecture is novel and important as large cost savings are expected due to increased efficiency in IMR operations. Operator friendliness has been an important aspect in the design, and has led to the following contributions: • A velocity scaling function to avoid joystick wind-up • Adaptive joystick deadband for straight line manouvers • An orbit control mode for inspection of subsea equipment with a rectangular footprint This paper is organized as follows: Section 2 gives a short synopsis of HCA, and HITL control systems. Section 3 describes the different control modes and high level architecture of the control system. The control modes are presented in Section 4 and 5. Section 6 concludes the work. 2. GUIDELINES AND HUMAN IN THE LOOP This section will give a synopsis of the concepts relevant for this work. Starting with a short introduction to HCA, and then an overview of the concept of having a human in the control loop, and distinguishing this from an automated feedback loop. 2.1 Guidelines for Design and Implementation Designers of new automation systems often have a tendency to be too technology centered, trying to automate every part of the system (Norman, 1990; Sheridan, 2001). However, there are often tasks that are either too difficult or too expensive to automate. A human operator is therefore needed to monitor the system, to take over in case of an incident and to perform the tasks that are not automated. A more thoughtful approach to function allocation is the compensatory principle. In this approach, tasks are allocated to man or machine according to an assumption about who is best qualified to perform the task. This method originates from Fitts list (Fitts, 1951), 168
which in spite of its age and criticism has persisted through history (Winter and Dodou, 2014). Both the technology centered approach and function allocation tend to lead to problems in the human-machine relation, described as the ironies of automation in Bainbridge (1983). There is literature to be found on design principles of HCA, especially in the aviation domain. However, when it comes to functional design and implementation of a HITL motion control system there is a need for more tangible guidelines. Billings (1996) and Atoyan et al. (2006) provide guidelines for designing a HCA system. Such system can be viewed as a three piece system; the human operator, the automation system and the human-machine interface. HCA is a term often describing automation interacting with humans, where the goal is to optimize the overall performance of the system (Sheridan, 1995). This includes handling of both automation and human errors. Shared control is a similar term that most often is used to describe an automation system where the control is shared between an automation system and a human operator. The presented work is focusing on the development of a control architecture supporting shared control for subsea IMR tasks. While many of the guidelines from the literature are related to training of the operator, and design of the human machine interface, the following four guidelines are addressing the ROV control system. These are proposed as the guidelines for designing the control system in the presented work (Billings, 1996; Atoyan et al., 2006). 1) The automated systems must be predictable 2) Provide the user with adaptable automation 3) The automated systems must also monitor the human operators 4) The automated system must be comprehensible to pilots Guideline number two suggests that the operator can change the level of automation during mission execution. This switch between control modes must be intuitive and easy. To achieve this, the control system should be stable during a switch between control modes (guideline 1). A set of more detailed guidelines for predictable control system behavior during operation is proposed: 1a) The velocity and position references during switching should be continuous 1b) A command from the operator, should always lead to a vehicle response 1c) The same operator command should lead to similar response, despite of the current operating mode 1d) If a stop is commanded, the vehicle should stop as fast as possible, without reversing 2.2 Human in the Loop and Joystick Control HITL control refers to the situation where a human is present in the control loop, fulfilling one or several control functions. The classic (closed) control loop consists of a controller and a process. The controller receives feedback from the process, and controls the actuators to drive it to the desired set-point. In ROV-control a human typically takes the place of the controller, and use video feedback from the vehicle to control the thrusters on the vehicle. A block diagram for such control loop can be seen in
2016 IFAC CAMS Sept 13-16, 2016. Trondheim, NorwayEirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172
169
Fig. 1. Human in the Loop in traditional ROV control Figure 1. Many ROV manufacturers do however supply their vehicles with “auto-functions” for heading and depth. In the traditional ROV control system, the operator uses video feedback to estimate the current position of the ROV. This is denoted the perceived position. ROV operations in the high-end range often use transponders to show the ROV position in a chart-plotter. The operator uses the joystick to command a thruster force output moving the ROV towards the desired position. The precision of this control is very dependent on the skill of the operator; how well she manages to estimate the position based on video feedback, and how good her “internal model” of the dynamic response of the vehicle under actuation is. This internal model is one of the factors that distinguishes skilled operators. In the case of environmental loads such as currents, the operator need to constantly account for these. Varying environmental forces are acting on the ROV, and its cable. Human perception of these forces are only possible by coupling the motions observed in a video-image and applied thrust with the pilots internal model of motion. This is a challenging task, and makes the performance of direct control sub-optimal. This is a motivation for introducing automation. An automated control loop will be faster and more precise as a computer is capable of receiving and processing sensor feedback faster than a human. The human is however indispensable in mission planning, and execution because of the cognitive abilities necessary for replanning, equipment diagnostics and situational awareness. The question is then how to best involve the human in the control loop, while at the same time utilizing automated control and navigation. 3. SYSTEM ARCHITECTURE In the following an adaptable shared control system enabling several control modes is presented. The overall architecture with four operating modes is illustrated in Figure 2. The different control modes are realized by switching between control loops. References for the control loops can be generated either by system, human, or a combination of these. The control system is described with respect to a typical offshore inspection consisting of several phases:
Fig. 2. Adaptable shared control system architecture enabling different operating modes The main objective of an inspection mission is videocamera based visual inspection. In this phase the orientation of the ROV is important, as the video camera must be facing the subsea equipment. Two different modes for inspection is proposed, position and orbit control. Both are using a 4-DOF position controller. The controller is controlling north and east position, as well as depth/altitude and heading. The difference between the modes is that the reference is given by the joystick in two different ways. In the first mode (position control) the output from the joystick is interpreted as a velocity reference in the vehicle frame. This velocity reference is then fed into a position reference filter to get the position. The other control mode is an orbit control mode. In this mode the joystick output is mapped to a cylindrical-like coordinate system where the object of interest is placed at the origin. In this mode the ROV will always face towards the center of the object of interest, while moving around it. The joystick is used to control the radius of the orbit, the angle of observation and the depth. The four control modes that are suggested are meant to extend the current control system of the ROV where the joystick motions are mapped directly to a thrust output on the ROV. It is important when designing a human centered control system that it is both easy for the operator to engage the automatic control system, as well as to intervene and do corrections. In the proposed design the operator should be able to easily switch between all control modes, according to guideline 2. It is important that the vehicle acts according to the operators anticipation during a mode change. This is reflected in guideline 1a. The proposed design is found as a block diagram in Figure 2. The dashed lines represent switch options. 3.1 Implementation Aspects
• Transit from A to B • Inspection of subsea equipment
In the transit phase a suitable control scheme will be an autopilot, controlling the heading, velocity and depth/altitude of the ROV. The reference from this controller can either come directly from the operator via the joystick, or by an automated guidance system with preprogrammed waypoints. 169
The control system proposed in this article is implemented and tested in a simulator using “perfect feedback”. In the simulation the ROV moved in a 3D virtual world, and was controlled by an operator. The 3D world is shown in Figure 6. The sensor-data used by the control algorithms are unbiased and without noise. This is however not the situation in a real subsea IMR operation. It is assumed that in a
2016 IFAC CAMS 170 Sept 13-16, 2016. Trondheim, NorwayEirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172
real scenario, the ROV will be equipped with a sensor suite and navigation filters, together capable of estimating the velocity and position of the ROV without drift in the estimates. Such drift would cause the ROV to drift correspondingly while stationkeeping. This would cause an unpredictable behavior where the operator would need to intervene in the automatic execution. The estimates do not need to be globally corrected, as the operator is working on the basis of camera feedback given in a local frame. The system is comprised of a set of control and guidance algorithms that individually are stable. Switching between the modes are initiated by the human operator. This implies that the switching happens relatively slowly. It is therefore assumed that the switching between modes does not introduce any unstable behavior. 4. TRANSIT MODES The autopilot and waypoint modes shown in Figure 2 are used when the ROV is in transit between work locations. These modes are controlling the vehicle while the operator can rest and observe the operation. The automatic controller is taking care of the heading, forward speed, and depth/altitude. When waypoint mode is used, the heading reference is made by the waypoint guidance system. In the autopilot mode the heading reference is controlled by using the joystick, and the operator takes the role as a guidance system. Both modes use the autopilot controller for allocating thrust to the ROV. The waypoint control loop is illustrated in green and the autopilot loop in blue in Figure 2. The autopilot controller is a PID-controller with a feed-forward velocity reference. The waypoint guidance scheme is a lookahead based line-of-sight steering law from Fossen (2011). If the operator wants to step out of the waypoint mode, e.g. to take a closer look at something odd along the route, the controller saves the current position as a waypoint. This allows the operator to easily go back to the predefined route even after the aborting the execution. This feature is meant to lower the threshold for the operator to intervene in the automatic execution. This adaptable behavior is according to guideline 2. 5. POSITIONING MODES During a subsea inspection the operator control objective is position control, and the position mode or orbit mode in Figure 2 is used. External disturbances are acting on the ROV so continuous control action is needed to keep the ROV at the desired position. The proposed position controller enables the operator to change position and orientation using the Joystick, and let the control system handle the force allocation to keep the vehicle on position. The designed position control system will account for external disturbances and map joystick inputs to ROV movements in an intuitive way. Both in position and orbit mode the joystick deflection is interpreted as a desired velocity. This velocity is then transformed and integrated to a position reference. In the position mode the velocity is in the vehicle reference frame and is directly transformed into the global reference frame (red control loop in Figure 2). In the orbit mode the velocity is given in a cylindrical 170
like coordinate system where the center of the orbit is placed in the origin (yellow loop in Figure 2). In order to make the position and velocity references continuous during a switch between modes, the mode that are being switched to is initialized using the position, and velocity from the current mode. 5.1 Position mode A nonlinear PID-controller with a feed forward reference is applied for position control. This is a well known and relatively simple controller. This corresponds with guideline 4, calling for a comprehensible control system. For position reference generation the approach proposed by Dukan and Sørensen (2012) is used. This is a low-pass filter cascaded with a mass, damper and spring system applied to the joystick output. The filter output is a smooth position reference as well as a reference velocity used as a feed-forward to the position controller. 5.2 Orbit Control Mode The orbit control mode enables the operator to circle around an object of interest during an inspection. This maneuver is difficult to perform using manual control because it changes the heading and position simultaneously. The idea of referencing joystick commands to a coordinate system using cylindrical coordinates is described in Yoerger et al. (1986). Simulator testing revealed that a cylindrical orbit was very handy when inspecting small objects (approximated by points), or larger objects with a circular base. However, for the offshore inspection scenario, most objects are quite large and rectangular in shape. This made the circular orbit unpractical as the distance to the object change a lot during one lap. To adapt the orbit for rectangular objects, it is proposed to change the base shape from a circle to a super-ellipse. The difference in distance between path and object can be seen in Figure 3. The standard parametrization of the super-ellipse results in an along-path velocity that is varying considerably. This makes it unsuitable to be used as a path. A new parametrization that solves this problem and results in constant along-path velocity is proposed. This is done to make the behavior more predictable (guideline 1). The geometric shape of the super-ellipse is documented in Weisstein (2016). This base shape has several shaping parameters, making it possible to generate a suitable path for the ROV to orbit around a subsea structure. The super-ellipse shape is can be described by the following equations: x(φ) = rA cos2/s (φ) 2/s
(1a)
(1b) y(φ) = rB sin (φ) Where A, and B scale the shape along the x-, or y-axis, while r sets the size of the shape. φ is the control parameter. The parameter s is sometimes called the squareness parameter and for s > 2 it controls the squareness of the shape. s = 1 produces a diamond shape, while s = 2 produces an ellipse, or a circle if A = B. Using this base shape will in other words give the possibility to adapt the orbit path to a lot of different shaped equipment. The base shape can be rotated by using a 2D rotation matrix, and
2016 IFAC CAMS Sept 13-16, 2016. Trondheim, NorwayEirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172
171
4
6
Structure Outline Circular Orbit Superellipse Orbit
5
No Velocity scaling Velocity scaling
3.5
4 3
Position error [m]
North [m]
3 2 1 0 -1 -2
2.5 2 1.5 1
-3 0.5
-4 -4
-2
0
2
4
6
0 0
East [m]
5
10
15
20
25
30
35
40
45
Time [s]
Fig. 3. Circular and Super-ellipse path around a rectangular subsea structure placed in the North-East plane by adding the coordinates of the desired origin of the shape. The r parameter is controlled in real-time by the operator by pushing the joystick forwards or backwards. This corresponds to moving closer or away from the structure, or moving the ROV forward and backward. When the joystick is moved sideways, the vehicle will move along the structure following the path. When using a cylinder as a base shape it was practical to control this motion along the path by using the angle parameter φ. For a cylinder the relation between one step along the path was proportional to one angle step. For the super-ellipse however, this does not hold as the along-path velocity is dependent on φ. A velocity unproportional to joystick input makes it very hard to control the vehicle. The proposed solution is to map the joystick y-axis to the along-path speed and then calculate the a new angle parameter based on this speed. This was done by substituting the variable θ for ω in (1). The new path variable is calculated numerically using the following relations: s = x (φ)2 + y (φ)2 (2) t Ud dt (3) ω= 0 s where Ud is the commanded along-path velocity, x (φ) and y (φ) is defined in (1). The denominator in (3) is infinity at the x-, and y-axis of the shape, causing the fraction to go towards zero. This is solved in the implementation by saturating the denominator at a high threshold. This may cause the along-path velocity to be slightly higher at one timestep when crossing the x- and y-axis. Simulator testing shows that this is not noticeable by the operator. 5.3 Joystick Wind-up Testing revealed that in presence of currents the vehicle may move slower than the filter reference. This may be hard for the operator to notice, and may cause her to lose control over the vehicle. We named this problem joystick wind-up because of its similarities to integrator wind-up. The reference filter is tuned so the maximum reference velocity is slightly less than the ROV maximum velocity. 171
Fig. 4. Position error with, and without velocity scaling during a move towards the current However, the reference filter does not take into account current forces. When the vehicle is travelling countercurrent, the vehicle will lag behind the joystick generated reference. This is because the ROV can not sustain the speed set by the reference filter. If the vehicle is far from the setpoint this might cause confusion for the operator as he/she does not get the expected feedback when moving the joystick. The controller is chasing a position far away from the vehicle and the vehicle will move in this direction at maximum possible speed, no matter if the operator changes the setpoint slightly. This is the same problem that can lead to integral wind-up in PID-controllers. In order to mitigate this issue, a lag based velocity scaling function is proposed for the reference filter. The idea is to multiply the reference speed with a scaling-factor when a lag is detected. The scaling parameter is calculated and applied in the global frame, as the current-forces are acting in this frame. The scaling factor, S is calculated as follows: 1 if |ηdi − ηi | ≤ 1m 0.5 (4) Si = if ηdi − ηi | > d |ηdi − ηi |
d is the lag treshold and S = Diag(S1 , S2 , S3 , 1, 1, 1) as we only apply scaling to the translational degrees of freedom. Figure 4 shows how the velocity scaling ensures that the position error stays within a reasonable bound, making the vehicle controllable by the joystick, following guideline 1b. 5.4 Adaptive Deadband When the operator attempts to move the vehicle at high velocity and along a straight line using the joystick, it is hard to keep the vehicle moving along the straight line. The explanation is that when making a somewhat coarse movement in forward direction to obtain a high speed, it is hard to avoid small movements in the transverse directions. To counteract such unintended commands it is possible to introduce a deadband on the joystick output. A regular deadband however tend to remove precision for small deflections of the joystick. To keep precision and counteract unintended motions in high speed an adaptive deadband is proposed. This deadband, D is calculated as:
2016 IFAC CAMS 172 13-16, 2016. Trondheim, Norway Eirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172 Sept
D=k
6
ηijs
(5)
i=1
where k is a tunable parameter controlling the size of the adaptive deadband envelope. The deadband is applied to the joystick output as follows: 0 if ηi < D js ηi = (6) ηijs if ηi ≥ D When the deadband is applied to the joystick output before it enters the reference filter it is significantly easier to make a straight line reference for the ROV at high speeds. Figure 5 shows two test-runs where the operator wants the ROV to move at high speed in a straight line. The figure shows a clear improvement in the horizontal plane, but also applies to a path in the vertical direction. -1 With Adaptive deadband No Deadband
North
-2
-4
-6
-4
-2
0
East
2
4
IMR operations, making the operations faster, safer and easier. In a future scenario we might see highly automated vehicles remotely operated from shore performing IMR operations subsea. REFERENCES
-3
-5
Fig. 6. Operators view during testing of control system
6
Fig. 5. Position in the North-East plane during a straight line move with, and without adaptive deadband 6. CONCLUSIONS Transit and position control modes for an ROV in IMR operation are described in the previous sections. A control system architecture is designed to enable adaptable shared control, allowing to switch between several modes of operation. The modes are designed to make the job for the human operator easier during an IMR operation. Less operator attention and effort is required as the ROV controller handles the external disturbances, as well as aiding in path following. For instance, the orbit control mode allows the operators to automatically observe the subsea equipment from all angles. The control system allows the operator to focus on the actual work, such as observing, or controlling the ROV tools or arms. The architecture including four control modes is tested in a simulation environment, using the flexibility of the control architecture to switch between the modes. The system is still at a testing and development stage, but a qualitative review has proved it easy to operate, even without training. Topics for further work will include implementation of a more refined human machine interface, automated collision avoidance and other operator aid functionality for IMR operations. The system will be further verified by laboratory experiments. Guidelines for design of human centered automation for aviation was used in the design phase. The guidelines suggested that the system should include adaptable automation and that it should be predictable and comprehensive for the human operators. The last guideline suggested that the automation system should also monitor the human to avoid errors. Guidelines to ensure predictable behavior of the system are proposed. The goal of the work is to introduce human friendly automation for ROVs performing 172
Atoyan, H., Duquet, J.R., and Robert, J.M. (2006). Trust in new decision aid systems. In Proceedings of the 18th Conference on L’Interaction Homme-Machine, IHM ’06, 115–122. ACM, New York, NY, USA. Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. Billings, C.E. (1996). Human centered aviation automation: Principles and guidelines. Technical Report NASA TM 110381, NASA Ames Reasearch Centre. Candeloro, M., Valle, E., Miyazaki, M.R., Skjetne, R., Ludvigsen, M., and Srensen, A.J. (2015). Hmd as a new tool for telepresence in underwater operations and closed-loop control of rovs. In Proc. MTS/IEEE OCEANS. Dukan, F. and Sørensen, A.J. (2012). Joystick in closedloop control of rovs with experimental results. Navigation, Guidance and Control of Underwater Vehicles, 3, 293–300. Fossen, T.I. (2011). Handbook of Marine Craft Hydrodynamics and Motion Control. John Wiley & Sons, Ltd. Norman, D.A. (1990). The problem of automation: Inappropriate feedback and interaction, not overautomation. Philosophical Transactions of the Royal Society of London, 585 – 593. Offshore Engineer (2015). Sea change. Offshore Engineer, October 2015, 50–52. Schjølberg, I. and Utne, I.B. (2015). Towards autonomy in ROV operations. 4th IFAC Workshop on Navigation, Guidance and Controlof Underwater Vehicles, NGCUV 2015, 48(2), 183 – 188. Sheridan, T.B. (1995). Human centered automation: oxymoron or common sense? IEEE International Conference on Systems, Man and Cybernetics, 1995. Intelligent Systems for the 21st Century., 1, 823 – 828. Sheridan, T.B. (2001). Rumination on automation, 1998. Annual Reviews in Control, 25, 89 – 97. Weisstein, E.W. (2016). Superellipse. In MathWorld. URL http://mathworld.wolfram.com/Superellipse.html. Winter, J.C.F. and Dodou, D. (2014). Why the fitts list has persisted throughout the history of function allocation. Cognition, Technology & Work, 16(1), 1–11. Yoerger, D., Newman, J.B., and Slotine, J.J.E. (1986). Supervisory control system for the jason rov. IEEE Journal of Oceanic Engineering, 11, 392 – 400.
ScienceDirect IFAC-PapersOnLine 49-23 (2016) 167–172
Adaptable Joystick Control System for Adaptable Joystick Control System for Adaptable Remotely Joystick Control System for Underwater Operated Vehicles Underwater Remotely Operated Vehicles Underwater Remotely ∗,∗∗ Operated Vehicles ∗ Eirik Eirik Eirik Eirik
Hexeberg Henriksen ∗,∗∗ Ingrid Schjølberg ∗ ∗,∗∗ Ingrid ∗ Hexeberg Henriksen Schjølberg ∗∗ Hexeberg Henriksen Schjølberg ∗,∗∗ Ingrid Tor Berge Gjersvik Hexeberg Henriksen Ingrid Schjølberg ∗ ∗∗ ∗∗ Tor Berge Gjersvik Tor Tor Berge Berge Gjersvik Gjersvik ∗∗ ∗ Centre for Autonomous Marine Operations and Systems ∗ for Autonomous Marine Operations and Systems ∗∗ ∗ Centre for Autonomous Marine and ∗ Centre Department of Petroleum Engineering, and Applied Geophysics Centre forofAutonomous Marine Operations Operations and Systems Systems ∗∗ ∗∗ Department Petroleum Engineering, and Applied Geophysics Department of Petroleum Engineering, and Applied Geophysics ∗∗ Norwegian University of Science and Technology, NO-7491, Department of Petroleum Engineering, and Applied Geophysics Norwegian University of Science and Technology, NO-7491, Norwegian University of Science and Technology, NO-7491, Trondheim, Norway Norwegian University of Science and Technology, NO-7491, Trondheim, Norway Trondheim, (e-mail:[email protected], [email protected], Trondheim, Norway Norway (e-mail:[email protected], [email protected], (e-mail:[email protected], [email protected], [email protected]) (e-mail:[email protected], [email protected], [email protected]) [email protected]) [email protected]) Abstract: Current commercial remotely operated vehicles (ROVs) used for inspection, mainteAbstract: Current commercial (ROVs) used for inspection, mainteAbstract: Current commercial remotely operated vehicles (ROVs) used for inspection, maintenance and repair tasks of subsea remotely petroleumoperated facilitiesvehicles are operated with a low of automation. Abstract: Current commercial remotely operated vehicles (ROVs) used for level inspection, maintenance and repair tasks of subsea petroleum facilities are operated with aaoperators low level of automation. nance and repair tasks of subsea petroleum facilities are operated with low level of automation. Precise and efficient operation of the vehicles is hard, and the vehicle need extensive nance and repair tasks of subsea petroleum facilities are operated with a low level of automation. Precise and efficient operation of theAvehicles is hard, and the vehicle system operators Precise and efficient operation of is and the operators need extensive training to operate efficiently. properly designed automation hasneed the extensive potential Precise and efficientthese operation of the theAvehicles vehicles is hard, hard, andautomation the vehicle vehicle system operators need extensive training to operate these efficiently. properly designed has the potential training to operate these efficiently. A properly designed automation system has the potential to lower the required skill and experience level for the operator, increase operation efficiency training to operate these efficiently. A properly designed automation system has the potential to lower the required skill and experience the operator, increase operation to lower the skill and level for the increase efficiency and counteract operator fatigue. This paperlevel usesfor theory from development of human efficiency centered to lower the required required skill and experience experience level for the operator, operator, increase operation operation efficiency and counteract operator fatigue. This paper uses theory from development of human centered and counteract operator fatigue. This paper uses theory from development of human centered automation (HCA) in the aviation industry to propose a new human centered control system and counteract operator fatigue. This paper uses theory from development of human centered automation (HCA) in the aviation industry to propose a new human centered control system automation (HCA) in the aviation industry to propose a new human centered control enabling shared control of ROVs. automation (HCA) in the aviation industry to propose a new human centered control system system enabling shared control of ROVs. enabling shared control of ROVs. The control system is implemented in a simulator and evaluated qualitatively. The human enabling shared control of ROVs. The control system is implemented aa simulator and evaluated qualitatively. human The control system is in and qualitatively. The human centered control system includes fourin of operation; position control, objectThe of interest The control system is implemented implemented inmodes a simulator simulator and evaluated evaluated qualitatively. The human centered control system includes four modes of operation; position control, object of interest centered control system includes four modes of operation; position control, object of interest orbit control, autopilot mode, and waypoint guidance mode. The main contributions of this centered control system includes four modes of operation; position control, object of interest orbit control, autopilot mode, and waypoint guidance mode. The main contributions of this orbit control, autopilot mode, and waypoint guidance mode. The main contributions of work are as follows: a human centered approach in ROV control system design, development orbit control, autopilot mode, centered and waypoint guidance mode. Thesystem main contributions of this this work are as follows: a human approach in ROV control design, development work are as follows: a human approach in ROV control design, development of a reference velocity scaling centered for predictable position control, an system adaptive joystick deadband work are as follows: a human centered approach in ROV control system design, development of a reference velocity scaling for predictable position control, adaptive joystick deadband of a velocity scaling for predictable position an adaptive joystick deadband function, an orbit control mode a super-ellipse as basean shape. Finally, guidelines for of a reference reference velocity scaling for using predictable position control, control, an adaptive joystick deadband function, an orbit control mode using aa super-ellipse as base shape. Finally, guidelines for function, an orbit control mode using super-ellipse as base shape. Finally, guidelines for predictable control system behavior are suggested. function, an orbit control mode using a super-ellipse as base shape. Finally, guidelines for predictable control system behavior are suggested. predictable control system behavior are suggested. predictable control system behavior are suggested. © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: Human Centered Design, Human Machine Interface, Remote Control, Unmanned Keywords: Human Human Centered Design, Human Machine Machine Interface, Remote Remote Control, Unmanned Unmanned Keywords: Underwater Vehicle,Centered Subsea, Design, IMR Human Keywords: Human Centered Design, Human Machine Interface, Interface, Remote Control, Control, Unmanned Underwater Vehicle, Subsea, IMR Underwater Underwater Vehicle, Vehicle, Subsea, Subsea, IMR IMR 1. INTRODUCTION 1. INTRODUCTION INTRODUCTION 1. 1. INTRODUCTION Most inspection, maintenance and repair (IMR) operaMost on inspection, maintenance and repair repair (IMR)areoperaoperaMost inspection, maintenance and (IMR) tions subsea petroleum extraction facilities perMost inspection, maintenance and repair (IMR)areoperations on subsea petroleum extraction facilities pertions on subsea petroleum extraction facilities are performed using remotely operated vehicles (ROV). Curtions onusing subsea petroleum extraction facilities are Curperformed remotely operated vehicles (ROV). formed using remotely operated vehicles (ROV). Currently there over 5000 subsea vehicles Xmas Trees installed formed usingare remotely operated (ROV). Currently there are over 5000 subsea Xmas Trees installed rently are over subsea Xmas on the there seafloor the 5000 world’s oceans. AllTrees these installed need to rently areof subsea Xmas Trees installed on inspected the there seafloor ofover the 5000 world’s oceans. All theseThese needare to on the seafloor of the world’s oceans. All these need to be and maintained yearly by ROVs. on the seafloor ofmaintained the world’syearly oceans. All theseThese needare to be inspected and by ROVs. be inspected and maintained yearly by ROVs. These are unmanned underwater vehicles controlled from These a surface be inspected and maintained yearly by ROVs. are unmanned underwater vehicles controlled from surface unmanned underwater vehicles aaa surface vessel through an umbilical cable.controlled This classfrom of vehicles has unmanned underwater vehicles controlled from surface vessel through an umbilical umbilical cable. This class of vehicles vehicles has vessel through an cable. This class of has had a rapid development curve from the 1970s, when they vessel through an umbilical cable. This class of vehicles has had a rapid development curve from the 1970s, when they had aafirst rapid development curve from 1970s, they were introduced for work in the the oil and gaswhen industry. had rapid development curve from the 1970s, when they were first first introduced introduced ROVs for work work in the the oil have and gas gas industry. were for in oil and However used today not industry. followed were firstcommercial introduced ROVs for work in the oil have and gas However commercial used today not industry. followed However commercial ROVs used today have not followed the development of other industries when it comes to auHowever commercial ROVs used today have not followed the development of other industries when it comes to auauthe development of other industries when it comes to tomation (Offshore Engineer (2015)). ROVs are currently the development of Engineer other industries when it are comes to automation (Offshore (2015)). ROVs currently tomation (Offshore ROVs are operated in a directEngineer control (2015)). mode. The operator uses a tomation Engineer (2015)). ROVs are currently currently operatedto(Offshore incontrol direct control mode.byThe The operator uses aa operated in aa direct control mode. operator uses joystick forces produced the ROV-propellers, operated in a direct control mode. The operator uses a joystick to control control forces forces produced by the the ROV-propellers, ROV-propellers, joystick to produced by and a master-arm is used for controlling the position of joystick to control forces produced by the ROV-propellers, and manipulator a master-arm master-armarms is used used for controlling controlling the position position of and a is for the of the (slave-arm). IMR operations often and a master-armarms is used for controlling the position of the manipulator (slave-arm). IMR operations often the manipulator arms (slave-arm). operations often require at least two operators (ROVIMR pilots). These pilots the manipulator arms (slave-arm). IMR operations often require at least least two operators (ROV pilots). These pilots require at operators pilots). These need extensible training to be(ROV able to control the pilots ROV require at least two two operators pilots). These pilots need extensible extensible training tosituational be(ROV able to to control the ROV need training to be able control the ROV efficiently, and a persistent awareness is critical. need extensible training to be able to control the ROV efficiently, and a persistent situational awareness is critical. efficiently, and aa persistent situational awareness is critical. efficiently, and persistent situational awareness is critical. This work is supported by the Research Council of Norway,
Introducing more automation in the ROV control system Introducing more automation the ROV control system Introducing more automation in the control system has the potential relieve thein from the tedious Introducing more to automation inoperator the ROV ROV control system has the potential to relieve the operator from the tedious has the potential to relieve the operator from the tedious task of manual control during long operations. This will has the potential to relieve the operator from the tedious task of manual manual control during long operations. Thiswork will task of control during long operations. This will counteract operator fatigue and make the operator task of manual control during long operations. Thiswork will counteract operator fatigue and make the operator counteract operator fatigue and make the operator work faster with less training (Schjølberg and Utne, 2015). counteract operator fatigue and make the operator work faster with less training (Schjølberg and Utne, 2015). faster with training (Schjølberg and Utne, 2015). faster withofless less andshared Utne, control 2015). for The goal thetraining work is(Schjølberg to introduce The goal of the work is to introduce shared control for The goal of the work is to introduce shared control for ROVs performing IMR operations, making the operations The goal of the work isoperations, to introduce shared control for ROVs performing IMR making the operations ROVs performing IMR operations, making the operations faster, safer and easier. ROVs performing IMR operations, making the operations faster, safer and faster, and easier. easier. faster, safer safer easier. Yoerger et al.and (1986) introduce the idea of relating joystick Yoerger et al. (1986) introduce the idea of and relating Yoerger et al. (1986) introduce the idea relating joystick command to different reference use joystick this to Yoerger et al. (1986) introduce theframes idea of of and relating joystick command to different reference frames use thisthat to command to different reference frames and use to propose different control modes. The work showed command to different reference frames and showed use this thisthat to propose different control modes. The work propose different control modes. The work showed that performance is improved with a closed loop control system propose different control modes. The work showed that performance is improved with a closed loop control system performance improved with aa closed loop control using shared is supervisory was performance isor improved withcontrol. closedThis loop evaluation control system system using shared or supervisory control. This evaluation was using shared or supervisory control. This evaluation was based on simulation with a pilot in the loop. This very using shared or supervisory control. This evaluation was based on simulation with a pilot in the loop. This very based on simulation with a pilot in the loop. This very early work did not consider human performance, and the based on simulation with a pilot in the loop. This very early work did not consider performance, and the early work did not consider human performance, and the conclusion was based on the human track following capabilities of early work did not consider human performance, and the conclusion was based on the track following capabilities of conclusion was based on the track following capabilities of the control system. Dukan and Sørensen (2012) focused on conclusion was based on the track following capabilities of the control system. Dukan and Sørensen (2012) focused on the control system. Dukan and Sørensen (2012) focused on methods for relating joystick commands to ROV motions the control system. Dukan and Sørensen (2012) focused on methods for relating to ROV motions methods for joystick commands to motions and references. Three joystick differentcommands schemes were proposed and methods for relating relating joystick commands to ROV ROV motions and references. Three different schemes were proposed and and references. Three different schemes were proposed and experimentally tested.different Two of schemes the methods use a filtered and references. Three were proposed and experimentally tested. Two of the methods use a filtered experimentally tested. Two of the methods use a filtered joystick command as a velocity reference, one is integrated experimentally tested. Two of the methods use a filtered joystick command as a velocity one is integrated joystick command as reference, one to a position reference and one reference, is used directly. The third joystick command as aa velocity velocity reference, one is is integrated integrated to a position reference and one is used directly. The third to a position reference and one is used directly. The method relates the filtered joystick command directly to to a position reference and one is used directly.directly The third third method relates the filtered joystick command to method relates the filtered joystick command directly to thrust forces onthe thefiltered ROV. joystick The control system switches method relates command directly to thrust forces on the ROV. The control system switches thrust forces on the ROV. The control system switches to position control when the velocity or thrust reference thrust forces on the ROV. The control system switches to position control when the velocity or thrust reference to to position position control control when when the the velocity velocity or or thrust thrust reference reference
This and work is by the Council Norway, Statoil FMC Technologies the project Next of Generation work is supported supported bythrough the Research Research Council of Norway, This This and work is supported bythrough the Research Council of Norway, Statoil FMC Technologies the project Next Generation Subsea Inspection, Maintenancethrough and Repair, 234108/E30. The work Statoil and FMC Technologies the project Next Generation Statoil and FMC Technologies through the project Next Generation Subsea Inspection, Maintenance and Repair, 234108/E30. The is associated with CoE AMOS, 223254. Subsea Inspection, Maintenance and Repair, 234108/E30. The work work Subsea Inspection, Maintenance and Repair, 234108/E30. The work is associated with CoE AMOS, 223254. is associated with CoE AMOS, 223254. is associated with CoE AMOS, 223254. Copyright © 2016, 2016 IFAC 167Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright © 2016 IFAC 167 Copyright ©under 2016 responsibility IFAC 167Control. Peer review of International Federation of Automatic Copyright © 2016 IFAC 167 10.1016/j.ifacol.2016.10.338
2016 IFAC CAMS 168 Sept 13-16, 2016. Trondheim, NorwayEirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172
is zero. This is to keep the vehicle in a fixed position. Candeloro et al. (2015) takes an alternative approach to ROV-control by using a head mounted display with internal motion sensors to control an ROV. This work develops a human in the loop (HITL) control system to be used for subsea IMR tasks. An ROV control system architecture for IMR operations is proposed. The architecture is developed using theory from HCA originating in the aviation industry. The different modes in the architecture include concepts from previous research. To investigate the usability, the control system was implemented in a simulator and tested with an operator in the loop. This testing was done in a qualitative manner with focus on user friendly operation of the ROV, and revealed weaknesses in the system. The presented work analyses these weaknesses, and proposes solutions for making the system more user friendly and robust. The contributions in the presented work are to bring in theory from HCA from the aviation industry to ROV control system design. This is used as a basis for proposing design guidelines for the ROV control system. The theory and guidelines form the fundament for a new ROV control architecture with a higher level of automation than what exists in current commercial ROVs. The control architecture is novel and important as large cost savings are expected due to increased efficiency in IMR operations. Operator friendliness has been an important aspect in the design, and has led to the following contributions: • A velocity scaling function to avoid joystick wind-up • Adaptive joystick deadband for straight line manouvers • An orbit control mode for inspection of subsea equipment with a rectangular footprint This paper is organized as follows: Section 2 gives a short synopsis of HCA, and HITL control systems. Section 3 describes the different control modes and high level architecture of the control system. The control modes are presented in Section 4 and 5. Section 6 concludes the work. 2. GUIDELINES AND HUMAN IN THE LOOP This section will give a synopsis of the concepts relevant for this work. Starting with a short introduction to HCA, and then an overview of the concept of having a human in the control loop, and distinguishing this from an automated feedback loop. 2.1 Guidelines for Design and Implementation Designers of new automation systems often have a tendency to be too technology centered, trying to automate every part of the system (Norman, 1990; Sheridan, 2001). However, there are often tasks that are either too difficult or too expensive to automate. A human operator is therefore needed to monitor the system, to take over in case of an incident and to perform the tasks that are not automated. A more thoughtful approach to function allocation is the compensatory principle. In this approach, tasks are allocated to man or machine according to an assumption about who is best qualified to perform the task. This method originates from Fitts list (Fitts, 1951), 168
which in spite of its age and criticism has persisted through history (Winter and Dodou, 2014). Both the technology centered approach and function allocation tend to lead to problems in the human-machine relation, described as the ironies of automation in Bainbridge (1983). There is literature to be found on design principles of HCA, especially in the aviation domain. However, when it comes to functional design and implementation of a HITL motion control system there is a need for more tangible guidelines. Billings (1996) and Atoyan et al. (2006) provide guidelines for designing a HCA system. Such system can be viewed as a three piece system; the human operator, the automation system and the human-machine interface. HCA is a term often describing automation interacting with humans, where the goal is to optimize the overall performance of the system (Sheridan, 1995). This includes handling of both automation and human errors. Shared control is a similar term that most often is used to describe an automation system where the control is shared between an automation system and a human operator. The presented work is focusing on the development of a control architecture supporting shared control for subsea IMR tasks. While many of the guidelines from the literature are related to training of the operator, and design of the human machine interface, the following four guidelines are addressing the ROV control system. These are proposed as the guidelines for designing the control system in the presented work (Billings, 1996; Atoyan et al., 2006). 1) The automated systems must be predictable 2) Provide the user with adaptable automation 3) The automated systems must also monitor the human operators 4) The automated system must be comprehensible to pilots Guideline number two suggests that the operator can change the level of automation during mission execution. This switch between control modes must be intuitive and easy. To achieve this, the control system should be stable during a switch between control modes (guideline 1). A set of more detailed guidelines for predictable control system behavior during operation is proposed: 1a) The velocity and position references during switching should be continuous 1b) A command from the operator, should always lead to a vehicle response 1c) The same operator command should lead to similar response, despite of the current operating mode 1d) If a stop is commanded, the vehicle should stop as fast as possible, without reversing 2.2 Human in the Loop and Joystick Control HITL control refers to the situation where a human is present in the control loop, fulfilling one or several control functions. The classic (closed) control loop consists of a controller and a process. The controller receives feedback from the process, and controls the actuators to drive it to the desired set-point. In ROV-control a human typically takes the place of the controller, and use video feedback from the vehicle to control the thrusters on the vehicle. A block diagram for such control loop can be seen in
2016 IFAC CAMS Sept 13-16, 2016. Trondheim, NorwayEirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172
169
Fig. 1. Human in the Loop in traditional ROV control Figure 1. Many ROV manufacturers do however supply their vehicles with “auto-functions” for heading and depth. In the traditional ROV control system, the operator uses video feedback to estimate the current position of the ROV. This is denoted the perceived position. ROV operations in the high-end range often use transponders to show the ROV position in a chart-plotter. The operator uses the joystick to command a thruster force output moving the ROV towards the desired position. The precision of this control is very dependent on the skill of the operator; how well she manages to estimate the position based on video feedback, and how good her “internal model” of the dynamic response of the vehicle under actuation is. This internal model is one of the factors that distinguishes skilled operators. In the case of environmental loads such as currents, the operator need to constantly account for these. Varying environmental forces are acting on the ROV, and its cable. Human perception of these forces are only possible by coupling the motions observed in a video-image and applied thrust with the pilots internal model of motion. This is a challenging task, and makes the performance of direct control sub-optimal. This is a motivation for introducing automation. An automated control loop will be faster and more precise as a computer is capable of receiving and processing sensor feedback faster than a human. The human is however indispensable in mission planning, and execution because of the cognitive abilities necessary for replanning, equipment diagnostics and situational awareness. The question is then how to best involve the human in the control loop, while at the same time utilizing automated control and navigation. 3. SYSTEM ARCHITECTURE In the following an adaptable shared control system enabling several control modes is presented. The overall architecture with four operating modes is illustrated in Figure 2. The different control modes are realized by switching between control loops. References for the control loops can be generated either by system, human, or a combination of these. The control system is described with respect to a typical offshore inspection consisting of several phases:
Fig. 2. Adaptable shared control system architecture enabling different operating modes The main objective of an inspection mission is videocamera based visual inspection. In this phase the orientation of the ROV is important, as the video camera must be facing the subsea equipment. Two different modes for inspection is proposed, position and orbit control. Both are using a 4-DOF position controller. The controller is controlling north and east position, as well as depth/altitude and heading. The difference between the modes is that the reference is given by the joystick in two different ways. In the first mode (position control) the output from the joystick is interpreted as a velocity reference in the vehicle frame. This velocity reference is then fed into a position reference filter to get the position. The other control mode is an orbit control mode. In this mode the joystick output is mapped to a cylindrical-like coordinate system where the object of interest is placed at the origin. In this mode the ROV will always face towards the center of the object of interest, while moving around it. The joystick is used to control the radius of the orbit, the angle of observation and the depth. The four control modes that are suggested are meant to extend the current control system of the ROV where the joystick motions are mapped directly to a thrust output on the ROV. It is important when designing a human centered control system that it is both easy for the operator to engage the automatic control system, as well as to intervene and do corrections. In the proposed design the operator should be able to easily switch between all control modes, according to guideline 2. It is important that the vehicle acts according to the operators anticipation during a mode change. This is reflected in guideline 1a. The proposed design is found as a block diagram in Figure 2. The dashed lines represent switch options. 3.1 Implementation Aspects
• Transit from A to B • Inspection of subsea equipment
In the transit phase a suitable control scheme will be an autopilot, controlling the heading, velocity and depth/altitude of the ROV. The reference from this controller can either come directly from the operator via the joystick, or by an automated guidance system with preprogrammed waypoints. 169
The control system proposed in this article is implemented and tested in a simulator using “perfect feedback”. In the simulation the ROV moved in a 3D virtual world, and was controlled by an operator. The 3D world is shown in Figure 6. The sensor-data used by the control algorithms are unbiased and without noise. This is however not the situation in a real subsea IMR operation. It is assumed that in a
2016 IFAC CAMS 170 Sept 13-16, 2016. Trondheim, NorwayEirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172
real scenario, the ROV will be equipped with a sensor suite and navigation filters, together capable of estimating the velocity and position of the ROV without drift in the estimates. Such drift would cause the ROV to drift correspondingly while stationkeeping. This would cause an unpredictable behavior where the operator would need to intervene in the automatic execution. The estimates do not need to be globally corrected, as the operator is working on the basis of camera feedback given in a local frame. The system is comprised of a set of control and guidance algorithms that individually are stable. Switching between the modes are initiated by the human operator. This implies that the switching happens relatively slowly. It is therefore assumed that the switching between modes does not introduce any unstable behavior. 4. TRANSIT MODES The autopilot and waypoint modes shown in Figure 2 are used when the ROV is in transit between work locations. These modes are controlling the vehicle while the operator can rest and observe the operation. The automatic controller is taking care of the heading, forward speed, and depth/altitude. When waypoint mode is used, the heading reference is made by the waypoint guidance system. In the autopilot mode the heading reference is controlled by using the joystick, and the operator takes the role as a guidance system. Both modes use the autopilot controller for allocating thrust to the ROV. The waypoint control loop is illustrated in green and the autopilot loop in blue in Figure 2. The autopilot controller is a PID-controller with a feed-forward velocity reference. The waypoint guidance scheme is a lookahead based line-of-sight steering law from Fossen (2011). If the operator wants to step out of the waypoint mode, e.g. to take a closer look at something odd along the route, the controller saves the current position as a waypoint. This allows the operator to easily go back to the predefined route even after the aborting the execution. This feature is meant to lower the threshold for the operator to intervene in the automatic execution. This adaptable behavior is according to guideline 2. 5. POSITIONING MODES During a subsea inspection the operator control objective is position control, and the position mode or orbit mode in Figure 2 is used. External disturbances are acting on the ROV so continuous control action is needed to keep the ROV at the desired position. The proposed position controller enables the operator to change position and orientation using the Joystick, and let the control system handle the force allocation to keep the vehicle on position. The designed position control system will account for external disturbances and map joystick inputs to ROV movements in an intuitive way. Both in position and orbit mode the joystick deflection is interpreted as a desired velocity. This velocity is then transformed and integrated to a position reference. In the position mode the velocity is in the vehicle reference frame and is directly transformed into the global reference frame (red control loop in Figure 2). In the orbit mode the velocity is given in a cylindrical 170
like coordinate system where the center of the orbit is placed in the origin (yellow loop in Figure 2). In order to make the position and velocity references continuous during a switch between modes, the mode that are being switched to is initialized using the position, and velocity from the current mode. 5.1 Position mode A nonlinear PID-controller with a feed forward reference is applied for position control. This is a well known and relatively simple controller. This corresponds with guideline 4, calling for a comprehensible control system. For position reference generation the approach proposed by Dukan and Sørensen (2012) is used. This is a low-pass filter cascaded with a mass, damper and spring system applied to the joystick output. The filter output is a smooth position reference as well as a reference velocity used as a feed-forward to the position controller. 5.2 Orbit Control Mode The orbit control mode enables the operator to circle around an object of interest during an inspection. This maneuver is difficult to perform using manual control because it changes the heading and position simultaneously. The idea of referencing joystick commands to a coordinate system using cylindrical coordinates is described in Yoerger et al. (1986). Simulator testing revealed that a cylindrical orbit was very handy when inspecting small objects (approximated by points), or larger objects with a circular base. However, for the offshore inspection scenario, most objects are quite large and rectangular in shape. This made the circular orbit unpractical as the distance to the object change a lot during one lap. To adapt the orbit for rectangular objects, it is proposed to change the base shape from a circle to a super-ellipse. The difference in distance between path and object can be seen in Figure 3. The standard parametrization of the super-ellipse results in an along-path velocity that is varying considerably. This makes it unsuitable to be used as a path. A new parametrization that solves this problem and results in constant along-path velocity is proposed. This is done to make the behavior more predictable (guideline 1). The geometric shape of the super-ellipse is documented in Weisstein (2016). This base shape has several shaping parameters, making it possible to generate a suitable path for the ROV to orbit around a subsea structure. The super-ellipse shape is can be described by the following equations: x(φ) = rA cos2/s (φ) 2/s
(1a)
(1b) y(φ) = rB sin (φ) Where A, and B scale the shape along the x-, or y-axis, while r sets the size of the shape. φ is the control parameter. The parameter s is sometimes called the squareness parameter and for s > 2 it controls the squareness of the shape. s = 1 produces a diamond shape, while s = 2 produces an ellipse, or a circle if A = B. Using this base shape will in other words give the possibility to adapt the orbit path to a lot of different shaped equipment. The base shape can be rotated by using a 2D rotation matrix, and
2016 IFAC CAMS Sept 13-16, 2016. Trondheim, NorwayEirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172
171
4
6
Structure Outline Circular Orbit Superellipse Orbit
5
No Velocity scaling Velocity scaling
3.5
4 3
Position error [m]
North [m]
3 2 1 0 -1 -2
2.5 2 1.5 1
-3 0.5
-4 -4
-2
0
2
4
6
0 0
East [m]
5
10
15
20
25
30
35
40
45
Time [s]
Fig. 3. Circular and Super-ellipse path around a rectangular subsea structure placed in the North-East plane by adding the coordinates of the desired origin of the shape. The r parameter is controlled in real-time by the operator by pushing the joystick forwards or backwards. This corresponds to moving closer or away from the structure, or moving the ROV forward and backward. When the joystick is moved sideways, the vehicle will move along the structure following the path. When using a cylinder as a base shape it was practical to control this motion along the path by using the angle parameter φ. For a cylinder the relation between one step along the path was proportional to one angle step. For the super-ellipse however, this does not hold as the along-path velocity is dependent on φ. A velocity unproportional to joystick input makes it very hard to control the vehicle. The proposed solution is to map the joystick y-axis to the along-path speed and then calculate the a new angle parameter based on this speed. This was done by substituting the variable θ for ω in (1). The new path variable is calculated numerically using the following relations: s = x (φ)2 + y (φ)2 (2) t Ud dt (3) ω= 0 s where Ud is the commanded along-path velocity, x (φ) and y (φ) is defined in (1). The denominator in (3) is infinity at the x-, and y-axis of the shape, causing the fraction to go towards zero. This is solved in the implementation by saturating the denominator at a high threshold. This may cause the along-path velocity to be slightly higher at one timestep when crossing the x- and y-axis. Simulator testing shows that this is not noticeable by the operator. 5.3 Joystick Wind-up Testing revealed that in presence of currents the vehicle may move slower than the filter reference. This may be hard for the operator to notice, and may cause her to lose control over the vehicle. We named this problem joystick wind-up because of its similarities to integrator wind-up. The reference filter is tuned so the maximum reference velocity is slightly less than the ROV maximum velocity. 171
Fig. 4. Position error with, and without velocity scaling during a move towards the current However, the reference filter does not take into account current forces. When the vehicle is travelling countercurrent, the vehicle will lag behind the joystick generated reference. This is because the ROV can not sustain the speed set by the reference filter. If the vehicle is far from the setpoint this might cause confusion for the operator as he/she does not get the expected feedback when moving the joystick. The controller is chasing a position far away from the vehicle and the vehicle will move in this direction at maximum possible speed, no matter if the operator changes the setpoint slightly. This is the same problem that can lead to integral wind-up in PID-controllers. In order to mitigate this issue, a lag based velocity scaling function is proposed for the reference filter. The idea is to multiply the reference speed with a scaling-factor when a lag is detected. The scaling parameter is calculated and applied in the global frame, as the current-forces are acting in this frame. The scaling factor, S is calculated as follows: 1 if |ηdi − ηi | ≤ 1m 0.5 (4) Si = if ηdi − ηi | > d |ηdi − ηi |
d is the lag treshold and S = Diag(S1 , S2 , S3 , 1, 1, 1) as we only apply scaling to the translational degrees of freedom. Figure 4 shows how the velocity scaling ensures that the position error stays within a reasonable bound, making the vehicle controllable by the joystick, following guideline 1b. 5.4 Adaptive Deadband When the operator attempts to move the vehicle at high velocity and along a straight line using the joystick, it is hard to keep the vehicle moving along the straight line. The explanation is that when making a somewhat coarse movement in forward direction to obtain a high speed, it is hard to avoid small movements in the transverse directions. To counteract such unintended commands it is possible to introduce a deadband on the joystick output. A regular deadband however tend to remove precision for small deflections of the joystick. To keep precision and counteract unintended motions in high speed an adaptive deadband is proposed. This deadband, D is calculated as:
2016 IFAC CAMS 172 13-16, 2016. Trondheim, Norway Eirik Hexeberg Henriksen et al. / IFAC-PapersOnLine 49-23 (2016) 167–172 Sept
D=k
6
ηijs
(5)
i=1
where k is a tunable parameter controlling the size of the adaptive deadband envelope. The deadband is applied to the joystick output as follows: 0 if ηi < D js ηi = (6) ηijs if ηi ≥ D When the deadband is applied to the joystick output before it enters the reference filter it is significantly easier to make a straight line reference for the ROV at high speeds. Figure 5 shows two test-runs where the operator wants the ROV to move at high speed in a straight line. The figure shows a clear improvement in the horizontal plane, but also applies to a path in the vertical direction. -1 With Adaptive deadband No Deadband
North
-2
-4
-6
-4
-2
0
East
2
4
IMR operations, making the operations faster, safer and easier. In a future scenario we might see highly automated vehicles remotely operated from shore performing IMR operations subsea. REFERENCES
-3
-5
Fig. 6. Operators view during testing of control system
6
Fig. 5. Position in the North-East plane during a straight line move with, and without adaptive deadband 6. CONCLUSIONS Transit and position control modes for an ROV in IMR operation are described in the previous sections. A control system architecture is designed to enable adaptable shared control, allowing to switch between several modes of operation. The modes are designed to make the job for the human operator easier during an IMR operation. Less operator attention and effort is required as the ROV controller handles the external disturbances, as well as aiding in path following. For instance, the orbit control mode allows the operators to automatically observe the subsea equipment from all angles. The control system allows the operator to focus on the actual work, such as observing, or controlling the ROV tools or arms. The architecture including four control modes is tested in a simulation environment, using the flexibility of the control architecture to switch between the modes. The system is still at a testing and development stage, but a qualitative review has proved it easy to operate, even without training. Topics for further work will include implementation of a more refined human machine interface, automated collision avoidance and other operator aid functionality for IMR operations. The system will be further verified by laboratory experiments. Guidelines for design of human centered automation for aviation was used in the design phase. The guidelines suggested that the system should include adaptable automation and that it should be predictable and comprehensive for the human operators. The last guideline suggested that the automation system should also monitor the human to avoid errors. Guidelines to ensure predictable behavior of the system are proposed. The goal of the work is to introduce human friendly automation for ROVs performing 172
Atoyan, H., Duquet, J.R., and Robert, J.M. (2006). Trust in new decision aid systems. In Proceedings of the 18th Conference on L’Interaction Homme-Machine, IHM ’06, 115–122. ACM, New York, NY, USA. Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. Billings, C.E. (1996). Human centered aviation automation: Principles and guidelines. Technical Report NASA TM 110381, NASA Ames Reasearch Centre. Candeloro, M., Valle, E., Miyazaki, M.R., Skjetne, R., Ludvigsen, M., and Srensen, A.J. (2015). Hmd as a new tool for telepresence in underwater operations and closed-loop control of rovs. In Proc. MTS/IEEE OCEANS. Dukan, F. and Sørensen, A.J. (2012). Joystick in closedloop control of rovs with experimental results. Navigation, Guidance and Control of Underwater Vehicles, 3, 293–300. Fossen, T.I. (2011). Handbook of Marine Craft Hydrodynamics and Motion Control. John Wiley & Sons, Ltd. Norman, D.A. (1990). The problem of automation: Inappropriate feedback and interaction, not overautomation. Philosophical Transactions of the Royal Society of London, 585 – 593. Offshore Engineer (2015). Sea change. Offshore Engineer, October 2015, 50–52. Schjølberg, I. and Utne, I.B. (2015). Towards autonomy in ROV operations. 4th IFAC Workshop on Navigation, Guidance and Controlof Underwater Vehicles, NGCUV 2015, 48(2), 183 – 188. Sheridan, T.B. (1995). Human centered automation: oxymoron or common sense? IEEE International Conference on Systems, Man and Cybernetics, 1995. Intelligent Systems for the 21st Century., 1, 823 – 828. Sheridan, T.B. (2001). Rumination on automation, 1998. Annual Reviews in Control, 25, 89 – 97. Weisstein, E.W. (2016). Superellipse. In MathWorld. URL http://mathworld.wolfram.com/Superellipse.html. Winter, J.C.F. and Dodou, D. (2014). Why the fitts list has persisted throughout the history of function allocation. Cognition, Technology & Work, 16(1), 1–11. Yoerger, D., Newman, J.B., and Slotine, J.J.E. (1986). Supervisory control system for the jason rov. IEEE Journal of Oceanic Engineering, 11, 392 – 400.