Ecological Interface for Collaboration of Multiple UAVs in Remote Areas

13th IFAC/IFIP/IFORS/IEA Symposium on 13th IFAC/IFIP/IFORS/IEA Symposium on 13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and Evaluation of Human-Machine Systems 13th IFAC/IFIP/IFORS/IEA Symposium on Analysis, Design, and of Analysis, Design, and Evaluation Evaluation of Human-Machine Human-Machine Systems Available onlineSystems at www.sciencedirect.com Aug. 30 Sept. 2, 2016. Kyoto, Japan Analysis, Design, and Evaluation of Human-Machine Systems Aug. 30 Sept. 2, 2016. Kyoto, Japan Aug. 30 - Sept. 2, 2016. Kyoto, Japan Aug. 30 - Sept. 2, 2016. Kyoto, Japan

ScienceDirect

IFAC-PapersOnLine 49-19 (2016) 450–455

Ecological Interface for Collaboration of Ecological Interface for Collaboration of Ecological Interface for Collaboration of Multiple UAVs in Remote Areas Multiple UAVs in Remote Areas Multiple UAVs in Remote Areas S. van Lochem, ∗∗∗ C. Borst, ∗∗∗ G.C.H.E. de Croon, ∗∗∗ S. vanM.M. Lochem, C. Borst, de Croon, ∗ ∗ G.C.H.E. ∗,1 S. Lochem, C. G.C.H.E. de ∗ van ∗Paassen and M. Mulder S. van vanM.M. Lochem, C. Borst, Borst, G.C.H.E. de Croon, Croon, ∗ ∗,1 ∗ and ∗,1 van Paassen Mulder M.M. van Paassen M. Mulder ∗ and M. ∗,1 M.M. van Paassen and M. Mulder ∗ Control and Simulation, Aerospace Engineering, Delft University ∗ ∗ Control and Simulation, Aerospace Engineering, Delft University Simulation, Aerospace Engineering, Delft ∗ Control and Technology, 2629 HS, Delft, The Netherlands Control and Simulation, Aerospace Engineering, Delft University University Technology, 2629 HS, Delft, Delft, The Netherlands Netherlands Technology, Technology, 2629 2629 HS, HS, Delft, The The Netherlands

of of of of

Abstract: Unmanned Aerial Vehicles (UAVs) can be used to access remote areas, e.g., Abstract: Unmanned Aerial Vehicles (UAVs) be used to access remote areas, e.g., Abstract: Unmanned Aerial Vehicles can be used to remote areas, for surveillance missions. Collaboration betweencan them overcome communication Abstract: Unmanned Aerial Vehicles (UAVs) (UAVs) can be can used help to access access remote areas, e.g., e.g., for surveillance missions. Collaboration between them can help overcome communication for surveillance missions. Collaboration between them can help overcome communication constraints by building airborne relay networks thatthem allow can beyond line of sight communication. for surveillance missions. Collaboration between help overcome communication constraints by building airborne relay that allow beyond of sight communication. constraints by building relay networks that allow line of communication. This research whether a networks single human can line supervise multiple UAVs in constraints by investigates building airborne airborne relay networks that operator allow beyond beyond line of sight sight communication. This research investigates whether a single human operator can supervise multiple UAVs in in This research investigates whether a single human operator can supervise multiple UAVs aThis collaborative surveillancewhether task under communication constraints. We designed an ecological research investigates a single human operator can supervise multiple UAVs in a collaborative surveillance task under communication constraints. We designed an ecological a collaborative surveillance task under communication constraints. We designed an ecological interface to support operatorstask in their task and increase system flexibility. A preliminary humana collaborative surveillance under communication constraints. We designed an ecological interface to support operators in their task and increase system flexibility. A preliminary humaninterface to support operators in their task and increase system flexibility. A preliminary humanin-the-loop study was done to investigate operator task performance and evaluate interface interface to support operators in their task and increase system flexibility. A preliminary humanin-the-loop study was done to investigate operator task performance and evaluate interface in-the-loop study was done to investigate operator task performance and evaluate interface components. It was shown that operators are able to successfully operate surveillance missions in-the-loop study was done to investigate operator task performance and evaluate interface components. It was was shown that operators are able able to to successfully successfully operate surveillance missions components. It operators are surveillance missions under communicationandthat battery constraints. did,operate however, not succeed to do components. It was shown shown that operators are ableParticipants to successfully operate surveillance missions under communicationand battery constraints. Participants did, however, not succeed to do under communicationand battery constraints. Participants did, however, not succeed to this without separation and conflicts andconstraints. communication losses, which indicates not thatsucceed the interface under communicationbattery Participants did, however, to do do this without separation conflicts and communication losses, which indicates that the interface this without separation conflicts and communication losses, which indicates that the interface lacks elements representing endurance and separation assurance. To an extent, the interface this without separation conflicts and communication losses, which indicates that lacks elements representing endurance and separation assurance. To an extent, the interface lacks elements representing endurance and separation assurance. an extent, the design turned out to be scalable, with a few visualizations that clutter for large lacks elements representing endurance and remaining separationvisualizations assurance. To To an cause extent, the interface interface design turned out to be scalable, with a few remaining that cause clutter for large large design turned out to be scalable, with a few remaining visualizations that cause clutter for numbers of UAVs. More advanced ways of remaining displaying visualizations information on request and grouping of design turned out to be scalable, with a few that cause clutter for large numbers of UAVs. More advanced ways of displaying information on request and grouping of numbers of UAVs. More advanced ways of displaying information on request and grouping of c and IFAC select information isMore warranted to further improve the information interface. Copyright 2016 numbers of UAVs. advanced ways of displaying on request grouping of cc select information is warranted to further improve the interface. Copyright 2016 IFAC select information is warranted to further improve the interface. Copyright 2016 IFAC c select information is warranted to further improve the interface. Copyright 2016 IFAC © 2016, IFAC (International Federation of Automatic Control) Hosting by Elsevier Ltd. All rights reserved. Keywords: human-machine interface, supervisory control, ecological interface design Keywords: human-machine interface, interface, supervisory control, control, ecological interface interface design Keywords: Keywords: human-machine human-machine interface, supervisory supervisory control, ecological ecological interface design design 1. INTRODUCTION Work Domain Analysis (WDA) was performed to analyze 1. Work Domain Analysis (WDA) was performed to analyze 1. INTRODUCTION INTRODUCTION Work Domain Analysis (WDA) was performed to the work domain constraints, visualizations 1. INTRODUCTION Work Domain Analysis (WDA) and was several performed to analyze analyze the work domain constraints, and several visualizations the work domain constraints, and several visualizations UAV operations grow exponentially (FAA, 2014) and the werework designed to map these constraints on visualizations the interface. domain constraints, and several UAV operations grow exponentially (FAA, 2014) and were designed to map these constraints on the interface. UAV technologies operations grow grow exponentially (FAA, search 2014) and to these constraints on new enableexponentially them to perform and were Directdesigned manipulation implemented having a UAV operations (FAA, 2014) designed to map mapwas these constraintsthrough on the the interface. interface. new them to perform perform search and were Direct manipulation was implemented through new technologies technologies enable them to and Direct manipulation was implemented through having a rescue, explorationenable and surveillance missions. search Unmanned tablet-based touch screen platform (Android). Ahaving human-a new technologies enable them to perform search and Direct manipulation was implemented through having a rescue, exploration and surveillance missions. Unmanned tablet-based touch screen platform (Android). A humanrescue, exploration and surveillance missions. Unmanned tablet-based touch screen platform (Android). A humanoperations do not expose human pilots to dangers, have tablet-based in-the-loop evaluation was platform done to investigate whether the rescue, exploration and surveillance missions. Unmanned touch screen (Android). A humanoperations do not expose human pilots to dangers, have in-the-loop evaluation done to investigate whether the operations do human pilots to dangers, have evaluation was done to whether the aoperations longer endurance, and enable remote areas. current GCS interface was design supports operator problemdo not not expose expose human access pilots to dangers, have in-the-loop in-the-loop evaluation was done to investigate investigate whether the aa longer endurance, and enable access to remote areas. current GCS interface design supports operator problemlonger endurance, and enable access to remote areas. current GCS interface design supports operator problemHaving multiple UAVs that operate as a team, can further solving performance. a longer endurance, and enable access to remote areas. current GCS interface design supports operator problemHaving that as team, can further performance. Having multiple multiple UAVs that operate operate as a team,to can further solving solving performance. enhance mission UAVs performance and robustness failures. Having multiple UAVs that operate as aa team, can further solving performance. enhance mission performance and robustness to failures. In the following, we first discuss UAV team collaboration enhance mission performance and robustness to failures. enhance mission performance and robustness to failures. In the following, first discuss UAV team collaboration In the following, we first UAV Successful team performance requires individual UAVs in remote areas, we followed by an introduction to our proIn the following, we first discuss discuss UAV team team collaboration collaboration Successful team performance requires individual UAVs in remote areas, followed by an introduction to our proSuccessful team performance requires individual UAVs in remote areas, followed by an introduction to to collaborate. Communication is crucial, to share UAVs state posed interface. Results of the preliminary evaluation are Successful team performance requires individual in remote areas,Results followedofby anpreliminary introduction to our our proproto collaborate. Communication is crucial, to share state posed interface. the evaluation are to collaborate. Communication is crucial, to share state posed interface. Results of the preliminary evaluation are information between the airborne vehicles, andshare including presented, with aResults discussion and conclusions. to collaborate. Communication is crucial, to state posed interface. of the preliminary evaluation are information between the airborne vehicles, and including presented, with aa discussion and conclusions. information between the airborne vehicles, and including presented, with discussion and conclusions. the human operator supported by a Ground Control information between the airborne vehicles, and including presented, with a discussion and conclusions. the human operator supported byto aa a Ground Ground Control the human operator Control Station (GCS). This supported often leadsby dependence on the human operator supported byto a a Ground Control Station (GCS). This often leads dependence on Station (GCS). This often leads to a dependence on Line-of-Sight (LOS) communications (Olsson et al., 2010), 2. TEAM COLLABORATION IN REMOTE AREAS Station (GCS). This often leads to a dependence on Line-of-Sight (LOS) communications (Olsson et al., 2010), 2. TEAM COLLABORATION IN REMOTE AREAS Line-of-Sight (LOS) communications (Olsson et al., 2010), 2. limited by obstacles and small communication ranges. To Line-of-Sight (LOS) communications (Olsson et al., 2010), 2. TEAM TEAM COLLABORATION COLLABORATION IN IN REMOTE REMOTE AREAS AREAS limited by obstacles and small communication ranges. To limited by obstacles and small communication ranges. To enable communication also in remote areas, UAVs can limited by obstacles and small communication ranges. To Sharing information between UAVs and GCS, such as enable also areas, UAVs enablea communication communication also in in etremote remote areas, Algorithms UAVs can can Sharing form relay network (Palat al., 2005). information between UAVs and GCS, such as enable communication also in remote areas, UAVs can Sharing information between UAVs and GCS, as flight states, operational modes and sensed data, is crucial form aa relay (Palat et al., Algorithms information between UAVs and data, GCS, issuch such as form developed relay network network (Palat etnetworks al., 2005). 2005). Algorithms Sharing flight states, operational modes and sensed crucial were to optimize these for reachability form a relay network (Palat et al., 2005). Algorithms flight states, operational modes and sensed data, is crucial for any mission. Often a centralized system architecture were developed to optimize these networks for reachability flight states, operational modes and sensed data, is crucial were developed to optimize these networks for reachability any mission. aa centralized system architecture and coverage Interest (ROIs) (Cetin and for were developedof toRegions optimizeof these networks for reachability for any Often system architecture is adopted, whereOften all UAVs communicate a GCS, and of of Interest (Cetin and any mission. mission. Often a centralized centralized systemwith architecture and coverage coverage of Regions Regions of computational Interest (ROIs) (ROIs)demand (Cetin and and for Zagli, 2012). However, high is adopted, where all UAVs communicate with a GCS, and coverage of Regions of Interest (ROIs) (Cetin is adopted, where all UAVs communicate with a that coordinates the activities of all individual vehicles. Zagli, 2012). However, high computational demand and is adopted, where all UAVs communicate with a GCS, GCS, Zagli, 2012). However, high computational demand and that coordinates the activities of all individual vehicles. inflexibility toHowever, unexpected mission changes,demand often still Zagli, 2012). high computational and that coordinates the activities of all individual This system architecture is considered to leadvehicles. to the inflexibility to unexpected mission changes, often still that coordinates the activities of all individual vehicles. inflexibility to unexpected mission changes, often still system architecture is considered to lead the require a human operator as mission the mainchanges, decision-maker. inflexibility to unexpected often still This This system is to to the best collaborative performance but can also sufferto require This system architecture architecture is considered considered to lead lead tofrom the require a a human human operator operator as as the the main main decision-maker. decision-maker. best collaborative performance but can also suffer from require a human operator as the main decision-maker. best collaborative performance but can also suffer from communication constraints (Godwin et al., 2007). In this paper, Ecological Interface Design (EID) (Vicente best collaborative performance but can also suffer from communication constraints (Godwin et al., 2007). In paper, Interface Design constraints (Godwin et 2007). In this this paper, Ecological Ecological Interface Design (EID) (Vicente and Rasmussen, 1992; Vicente, 2002; Borst(EID) et al.,(Vicente 2015) is communication communication constraints (Godwin et al., al., 2007). downIn this paper, Ecological Interface Design (EID) (Vicente Communication between UAV and GCS comprises and Rasmussen, 1992; Vicente, 2002; Borst et al., 2015) is and Rasmussen, 1992; Vicente, 2002; Borst et al., 2015) is Communication between UAV and GCS comprises downapplied to support the operator in the control of multiple and Rasmussen, 1992; Vicente, 2002; Borst et al., 2015) is Communication between UAV and GCS comprises downlink of telemetryand sensed dataGCS as well as uplink of applied to support the operator in the control of multiple Communication between UAV and comprises downapplied to support the operator in the control of multiple UAVs. The mission aimed at conducting a surveillance link of telemetryand sensed data as well as uplink of applied to support the operator in the control of multiple link of telemetryand sensed data as well as uplink of commands. Small UAVs (<5kg) generally use Wi-Fi sigUAVs. The mission aimed at conducting a surveillance link of telemetryand sensed data as well as uplink of UAVs. The mission aimed at conducting a surveillance commands. Small UAVs (<5kg) generally use Wi-Fi sigtask of The one mission or more aimed ROIs, at in conducting a remote area, requiring nals UAVs. a surveillance commands. Small UAVs (<5kg) generally use Wi-Fi sig(2.4GHz), which are constrained to line-of-sight. The task of one or more ROIs, in a remote area, requiring commands. Small UAVs (<5kg) generally use Wi-Fi sigtask of one or more ROIs, in a remote area, requiring are constrained line-of-sight. The the operator buildROIs, a relay communicatask of one orto more in anetwork remote for area, requiring nals nals (2.4GHz), which are to line-of-sight. The link (2.4GHz), budget is which very limited becauseto severe payload the operator toour build relay network for communica(2.4GHz), which are constrained constrained toof line-of-sight. The the operator build aa relay network communication, extendingto previous work (Fuchsfor et al., 2014). A nals link budget is very limited because of severe payload the operator to build a relay network for communicalink budget is very limited because of severe payload limitations. Flights beyond the maximum communication tion, extending our previous work (Fuchs et al., 2014). A link budget is very limited because of severe payload tion, extending our our previous previous work work (Fuchs (Fuchs et et al., al., 2014). 2014). A A limitations. Flights the maximumwhere communication tion, extending 1 E-mail: limitations. Flights beyond the communication range lead to a lossbeyond of communication, the UAV [email protected] limitations. Flights beyond the maximum maximumwhere communication 1 1 E-mail: range lead to a loss of communication, the UAV UAV [email protected] range lead to a loss of communication, where E-mail: [email protected] 1 E-mail: [email protected] range lead to a loss of communication, where the the UAV Copyright © 2016, 2016 IFAC 461Hosting by Elsevier Ltd. All rights reserved. 2405-8963 © IFAC (International Federation of Automatic Control) Copyright 2016 461 Copyright ©under 2016 IFAC IFAC 461Control. Peer review© of International Federation of Automatic Copyright © 2016 responsibility IFAC 461 10.1016/j.ifacol.2016.10.620

2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan

S. van Lochem et al. / IFAC-PapersOnLine 49-19 (2016) 450–455

451

represented in the current interface: all UAVs are assumed to automatically return to the ground station once they have a near empty battery level.

Region of Interest Surveillance UAV

Abstract Function The (camera) sensor coverage per UAV is indicated around surveillance waypoints on the map using a shading which changes color depending on status. Areas that are being covered are green, ones that are expected to be covered are yellow and for UAVs without communication a red color is given because sensed data cannot be sent to the GCS. Coverage areas are circular because surveillance UAVs typically ‘circle around’ their assigned waypoint. The radius depends on the UAV altitude and the field-of-view of the on-board camera.

Relay UAV #2

Relay UAV #1

Fig. 1. A UAV relay and surveillance network. continues to fly autonomously, uncontrollable from the GCS, possibly leading to a crash. To reach remote areas, the LOS range is extended through a relay of the communication signal. An example is the “Tour de France” bicycle race where multiple aircraft are used to relay live video streams, filmed from motorcycles, to the ground station. In our application, chains of relay UAVs can be used, as illustrated in Figure 1. To form such networks in an optimal way, coordination is required for positioning and task allocation – commanding the UAV to perform a surveillance or communication relay task – of the whole UAV team. To maximize communication range, the UAVs that act as relay units can be placed as close as possible to the communication range (assumed to be circular). Note that the battery/energy requirements of a relay UAV can be different from a UAV that is assigned to have a surveillance role. Clearly, human operators need a good GCS interface to support their decision-making. 3. PROPOSED GCS INTERFACE A work domain analysis (WDA) was conducted to reveal the surveillance mission constraints. Here we describe the main WDA findings and how these affected the ecological interface. Whereas our earlier research focused on higherlevel information (Fuchs et al., 2014), here our aim was to study those lower-level information variables that affect communication most: UAV position and heading, battery level, communication status and altitude. Also some higher-level information, such as communication range and (ground) coverage is visualized to be better able to perform the overall surveillance mission. Figure 2 illustrates the proposed interface design, the main elements of which will be briefly discussed next. In order to keep the interface as scalable as possible, that is, still usable for larger numbers of UAVs, information is presented close to the individual vehicle icons. Functional Purpose The surveillance mission goal is to obtain ground coverage of one or more ROIs, which are indicated on the map-view display by a colored shading. The functional purpose to “safely return home” is not 462

Locomotion is present in the display in the form of movement of UAV icons on the map. Furthermore, collaboration of UAVs can be detected through the relay status and communication information (i.e., (dashed) relay communication range circles). Separation between UAVs is shown through coloring the icons and labels on an altitude tape on the right-hand side. UAVs that fly at unique altitudes are colored gray; UAVs that fly at the same altitude but with sufficient horizontal separation are colored blue. In case of a separation conflict, the involved UAVs are colored red, and lines between the UAV icons to depict conflicting pairs. Group labels are used on the altitude tape to indicate that UAVs are located at (approximately) the same altitude. This grouping of labels is also needed to prevent a cluttered altitude tape and thus to keep the design scalable. Once a group label is clicked, the involved UAV icons become yellow (indicates the relation with the selected altitude label group) and individual labels are shown on the left side of the altitude tape. Generalized Function The mission flight plan is indicated on the map using waypoints which contain labels indicating which UAV they belong to. The generalized function of communication is represented in the interface by a small communication icon, as shown in Figure 3, included in the UAV icons. This icon was designed to match the human mental model of the information it represents: three full (blue) bars for high signal reception, less bars when signal reception decreases, and a cross in case of complete communication loss. Physical Function Aircraft icons are used to show the status of UAVs on the map. Apart from the communication status icon these contain information about heading (attitude is irrelevant because autonomous navigation capabilities were assumed), position and battery status. The latter is shown in a way that matches the operator mental model: a high battery level corresponds with a full green icon, a low level with an (almost) empty red icon, Figure 3. The communication area is indicated using outer boundary circles. The maximum communication range of the UAVs is assumed to be equal to that of the ground station, so communication is possible when the UAV is located within the circle. The communication area of the ground station is displayed around a “home” icon, indicating the GCS position. In case a UAV has been assigned a communication relay task, an extra range circle is drawn around it, extending the area in which communication with the GCS is still possible.

2016 IFAC/IFIP/IFORS/IEA HMS 452 Aug. 30 - Sept. 2, 2016. Kyoto, Japan

S. van Lochem et al. / IFAC-PapersOnLine 49-19 (2016) 450–455

No aircra� selected No aircra� selected Aircra� A Aircra� B Aircra� C

Home comm. range Relay comm. range

COMM GUARANTEED

COMM RANGE

COVERAGE

ROI 5 SW C

C

Distance home: 850m Al�tude: 70m Task: RELAY Ba�ery: 12.4V B

SW

B

B

4 3

A

2

A

D

AC

R

R

DE

D E

E

1

Land

Take-off Go home

Fig. 2. Scheme of the proposed interface;  1 is a separation conflict between two UAVs;  2 shows a selected icon with UAV information window, the UAV acts as a relay circling around waypoint B;  3 is a UAV surveilling the area around waypoint A;  4 shows a UAV outside the communication zone, and  5 is the (joint) altitude tape. will be discussed as one group in the following. Each UAV in the system had one waypoint available that could be moved on the map by the operator. By default, the fixed-wing UAVs circled around their waypoints, yielding a circular ground coverage area, see Figure 2.

Fig. 3. Battery- and communication icons. Physical Form Locations of the ground station and UAVs are indicated on the map with respectively a “home”- and aircraft icons. The latter are also used to indicate the UAVs headings. When selecting a UAV, its battery properties are shown as an exact numerical voltage in the UAV information window. 4. HUMAN-IN-THE-LOOP EVALUATION A preliminary evaluation was performed to test the interface design, with focus on problem-solving activities of operators. This was done for simulated surveillance missions in which live sensor coverage of remote ROIs is required under communication and battery constraints. 4.1 Participants and Task Ten subjects – five staff members with experience in UAV control, and five aerospace students – were asked to perform six mission scenarios. Because the results were not significantly different for these two groups, participants 463

The objective was to get maximum ground coverage of ROIs for as long as possible, without losing communication link between UAVs and the GCS. The maximum communication range for both UAVs and GCS was set at 1km; an inverse fourth-power relation between antenna distance and signal strength was implemented in the simulation. Operators were allowed to choose between two predetermined roles (and corresponding flight levels) for the UAVs, namely to act as a surveillance or relay UAV, with altitudes of 40m and 60m, respectively. These altitudes are small relative to the maximum communication range, so it can be safely assumed that the latter is not influenced by the UAV altitude. Note that since a relay UAV flies at a higher altitude, and also needs to relay all communication data, its battery is programmed to run out faster. 4.2 Scenarios A set of practice scenarios was used to explain the working principle of the interface and get subjects acquainted with the required interaction. Participants were then required to conduct six test scenarios, in the same order. These scenarios differed in (initial) waypoint- and ROI locations, the number of UAVs, their battery levels and the possible

S. van Lochem et al. / IFAC-PapersOnLine 49-19 (2016) 450–455

(very difficult). After the final scenario some extra (open) questions were presented to ask their general opinion about the interface, the touch screen, and simulation.

Table 1. Description (D) and expected solution (S) strategy for all six scenarios. 1

D S

2

D S

3

D S

4

D S

5

D S

6

D

S

453

3 UAVs, all with full battery. One ROI far outside home communication range. Build a relay chain of two nodes and send the survey UAV first (has to fly farthest).

During the scenario runs also data such as task assignment to UAVs (relay or surveillance), communication status, (waypoint) positions and battery levels were logged. These data were used to check whether the found solutions corresponded with the expected strategies. Mission success in reaching the objective to maximize ground coverage of the ROIs was assessed based on how performance scores developed over time, as well as their final values.

3 UAVs, all have a full battery except for one. Two ROIs outside home range, of equal point value, far apart from each other. Choose a UAV with full battery and assign a position such that it can be a relay for the other two. In-flight start with 3 UAVs. #1 has lower battery. #3 starts outside communication range. #2 has battery failure after 1 min. First choose UAV 2 to relay. After battery failure, make sure all UAVs are within home range.

5. EVALUATION RESULTS 5.1 Operator Performance

4 UAVs, all with lower battery except for one. Large ROI (just) outside home range. Use the UAV with full battery to relay for the other three, which survey the ROI.

Figure 4 shows the mission success counts per scenario. Out of the total of 60 runs, seven were marked as ‘failed’ (more points were lost to communication and separation losses than obtained from coverage) and 20 runs were ‘flawless’ (no points lost at all).

3 UAVs, all with a low battery. Two ROIs, one inside home range (1pt) and one outside (2pts). Because batteries are low, go for less dependence on relay. Send one UAV to the ROI inside home range and use the other two to reach the remote one.

10

Flawless Success Fail

9

6 UAVs, all with a full battery except for one with a much lower level. Two ROIs, one inside home range (1pt) and one outside (2pts). Spare batteries by keeping UAVs on the ground. Substitute UAVs with full battery halfway during the mission for relay.

8

Number of scenario runs

2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan

7

6 5

existence of battery failures. Table 1 summarizes the scenarios and expected solution strategies. Note that all UAVs automatically return home and land there once they have a near empty battery. To motivate participants, points were awarded for every half second a UAV had its maximum possible coverage within a ROI. A difference in points between ROIs (indicated using numbers in the circles) was used to force participants to choose where to send their vehicles. Penalty points were subtracted for loss of communication (2) and separation conflicts (2 per conflict pair). The horizontal separation was set to be twice the surveillance radius to prevent overlap of ground coverage and assure safe flight. 4.3 Apparatus This study was carried out in the ATM-Lab of the Faculty of Aerospace at TU Delft. A Samsung Galaxy Note 10.1 tablet was used to run the interface which was build as an Android application. Through WiFi the tablet was connected to a Paparazzi UAV ground control station on a PC, which was used to simulate multiple UAVs. 4.4 Measurements A questionnaire was completed by all participants after each scenario run. It consisted of a set of open questions about the participant decisions and control actions, to determine their strategy and system understanding. Subjects were also asked to rate individual interface elements on a Likert-scale from 1 (not useful) to 10 (very useful), as well as the scenario difficulty on a scale from 1 (very easy) to 5 464

4

3 2

1 0

1

2

4 3 Scenario number

5

6

Fig. 4. Mission success counts per scenario. Figure 5 shows one typical example per scenario of the performance score development. Towards the end of all scenarios a continuous scoring of points occurred. Subjects had little difficulties in scenarios 1, 2 and 4. In others, problems were encountered such as a relay chain breakdown at the end of scenario 5 and other setbacks such as temporary communication losses and separation conflicts. A frequent cause for communication loss could be attributed to mistiming between communication relay availability and other UAVs leaving the communication zone. Figure 6 shows relatively high numbers of short communication losses for all scenarios. Another cause was a failure to sustain relay chains to preserve communication connection, which generally resulted in longer communication loss durations. Participants had great difficulty in determining the time when the relay UAV batteries would be empty, or when UAVs would return to maintained communication areas. Once relay UAVs drop out, connection losses are irrecoverable causing long periods without any communication for all UAVs involved. The strategy to first send out the UAVs that had to fly the farthest led to a higher performance score: reducing

2016 IFAC/IFIP/IFORS/IEA HMS 454 Aug. 30 - Sept. 2, 2016. Kyoto, Japan

90 80

Sc. Sc. Sc. Sc. Sc. Sc.

Figure 7 shows the subjective scenario difficulty ratings, which roughly indicates the expected difficulty increase from scenario 1 to 6. Note that scenario 3 is an exception as it was the only one which involved an in-flight start and a battery failure.

1 2 3 4 5 6

70 60

5

50 40

4.5

30

4 Median rating

Normalized performance score (%)

100

S. van Lochem et al. / IFAC-PapersOnLine 49-19 (2016) 450–455

20 10 0 0

10

20

30 40 50 60 70 Scenario progress (%)

80

90 100

Fig. 5. Performance score development (per scenario, normalized with minimum and maximum scores).

Number of communication losses

25

20

3 2.5 2 1.5 1

t <5 5 ≤ t <10 10 ≤ t <20 20 ≤ t <40 t ≥ 40

1

2

4 3 Scenario number

5

6

Fig. 7. Median of the difficulty ratings per scenario (1=very easy, 5=very difficult).

15

Four elements have a relatively constant rating over the scenarios: the altitude tape, altitude value, communication range- and coverage circles. As expected, the altitude value was not found very useful, whereas the altitude tape scored much higher. The analog overview it provides, facilitates altitude comparisons between UAVs, and with that it allowed operators to solve separation conflicts through altitude change commands given on the tape.

10

5

0

3.5

1

2

4 3 5 Scenario number

6

Fig. 6. Communication loss duration (t) frequencies. travel time and separation conflicts. Possibly the difficulty some operators had to determine a strategy in advance (e.g., number of relay UAVs needed) caused a difference in trajectory planning and task assignment. These problems mainly occurred in scenarios 1, 4 and 6, which were either scenarios with ROIs far away from the GCS, or ones which required the UAVs to fly relatively close to each other. With few exceptions, participants used the expected strategies (Table 1), which indicates that they understood the situations and were able to find appropriate solutions. The most logical choice for relay-task assignment would be to use the UAV with the highest battery level. The most common ‘mistake’ was to assign this task to a non-optimal candidate. A probable reason was that subjects missed a general overview of all UAVs states, which caused them to send take-off commands to UAVs without knowing their task in advance. The order of sending out UAVs, and the initial locations they were sent to, were very determining factors to mission success. Most participants recognized these mistakes once the UAVs were in flight and often solved this by switching tasks, losing valuable time.

The communication range circles were found to be very useful, which can be explained by the observation that participants relied on them to determine their strategy for collaboration between UAVs to reach remote ROIs (number of relays required, relay placement, etc.). Combined with UAV icon positions on the map, an instant indication of communication possibility is provided, as well as insight in possible solutions in case of communication loss. The coverage circles were generally found useful. It was, however, observed that subjects did use them to prevent separation conflicts. This is explained by the fact that the horizontal separation standard equals twice the coverage radius, meaning that conflicts can (partly) be avoided by preventing overlap of coverage circles. This coupling resulted in a main use for vehicle separation instead of waypoint placement for the abstract function of coverage.

5.2 Participant Feedback

For other interface elements, more variations exist over the scenarios. Both the battery icon and -voltage roughly follow the trend of the scenario difficulty ratings: more difficult scenarios required participants to watch the battery level more closely. This also explains that the ratings for the battery voltage increase faster than for the -icon. In general the battery icon was found to be significantly more useful than the voltage level as it provides a status message on which operators can rely instantly.

Using the questionnaire the usefulness ratings of the individual interface elements, as well as their variation over the test scenarios, was indicated by the participants.

The communication icon was considered to be not useful at all: many subjects indicated that they never looked at it. To the participants it only mattered whether if UAVs could communicate, or not, and the received signal level was

465

2016 IFAC/IFIP/IFORS/IEA HMS Aug. 30 - Sept. 2, 2016. Kyoto, Japan

S. van Lochem et al. / IFAC-PapersOnLine 49-19 (2016) 450–455

455

considered irrelevant. The combination of UAV position and the communication range circles on the map turned out to be a better cue than the communication icon.

sideration that the current interface was tailor-made for this particular application, and excluded much information that would be relevant for other purposes.

In general, conflict indication was considered to be somewhat useful, especially in scenario 4, where many UAVs were supposed to fly closely together, causing more conflicts. Participants commented that the conflict detection and indication is clear but does not help for prevention because it does not provide a resolution advice.

Better solutions are needed, e.g., in grouping of UAV icons on the map and display of (detailed) information only when relevant. For example, by only showing battery level values on request or in case it is low. Providing commands directly to UAVs through their icons, instead of using generic buttons in combination with icon selection, are expected to prevent mix-up in commands to the vehicles.

The drop-down selection menu was found to be useful. Participants preferred, however, to act more directly on the UAVs through their icons and labels (direct manipulation). A solution for selection of overlapping icons would in fact make the drop-down menu obsolete. The open answers in the questionnaire and notes taken during the tests, yielded several additional remarks on the interface. An often recurring comment was that the way control input is given is not as desired and future paths of UAVs are unclear. This is caused by the (simplified) path planning possibilities of the scenario design, which allowed operators to change the location of one waypoint per UAV, which they would fly to and circle around. Problems regarding a cluttered display occurred mainly at scenario 6, in which a considerably higher number of UAVs was available. The main comments were about the information window and size of the UAV icons which blocked information on the map, resulting in problems moving waypoints.

7. CONCLUSIONS An interface was developed to support human operators in their control of multiple UAVs, in collaborative surveillance tasks under communication constraints. The humanin-the-loop evaluation showed that operators were indeed able to build and maintain communication relay networks in order to access remote ROIs for surveillance. All participants were able to identify the problems present in the scenarios and to select the best available UAVs for the relay and surveillance tasks. The interface still lacks, however, elements that support separation assurance and battery management. The lack of preview information led operators to make choices without fully understanding the consequences of their decisions on mission safety and performance. Especially visualizing the battery implications for UAV endurance, as reported in (Fuchs et al., 2014), would complement the current interface very well. REFERENCES

6. DISCUSSION The evaluation showed that our subjects had difficulty in maintaining the relay network in terms of endurance (battery) and path planning. Although battery level was presented, implications on higher-level constraints, such as when would it be appropriate to have a ‘fresh’ UAV to take over the relay role, were unclear. This made it difficult to assess the risk involved with the choice to survey a ROI beyond the home communication range. Especially when workload increases, subjects lost track of the individual UAV battery levels and sometimes gave commands to the ‘wrong’ UAV as they mixed them up. The aspect of signal strength was not well included in the interface, and subjects mainly treated communication as a binary aspect, leading them to place relay UAVs at the very edges of the communication ranges. Visualizations such as a shading inside the communication region, that is dependent on the received communication signals, could have better supported operators in maintaining a more robust network. Furthermore, indication of received sensor data quality at the ground station could make them aware of the consequences of the structure of their relay network and encourage them to more carefully place relay UAVs. To some extent our interface has shown to be scalable, where the concept of (status) information display on UAV icons yielded the best results. Once the number of UAVs was significantly increased, however, like in scenario 6, information started to overlap and clutter became a nuisance. Scalability is expected to remain an important issue for this work domain, especially when taken into con466

Borst, C., Flach, J.M., and Ellerbroek, J. (2015). Beyond Ecological Interface Design: Lessons From Concerns and Misconceptions. IEEE Trans. on Human-Machine Systems, 45(2), 164–175. Cetin, O. and Zagli, I. (2012). Continuous Airborne Communication Relay Approach Using Unmanned Aerial Vehicles. J. of Intelligent and Robotic Systems: Theory and Applications, 65(1-4), 549–562. FAA (2014). FAA Aerospace Forecast: FY 2014-2034. Fuchs, C., Borst, C., de Croon, G.C.H.E., van Paassen, M.M., and Mulder, M. (2014). An Ecological Approach to the Supervisory Control of UAV Swarms. Int. journal of Micro Air Vehicles, 6(4), 211–229. Godwin, M.F., Spry, S.C., and Hedrick, J.K. (2007). A Distributed System for Collaboration and Control of UAV Groups : Experiments and Analysis. In Cooperative Systems, 139–156. Springer Berlin Heidelberg. Olsson, P.M., Kvarnstr¨om, J., Doherty, P., Burdakov, O., and Holmberg, K. (2010). Generating UAV communication networks for monitoring and surveillance. In 11th Int. Conf. on Control, Automation, Robotics and Vision, ICARCV 2010, Dec., 1070–1077. Palat, R.C., Annamalai, A., and Reed, J.H. (2005). Cooperative Relaying for Ad-hoc Ground Networks Using Swarm UAVs. Proc. IEEE Military Communications Conf. MILCOM, 2005, 1–7. Vicente, K.J. (2002). Ecological interface design: progress and challenges. Human factors, 44(1), 62–78. Vicente, K.J. and Rasmussen, J. (1992). Ecological Interface Design: Theoretical Foundations. IEEE Trans. on Systems, Man and Cybernetics, 22(4), 589–606.