Experimental Study of Situation-Adaptive Human-Automation Collaboration for Takeoff Safety

Copyright © IFAC Man-Machine Systems, Kyoto, Japan, 1998

EXPERIMENTAL STUDY OF SITUATION-ADAPTIVE HUMAN-AUTOMATION COLLABORATION FOR TAKEOFF SAFETY

Makoto Itoh', Yasuhiko Takae", Toshiyuki Inagaki', Neville MoraY;

*Institute of Infonnation Sciences and the Center for TARA, University ofTsukuba, Tsukuba 305-8573 Japan ** Doctoral Program in Engineering, University of Tsukuba, Tsukuba 305-8573 Japan # Department of Psychology, University of Surrey,

Guildford, Surrey GU2 5XH, United Kingdom

Abstract: This paper investigates usefulness of dynamic and flexible allocation of authority for decision and control during takeoff of an aircraft. Even though one of the authors ' previous mathematical analysis claims that the authority should be exchanged between humans and automated systems depending on situations, it is not obvious whether human pilots are willing to accept the trading of authority. This paper gives an experimental study for the efficacy of such situation-adaptive autonomy. Copyright ©19981FAC. Keywords: Aircraft operations, Automation, Decision making, Human-centered design, Human factors, Man/machine systems, Safety, Supervisory control

be given authority for decision and control when safety is a factor.

I. INTRODUCTION

Human-centered automation (Billings, 1991 ; Woods, 1989) is one of the central issues in the study of a human-machine system with a supervisory control

By taking, as an example, a rejected takeoff (RTO) problem of an aircraft, it has been proven in Inagaki

configuration (Sheridan, 1992). It is often claimed that human supervisors must bear effective authority over

takeoff should be traded between humans and

(1997) that the authority of decision making during

automation (Woods, 1989). It is not obvious, however, whether the humans must have full authority at all times in every occasion. The humans in a large-complex

automated systems in a situation-adaptive manner. It is not clear, however, whether human pilots are willing

system may face extremely difficult situations. Inagaki (1993) has thus proposed the concept of situation-

an experiment to examine the usefulness of SAA for attaining takeoff safety. How humans feel toward SAA

adaptive autonomy (SAA). Mathematical analyses (Inagaki, 1993, 1997) have shown that computers may

is analyzed from a viewpoint of human acceptance of

to accept the trading of authority. This paper conducts

SAA.

323

RTO action init"ated

2. PROBABILITY MODEL

RTO action corn eted

Consider the situation where an aircraft is taking off. Suppose an engine failure warning is given during the takeoff run. Human pilots have two alternatives: GO (continue the takeoff) and NO-GO (abort the takeoff). The aircraft can stop by the end of the runway if the human pilot initiates procedures to abort the takeoff before VI. The aircraft, on the other hand, may cause an overrun if the takeoff is rejected after achieving V I. The standard procedure for the decision, upon an engine

RTO initiated with delay

engine failure

wr

failure, is: (i) If the airspeed is less than VI' then NOGO. (ii) If V, has already been achieved, then GO. Note here that VI should be regarded as the maximum speed by which the human pilot must initiate RTO actions, instead of just deciding whether to go or not.

RTO action

'Of

RTO completed with delay

RTOaction

"I'

+

RTO initiated with delay

Inagaki (1997) has proven the necessity of dynamic trading of authority for attaining takeoff safety. This section summarizes the results of his work.

RTO completed with delay

(2) V EF before but near V I engine failure warning

!

Let L AS denote the conditional expected risk when a decision is made by an automated system (AS) . An engine failure warning can be false . There are two policies, which are taken by a human when he/she hesitates to say either that the warning is correct or that

RTO action initiated

1+ RTO initiated with delay

it is incorrect: (i) trustful policy (TP), in which the warning is trusted, and (ii) distrustful policy (DP), in which the warning is distrusted. Let L TP or L DP denote the conditional expected risk when the decision is made by a human pilot with TP or DP, respectively.

RTO action completed

+ RTO completed with delay

Fig. I. Engine failure and rejected takeoff (after Inagaki, 1997) This paper conducts an experiment to investigate the effectiveness of SA A for phases (ii) and (iii).

In Inagaki (1997), the RTO problem was analyzed by distinguishing some takeoff rolling phases (Fig. I): (i) If an engine failure warning is given at an airspeed VEF which is far below VI (Fig. 1(1», then L DP S L TP S LAS• This result suggests that the takeoffshould not be fully automated. (ii) If a warning is given before but near

3. RESEARCH VEHICLE A flight simulator has been developed for the RTO experiments. The flight simulator runs on a graphic workstation (SGI), and its interface is designed with VAPS (Virtual Prototypes, Inc.). Fig. 2 depicts the

VI (Fig. 1(2», then L DP < L Tr No fixed order relation is found between L AS and L TP' or between L AS and L Dr (iii) If an engine failure warning is given almost at VI' and no human pilot can initiate RTO actions by V I but AS can (Fig. 1(3», thenL DP
interface of the simulator, which consists of the outside view seen from the windshield (top), PFD (Primary Flight Display: left), and EICAS (Engine Instrument and Crew Alerting System: right). PFD gives: (a) the aircraft speed, (b) VI' (c) V R at which the liftoff procedure must be initiated, and (d) remaining length of the runway. Callouts are given by the computer at

assume that a human pilot must always bear full authority for decision and control. The above results (i) - (iv) suggest the necessity ofSAA.

80 knots, VI' and VR • EICAS gives parameter values

324

Fig. 2. Interface for RTO experiment

for each engine, such as N J (which indicates engine

conditions with the simulated aircraft is as follows:

power in terms of rpm) and EGT (Exhaust Gas Temperature). Subjects are requested to lift off the aircraft safely or to reject the takeoff if necessary.

( 1) Pull the trigger to start a takeoff run. The two engines are then set at full power. (2) Maintain directional control by twisting (if

The simulated aircraft has two engines and an engine

necessary) the joystick for compensating a crosswind effect.

may fail during a takeoff run. A yaw movement is observed when an engine fails . A crosswind, however,

(3) Pull the trigger when he/she decides to "GO." (4) Pull the stick when the airspeed has reached VR.

may also produce a yaw movement. Thus, the yaw movement may not always indicate that an engine has failed. There are some cues to detect the engine failure:

The standard procedure for the GOINO-GO decision,

(1) an engine failure warning in red, which will be given at the top right of EICAS, (2) decline in engine

Left Lever (Thrust Lever)

parameter values, and (3) change in rate of acceleration

Trigger

DD DD DD DD

of airspeed. Subjects are told to control the aircraft with a control box which is shown in Fig. 3. The control box consists of a joystick, two levers, and some switches (including a trigger on the joystick).

Three-Axis Stick

The takeoff procedure under normal operating

Fig. 3. Location of switches on the control box

325

upon an engine failure, is specified as follows : (1) If an engine fails at a speed which is less than V I'

with the takeoff-simulator. Participants were randomly assigned to one of Order of Condition.

then pull the left lever to cut off the engines for rejecting the takeoff.

Training: At the beginning, subjects received a written

(2) If VI has already been achieved, then pull the trigger

description on the purpose of the experiment.

to continue the takeoff.

Explanations on technical terms, interface, and takeoff procedures were also given. Each subject received at

There are four types of outcomes for each trial.

least ten trials with an engine failure in phase I or after

(i) Successful takeoff: The aircraft attains height of 35

V I to learn how to manipulate takeoff procedures.

feet at the end of the runway. (ii) Successful RTO: The takeoff is aborted and the

Subjects were given a few trials without any engine failure. Each subject proceeded to the next stage ifhis

aircraft stops on the runway with no damage. (iii) Marginal takeoff: The altitude (screen height)

or her actions were correct in more than 9 trials out of the last 10.

achieved at the end of the runway is less than 35 feet. (iv) Overrun accident: The aircraft can not stop safely by the end of the runway due to a late NO-GO decision.

Data Collection : Every subject performed 30 trials under each control mode. Each set of trials consisted

SAA carries out the takeoff procedure if an engine fails

of 12 trials in phase 2, 12 trials in phase 3, and 6 dummy trials composed of two phase-I trials, two after-VI

at an airspeed above VI ' IfSAA has the authority, a human pilot is not allowed to make NO-GO decision.

trials, and two trials with no engine failure . The

SAA never aborts a takeoff once V I has been achieved.

parameters, such as VI' VR' VEF , site of an engine failure, and direction of a crosswind, were randomized at every

It could be said that SAA is GO-Minded. If there is no engine failure, SAA never flies the aircraft.

trial. At the end of each trial, subjects were requested to give a subjective rating of "self-confidence" (SC) to

4. EXPERIMENT

represent how confident they were in their ability to perform the task manually. An I I-point rating scale with "0" indicating "not at all" and "10" indicating "completely" was used. In addition to SC, subjects were requested to give subjective ratings of "trust" (T)

4 . 1 Methods The experimental design is a 2 x 2 x 2 factorial design,

in the automation and "reliance" (R) on SAA. Subjects gave subjective ratings after they were told whether

mapping onto Mode of Control x Phase of Takeoff x Order of Condition. Mode of Control is Manual (M) or SAA. Phase of Takeoff refers to phase 2 (Fig. 1(2)) or phase 3 (Fig. 1(3)). In phase 2, an engine fai Is at

there had been an overrun accident or a marginal takeoff, the value of root mean square error for the directional control of the aircraft, and whether SAA had took over the control of the aircraft.

some time point in the time interval [TVI-I .68, TVI-0.84], where TVI denotes the time point when VI is achieved. The time interval [T v l -0.84, T v l ] is defined as phase 3. These time intervals were set based on a preliminary

Performance Measure : Several performance measures were recorded: (I) response time, (2) outcomes of trials, (3) Self-Confidence (SC) , (4) Trust (T), and (5)

experiment. Order of Condition refers to the fact that half the subjects received all the M trials before the SAA trials (M->SAA), and half the opposite (SAA-

Reliance (R).

>M). Mode of Control and Phase ofTakeoffare withinsubjects factors, and Order of Condition is a between-

4.2 Results and Discussion

subjects factor. Before discussing effectiveness of SAA, it may be Subjects were 12 vol\J?teer undergraduate and graduate

useful to examine whether the experimental condition,

students who were paid ¥ 1,500 as a basis for participating a two-hour session. They could win

such as the definition of phases, was appropriate or not. The probability of how often SAA performed the takeoff

additional bonuses (¥200 or ¥500) according to their

procedure was evaluated. Table I gives the summary

performance. None of them had any prior experience

of the data. People could hardly initiate RTO actions

326

00.30 ~

o", 0.50

Table I Probability of takeovers by SAA

... 1-

... 1-

.~ ~ 0.25

.12 20.20 5 ~ c..~

00::

00::

o '"

Order

Phase 2 mean

s.d.

c.. ~

Phase 3 mean

o u~ ~

~

s.d.

0

'" 0.00 ........._ _...1.-_ 2 3 Phase

M->SAA 0.486 0.281 0.945 0.101

~

~

ORDER CSAA->M 0 M ->SAA

u ~

"'0.10 .....-'--_........M SAA Control

(a)

(b)

SAA->M 0.514 0.244 0.972 0.043 Fig. 5. Effects on successful RTOs. before V I if an engine failure occurred in phase 3. On the other hand, subjects were able to reject the takeoff

can be interpreted as follows. As humans experience

with a probability of about 50 percent if an engine failed

more takeoffs with an engine failure at a high speed,

in phase 2. The results indicate that the definition of

the proportion of success in RTO under M mode

the phases in this experiment meets the assumptions in

approaches to that under SAA mode. If people are not

the mathematical model.

experienced at making the GOINO-GO decision, on the other hand, an RTO is more successful under SAA

Data on reaction time were not used in the analyses,

mode than under M mode.

because subjects pulled the trigger sometimes too late for recording the exact time ofmaking "GO" decision.

On the marginal takeoffs, an ANOVA showed that there

As stated in section 3, each trial ends with one of the

(F( I, I 0)=25.18, p
four types of outcomes. On the successful takeoffs, an

of marginal takeoffs with an engine failure was greater

ANOVA gave two significant main effects. Subjects

in phase 2 than in phase 3.

was a significant main effect of Phase of Takeoff

lifted off the aircraft more successfully under SAA mode than under M mode (F(l , I 0)=6.50, p<0.03) (Fig.

There were three overrun accidents under SAA mode,

4 (a» . The proportion of success in takeoff with an

but there were many (52 accidents out of 288 trials)

engine failure was higher in phase 3 than in phase 2

under M mode.

(F(l, I 0)=595 .56, p
If the GOINO-GO decision is made in a too GO-Minded

Phase ofTakeoff (F(l , 10)=4.17, p<0.069), which shows

manner, takeoffs with an engine failure tend to be

that subjects lifted off the aircraft more successfully

continued even when they can be aborted. AnANOVA

under SAA mode than under M mode when an engine

on the probability of continuing an takeoff with an

failed in phase 3.

engine failure in phase 2 did not give the significant

An ANOVA on successful RTOs showed that the

SAA is not too GO-minded.

main effect of Mode of Control. The result shows that proportion of success in RTO with engine failure was higher in phase 2 than in phase 3 (F(l, 10)=46.96,

The above results illustrate efficacies of SAA under

p
time-critical conditions. It is important to know here

of Control and Order of Condition was significant

how people felt toward SAA and whether they were

(F(l,IO)=14.96, p
willing to accept it. An AN OVA on SC showed that the interaction between Mode of Control and Phase of

::: 0.4 o

.,

'O~

.12:2

0.3

1: '"

8.~ o ~ ... :>

Takeoff was significant (Fig. 6). Sheffe test showed

~I

/

~ '" 0.2

M

SAA Mode

.,o

... ~ 0_

MODE

1: '"

CSAA

.-g ...-;0.5 o '" 0..13

o u ... :> ~ '"

that there was a significant difference between the modes when an engine failed in phase 2. The result implies that subjects lost SC to some extent under M

oM

mode when an engine failed in phase 2. The subjects

0 Phase

were likely to hesitate in deciding which alternative to

(b)

choose, and thus they sometimes caused overrun

(a)

accidents. On the other hand, if subjects were supported Fig. 4. Effects on successful takeoffs .

by SAA, they did not lose SC even when an engine

327


u

5. CONCLUSION

7.5

c

/



<::

c

0

7.0

U

...:.

-.;

'"

MODE

This paper found that SAA was useful for attaining

CSAA

takeoff safety. As a whole, SAA in this experiment

OM

was trusted and accepted by subjects. However, SAA

6.5

2

should be implemented with care, because human

3

Phase

reliance on SAA may be lost if intentions of a human

Fig. 6. Two-way interaction between Phase of Takeoff

and SAA are contradictory even though SAA is reliable. Note that SAA takes only actions to continue the takeoff,

and Mode of Control on Self-Confidence

and never takes action for NO-GO. It means VI has been already achieved at the time moment when SAA

failed in phase 2, because SAA could avoid overrun

begins to control the aircraft. Thus, one reason for the decline of reliance is that humans may not understand that it is too late to reject the takeoff when the humans

accidents. T and R were measured only when SAA had the

choose "NO-GO." SAA should therefore be more capable of expressing what actions are to be taken and

authority to make GOINO-GO decision. ANOVAs on them were not straightforward, because some subjects

why. How to design human-machine interface to give

experienced few cases in which SAA took authority when an engine failed in phase 2. Thus, the data ofT and R with an engine failure in phase 3 were analyzed.

SAA the capability must be studied for making SAA acceptable to humans.

The mean value ofT was 8.13 for group M->SAA, and 9.25 for group SAA->M. The mean values ofR were

ACKNOWLEDGMENTS

7.83 and 9.02 for groups M->SAA and SAA->M, respectively. ANOVAs showed no significant effect, which implies that Order of Condition affected neither

This work has been partially supported by the Center for TARA at the University of Tsukuba, Grant-in-Aid

T nor R.

for Scientific Research 07650454, 08650458, and 09650437 of the Japanese Ministry of Education, Science, Sports and Culture, and the First Toyota High-

The data of subjective ratings under SAA mode were broken down depending on whether or not decisions

Tech Research Grant Program.

made by a subject and SAA were contradictory (Table 2). ANOVAs showed that SC and R were lower when the human and SAA did not have the same intention than when they did (F(l ,208)=11.20, p
REFERENCES Billings, C. E. (1991) Human-centered aircraft automation : A concept and guidelines, NASA TM-

effect on T. The results on the ANOVAs can be interpreted as follows. A subject may lose his or her

103885. Inagaki, T. (1993). Situation-adaptive degree of automation for system safety. Proc. ROMAN, 231-236.

confidence when SAA does not accept his or her decision. The reliance on SAA may also decline if the decision of SAA conflicts with the subject's one, ~ven

Inagaki, T. (1997). To go or not to go: Decision under

when the subject thinks that SAA is trustworthy.

time-criticality and situation-adaptive autonomy for

Table 2 Subective ratings under SAA mode

takeoff safety. Proc. lASTED Int 'I Con! Applied Modeling and Simulation, Banff, 144- 147. Sheridan, T. B. (1992). Telerobotics, automation, and

decisions (Man, Auto) conflicting (NO-GO.GO)

coincident (GO. GO)

T

SC

human supervisory control. MIT Press. Woods, D. D. (1989). The effects of automation on

R

mean

s.d.

mean

s.d.

mean

s.d.

6.34

1.79

8.34

1.54

7.95

1.68

7.28

1.80

8.68

1.24

8.49

humans' role: Experience from non-aviation industries . In: Flight Deck Automation: Promises and

Realities (Norman and Orlady, Eds.), NASA CP10036,61-85.

1.49

328