Development of Adaptive Human-Machine Interface to Match Human Satisfaction

DEVELOPMENT OF ADAPTIVE HUMAN-MACHINE INTERFACE TO...

14th World Congress of IFAC

M-6b-02-6

Copyright © 1999 IFAC 14th Triennial World Congress, Beijjng, P.R. Chlna

DEVELOPMENr-I' OF ADAPI'IVE HUMAN-MACHINE INTERFACE TO MATCH HUMAN SA'"fISFACTION

IIisashi Matsunaga and Hiromu Nakazawa

Department ofMechanical Engineering, School ofScience and Engineering, Waseda University~ Tokyo, Japan. Tel / Fax: +81-3-5273-4659 E-mail... [email protected]~waseda.ac.jp

Abstract: The 21st century is likely to see many robots and computer systems coming into the sphe.rc of daily life. Such robots and systems must possess the ability to coope.rate and coexist along with humans. Adaptive. human-machine interfaces that can adapt to the current operator and the current context are requested . In this pape·r an adaptive human-machine interface based on satisfaction measures estimated by e.lectroencephalngram (EEG) measurement has bee,u proposed~ The results of expe-riments showed the possibility of the adaptive human-machine interface based on satisfaction measures. Copyright © 19991FAC Keywords: Human-centered design, Human-maehine interface, IInman factors, Systems design, Systems methodology, User interfaces

1 ~ INTRODUCTION

intelligent quotient but a high emotional quotient as we.]).

Ever since the Industrial Re.volution, the major focus of system design has he.en on designing systems to be automate.d as much as possible. Yet, today, various contradiction s have become apparent in sectors which arc considere,d to have heen successful in adapting unmanned and automation systems (Nakazawa, 1993). The design of a system should be bumanoriented while balancing humans and the automated machine system.

Since the human-machine interface is what connects a human operator with an automatic system, a humanoriented interfac£ should be develope.d. Historically, it has been the user who has had to adapt to the syste-m. However, as systems and tasks increase in <.~rnplexity, user pe,rformance must suffe-r if users must change their beh avior tu suit the system (Norcio and Stan)e-y, 1989). Meanwhile, the. 21st century is likely to see many rooots and computer systems coming into the sphere of daily life. Such robots and systems must posse.ss the ability to cooperate and coexist along with humans. That is, they filUst have not only a high

Therefore, the idea become-s relevant that it is not humans that must meet the syste.m but the system that

must adjust to humans. A human-machine interface that can adapt lo individual users and varying contexts (Norcio and Stanley, 1989) should be developed. Such an interface will e,nable use-IS to focus on their primary tasks and to fully use their useful abilities. Although some· studies have been made to develop adaptive human-machine inteTfaces such as one that adapts to mental workload (Arai et al., 1994), no study has been made. that considers human satisfaction. In this pape·r, an adaptive human-machine inte.rface based on satisfaction measures estimated by frontal

EEG measurement (Matsunaga and Nakazawa, 1997) has been proposed~ The. real-time. prototype system ,vas developed using a mental t ask at the first stage of the realization of the adaptive human-mac.hine interface to match human satisfaction, and the method for adaptation was examined.

6529

Copyright 1999 IFAC

ISBN: 0 08 043248 4

DEVELOPMENT OF ADAPTIVE HUMAN-MACHINE INTERFACE TO...

2. SATISFACTION MEASUREMENT SYSTEM

14th World Congress of IFAC

scales, based on this relation, was then used to evaluate the level of affcL'1ive satisfaction.

2.1 An indicator for satisfaction measurement Since the state. of one 1s mind is supported by the brain, it appears on EEG as weB as in subjective experience. To measure the- state of mind, it is necessary to obtain the relation between EEG and subjective psychological experience. Once this relation is obtained, psychological information on satisfaction can be evaluated by measuring EEG. Then, an indicator for

measuring satisfaction was obtained, which was combine,d psychological and physiological information (Matsunaga and Nakazawa, 1998a). Fig.l shows a flowchart for obtaining this eOTre.lalion. The proce,dure is briefly desc-ribed below.

•

Relation between psychological and physiological information

Fig_l~

Flowchart for obtaining correlation between EEG and subjective. experience.

First, a language-based model of satisfaction was construete-d at the psychological level in order to properly evaluate subjective satisfaction as affe,ctive state~ There have been many studies on self-reported mood, where two hroad dimensions consistently emerge: ple,asantness-unple,asantness and arousal as tb e first two dimensions not rotated, and positive affect and negative affect as the first two Varimax rotated dimensions~ Watson and Te.llege.n (1986) reanalyzed a large number of studies on self-reported mood and showed that positive affect and negative affe.ct represe.nt the major dime·nsions of e-mo1ional experience. There is however no Japanese-language measure to evaluate affective satisfaction. Then, the relation betwee.n multiple mood scales (Terasaki et aI., 1992) and satisfaction was e,xamined. The multiple mood se-ale.s, which is composed of eight emotional moods (e-ach mood consists of 10 Japanese adjectives), was constructed in orde.r to measure multiple mood states. Using principal conlpone.nl analysis, the. re-Iation betwee,n satisfaction and the eight emotional moods was estabHshed on a twodimensional plane consisting of the pleasantncssunple.asantness and arousal axe,s, and also of the, positive and negative affect axes~ The nlultiple mood

Second, single channel frontal EEG was measured from bipolar de,rivation using a wireless syste.m on subjects with ope-ll eyes. EEG measurements are usually taken from referential derivation using a multi-channel system on a subject who stays still with closed eyes. However, it takes a considerable time to connect the electrodes for a multi-channel system, and the system also requires special te·chnicians for the operation and maintenance- of each individual measurement. Furthermore, EEG results vary widely depending on measurement conditions such as the derivation method, or on whether the subjccfs eyes are closed or open. Thus, measure-ment results from subjects with closed eyes can be interpreted only in that setting, and are not applicable to subjects with open eyes, which is the normal awakened state. Using FFf to de.termine the, power spe.ctrurn and examining the appearance rate within the total power, it was found that the appe,arance rate of the er band was positively correlated \Vith affective sati~factioD~ while that of the 13 band was negatively corre.lated. This resuJ1 reflects asymmetries in frontal EEG activity. Whe·eIer et aL (1993) reported that the left frontal region is assoc;a1ed with positive emotion while the right associated with negative emotion4 If the right and left frontal EEGs are activated diffe,re-ntly for satisfying and unsatisfying tasks, this would explain the above correlation. OuT result is also supported by Davidson and Fox (1982) where infants sho\ved greater activation of the left frontal than ofthe right frontal area in response. to happy video stimuli but no differe,Dce -iD response to sad vide,o stimllli4 Furthe.rmore,~ a study of EEG tracings using bipolar de-rivation on individuals who showe·d anxiety states of varying intensity (Cohn, 1946) indicated increased frontal activity and also supports OUT result.

2~2

Development of a satisfaction fneasurement .}yslem

As mentioned in the previous section, satisfaction can be measured by focusing on the appearance rate, which is a relative power obtained by carrying out power spectrum analysis via FFT on frontal EEG measure-ment from bipolar derivation. Although the appe,arance rate is convenient for statistical tre.atment and gives an indicator of satisfaction, it seems to overly compress the EEG data. Therefore" a satisfaction measurement system was dcve-loped (Matsunaga and Nakazawa, 1997) that employed a ne,ural network possessing excellent pattern recognition~ Even without an established model that correlates between psychological and physiological states such as the one in the- study, it is possible. using a neural ne·twork to

6530

Copyright 1999 IFAC

ISBN: 0 08 043248 4

14th World Congress ofIFAC

DEVELOPMENT OF ADAPTIVE HUMAN-MACHINE INTERFACE TO...

acquire experienced knowledge and to reflect individual difference-s.. The net\vork model employed in the study is a three-layered hierarchical model. Learning took place based on the error back propagation method. For the- input signal for le-arning, a normalized power spectrum was use,d, which includes not only information on the appearance rate but also other unidentific.d information like the shape of the power spectrum. Two output signals we·re used, with the corresponding training data as (1.0, 0.0) fOT satisfaction, and (O.O~ 1.0) for dissatisfaction. The input signals were selected from 10 subjects based on tbe,ir subje.ctive e,valuations. Satisfaction degree is the share that the first unit of the output layer has within the sum of the two output units. Fig. 2 shows the structure of the neural network.

Fig.2. Structure of the neural network Fig. 3 shows the, configuration of the developed satisfaction measurement system. The system consists of the detector, the signal processing part which derives tbe powe,r spectrum using :FFT, and the neural network which determines the satisfaction level. Because the system employs a headband with pad electrodes and a wireless transmitting system~ it has the following advantages. ~rhe. e.le.ctrodes can be e.asily attached and detache.d; it is not overly sensitive to noises; the subject is not physically restrained or uncomfortable. The system has another neural network carrying out noise ide.ntification based on the power spectrum. Furtbcrmore~ the system is equipped with a feedback device that uses micro vibrations from an e.ccentric-weight motor. Thus, the worker/operator can work without having his/her condition noticed by othe,r people. Analysis time for satisfaction data can be chosen from either 8~53 or 4.27sec.

2.3 Exafflple ofsatisfaction

"leaSUrenlents

The validity of tbe system can be confirmed if operations are- e-va)uate.d to inde-ed cause either satisfaction or dissatisfaction as they are set lip. For thi s purpose, measureme,nts were taken on su bje.cts pe.rforming various operations but whose. data were not used for neural network learning. Among these,. one e.xample is shown below. Fig. 4 shows measurement results when two subjects were shown thre-e types of video program. uComedyu is a Mr. Bean TV program. Itpear" indicate.s a program with a crueJ conte·nt showing animals being killed and human corpses lying on the ground. "Documentary" is about the cruise ship Titanic. It shows that satisfaction is highest for comedy, next fOT Titanic and lowest for the fear progyam. This result confirmed the subjects' psychological experie.nces.

This result demonstrates that the system was able to successfully evaJuate satisfaction level.. Measurements were· also taken for other operations and subje.cts, further confirming the system's validity (Matsunaga and Nakazawa, 1998b). Furthermore, the authors are presently making a comparison between our system and other study results as well as Mushats emotion spectrum analysis method (1997) which can extract feature assoc.iated with emotions from multichannel EEGs, and are obtaining positive Jcsults~ There is also a pla.n underway to market the satisfaction measurement system in cooperation with Techna Electronics corp. (Tokyo" Japan).

j li

~~lllllj ••ilt_'lill

o

10

15

20

2S

:'ill

~

40

45

t ttin.

I

nmr.

r'ig. 4. Results of satisfaction measurement when two subjc.cts were shown three type,s of video program 3~

ADAPTIVE HUMAN-MACIDNE INTERFACE TO MAT'CH HUMAN SATlSfAC~rION

3.1 Proposal ofAdaptive human-machine interface to lnatch hu""an

5ati~faction

Oft/off

Fig. 3. Configuration of the satisfaction measuremc·nt syste.m

An adaptive human-machine inte.rface to match human satisfaction can be proposed based on the real-time satisfaction measures by the developed satisfaction measurement system. Fig.5 sho\vs the,

6531

Copyright 1999 IFAC

ISBN: 0 08 043248 4

DEVELOPMENT OF ADAPTIVE HUMAN-MACHINE INTERFACE TO...

configuration of the adaptive human-machine interface. to match human satisfaction. This adaptive human-machine interface. bas the conventional interface part which gets the input from the operator and displays the information from the system to the operator, as well as the adaptive part \vhich gets the operator's satisfaction information from the. satisfaction me.asurement system and understands the state of his/her satisfaction and send the signals to the controlle.rs of the machine syste·m in order to change the state of the system~ It is possible using this interface 1u take adaptive action to the human operator based on his/he.r satisfaction change~

14th World Congress of IFAC

.. Whe.n there is no satisfaction change, the interval remains constant. .. The interval changes when the satisfaction le.vel changes. • The. amount by which the interval changes at one tiule is constant. • To initially set off the aJgorithm~ when the subject first registers dissatisfaction during the mental task, the interval is shortened. .. When the interval becomes ze,ro, the previous interval is maintained. Table. 1 shows the algorithm.

Wirele...o;s transmitting

Satisfaction measurement system

Fig. 5. Configuration of the adaptive bumanmachine interface to match human satisfaction

3.2 Development of adaptive human-machine interface to match human satisfaction A prototype system which uses a pacing mental task was de-veloped based on the above concept to test the adaptive human-machine interface. Pacing stress is said to be a kind of stressor th at elicits physiological and psycho-logical changes. However, the results of many studies on pacing stre.ss are· not unanimous. Gregov et at (1997) reported that paced industrial workc-rs were less satisfied with their jobs than unpaced workers, while Khaleque and Hossain (1993) found that satisfaction was highest for the automated workers of a sanitary ware factory, next for the manual workers and lowest for the se.mi-automate.d

workers. Broadbent and Gath (1981) found that repetitive workers of a motor-ear manufacturing plant showed lower job satisfaction than non-re.petitive ones, with no difference in job satisfaction between those who were. paced by an assembly line and those who were not. The subject is presented on tbe CRT monitor with a series of addition problems (Le., the addition of twodigit numbers; for example, 15+30 =). He/she then type.s the answe,r OD the keyboard . Fig.. 6 shows the. outlook of the experiment. The interval bct""reen the presentation of one problem and the next is made to change following an algorithm based on the subjectfs satisfaction level. The algorithm was experimentally develope.d, and was based on the following rule:

Satisfaction

intonnation The prototyfX system changes intervals between problems based on the subje(.:1~s satisfaction changes. Fig~

6. Experimental view

Table 1 The algorithm for adaptation to match human satisfaction Previous act ion ~

.---9" .---9"

Satisfaction change

---r-

---....

---...

~

~

~

----------....

~

----... ----... ----... ~

Present action ~ ~

---....

---.... ---...

---....

~

---....

~

~

---.....

~

: Interval is increased~ increase in satisfaction Interval is maintained; no change in satisfaction : Interval is decrea..--:ed; decrease in satisfaction

---... = ~

6532

Copyright 1999 IFAC

ISBN: 0 08 043248 4

14th World Congress of IFAC

DEVELOPMENT OF ADAPTIVE HUMAN-MACHINE INTERFACE TO...

Analysis time for satisfaction was 8~53 sec. Whether the subje.ct's satisfaction level JULTeased or decTc-ased was de.termined based on the following manner. Prior to the task the satisfaction of the subject who rested for about two minute~s (128 sec.) with eyes open was measured and averaged. When satisfaction during the task decreased by 20 percent from the average-d satisfaction, this was assumed to be a dissatisfied state. Conversely, a satisfaction increase by 20 percent indicated a satisfied state. When one state· was registered four or more- times in the previous seven measurements, the interval was changed . The initial interval was six seconds, and any single interval change took place in two-second increments. When an interval change occurred, this interval was kept constant for about 30 seconds (8.53 X 4). Fig.? shows the result of a subject A working at the task with and without adaptive control. The work unde-r e.ach condition lasted continuously for 45 minutes.. The experiment with adaptive control was conducted t\Vice. It shows that satisfaction becomes high when the. inte-rval changes with adaptive c.ontrol while it is low when the interval is kept constant at six seconds without adaptive controL

25

f

-,

,

3.3 Evaluation of the adaptive hunlon-lllochine interface to match human satisfaction

rnin..

(a) With adaptive. control

I

x: t

min_

il~,1JI I I I ,i ~ j{t

(a) With adaptive control

11

o

XI

2'>

t min_

~l!~! !'i!(!1 fit l

(b) With adaptive control

(c) Without adaptive control (at subjectls own pace) t"'ig.8. Comparison of satisfaction with adaptivecontrol and at subject's own pace (subject B) These two measurement results demonstrate the possibility of the adaptive human-machine intedace in heightening or maintaining satisfaction leve-L In this adaptive metbod~ however, determination of whether the subject becomes satisfied or dissatisfied depends on the comparjson between the- current satisfaction and the averaged satisfaction measured prior to the task. So unless the satisfaction becomes lower than that at resting state~ the initial state (six sec. interval in tbis study) is not changed. Also the measurement prior to the work at re.sting state is not practical in future application . Hence, another method ,vas develope.d for determination of subject's satisfied or dissatisfied state. The average of the- previous seven measurements (8.53 x7 sec~ ) a re used as the based satisfaction. It is

(b) With adaptive. control

maintained until when the inte.rval is changed, and is replaced by tbe average- of the. seven me·asureme·nts prior to the inte-rval change. It is possible using this me.thod to detect the re,c.ent satisfaction change- and to take better adaptive action. Fig.9 shows an example- of the. measurement result using this method.

~

Il

I min.

(c) Without adaptive control

Fig.7. Comparison of satisfaction with and v.'ithout adaptive control (subject A) Xl f mid.

Fig.8 shows the result of a different subject B with adaptive control and at the subjeces own pace. It shows that satisfaction becomes high when interval change occurs based on adaptive. control while at the with subject's own pace remains low.

Fig~9.

An example of satisfaction measurement with adaptive control using varying based satisfaction (Subject C)

6533

Copyright 1999 IFAC

ISBN: 0 08 043248 4

DEVELOPMENT OF ADAPTIVE HUMAN-MACHINE INTERFACE TO...

4. DISCUSSION When the. experiments were conducted usingtbe same mental task above, no significant difference was found betwee,n

se,If-paced

mode

and

machine-paced

(constant at slow or blst speed) mode. In this sense, the adaptive system may be useful in detecting individual human satisfaction and in taking adaptive action ~ However, satisfaction with adaptive control can not simply be compared with that under selfpaced or machine~paced mode because the varying context affects the satisfaction le.veL Still the adaptive control is thought to have the possibility that increases satisfaction rather than other modes be.cause it can detect the satisfaction change in real-time and change the state of the system. In future, the relation between pacing stress and satisfac.tion will be examined together with the development of the adaptive human-machine interface~ which will be useful to design human-oriented systems balancing automation and useful human abilities.

5. CONCLUSION In this paper, a re.aI-time. adaptive human-machine interface to match human satisfaction has been proposed base.d on the satisfaction measures as e,vaJuale.d by the satisfaction measurement system. A~ a first step towards a machine system th a t adjusts to human satisfaction, a prototype- system was developed using a me,ntal task. The results of the expe-rime,nts showed the possibility of the adaptive, bumanmachine interface. The authors have been developing more realistic prototype systems, for instance a simulation system of manipulation task as we.ll as a robot system that can detect human satisfaction and take adaptive action.

ACKNOWLEDGEMENTS This research is supporte.d in part by t'Re.search for the. Future" Program of the Japan Socie-ty for the Promotion of Science (JSPS) under "Science. of Human Based Synthesis" Project (Project numbe,r

96POO703), and Grant-in-Aid fur JSPS fellows.

REFERENCES Arai, T.,

Klllx)~ O~,

Takahashi, M. and Yoshikawa, H.

(1994). Development of adaptive human interface- based on the physiological measures, Human InterfaceN&R, 9, 191-196. [Japanese,] Broadbent, D. E. and Gath, D. (1981). Symptom le,vels in assembly-line workers. In: Machine Pacing and Occupational Stress (G. Salvendy and M.J. Snlith(eds.)}, T'ay]or & t~rancis Ltd~ London, 243-252.

14th World Congress of IFAC

Cohn, R. (1946). The influence of emotion on the human electroencephalogram, The Journal of Nervous and Mental Disease? 104(4),351-357. Davidson, R~ l . and Fox, N. A. (1982). A'iymmetrical Brain Activity Discriminates Between Positive and Negative Affective Stimuli in Human Infants, Science, 218, 1235-1237. Gregov, Lj., Mane.nica, I., and Prorokovic, A. (1997). Job satisfaction and life satisfaction of paced and unpaced job workers, Proceedings of the 13th triennial Congress of the International Ergonomics Association, 1, 477-479.. Kbaleque, A. and Hossain, M.M. (1993). Job satisfaction, fatigue and mental health of manual, semi-automated and automated workers. In: The Ergonomics ofManual Work (W.S. Marras, W. Karwowski, J. L. Smith and L. Pacholski (Eds.)) Taylor & Francis Lld, I-Ondon,441-443. Matsunaga, 11. and Nakazawa, H.(1997). A Study on Human~Oriented Manufacturing System (HOMS) -Development of SatisfaL'1ion Measurement Syste.m (SMS) and Evaluation of Element Technologies of HOMS using SMS, Proc. 1nl. Conf. on Manufacturing Milestones toward the 21st Century, Tokyo, 217-222. Matsunaga, H. and Nakazawa, H. (1998a). Experimental study for satisfaction measurement ~ Re1 ation between subjective satisfaction and frontal EEG measured from bipolar derivation, Jpn. J. Ergonomics, 34(4), 191-201. [Japanese] Matsunaga, H. and Nakazawa, H. (1998b). Satisfaction measurement. In: Human Factors in Organizatioftal Design and Managemeltl- V7 (p.Vink, E.A. Koningsveld and S~ Dbondt (eds. »), Elsevier Science~ 623-628. Musha, T., Terasaki, y~,. Ilaque, It A. and Ivanitsky, A. (1997). Feature extraction from EEGs associated with emotions, Artifu:ial Life and Robotics, 1, 15-19~ Nakazawa, H~(1993). Alternative Human Role in Manufacturing, AI & &~OC~, SpringeT-Verlag London~ London, 7, 151~156, Norcio, A . F. and Stan]ey, J. (1989) . Adaptive human-c.omputer interfaces: A literature survey and perspective, IEEE transactions on systems, man and cybernetics, 19(2), 399-408. Terasaki, M~, Kishimoto, Y. and Koga, A~ (1992). Construction of a multiple mood scale, The Japanese Journal of Psychology, 62(6)~ 350356. (Japane.se] WalsoD, D. and Tc.l1cgen, A. (1986). Toward a consensual structure of mood, Psychological Bulletin, 98(2), 219-235. Wheeler, R~ E., Davidsoo, R. J~ and 'fOlllarkeu, A.(1993). Frontal brain asymmetry and emotional reactivity -A biologic.al substr ate of afie,ctive style, P.sychophysiology, 30~ 82-89~

6534

Copyright 1999 IFAC

ISBN: 0 08 043248 4