An Experimental Study on Human Cognitives at Man-Machine Interface

C()p\T i ~ h t

©

I F.\ C 11 t h Triclllli,d \ \"orld

(:o ll grt'ss. T alli llll. ES1()llia. l'SSR. I ~'~IO

AN EXPERIMENTAL STUDY ON HUMAN COGNITIVES AT MAN-MACHINE INFERFACE H . Yoshikawa, H. Shimoda, Y. Nagai, S. Kojima and K. Inoue Kvoto University. 6 11 Gokaslw. Uji. japan

Abstract. A basic psychology experiment has been conducted to observe psychophysiological characteristics of on-line human cognitive behavior, where cognitive tasks on memorization and pattern classification were given to subjects by personal computer using simple state transition model. Three types of subjects' data, eye movement data by eye mark recorder, physio-elctric signals by polygraph and verbal reports, were ana l yzed with respect to general characteristics of human cognitive behavior from the obtained characteristics of various bio-informatic data such as saccade, eye attention, pupil reaction and blinking , skin potential r esponse and hea r t rate . It was found that t he psycho-physiological measureme nt utilized was useful to the objective and detailed a nalysis for the related human cognitive process. Keywords.

Man-machine system; Human factors; Cognitives; Eye-movement; Polygraph.

INTRODUCTION

asked to verbalize all that in his mind during congitive tasks (concurrent verbalization), and after the test, he was asked to review his thinking process (retrospective verbalization) (Ericcson, 1980) .

With regards to safe and reliable operation of complex, large-scale modern technology systems such as nuclear power plant (NPP), it is important to understand cognitive factors of operators who supervise the plant system, ie, monitor system status, diagnose its anomaly and select proper counteraction through man-machine interface (MMI). The relevant cognitive process is understood as online event-driven information processing , where the operator manipulates perceived " out-world" information and two types of pre-acquired knowledge (mental model), ie., static model on system structure and function of the plant to be controlled, and dynamic model to predict event transition in the plant process and to select counteraction to control the plant. The potential source of human errors in plant operation lies in (A) structures and contents of the out-world information and the two mental models . and (B) manipulation process of the mental models. Therefore, it is important to understand the characteristics of those A and B, and to reflect them to the improvement of various problems at MMI . The authors conducted a basic laboratory experiment to study generic characteristics of the related human cognitive information processing and to develop effective methodology for measuring and analyzing the relevant cognitive characteristics as objectively as possible .

Video records of EMR data and verbal protocol are later processed to obtain movement of eye mark, its velocity distribution , variation of pupil diameter, etc. Additional device was developed to EMR to measure real-time cross section area of pupil and to obtain eye blinking data easily. The voice records of subjects are documented for experimentor's verbal protocol analysis . Since it was found impossible to measure EEG data and EMR data simultaneously because of the limitation of NAC ' s EMR, only ECG, EDA and respiration curve were recorded. Both skin potential response (SPR) and instantaneous heart rate (HR) were utilized as psycho-physiological measures from the polygraph. Three types of experiments were made to each subject in sequence of more than twenty tests for each experiment . As to the subject of cognitive tasks in those experiment, simple state transltlon model of three-input (keys 1, 2 and 3) and threestate ( : circle, triangle and square shapes) is commonly utilized for lea r ning and pattern classification tasks . The structure of the state transition model is illustrated by graph in Fig. 1 where arrows with numbers mean state transition by input keys . The CRT displays for the three experiments are explained in Fig . 2.

EXPERIMENTAL METHOD In the first test series of learning (Experiment 1), the s ubject is asked to understand the hidden model by his own trial of pushing keys and seeing the changes of shape. The limit of working memory is said to be 7 ± 2 chunks with decay time as short as 7 seconds (Card, 1983) . The purpose of this Experiment 1 is to see how the subject memorizes the rules of 3 input keys x 3 shapes = 9 items (more than magical number 7) effectively .

The experimental set up is made to observe subject's process of performing simple on-line cognitive tasks of learning (memorization) and pattern classification which are selected as basic elements of cognitive task at MMI . Those cognitive tasks are generated using simple state transition model, and given to the subject with the conversation mode to CRT display of personal computer (PC). The measured protocols taken from t he subjects are (i) eye movement data using eye mark recorder (£HR), (ii) several physio - electric signals (EEG, ECG, EDA, etc) using polygraph and (iii) two types of verbal reports. The subject is

The Experiments 2 and 3 are the test series on pattern classification. In those exper iments, the key and state on the CRT screen are automatically changed with time (random variation) according to

449

state transition arr ows . Th e purpose of these two experiments are to kn ow th e s ubj ects ' st rategy to solve the problem, and their e ye movement data dur i ng the course of problem solving. The ob taine d data by EMR and polygraph, and the ope r ation record co ncer ning the subject's conversation with CRT a r e processed and depi c ted as time-line chart shown in Fig. 3 . In Fig . 3, th e verbal r eco rd, c ues to disti ngu is h the samples, the operation r eco rd on key input and the cha ng e o f shape ( ) , c hanges of eye fixation, pupil area, hea rt r ate (HR) and skin potential response ( SPR) are traced on the same time axis. One can easily interpret the details of s ubj ect ' s problem so lving pr ocess by correl ating EMR data a nd physiological data wi th the men t al process estimated by verbal report.

Fig . l State transition model used for cognitive tasks .

EXPERIMENTAL RESULT The experiments wer e cond ucted for 6 s ubjects (st udent s , 5 males, 1 female) . The analyzed results on ge ne r ic cha r acte ri st i cs of the related on- line cog ni t i ve information processi ng are pr ese nt ed.

[Learning ] Experiment 1

@!J State

Ex periment 1 (Learning)

Ke y O]

[1]

Observed memor iza tion types and scaling diffic ulty. The time to solve eac h test of Experiment 1 (to t a l 23 tests) were different by subject and by structure of state transition model , a nd for the way of memor ization , two types of memo ri zatio n were obse r ved , namely (A) statedepend ent memorization ( memorize the state associated with input ke y and t he cha nged state), and (B) input -dependent memorization (memorize the sequence of state transition for a specified input key) .

DJ

Under s tand

GlY e up

[Pattern Classification ] Experiment 2, 3

@]

On e as pe ct of scali ng the dif fic ulty of the state tr ansition model is the following entro py value I wh ich is the average i nforma t ion quantity per one state transition ,

S ta t e Ke y O]

[1] III

(1)

whe r e Pi - is transition pr o bability f r om s tate i to j, a ~d Pi is the probability of state i in sta ti ona r y sta te. If the key selection is assumed t o occu r with Pij = 1/ 3 , then the pro bab i lity Pi is given by

Samples @.

Y\•

for Experiment 2 «~

• j~i

•@J -r- Oe ,~-r-'Oe

~

and

• Y\ ,re"- er

Li Pi =

(2)

L j a ij Pi = Pi

(3)

wh e r e paramete r a- - is taken as 1/3 if transition from i to j is li~Jly, while 0 if it is unlikely . The larger the entropy value, the more diffi c ult the state transition model might be perceived by s ub jects.

Samples for Experiment 3 Fig.2 CRT display for cognitive tasks on learning and patter n class ification.

Neg le cting t he obse r ved personal differences concer ning which t ype o f memorization is likely to use , the relat i on between problem solving time versus entropy is plotted as shown in Fig. 4, for the mean values an d the s tandard devia ti ons of problem solving time . The general tendency see n in Fig. 4 is t hat the larger t he entropy valu e , the larger the ave r age pro blem solving time and the standa rd deviation . Especia lly for the s ubj ects wh o take the memorization type A, there we re more positive correlation s hip between entropy a nd the pro bl em solving time than that in Fig . 4.

the hidden model. and th e s ub ject is requested to find the right model by choosing from the three models di s pl ayed as "samples" on the screen (see Fig. 2). Th e frequency of automat i c key change is varied as 1. 2 and 3 seconds i n experi ment 2, while it is fixed as 1 second i n ex periment 3. The difference bet~een Experiment 2 a nd 3 is that th e s hape s of the three sample stat e transi tion mode ls i n experime nt 2 a r e different with each other, wh i le those of Experiment 3 are the same shape with different key number allocation for the

Another difficu l ty scale is r e lated to how man y conspicuous patterns can be f ound by pushing the same key many times. We selec ted the five ca nd idates of such pattern as shown in Fig. 5 ,

-l50

VERBAL RECORD

CUES

(%)(%)

,

INPUT STATUS

I

I

-Lo

06 I I 00 I 0

0 0 I I 0 0

0 0 I I 0 0

OUT

(%)(%)

I

I

I

I

delete sample 2

(%)

,@ , I

I

Q>

I

t

I

I

I

LOOK

I

I

o

o

()

I

o

I

0 0 oo~Oo Cl, ,,, ,,, ,,, , , , , FIGURE r-r , , ,- , AT SAMPLE, SAMPLE 2 ' ..!. , ,I , , .!

,,

Cl

sure it's 1 because the path from L;. to 0

can omit 3 because L;. makes double loop

I

I

" , ~ lA,

Cl Cl Cl

I

J

I

1_

. I ~ _'

CI~ Cl

: : : If: I

-,-.- T

I

@)(%)/'

I

I

I

I

I

Cl

r-r .. -'- " -'

I _I

"

I

'.

SAMPLE 3~'--~-L~~~~~~~~~~'~~L'.r'~~'~~~~--~----.-

0:00

0:05

0:10

0:15 0:20\ 0:25

delete sample 3

0:30

o:y

0:40

0:45

compare samples

stop blinking

stop blinking

~~liOi0

Area of Pupil

/'/ heart rate rise

Skin Potential

lJ\vu~ 0 6 \

I

skin potential change

400

300

300 250

200

200

150 100

T

j

I1 t

The relations between problem solving time versus number of those patterns in the model are plotted in Fig. 6 (a) to (e), respectively for the above five patterns. We can see f rom Fig. 6 that the more patterns of no-operation, loop and reset, the easier to under stand, while consecutive and toand-fro, not. It was also found that when there exists such "easier" patterns in the model, the subject with the memorization type B has the tendency of "premature formation of ce rt ainty" that he feels confident that he solved the problem even if there still remains to be solved actually. This is thought to undermine the cause of error.

150 100 50

0

0 0.1

0. 9

1.1

!.3

1.5

1.1

Entropy (bi t)

Fig.4

\

skin potential change

namely (i) loop pattern, (ii) consecutive pattern, (iii) to-and-fro pattern, (iv) reset patte rn and (v) no-operation pattern. The problem is by which pattern the subjec t would perceive the model easier to understand.

350

250

0. 5

~

Time-line chart used for analyzing a subjects ' cognitive process in Experiment 2 .

400 Problem Solving 350 Time (sec)

50

heart rate rise

I--------=:0-------------0---

Heart Rate

Fig.3

0:50

Relation between problem solving time versus entropy value in Experiment 1.

Mental image effect. Concerning the effect of showing the graphs of state transition model (see Fig. 1) to subjects, the variation of tendency in

451

Problem 300 Solving Time (sec) 210 200 110 100

(a) Loop Pattern

50

(b) Consecutive Pattern

n-

()

•

• /\

(c) To-and fro Pattern

(a) No-operation pattern

Number of Pallerns

300 Problem Solving 250 Time (sec)

• •

200 150 100

(d) Reset Pattern

50

Ij

n-

•

(b) Loop pattern Problem Solving Time (sec)

Number of Pallerns

300 210 200 150

(e) No-operation Pattern 100

Fig.S

Five patterns of state transition by pushing the same input key.

j

j

0

I

50

o-

(c) Reset pattern the relation between the problem solving time versus entropy is shown in Fig . 7 (a) a nd (b), where (a) is prior to showing , while (b) afterwards. We can see by comparing the both (a) and (b), that the average problem so l ving time i s much s hort ened with less standard deviation, after the image of the problem is indicated to subjects by such graphical forms.

Number of Pallerns

300 Problem Solving 210 Time (sec) 200 . 110 100 . 50

Types of human e rr or. The types of human e rr or observed in the Experiment 1 are summarized as follows; (i) memorization failure at perceiv ing phase, (ii) temporal data missing in working memory, and (iii) automatic extension of partial rule of problem so l ving at inference stage . The l ast type might be the reason of afo re -mentioned "premature confi dence formation".

I

j I

(d ) Consecutive pattern

Number of Pallerns

Problem 300 -, - - - -- - -- - -- - - - , Solving Time (sec) 210

Experiments 2 a nd 3 (Pattern Classification) Problem solving st r ategv I n those experiments 2 and 3 where the subjects had to decide wh ich of the three samples is the right one by looking at the automatic c hange of the state t ran si ti on, all the subjects took the strategy of " step- wise elimination" method . In the both experiments , most of the subjects fi nd one or two cues t o elimi nat e wr ong samples by comparing the three sample models f irst, and then they loo k at the state c hange until the wrong samples are eliminated one by one . To be compared wi th experiment 2 , it seemed rather difficult for the subjects to find such cues in experiment 3, because the shapes of the sam pl e mod els are the same struc ture with different i nput key allocation . At any rate , it was fou nd that the

100 110 100 10

II

(e ) To-and -fro pattern Fig.6

452

Number of Pallerns

Rel ation be t ween problem so lving time versus number of patterns in state tra nsition model .

Problem SolvIng

400

CONCLUSION

150 TIme (sec) lOO

With the purpose of developing objective analysis methodology of cognitive factors at MMI, the authors conducted a basic laboratory experiment to examine subjects' cognitive characteristics using a simple state transition model as online cognitive tasks of learning and pattern classification. Eye mark record er and polygraph were used to obtain the psycho-physiological data from the subjects, together with the verbal reports. The general observation from the experiments thus far conducted and the analysis for those data is that the human tends to take a problem solving strategy with cognitive load as low as possible and that there is a limitation of cognitive information pr ocessing speed for each task. Mental image to the problem is the unnegligible factor for improving human cognitive performance. It was found that personal cognitive load and the processing speed could be estimated as objectively as possible, by utilizin g bioinformatic measurement as proposed in the present paper.

250 200

j

If!

150

100 50 00. 5

0. 1

0. 9

\.I

\.!

1.5

1. 1

Entropy (bit)

(a) Prior to Showing State Transition Model

Problem 400 SolvIng 350 TIme (sec)

lOO 250 200

ISO

100

REFERENCES

SO

o -I --~--~--~~---r--~ 0.5 0.1 0.9 I. 1 I. 3 l. 5 l. 1

Card

S .K. (1983). The psychology of humancomputer interaction, Lawren ce Erlbaum Associates, Hillsdale , New Jersey. Ericsson, K.A., and H. A. Simon (1980). Verbal report as data, Psychological Review, 87, 215.

Entropy(blt)

(b) After Showing State Transition Model

Fig.7

Relation between problem solving time versus entropy value in Experiment 1.

subjects would prefer the above-mentioned st r ategy of lighter cognitive load to more accurate but time-consuming method of memorizing all the rules of state transi tion completely by observing automatic change. Information change effect . In case of Experiment 2, the cycles of automatic state transition were varied as 1, 2 and 3 seconds. We investigated the speed effect of this information change by examining a s ub ject ' s performance r ecords which are depicted by time-line chart shown in Fig. 3. The points of analysis by the time-line charts were (i) the subject ' s eye fixation on whether sam ples or state shapes, (ii) the cues which shou ld be potentially utilized by the subject to eliminate wrong samples, and (iii) the detailed process of his problem solving . The result of missing rate of cu e occurence and rate of c ue detection for all the test series of Experiment 2 are given in Table 1. It is seen from Table 1 that time cycle of 1 second is too fast for the subject to have good performance.

TABLE 1

Effect of Cyclic Information To A Subject's Performance

Time cycle (second) 1 2 3

Missing rate of cue occurence 0.475 0.190 0 .1 29

Change

Rate of cue detection 0.477 0.644 0.609

453