The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload

HUMAN MENTAL WORKLOAD P.A. Hancock and N. Meshkati (Editors) Elsevier Science Publishers B.V. (North-Holland), 1988

185

THE SUBJECTIVE YORKLOAD ASSESSUENT TECHNIQUE: A SCALING PROCEDURE FOR HEASURING MENTAL WORKLOAD Gary B. R e i d H a r r y 6. Armstrong Aerospace M e d i c a l Research L a b o r a t o r y Wright P a t t e r s o n AFB, Ohio

Thomas E. Nygren Department o f Psychology Ohio S t a t e U n i v e r s i t y Mental workload i s proposed t o be a m u l t i d i m e n s i o n a l cons t r u c t t h a t can be l a r g e l y e x p l a i n e d by t h r e e component factors: Time Load, Mental E f f o r t Load, and P s y c h o l o g i c a l S t r e s s Load. I n t h i s paper, we d e s c r i b e a s u b j e c t i v e s c a l i n g approach, t h e S u b j e c t i v e Workload Assessment Techn i q u e (SWAT), t h a t c a p t u r e s t h i s m u l t i d i m e n s i o n a l n a t u r e o f mental workload. We d e s c r i b e t h e SWAT procedure as a twophased method t h a t i n c l u d e s ( a ) a s c a l e developnent phase based on c o n j o i n t measurement and nonmetric s c a l i n g , and ( b ) an event s c o r i n g phase. The developnent o f SWAT and i t s measurement f o u n d a t i o n s a r e discussed. Recent r e s e a r c h i l l u s t r a t i n g SWAT'S widespread u t i l i t y and i t s s e n s i t i v i t y as a measure o f perceived mental workload i s summarized. INTRODUCTION Mental workload i s a c o n s t r u c t t h a t has c o n s i d e r a b l e i n t u i t i v e appeal. Almost everyone can t h i n k of examples where two o r more i n d i v i d u a l s perform e s s e n t i a l l y t h e same a t h l e t i c , w o r k - r e l a t e d , o r academic t a s k t o t h e same o b j e c t i v e l y measured performance l e v e l . Yet, i t i s c l e a r t o t h e i n d i v i d u a l s and t o o b s e r v e r s t h a t some o f t h e s e people must expend much more e f f o r t t h a n o t h e r s t o a c h i e v e t h i s same l e v e l o f performance. The f e e l i n g o f expended e f f o r t appears t o somehow be r e l a t e d t o t h e c o n s t r u c t c a l l e d work1 oad. D e s p i t e t h e importance t h a t many i n v e s t i g a t o r s have a t t a c h e d t o mental workload as a measurable c o n s t r u c t , i t has been p e r p l e x i n g f o r s c i e n t i s t s t o study. As t h e above example i l l u s t r a t e s , t h i s i s p a r t l y because performance measures cannot, o f themselves, d e s c r i b e workload. Operators o f t e n i n c r e a s e e f f o r t as a t a s k becomes more demanding, t h u s i n c r e a s i n g perceived workload, w h i l e s t i l l m a i n t a i n i n g h i g h performance. From t h e p e r s p e c t i v e o f measurement, t h i s i m p l i e s t h a t mental w o r k l o a d i s o n l y mode r a t e l y c o r r e l a t e d w i t h performance measures, and a d d i t i o n a l measures must be developed i f t h e c o n s t r u c t i s t o be adequately described. A g r e a t deal o f research has been conducted o v e r t h e past s e v e r a l y e a r s t o assess t h e f u n c t i o n a l r e l a t i o n s h i p between mental workload and a h o s t o f p h y s i o l o g i c a l , b e h a v i o r a l , and s u b j e c t i v e measures. Recently, s e v e r a l s u b s t a n t i a l papers have presented d i s c u s s i o n s o f ways t o b o t h measure mental w o r k l o a d (O'Donnell & Eggemeier, 1986) and adequately d e s c r i b e i t s m u l t i d i m e n s i o n a l c h a r a c t e r i s t i c s (Gopher & Donchin, 1986). Yet, t h e most fundamental i s s u e r e l a t e d t o t h e s t u d y o f workload, a p r e c i s e d e f i n i t i o n o f t h e term, has remained e l u s i v e and has spawned c o n s i d e r a b l e debate among r e s e a r c h e r s (Moray, 1979).

186

G.B. Reid and T.E. Nygren

One r e a s o n t h a t t h e c o n c e p t o f m e n t a l w o r k l o a d , d e s p i t e i t s d e f i n i t i o n a l e l u s i v e n e s s , i s b o t h t h e o r e t i c a l l y i n t e r e s t i n g and so e a s i l y a c c e p t e d i s t h a t t h e c o n c e p t o f p h y s i c a l w o r k l o a d has been e f f e c t i v e l y u s e d f o r decades. S c i e n t i s t s w o r k i n g i n t h e d i s c i p l i n e s o f e r g o n o m i c s and work p h y s i o l o g y have d e v e l o p e d many measures o f p h y s i c a l work t h a t r e l a t e amount o f work a c c o m p l i s h e d t o t h e e n e r g y c o s t (e.g., oxygen c o n s u m p t i o n A m a j o r work d e s c r i b i n g t h e s e a s s o c i a t e d w i t h l i f t i n g a g i v e n mass). r e s e a r c h methods i s e d i t e d b y S i n g l e t o n , Fox, and W h i t f i e l d ( 1 9 7 3 ) . However, t e c h n o l o g i c a l i n n o v a t i o n s have caused an i n c r e a s i n g amount o f work i n o u r s o c i e t y t o be a s s o c i a t e d w i t h t a s k s t h a t r e q u i r e l i t t l e o r no physical e f f o r t . I n c r e a s i n g l y , t a s k s a r e h e a v i l y loaded w i t h mental a c t i v i t i e s such as i n f o r m a t i o n p r o c e s s i n g , d e c i s i o n making, and s y s t e m monitoring. R e s e a r c h e r s have, t h e r e f o r e , expanded t h e c o n c e p t o f w o r k l o a d t o i n c l u d e m e n t a l work as w e l l as p h y s i c a l work. The e x p a n s i o n o f t h e c o n s t r u c t o f w o r k l o a d t o i n c l u d e m e n t a l a c t i v i t y i s c l o s e l y r e l a t e d t o t h e o r i e s o f a t t e n t i o n and i n f o r m a t i o n p r o c e s s i n g t h a t h a v e been a t o p i c o f r e s e a r c h f o r c o g n i t i v e p s y c h o l o g i s t s (Gopher & Donchin, 1986). Many models h a v e been proposed and d e b a t e d b u t , t o o v e r s i m p l i f y t h e argument, t h e essence o f t h e m a j o r t h e o r i e s i s t h a t t h e human i n f o r m a t i o n p r o c e s s i n g system has a f i n i t e c a p a c i t y o r c a p a c i t i e s , and d i f f e r e n t t a s k s i t u a t i o n s r e q u i r e v a r y i n g degrees o f c a p a c i t y expenditure. I f a person i s i n a h i g h w o r k l o a d s i t u a t i o n , t h e n h e o r she h a s l i t t l e "spare capacity." C o n v e r s e l y , i n a l o w w o r k l o a d s i t u a t i o n , a subs t a n t i a l p o r t i o n o f t h e p e r s o n ' s c a p a c i t y i s untapped. The e x a c t n a t u r e o f t h i s c a p a c i t y o r c a p a c i t i e s has been t h e t o p i c o f c o n s i d e r a b l e d e b a t e (Norman & Bobrow, 1975; Navon & Gopher, 1979; Wickens, 1984; K a n t o w i t z , 1985). I n a d d i t i o n t o t h e study o f t h e t h e o r e t i c a l p r i n c i p l e s t h a t u n d e r l i e t h e c o n s t r u c t o f w o r k l o a d , t h e consequences o f t h e c o n s t r u c t a r e o f c o n c e r n t o systems d e s i g n e r s and e v a l u a t o r s . Modern systems a r e b e i n g d e s i g n e d and put i n t o o p e r a t i o n t h a t i n c o r p o r a t e t h e v i r t u a l e x p l o s i o n o f e n g i n e e r i n g t e c h n o l o g i e s w h i c h have become a v a i l a b l e i n r e c e n t y e a r s . As p r e v i o u s l y n o t e d , t h e s e systems g e n e r a l l y p l a c e o p e r a t o r s i n a d i f f e r e n t t y p e o f work e n v i r o n m e n t f r o m what h a s been t r u e h i s t o r i c a l l y . The o p e r a t o r ' s r o l e i s i n c r e a s i n g l y t h a t o f a system m o n i t o r , i n f o r m a t i o n manager, and d e c i s i o n maker. Many o f t h e manual t a s k s t h a t o p e r a t o r s have p r e v i o u s l y p e r f o r m e d a r e now b e i n g automated. Due t o t h e advances i n computer t e c h n o l o g y and e l e c t r o n i c s e n s i n g , more i n f o r m a t i o n i s a v a i l a b l e t o d i s p l a y t o o p e r a tors. B u t , space t o d i s p l a y t h i s v a s t q u a n t i t y o f i n f o r m a t i o n a t an o p e r a t o r ' s work s t a t i o n has become overcrowded. To r e l i e v e t h i s o v e r c r o w d i n g , mu1 t i m o d e d i s p l a y s and mu1 t i f u n c t i o n s w i t c h e s have been d e v e l oped and i n s t a l l e d i n o p e r a t o r work s t a t i o n s . These advances, a l t h o u g h p r o v i d i n g o p e r a t o r s w i t h more c o m p l e t e i n f o r m a t i o n t h a n was p r e v i o u s l y p o s s i b l e , p l a c e new demands on them. O p e r a t o r s n o t o n l y must a s s i m i l a t e t h e i n f o r m a t i o n p r e s e n t e d t o them, b u t t h e y must f r e q u e n t l y a l s o d e c i d e w h a t i n f o r m a t i o n i s needed and where i t s h o u l d be d i s p l a y e d . On o c c a s i o n , as modern systems have come i n t o o p e r a t i o n , o p e r a t o r s have c o m p l a i n e d t h a t t h e w o r k l o a d a s s o c i a t e d w i t h o p e r a t i n g t h e s e systems i s excessive. A l s o , some a c c i d e n t s and n e a r a c c i d e n t s h a v e r a i s e d q u e s t i o n s a b o u t t h e l e v e l o f o p e r a t o r w o r k l o a d and s y s t e m s a f e t y . Such q u e s t i o n s have had t o be a d d r e s s e d i n human f a c t o r s e v a l u a t i o n s w h i c h , i n t u r n , h a v e d e m o n s t r a t e d a need f o r r e l i a b l e and e f f e c t i v e methods f o r m e a s u r i n g w o r k load. Thus, a c o n s i d e r a b l e amount o f r e s e a r c h i n r e c e n t y e a r s has been

The Subjective Workload Assessment Technique

187

d i r e c t e d toward developnent o f s e n s i t i v e and r e l i a b l e workload measurement i n s t r u m e n t s (cf., O'Donnell & Eggemeier, 1986). Other c h a p t e r s i n t h i s book d e s c r i b e t h e c u r r e n t s t a t e o f much o f t h i s research. Many o f t h e s e measurement procedures a r e very promising, b u t g e n e r a l l y t h e y a r e s t i l l l a r g e l y r e s t r i c t e d t o r e s e a r c h environments. It i s c l e a r , however, t h a t each of t h e s e r e p o r t e d measures has s t r e n g t h s and weaknesses. It appears e q u a l l y c l e a r t h a t because o f t h e c o m p l e x i t y o f t h e workload c o n s t r u c t , i t i s u n l i k e l y t h a t any s i n g l e measure w i l l be c o m p l e t e l y adequate i n prov i d i n g t h e t y p e o f a p p l i e d measurement mechanism t h a t i s d e s i r e d and a t t h e same t i m e be a p p l i c a b l e t o a l l k i n d s o f a p p l i e d work s i t u a t i o n s . These v a r i o u s measures o f mental w o r k l o a d a r e c u s t o m a r i l y d i v i d e d i n t o three classes: ( 1 ) s u b j e c t i v e , ( 2 ) p h y s i o l o g i c a l , and ( 3 ) b e h a v i o r a l o r performance. One o f t h e s e classes, s u b j e c t i v e measures, has c o n s i d e r a b l e appeal f o r a p p l i e d s i t u a t i o n s . The remainder o f t h i s c h a p t e r w i l l be focused upon t h i s c l a s s o f measures and upon one s u b j e c t i v e measure i n particular. S u b j e c t i v e Measurement o f Workload Although c o n s i d e r a b l e e f f o r t has been expended t o d e v e l o p automated and o b j e c t i v e measures o f workload, one method t h a t c o n t i n u e s t o be p o p u l a r i n o p e r a t i o n a l e v a l u a t i o n s i s s i m p l y t o ask t h e o p e r a t o r t o r e p o r t how h a r d he o r she i s working. T h i s i s , i n f'act, how we can d e f i n e a s u b j e c t i v e measure o f workload. It i s one t h a t i s based on a s u b j e c t ' s d i r e c t e s t i mate o r comparison judgment o f t h e workload experienced a t a g i v e n moment. W i l l i g e s and W i e r w i l l e r e p o r t e d i n t h e i r 1979 r e v i e w paper t h a t s u b j e c t i v e measures a r e t h e most f r e q u e n t l y used methods f o r workload assessment. The i n t e r v e n i n g e i g h t y e a r s o f workload r e s e a r c h have n o t dramatically altered the situation. There a r e s e v e r a l reasons f o r t h e p o p u l a r i t y o f s u b j e c t i v e w o r k l o a d measures. The f i r s t and probably t h e most i m p o r t a n t i s t h a t s u b j e c t i v e measures enjoy high face v a l i d i t y . Operators and d e s i g n e n g i n e e r s can r e a d i l y accept t h a t i f o p e r a t o r s t h i n k t h a t t h e r e i s t o o much work a s s o c i a t e d w i t h t h e o p e r a t i o n o f a c e r t a i n system, t h e n d e s i g n a l t e r n a t i v e s must be found. Secondly, s u b j e c t i v e measures a r e somewhat more d i r e c t t h a n many o f t h e o t h e r measures. I f someone wants t o know how much w o r k l o a d i s r e q u i r e d i n a c e r t a i n i n s t a n c e , measures o f p h y s i o l o g i c a l and b e h a v i o r a l v a r i a b l e s r e q u i r e knowledge o f t h e f u n c t i o n a l r e l a t i o n s h i p between t h e s e v a r i a b l e s and workload. A complete u n d e r s t a n d i n g o f t h e s e r e l a t i o n s h i p s i s n o t c u r r e n t l y a v a i l a b l e , a l t h o u g h progress toward t h i s end i s b e i n g made (see o t h e r c h a p t e r s i n t h i s volume; O'Donnell & Eggemeier, 1986; Gopher & Donchin, 1986). Conversely, i n t h e same s i t u a t i o n , i f o p e r a t o r s a r e asked t o assess t h e degree o f workload, t h e y can d e s c r i b e i n a t l e a s t a general o r d i n a l way, how h a r d t h e y a r e working. I n debating t h e issue o f what i s workload, Johanssen, Moray, Pew, Rasmussen, Sanders, and Wickens (1979) have concluded t h a t i f an o p e r a t o r t h i n k s he i s l o a d e d down and under s t r e s s i n a s i t u a t i o n , t h e n one must conclude t h a t h e i s , r e g a r d l e s s o f what o t h e r i n d i c e s m i g h t l e a d you t o conclude. This l o g i c i m p l i e s y e t a n o t h e r reason f o r t h e p o p u l a r i t y o f s u b j e c t i v e measures. O f t e n t h e approach used t o v a l i d a t e o b j e c t i v e measures i s t o demonstrate t h a t t h e s e measures can, i n f a c t , p r e d i c t o r a r e c o r r e l a t e d w i t h subj e c t i v e measures. F i n a l l y , t h e ease a s s o c i a t e d w i t h o b t a i n i n g s u b j e c t i v e measures makes them v e r y a d a p t a b l e t o o p e r a t i o n a l environments l i k e t h e system d e s i g n e v a l u a t i o n s p r e v i o u s l y mentioned. Instrumentat i o n

188

G.B. Reid and T.E. Nygren

requirements a r e minimal and t h e t i m i n g o f d a t a c o l l e c t i o n can be t a i l o r e d t o f i t t h e p a r t i c u l a r operational s i t u a t i o n . D e s p i t e t h e p o p u l a r i t y and u s e f u l n e s s o f s u b j e c t i v e measures f o r operat i o n a l s i t u a t i o n s , u n t i l r e c e n t l y , t h e y were t h e l e a s t researched c l a s s o f workload measures. W i l l i g e s and W i e r w i l l e (1979) n o t e d i n t h e i r r e v i e w t h a t u p t o t h a t p o i n t , i n most cases s u b j e c t i v e measures r e p r e s e n t e d " a s i t u a t i o n - s p e c i f i c , a d j u n c t measurement i n s t r u m e n t w i t h no accompanying v a l i d i t y o r r e l i a b i l i t y data" (p. 552). They a l s o observed t h a t " g i v e n t h e widespread use and general a p p l i c a b i l i t y o f r a t i n g s c a l e s as a t e c h n i q u e o f workload assessment, i t i s s u r p r i s i n g t h a t a r i g o r o u s l y developed workload r a t i n g s c a l e has n o t been developed" ( W i l l i g e s & W i e r w i l l e , 1979, p. 552). I n response t o t h i s recognized need, r e s e a r c h e r s a t t h e U.S. A i r F o r c e ' s H a r r y G. Armstrong Aerospace Medical Research L a b o r a t o r y have developed SWAT, t h e S u b j e c t i v e Workload Assessment Technique (Reid, Shingledecker, Nygren, ti Eggemeier, 1981). SWAT i s a s c a l i n g procedure t h a t has been developed f o r use i n a p p l i e d s e t t i n g s . What d i s t i n g u i s h e s SWAT f r o m most o t h e r s u b j e c t i v e r a t i n g methods i s t h a t i t was r i g o r o u s l y developed t o be r o o t e d i n f o r m a l measurement t h e o r y , s p e c i f i c a l l y c o n j o i n t measurement theory. The o v e r r i d i n g p r i n c i p l e s t h a t have guided t h e developnent o f SWAT have been ( a ) t o d e v e l o p as p r e c i s e a measure as p o s s i b l e w h i l e m i n i m i z i n g t h e i n t r u s i v e n e s s o f t h e d a t a c o l l e c t i o n procedure on t h e o p e r a t i o n a l s i t u a t i o n , ( b ) t o place minimal measurement c o n s t r a i n t s on t h e c o m p l e x i t y o f t h e judgmental t a s k t h a t i s r e q u i r e d o f t h e o p e r a t o r s making workload e v a l u a t i o n s , and ( c ) t o p r o v i d e a mechanism f o r t e s t i n g t h e v a l i d i t y o f t h e formal measurement model t h a t i s assumed by t h e u n d e r l y i n g a d d i t i v e model i n SWAT. One c r i t i c i s m o f t e n made o f s u b j e c t i v e measures i s t h a t t h e y a r e based on t h e s u b j e c t s ' a b i l i t y t o r e p o r t d i r e c t numerical e s t i m a t e s o f workload o r dimensional components o f workload. What i s o f t e n assumed i n t h e s e approaches, w i t h o u t v e r i f i c a t i o n , i s t h a t t h e s u b j e c t s ' judgments have i n t e r v a l o r r a t i o - s c a l e properties. That i s , i f i t i s assumed i n t h e s c a l i n g procedure t h a t we have i n t e r v a l measurement, t h e n i f a s u b j e c t were t o g i v e workload r a t i n g s o f 2, 4, and 6 t o t h r e e d i f f e r e n t t a s k s (A, B, and C ) , t h e d i f f e r e n c e s i n perceived workload between A and B and between B and C would be i n f e r r e d t o be equal. I n o t h e r words, i t i s assumed t h a t s u b j e c t s can make a c c u r a t e e q u a l - i n t e r v a l judgments on t h e workload scales. I n c o n t r a s t , procedures such as b i s e c t i o n o r magnitude e s t i m a t i o n assume t h a t s u b j e c t s can make r a t i o judgments. Thus, t h e s e t e c h n i q u e s assume t h a t a s u b j e c t can make r e l i a b l e i n f e r e n c e s t h a t B i s t w i c e as much work as A and t h a t C i s t h r e e t i m e s as much work as A. Any t i m e a s c a l i n g procedure r e q u i r e s s u b j e c t s t o b i s e c t an i n t e r v a l (e.g., f i n d t h e task t h a t i s h a l f as much work as t h e s t a n d a r d t a s k ) o r t o judge r a t i o s (e.g., t w i c e as l a r g e , etc.), i t i s assuming t h a t t h e numerical e s t i m a t e s a r e on a r a t i o scale. C l e a r l y , t h e s e t e c h n i q u e s make s t r o n g assumptions which must be e m p i r i c a l l y t e s t e d . A t t h e very l e a s t , such s u b j e c t i v e measurement procedures f o r c e a d i f f i c u l t judgment t a s k on t h e s u b j e c t s and would r e q u i r e t h a t t h e y be w e l l - t r a i n e d i n t h e use o f t h e scale. The SWAT procedure t o be d e s c r i b e d below does n o t make s t r o n g assumptions about s u b j e c t s ' a b i l i t i e s t o make judgements. Rather, i n SWAT, s c a l e

The Subjective Workload Assessment Technique

189

d e v e l o p n e n t i s based o n l y on o r d i n a l i n f o r m a t i o n t h a t i s i n f e r r e d f r o m r a n k i n g s o r p a i r e d c o m p a r i s o n judgments. Hence, SWAT o n l y r e q u i r e s t h a t t h e s u b j e c t can m e a n i n g f u l l y o r d e r t h e a l t e r n a t i v e s w i t h r e s p e c t t o t h e l e v e l o f perceived workload. The SWAT s c a l e was d e v e l o p e d on t h e b a s i s o f a m i n i m a l s c a l i n g method t h a t a l s o has f a c e v a l i d i t y . I n SWAT, one o b t a i n s a c t u a l w o r k l o a d o r d e r i n g s t o produce a w o r k l o a d s c a l e , i n a manner s i m i l a r t o t h e way m a r k e t i n g r e s e a r c h e r s , f o r example, u s e o b s e r v e d preference o r d e r i n g s t o o b t a i n a s t r e n g t h o f preference scale. In the r e m a i n d e r o f t h i s c h a p t e r , we w i l l d e s c r i b e t h e d e v e l o p n e n t p r o c e s s , t h e measurement b a s i s , and t h e p h i l o s o p h y t h a t has been used i n t h e d e s i g n o f SWAT, as w e l l as p r e s e n t some o f t h e d a t a t h a t h a v e been o b t a i n e d u s i n g t h i s procedure.

MENTAL WORKLOAD OPERATIONALLY DEFINED As p r e v i o u s l y mentioned, a d e f i n i t i o n o f t h e t h e o r e t i c a l c o n s t r u c t , m e n t a l w o r k l o a d , t h a t a l l w o r k l o a d s c i e n t i s t s c a n a c c e p t does n o t y e t e x i s t . T h i s was even more t r u e i n 1980 when SWAT d e v e l o p n e n t began. One e x p l a n a t i o n as t o why t h e p r e c i s e d e f i n i t i o n h a s been s o e l u s i v e may r e l a t e t o one a s p e c t o f w o r k l o a d t h a t many s c i e n t i s t s do a g r e e upon. Many researchers i n t h e f i e l d b e l i e v e t h a t mental workload i s n o t a s i n g l e u n i d i m e n s i o n a l phenomenon, b u t i s a c o n s t r u c t composed o f s e v e r a l e l e m e n t s or d i m e n s i o n s . A t t h i s p o i n t , however, t h e agreement among r e s e a r c h e r s a p p e a r s t o end. I n an a t t e m p t t o d e v e l o p a consensus d e f i n i t i o n o f m e n t a l w o r k l o a d , we c o n d u c t e d a l i t e r a t u r e r e v i e w and n o t e d what many s c i e n t i s t s b e l i e v e d t o be t h e c r i t i c a l components t h a t go i n t o t h e p e r c e p t i o n o f ment a l wprkload. T a b l e 1 p r e s e n t s t h e e l e m e n t s t h a t went i n t o o v e r 20 s c i e n tists d e f i n i t i o n o f m e n t a l w o r k l o a d i n 1980. These d e f i n i t i o n s were s t u d i e d f o r a r e a s o f agreement; i n many cases, a t r a n s l a t i o n was p e r f o r m e d t o c a p t u r e t h e essence o f an i n v e s t i g a t o r ' s d e f i n i t i o n w h i l e p u t t i n g a l l o f t h e d e f i n i t i o n s i n common terms. Based on o u r r e v i e w and d e s p i t e t h e d i s a g r e e m e n t as t o a p r e c i s e d e f i n i t i o n o f m e n t a l w o r k l o a d , i t c a n b e observed, i f our t r a n s l a t i o n i s a c c u r a t e , t h a t t h r e e v a r i a b l e s a p p e a r i n a majority o f the definitions. The l i t e r a t u r e r e v i e w c l e a r l y i n d i c a t e d t h a t a l m o s t e v e r y o n e t h o u g h t t h a t i n some way t i m e p r e s s u r e i s a m a j o r component o f w o r k l o a d . This i s , t o some e x t e n t s u p p o r t e d by t h e p r a c t i c e w i t h i n t h e a i r c r a f t i n d u s t r y o f u s i n g t i m e l i n e a n a l y s i s as t h e p r i n c i p a l way o f e v a l u a t i n g t h e adequacy o f a c o c k p i t design w i t h r e s p e c t t o o p e r a t o r workload. As a r e s u l t o f t h e g e n e r a l agreement r e g a r d i n g t i m e , t h e c o n c e p t u a l framework t h a t was d e v e l oped f o r SWAT i n c l u d e d Time Load as t h e f i r s t f a c t o r or d i m e n s i o n . Time Load, as o p e r a t i o n a l l y d e f i n e d f o r our purposes, means b o t h t i m e a v a i l a b l e and t a s k o v e r l a p . C l e a r l y , i f t h e t i m e r e q u i r e d t o p e r f o r m a t a s k exceeds t h e t i m e a v a i l a b l e , t h e o p e r a t o r has a t i m e l o a d problem. Another t i m e f a c t o r , w h i c h on o c c a s i o n i s o v e r l o o k e d , i n v o l v e s t a s k o v e r l a p . I f an o p e r a t o r i s p e r f o r m i n g a complex t a s k , i t may be made u p o f many component t a s k s or s u b t a s k s . Each o f t h e s e t a s k s h a s i t s own t i m e demands. I f we assume t h a t t h e o p e r a t o r has t h e s k i l l s or a b i l i t i e s demanded by t h e t a s k s , as l o n g as t h e t a s k s c a n b e c o m p l e t e d s e q u e n t i a l l y , t h e o p e r a t o r can m a i n t a i n p e r f o r m a n c e a t an a c c e p t a b l e l e v e l . I f , o n t h e o t h e r hand, t h e t a s k s s t a r t t o compete f o r t h e o p e r a t o r ' s t i m e r e s o u r c e s , h e or sh? w i l l b e f o r c e d t o e v a l u a t e t h e t a s k s f o r p r i o r i t y and a l l o w some t a s k s performance t o d e t e r i o r a t e and/or t h e i r completion t o be delayed. Under t h i s s e t o f c i r c u m s t a n c e s , we c o n t e n d t h a t t h e o p e r a t o r i s a l s o u n d e r a t i m e load.

190

G.B. Reid and T.E. Nygren TABLE 1.

TIME LOAD

ELEMENTS DEFINING MENTAL WORKLOAD

M E N T A L EFFORT LOAD

PSYCHOLOGICAL STRESS LOAD

The Subjective Workload Assessment Technique

191

The second d i m e n s i o n t h a t t e n d e d t o emerge f r o m i n s p e c t i o n o f t h e o p i n i o n s o f r e s e a r c h e r s i n o u r l i t e r a t u r e r e v i e w i s one t h a t d e a l s w i t h t a s k f a c t o r s such as d i f f i c u l t y , c o m p l e x i t y , o r e f f o r t . T h i s d i m e n s i o n i s r e l a t e d t o t h e w e a l t h o f r e s e a r c h i n c o g n i t i v e p s y c h o l o g y where t h e demands a s s o c i a t e d w i t h v a r i o u s l e v e l s o f a t a s k have been m a n i p u l a t e d by such t h i n g s as t h e number o f e l e m e n t s t h a t t h e s u b j e c t must p r o c e s s , t h e f o r c i n g f u n c t i o n d r i v i n g a t r a c k i n g t a s k , i n d u c t i v e reasoning, d e d u c t i v e reasoning, o r memory r e t r i e v a l . T h i s d i m e n s i o n a l s o t e n d s t o encompass t h e c o n c e p t o f mental c a p a c i t y o r c a p a c i t i e s r e f e r r e d t o previously. I n a p p l y i n g t h i s model, one assumes t h a t t h e human o p e r a t o r h a s a l i m i t e d c a p a c i t y . P e r f o r m a n c e o f one t a s k may consume a c e r t a i n amount o f an o p e r a t o r ' s r e s o u r c e s , w h i l e a n o t h e r t a s k may consume o t h e r r e s o u r c e s . The i m p l i c a t i o n i s t h a t t h e r e s o u r c e s t h a t a r e n o t expended i n t a s k p e r f o r m a n c e a r e h e l d i n r e s e r v e t o b e used f o r o t h e r t a s k s o r as a way t o prt more e f f o r t toward accomplishing a c u r r e n t task. The e x a c t n a t u r e o f t h i s l i m i t e d c a p a c i t y i s t h e s u b j e c t o f a l a r g e body of r e s e a r c h b u t t h e b a s i c n o t i o n o f l i m i t e d c a p a c i t y f o r work seems t o b e i n h e r e n t i n t h e c o n c e p t u a l i z a t i o n o f m e n t a l w o r k l o a d ( D o n c h i n & Gopher, 1986). The second d i m e n s i o n p o s t u l a t e d f o r t h e SWAT framework was, t h e n , c a l l e d M e n t a l E f f o r t Load. Mental E f f o r t Load i n v o l v e s such processes as p e r f o r m i n g c a l c u l a t i o n s , m a k i n g d e c i s i o n s , a t t e n d i n g t o i n f o r m a t i o n sources, p l a c i n g i n f o r m a t i o n i n s h o r t t e r m memory and r e t r i e v i n g i t , r e t r i e v i n g r e l e v a n t i n f o r m a t i o n f r o m l o n g t e r m memory, and e s t i m a t i o n . T h i s l i s t o f f e r s a s u g g e s t i o n as t o t h e k i n d s o f p r o c e s s e s t h a t a r e a s s o c i a t e d w i t h t h i s d i m e n s i o n and i s n o t intended t o be i n c l u s i v e . I n essence, M e n t a l E f f o r t Load i s t h e d i m e n s i o n t h a t i s u s e d t o a c c o u n t f o r most o f t h e c a p a c i t y e f f e c t s d i s c u s s e d e a r l ie r . The t h i r d commonly o b s e r v e d c h a r a c t e r i s t i c o f w o r k l o a d d e a l s w i t h t h e g e n e r a l c o n c e p t o f p s y c h o l o g i c a l s t r e s s and seems t o encompass a number o f o p e r a t o r v a r i a b l e s such as m o t i v a t i o n , t r a i n i n g , f a t i g u e , h e a l t h , and emot i o n a l state. T h i s d i m e n s i o n may b e r e p r e s e n t e d b y such s p e c i f i c s t r e s s o r s as f e a r o f p h y s i c a l harm, f e a r o f f a i l u r e , t e n s i o n , u n f a m i l i a r i t y , and d i s o r i e n t a t i o n , t o name a few. I n addition, physical stressors such as t e m p e r a t u r e , v i b r a t i o n , G - f o r c e s , and n o i s e may b e i n c l u d e d . These a r e s t r e s s o r s t h a t a r e known t o a f f e c t p e r f o r m a n c e when t h e y a r e present i n moderate t o h i g h l e v e l s . However, a t l o w l e v e l s t h e y may o n l y be a s o u r c e o f i r r i t a t i o n t o t h e o p e r a t o r . I n t h e s e s i t u a t i o n s , some d e g r e e o f e f f o r t may be r e q u i r e d b y t h e o p e r a t o r t o manage h i s o r h e r d i s c o m f o r t and, t h u s , a f f e c t t h e p e r c e i v e d w o r k l o a d . Presence o f v a r i a b l e s such as t h e s e i s d e f i n e d as b e i n g p a r t o f t h e m e n t a l w o r k l o a d d i m e n s i o n , psychological stress. Hence, we c a l l e d t h e t h i r d d i m e n s i o n P s y c h o l o g i c a l S t r e s s Load and d e f i n e d i t as a n y t h i n g t h a t c o n t r i b u t e s t o an o p e r a t o r ' s confusion, f r u s t r a t i o n , and/or anxiety.

It i s i m p o r t a n t t o emphasize t h a t t h e above summary d e f i n i t i o n o f m e n t a l w o r k l o a d was n o t proposed t o end t h e t h e o r e t i c a l d e b a t e c o n c e r n i n g a p r e cise definition. R a t h e r , t h e d e f i n i t i o n i s i n t e n d e d t o c a p t u r e most o f t h e i m p o r t a n t components t h a t a p p e a r t o i n f l u e n c e p e o p l e ' s percept% of workload. The p l r p o s e o f a t h r e e - d i m e n s i o n a l d e f i n i t i o n , r a t h e r t h a n one t h a t a t t e m p t e d t o i n c l u d e a l l r e l e v a n t d i m e n s i o n s , was t o make t h e measSWAT i s i n t e n d e d urement o f w o r k l o a d f e a s i b l e i n o p e r a t i o n a l s i t u a t i o n s . t o be a pragmatic approach t o t h e e s t i m a t i o n o f mental workload i n operational situations. An o v e r r i d i n g c o n c e r n i n d e v e l o p i n g t h i s p r o c e d u r e was t o minimize i n t r u s i o n t o operators w h i l e providing t h e best possible

192

G.B. Reid and T. E. N-vgren

mechanism f o r d i s c r i m i n a t i o n o f w o r k l o a d l e v e l s , e s p e c i a l l y d i f f e r e n t i a t i n g between p e r c e p t i o n s o f m o d e r a t e and h i g h l e v e l s . These o b j e c t i v e s seemed t o b e most amenable t o t h e u s e o f a s u b j e c t i v e s c a l i n g procedure. Since t h e working d e f i n i t i o n postulated a multidimens i o n a l c o n s t r u c t , a m u l t i d i m e n s i o n a l a l t e r n a t i v e t o t r a d i t i o n a l unidimens i o n a l s c a l i n g approaches was used. A s c a l i n g approach c a l l e d c o n j o i n t m e a s u r e m e n t / c o n j o i n t s c a l i n g a p p e a r e d t o b e f e a s i b l e and h a d been u s e d i n c l o s e l y a s s o c i a t e d e f f o r t s m e a s u r i n g systems o p e r a b i l i t y ( D o n n e l l & O'Connor, 1978; D o n n e l l , 1979). It was chosen as an a p p r o a c h w i t h good p o t e n t i a l f o r m e a s u r i n g t h e complex c o n s t r u c t o f w o r k l o a d . Because o f t h e r e l a t i v e r e c e n c y o f i t s d e v e l o p n e n t and i t s f u n d a m e n t a l r e l a t i o n s h i p t o SWAT, a b r i e f o v e r v i e w o f t h e c o n j o i n t measurement m e t h o d o l o g y w i l l b e presented.

CONJOINT MEASUREMENT AND CONJOINT SCALING I n many judgment and d e c i s i o n m a k i n g s i t u a t i o n s where a s u b j e c t i v e s c a l i n g t e c h n i q u e seems p a r t i c u l a r l y r e l e v a n t o r u s e f u l , i t i s o f t e n assumed t h a t t h e v a r i a b l e o f i n t e r e s t , i n t h i s c a s e m e n t a l w o r k l o a d , i s a complex phenomenon t h a t i s a c t u a l l y c o m p r i s e d o f s e v e r a l p e r c e p t u a l l y i n d e p e n d e n t d i m e n s i o n s (i.e., Time Load, M e n t a l E f f o r t Load, P s y c h o l o g i c a l S t r e s s Load).' It i s a l s o o f t e n t h e c a s e t h a t s c i e n t i s t s w o u l d l i k e t o know t h e c o m p o s i t i o n r u l e t h a t p e o p l e a c t u a l l y use t o combine i n f o r m a t i o n f r o m t h e s e p e r c e i v e d d i m e n s i o n s o r f a c t o r s i n t o t h e more complex c o n s t r u c t . C o n j o i n t measurement t h e o r y p r o v i d e s a p o w e r f u l m e t h o d o l o g y f o r accomplishing this. I t s power l i e s i n t h e f a c t t h a t i t uses o n l y o b s e r v e d o r d i n a l o r rank o r d e r i n f o r m a t i o n a b o u t t h e complex c o n s t r u c t i n o r d e r t o e m p i r i c a l l y e s t a b l i s h a combination r u l e t h a t f i t s a respondent's data. Axiom T e s t s f o r C o n j o i n t Measurement A l t h o u g h t h e m a t h e m a t i c a l f o u n d a t i o n s f o r c o n j o i n t measurement t h e o r y h a v e been i n e x i s t e n c e f o r many y e a r s ( H o l d e r , 1 9 0 1 ) , p r o c e d u r e s f o r d e v e l o p i n g s c a l e s were i m p r a c t i c a l u n t i l t h e developnent o f numerical a n a l y s i s a l g o r i t h m s f o r u s e on modern compcters. I n 1964, Luce and Tukey p u b l i s h e d t h e f i r s t a r t i c l e t h a t described a s e t o f s u f f i c i e n t conditions f o r addit i v e c o n j o i n t measurement i n t w o f a c t o r s . I n t h e i r c l a s s i c a l work on measurement t h e o r y , K r a n t z , Luce, Suppes, and T v e r s k y ( 1 9 7 1 ) b u i l t on t h i s work and e a r l i e r i n d e p e n d e n t work by K r a n t z ( 1 9 6 4 ) and T v e r s k y (1967) and extended t h e t h e o r y o f a d d i t i v e c o n j o i n t s t r u c t u r e s i n t o a general t h e o r y o f p o l y n o m i a l c o n j o i n t measurement f o r s i m p l e p o l y n o m i a l c o m p o s i t i o n r u l e s i n t h r e e o r more f a c t o r s . The g e n e r a l t h e o r y as o u t l i n e d by K r a n t z e t a l . ( 1 9 7 1 ) p r o v i d e s f o r a s e r i e s o f axioms, w h i c h can be t e s t e d o n a s e t o f d a t a , t o d i s c r i m i n a t e 'It i s i m p o r t a n t t o d i s t i n g u i s h t h e c o n c e p t s t a t i s t i c a l i n d e p e n d e n c e f r o m p e r c e p t u a l independence. What i s c r i t i c a l f o r a v a l i d a d d i t i v e r e p r e s e n t a t i o n o f a psychological c o n s t r u c t i s n o t t h a t t h e dimensions o r f a c t o r s a r e completely u n c o r r e l a t e d i n t h e r e a l world, b u t r a t h e r t h a t t h e i n d i v i d u a l d e c i s i o n maker p e r c e i v e s them a s b e i n g p e r c e p t u a l l y i n d e p e n d e n t . I n o t h e r words, t h e i n d i v i d u a l c a n a l w a y s m e a n i n g f u l l y e v a l u a t e d i f f e r ences i n one f a c t o r w i t h t h e o t h e r s h e l d c o n s t a n t . (See K r a n t z e t a l . , 1971, f o r a f u r t h e r d i s c u s s i o n o f t h i s p r o p e r t y . )

The Subjective Workload Assessment Technique

193

among f o u r s i m p l e polynomial models t o determine which o f them b e s t f i t t h e s e t of data. F o r example, i n our case we l e t T, E, and S r e p r e s e n t t h e t h r e e proposed workload dimensions o f Time Load, Mental E f f o r t Load, and P s y c h o l o g i c a l S t r e s s Load. As shown i n d e t a i l l a t e r i n o u r s p e c i f i c d i s c u s s i o n of SWAT, t h r e e l e v e l s f o r each o f t h e s e dimensions can be d e f i n e d and l a b e l e d as t i , t 2 , t g ; e l , e2, e3; and s1, s2, and s3, respectively. F i n a l l y , l e t g ( t l ) , h ( e l ) , and k ( s 1 ) i l l u s t r a t e t h e s u b j e c t i v e s c a l e values a s s o c i a t e d w i t h t h r e e o f t h e l e v e l s f o r a g i v e n individual. These l e v e l s o f t h e t h r e e f a c t o r s combine t o f o r m a u n i q u e workload c o m b i n a t i o n (1,1,1), and i t s o v e r a l l judged value, f ( t 1 , e l , 51). can be found v i a e i t h e r : an a d d i t i v e model, i f f ( t i , el, s i ) = g(ti)+h(ei)+k(sl),

(1)

a m u l t i p l i c a t i v e model, i f f ( t i , el, s i ) = g(ti)*h(ei)*k(sl),

(2)

a d i s t r i b u t i v e model, i f f ( t i , e l , s i ) = g ( t )*[h(e i ) + k ( s

111,

(3)

o r a d u a l - d i s t r i b u t i v e model, i f f(t1,

el, si) = g(t

(4)

Note t h a t i n t h e l a t t e r t h r e e models, t h e o v e r a l l value o f t h e combined e f f e c t o f t h e t h r e e f a c t o r s , f ( t 1 , e l , s l ) , c o u l d be c o m p l e t e l y erased i f one o f t h e m u l t i p l i c a t i v e f a c t o r s has a z e r o l e v e l . I n t h i s case i t would n o t m a t t e r what t h e l e v e l s o f t h e o t h e r f a c t o r s were. F o r an a d d i t i v e model, o f course, t h i s i s n o t t h e case, s i n c e a zero l e v e l o f a f a c t o r would make o n l y t h a t f a c t o r i r r e l e v a n t f o r t h e combined s t i m u l u s e f f e c t . Since i n many a p p l i c a t i o n s one would n o t expect t o f i n d a m u l t i p l i c a t i v e f a c t o r w i t h t h i s z e r o l e v e l p r o p e r t y , most t h e o r e t i c a l and e m p i r i c a l r e s e a r c h i n c o n j o i n t measurement has focused on t h e a d d i t i v e model. The K r a n t z e t a l . (1971) axioms d e f i n e f i v e o r d i n a l p r o p e r t i e s t h a t a r e u s e f u l i n d i f f e r e n t i a t i n g among t h e models i n Equations 1-4. In addition, a l l a r e necessary a l t h o u g h n o t s u f f i c i e n t f o r t h e a d d i t i v e model. These a r e s i m p l e or s i n g l e f a c t o r independence, j o i n t f a c t o r independence, d o u b l e cancel 1 a t i on, d i s t r i b u t i v e cancel 1a t i o n , and d u a l - d i s t r i b u t i ve cancellation. I t i s c l e a r f r o m t h e r e s u l t s o f a r e c e n t Monte C a r l o s t u d y (Nygren, 1985) t h a t t h e c r i t i c a l axioms t h a t a r e used t o assess a d d i t i v i t y a r e s i m p l e independence, j o i n t independence, and d o u b l e c a n c e l l a t i o n . Simple or s i n g l e f a c t o r independence means t h a t t h e o r d e r i n g o f t h e l e v e l s o f one f a c t o r (e.g., Time Load) must s t a y t h e same a t a l l t h e l e v e l s o f t h e o t h e r f a c t o r s (e.g., E f f o r t Load and S t r e s s Load). Note t h a t t h i s s i m p l e independence i s an axiom t h a t d e s c r i b e s a m o n o t o n i c i t y ; i t does n o t r e f e r t o s t a t i s t i c a l independence. Hence, t h e o r e t i c a l l y i t would be q u i t e p o s s i b l e , f o r example, t o f i n d t h a t Time Load was independent (monotonic) o f E f f o r t and S t r e s s , b u t t h a t E f f o r t was n o t independent o f S t r e s s and Time or S t r e s s was n o t independent o f E f f o r t and Time. To t h e e x t e n t t h a t t h e m o n o t o n i c i t y or s i m p l e independence p r o p e r t y h o l d s , an a d d i t i v e model i s supported. For s i m p l i c i t y , we l e t ( l , l , l ) , (1,1,2), (1,1,3), ,

...

G.B. Reid and T.E. Nygren

194

(3,3,3) r e p r e s e n t t h e s t i m u l i f o r m e d b y c o m b i n i n g a1 p o s s i b l e l e v e l s o f t h e Time Load, E f f o r t Load, and S t r e s s Load f a c t o r s . As an i l l u s t r a t i o n o f t h e s i m p l e independence axiom, suppose t h a t i t was f o u n d t h a t an i n d i v i d u a l o r d e r e d t h e s t i m u l u s c o m b i n a t i o n s ( l , [ l , 1 ) < (2,[1,1]) but (1,[2,2]) > (2,[2,2]). These o r d e r i n g s w o u l d be a v i o l a t i o n o f s i n g l e f a c t o r i n d e p e n d e n c e f o r Time Load o f E f f o r t Load and S t r e s s Load, because t h e o r d e r i n g o n t h e Time Load f a c t o r i s " < " i n one c a s e (1 < 2 ) , b u t ">" i n t h e o t h e r c a s e when t h e c o m b i n a t i o n o f E f f o r t and S t r e s s changes f r o m [1,1] t o [2,2]. Note t h a t t h i s i s very s i m i l a r t o f i n d i n g an i n t e r a c t i o n i n an a n a l y s i s o f v a r i a n c e where a dependent v a r i a b l e c a n n o t b e e x p l a i n e d b y m a i n e f f e c t s alone. J o i n t f a c t o r i n d e p e n d e n c e i s s a t i s f i e d when t h e o r d e r i n g o f a l l combinat i o n s o f t h e l e v e l s o f any t w o o f t h e f a c t o r s (e.g., Time Load and E f f o r t Load) s t a y t h e same f o r a l l l e v e l s o f a t h i r d v a r i a b l e ( S t r e s s Load). In a manner comparable t o t h a t f o r s i n g l e f a c t o r independence, t h e r e a r e t h r e e f o r m s o f j o i n t f a c t o r independence--Time and E f f o r t j o i n t l y i n d e p e n d e n t o f S t r e s s , S t r e s s and E f f o r t j o i n t l y i n d e p e n d e n t o f Time, and Time and S t r e s s j o i n t l y i n d e p e n d e n t o f E f f o r t . 2 The o r d e r i n g s ([2,2],2) > ([1,1],2) b u t ([2,21,3) < ([1,1],3) represent a v i o l a t i o n o f j o i n t f a c t o r i n d e p e n d e n c e f o r Time and E f f o r t o f S t r e s s , because [2,2] and [1,1] produce o p p o s i t e o r d e r i n g s i n t h e c o r r e s p o n d i n g p a i r s when combined w i t h l e v e l s 2 and 3 o f t h e S t r e s s Load f a c t o r . F i n a l l y , d o u b l e c a n c e l l a t i o n i s d e f i n e d f o r a p a i r o f f a c t o r s each w i t h t h r e e l e v e l s and i s s a t i s f i e d i f t h i s 3 x 3 m a t r i x i s c o n s i s t e n t w i t h r e g a r d t o t h e o r d e r i n f o r m a t i o n i n i t s d i a g o n a l s ( K r a n t z e t al., 1971; K r a n t z and T v e r s k y , 1971).3 The t e r m " c a n c e l l a t i o n " i s u s e d s i n c e what t h e axiom r e a l l y i m p l i e s i s t h a t t h e psychological value o f a shared l e v e l o f a f a c t o r c a n b e e l i m i n a t e d o r " c a n c e l e d " f r o m each o f t w o s t i m u l u s combinations w i t h o u t a f f e c t i n g t h e i r o r d e r i n g with respect t o workload. Such a p r o p e r t y must, o f c o u r s e , h o l d i n an a d d i t i v e model s i n c e i t h o l d s a l g e b r a i c a l l y i n t h e a d d i t i o n o f r e a l numbers. I n p r a c t i c e , t h e way t h e a x i o m t e s t i n g p r o c e d u r e w o r k s i s t h a t , g i v e n a complex c o n s t r u c t made u p o f t h r e e d i m e n s i o n s as i n t h e c a s e o f SWAT, s u b j e c t s a r e r e q u i r e d t o order t h e s t i m u l u s c o n d i t i o n s t h a t a r e generated b y f o r m i n g a l l 27 c o m b i n a t i o n s o f t h e t h r e e l e v e l s o f each d i m e n s i o n i n a 3 x 3 x 3 design. These r a n k o r d e r d a t a a r e t h e n s u b j e c t e d t o t h e i n d e pendence and c a n c e l 1 a t i o n axiom t e s t s . The o b t a i n e d t h r e e - d i m e n s i o n a l ZLet A1, A z , and A 3 r e p r e s e n t t h r e e f a c t o r s i n a c o n j o i n t d e s i g n . Then we can d e f i n e s i m p l e i n d e p e n d e n c e and j o i n t i n d e p e n d e n c e i n an A1 x A2 x A 3 d e s i g n as i s inde i f and&i,

A1

n d e n t o f A 2 and A 3 whenever ( a l , a2, a 3 ) > ( b i , b2, b3) > ( b i , b2, b3), and

A 1 and A 2 a r e

( b l , bz, a3)

i n t l y i n d e p e n d e n t o f A3 whenever ( a i , a2, i3' oand o n l y if ( a l , az, b3) > ( b i , b2. b3).

82,

a3) >

3Double C a n c e l l a t i o n i s s a t i s f i e d i f ( a l , bz, a3) > ( b l , c2, a3), and ( b l , a2, a3) > ( c i r bz, a 3 ) , t h e n t h i s i m p l i e s ( a l , a2, a 3 1 > ( c i , C 2 , C3).

a3)

The Subjective Workload Assessment Technique

195

d a t a m a t r i x i s examined f o r c o m p l i a n c e w i t h t h e o r d e r i n g s e x p e c t e d among t h e s t i m u l u s c o m b i n a t i o n s o f t h e l e v e l s o f Time, E f f o r t , and S t r e s s when t h e axioms a r e s a t i s f i e d . I f t h e s u b j e c t ' s rank o r d e r data a r e consist e n t , t h e p r o p e r t i e s o f t h e models i n E q u a t i o n s 1-4 can b e t e s t e d t o h e l p d e t e r m i n e a b e s t - f i t t i n g model. Even t h o u g h t h i s p r o c e d u r e c a n examine t h e f u l l r a n g e o f models i n E q u a t i o n s 1-4, i n a p p l i c a t i o n s a s s o c i a t e d w i t h t h e s c a l e d e v e l o p n e n t phase o f SWAT, t h e p r o c e d u r e i s u s e d e s s e n t i a l l y t o v e r i f y t h e adequacy o f a n a d d i t i v e model. T h i s i s based o n t h e f i n d i n g t h a t o v e r f i v e y e a r s and an e s t i m a t e d number o f c a r d s o r t s t h a t exceeds one t h o u s a n d , l e s s t h a n one p e r c e n t have been a n a l y z e d t h a t were b e t t e r r e p r e s e n t e d by one o f t h e n o n a d d i t i v e p o l y n o m i a l models i n E q u a t i o n s 2-4. Conjoint Scaling It i s o f t e n t h e case, however, t h a t i n a d d i t i o n t o k n o w i n g t h e c o m p o s i t i o n r u l e t h a t d e s c r i b e s t h e way s u b j e c t s combine d i m e n s i o n s t o f o r m a complex phenomenon, t h e i n v e s t i g a t o r w o u l d a l s o l i k e t o have s c a l e v a l u e s t o r e p r e s e n t t h e s u b j e c t i v e v a l u e s o f v a r i o u s l e v e l s of b o t h t h e complex phenomenon and i t s component d i m e n s i o n s . T h i s i s p r e c i s e l y t h e case f o r SWAT and w o r k l o a d . The p r o c e d u r e t h a t i s used f o r t h i s purpose i s o f t e n c a l l e d numerical c o n j o i n t s c a l i n g i n order t o d i f f e r e n t i a t e i t from t h e a x i o m a t i c c o n j o i n t measurement p r o c e d u r e d e s c r i b e d above. Before t h e development o f m u l t i d i m e n s i o n a l s c a l i n g c o m p u t e r a l g o r i t h m s , i t was e s s e n t i a l l y impossible t o simultaneously f i n d these subjective scale v a l u e s f o r t h e l e v e l s o f b o t h t h e component d i m e n s i o n s and t h e i r combined effect. The s c a l i n g r o u t i n e i n SWAT t h a t i s used t o e s t a b l i s h a s c a l e f o r m e n t a l w o r k l o a d a c t u a l l y c o n t a i n s t w o such d i s t i n c t s c a l i n g p r o c e d u r e s . They a r e based o n m o d i f i c a t i o n s o f t w o n o n m e t r i c s c a l i n g a l g o r i t h m s , MONANOVA ( K r u s k a l , 1965) and NONMETRG (Johnson, 1973).

A n o n m e t r i c s c a l i n g p r o c e d u r e i s one t h a t a t t e m p t s t o f i n d t h e b e s t f i t t i n g s e t o f i n t e r v a l - s c a l e d values f o r t h e l e v e l s o f t h e perceptually i n d e p e n d e n t d i m e n s i o n s and t h e i r r e s u l t a n t combined e f f e c t based o n l y o n t h e rank o r d e r r e l a t i o n s h i p s t h a t a r e p r e s e n t i n t h e d a t a . Thus, nonm e t r i c s c a l i n g methods d i f f e r f r o m m e t r i c s c a l i n g p r o c e d u r e s i n t h a t t h e y d o n o t assume a l i n e a r r e l a t i o n s h i p between o b s e r v e d d a t a and f i n a l s c a l e values. N o n m e t r i c p r o c e d u r e s d o n o t need t o make t h e sometimes q u e s t i o n a b l e a s s u m p t i o n t h a t t h e r e s p o n d e n t c a n and w i l l make r e l i a b l e r a t i n g s t h a t have i n t e r v a l - s c a l e p r o p e r t i e s when j u d g i n g a complex c o n s t r u c t l i k e mental workload. A nonmetric s c a l i n g procedure o n l y r e q u i r e s t h e d a t a t o be r e l i a b l y r a n k o r d e r e d . The c o m p a r i s o n o f n o n m e t r i c t o m e t r i c i s , t h e n , e q u i v a l e n t t o f i n d i n g a b e s t - f i t t i n g monotonic f u n c t i o n r a t h e r t h a n a l i n e a r f u n c t i o n r e l a t i n g t h e scaled v a r i a b l e s t o t h e observable data. G i v e n t h e proposed a d d i t i v e c o m p o s i t i o n r u l e , each o f t h e s c a l i n g a l g o r i t h m s i n SWAT f i n d s a s e t o f s c a l e v a l u e s f o r t h e t w e n t y - s e v e n w o r k l o a d c o m b i n a t i o n s (3 x 3 x 3 ) such t h a t ( a ) t h e y a r e a d d i t i v e c o m b i n a t i o n s o f t h e s c a l e v a l u e s f o r t h e t h r e e l e v e l s o f t h e Time, E f f o r t , and S t r e s s f a c t o r s , and ( b ) t h e 27 s c a l e v a l u e s a r e as m o n o t o n i c as p o s s i b l e w i t h t h e s u b j e c t ' s o r i g i n a l r a n k o r d e r i n g o f t h e 27 w o r k l o a d c o m b i n a t i o n s . Though i t may n o t seem a t f i r s t t o be i n t u i t i v e l y r e a s o n a b l e , t h e r e s t r i c t i o n o f a n a d d i t i v e model c o u p l e d w i t h t h e o v e r d e t e r m i n a t i o n o f o r d e r i n g s among s t i m u l u s s c a l e v a l u e s b a s e d on t h e o b s e r v a b l e r a n k o r d e r i n g s , a r e s u f f i c i e n t t o a l l o w t h e nonmetric s c a l i n g algorithms t o f i n d a unique, bestf i t t i n g s e t o f s t i m u l u s values w i t h i n t e r v a l - s c a l e properties. The

196

G.B. Reid and T.E. Nygren

d e f i n i t i o n o f b e s t - f i t t i n g i s what d i f f e r e n t i a t e s t h e MONANOVA-based and NONMETRG-based procedures. The f i r s t s c a l i n g a l g o r i t h m t h a t i s used i n t h e SWAT program i s based on a m o d i f i c a t i o n o f K r u s k a l ' s monotonic t r a n s f o r m a t i o n procedure, MONANOVA ( K r u s k a l , 1965). MONANOVA performs a n o n m e t r i c s c a l i n g o f t h e d a t a v i a t h e w i d e l y used STRESS-based l e a s t - s q u a r e s approach. The s c a l i n g a n a l y s i s i s performed e i t h e r on each i n d i v i d u a l d a t a s e t s e p a r a t e l y o r on an a v e r age data m a t r i x as s e l e c t e d by t h e i n v e s t i g a t o r . T h i s procedure produces s c a l e values f o r each o f t h e l e v e l s o f t h e f a c t o r s and f o r t h e s t i m u l u s combinations produced by combining a l l o f t h e l e v e l s o f a l l o f t h e f a c tors. A n o r m a l i z a t i o n o f t h e s c a l e f o r t h e s t i m u l u s combinations r e s c a l e s t h e combinations s o t h a t t h e l o w e s t s c a l e value ( f o r s t i m u l u s c o m b i n a t i o n (1, 1, 1) i s zero and t h e h i g h e s t s c a l e v a l u e (3, 3, 3 ) i s 100. T h i s n o r m a l i z a t i o n i s p a r t i c u l a r l y u s e f u l i f t h e s t i m u l i a r e designed, as i n t h e case o f workload, such t h a t t h e l o w e s t (1, 1, 1) and t h e h i g h e s t (3, 3, 3 ) s t i m u l u s combinations a r e meaningful anchors f o r t h e complex phenomena under i n v e s t i g a t i o n . The SWAT procedure begins by rank o r d e r i n g t h e d a t a f r o m t h e s m a l l e s t t o t h e l a r g e s t , i f t h e y a r e n o t a l r e a d y i n t h a t form. Because t h e procedure i s nonmetric, f r o m t h i s p o i n t on o n l y rank o r d e r s o f t h e d a t a and n o t t h e d a t a values themselves a r e used. An a r b i t r a r y s e t o f i n i t i a l s c a l e values f o r t h e l e v e l s o f t h e f a c t o r s a r e formed t o produce i n i t i a l e s t i m a t e s o f t h e 27 s t i m u l u s combinations. From t h e s e i n i t i a l s c a l e values, a m a t r i x o f what a r e c a l l e d d i s p a r i t i e s i s formed. D i s p a r i t i e s are transformed d a t a values t h a t a r e monotonic w i t h t h e o r i g i n a l d a t a and as c l o s e as p o s s i b l e t o t h e i n i t i a l s e t o f workload s c a l e values. Next, a badness-off i t measure, STRESS, i s computed t o determine how c l o s e l y t h e monotonic a l l y t r a n s f o r m e d d i s p a r i t y values match t h e e s t i m a t e d s c a l e values f r o m t h e a d d i t i v e model. STRESS i s computed by f i n d i n g t h e square r o o t o f t h e sum o f t h e squared d e v i a t i o n s between t h e d i s p a r i t y values and t h e e s t i mated s t i m u l u s values. I f t h e o r i g i n a l rank d a t a a r e i n p e r f e c t agreement w i t h an a d d i t i v e r e p r e s e n t a t i o n , t h e n m o n o t o n i c a l l y t r a n s f o r m e d d i s p a r i t i e s w i l l be found t h a t , when s u i t a b l y normalized, a r e i d e n t i c a l t o t h e e s t i m a t e d s t i m u l u s s c a l e values, p r o d u c i n g a STRESS v a l u e o f zero. S u b j e c t s ' d a t a are, however, g e n e r a l l y n o t w i t h o u t some random e r r o r . In T y p i c a l l y then, t h e a l g o r i t h m w i l l n o t f i n d a STRESS v a l u e o f zero. t h e s e cases, t h e a l g o r i t h m works i t e r a t i v e l y . Following t h e comprtation o f STRESS, t h e e s t i m a t e d s t i m u l u s s c a l e values a r e r e c a l c u l a t e d v i a a leaSt-SqUareS e s t i m a t i o n procedure s i m i l a r t o t h a t employed i n s t a n d a r d regression analysis. The p a r t i a l d e r i v a t i v e o f STRESS w i t h r e s p e c t t o each s c a l e value i s found and a numerical a n a l y s i s procedure known as t h e method of g r a d i e n t s i s used t o f i n d a new s e t o f b e s t - f i t t i n g ( i n t h e l e a s t squares sense) s t i m u l u s s c a l e values. New d i s p a r i t i e s a r e formed, a new STRESS v a l u e i s computed, and t h e i t e r a t i v e process i s c o n t i n u e d u n t i l no improvement i n STRESS can be found. Following the l a s t iteration, the e s t i m a t e d s c a l e values f o r t h e 27 s t i m u l u s combinations a r e found and a r e n o r m a l i z e d as p r e v i o u s l y described, s o t h a t c o m b i n a t i o n (1, 1, 1) has a s c a l e v a l u e of z e r o and (3, 3, 3 ) has a s c a l e v a l u e o f 100. S c a l i n g employing a m o d i f i c a t i o n o f Johnson's (1973) n o n m e t r i c monotone r e g r e s s i o n procedure i s t h e f i n a l s t e p i n SWAT. It may a t f i r s t seem redundant t o perform two s c a l i n g procedures i n SWAT, s i n c e b o t h w i l l y i e l d i d e n t i c a l r e s u l t s f o r p e r f e c t l y a d d i t i v e data. A problem w i t h t h e s c a l i n g

The Subjective Workload Assessment Technique

I97

a l g o r i t h m d e s c r i b e d above and STRESS-based a1 g o r i t h m s i n g e n e r a l , however, i s t h a t t h e y a r e prone, i n a number of common n o n a d d i t i v e cases, t o p r o duce s c a l i n g s o l u t i o n s t h a t f o r c e t i e s i n t h e s c a l e v a l u e s f o r t h e l e v e l s of some of t h e f a c t o r s . T h i s produces a d e g e n e r a t e s o l u t i o n t h a t has t h e appearance o f a p e r f e c t f i t t o a n a d d i t i v e model. N i c k e r s o n and M c C l e l l a n d ( 1 9 8 4 ) p r o v i d e examples o f seven such common s i t u a t i o n s . It i s c l e a r f r o m t h e i r and o u r p r e v i o u s work (cf., Nygren, 1985) t h a t u n l e s s one examines t h e d a t a c a r e f u l l y w i t h r e s p e c t t o t h e c o n j o i n t axioms f o u n d i n SWAT, a z e r o l e v e l of STRESS o b t a i n e d f r o m MONANOVA-based s c a l i n g a l o n e m i g h t l e a d t h e u s e r t o an e r r a n t c o n c l u s i o n o f a d d i t i v i t y among t h e f a c t o r s , as w e l l as t o poor e s t i m a t e s o f t h e s t i m u l u s s c a l e v a l u e s . The second s c a l i n g p r o c e d u r e i n SWAT, t h e n , i s u s e d t o p r o v i d e a n o t h e r s c a l i n g of t h e d a t a , t h i s t i m e based on a b a d n e s s - o f - f i t measure o t h e r t h a n STRESS. T h i s measure, THETA, d i f f e r s f r o m STRESS i n t h a t i t i s based on a p a i r w i s e method i n w h i c h t h e d i f f e r e n c e s i n s c a l e v a l u e s f o r a l l p o s s i b l e p a i r s o f s t i m u l i ( 3 5 1 p a i r s f o r t h e 27 s t i m u l i i n SWAT) a r e compared w i t h t h e d i f f e r e n c e s i n t h e o r i g i n a l ranks. As i n t h e p r e v i o u s s c a l i n g a l g o r i t h m , t h i s r o u t i n e s t a r t s by f i n d i n g a s e t o f e s t i m a t e s o f t h e s t i m u l u s s c a l e values. F o r e f f i c i e n c y , i t uses t h e f i n a l e s t i m a t e s f o u n d by t h e p r e v i o u s STRESS-based procedure. I f t h e d a t a do, i n f a c t , c o n f o r m t o a n a d d i t i v e model, t h e p r o c e d u r e s t o p s a f t e r one i t e r a t i o n , s i n c e t h e s c a l e v a l u e s have a l r e a d y been d e t e r m i n e d . If t h e data are n o t a d d i t i v e , t h e n t h e b a d n e s s - o f - f i t measure THETA i s computed by summing t h e d i f f e r e n c e s i n s c a l e values f o r a l l p a i r s o f s t i m u l i f o r which t h e o r i g i n a l r a n k s a r e n o t i n t h e same o r d e r as t h e e s t i m a t e d s c a l e v a l u e s . T h i s sum i s t h e n n o r m a l i z e d by d i v i d i n g by t h e sum o f a l l d i f f e r e n c e s i n s c a l e v a l u e s and t a k i n g t h e s q u a r e r o o t . The n u m e r a t o r o f t h i s t e r m , and t h u s THETA, w i l l b e z e r o i f a l l p a i r s o f r a n k s and p a i r s o f e s t i m a t e d s c a l e v a l u e s a r e i n t h e same o r d e r . As i n t h e c a s e o f STRESS, t h e p a r t i a l d e r i v a t i v e o f THETA ( a c t u a l l y THETA-squared) i s t a k e n w i t h r e s p e c t t o each s c a l e v a l u e i n o r d e r t o f i n d new e s t i m a t e s t h a t w i l l m i n i m i z e t h e d i f f e r ences i n s c a l e v a l u e s f o r w h i c h t h e r e a r e i n c o r r e c t p a i r w i s e o r d e r i n g s . The i t e r a t i v e p r o c e d u r e i s t h e n c o n t i n u e d u n t i l n o s i g n i f i c a n t improvement i n t h e e s t i m a t e d s c a l e v a l u e s t h a t w i l l m i n i m i z e THETA can be found. It i s i m p o r t a n t t o n o t e t h a t t h i s THETA measure i s s t r o n g l y r e l a t e d t o K e n d a l l ' s Tau c o e f f i c i e n t , a l t h o u g h t h e y a r e n o t a s i m p l e f u n c t i o n o f o n e another, I n SWAT f o r example, f o r a s e t o f r a n k s t h a t f i t an a d d i t i v e model, Tau w i l l b e 1.0, i n d i c a t i n g t h a t a l l 351 p a i r s o f e s t i m a t e d s c a l e v a l u e s a r e i n t h e same o r d e r as t h e 3 5 1 p a i r s o f ranks. For nonadditive d a t a , i t i s s t i l l p o s s i b l e f o r THETA t o b e 0.0 ( b y p r o d u c i n g t i e d s c a l e v a l u e s ) b u t f o r Tau t o n o t b e e q u a l t o 1.0. The m a j o r a d v a n t a g e o f THETA o v e r STRESS i s found, t h e n , i f e r r o r (i.e., n o n a d d i t i v i t y ) occurs i n a s u b j e c t ' s d a t a ; t h e THETA-based s c a l i n g i s much more l i k e l y t o d e t e c t e r r o r t h a n i s t h e STRESS-based s c a l i n g . G e n e r a l l y , however, t h e t w o p r o c e d u r e s w i l l produce v e r y s i m i l a r r e s u l t s . I n a p p l i c a t i o n s o f SWAT where t h e y d i f f e r s i g n i f i c a n t l y i n t h e i r e s t i m a t e s o f s c a l e values, t h e r e s e a r c h e r h a s a much b e t t e r chance o f d i a g n o s i n g why t h e e r r o r o r nona d d i t i v i t y o c c u r r e d t h a n i f h e o r s h e had u s e d o n l y one o f t h e t w o s c a l i n g methods. F i n a l l y , i t i s , o f course, obvious t h a t b o t h s c a l i n g procedures w i l l a l w a y s y i e l d a s e t o f a d d i t i v e s c a l e v a l u e s t h a t a r e o n l y as good as t h e e f f o r t t h a t went i n t o t h e c r e a t i o n o f t h e d e s c r i p t i o n s o f t h e l e v e l s o f t h e f a c t o r s themselves. I n t h e n e x t s e c t i o n , we d i s c u s s t h i s s c a l e d e v e l o p n e n t procedure.

198

G.B. Reid and T.E. Nygren

SCALE DEVELOPMENT Any t i m e an i n v e s t i g a t o r wants t o use a r a t i n g scale, he o r she must ( a ) d e v e l o p a s e t o f d e s c r i p t o r s t o r e p r e s e n t t h e d i f f e r e n t p o i n t s on t h e s c a l e , and ( b ) t r a i n r a t e r s as t o what t h e meanings o f t h e d e s c r i p t o r s a r e and how t h e y a r e t o be used t o r a t e some event. I n p r a c t i c e , t h i s process i s o f t e n not given t h e l e v e l o f a t t e n t i o n t h a t theory requires. In c l a s s i c a l u n i d i m e n s i o n a l s c a l i n g such as T h u r s t o n i a n s c a l i n g o r L i k e r t s c a l i n g , t h e i n v e s t i g a t o r should, w i t h t h e a i d o f s u b j e c t m a t t e r e x p e r t s , w r i t e d e s c r i p t o r s and t h e n s u b j e c t t h e s e d e s c r i p t o r s t o an e v a l u a t i o n by a sample o f s u b j e c t s f r o m t h e p o p l a t i o n t h a t w i l l u l t i m a t e l y use t h e scale. The e v a l u a t i o n i s used t o s e l e c t which o f t h e c a n d i d a t e s c a l e i t e m s have t h e g r e a t e s t d i s c r i m i n a b i l i t y . T h i s process c o n t i n u e s u n t i l a s e t o f d e s c r i p t o r s i s a r r i v e d a t f o r each o f t h e p o i n t s on t h e s c a l e . A f t e r t h e s c a l e has been developed, t h e n each new sample o f s u b j e c t s t h a t a r e r e q u i r e d t o use t h e s c a l e must be c a r e f u l l y t r a i n e d s o t h a t t h e y understand t h e meaning o f t h e d e s c r i p t o r s i n t e n d e d by t h e o r i g i n a l g r o u p o f judges. I n SWAT, s c a l e developnent i s an a p p l i c a t i o n o f t h e c o n j o i n t measurement procedure o u t l i n e d above. The process o f d e t e r m i n i n g a c o m p o s i t i o n r u l e means t h a t each t i m e a workload i n v e s t i g a t i o n i s conducted t h e s u b j e c t s d e f i n e t h e r e l a t i v e w e i g h t s and t h e c o m p o s i t i o n r u l e t h a t f i t t h e i r perc e p t i o n s o f workload. T h i s i s d i f f e r e n t f r o m o t h e r s c a l i n g methods i n t h a t ( a ) t h e same s u b j e c t s d e f i n e t h e s c a l e and a p p l y i t t o r a t i n g s o f e v e n t s , and ( b m e s c a l e developnent process i s used t o t r a i n s u b j e c t s t o understand t h e meaning o f t h e d e s c r i p t o r s . As d e s c r i b e d e a r l i e r , f o r t h e purpose o f SWAT, workload i s d e f i n e d as b e i n g composed o f t h r e e l e v e l s o f each o f t h e t h r e e dimensions: Time Load, Mental E f f o r t Load, and P s y c h o l o g i c a l S t r e s s Load. Oescri p t i o n s o f t h e s e dimensions a r e presented i n Table 2 and t h e i r c o m b i n a t i o n i n t o t h e t h r e e dimensional workload c o n s t r u c t i s r e p r e s e n t e d i n F i g u r e 1. Each o f t h e c e l l s o f t h i s m a t r i x i n F i g u r e 1 i s r e p r e s e n t e d by a c o m b i n a t i o n o f one o f t h e d e s c r i p t o r s f o r each o f t h e dimensions, y i e l d i n g a t o t a l o f 27 combinations. These d e s c r i p t o r s a r e t y p e d on a s e t o f i n d e x c a r d s s o t h a t each c e l l i s r e p r e s e n t e d by a s e p a r a t e card. T h i s deck o f c a r d s i s t h e medium employed i n o b t a i n i n g t h e r a t e r ' s judgment o f t h e r e l a t i v e workload each c o m b i n a t i o n r e p r e s e n t s t o him o r her. S u b j e c t s a r e r e q u i r e d t o go t h r o u g h a c a r d s o r t procedure where t h e y p l a c e t h e c a r d s r e p r e s e n t i n g t h e 27 c e l l s o f t h e t h r e e - d i m e n s i o n a l m a t r i x i n rank o r d e r b e g i n n i n g w i t h t h e c o m b i n a t i o n o f d e s c r i p t o r s t h a t r e p r e s e n t s t h e l o w e s t workload s i t u a t i o n (1, 1, 1) and e n d i n g w i t h t h e c o m b i n a t i o n t h a t r e p r e s e n t s t h e h i g h e s t workload s i t u a t i o n (3, 3, 3 ) . w i t h an o r d e r i n g o f t h e 25 o t h e r s t i m u l i i n between. The s u b j e c t s a r e encouraged t o t h i n k o f s i t u a t i o n s f r o m t h e i r own e x p e r i e n c e s t h a t would have been a p p r o p r i a t e l y d e s c r i b e d by a p a r t i c u l a r combination. They t h e n compare t h a t s i t u a t i o n w i t h a s i t u a t i o n r e c a l l e d f o r a n o t h e r c o m b i n a t i o n and make a judgment as t o which o f t h e s i t u a t i o n s r e p r e s e n t s t h e h i g h e r p e r c e i v e d workload. S u b j e c t s t h e n place t h e s e t w o c a r d s i n t h e proper o r d e r and s e l e c t a n o t h e r c a r d and r e p e a t t h e same d e c i s i o n process. The s u b j e c t s a r e i n s t r u c t e d t o t r y t o imagine a s i t u a t i o n f o r each c a r d b u t i f t h e y cannot t h i n k o f an e v e n t t h a t c o u l d have been d e s c r i b e d by a c e r t a i n combination, t h e y a r e requested t o place t h e c a r d i n t h e i r o r d e r i n g a t t h e p o i n t where i t would f a l l i f an e v e n t d i d e x i s t t h a t would be p r o p e r l y

The Subjective Workload Assessment Technique TABLE 2.

.

SWAT

199

DIMENSIONS

Time Load

1.

O f t e n have spare time. I n t e r r u p t i o n s or o v e r l a p among a c t i v i t i e : o c c u r i n f r e q u e n t l y or n o t a t a l l .

2.

O c c a s i o n a l l y have s p a r e t i m e . a c t i v i t i e s occur frequently.

3.

Almost never have spare t i m e . I n t e r r u p t i o n s o r o v e r l a p among a c t i v i t i e s a r e very f r e q u e n t , o r o c c u r a l l t h e t i m e .

I.

I n t e r r u p t i o n s or o v e r l a p among

Mental E f f o r t Load

1.

Very l i t t l e conscious mental e f f o r t or c o n c e n t r a t i o n r e q u i r e d . A c t i v i t y i s almost a u t o m a t i c , r e q u i r i n g l i t t l e o r no a t t e n t i o n .

2.

Moderate conscious mental e f f o r t or c o n c e n t r a t i o n r e q u i r e d . Complexity o f a c t i v i t y i s m o d e r a t e l y h i g h due t o u n c e r t a i n t y , u n p r e d i c t a b i l i t y , or u n f a m i l i a r i t y . C o n s i d e r a b l e a t t e n t i o n required.

3.

E x t e n s i v e mental e f f o r t and c o n c e n t r a t i o n a r e necessary. complex a c t i v i t y r e q u i r i n g t o t a l a t t e n t i o n .

I I.

Very

P s y c h o l o g i c a l S t r e s s Load

1.

L i t t l e c o n f u s i o n , r i s k , f r u s t r a t i o n , or a n x i e t y e x i s t s and can be e a s i l y accommodated.

2.

Moderate s t r e s s due t o c o n f u s i o n , f r u s t r a t i o n , or a n x i e t y n o t i c e a b l y adds t o workload. S i g n i f i c a n t compensation i s r e q u i r e t o m a i n t a i n adequate performance.

3.

High t o very i n t e n s e s t r e s s due t o c o n f u s i o n , f r u s t r a t i o n , o r a n x i e t y . High t o extreme d e t e r m i n a t i o n and s e l f - c o n t r o l r e q u i r e d

r e p r e s e n t e d by t h a t s e t o f d e s c r i p t o r s . The o r d e r o f t h e combinations t h a t r e s u l t s f r o m t h i s c a r d s o r t procedure i s t h e n used as t h e i n p r t d a t a f o r t h e c o n j o i n t measurement a n a l y s i s . Given t h e rank o r d e r t h a t t h e s u b j e c t s have d e r i v e d f o r t h e combinations o f t h e l e v e l s o f t h e t h r e e dimensional c o n s t r u c t , t h e a l g o r i t h m i s used t o search f o r a s e t o f a d d i t i v e s c a l e values t h a t d e s c r i b e s t h e order o f t h e l e v e l s o f t h e t h r e e composite dimensions. T h i s a n a l y s i s can be performed on each s u b j e c t ' s o r d e r i n g or on a consensus o r d e r i n g o b t a i n e d by a v e r a g i n g a group o f subjects' orderings. The advantages o f u s i n g an average o r d e r f o r i n p r t w i 11 be d i s c u s s e d 1a t e r . The adequacy o f t h e o b t a i n e d s o r t s has been evaluated, and c o n t i n u e s t o b e e v a l u a t e d , by a n a l y z i n g t h e number o f axiom v i o l a t i o n s t h a t a r e present i n a s e t o f data. T e c h n i c a l l y , an a x i o m a t i c a n a l y s i s i s d e t e r m i n i s t i c so t h a t one axiom v i o l a t i o n i s s u f f i c i e n t t o i n v a l i d a t e t h e model b e i n g

G.B. Reid and T.E. Nygren

200

F i g u r e 1.

Three-Dimensional Workload C o n s t r u c t

tested. T h i s c r i t e r i o n i s very i m p r a c t i c a l because people do n o t g i v e e r r o r f r e e d a t a very o f t e n . Because o f t h i s d i f f i c u l t y , work on an e r r o r t h e o r y f o r c o n j o i n t measurement i s i n progress (Nygren, 1985, 1986). I n t h e meantime, " r u l e s o f thumb" have been e s t a b l i s h e d based on e x t e n s i v e e x p e r i e n c e w i t h s e t s o f c a r d s o r t d a t a (Reid, P o t t e r , & B r e s s l e r , 1987). B a s i c a l l y , t h e r u l e s a l l o w f o r u p t o a p p r o x i m a t e l y a 5 percent t o 10 perc e n t v i o l a t i o n r a t e f o r t h e independence axioms as l o n g as t h e s e i n c o n s i s tencies i n v o l v e adjacent o r near-adjacent pairs. A n a l y z i n g Card S o r t Data The f i r s t s t e p i n a n a l y z i n g c a r d s o r t d a t a i s t o determine t h e l e v e l of agreement among a p a r t i c u l a r group o f s u b j e c t s . A Kendall ' s C o e f f i c i e n t I f t h e W-value i s s u f f i o f Concordance (W) i s used f o r t h i s p r p o s e . c i e n t l y l a r g e (maximum value f o r p e r f e c t i n t e r s u b j e c t agreement i s l.O), t h e s u b j e c t s a r e placed i n t o a s i n g l e group a n a l y s i s where a l l o f t h e i r d a t a a r e averaged. A " r u l e o f thumb" t h a t has been e s t a b l i s h e d i s t h a t i f t h e W i s .75 o r h i g h e r , t h e r e i s s u f f i c i e n t agreement t o make a s i n g l e scale t h a t w i l l represent a l l of t h e subjects without i n c u r r i n g a l a r g e An e x c e p t i o n t o t h i s pracchance o f m i s r e p r e s e n t i n g any s i n g l e s u b j e c t . t i c e would be f o r a s i t u a t i o n where t h e focus o f t h e i n v e s t i g a t i o n pert a i n s t o an i n d i v i d u a l d i f f e r e n c e s v a r i a b l e . T h i s t y p e o f a n a l y s i s would probably be b e s t accomplished u s i n g s c a l e s f o r each i n d i v i d u a l s u b j e c t . I n t h e event t h a t t h e o v e r a l l K e n d a l l ' s C o e f f i c i e n t o f Concordance i s l o w e r t h a n t h e proposed c u t o f f , a procedure c a l l e d SWAT p r o t o t y p i n g (Reid, Eggemeier, & Nygren, 1982) has been developed t h a t i n c o r p o r a t e s t h e advant a g e s o f an average s c a l e w h i l e a d j u s t i n g t o t h e i n d i v i d u a l s ' w e i g h t s f o r t h e composite dimension. D u r i n g t h e c a r d s o r t procedure, t h e comparisons t h a t each s u b j e c t must make between each o f t h e c e l l s o f t h i s m a t r i x , i n many cases, a r e very f i n e d i s c r i m i n a t i o n s . The degree o f v a r i a b i l i t y found f o r comparisons o f p a r t i c u l a r p a i r s o f c e l l s o r i n c o n s i s t e n c i e s f o u n d f o r s i m i l a r s t i m u l u s p a i r s can b e viewed as random o r u n s y s t e m a t i c error. I f t h e s u b j e c t s agree as t o t h e b a s i c s t r u c t u r e o f t h e c o n s t r u c t , t h e n a process o f a v e r a g i n g t h e i r i n d i v i d u a l o r d e r i n g s w i l l r e s u l t i n an o r d e r t h a t tends t o cancel o u t t h e s e random e r r o r s . W h i l e i t i s t r u e t h a t t h i s process w i l l a l s o mask some amount o f v a r i a t i o n t h a t i s a r e s u l t o f an i n d i v i d u a l I s unique c o n t r i b u t i o n , t h i s t h r e a t i s m i n i m i z e d by d e t e r m i n i n g t h e e x t e n t o f agreement among i n d i v i d u a l s and d i v i d i n g t h e s u b j e c t s i n t o homogeneous subgroups, i f a p p r o p r i a t e .

The Subjective Workload Assessment Technique

20 1

I f some s u b j e c t s ' o r d e r i n g s a r e based on a model o f w o r k l o a d t h a t p l a c e s t h e g r e a t e s t w e i g h t on t i m e r e l a t e d f a c t o r s and p l a c e s a m o d e r a t e w e i g h t o n f a c t o r s r e l a t e d t o m e n t a l e f f o r t , and v e r y l i t t l e w e i g h t on f a c t o r s t h a t r e l a t e t o psychological stress, w h i l e another s u b j e c t ' s ordering i s b a s e d on r e l a t i v e w e i g h t i n g s t h a t a r e i n t h e r e v e r s e o r d e r , t h e n t h e l e v e l o f agreement i n t h e t w o o r d e r i n g s w i l l b e low. F o r t h e p l r p o s e o f SWAT p r o t o t y p i n g , s i x h y p o t h e t i c a l o r d e r i n g s have been d e v e l o p e d w h i c h a r e based on a s t r i c t c o m p l i a n c e t o a r u l e d e f i n i n g t h e r e l a t i v e i m p o r t a n c e f o r each o f t h e t h r e e d i m e n s i o n s . The f i r s t p r o t o t y p e o r d e r i n g i s b a s e d on a r e l a t i v e w e i g h t i n g scheme w h i c h p l a c e s t h e g r e a t e s t emphasis o n t i m e , t h e second on e f f o r t , and t h i r d on p s y c h o l o g i c a l s t r e s s . If a subject o r d e r e d a c a r d deck a c c o r d i n g t o t h i s TES w e i g h t i n g scheme, t h e n t h e o r d e r i n g o f t h e combinations would be l i k e t h e o r d e r represented i n T a b l e 3 where, as can be seen, t h e l e v e l s o f s t r e s s change f a s t e s t , w h i l e t h e l e v e l s o f m e n t a l e f f o r t i n c r e a s e more s l o w l y , and t h e l e v e l o f t i m e increases a t t h e slowest rate. I n t h e same manner, an o r d e r i n g can b e e s t a b l i s h e d f o r t h e o t h e r r e l a t i v e w e i g h t i n g s TSE, ETS, EST, STE, and SET. R a t e r s ' r a n k o r d e r i n g s a r e c o r r e l a t e d w i t h each o f t h e s e p r o t o t y p e o r d e r i n g s u s i n g a Spearman's Rho t o d e t e r m i n e t h e r e l a t i v e i m p o r t a n c e each s u b j e c t p l a c e s on each o f t h e t h r e e d i m e n s i o n s . U s u a l l y , p r o t o t y p i n g

TABLE 3.

~

I

Rank Order 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Card Label

N

a

W F

J

C X S M U G Z V

Q

zz

K E R H P D Y A 0 L T

I

TES WEIGHTING SCHEME Descri ptor Colnbi n a t i on tff o r t St ress

rime 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3

1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3 1 1 1 2 2 2 3 3 3

1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3

Scale Va 1ues 0.0 24.4 51.4 7.6 32.0 59.0 27.7 52.1 79.1 6.5 30.9 57.9 14.1 38.5 65.5 34.2 58.6 85.6 20.9 45.2 72.3 28.5 52.9 79.9 48.6 73.0 100.0

202

G.B. Reid and T.E. Nygren

a l l o w s a group o f s u b j e c t s t o be d i v i d e d i n t o two or t h r e e homogeneous subgroups. Once i t has been determined how many groups a r e needed t o r e f l e c t t h e w e i g h t i n g s f o r a p a r t i c u l a r group o f s u b j e c t s , t h e n t h e c o n j o i n t a n a l y s i s i s performed. A s e p a r a t e a n a l y s i s i s r e q u i r e d f o r each subgroup found i n the prototyping analysis. Since t h e procedure i s t h e same f o r each group, f o r t h e sake o f t h i s i l l u s t r a t i o n , we w i l l assume t h a t t h e s u b j e c t s have s u f f i c i e n t l y h i g h agreement t o preclude p r o t o t y p i n g . The o u t p u t from t h e c o n j o i n t a n a l y s i s t h e n p r o v i d e s a scale, r a n g i n g f r o m z e r o t o 100, t h a t l i s t s a s c a l e value f o r each o f t h e c e l l s o f t h e t h r e e dimensional m a t r i x i n F i g u r e 1 r e p r e s e n t i n g combinations o f l e v e l s o f Time Load, E f f o r t Load, and S t r e s s Load. This s c a l e can t h e n be used t o a s s i g n s c a l e values t o new s i t u a t i o n s v i a t h e p o r t i o n o f t h e SWAT procedure t h a t i s c a l l e d event scoring. S t a b i l i t y o f S u b j e c t s ' Judgments The f i r s t two m a j o r q u e s t i o n s t h a t had t o be answered i n t h e developnent o f SWAT were whether ( a ) s u b j e c t s c o u l d perform t h e 27 c e l l c a r d s o r t , and ( b ) i f t h e y could, whether t h e c a r d s o r t s change f r o m day t o day as a f u n c t i o n o f t h e i n d i v i d u a l s ' c u r r e n t experiences or whether t h e y would be s t a b l e w i t h i n i n d i v i d u a l s across time. Although t h e o r d e r i n g procedure can be r a t h e r u n e x c i t i n g and d i f f i c u l t f o r some i n d i v i d u a l s , i t has been shown t o be an e f f e c t i v e and r e l i a b l e way t o o b t a i n t h e needed judgments. The f i r s t assessment o f t h e s t a b i l i t y o f s u b j e c t s ' workload o r d e r i n g s was performed w i t h 30 A i r F o r c e p i l o t s who were p a r t i c i p a t i n g i n a s t u d y o f a i r - t o - a i r combat i n a h i g h f i d e l i t y s i m u l a t o r . T h i s s t u d y was run u s i n g d i f f e r e n t s i m u l a t o r c o n f i g u r a t i o n s o v e r s e v e r a l months. S u b j e c t s performed t h e c a r d s o r t p r i o r t o t h e b e g i n n i n g o f t h e s t u d y and were rechecked f o u r months l a t e r a t t h e b e g i n n i n g o f phase two o f t h e study. The recheck was conducted u s i n g n i n e p a i r e d comparisons. The p a i r s were formed by p l a c i n g two o f t h e c a r d combinations on a s i n g l e sheet o f paper. The p a i r s were s e l e c t e d s o t h a t t h e y r e p r e s e n t e d t h e f u l l range o f c a r d combinations. Based on t h e o r i g i n a l c a r d s o r t , p r e d i c t i o n s o f t h e s e l e c t i o n s were made f o r each s u b j e c t . These p r e d i c t i o n s were found t o b e c o r r e c t 80 percent o f t h e time. Comparable r e s u l t s were found i n two o t h e r s t u d i e s . One o f t h e s e i n v e s t i In this g a t i o n s was done u s i n g p a i d s t u d e n t s f r o m a l o c a l u n i v e r s i t y . s t u d y , s i x s u b j e c t s performed t h e c a r d s o r t on two d i f f e r e n t occasions separated by a year. The o r d e r i n g s f r o m t h e s e two a d m i n i s t r a t i o n s were c o r r e l a t e d u s i n g a Spearman's Rho c o e f f i c i e n t , and f i v e o f t h e s i x o r d e r i n g s were found t o c o r r e l a t e .90 or g r e a t e r , w i t h t h e o t h e r s u b j e c t ' s c o r r e l a t i o n b e i n g .53. A t h i r d check was performed on 22 m i l i t a r y subj e c t s p a r t i c i p a t i n g i n an i n v e s t i g a t i o n o f c o n t r o l room designs. This recheck was performed a f t e r two months and a g a i n r e s u l t e d i n very h i g h correlations. Twenty-one o u t o f 22 s u b j e c t s had o r d e r i n g s c o r r e l a t i n g .90 or g r e a t e r . The o t h e r s u b j e c t had a d d i t i v e o r d e r i n g s each t i m e , b u t t h e y i n d i c a t e d a s h i f t i n t h e r e l a t i v e w e i g h t i n g s o f t h e t h r e e dimensions. The c o r r e l a t i o n s f o r t h e s u b j e c t s i n t h e s e s t u d i e s show, i n g e n e r a l , remarkab l e c o n s i s t e n c y f o r most s u b j e c t s over extended p e r i o d s o f time. The o c c a s i o n a l i n c o n s i s t e n t s u b j e c t c r e a t e s some cause f o r concern s i n c e an e x p l a n a t i o n f o r t h e i n c o n s i s t e n c y has n o t been e s t a b l i s h e d . However, i t remains very e n c o u r a g i n g f o r t h e SWAT procedure t h a t when t h e s e low

The Subjective Workload Assessment Technique c o r r e l a t i o n s a r e sometimes found f o r s u b j e c t s , i t i s a s h i f t i n t h e r e l a t i v e importance o f t h e f a c t o r s i n and n o t due t o e x c e s s i v e e r r o r o r n o n a d d i t i v i t y i n a d d i t i v e model h o l d s w e l l f o r s u b j e c t s , even i f t h e t h e f a c t o r s may s h i f t s l i g h t l y o v e r time.

203

almost always due t o d e t e r m i n i n g workload t h e rank data. The r e l a t i v e weights o f

EVENT SCORING D e s p i t e t h e importance o f t h e s c a l e developnent phase, i t i s t h e event s c o r i n g phase t h a t people g e n e r a l l y a r e t h i n k i n g about when t h e y r e f e r t o a s c a l i n g procedure. Event s c o r i n g s i m p l y r e f e r s t o t h e experiment o r c o n d i t i o n t h a t an i n v e s t i g a t o r wants t o e v a l u a t e r e g a r d i n g mental workload. I n t h e experiment, o p e r a t o r s a r e asked t o p r o v i d e judgements t h a t can be c o n v e r t e d i n t o s c a l e values r e p r e s e n t i n g t h e degree o f mental workl o a d a s s o c i a t e d w i t h t a s k performance. F o r example, i f SWAT was used t o e v a l u a t e t h e mental workload a s s o c i a t e d w i t h u s i n g two a l t e r n a t i v e designs o f a power p l a n t c o n t r o l panel, s u b j e c t s m i g h t be r e q u i r e d t o p e r f o r m a s i m u l a t e d s c e n a r i o u s i n g each o f t h e panel c o n f i g u r a t i o n s . The s c e n a r i o would be segmented i n t o component t a s k s and as t h e s u b j e c t s performed t h e t a s k s t h e y would be asked t o apply t h e p r e v i o u s l y l e a r n e d d e s c r i p t o r s t o e v a l u a t e each t a s k w i t h r e g a r d t o t h e l e v e l o f Time Load, Mental E f f o r t Load, and P s y c h o l o g i c a l S t r e s s Load (e.g., 2, 3, 2). D u r i n g d a t a a n a l y s i s t h e i n v e s t i g a t o r would c o n v e r t t h e s e values i n t o an o v e r a l l workload s c o r e (e.g., 64.4) by f i n d i n g t h e s c a l e value a s s o c i a t e d w i t h t h e c o m b i n a t i o n d u r i n g t h e s c a l e developnent phase. These scores would t h e n be used as t h e dependent v a r i a b l e i n an a n a l y s i s o f t h e d i f f e r e n c e i n workload a s s o c i a t e d w i t h t a s k s performance as a f u n c t i o n o f t h e t y p e o f d i s p l a y c o n f i g u r a t i o n used. When we decided t o d e v e l o p a new s u b j e c t i v e workload measure, t h e p r i m a r y m o t i v a t i o n was t h a t , a l t h o u g h a number o f reasonable measures had been proposed, none o f them had been e x t e n s i v e l y t e s t e d and evaluated. Theref o r e , we have attempted t o c o l l e c t a d a t a base t h a t w i l l p r o v i d e i n v e s t i g a t o r s w i t h i n f o r m a t i o n r e l a t i v e t o SWAT'S u t i l i t y as a measure o f o p e r a t o r workload. T h i s d a t a base has been c o l l e c t e d b o t h i n l a b o r a t o r y and f i e l d s i t u a t i o n s . The l a b o r a t o r y r e s e a r c h has been c e n t e r e d on t h e m a n i p u l a t i o n o f t a s k v a r i a b l e s and temporal v a r i a b l e s t o see i f SWAT i s s e n s i t i v e t o changes i n demand l e v e l s . I n a d d i t i o n , SWAT has been used i n several operational tests. The o p e r a t i o n a l t e s t s have been c a r r i e d o u t t o more d i r e c t l y e v a l u a t e t h e u t i l i t y o f SWAT i n t h e environment t h a t i s most relevant t o i t s intended application. Although t h e s e t e s t s were u s u a l l y designed w i t h o t h e r o b j e c t i v e s i n mind, i n many cases t h e d a t a can be looked a t i n terms o f an e v a l u a t i o n o f t h e degree o f success a s s o c i a t e d w i t h t h e SWAT a p p l i c a t i o n . L a b o r a t o r y i n v e s t i g a t i o n s o f SWAT have c e n t e r e d around t h e use o f a n assessment b a t t e r y named t h e C r i t e r i o n Task Set (CTS). The CTS ( S h i n g l e d e c k e r , C r a b t r e e , & Acton, 1982; Shingledecker, 1984; Eggemeier, 1987) i s a b a t t e r y o f t a s k s t h a t has been developed t o p r o v i d e a s t a n d a r d i z e d s e t o f t e s t s and procedures t o be used i n t h e e v a l u a t i o n o f workload measures. The b a t t e r y , as c u r r e n t l y c o n f i g u r e d , i s composed o f n i n e t a s k s t h a t have been s e l e c t e d t o be s e n s i t i v e t o d i f f e r e n t components o f t h e human i n f o r m a t i o n processing system and t o be c l e a r l y r e p r e s e n t a t i v e of o p e r a t i o n a l A i r F o r c e tasks. The model t h a t guided s e l e c t i o n o f t h e n i n e t a s k s i s based p r i m a r i l y on a m u l t i p l e resources model o f i n f o r m a t i o n proc e s s i n g (Navon & Gopher, 1979; Wickens, 1980). The t a s k s were s e l e c t e d so

G.B. Reid and T.E: Nygren

204

t h a t each one p r i m a r i l y places a demand on one o f t h e proposed p r o c e s s i n g resources. Table 4 l i s t s t h e t a s k s t h a t a r e c u r r e n t l y i n t h e b a t t e r y and t h e associated processing function. TABLE 4.

CTS TASKS AND ASSOCIATED PROCESSING FUNCTIONS*

Task

Processing F u n c t i o n

V i s u a l D i s play Moni t o r i ng Continuous R e c o g n i t i o n Memory Search L i n g u i s t i c Processing Mathematical P r o c e s s i n g Spatial Processing Grammatical Reasoning Unstable Tracking I n t e r v a l Production

Visual Perceptual Input Working Memory Encodi ng/Storage Working Memory S t o r a g e / R e t r i e v a l Symbolic I n f o r m a t i o n Mani p l a t i o n Symbolic I n f o r m a t i o n Mani p l a t i o n Spatial Information Manipulation Reasoning Manual Response Speed/Accuracy Manual Res ponse T i m i ng

*From Eggemeier, 1987. P a r a m e t r i c r e s e a r c h has been conducted (Shingledecker, 1984; Eggemeier & Amell, 1986; Amell, Eggemeier, & Acton, 1987) t o e s t a b l i s h t h r e e d i s t i n c t i v e l y d i f f e r e n t l e v e l s f o r each task. A d d i t i o n a l l y , t h i s r e s e a r c h was used t o e s t a b l i s h t h e s u b j e c t t r a i n i n g requirements and o t h e r r e l e v a n t aspects o f t h e s t a n d a r d i z e d t e s t - a d m i n i s t r a t i o n procedures. An advantage o f u s i n g t h i s b a t t e r y as t h e c e n t r a l element o f SWAT e v a l u a t i o n i s t h a t t h e t a s k s were s e l e c t e d s y s t e m a t i c a l l y s o t h a t , when t h e t h e e n t i r e b a t t e r y i s used, d i f f e r e n t demand c h a r a c t e r i s t i c s a r e obtained. A second m a j o r advantage i s t h a t , because t h e b a t t e r y has a s t a n d a r d a d m i n i s t r a t i o n format, o t h e r i n v e s t i g a t o r s can e a s i l y r e p l i c a t e t h e r e s e a r c h o r can e v a l u a t e o t h e r mental w o r k l o a d measures under comparable c o n d i t i o n s . Several independent s t u d i e s have used t h e CTS t o i n v e s t i g a t e SWAT'S m e t r i c p r o p e r t i e s , and d a t a f r o m some o f them w i l l be presented here. One s t u d y i n p a r t i c u l a r used 104 s u b j e c t s t o perform a l l e i g h t o f t h e CTS t a s k s t h a t a r e composed o f m u l t i p l e l e v e l s , t h e I n t e r v a l P r o d u c t i o n Task i s n o t composed o f l e v e l s ( S c h l e g e l & G i l l i l a n d , 1987). The procedure t h a t was f o l l o w e d had t h e s u b j e c t s r e p o r t f o r a one-hour s e s s i o n f o r f o u r consecut i v e days. A f t e r a weekend, t h e y r e t u r n e d f o r one f i n a l p r a c t i c e day A l l subjects p a r t i c i p a t e d f o l l o w e d by f o u r c o n s e c u t i v e days o f t e s t i n g . i n one two-hour s e s s i o n each t e s t i n g day. The f i r s t and t h i r d t e s t i n g days were r e p l i c a t i o n s t h a t f o l l o w e d t h e same t e s t i n g procedure used d u r i n g t h e f o u r t r a i n i n g days. The day between t h e two d a t a runs was a s t r e s s o r day where t h e t h e s u b j e c t s performed t h e CTS t a s k s under one o f several s t r e s s o r conditions. T r i a l s on each day were t h r e e minutes l o n g w i t h t h e l e v e l s o f each t a s k b e i n g presented i n ascending order. A fixed sequence o f t h e n i n e CTS t a s k s was used f o r a l l d a t a runs and was as f o l lows: (1) Memory Search, ( 2 ) I n t e r v a l P r o d u c t i o n , ( 3 ) Continuous R e c a l l , ( 4 ) L i n g u i s t i c Processing, ( 5 ) P r o b a b i l i t y M o n i t o r i n g (renamed D i s p l a y M o n i t o r i n g ) , ( 6 ) Grammatical Reasoning, ( 7 ) Mathematical Processing, ( 8 ) U n s t a b l e Tracking, and ( 9 ) S p a t i a l Processing. F i g u r e 2 presents p l o t s o f SWAT r a t i n g s across t h e two r e p l i c a t i o n s o f t h e three d i f f i c u l t y levels. A n a l y s i s o f v a r i a n c e on t h e d a t a i n d i c a t e d t h a t t h e main e f f e c t f o r l e v e l s was s t a t i s t i c a l l y s i g n i f i c a n t ( p < .05) f o r a l l

The Subjective WorkloadAssessment Technique

UNSTABLE TRACKING

VISUAL DISPLAY MONITORING

205

SPATIAL PROCESSING

:1-

LINGUISTIC PROCESSING

MATHEMATICAL PROCESSING

J

I

::i

GRAMMATICAL REASONINQ

MEMORY SEARCH

.

CONTINUOUS RECOGNITION

F

*.~

Figure 2.

SYAT R a t i n g P l o t s

e i g h t t a s k s ( S c h l e g e l & G i l l i l a n d , 1987). The r e p l i c a t i o n on t e s t day t h r e e p r o v i d e d a check f o r s t a b i l i t y ( o f t e n c a l l e d r e l i a b i l i t y ) o f t h e SWAT measure. The b a s i c a l l y f l a t shape o f a l l o f t h e curves i n F i g u r e 2 demonstrates t h a t g e n e r a l l y SWAT i s a v e r y s t a b l e measure. An a n a l y s i s o f v a r i a n c e i n d i c a t e d t h a t seven o f t h e t a s k s were n o t s t a t i s t i c a l l y d i f f e r e n t on day t h r e e from day one ( p > .05). The one e x c e p t i o n was s p a t i a l p r o c e s s i n g which had s i g n i f i c a n t l y l o w e r SWAT r a t i n g s on day t h r e e , p o s s i b l y r e f l e c t i n g t h a t l e a r n i n g was s t i l l t a k i n g place.

206

C.B. Reid and T.E. Nygren

A n o t h e r s t u d y t h a t was based o n t h e u s e o f a C T S t a s k c o n s i s t e d o f a d u a l t a s k e x p e r i m e n t i n w h i c h t h e u n s t a b l e t r a c k i n g t a s k was used t o r e p r e s e n t a f l y i n g t a s k , and a r a d i o c o m m u n i c a t i o n s t a s k was u s e d as a s e c o n d a r y t a s k ( R e i d , S h i n g l e d e c k e r , & Eggemeier, 1981). Results indicated that b o t h t h e s e c o n d a r y t a s k and t h e SWAT r a t i n g s d i f f e r e n t i a t e d among t h e three d i f f i c u l t y levels o f the unstable tracking task. This concurrent r e s u l t was i n t e r p r e t e d as p r o v i d i n g s u p p o r t i v e e v i d e n c e t h a t t h e measures were r e f l e c t i n g a c t u a l d i f f e r e n c e s i n s u b j e c t s ' m e n t a l w o r k l o a d . Some s t u d i e s h a v e been u n d e r t a k e n t o i n v e s t i g a t e f a c t o r s r e l a t e d t o p r o c e d u r a l a s p e c t s o f SWAT a d m i n i s t r a t i o n . One such s e r i e s o f e x p e r i m e n t s was d i r e c t e d t o w a r d i n v e s t i g a t i o n o f t h e d e l a y between t a s k p e r f o r m a n c e and t h e t i m e when s u b j e c t s p r o v i d e a c t u a l w o r k l o a d r a t i n g s . Frequently, i n operational situations, t h e portion o f a task t h a t i s o f greatest i n t e r e s t t o an i n v e s t i g a t o r o c c u r s a t p r e c i s e l y t h e same t i m e t h a t an o p e r a t o r c a n n o t b e i n t e r r u p t e d t o c o l l e c t a w o r k l o a d r a t i n g . The i n v e s t i g a t o r , t h e n , must w a i t f o r a b r e a k i n t h e o p e r a t o r s ' a c t i v i t i e s i n o r d e r t o o b t a i n a r a t i n g f o r t h a t event. I f t h e d e l a y between t a s k p e r f o r m a n c e and a s s i g n m e n t o f t h e w o r k l o a d r a t i n g s i s v e r y l o n g , i t w o u l d seem r e a s o n a b l e t o e x p e c t an e f f e c t o f t h e d e l a y on t h e a s s i g n e d r a t i n g s . N o t e s t i n e (1984) i n v e s t i g a t e d a delay i n r e p o r t i n g o f workload r a t i n g s o f u p t o 30 m i n u t e s . I n t h i s e x p e r i m e n t , s u b j e c t s p e r f o r m e d a d i s p l a y monit o r i n g t a s k and s u p p l i e d SWAT r a t i n g s i m m e d i a t e l y a f t e r t a s k c o m p l e t i o n or a f t e r e i t h e r a 15- or 3 0 - m i n u t e d e l a y . A n a l y s i s o f t h e SWAT r a t i n g s d a t a revealed t h a t t h e r a t i n g s f o r t h e three l e v e l s o f t h e d i s p l a y monitoring t a s k w e r e s i g n i f i c a n t l y d i f f e r e n t f r o m one a n o t h e r ( p < .05), b u t t h a t t h e y were n o t s t a t i s t i c a l l y d i f f e r e n t as a f u n c t i o n o f t h e r e s p o n s e d e l a y . Eggemeier, C r a b t r e e , and L a P o i n t e ( 1 9 8 3 ) u s e d t h e same e x p e r i m e n t a l d e s i g n i n an e x p e r i m e n t i n w h i c h s u b j e c t s p e r f o r m e d a s h o r t - t e r m memory t a s k . Because t h e N o t e s t i n e (1984) e x p e r i m e n t h a d u s e d a p e r c e p t u a l t a s k , i t was t h o u g h t t h a t a n e f f e c t m i g h t b e o b s e r v e d i f t h e t a s k demand t a p p e d t h e same p r o c e s s i n g r e s o u r c e t h a t c o u l d b e e x p e c t e d t o a f f e c t r e t e n t i o n o f t h e rating. The s h o r t - t e r m memory t a s k r e q u i r e d s u b j e c t s t o k e e p t r a c k o f t h e number o f o c c u r r e n c e s o f f o u r l e t t e r s i n a sequence o f l e t t e r s . Task d i f f i c u l t y was m a n i p l l a t e d by h a v i n g t h e l e t t e r s appear a t 1-second, 2-second, or 3-second i n t e r v a l s . A g a i n , t h e d a t a i n d i c a t e d a s i g n i f i c a n t main e f f e c t f o r t a s k l e v e l s , b u t t h e d e l a y c o n d i t i o n s d i d n o t d i f f e r s i g n i f i c a n t l y f r o m t h e immediate r a t i n g c o n d i t i o n . I n b o t h t h e Eggemeier e t a l . (1983) and N o t e s t i n e ( 1 9 8 4 ) s t u d i e s , t h e i n t e r v e n i n g t a s k was a t a s k t h a t was d i s s i m i l a r f r o m t h e t a s k t o b e rated. T h e r e f o r e , Eggemeier, Me1 v i l l e , and C r a b t r e e ( 1 9 8 4 ) c o n d u c t e d a t h i r d e x p e r i m e n t i n w h i c h t h e p r i m a r y v a r i a b l e was t h e t y p e o f i n t e r v e n i n g task. One c o n d i t i o n was a n o i n t e r v e n i n g t a s k c o n d i t i o n , a second was a d i f f i c u l t i n t e r v e n i n g t a s k c o n d i t i o n , a t h i r d was an easy i n t e r v e n i n g t a s k c o n d i t i o n , and a f o u r t h was a m i x e d d i f f i c u l t y i n t e r v e n i n g t a s k c o n d i tion. I n t h i s e x p e r i m e n t o n l y one d e l a y c o n d i t i o n , 1 4 m i n u t e s , was used, and t h e t a s k was a v a r i a t i o n o f t h e s h o r t - t e r m memory t a s k used i n t h e previous study. I n t h i s v a r i a t i o n , s u b j e c t s k e p t t r a c k o f f i v e l e t t e r s w i t h t h e l e t t e r s i n t h e t a s k sequence p r e s e n t e d a t a s i n g l e r a t e f o r 500 m i l l i s e c o n d s each. A g a i n , SWAT was f o u n d t o d i s c r i m i n a t e between t h e l e v e l s of t h e memory t a s k b u t n o s i g n i f i c a n t d i f f e r e n c e s w e r e f o u n d as a f u n c t i o n o f t h e t y p e o f i n t e r v e n i n g task.

The Subjective Workload Assessment Technique

201

Taken as a whole, t h e n , t h i s s e r i e s o f e x p e r i m e n t s p r o v i d e s s u p p o r t f o r t h e s e n s i t i v i t y of SWAT t o m a n i p u l a t i o n s o f t a s k i n d u c e d w o r k l o a d . The d a t a a l s o s u p p o r t t h e c o n c l u s i o n t h a t i n o p e r a t i o n a l s i t u a t i o n s , where task c o n s t r a i n t s r e q u i r e t h a t t h e i n v e s t i g a t o r delay o b t a i n i n g workload r a t i n g s , t h e d e l a y may n o t h a v e a l a r g e i m p a c t on t h e s u b j e c t s ' r a t i n g s . On t h e o t h e r hand, because of p o t e n t i a l l o s s f r o m s h o r t - t e r m memory, i t w o u l d b e p r u d e n t t o o b t a i n r a t i n g s as soon as p o s s i b l e a f t e r c o m p l e t i o n o f a r e l e v a n t event. One a t t r i b u t e t h a t i s g e n e r a l l y t h o u g h t t o be d e s i r a b l e f o r a w o r k l o a d measure i s i t s d i a g n o s t i c i t y . D i a g n o s t i c i t y i s t h e a b i l i t y o f a measure t o r e f l e c t t h e cause o r causes t h a t u n d e r l i e an i n c r e a s e i n m e n t a l w o r k l o a d . Some have a r g u e d t h a t s u b j e c t i v e measures a r e u s u a l l y t h o u g h t t o b e poor i n t h e i r a b i l i t y t o p r o v i d e d i a g n o s t i c i n f o r m a t i o n ( c f . , the d i s c u s s i o n i n Gopher & Donchin, 1986). We a r g u e t h a t t h e m u l t i d i m e n s i o n a l c h a r a c t e r i s t i c o f SWAT p r o v i d e s an o p p o r t u n i t y t o i m p r o v e i t s d i a g n o s t i c c a p a b i l i t y . Because r a t i n g s a r e o b t a i n e d on t h e i n d i v i d u a l d i m e n s i o n s and t h e s c a l i n g a l g o r i t h m p r o v i d e s s c a l e v a l u e s f o r t h e component s c a l e s as w e l l as t h e i r o v e r a l l a d d i t i v e e f f e c t , s e p a r a t e i n d i v i d u a l a n a l y s e s can b e p e r f o r m e d t o a s c e r t a i n w h i c h d i m e n s i o n i s c h a n g i n g t h e most as t a s k demand increases. P o t t e r and A c t o n (1985) p e r f o r m e d a s u b s c a l e a n a l y s i s i n a s t u d y u s i n g t h e c o n t i n u o u s r e c a l l t a s k f r o m t h e CTS. They showed i n t h i s e x p e r i m e n t t h a t , a l t h o u g h a l l t h r e e o f t h e component s c a l e s were s e n s i t i v e t o t a s k demand, t h i s e f f e c t o c c u r r e d a t d i f f e r e n t l e v e l s o f demand. The M e n t a l E f f o r t Load s c a l e i n c r e a s e d s u b s t a n t i a l l y a t t h e l o w e s t l e v e l s o f t a s k m a n i p u l a t i o n and t h e n r e m a i n e d f a i r l y c o n s t a n t t h r o u g h t h e m i d d l e a n d h i g h e r demand mani p u l a t i o n s . Time Load and P s y c h o l o g i c a l S t r e s s Load s c a l e s , on t h e o t h e r hand, s t a r t e d o u t t o i n c r e a s e v e r y s l o w l y a t t h e l o w m a n i p u l a t i o n s and changed most i n r e s p o n s e t o t h e m o d e r a t e t o h i g h manipul a t i o n s o f t a s k demand. T h i s d i f f e r e n t i a l s e n s i t i v i t y was i n t e r p r e t e d as b e i n g s u p p o r t i v e o f appropriateness o f t h e i n d i v i d u a l dimensions s e l e c t e d f o r SWAT. A f o l l o w - u p e f f o r t ( P o t t e r , 1986) was c o n d u c t e d t o t r y t o independently manipulate t h e t i m e load dimension through a r a t e o f pres e n t a t i o n m a n i p u l a t i o n and t h e m e n t a l e f f o r t l o a d d i m e n s i o n t h r o u g h t a s k demand m a n i p u l a t i o n . Two t a s k s w e r e used: a memory s e a r c h t a s k , and t h e c o n t i n u o u s r e c a l l t a s k f r o m t h e CTS. Task d i f f i c u l t y f o r t h e memory s e a r c h t a s k was m a n i p u l a t e d by v a r y i n g t h e number o f i t e m s ( l e t t e r s ) t h a t a s u b j e c t h e l d i n memory. Task d i f f i c u l t y i n t h e c o n t i n u o u s r e c o g n i t i o n t a s k was m a n i p u l a t e d by v a r y i n g t h e number o f d i g i t s b e i n g h e l d i n memory as w e l l as how many back i n a c o n t i n u o u s s t r e a m o f numbers t h e s u b j e c t h a d t o remember. F i g u r e 3 shows t h e r e s u l t f o r t h e memory s e a r c h t a s k . As c a n be seen f r o m t h i s f i g u r e , t h e t i m e l o a d d i m e n s i o n does seem t o v e r y c l e a r l y r e f l e c t t h e manipulation o f rate. Although t h e e f f o r t dimension does c l e a r l y r e f l e c t t h e t a s k d i f f i c u l t y m a n i p u l a t i o n , t h e r e i s a l s o , a p p a r e n t l y , some e f f e c t o f t i m e p r e s e n t i n t h e s c o r e s . Also, t h e r a t i n g s on t h e psychological s t r e s s dimension i n c r e a s e even though no attempted m a n i p u l a t i o n o f t h i s d i m e n s i o n was u n d e r t a k e n . Since i t i s impossible t o know w h e t h e r or n o t p s y c h o l o g i c a l s t r e s s was b e i n g a f f e c t e d i n some way o r whether increased d i f f i c u l t y a c t u a l l y r e q u i r e s a d d i t i o n a l processing time, i t c a n o n l y be o b s e r v e d t h a t t h e p a t t e r n p r e s e n t i n t h e s e c u r v e s s u p p o r t s , t o some d e g r e e , t h e d i f f e r e n t i a l s e n s i t i v i t y o f t h e d i m e n s i o n s . Adding weight t o t h i s i n t e r p r e t a t i o n i s t h e f a c t t h a t t h e t y p e o f scale (time, e f f o r t , s t r e s s ) i n t e r a c t e d w i t h t h e experimental manipulations o f r a t e of presentation, d i f f i c u l t y l e v e l , and r a t e by d i f f i c u l t y .

C.B. Reid and T. E. Nygren

208 TIME LOAD

PSYCHOLOGICAL STRESS LOAD

MENTAL EFFORT LOAD

TASK DIFFICULTY (MEMORY SET SIZE) - 1

7Y

- 4 -7

6-

3 A

s

i a 3

Q

=

54-

3211 1 I I I SLOW MEDIUM FAST SLOW MEDIUM FAST SLOW MEDIUM FAST 1

I

1

PRESENTATION RATE

F i g u r e 3.

R e s u l t s f o r Memory Search Task

SWAT has a l s o been used i n two experiments t o e v a l u a t e t h e e f f e c t s of s t r e s s o r s on work1 oad. The f i r s t experiment (A1 bery, Ward, & G i 11 , 1985) was designed t o e v a l u a t e t h e p o s s i b i l i t y t h a t a high-G environment l i k e t h a t found i n a modern f i g h t e r a i r c r a f t c o u l d c o n t r i b u t e t o o p e r a t o r ment a l workload. Impairment c o u l d occur as a r e s u l t o f reduced b l o o d f l o w o r from t h e conscious e f f o r t s t h a t must be expended i n c o u n t e r a c t i n g t h e blood f l o w effects. To i n v e s t i g a t e t h i s phenomenon, s u b j e c t s were r e q u i r e d t o s o l v e a two-dimensional maze problem on a CRT w h i l e b e i n g exposed t o G f o r c e s o f 1.5, 3.0, 5.0, and 6.0 Gs i n a human c e n t r i f u g e . The r e s u l t s o f t h i s s t u d y i n d i c a t e d t h a t scores a s s o c i a t e d w i t h maze performance were n o t a f f e c t e d by t h e G l e v e l s . However, SWAT r a t i n g s f o r moderate and high-G l e v e l s (5.0 and 6.0 Gs) were s i g n i f i c a n t l y h i g h e r t h a n SWAT r a t i n g s f o r t h e l o w e r G l e v e l s (1.5 and 3.0 Gs). Another study a l s o i n v e s t i g a t e d a p o t e n t i a l e n v i r o n m e n t a l s t r e s s o r ( A l b e r y , Repperger, Reid, Goodyear, Ramirez, & Roe, 1987). I n t h i s study l o w (90 dB) t o moderate (100 dB) n o i s e was used t o p r o v i d e a s t r e s s o r w h i l e t h e s u b j e c t performed a s i n g l e a x i s compensatory t r a c k i n g t a s k t h a t represented a f l y i n g task. Task d i f f i c u l t y was m a n i p u l a t e d by p r e s e n t i n g f i v e d i f f e r e n t f o r c i n g f u n c t i o n s f o r each o f t h r e e t r a c k i n g p l a n t dynami c s . B o t h performance measures and SWAT r a t i n g s e f f e c t i v e l y d i s c r i m i n a t e d between t h e t h r e e d i f f e r e n t p l a n t dynamics. The n o i s e s t r e s s o r d i d n o t have a measurable e f f e c t on t h e s u b j e c t s ' performance b u t t h e l e v e l s o f n o i s e d i d produce a s t a t i s t i c a l l y s i g n i f i c a n t e f f e c t on t h e SWAT r a t i n g s . Simul a t i on S t u d i e s As a r u l e , s i m u l a t i o n s t u d i e s do n o t have t h e degree o f e x p e r i m e n t a l c o n t r o l t h a t c h a r a c t e r i z e s l a b o r a t o r y experiments. Because t h e t a s k s a r e s o much more complex, t h e same degree o f p r e c i s i o n u s u a l l y cannot be achieved. On t h e o t h e r hand, t h e degree o f r e a l i s m p o s s i b l e i n a

The Subjective Workload Assessment Technique

209

s i m u l a t i o n study provides an environment t h a t can be used t o v e r i f y g e n e r a l i z a t i o n s o f l a b o r a t o r y r e s u l t s t o more " r e a l w o r l d " s i t u a t i o n s . SWAT h a s been used i n t w o k i n d s o f s i m u l a t i o n s . The f i r s t g r o u p o f s t u d i e s was d e s i g n e d and e x e c u t e d f o r t h e p l r p o s e o f e v a l u a t i n g m e n t a l w o r k l o a d measures, and t h e second g r o u p c o n s i s t e d o f f i e l d - t y p e e v a l u a t i o n s o f o p e r a t i o n a l systems. I n t h e second g r o u p o f s t u d i e s , SWAT was u s e d as a dependent v a r i a b l e f o r c o n s i d e r i n g m e n t a l w o r k l o a d i n t h e system e v a l u a t i o n . The f i r s t g r o u p o f s i m u l a t i o n s was c o n d u c t e d p r i m a r i l y t o i n v e s t i g a t e a number o f p h y k i o l o g i c a l w o r k l o a d measures. SWAT was i n c l u d e d i n t h e i n v e s t i g a t i o n s because t h e c h a r a c t e r i s t i c s o f ease o f a p p l i c a t i o n and l a c k o f i n s t r u m e n t a t i o n made i t a s i m p l e and i n e x p e n s i v e a d d i t i o n t o t h e study. The f i r s t e x p e r i m e n t i n t h i s s e t used B-52 p i l o t i n g t a s k s f o r c r e a t i n g d i f f e r e n t w o r k l o a d c o n d i t i o n s i n a 8-52 s i m u l a t o r ( T h i e s s e n , Lay, & S t e r n , 1987). The s c e n a r i o was w r i t t e n t o have t h r e e l e v e l s o f w o r k l o a d r e p r e s e n t e d . The l e v e l s were d e f i n e d as f o l l o w s : Low

S t r a i g h t and L e v e l F l i g h t

Medium

Normal Descent and ILS Approach

H i gh

D e s c e n t t o I L S Approach w i t h S u c c e s s i v e E n g i n e F a i l u r e s , Runaway T r i m , and C r o s s w i n d s

The s c e n a r i o was p r e s e n t e d i n t w o s i m u l a t i o n r u n s o f a p p r o x i m a t e l y 15 m i n u t e s each. L i n e p i l o t s f r o m a 8-52 s q u a d r o n s e r v e d as s u b j e c t s a n d f l e w t h e s c e n a r i o s i n a C u r t i s W r i g h t DEHMEL f l i g h t s i m u l a t o r a t C a r s w e l l A i r F o r c e Base, Texas. The t o p l e f t panel o f F i g u r e 4 shows t h a t t h e SWAT r a t i n g s c l e a r l y d i f f e r e n t i a t e d between t h e t h r e e w o r k l o a d c o n d i t i o n s . The SWAT s c o r e s were a m o n o t o n i c f u n c t i o n o f t h e a p r i o r i d e f i n e d l e v e l s o f t a s k demand. The second s t u d y e v a l u a t e d t h e w o r k l o a d o f t a s k s p e r f o r m e d by a B-52 t a i l gunner ( T h i e s s e n e t al., 1987). I n t h i s study, t h e s i m u l a t i o n scenario was w r i t t e n t o c r e a t e t h r e e w o r k l o a d l e v e l s f o r t h i s p r e d o m i n a n t l y percept u a l m o t o r and c o m m u n i c a t i o n t a s k . The l e v e l s o f w o r k l o a d d e f i n e d by t h e s c e n a r i o were: Low

H o s t i l e t a r g e t e n c o u n t e r s a t h i g h a l t i t u d e , enemy t e r r i tory, automatic t a r g e t a c q u i s i t i o n ;

Medium

H o s t i l e t a r g e t encounters low-level, acquisition;

High

H o s t i l e t a r g e t encounters a t low-level w i t h radar system ma1 f u n c t i o n s .

manual t a r g e t

The s c e n a r i o s were i m p l e m e n t e d on a gunner s t a t i o n t r a i n e r c o n s i s t i n g o f a r a d a r scope, i n d i c a t o r s , s w i t c h e s , and an i n s t r u c t o r s t a t i o n . The 13 s u b j e c t s were drawn f r o m o p e r a t i o n a l c r e w s and each s u b j e c t " f l e w " a one h o u r m i s s i o n w i t h a p p r o x i m a t e l y 30 m i n u t e s o f t h e t i m e d e d i c a t e d t o t h e a c t u a l t a r g e t p r e s e n t a t i o n segments d e f i n e d by t h e w o r k l o a d l e v e l s . The b o t t o m panel o f F i g u r e 4 i l l u s t r a t e s t h a t SWAT was s e n s i t i v e t o t h i s w o r k l o a d manipulation. SWAT r a t i n g s f o r each o f t h e t h r e e l e v e l s w e r e s i g n i f i c a n t l y d i f f e r e n t f r o m one a n o t h e r . A t h i r d s i m u l a t i o n s t u d y r e p o r t e d by

G. B. Reid arid T.E. Nygreri

210

100

IW

90 -

90

-

-

80

-

10

-

60

~.

5

0

80 70

w

~

3

60

-

>

50

-

2

40-

a

t

30

20

P

w

3

Q>

,,," /'

c a

/

uI

,A

~

-

0-

40

,d'

~

, '

~

, / '

30-

20 10

d"

O T

,,Y

_/'

U

~

-

0 7

HLDLOW

HED H ~ G H

Him

MISSION TYPE

Figure 4.

SWAT Results

Thiessen e t a l . (1987) i n v e s t i g a t e d a f i g h t e r a i r defense mission. In t h i s study, 13 s u b j e c t s f l e w an F-16 s i m u l a t o r w i t h a 36" x 48" wide a n g l e f i e l d o f view v i s u a l s i m u l a t i o n . The c o c k p i t was a f i x e d base F-160 w i t h m a j o r c o n t r o l s and d i s p l a y s f u n c t i o n a l . The s i m u l a t i o n i n c l u d e d f o u r d e f e n s i v e c o u n t e r a i r s c e n a r i o s designed t o p r o v i d e workload r a n g i n g f r o m l o w t o h i g h d e f i n e d as f o l l o w s : Low

An F-16 chases t h r e e enemy a i r c r a f t making an "S" weave escape.

Medium Low

F i v e enemy a i r c r a f t approach t h e F-16 head-on.

Medium High

An 'IS" One enemy f i g h t e r approaches t h e F-16 head-on. weave p a t t e r n ; two enemy f i g h t e r s approach head-on; f o u r enemy bombers approach t h e F-16 head-on b e h i n d t h e fighters.

H i gh

Seven enemy a i r c r a f t approach t h e F-16; two o f t h e a i r c r a f t s p l i t i n o p p o s i t e d i r e c t i o n s t o c a t c h t h e F-16 i n a p i n c h e r maneuver.

Posthoc t e s t s f o l l o w i n g a n a l y s i s o f v a r i a n c e on t h e SWAT r a t i n g s showed a s i g n i f i c a n t e f f e c t only for t h e h i g h workload condition. The means (on t h e 0 t o 100 SWAT s c a l e ) f o r t h e f o u r l e v e l s were 30, 41, 42, and 72. Although t h e SWAT scores t e n d t o go u p as a monotonic f u n c t i o n o f t h e

The Subjective Workload Assessment Technique

21 1

w o r k l o a d mani p u l a t i o n , t h e p o s t h o c t e s t s r e v e a l e d t h a t t h e m a g n i t u d e o f t h e d i f f e r e n c e s between l e v e l s was t o o s m a l l t o b e s t a t i s t i c a l l y r e l i a ble. T h i s f i n d i n g was s u b s t a n t i a t e d by f i v e p h y s i o l o g i c a l measures and a p e r f o r m a n c e measure. The l a c k o f s i g n i f i c a n t d i f f e r e n c e s i n a l l o f t h e s e measures must b e i n t e r p r e t e d as i n d i c a t i n g t h a t t h e i n t e n d e d m a n i p u l a t i o n was n o t as s t r o n g as t h e i n v e s t i g a t o r s h a d i n t e n d e d . However, even i n t h i s s i t u a t i o n , SWAT proved t o be as s e n s i t i v e as any o f t h e dependent v a r i a b l e s used. As a s e t , t h e s e s i m u l a t i o n s t u d i e s p r o v i d e s u b s t a n t i a l s u p p o r t f o r t h e s e n s i t i v i t y o f SWAT. T h i s was an i m p o r t a n t s e t o f e x p e r i m e n t s s i n c e i t i n v o l v e d t h e d i f f i c u l t amalgamation of l a b o r a t o r y c o n t r o l w i t h t h e r e a l i s m of an o p e r a t i o n a l t a s k . The s i m i l a r i t y o f t h e s e d a t a t o t h e r e s u l t s o b t a i n e d i n t h e psychology l a b o r a t o r y p r o v i d e evidence t o support t h e a s s e r t i o n t h a t SWAT i s s e n s i t i v e t o v a r i a t i o n s i n w o r k l o a d a c r o s s a w i d e v a r i e t y o f t a s k s and o p e r a t i o n a l c o n d i t i o n s . Another important f u n c t i o n o f these s i m u l a t i o n s t u d i e s i s t o b r i d g e t h e g a p between t h e r e s e a r c h f i n d i n g s and o p e r a t i o n a l a p p l i c a t i o n s . As p r e v i o u s l y s t a t e d , t h e o b j e c t i v e o f t h e d e v e l o p n e n t o f SWAT was t o p r o v i d e a measurement t o o l f o r u s e i n o p e r a t i o n a l e n v i r o n m e n t s l i k e f l i g h t t e s t s a n d o p e r a t i o n a l t e s t and e v a l u a t i o n (OT&E). I n t h e s e a p p l i c a t i o n s , t h e o b j e c t i v e s of t h e t e s t s a r e n o t r e l a t e d t o t h e e v a l u a t i o n o f w o r k l o a d measures. I n t h e s e i n s t a n c e s , t h e SWAT measure i s needed as a dependent v a r i a b l e t o e v a l u a t e such f a c t o r s as a l t e r n a t i v e s y s t e m c o n f i g u r a t i o n s , p r o c e d u r e s , o r v a r i o u s crew f a c t o r s . T a b l e 5 p r o v i d e s a sample o f e v a l u a t i o n s where SWAT has been employed. These e v a l u a t i o n s t y p i c a l l y d e a l w i t h new o r e x p e r i m e n t a l systems and, t h e r e f o r e , even when t h e systems a r e c o m m e r c i a l r a t h e r than defense-related, t h e data are considered sensitive. Because o f t h e s e n s i t i v i t y o f most o f t h e s e a p p l i c a t i o n s , t h e r e s u l t s o f t h e t e s t c a n n o t R a t h e r , t h e b r e a d t h of t h e t y p e s o f e v a l u a t i o n s and t h e be d e t a i l e d here. number o f a p p l i c a t i o n s a r e p r e s e n t e d t o d e m o n s t r a t e t h e u t i l i t y o f SWAT a s a w o r k l o a d dependent v a r i a b l e i n many t y p e s o f o p e r a t i o n a l e v a l u a t i o n s .

SUMMARY AND CONCLUSIONS Because o f t h e c o m p l e x i t y o f t h e c o n s t r u c t known as m e n t a l w o r k l o a d , i t s measurement p r e s e n t s a f o r m i d a b l e c h a l l e n g e . I n f a c t , i t has been a r g u e d t h a t , because o f t h i s c o m p l e x i t y , no s i n g l e measure i s l i k e l y t o adeq u a t e l y encompass a l l components o f w o r k l o a d i n a l l a p p l i e d s i t u a t i o n s (Eggemeier, 1984). The a l t e r n a t i v e t o a s i n g l e i n d e x o f w o r k l o a d i s a b a t t e r y o f measures, each o f w h i c h i s s e n s i t i v e u n d e r d i f f e r e n t c o n d i t i o n s o f task t y p e o r subject type, o r i s s e l e c t i v e l y s e n s i t i v e t o p a r t i c u l a r components t h a t c o m p r i s e t h e c o n s t r u c t . J u s t such a b a t t e r y i s u n d e r d e v e l o p n e n t as p a r t o f t h e r e s e a r c h program a t t h e H a r r y G. A r m s t r o n g Aerospace M e d i c a l R e s e a r c h L a b o r a t o r y . The o b j e c t i v e o f t h i s r e s e a r c h program i s t o d e v e l o p a b a t t e r y o f m e t r i c s and s u b j e c t t h e m t o r i g o r o u s t e s t i n g i n o r d e r t o d e f i n e ( a ) t h e c o n d i t i o n s u n d e r w h i c h each measure i s u s e f u l as a measure o f m e n t a l w o r k l o a d , ( b ) t h e c h a r a c t e r i s t i c s o f t h e measures, and ( c ) t h e i n t e r r e l a t i o n s h i p s among t h e v a r i o u s measures. The S u b j e c t i v e W o r k l o a d Assessment T e c h n i q u e (SWAT) i s a component o f t h i s battery. M e t r i c s r e p r e s e n t i n g o t h e r c l a s s e s o f measures i n c l u d i n g p h y s i o l o g i c a l measures a n d b e h a v i o r a l ( p e r f o r m a n c e ) measures a r e a l s o u n d e r d e v e l o p n e n t ( c f . , t h e c h a p t e r s by W i l s o n and O ' D o n n e l l , and by Eggemeier i n t h i s volume).

212

G.B. Reid and T.E. Nygren TABLE 5.

Category

SWAT APPLICATIONS STUDIES

System

Sinul ation Aircraft

F-16/F-15 A i r - t o - A i r KC-135 F l i g h t Deck M o d e r n i z a t i o n A-300 Approach and Landing ( S c h i c k & Hahn, 1987) 8-52 Long M i s s i o n ( S k e l l y & P u r v i s , 1985) DC-10 Approach and Landing ( B i f e r n o & Reid, 1983) 8-52 CG/Fuel Level A d v i s o r y System H e l i c o p t e r NOE (Haworth, Bivens, S h i l v e y , & Delgado, 1987) General A v i a t i o n T r a i n i n g ( H a s k e l l & Reid, 1985)

Control Center

Ground Launch M i s s i l e ( C r a b t r e e , Bateman, & Acton, 1984; Acton & Crabtree, 1985) Nuclear Power P l a n t T r a i n i n g (Beare & D o r r i s , 1984)

O i l R e f i n e r y ( B e v i l l e E n g i n e e r i n g , Inc.,

1986)

Operational Aircraft

F-16 F1 ight T e s t * A-10 F1 i g h t T e s t * Laser Guided M i s s i l e F l i g h t Test* (Ossard, A m a l b e r t i & Poyot, 1987)

Control Center

C-1412 A i r Drop/Ai r Land** KC-10 Boom Operator**

(Dodge, Wong, & Brown, 1984)

Command and C o n t r o l Center** 1984)

* **

( C o u r t r i ght & Kuperman,

F l i g h t Test O p e r a t i o n a l T e s t & E v a l u a t i o n (OT&E)

SWAT i s a s c a l i n g procedure t h a t r s designed t o a l l o w t h e m e a n i n g f u l assignment o f numbers t o i n d i v i d u a l s s u b j e c t i v e i m p r e s s i o n s o f t h e mental workload a s s o c i a t e d w i t h p e r f o r m i n g v a r i o u s t a s k s . As a s u b j e c t i v e measu r e we see t h e t e c h n i q u e as h a v i n g t h e f o l l o w i n g d i s t i n c t advantages: (1) i t i s based on f o r m a l p r o p e r t i e s o f c o n j o i n t measurement t h e o r y , ( 2 ) t h e u n d e r l y i n g assumption o f a d d i t i v i t y o f t h e t h r e e w o r k l o a d dimensions i s t e s t a b l e f o r b o t h i n d i v i d u a l and group d a t a , ( 3 ) o n l y o r d i n a l ( r a n k o r d e r )

The Subjective Workload Assessment Technique

213

d a t a i s r e q u i r e d , ( 4 ) t h e rank o r d e r i n g t a s k o f t h e 27 workload combinat i o n s has f a c e v a l i d i t y , ( 5 ) t h e SWAT s c a l i n g a l g o r i t h m s i m u l t a n e o u s l y produces i n t e r v a l - s c a l e d e s t i m a t e s o f t h e l e v e l s o f t h e t h r e e workload dimensions as w e l l as e s t i m a t e s o f t h e i r combined e f f e c t s , ( 6 ) i n d i v i d u a l d i f f e r e n c e s e s t i m a t e s o f t h e importance o f each dimension f o r e v a l u a t i n g workload can be o b t a i n e d , t h u s a l l o w i n g s u b j e c t s t o be prototyped, and ( 7 ) once t h e s c a l e has been o b t a i n e d , v a r i o u s t a s k s o r subtasks can be e a s i l y scored v i a a n o n i n t r u s i v e procedure we c a l l event s c o r i n g .

SWAT i s i n t e n d e d t o be a g l o b a l measure o f workload t h a t i s a p p l i c a b l e i n a l a r g e range o f s i t u a t i o n s . That i s , i t i s expected t h a t SWAT s h o u l d be g e n e r a l l y s e n s i t i v e t o i n c r e a s e s i n workload and t h e r e f o r e be h e l p f u l i n i d e n t i f y i n g areas o f concern w i t h i n a t a s k o r system d e s i g n t h a t r e q u i r e more i n t e n s e i n v e s t i g a t i o n . Because SWAT r a t i n g s a r e r e l a t i v e l y s i m p l e t o o b t a i n , SWAT can be used as t h e measure t h a t p r o v i d e s c o n t i n u i t y t h r o u g h o u t a system d e s i g n o r f a m i l y o f s t u d i e s . An e a r l y s t u d y m i g h t i d e n t i f y a p a r t i c u l a r phase o f a system o p e r a t i o n as b e i n g r e l a t i v e l y h i g h i n workload. That phase m i g h t t h e n be more t h o r o u g h l y s t u d i e d u s i n g one o r more o t h e r workload measurement t e c h n i q u e s such as one o f t h e p h y s i o l o g i c a l measures o r a secondary t a s k . I n most i n s t a n c e s d u r i n g t h i s more focused study, SWAT c o u l d a l s o be o b t a i n e d t o p r o v i d e supplemental i n f o r m a t i o n t h a t can c o n f i r m t h a t t h e s u b j e c t i v e i m p r e s s i o n has n o t been a l t e r e d by t h e change i n study c o n d i t i o n s w h i l e g e t t i n g t h e i n c r e a s e d focused s e n s i t i v i t y and d i a g n o s t i c i t y o f t h e a d d i t i o n a l measures. A l i m i t a t i o n o f s u b j e c t i v e measures o f workload i s t h a t t h e y p r o v i d e r e l a t i v e information. Under c u r r e n t c o n d i t i o n s , we a r e sometimes r e s t r i c t e d i n s a y i n g o n l y t h a t one t a s k has more o r l e s s workload t h a n another. Research i s needed t o c l e a r l y d e f i n e t h e degree o f i n f l u e n c e o f f a c t o r s such as t h e number and range o f t a s k l e v e l s present i n an i n v e s t i g a t i o n o r t h e o r d e r e f f e c t s o f t h e v a r i o u s c o n d i t i o n s i n an i n v e s t i g a t i o n . Once t h i s aspect o f t h e measurement process i s understood t h e n i t may be possib l e t o e s t a b l i s h what has been l a b e l e d a " r e d l i n e " f o r mental workload. The t e r m " r e d l i n e " was chosen t o i m p l y t h a t , i f an o p e r a t o r ' s w o r k l o a d exceeds a c e r t a i n value, t h e p r o b a b i l i t y o f performance breakdown i s i n c r e a s e d , r a t h e r t h a n t o i m p l y t h a t t h e r e i s a value t h a t w i l l d e f i n i t e l y r e s u l t i n performance breakdown. I n most i n s t a n c e s when i n v e s t i g a t o r s a r e concerned about measuring workload, t h e i m p l i e d q u e s t i o n i s , "Is t h e workl o a d t o o h i g h ? " I n o r d e r t o answer t h i s q u e s t i o n , i t w i l l be necessary t o d e v e l o p m e t r i c s t h a t a r e a b s o l u t e measures o f a known range o f a charact e r i s t i c o f operators. U n t i l t h i s goal can be achieved, i t i s d e s i r a b l e t o b u i l d a d a t a base o f measurement values t h a t a r e a s s o c i a t e d w i t h i n c r e a s e d e r r o r r a t e s and/or performance breakdown. Considerable data r e l a t i n g each workload measure t o performance d e g r a d a t i o n i s needed t o e s t a b l i s h t h e l o c a t i o n o f t h e t o l e r a n c e l e v e l s i n d i c a t e d by t h e v a r i o u s measures. F i n a l l y , i n o r d e r t o make t h e complementary use o f m u l t i p l e measures p o s s i b l e , e x t e n s i v e r e s e a r c h w i l l be necessary t o d e f i n e t h e r e l a t i o n s h i p between s u b j e c t i v e and t h e v a r i o u s o t h e r measures o f mental workload t h a t a r e presented i n t h i s volume. ACKNOYLEDGEHENTS T h i s work i n c l u d i n g most o f t h e developnent o f t h e SWAT s c a l i n g a l g o r i t h m was supported i n p a r t by a c o n t r a c t t h r o u g h t h e U. S. Air F o r c e and t h e second author. The a u t h o r s would l i k e t o thank Ms. J. B r e s s l e r f o r t h e s u p p o r t p r o v i d e d i n p r e p a r a t i o n o f t h i s m a n u s c r i p t . We would a l s o l i k e t o

G. B. Reid arid

214

T.E. Nygren

e x p r e s s a p p r e c i a t i o n t o s e v e r a l c o l l e a g u e s , Dr. H.A. P o l z e l l a , Mr. S.S. P o t t e r , and Dr. M.L. F r a c k e r who reviews o f t h e manuscript a t v a r i o u s stages.

C o l l e , D r . D.P. provided c r i t i c a l

REFERENCES

c11

A c t o n , W.H. and C r a b t r e e , M.S., W o r k l o a d assessment t e c h n i q u e s i n s y s t e m r e d e s i g n , P r o c e e d i n g s o f t h e IEEE N a t i o n a l Aerospace and E l e c t r o n i c s Conference (1985).

c21

A l b e r y , W.B., Ward, S.L., and G i l l , R.T., The e f f e c t o f a c c e l e r a t i o n s t r e s s on human w o r k l o a d , H a r r y G. A r m s t r o n g Aerospace M e d i c a l R e s e a r c h L a b o r a t o r y T e c h n i c a l R e p o r t , (AAMRL-TR-85-039), (1985).

C31

R e i d , G.B., Goodyear, C., Ramirez, A l b e r y , W.B., Repperger, D.W., L.E., and Roe, M.M., E f f e c t o f n o i s e o n a d u a l t a s k : S u b j e c t i v e and o b j e c t i v e workload c o r r e l a t e s , Proceedings o f t h e IEEE National Aerospace and E l e c t r o n i c s C o n f e r e n c e (1987).

C41

Eggemeier, F.T., and Acton, W.H., The c r i t e r i o n t a s k A m e l l , J.R., set: An u p d a t e d b a t t e r y , paper p r e p a r e d f o r p r e s e n t a t i o n a t t h e T h i r t y - F i r s t Annual M e e t i n g o f t h e Human F a c t o r s S o c i e t y (1987).

C5l

Beare, A.N. and D o r r i s , R.E., The e f f e c t s o f s u p e r v i s o r e x p e r i e n c e and t h e presence o f a s h i f t t e c h n i c a l a d v i s o r on t h e Derformance o f two-man c r e w s i n a n u c l e a r power p l a n t s i m u l a t o r , P r o c e e d i n g s o f t h e Human F a c t o r s S o c i e t y T w e n t y - E i g h t h Annual M e e t i n g T 9 8 4 ) 242-246.

C61

B e v i l l e Engineering, sol id a t i o n (1986).

171

DC-10 s t u d y - Does t h e d i s t a n c e o f a B i f e r n o , M. and R e i d , G.B., touch-panel c o n t r o l i n f l u e n c e o p e r a t o r performance and/or workload? ( u n p u b l i s h e d r e p o r t , 1983).

C81

C o u r t r i g h t , J.F. and Kuperman. G.. Use o f swat i n u s a f s v s t e m t & e. P r o c e e d i n g s o f t h e Human F a c t o r s S o c i e t y T w e n t y - E i g h t h A n n u a i ' M e e t i n g ( 1 9 8 4 ) 7 00-703.

C91

C r a b t r e e , M.S., Bateman, R.P.. and Acton. W.H.. Benefits o f usinq o b j e c t i v e and s u b j e c t i v e - w o r k l o a d measures-, P r o c e e d i n g s o f t h e Human F a c t o r s S o c i e t y Twenty-Ei g h t h Annual M e e t i n g (1984) 950-953.

ClOl

Dodge, D.C., Wong, T.J., and Brown, K.W., Boom c o n t r o l s y s t e m improvement s t u d y Phase I 1 - S u p p l e m e n t a l i n d i c a t i o n system. R e p o r t No. MDC 59732 ( D o u g l a s A i r c r a f t Company, McDonnell D o u g l a s , 1984).

Inc.,

Human f a c t o r s a n a l y s i s o f r e f i n e r y c o n -

-

An a p p l i c a t i o n o f d e c i s i o n - a n a l y t i c t e c h n i q u e s t o t h e t e s t and e v a l u a t i o n phase o f a m a j o r a i r s y s t e m : Phase 111. McLean, V i r g i n i a : D e c i s i o n s and D e s i g n s , Inc., TR-PR-79-6-91, 1979.

c111 D o n n e l l , M.L.,

C121

D o n n e l l , M.L. and D'Connor, M.F., The a p p l i c a t i o n o f d e c i s i o n a n a l y t i c t e c h n i q u e s t o t h e t e s t and e v a l u a t i o n phase o f t h e a c q u i s i t i o n o f a m a j o r a i r s y s t e m : Phase 11. McLean, V i r g i n i a : D e c i s i o n s and D e s i g n s , Inc., TR-78-3-25, 1978.

The Subjective Workload Assessment Technique

215

El31

Eggemeier, F.T., W o r k l o a d m e t r i c s f o r s y s t e m e v a l u a t i o n , P r o c e e d i n g s o f t h e D e f e n s e Research G r o u p Panel V I I I Workshop " A p p l i c a t i o n s o f System Ergonomics t o Weapon System Developnent," Shrivenham, E n g l a n d ( 1 9 8 4 ) C/5-C/20.

El41

Eggemeier, F.T., P r o p e r t i e s of w o r k l o a d assessment t e c h n i q u e s , i n : Hancock and N. M e s h k a t i ( e d s . ) , Human M e n t a l W o r k l o a d (Amsterdam, The N e t h e r l a n d s , E l s e v i e r , 1987).

El51

Eggemeier, F.T. and A m e l l , J.R., Visual p r o b a b i l i t y monitoring: E f f e c t s o f d i s p l a y l o a d and s i g n a l d i s c r i m i n a b i l i t y P r o c e e d i n s o f t h e Human F a c t o r s S o c i e t y T h i r t i e t h Annual M e e t i n g (i9-

[161

Eggemeier, F.T., M e l v i l l e , B.E., and C r a b t r e e , M.S., The e f f e c t o f i n t e r v e n i n g t a s k p e r f o r m a n c e on s u b j e c t i v e w o r k l o a d r a t i n g s , Proc e e d i n g s o f t h e Human F a c t o r s S o c i e t y T w e n t y - E i g h t h Annual M e e t i n g ( 1 9 8 4 ).

[17]

and L a P o i n t e , P.A., The e f f e c t o f Eggemeier, F.T., C r a b t r e e , M.S., d e l a y e d r e p o r t on s u b j e c t i v e r a t i n g s o f m e n t a l w o r k l o a d , P r o c e e d i n s o f t h e Human F a c t o r s S o c i e t y Twenty-Seventh Annual M e e t 4

P.

-

[I81

Gopher, D. and Donchin, E., W o r k l o a d -- An e x a m i n a t i o n o f t h e concept, i n : K. R. B o f f , L. Kaufman, and J. P. Thomas (eds.), Handbook o f p e r c e p t i o n and human performance, V o l 2: Cognitive P r o c e s s e s and P e r f o r m a n c e (New York, W i l e y I n t e r s c i e n c e , 1986).

[19]

H a s k e l l , B. and R e i d , G.B., An i n v e s t i g a t i o n o f t h e s u b j e c t i v e p e r c e p t i o n o f w o r k l o a d and p e r f o r m a n c e i n l o w - t i m e p r i v a t e p i l o t s , A v i a t i o n Space and E n v i r o n m e n t a l M e d i c i n e , ( i n p r e s s , 1985).

[20]

Haworth, L.A., B i v e n s , C.C., S h i v e l y , R.J., and Delgado, D., Advanced c o c k p i t and c o n t r o l c o n f i g u r a t i o n s f o r s i n g l e p i l o t h e l i c o p t e r - n a p - o f - t h e - e a r t h f l i g h t , Paper p r e s e n t e d a t t h e A m e r i c a n H e l i c o p t e r S o c i e t y F o r t y - T h i r d Annual Forum and T e c h n o l o g y D i s p l a y (1987).

[21]

H o l d e r , O., D i e axiome d e r Q u a n t i t a t und d i e L e h r e vom h i s s , B e r i c h t e uber d i e Verhandlungen d e r K o n i g l i c h Sachsischen Gesellens c h a f t d e r W i s s e n s c h a f t e n zu L e i p z i g , M a t h e m a t i s c h P h y s i s c h e C1 a s s e 53 ( 1 9 0 1 ) 1-64.

[22]

Johannsen, G., Moray, N., Pew, R., Rasmussen, J., Sanders, A., and Wickens, C., F i n a l r e p o r t o f t h e e x p e r i m e n t a l p s y c h o l o g y group, in: N. Moray (ed.), M e n t a l Workload: I t s Theory and Measurement (New York, Plenum P r e s s , 1979).

[23]

Johnson, R.M., Pairwise nonmetric P s y c h o m e t r i k a 38 ( 1 9 7 3 ) 11-18.

[24]

K a n t o w i t z , B.H., Channels and s t a q e s i n human i n f o r m a t i o n p r o c e s s i n g : - A l i m i t e d a n a l y s i s o f t h e o r y and m e t h o d o l o g y , J o u r n a i o f M a t h e m a t i c a l P s y c h o l o c y 2 9 ( 1 9 8 5 ) 135-174.

multidimensional

scaling,

G.B. Reid and T.E. Nygren Krantz. D.H., C o n j o i n t measurement: The Luce-Tukey axiomat z a t i o n and some e x t e n s i o n s , J o u r n a l o f Mathematical Psychology 1 (1964) 248-277. Krantz. D.H., Luce, R.D., Suppes, P., and Tversky, A., o f Measurement, Vol. 1, (New York: Academic Press 1971).

Faun

ations

and Tversky, K r a n t z , D.H. - . A.. . C o n j o i n t measurement a n a l y s i s o f comp o s i t i o n r u l e s i n psychology, - P s y c h o l o g i c a l Review- 78 (1971) 151-169. A n a l y s i s o f f a c t o r i a l experiments by e s t i m a t i n q K r u s k a l , J.B., monotone t r a n s f o r m a t i o n s o f t h e data, J o u r n a l o f t h e Royal S t a t i s t i : c a l S o c i e t y , S e r i e s B, 2 7 (1965) 251-263. and Tukey, J.W., Simultaneous c o n j o i n t measurement: A Luce, R.D. new t y p e o f fundamental -measurement, J o u r n a l o f Mathematical Psychology 1 (1964) 1-27. Moray, N. Workload: 979).

(ed.), Models and measures o f mental workload, i n : Mental I t s Theory and Measurement (New York, Plenum Press,

Navon, D. and Gopher, D., On t h e e c o n o w o f t h e human p r o c e s s i n g system, P s y c h o l o g i c a l Review 86 (1979) 214-255. Nickerson, C.A. and McClelland, G.B., S c a l i n g d i s t o r t i o n i n numeric a l c o n j o i n t measurement, A p p l i e d P s y c h o l o g i c a l Measurement 8 (1984) 183-198. Norman, D. and Bobrow, D . , On d a t a l i m i t e d and r e s o u r c e l i m i t e d processing, Journal o f C o g n i t i v e Psychology 7 (1975) 44-60. N o t e s t i n e , J.C., S u b j e c t i v e workload assessment i n a p r o b a b i l i t y m o n i t o r i n g t a s k and t h e e f f e c t o f delayed r a t i n g s , Proceedin s o f t h e Human F a c t o r s S o c i e t y Twenty-Eighth Annual M e

-

Nygren, T.E., An e x a m i n a t i o n o f c o n d i t i o n a l v i o l a t i o n s o f axioms f o r a d d i t i v e c o n j o i n t measurement, A p p l i e d P s y c h o l o g i c a l Measurement 9 (1985) 249-264.

A two s t a g e a l g o r i t h m f o r a s s e s s i n g v i o l a t i o n s o f Nygren, T.E., additivity via axiomatic and numerical conjoint analysis, Psychometrika, 51 (1986) 483-491. O'Donnell, R.D. and Eggemeier, F.T., Workload assessment methodology, i n : K.R. B o f f , L. Kaufman, and J.P. Thomas (eds.) Handbook o f r c e p t i o n and human performance, Val. 2: C o g n i t i v e Processes and rformance (New York, W i l e y I n t e r s c i e n c e , 1986). Ossard, G., A m a l b e r t i , R., and Poyot, G., E v a l u a t i o n de l a charge de t r a v a i l du p i l o t e i n d u i t e par un systeme d'arme guide l a s e r , ( M i n i s t e r e de l a Defense: C e n t r e d'Etudes e t de Recherches de Medecine Aerospatiale, L a b o r a t o i r e d'Etudes M e d i c o p h y s i o l o g i q u e s 16/330, 1987).

The Subjective Workload Assessment Technique

217

[39]

Potter, S.S., S u b j e c t i v e workload assessment t e c h n i q u e (SWAT) s u b s c a l e s e n s i t i v i t y t o v a r i a t i o n s i n t a s k demand and p r e s e n t a t i o n r a t e , u n p u b l i s h e d masters t h e s i s , W r i g h t S t a t e U n i v e r s i t y , Dayton, Ohio (1986).

[40]

P o t t e r , S.S. and Acton, W.H., R e l a t i v e c o n t r i b u t i o n s o f SWAT dimens i o n s t o o v e r a l l s u b j e c t i v e workload r a t i n g s , Proceedings o f t h e T h i r d Symposium on A v i a t i o n Psychology, (Columbus, Ohio, Ohio S t a t e I n i v e r s i t y , 1985) 231-238.

[41]

Reid, G.B., Eggemeier, F.T., and Nygren, T.E., An i n d i v i d u a l d i f f e r ences approach t o SWAT s c a l e d e v e l o p e n t , Proceedings o f t h e Human F a c t o r s S o c i e t y Twenty-Sixth Annual M e e t i n g (1982) 639-642. Reid, G.B., P o t t e r , S.S., and B r e s s l e r , J.R., U s e r ' s guide f o r t h e s u b j e c t i v e workload assessment t e c h n i q u e (SWAT), H a r r y G. Armstrong Aerospace Medical Research L a b o r a t o r y Technical Report, (AAMRL-TR8 7 - i n process), W r i g h t - P a t t e r s o n A i r Force Base, Ohio (1987). Shingledecker, C.A., and Eggemeier, F.T., Application o f Reid, G.B., c o n j o i n t measurement t o workload s c a l e d e v e l o p e n t , Proceedin s o f t h e Human F a c t o r s S o c i e t y T w e n t y - F i f t h Annualj - 1 1

-

Reid, G.B., Shingledecker, C.A., Nygren, T.E., and Eggemeier, F.T., Developnent o f m u l t i d i m e n s i o n a l s u b j e c t i v e measures o f workload. Proceedings o f t h e I E E E I n t e r n a t i o n a l - Conference on C y b e r n e t i c s and S o c i e t y (1981) 403-406. and Hahn, R.L., The use o f s u b j e c t i v e workload assessS c h i c k , F.V. ment t e c h n i q u e i n a complex f l i g h t t a s k , A d v i s o r y Group f o r Aerospace Research and Developnent (AGARD), (AGARD-AG-282), (1987) 37-41. and G i l l i l a n d , K., Evaluation o f t h e c r i t e r i o n task Schlegel, R.E. set, (AAMRL-TR-87-in press), W r i g h t - P a t t e r s o n A i r Force Base, Ohio: A i r Force Aerospace Medical Research L a b o r a t o r y (1987).

A t a s k b a t t e r y f o r a p p l i e d human performance Shingledecker, C.A., assessment research, A i r Force Aerospace Medical Research L a b o r a t o r y T e c h n i c a l Report, AFAMRL-TR-84-071 (November 1984). Shingledecker, C.A., Crabtree, M.S., and Acton, W.H. , S t a n d a r d i z e d t e s t s f o r t h e e v a l u a t i o n and c l a s s i f i c a t i o n o f workload m e t r i c s , Proceedings o f t h e Human F a c t o r s S o c i e t y Twenty-Sixth Annual M e e t i n g (1982) 648-651. S i n g l e t o n , W.T., Fox, J.C. and W h i t f i e l d , D. (eds.), Man a t Work (London, T a y l o r and F r a n c i s , 1973).

Measurement o f

and P u r v i s , B.D., 8-52 w a r t i m e m i s s i o n s i m u l a t i o n : S k e l l y , J.J. S c i e n t i f i c p r e c i s i o n i n workload assessment, Proceedings o f t h e 1985 A i r F o r c e Conference on Technology i n T r a i n i n g and t d u c a t i o n ( T l l t ) , 71985) 105-109.

218

G. B. Reid and T.E. Nygren

[51]

T h i e s s e n , M.S., Lay, J.E., and S t e r n , J.A., Neuropsychological workl o a d t e s t b a t t e r y v a l i d a t i o n s t u d y , H a r r y G. A r m s t r o n g Aerospace Medical Research L a b o r a t o r y T e c h n i c a l Re p o r t , (AAMRL-TR-87-i n p r e s s ) , Wri g h t - P a t t e r s o n Air F o r c e Base, O h i o ( 1 9 8 7 ) .

[52]

A g e n e r a l t h e o r y o f p o l y n o m i a l c o n j o i n t measurement, T v e r s k y , A,, J o u r n a l o f M a t h e m a t i c a l P s y c h o l o g y 4 ( 1 9 6 7 ) 1-20.

1 5 3 1 Wickens, C.D., The s t r u c t u r e o f attentional R. N i c k e r s o n (ed.), A t t e n t i o n and P e r f o r m a n c e V I I I , J e r s e y , E r l b a u m P r e s s , 1980).

resources, (Hillsdale,

in: New

[541

Wickens, C.D., P r o c e s s i n g r e s o u r c e s i n a t t e n t i o n , i n R. Parasuraman and R. D a v i e s ( e d s . ) , V a r i e t i e s o f A t t e n t i o n , (New York, Academic P r e s s , 1984).

[55]

and W i e r w i l l e , W.W., B e h a v i o r a l measures o f a i r c r e w W i l l i g e s , R.C. m e n t a l w o r k l o a d , Human F a c t o r s 2 1 ( 1 9 7 9 ) 549-574.

[56]

W i l s o n , G.F. and O ' D o n n e l l , R.D., Measurement o f O p e r a t o r W o r k l o a d With t h e Neuropsychological Workload Test B a t t e r y , i n : P. Hancock and N. M e s h k a t i (eds.), Human M e n t a l W o r k l o a d (Amsterdam, The N e t h e r 1 ands, E l s e v i e r , 1987).