- Email: [email protected]
Control of Input Variables by Head Movements of Handicapped Persons
Copyright © IFAC :\nalysis, Design and E\'a luation of \ Ian - \ Iac:hinc SYStcms Badcn -Hadcll, Fede ra l Rcpub l'ir of CtTll1any 19~:!
CONTROL OF INPUT VARIABLES BY HEAD MOVEMENTS OF HANDICAPPED PERSONS A. Korn Fraun h ofer Institu t e for Iriform at ioll and Data Processing, KarlsTuh e, Federa l R ep ubh c of Germany
Abstract . For peop l e with only limited muscle functions the delegation of alpha-numeric or graphic information is sometimes very difficult. Considering se riously handicapped persons who cannot u se their arms or legs , head or eye movements can often be used for ·, .,ritten communi cation . For s uch persons we have developed a " Visually Coupled System (VCS) " which consists of a suitable frontlet and luminous sou rces (light dots) fastened to the head. A light dot is supe rpo sed on the visual field and serves as a mark clearly determining the line of sight . Another source of light is pictured on the target of a TV camera in order to measure the head position without making any contact . The task of the handicapped consists in s hi fting the opt i cal mark until it coincides with a preset environmental target. Because of the great accuracy of the eye -h ead coo r dination the measured preciSion for the fixation of a sma ll target or for tracing a simple optical pattern is about 20 minutes of arc . When using the VCS as an aid for writing letters th e performance for selecting a character out of a 5 xG matrix was 0 . 3- 0 .4 s per l etter . Keywords . Rehabilitation: communication: visually coupled system: eye -head coordination: head dynamics: writing aid .
INTRODUCTION
A fast and accurate coordination of eye and head movements is , simi l ar to the eye - h and coordination , a n atu r al physiological activity of man, closely linked to his per ception of a n d reaction to environmental stimuli . In visual search tasks eye and head movements generally will occur . Here th e process of visual acquisiti on can be divided in t o three motor outputs: saccadic eye movements , very fast head movements and compensato r y eye movements (Zangeme ister and others , 19 81) . Head sh ifting has , normally , a share of 80 to 90 % in total line-of-sight shifting.
Thank s to the rapid progress made in the development of electronic measuring systems , seriously handicapped persons are offered technical aids providing more advantages than the mechanical aids so far used. Su c h conventional aids such as suction and b l owing devices or mouth guided pens , very often call for an unnatural posture or head move ments and prevent any verba l commun ication. Assu~ing that the handicapped cannot write but h e is able to speak then the present possibilities of automatic speech recognition may be considered for the translation of the spoken words into writing. It is however difficult in this case to create graphic information such as drawings. In this case and for paralyzed patients who have lost their vo ic e and writing/typing ability , a communication dev i ce with transducer facili ties to detect the line of sight ha s been developed . This device is , essentia lly, a so- called " Visually Coupled System (VCS) " whi ch is a contro l system activated by head and/or eye movements , with a visua l feedback (Russo , 19 78: Korn, 19 80) . The interest in VCS is main ly gi ven by its many possible applications in the field of fl ight control. The VCS, developed in o ur insti tute , will subsequen tly be presented as an aid to the handi capped.
Pr ovided it will be possible to carry o ut coo rd inated head mov ements , a t ask of the handicapped , e . g . of those suffering from pa r aplegia , may consist in shifting an optical ma rk until it coincides with a preset environmental targ et . After this , the measured coo rdina tes of the head position can be assigned to the line o f sight. In the process , four different prob l ems have ~o be solved: How to measure the head position without incommoding the h andicapped by the measuring process . The v isua l feedback of the head position . The verification of the called - up in fo rmation.
28 1
282
A. Korn The starting of the p r ocess by the handicapped.
Frames of Reference These problems are individua lly solved as follows
METHODS Measurement of the line of sight The scheme of the experimental ar rangement is sho wn in Fig . 1. The head position is measured without making any contac t. A light-weight frontlet and a luminous sou rce Lo1 are fastened to the head . This so ur ce f light is picture d on the target o f a TV cam e ra. Any shift of the light dot caused by head movements is elect r onica lly measured by means of a Video - Analyzer whic h s ignals the x,y-co ordinates o f the light dot . Th e measurement of two degrees of freedom , namely shaking and nodding the head , is s u f fi c ient in most cases (Korn , 19 8 1 ) .
Working Space ( Laboratory)
Fig. 2 .
Li n e of Sight
Head Related Reference System
The two frames of ref e rence which must be considered f o r the d e t e r mination of the line of sight . In the wor king space a targe t F is watched by the subject whose h e ad ha s the point of r otation D . This po int is the orign of the second r eference system which is the head related ref erence system . Only r otatio n s of t h e head around the two axes z and y are measured .
TV Camera
Fig . 1.
Schematic view of the appa ratus to dete rmin e t h e line of sight by measuring head movements .
The visua l feedback of the head position is realized by supe rposing on t h e vis u a l field a light dot which i s the reflected be am of the light source L . This light source is 2 coupled with the head and its optics can be adapted in order to get a s harp im age in the plane of the target . Ou r investigations have been performed with the prototype shown in Fig . 2. Here the head had to be carry a l oad o f 160 grammes .
Fig. 3 .
A s imple Visually Cou pled System which is composed of two sources of light and a s i mp l e optics . The visual feedback i s reac h ed by a li g ht dot superposed on the visual field by means of the glass and t h e focusing adjustment pictured on the right hand side. Th e head has to ca rry a l oad of about 160 grammes.
The two frames of r eference wh ich must be considered f o r the interpretation of the measured data a re shown in Fig. 3 . On the left hand side the positions of the p o int of h ead rotation D, the eye A and the target F are described in the laboratory system or working space . On the right hand side all these points , together with the light dot , are related to a r eference system which is firmly coupl ed with the head. The output of the Video-Analyzer in Fig. 1 is a measure for the head r o tati o n a round two axes in this refer ence system.
Contro l o f a typewr iter by h ead movements In many cas es the visual feedback of the head position can b e performed by observing a cursor on a monitor which may be a h ome TV monitor. The c urs or is shifted by head movements and mu s t be made to coinc ide with a desired character on a foil which is f ix ed on the monitor screen . This procedure has the advan tage o f immediately marking the relevant target. If it s position r e la tive to the foil i s stored in a computer , the measur e d data can be used in order to
Contro l of Input Va riabl es by Head Movements control technical devices such as a type writer or a speech synthesizer as indicated in Fig . 4. Here the measured analog signals are fed inot a computer where this input is Lransformed into suitable control signals . We have investigated the control of a typewriter. For safety reasons, an acknowledge ment of the called - up information will frequently be necessary.This may be effected by optical or acoustic means . We have integrated a voice output unit (voice synthesizer) producing an acoustic acknowledgement by an appropriate access to the word - organized storage .
283
Fixation of a Grid a""
~
~
b
~
-<,.
It.
~
1)
"-
1:'
~
-'\
..,
I.
~
fV
· j2S ·
--<\- . . -- ~ -
-{
!:;
~
~
----i---......--
25'
---1 ~
Speech Synthesizer
x VideoAnalyzer
~
Y
Encoder Control Unit Storage
Head Rotation
i Visually Coupled System Fig. 4.
~
Typewriter
Electronic devices for analyzing a video signal and the generation of acoustic and opt ical pattern.
Considering that , for the release of a desired function , hand or foot movements are not feasible , we have coupled the starting pulse with the dwell time of the cursor in the different fields of our 5x6 input matrix. With a simulated typewriter a dwell time between 0.3 and 0 .4 s has proved to be suitable .
Fig. 5 .
•
.
Fixation accuracy when the subject has to shift an optical mark by head movements until it coincides with a point of the grid .
Wher eas with the registrations shown in Fig. 5 the writing process after each fixation is interrupted by lifting the writing aid at the plotter , the curves shown in Fig. 6 - 8 show a continuous registration of the head movements following the lines of the meandering pattern at bottom left in Fig. 6, the individual characters at the top of Fig . 7 , and the individual circuits of a printed board in Fig. 8 . Also with these continuous registrations the deviations from the should-be-value do not exceed a visual angle of 10 '.
RESULTS When using the VCS, the fixation precision was measured with the aid of a grid which was watched at a visual angle of 8° in a horizontal, and of 6° in a vertical direc tion. The angular changes of the head related reference system as compared with the laboratory system were measured with the aid of t h e experimental device shown in Fig . 1. Two analog voltages corresponded to the rotations of the head . A registration of these voltages with the aid of an x - y plotter is shown in Fig. 5 . Some elements of the pattern were fixed for approx. 1 s. The registration shows that the amplitudes of the fixation movements in both x and y directions are, i.a. , recorded at a visual angle of 10', this corresponding to space of less than 3 mm between elements at a distance of 1 m.
1
25'
AA Fig. 6.
Pattern copied
by Head Movements
Continuous registration of the head movements following the lines of the meandering pattern at bottom left.
284
A. Korn
are successively fi x ed by means of saccadic eye movements .
AHTE KDLN AHTE Fi g . 7 .
Co n t i nu ous r egistration of head moveme n ts f o llowin g the l ines of the indiv i d u a l c h aracters at the top .
Th e re l ease of the sta r ting pu l se for printi ng a c h aracter via the d we ll time when fi x ing the corresponding matri x element was found to be agreeab l e . The subjects so far Ilsed for experiments were members of the Institute .
l'
Pi.g . 9 .
Contr olling a typewriter by h eau mov e me nt s . The l etters of the words " GUTEN MORGEN " (good morning) are fixed successively by head movements with the aid of the Vis u ally Coupled System . A time of approx . 3 . 5 s wa s r eq uired for selecting 1 1 characters.
CONCLUSION
I /
Fi g . 8 .
Co n ti n uous r e g istration of h ead movemen t s f o ll o wing t he ind i vid u al c ir c u its o f a pr i n t ed boa r d at the t op .
Whe n co ntro lling a typ e wri t er b y h ead move ments , the c ha rac t e r s of the s imul ated k ey b o ard must b e fix ed s u ccessive ly wi t h t h e aid o f t he VCS . For the words " Guten Morgen " (good morning ) the r e gi s tr a t io n o f t h e co rres p o nd ing h ead moveme n ts is p r esented in Fig. 9. Fo r s el e cting 11 c h a r ac t ers , a tim e of appro x. 3.5 s wa s r equir e d , equ a l t o 0 . 3 s p e r c h a r ac t e r . A n egli gibl y s h o r te r tim e i s ne eded wh en t he s i ng l e c h a r acters
Our investigations with a prototype of a simple Visually Coupled Sy stem s how that a fast se l ection of alpha - numeric characters and a faithful reproduction of graphic pattern can be performed by head movement s . This prototype i s st ill uncomfortable in order to be us ed by seriously handicapped persons in daily life . Nev e rtheless we are sure that the optics , th e light sourc es , and the device for the mea surement of the two degrees of head rotati o n can b e r e duced to an acceptable s ize . If the visual feedback is achieved by a cursor , that is a light spot for example on a hom e TV monitor , than nothing but the rotation of the h ead around two axes must be measured which is also possible by simple non- optical methods l ike pote n tiometers or inertia sensors . Furt h er improvements can o n ly be reached by a c l ose cooperation wit h handicapped persons.
ACKNOWLEDGEMENTS Th is repo r t was supported by Th e German Federal Ministry of Defense
Control of Input Variables by Head Movements
REFERENCES Korn, A. (198 0) . Verfahren zum indirekten Messen der Blickrichtung. Patent No . 26 29073 of the German Patent Office. Korn, A. (1981) . Visual Search: Relation between detection performance and visual field size. Proc. of the 1s t European Annual Conference on Human Decision and Manual Control , Delft, May 1981 , 27 - 34. Russo , L. (1978) . Helmet mounted visually coupled systems . Proc . of the S.l.D . Vol.19/4, 181-18 5. Zangemeister , W.H., Lehmann, S ., Stark , L. (1981) . Sensitivity analysis and optimization f or a h ead movement model . BioI. Cybern . Vol . 41 , 33- 45.
285
CONTROL OF INPUT VARIABLES BY HEAD MOVEMENTS OF HANDICAPPED PERSONS A. Korn Fraun h ofer Institu t e for Iriform at ioll and Data Processing, KarlsTuh e, Federa l R ep ubh c of Germany
Abstract . For peop l e with only limited muscle functions the delegation of alpha-numeric or graphic information is sometimes very difficult. Considering se riously handicapped persons who cannot u se their arms or legs , head or eye movements can often be used for ·, .,ritten communi cation . For s uch persons we have developed a " Visually Coupled System (VCS) " which consists of a suitable frontlet and luminous sou rces (light dots) fastened to the head. A light dot is supe rpo sed on the visual field and serves as a mark clearly determining the line of sight . Another source of light is pictured on the target of a TV camera in order to measure the head position without making any contact . The task of the handicapped consists in s hi fting the opt i cal mark until it coincides with a preset environmental target. Because of the great accuracy of the eye -h ead coo r dination the measured preciSion for the fixation of a sma ll target or for tracing a simple optical pattern is about 20 minutes of arc . When using the VCS as an aid for writing letters th e performance for selecting a character out of a 5 xG matrix was 0 . 3- 0 .4 s per l etter . Keywords . Rehabilitation: communication: visually coupled system: eye -head coordination: head dynamics: writing aid .
INTRODUCTION
A fast and accurate coordination of eye and head movements is , simi l ar to the eye - h and coordination , a n atu r al physiological activity of man, closely linked to his per ception of a n d reaction to environmental stimuli . In visual search tasks eye and head movements generally will occur . Here th e process of visual acquisiti on can be divided in t o three motor outputs: saccadic eye movements , very fast head movements and compensato r y eye movements (Zangeme ister and others , 19 81) . Head sh ifting has , normally , a share of 80 to 90 % in total line-of-sight shifting.
Thank s to the rapid progress made in the development of electronic measuring systems , seriously handicapped persons are offered technical aids providing more advantages than the mechanical aids so far used. Su c h conventional aids such as suction and b l owing devices or mouth guided pens , very often call for an unnatural posture or head move ments and prevent any verba l commun ication. Assu~ing that the handicapped cannot write but h e is able to speak then the present possibilities of automatic speech recognition may be considered for the translation of the spoken words into writing. It is however difficult in this case to create graphic information such as drawings. In this case and for paralyzed patients who have lost their vo ic e and writing/typing ability , a communication dev i ce with transducer facili ties to detect the line of sight ha s been developed . This device is , essentia lly, a so- called " Visually Coupled System (VCS) " whi ch is a contro l system activated by head and/or eye movements , with a visua l feedback (Russo , 19 78: Korn, 19 80) . The interest in VCS is main ly gi ven by its many possible applications in the field of fl ight control. The VCS, developed in o ur insti tute , will subsequen tly be presented as an aid to the handi capped.
Pr ovided it will be possible to carry o ut coo rd inated head mov ements , a t ask of the handicapped , e . g . of those suffering from pa r aplegia , may consist in shifting an optical ma rk until it coincides with a preset environmental targ et . After this , the measured coo rdina tes of the head position can be assigned to the line o f sight. In the process , four different prob l ems have ~o be solved: How to measure the head position without incommoding the h andicapped by the measuring process . The v isua l feedback of the head position . The verification of the called - up in fo rmation.
28 1
282
A. Korn The starting of the p r ocess by the handicapped.
Frames of Reference These problems are individua lly solved as follows
METHODS Measurement of the line of sight The scheme of the experimental ar rangement is sho wn in Fig . 1. The head position is measured without making any contac t. A light-weight frontlet and a luminous sou rce Lo1 are fastened to the head . This so ur ce f light is picture d on the target o f a TV cam e ra. Any shift of the light dot caused by head movements is elect r onica lly measured by means of a Video - Analyzer whic h s ignals the x,y-co ordinates o f the light dot . Th e measurement of two degrees of freedom , namely shaking and nodding the head , is s u f fi c ient in most cases (Korn , 19 8 1 ) .
Working Space ( Laboratory)
Fig. 2 .
Li n e of Sight
Head Related Reference System
The two frames of ref e rence which must be considered f o r the d e t e r mination of the line of sight . In the wor king space a targe t F is watched by the subject whose h e ad ha s the point of r otation D . This po int is the orign of the second r eference system which is the head related ref erence system . Only r otatio n s of t h e head around the two axes z and y are measured .
TV Camera
Fig . 1.
Schematic view of the appa ratus to dete rmin e t h e line of sight by measuring head movements .
The visua l feedback of the head position is realized by supe rposing on t h e vis u a l field a light dot which i s the reflected be am of the light source L . This light source is 2 coupled with the head and its optics can be adapted in order to get a s harp im age in the plane of the target . Ou r investigations have been performed with the prototype shown in Fig . 2. Here the head had to be carry a l oad o f 160 grammes .
Fig. 3 .
A s imple Visually Cou pled System which is composed of two sources of light and a s i mp l e optics . The visual feedback i s reac h ed by a li g ht dot superposed on the visual field by means of the glass and t h e focusing adjustment pictured on the right hand side. Th e head has to ca rry a l oad of about 160 grammes.
The two frames of r eference wh ich must be considered f o r the interpretation of the measured data a re shown in Fig. 3 . On the left hand side the positions of the p o int of h ead rotation D, the eye A and the target F are described in the laboratory system or working space . On the right hand side all these points , together with the light dot , are related to a r eference system which is firmly coupl ed with the head. The output of the Video-Analyzer in Fig. 1 is a measure for the head r o tati o n a round two axes in this refer ence system.
Contro l o f a typewr iter by h ead movements In many cas es the visual feedback of the head position can b e performed by observing a cursor on a monitor which may be a h ome TV monitor. The c urs or is shifted by head movements and mu s t be made to coinc ide with a desired character on a foil which is f ix ed on the monitor screen . This procedure has the advan tage o f immediately marking the relevant target. If it s position r e la tive to the foil i s stored in a computer , the measur e d data can be used in order to
Contro l of Input Va riabl es by Head Movements control technical devices such as a type writer or a speech synthesizer as indicated in Fig . 4. Here the measured analog signals are fed inot a computer where this input is Lransformed into suitable control signals . We have investigated the control of a typewriter. For safety reasons, an acknowledge ment of the called - up information will frequently be necessary.This may be effected by optical or acoustic means . We have integrated a voice output unit (voice synthesizer) producing an acoustic acknowledgement by an appropriate access to the word - organized storage .
283
Fixation of a Grid a""
~
~
b
~
-<,.
It.
~
1)
"-
1:'
~
-'\
..,
I.
~
fV
· j2S ·
--<\- . . -- ~ -
-{
!:;
~
~
----i---......--
25'
---1 ~
Speech Synthesizer
x VideoAnalyzer
~
Y
Encoder Control Unit Storage
Head Rotation
i Visually Coupled System Fig. 4.
~
Typewriter
Electronic devices for analyzing a video signal and the generation of acoustic and opt ical pattern.
Considering that , for the release of a desired function , hand or foot movements are not feasible , we have coupled the starting pulse with the dwell time of the cursor in the different fields of our 5x6 input matrix. With a simulated typewriter a dwell time between 0.3 and 0 .4 s has proved to be suitable .
Fig. 5 .
•
.
Fixation accuracy when the subject has to shift an optical mark by head movements until it coincides with a point of the grid .
Wher eas with the registrations shown in Fig. 5 the writing process after each fixation is interrupted by lifting the writing aid at the plotter , the curves shown in Fig. 6 - 8 show a continuous registration of the head movements following the lines of the meandering pattern at bottom left in Fig. 6, the individual characters at the top of Fig . 7 , and the individual circuits of a printed board in Fig. 8 . Also with these continuous registrations the deviations from the should-be-value do not exceed a visual angle of 10 '.
RESULTS When using the VCS, the fixation precision was measured with the aid of a grid which was watched at a visual angle of 8° in a horizontal, and of 6° in a vertical direc tion. The angular changes of the head related reference system as compared with the laboratory system were measured with the aid of t h e experimental device shown in Fig . 1. Two analog voltages corresponded to the rotations of the head . A registration of these voltages with the aid of an x - y plotter is shown in Fig. 5 . Some elements of the pattern were fixed for approx. 1 s. The registration shows that the amplitudes of the fixation movements in both x and y directions are, i.a. , recorded at a visual angle of 10', this corresponding to space of less than 3 mm between elements at a distance of 1 m.
1
25'
AA Fig. 6.
Pattern copied
by Head Movements
Continuous registration of the head movements following the lines of the meandering pattern at bottom left.
284
A. Korn
are successively fi x ed by means of saccadic eye movements .
AHTE KDLN AHTE Fi g . 7 .
Co n t i nu ous r egistration of head moveme n ts f o llowin g the l ines of the indiv i d u a l c h aracters at the top .
Th e re l ease of the sta r ting pu l se for printi ng a c h aracter via the d we ll time when fi x ing the corresponding matri x element was found to be agreeab l e . The subjects so far Ilsed for experiments were members of the Institute .
l'
Pi.g . 9 .
Contr olling a typewriter by h eau mov e me nt s . The l etters of the words " GUTEN MORGEN " (good morning) are fixed successively by head movements with the aid of the Vis u ally Coupled System . A time of approx . 3 . 5 s wa s r eq uired for selecting 1 1 characters.
CONCLUSION
I /
Fi g . 8 .
Co n ti n uous r e g istration of h ead movemen t s f o ll o wing t he ind i vid u al c ir c u its o f a pr i n t ed boa r d at the t op .
Whe n co ntro lling a typ e wri t er b y h ead move ments , the c ha rac t e r s of the s imul ated k ey b o ard must b e fix ed s u ccessive ly wi t h t h e aid o f t he VCS . For the words " Guten Morgen " (good morning ) the r e gi s tr a t io n o f t h e co rres p o nd ing h ead moveme n ts is p r esented in Fig. 9. Fo r s el e cting 11 c h a r ac t ers , a tim e of appro x. 3.5 s wa s r equir e d , equ a l t o 0 . 3 s p e r c h a r ac t e r . A n egli gibl y s h o r te r tim e i s ne eded wh en t he s i ng l e c h a r acters
Our investigations with a prototype of a simple Visually Coupled Sy stem s how that a fast se l ection of alpha - numeric characters and a faithful reproduction of graphic pattern can be performed by head movement s . This prototype i s st ill uncomfortable in order to be us ed by seriously handicapped persons in daily life . Nev e rtheless we are sure that the optics , th e light sourc es , and the device for the mea surement of the two degrees of head rotati o n can b e r e duced to an acceptable s ize . If the visual feedback is achieved by a cursor , that is a light spot for example on a hom e TV monitor , than nothing but the rotation of the h ead around two axes must be measured which is also possible by simple non- optical methods l ike pote n tiometers or inertia sensors . Furt h er improvements can o n ly be reached by a c l ose cooperation wit h handicapped persons.
ACKNOWLEDGEMENTS Th is repo r t was supported by Th e German Federal Ministry of Defense
Control of Input Variables by Head Movements
REFERENCES Korn, A. (198 0) . Verfahren zum indirekten Messen der Blickrichtung. Patent No . 26 29073 of the German Patent Office. Korn, A. (1981) . Visual Search: Relation between detection performance and visual field size. Proc. of the 1s t European Annual Conference on Human Decision and Manual Control , Delft, May 1981 , 27 - 34. Russo , L. (1978) . Helmet mounted visually coupled systems . Proc . of the S.l.D . Vol.19/4, 181-18 5. Zangemeister , W.H., Lehmann, S ., Stark , L. (1981) . Sensitivity analysis and optimization f or a h ead movement model . BioI. Cybern . Vol . 41 , 33- 45.
285