Computer analysis of eye movements

computer methods and wograms in biomedicine II'slk

I~

%1 l l \ [ I } l (

Computer Methods and Programs in Biomedicine, 40 (1993) 181-187

Computer analysis of eye movements •

•

S e l i m S. H a c l s a h h z a d e

>g

a

" , J o h n S. A l l e n b, L a w r e n c e W. S t a r k c

ULandis & Gyr. Corporate R&D (4737). Gubelstr. 22. CH-6301 Zug. Switzerland hUniversity of Auckland. Anthropology Department. Private Bag. Auckland. New Zealand 'Neurology Unit. University of CaliJornia. Berkeley. CA 94720. USA

Abstract An interactive program package for the acquisition, analysis and plotting of human eye movements is introduced. It is shown that the programs described in this paper can be used by scientists in a wide range of disciplines in spite of their different data analysis requirements. An example dealing with smooth pursuit tracking is given.

Key words: Eye movements: Data acquisition; Biomedical analysis data

1. Introduction

Analysis of human eye movements is a broad spectrum endeavor that occupies, among others, bioengineers, neurophysiologists, neurologists, psychiatrists, educators, ergonomists, and pharmacologists. In most cases, the eye movement data are analyzed visually, which is very time consuming. Therefore, various efforts have already been made to automate the analysis process [I,2]. However, most of the existing programs are tailormade for very specific tasks and lack the generality, portability and ease of use one might expect from a general purpose package. The package described in this note was designed to counter these deficiencies and developed to perform general acquisition, analysis, and plotting tasks in a user friendly, interactive fashion. It has both auto*Corresponding author. Tel.: +41-42-244737; Fax: +41-42244693 E-Mail: selim @ nimbus.ethz.ch. 016%2607/93/$06.00 COMMET 01385

mated and manual data editing procedures, is coded in C and can run on most DOS or Unix systems to analyze saccades, fixations, and smooth pursuit movements•

2. Measurement methods

Two basically similar infrared reflectometry methods are used to measure eye movements• The first uses an infrared video camera to take an image of the eye with the aid of a diffuse infrared source reflected from the cornea of the subject whose head rests on a chin-rest and leans on a forehead-rest (see Fig. I). This image is then processed to result in a pure black and white image (no gray tones), where the pupil appears as a round black figure on a white background accompanied by a Purkinje image. The position of the eye is determined through the computation of the center of the pupil and the Purkinje image [3]. The

© 1993 Elsevier Scientific Publishers Ireland Ltd. All rights reserved

182

S.S. Hacisalih:ade et al./Comput. Methods" Program.~ Biomed 40 (1993) 181-187

EYE

MMI SYSTEM 2000

MONITOR

IBM PC

Fig. 1. Measurement configuration. (a) Subject, (b) infrared light source, (c) TV camera, (d) eye tracker (Micromeasurements System 2000) (e) monitor, (f) computer with the acquisition (MEMGET), analysis (MEMDAP) and plotting (MEMPLOT) programs.

pupil opening area as well as the vertical and horizontal position of the eye are recorded by a data acquisition program running on a PC with adequate interface. The same program can also record a visual stimulus (for visual tracking measurements) if present. The second method uses adjustable infrared LEDs and phototransistors mounted on a lensless eye-glass frame worn by the subject [41. The phototransistor currents drive a differential amplifier whose output corresponds to the eye position which is recorded on a magnetic tape as analog signals which have to be digitized later for automatic analysis.

3. M E M G E T M E M G E T is the eye movement data acquisition program. It uses the parallel ports of an IBM PC to receive data from the Micromeasurements System 2000, which corresponds to the angular eye position. The theoretical encoding resolution of 9

bits corresponds to 3 arc minutes in a visual field of 22 degrees, an order of magnitude better than the accuracy of the system and sufficient for most applications. The gain factor is different for every subject. Therefore, fastidious individual calibration is an absolute necessity to enable a reasonable quantitative analysis of the eye movements. The data are recorded in ASCII files in five columns: pupil area, horizontal eye position, vertical eye position, horizontal tracking stimulus, vertical tracking stimulus. The sampling rate for the TV system is 60 Hz. Sometimes this is not sufficient for the analysis of fast saccades. The size of the file depends on the duration of the measurement and the sampling frequency. In this context, infrared glasses can be used for eye position measurement with a 1 kHz sampling rate. M E M G E T greets the user and starts the calibration procedure. The calibration consists of measuring the eye positions while the subject looks at some grid points on a screen. In this fashion, the non-linear mapping of the regular grid on the

S.S. Hacisalihzade et al./Comput. Methods Programs Biomed 40 (1993) 181-187 screen to the measured irregular eye positions corresponding to that grid can be determined numerically. This mapping is then used to increase the accuracy of the eye position recording by correcting each actual measurement with the corresponding distortion factor which is determined by a two dimensional linear interpolation. The calibration typically takes 20 s and is useful to find out about the quality of recording: the straight lines made up of points on the calibration card should be mapped to horizontal or vertical straight lines with the measurement. A linear regression analysis tells the user whether the recorded calibration points lie on such lines or whether they deviate too much, in which case the calibration can be repeated after minor adjustments of the camera or the light source until satisfactory results are achieved. Once the calibration is completed, the program indicates its readiness to accept data and asks under what file name the data should be stored and how long the measurement should continue. An upper limit of 60 s has been set as the default, since much longer measurement sessions are not practical due to rapid onset of fatigue. After receiving satisfactory answers, M E M G E T begins to record the eye position data.

4. M E M D A P M E M D A P (Multipurpose Eye Movements Data Analysis Program) is the program used for the analysis of eye movements. It starts by asking the user to supply the name of the data file which is to be analyzed (e.g. ABCDE.DAT); note that all created data files from analysis will have the same name as the original file name but different extensions. The size of the file is determined and the data are read. Data collected during blinks must be removed, since the eye position during a blink is indeterminate (experience with the measurement methods described above shows that values which change rapidly and erratically, in other words not like eye movements at all, are recorded during blinks). They are removed by a nonlinear threshold median filter [5] where the output is the median value of the input window rather than the mean. This kind of filtering used frequently in bio-

183

logical data analysis to get rid of artifacts can remove blinks with little effect on the eye position waveform. The data may be further filtered linearly to reduce the high frequency measurement noise and quantization noise. Here the user either chooses the default option which is a three point moving average or a finite impulse response filter (FIR) which computes the output y[n] as a function of the input u[n] according to N-I yln] =

~ a aiuln - il i=0

The order of the filter N and the coefficients a0. an-l, have to be specified by the user. Empirical results show that choosing N = 5 or N = 7 and a i = 1 often give best results. The filtered data are stored in a file with the extension FIL. From this point on the program works with the filtered data. This preprocessed signal can now be used in many ways depending on the eye movement being studied. The velocities (horizontal, vertical and resultant) of the eye movements as well as that of a possibly present visual stimulus are computed and stored in files with the extension VEL. Furthermore, histograms of velocities and accelerations are also recorded in files with the extension VLH. If a visual stimulus is present in the data file, that is if the measurement was of a visual pursuit tracking, measures of the quality of tracking are computed and written in a file with the extension TRK. These measures are the position P and the velocity V RMS errors defined as

] ,!2,

P =

s(tk) - e(tk)) 2

t_ k

V=

[~( s(tk)-s(tk-l)-At e(lk) -- e ( l k - l)-

21

I/2

(2)

S.S. Hacisalihzade et al./Comput. Methods Programs Biomed 40 (1993) 181-187

184

The summations are over the duration of the measurement s(t) and e(t) indicate the time-histories of the stimulus and the eye position. RMS errors that would have resulted if the subject did not follow the target at all are computed according to

pO =

s2(tk

,

V° = [ ~ ( s(tk) -Ats(tk - ~) ) 2]

(3)

Thus, unitless RMS errors normalized with zero eye movement errors can now be defined as

p, = p/pO, V' = V/V°

(4)

All the variables in Eqs. 2 - 4 have both horizontal and vertical components. Subsequently, the resultant speed of the eye movements are analyzed. Eye movements are necessary for vision while looking at an object which spans more than several degrees in the subject's field of view, because detailed visual information can only be obtained through the fovea, the small (about one degree) central area of the retina which has the highest photoreceptor concentration. Therefore, the brain directs the eye to move in such a way as to foveate successively onto the points of interest [6]. While viewing a stationary object, the eyes alternate between fixations and saccades, very rapid eye movements. Each saccade leads to a new fixation. Typically there are about three saccades per second, but since saccades are so fast, they occupy only about 10% of the total viewing time. Vision is suppressed during the saccades and thus almost all the visual information is collected during the fixations [7]. Clearly, eye movements play a very important role in visual perception. It has been found about two decades ago that people have repetitive and idiosyncratic ways of inspecting and recognizing a particular familiar object; these patterns were named scanpaths [8]. M E M D A P recognizes and labels an eye movement as a fixation if for a required period it meets both of the following criteria: (1) eye movements

must not exceed a given speed, (2) the resulting eye position must not deviate more than a given distance. The duration and occurrence of each fixation is recorded. A histogram of fixation frequency versus duration as well as the shortest and longest duration of fixations are recorded in a file with the extension FXS. This file also includes the mean fixation duration with the standard deviation as well as the slowest and fastest eye movement during the fixation. A special feature, which is of utmost interest to researchers studying dyslexia, is the number of fixations following a movement in the direction of reading and following a movement opposite to the direction of movement (regressions) while reading is also included. A similar analysis of the resultant speed of the eye movements helps to classify the eye movements which are not fixations as smooth pursuit movements or saccades. After being decided whether an eye movement is a saccade or a smooth pursuit movement, statistical data similar to that of fixations (like duration histograms, minimum, maximum and mean durations as well as the velocity of the slowest and fastest movements) are recorded in files with the extensions SPS (for smooth pursuit statistics) and SCS (for saccade statistics). In some cases, one might encounter some eye movements during a session which cannot be classified automatically as either fixations, smooth pursuit movements or saccades. A scanpath may be represented as a sequence of fixations, connected with idealized straight lines. Therefore, the time when a fixation occurs, its duration and the average horizontal and vertical position of the eye during a fixation as well as the standard deviation from the mean point are recorded in a file with the extension SCP.

5. M E M P L O T For the most part, eye movements can be accurately plotted in a simple and straightforward manner. The simple requirements of most investigators can be met by a program capable of plotting horizontal signal vs. time, vertical signal vs. time and horizontal vs. vertical signals, with the incorporation of a target signal as necessary.

S.S. Hacisalihzade et al./Cornput. Methods" Programs Biomed 40 (1993) 181-187 i-IMS

EYF AND TARGEI POST'ION

185

ERROR

~

8

5%

(DEGREES)

10 o t,2

10 15 L

1---

0

I

1

J--.--

2

3

l

.__

4

1

~

5

.....

6

VELOC;rY

I

I

7

8

_ . ~

.....

l

9

RMS ERROR -

10

~Og

80[.

_~ . 6 0

.

. l

.

.

. 2

.

. ]

I

5 II~IE

l;

; 8

:'

__ 9

tO

lSECOeal~)

RMS

EYE AND TARGET POSITION

ERROR

=

24.3%

(DEGREES)

15 10 5 0 5 10 15

L, 1

__.

I

I

I

I

I

I

1

I

I

2

3

4

5

6

7

B

g

10

v~_~lrr

~

E2qAC~ -

78 6 I

75 -

0

-tO00

t

2

3

4

~

6

7

8

g

tO

TIME (SECONI~I

Fig. 2. Smooth pursuit eye movements of normal and schizophrenic subjects. (a) Position vs. time (normal), (b) velocity vs. time (normal), (c) position vs. time (schizophrenic), (d) velocity vs. time (schizophrenic). The continuous and broken lines represent the response and stimulus, respectively. A high number of saccadic intrusions are visible and quantifiable in terms of higher normalized RMS error values in schizophrenics. Velocities are normed and unitless.

MEMPLOT is the plotting utility for M E M D A P generated files. It uses flexible menus for flow control which allows the user to skip around. After the name of the file to be plotted, its length and number of columns are determined, data are read and control flow is passed to the proper plotting function as indicated by the file name extension. Plots of D A T files allow the user to quickly

examine unprocessed data files before running them through M E M D A P . The user can choose the desired type of plot from a menu (horizontal vs. time, vertical vs. time or horizontal vs. vertical) and can look at it on the monitor or send it to the plotter. Plotting functions for velocity are similar. The axes are scaled automatically by the plotting system. This is especially important in looking at

186

S.S. Hacisalihzade et al./Comput. Methods Programs Biorned 40 [ 19931 181-187

very saccadic tracings where the velocity can vary widely. MEMPLOT plots scanpaths in the following manner: the scanpath plot has its axes labeled in degrees. The axes can be rescaled continuously, because it is often desirable to plot scanpaths directly onto the picture, screen or drawing the subject was looking at when the recording was made. By making changes to the plot or the axes' extent, a match can be made for plotting the scanpath on a representation of the original stimulus. When the desirable rescaling of the axes has been reached, the plot can be sent to the plotter with or without axes, with or without fixation boxes, and with or without numbers. Fixation boxes are boxes drawn around each fixation. The area of each fixation box is proportional to the amount of time spent at a fixation relative to the total time spent fixating. The absolute size of the fixation boxes can be determined by the user. 'Numbers' are simply sequence labels placed by the fixations that number them in chronological order (see Fig. 3).

6. An example The program package described above has already been successfully used in research projects dealing with cognitive psychology, special education, medicine, engineering and aesthetics. The illustrations in Fig. 2 are taken from an anthropological/psychiatric application: studying the effects of ethnicity in the etiology and distribution of schizophrenia. That can be objectively studied through the use of a culture-neutral eye movement marker found in a high percentage of schizophrenics. The pursuit eye movements of Japanese schizophrenics were recorded (MEMGET), analyzed (MEMDAP) and displayed (MEMPLOT) to that end [9,10]. Indeed, it was found that the smooth pursuit deficiency in schizophrenics is culture independent.

7. Conclusions An eye movement acquisition and analysis package, written in the C language, that can be

m

".Z -'7,

~

"; ..

Fig. 3. A subject looking at the picture of the baby first fixates on his left eye (1), then on his left hand/knee (2), then on his mouth (3) and finally on his right hand/foot (4). The shortest fixation is on the left hand/knee (hence the smallest box) and the longest fixation is on the mouth (hence the largest box).

used on DOS or Unix based PC systems was described. It was designed around a specific video front end although it can also be used with others and infrared spectacles. Provisions were made for automatic and manual data editing to eliminate blinks, etc. The programs can analyze and plot pursuit eye movements, saccades, fixations and scanpaths. This package, about 4000 lines of C code, is already in use for a variety of general purpose applications like biological psychiatry, cognitive science, and bioengineering and is available free of charge.

8. Acknowledgements This work was partially supported by the Swiss National Science Foundation grant 5.521.330. 615/7 (S.S.H.), and by the National Science Foundation Graduate Fellowship Program and the Scottish Rite Schizophrenia Research Program (J.S.A). The authors thank Huy Ngo, Joe Terdiman and Shin Kao for their assistance during the project.

S.S. Hacisalihzade et aL/Comput. Methods Programs Biomed 40 (1993) 181-187

9. References [1] M. Juhola, Detection of nystagmus eye movements using a recursive digital filter, IEEE Trans. Biomed. Eng., 35 0988) 389-395. [2] R.W. Baloh, L. Langhofer, V. Honrubia and R.D. Yee, On-line analysis of eye movements with a digital computer, Aviat. Space Environm. Med. 51 (1980) 563-567. [3] K. Sherman, Micromeasurements System 2000 Eye Monitor operator's manual (Micromeasurements, Berkeley, 1987). [4] L.R Young and D. Sheena, Eye movement measurement techniques, Am. Psychol. 30 (1975) 315-330. [5] J.W. Tukey, Exploratory Data Analysis (AddisonWesley, Menlo Park, CA, 1977). [6] B.L. Zuber, Models of Oculomotor Behavior and Control (CRC Press, Boca Raton, FL, 1981).

187

[7] L.W. Stark, J.A. Michael and B.L. Zuber, Saccadic suppression: a product of the saccadic anticipatory signal, in: Attention in Neurophysiology, Eds. C.R. Evans and T.B. Mulholland (Butterworths, London, 1969). [8] D. Noton and L.W. Stark, Scanpaths in eye movements during pattern perception, Science 171 (1971) 308-31 I. [9] J.S. Allen, K. Matsunaga, T. Nakamura, F. Kitamura, T. Furukawa, S.S. Hacisalihzade, V.M. Sarich and L.W. Stark, Schizophrenia, eye movements, and biocultural heterogeneity, Hum. Biol. 62 (1990) 337-352. [10] J.S. Allen, K. Matsunaga, S.S. Hacisalihzade and L.W. Stark, Smooth pursuit eye movements of normal and schizophrenic subjects tracking an unpredictable target, Biol. Psychiatry 28 (1990) 705-720.