- Email: [email protected]
Ecological validity of neuropsychological assessment: Do neuropsychological indexes predict performance in everyday activities?
Applied & Preventive Psychology 2:209-215(1993). Cambridge UniversityPress. Printed in the USA.
Copyright © 1993 AAAPP 0962-1849/93 $5.00 + .00
Ecological validity of neuropsychological assessment: Do neuropsychological indexes predict performance in everyday activities? BARBARA A.,WILSON MRC Applied P~yehology Unit, Cambridge
Abstract
This paper discussesthe nature of neuropsychological assessment within the context of ecological validity. It considers the extent~tO which traditional neuropychological tests can predict performance in everyday life before going on to,describe several new procedures designed to map directly on to real life functioning. Key words: Autobiographical memory, Ecological validity, Neuropsychological tests, Unilateral neglect
difficulties that preclude or hinder attempts to compensate for or bypass her problems. While it can be demonstrated that neuropsychological indexes such as those provided by the range of assessments taken by the two preceding subjects indicate that one of them is likely to have less trouble compensating for problems encountered in everyday life, it remains true that these indexes do not give us sufficient detail to be able to predict what kinds of everyday problems are likely to be faced, nor do they tell us much about the nature and frequency of the problems. Sunderland, Harris, and Baddeley (1983) demonstrated these shortcomings when they studied a group of head-injured patients, their relatives, and a number of orthopedic controls. The authors questioned the usefulness of traditional memory tests for predicting difficulties in everyday life. They discovered that, although some neuropsychological tests such as the prose recall passage from the Wechsler Memory Scale (Wechsler, 1945) were sensitive in detecting organic impairment--that is, their ability to discriminate the head-injured subjects from controls--they were not able to predict the kinds of everyday memory failures reported by either the patients themselves or their relatives. The problem posed, then, for those interested in obtaining as complete a prediction as possible of the future problems likely to be encountered in the everyday lives of brain-injured subjects is that many neuropsychological tests, particularly those regarded as traditional or "laboratory oriented," can only indicate the likelihood of problems and not the specific nature of those problems; The preceding remarks should not be taken as a criti-
L.E. is a young man who sustained a brain hemorrhage when he was a first-year undergraduate at Cambridge University. This left him with a severe memory impairment and all the characteristics of the amnesic syndrome. His scores on all tests of episodic memory and new learning are in the very abnormal range, yet he lives alone, is in paid employment, and remains independent in all activities of daily living. Conversely, V.K., who is a young woman of similar age to L.E., and whose scores on the same tests of memory are in the normal or mildly impaired range, cannot live alone, is unemployed, and needs help with many of her daily activities. Judged simply on data provided by the neuropsychological memory tests taken by both of these subjects, we would have to conclude that these particular tests do not predict performance by the two subjects in their everyday life. However, by looking at evidence from a more complete neuropsychological assessment, we can find broader evidence suggesting that L.E. will be able to cope better in everyday life. We learn that he has focal brain damage and a pure amnesic syndrome so that, apart from memory problems, he has no other intellectual impairment. His attention, perceptual, language, reading, and organizational skills remain intact. V.K., on the other hand, has diffuse brain damage following a severe head injury and, although her memory problems are slight in comparison to L.E., she has numerous cognitive deficits including perceptual, word-finding, reasoning, and organizational Send correspondenceand reprint requests to Barbara A. Wilson, MRC Applied PsychologyUnit, 15 Chaucer Road, Cambridge CB2 2EF, U.K. 209
210
Wilson
cism of certain neuropsychological tests. It would be wrong to expect all neuropsychological assessment procedures to address likely outcomes connected with everyday performance. I argued elsewhere (Wilson, 1991b) that assessments are carried out to answer questions, and the assessment tools used are determined by the nature of the questions asked by the investigator. Thus, a theoretical question, such as whether or not implicit memory is a unitary concept, will require a different form of investigation than a question concerned with teasing out a problem that is likely to be present in the life of a particular patient with a particular injury to the brain. In the former, we would need to assess a large number of subjects on as many implicit memory tasks as possible, looking for double dissociations. In the latter, we would need to carry out neuropsychological tests aimed specifically at revealing problems that are likely to occur in normal everyday living. We would also need, in the latter investigation, to interview the patient and relatives to find out how problems are manifested in real life, observe the patient in a number of settings, and use self-report measures and rating scales. Neuropsychological tests may be used primarily to answer questions about the nature of a subject's intellectual functioning: which particular skills are impaired and which are intact, and how the subject compares to others of the same age. Although these are not direct questions about performance in everyday life, they must nevertheless be asked in order to build up a picture of a subject's cognitive strengths and weaknesses so that impossible demands are not made on the subject when planning treatment or rehabilitation prior to return to some form of everyday living. Other neuropsychological tests may indeed predict everyday performance even though they are not primarily designed for this purpose. For example, stroke patients who make a large number of errors on tests of letter cancellation tend to have more accidents than stroke patients who make fewer errors, presumably because failure on this test reflects the presence of unilateral neglect. Neglect is associated with failure to respond, attend, or orient to one side of space, thus causing omissions in cancellation tasks and, for some patients, accidents in real life. Because not all patients are affected in this way in real life, the letter cancellation test is a useful predictor only at the group level. It is important to differentiate between tests that predict real-life behaviors for individuals and those that predict the same for groups. While it is true that subjects who score poorly on the Wisconsin Card Sorting Test (WCST), for example, are likely to have problems finding and keeping employment, this is not true for all subjects who do badly on this test. Poor performance on block design tests are associated with poor dressing skills and other visuospatial tasks, yet
Newcombe (1987) reported the case of a surgeon who, despite a below-average score on this test following a gun shot wound, was able to continue surgical operations successfully for the rest of his working life. Similar examples of exceptional subjects can be obtained from almost every experienced neuropsychologist. What these neuropsychological tests fail to do of course is take into account such factors as motivation, personality, individual styles, family support, and previous reinforcement history. For this reason, they are unlikely to be successful at predicting real-life performance for each individual patient. One way to enhance their relevance to everyday life is to combine their indexes with information gathered from behavioral observations, interviews, rating scales, self-report measures, and other assessment procedures. For further discussion of these issues, see Hart and Hayden (1986), Wilson (1991b), and Wilson, Cockburn, and Baddeley (1989). Another way of enhancing the ecological validity of neuropsychological indexes is to devise tests that map directly on to everyday behaviors such as the Test of Functional Communication for Aphasic Adults (Holland, 1980) and the Cognitive Competency Test (Wang & Ennis, 1986). This is an approach that colleagues and I have been adopting for a number of years, and the remainder of this article will describe some of the procedures we have developed. The Rivermead Behavioural Memory Test When I began working at Rivermead Rehabilitation Centre in Oxford, England, in 1979, I assessed patients as I had been taught and reported the results at weekly ward rounds. I would say something like "Mrs. Smith is two standard deviations below the mean on the Rey-Osterreith figure" or "Mr. Brown is below the first percentile on the Ravens Matrices." Therapists would typically groan and say, "But is she safe to go home?" or "Can he go back to work?" I realized that I could not answer such questions posed by therapists because the tests I relied on were not representative of problems faced by patients in real life. The reports I wrote were typically filed away and not required in the process of rehabilitation. In short, the priorities of patients and relatives were different from mine. I considered whether or not it was possible to change this mismatch between information I was supplying and information wanted by others involved in work that would eventually lead to some form of rehabilitation that enabled patients to return to their homes and to a life outside Rivermead Rehabilitation Centre. I set about designing a memory test that would predict everyday memory problems and also monitor change over time. This aspect of my work culminated in the development of the Rivermead Behavioural Memory Test (RBMT), a standardized test, administered and scored like any other
Ecological Validity of Neuropsychological Assessment standardized test but including analogs of everyday tasks rather than clinical or experimental material such as paired associates or abstract drawings. Items for inclusion in the test were selected on the basis of observations of memory-impaired people at Rivermead and from the work reported by Sunderland et al. (1983). The 12 subtests of the RBMT include remembering a new name, recalling a new route (immediate and delayed), remembering a newspaper article (immediate and delayed), remembering to ask for an appointment when an alarm sounds, remembering to deliver a message, face and picture recognition tasks, orientation questions, and remembering to ask for a personal belonging at the end of the test. Following the pilot stages, Janet Cockburn from Oxford and Alan Baddeley from Cambridge joined the research project and the test was published in 1985 (Wilson, Cockburn, & Baddeley 1985). There are four parallel versions of the test, and both interrater and testretest reliability are high. Three methods were used to establish the validity of the RBMT (Wilson, Cockburn, Baddeley, & Hiorns 1989), the most important being establishing ecological validity. This was determined by correlating performance on the RBMT with memory failures observed by therapists during patients' daily rehabilitation programs. Therapists observed 80 patients for a mean of 35 hr per patient (range 16-55 hr). The correlation between therapists' observations and RBMT scores was -.75 (p < .001), suggesting that the RBMT is a good and ecologically valid measure of everyday memory performance. The fact that the RBMT has four parallel versions makes it a useful measure for monitoring change over time. Norms exist for all ages from 5 to 96 years. Children between 5 and 10 years of age are given slightly different versions that are more suited to their age (Wilson, Ivani-Chalian, Besag, & Bryant, 1993). Other studies have corroborated the usefulness of the RBMT as an ecologicaly valid test. Wilson (1991a), in a long-term (5-10 years) follow-up study of 54 patients referred for memory therapy between 1979 and 1985, found that the RBMT discriminated between patients who were independent (independence operationally defined as being either in paid employment or in full-time education or living alone) and those who were dependent (Z2 = 24.72, p < .001), whereas the Wechsler Memory Scale--Revised (WMS-R) did not discriminate. Schwartz and Macmillan (1989) found that the RBMT correlated with employment status, whereas a memory questionnaire did not. They concluded that the RBMT was a more objective measure of functional ability. Kotler-Cope (1990) found the WMS-R was not as valid a measure of everyday memory and had less ecological validity than the RBMT. Huppert and Beardsall (1991, in press) have also found the RBMT to be useful for the assessment of early dementia.
211
One of the major advantages of the RBMT is that it has good face validity: It "feels" like a test of memory to the subjects who are being assessed, unlike tests that rely on abstract drawings or pairs of words, regarded by some patients as being remote from everyday life. The Autobiographical Memory Interview Another assessment with good face validity is the Autobiographical Memory Interviews (AMI) (Kopelman, Wilson, & Baddeley, 1989, 1990). Until recently, autobiographical memory had not received a great deal of attention. In 1986, Alan Baddeley and I published a chapter about observations of autobiographical memory in 12 amnesic patients. We noted that some patients appeared to have good access to autobiographical memory at least up until a year or two prior to the onset of the amnesia. When retested a week later, they produced the same details as they had provided originally, even though they had no recall of the earlier assessment. These patients, although severely amnesic, were able to cope with social situations and did not present with management problems for the staff or their relatives. Another subgroup demonstrated "clouded" autobiographical memory in which they were able to recall events on one occasion but not on others. These patients tended to be anxious, frightened, or agitated and created difficulties for those looking after them. A third subgroup provided plenty of autobiographical information but gross confabulation, making errors and inconsistencies and producing bizarre and elaborate stories that were grossly inaccurate. We believed that a measurement of autobiographical memory could be of potential prognostic importance and might help in predicting response to treatment. Our subgroup patients with gross confabulation, for example, were all admitted to long-term psychiatric care on discharge from Rivermead. We were of the opinion that those with clouded autobiographical memory might be agitated or anxious because they had lost a sense of self due possibly to their loss of a remembered past. These remain tentative conclusions. One consequence of the 1986 study was the development of the AMI in collaboration with Michael Kopelman, a neuropsychiatrist working in London. The AMI is a semistructured interview rather than a formal test, although the procedure is standardized and objective scores can be obtained. The AMI has two sections called the personal semantic schedule and the autobiographical incidents schedule. The former assesses subjects' recall of facts from their own past life (e.g., names of school teachers, date and place of wedding); the latter assesses subjects' recall of specific events in their earlier life (e.g., an incident occurring at a subject's first school, first job, or recent holiday).
212
Wilson
Each section assesses memories across three broad chronological bands: childhood, early adult life, and recent past (during the last 12 months). Subjects typically find the test enjoyable and interesting. Is it in any way useful to assess autobiographical memory? After all, most neuropsychologists might argue that they manage without investigating this aspect of memory. We believe such an assessment is valuable for at least three reasons. First, it enables better understanding of the nature of any memory deficit observed--for example, both D.B. and K.J. (reported by Baddeley & Wilson, 1986) had very similar scores on all neuropsychological tests administered but were very different in their ability to access autobiographical memory and also very different in their degree of adaptation to their memory problem. Second, the information gathered by the AMI can help to improve advice and counseling. Third, that same information can provide an individual focus for subsequent management such as reminiscence therapy. The AMI also provides a useful tool for investigating cases in which anterograde and retrograde amnesia may be dissociated and for cases where there may be a dissociation between recall of personal semantic information and recall of personal episodic information for the same time periods. Evans, Wilson, Wraight, and Hodges (in press) reported, for example, the case of a woman assessed while in transient global amnesia. She was able to recall the personal semantic information from all time periods assessed but had very impoverished recall of episodic incidents for all periods. When reassessed 2 days later, she showed normal recall of the episodic incidents. Patients with progressive dysphasia may show the opposite pattern whereby personal episodic information is available for far longer than the personal semantic information (Hodges, personal communication).
The Behavioural Inattention Test The Behavioural Inattention Test (BIT), analagous to the RBMT, is designed to predict everyday problems arising from unilateral visual neglect. Such neglect is a heterogeneous and often transitory phenomenon in which patients fail to report, respond, or orient to stimuli on one side of space. Commonly associated with righthemisphere stroke, it is also seen to a lesser extent after left-hemisphere stroke. Patients with other conditions such as cerebral tumor and head injury may also exhibit unilateral neglect. Several studies have shown tfiat unilateral visual neglect is one of the major factors impeding functional recovery and rehabilitation success (Denes, Semenza, Stoppa, & Lis, 1982; Kinsella & Ford, 1985). Prior to the development of the BIT, there was no single standardized battery for the assessment of unilat-
eral visual neglect. Instead, a variety of procedures were (and still are) used, including the line crossing test (Albert, 1973), indented reading (Caplan, 1987), and position bias on Ravens Coloured Matrices (Costa, Vaughan, Horwitz, & Ritter 1969). Such single tests of neglect are inappropriate because (a) patients often vary in the way neglect presents itself and (b) an individual patient may show great variability in the degree of neglect, depending on such factors as fatigue, position of the tester, and presence or absence of distinctive stimuli in the testing room. For these reasons, a more substantial and varied battery of items is required, and these must allow for testing over periods of time. The immediately aforementioned individual tasks do not map closely on to the everyday demands faced by patients with unilateral visual neglect. Such patients often collide with objects, attend to only one side of the body, ignore food on one side of the plate, and experience difficulty with reading, writing, and drawing. The items in the BIT were selected partly as a result of observing stroke patients who exhibited everyday problems arising from neglect, partly from findings of other studies such as those by Diller and Gordon (1981), and partly from discussions with occupational therapists working with neglect patients. The complete BIT battery (Wilson, Cockburn, & Halligan 1987) comprises nine behavioral subtests that reflect aspects of daily life and six simple paper and pencil measures of neglect. In the former are picture scanning, telephone dialing, reading a menu, reading a newspapertype article, telling and setting the time, coin sorting, address and sentence copying, map navigation, and card sorting. The paper and pencil subtests include line crossing, letter and star cancellation, figure and shape copying, line bisection, and representational drawing. Details of the standardization and validation procedures can be found in Halligan, Cockburn, and Wilson (1991). Like the RBMT, a measure of ecological validity was obtained by using therapists' ratings of problems encountered by our patients: together with scores on a standardized Activities of Daily Living (ADL) scale. On both there were significant correlations suggesting that the BIT is an ecologically valid measure of everyday problems arising from neglect. Another study by Shiel (1990) investigated the relationship between unilateral visual neglect and ADL. She found that patients with neglect, as measured by the BIT, were significantly more likely to have problems with ADL tasks (p < .001) than those without neglect. She also found that (a) the behavioral subtests correlated higher with ADL scores than the paper and pencil subtests; (b) the higher correlations were with indoor mobility, transfer from bed to chair, and using the lavatory; and (c) the results were not due to hemianopia or motor problems.
Ecological Validity of NeuropsychologicalAssessment The Behavioral Assessment of the Dysexecutive Syndrome The Behavioral Assessment of the Dysexecutive Syndrome (BADS) is a test aimed at predicting everyday problems arising from dysexecutive syndrome (DES). Nick Alderman, Paul Burgess, Jon Evans, and I have just completed the pilot stage of this study (Alderman, Evans, Burgess, & Wilson, 1993). DES closely resembles what was once called the frontal lobe syndrome, a term that "is used to refer to an amorphous, varied group of deficits, resulting from diverse etiologies, different locations, and variable extents of abnormalities" (Stuss & Benson, 1984, p. 3). Baddeley and Wilson (1988) claimed that specification of a syndrome in terms of localization is unfortunate and potentially misleading. We do not classify memory, language, reading, or perceptual deficits in this way because it is recognized that to do so would be inadequate. The same inadequacy would seem to apply to frontal lobe damage where a functional definition is more appropriate. Although there is great variability in the extent and degree of cognitive impairment in patients with frontal lobe damage, certain features are highly characteristic. Rylander (1939) described them as "disturbed attention, increased distractability, a difficulty in grasping the whole of a complicated state of a f f a i r s . . , well able to work along routine lines [but] cannot learn to master new types of task" (p. 20). Shallice (1982) believes this pattern of deficits can be described as an impairment in attentional control. He explains this in terms of a model of attention whereby a supervisory system exerts an executive function. Baddeley (1986) believes Shallice's "supervisory system" is analagous to the central executive component of working memory (Baddeley & Hitch, 1974) and suggests the term dysexecutive syndrome as a functional characterization of patients with this particular pattern of deficits. Some of the most widely used neuropsychological tests to assess executive functioning include the WCST, verbal fluency, the Stroop test, and the Reitan Trail Making Test. Yet some patients with obvious impairments of executive functioning can score normally on some or all of these tests. Shallice and Burgess (1991) described three such people in some detail. The patients performed satisfactorily on tests thought sensitive to frontal lobe lesions yet were unable to act effectively on their own initiative as they organised virtually no non-routine activities. Shallice and Burgess made the point that in most neuropsychological tests the patient typically has a single explicit problem to tackle at any one time, the trials tend to be very short (1 minute or so or even less), task initiation is strongly prompted by the examiner and what constitutes suc-
213
cessful trial completion is clearly characterised. (pp. 727-728) Rarely are patients required to organize or plan their behavior over longer time periods or to set priorities in the face of two or more competing tasks. Yet it is these sorts of executive abilities that are a large component of many everyday activities. Shallice and Burgess's solution was to develop the sixelements and the multiple-errands tests, both being ecologically valid and aimed at tapping the more subtle "executive demands." The purpose of the six-elements test was to see how well patients could carry out six open-ended tasks (two sets of three: dictating a route, solving arithmetic problems, and writing down names of pictures) in a 15-min period. Each of the tasks was in itself relatively easy, but two simple rules had to be followed: Subjects were required to attempt (not necessarily complete) some items from each task, and they were not allowed to do two of the same set consecutively. The multiple-errands task was carried out in a pedestrian precinct, previously unknown to the subjects. They had to carry out a number of tasks in which minor, unforeseen events could occur. Errands included buying certain things as well as requiring subjects to be at a certain place at a certain time and to find out certain pieces of information (such as the shop with the most expensive item). Patients with DES both made more errors than controls and showed qualitatively different behavior (such as walking out of a shop without paying). The multipleerrands test has a high ecological validity and captures the nonroutine, problem-solving, planning, organization, and initiative required for everyday functioning. However, it also requires considerable time, planning, and organization on the part of the assessing neuropsychologist and, consequently, is not always practicable in clinical situations. Nick Alderman, Paul Burgess, Jon Evans, and I decided to modify the six-elements test in order to make it simpler for the more severely impaired and less intellectually able brain-injured patients who are more typical of neuropsychological referrals than the patients seen by Shallice and Burgess (1991). We also designed a range of other tests similar to real-life activities that would cause difficulties for some patients with DES. These included (a) a practical problem-solving task, whereby subjects have to extract a cork from a tube with the aid of certain pieces of equipment and by following a prescribed set of rules; (b) a paper and pencil problem-solving task adapted from the plan and search test in the StanfordBinet Test (Terman & Merrill, 1961), whereby the subject has to pretend he or she has lost some keys in a field, and a line is to be drawn to show how the subject would search the field to find the keys; (c) three tasks of increas-
214
Wilson
ing difficulty, demanding attention and the ability to follow rules using playing cards (e.g., say "yes" if the next card is the same color as the previous card, otherwise say "no"; (d) estimating how long it takes to do certain things such as train to be a medical doctor or blow up a party balloon; (e) a test involving the use of a map and planning a route round a zoo, again with certain rules needing to be followed and including two versions varying in level of difficulty; and (f) the simplified six-elements test already described. In addition, a questionnaire about everyday problems is given to each patient and to a carer or relative of the patient. Preliminary results from the pilot study, in which 33 controls and 36 brain-injured people were tested, suggest that (a) the BADS is a better predictor of everyday problems (as measured by the relatives/carers' questionnaire) than the WCST, (b) the modified six elements and the harder version of the zoo map tasks are the most sensitive individual subtests in the battery, (c) using scores from all subtests is best for predicting impaired everyday functioning, and (d) double dissociations are not uncommon. For example, one of our DES patients had no difficulty with the practical problemsolving task but experienced much difficulty with the paper and pencil problem-solving. The results of another patient showed the reverse of this pattern. From these and other examples, it would seem likely that it is better to administer several tasks in order to obtain a true measure of DES. Although we cannot provide a conclusive appraisal of the BADS while research is continuing on its effectiveness, it does hold promise of being a useful assessment battery that provides good prediction of everyday problems arising from impaired executive functioning. Concluding Remarks In 1989 a highly provocative paper was published, entitled "The Bankruptcy of Everyday Memory" (Banaji & Crowder, 1989), in which the authors claimed that the naturalistic study of memory has not been productive. Subsequently, a whole issue of the American Psychologist (1991) was devoted to (mostly very critical) replies to the original article. Ceci and Bronfenbrenner (1991) said, for example, that
by equating mental behaviour with the actions of inner chemical processes, (the authors of the original article) have overlooked a critical difference; namely, living beings are active agents who not only adapt to contexts but also modify, select and create them. (p. 27) A further comment by Morton (1991) argued that "the progress of research in memory is being impeded not by the everyday memory movement, but by an excessively restricted theoretical base" (p. 32). Perhaps the most pertinent message is that from Tulving (1991), who wrote, "there is no reason to believe that there is only one correct way of studying m e m o r y . . . [this] is not a zerosum game in which only one side can win" (p. 41). As argued earlier, we need a complete range of neuropsychological assessments in order to provide the clearest picture o f a brain-injured patient's present neuropsychological condition and possible future progress after returning to everyday life in a community. It is unhelpful for neuropsychologists who ask a particular set of questions to attack other neuropsychologists, asking a different set of questions, because their answers do not seem to apply to the first group. The way forward for the investigator is to be broad minded about testing and recognize that better diagnosis will follow from using tests from a wide range of sources, some of which will be more theoretically based while others will be more practically oriented. Also, as recommended earlier, the tester should be willing to combine information gathered from neuropsychological tests with information gathered from behavioral observations, interviews, rating scales, self-report measures, and other assessment procedures. Doors should not be shut just because a particular investigator has been educated in a particular way. Our education as researchers is never at an end and must involve willingness to open all kinds of doors in the search for a better deal for our patients. Finally, on the specific question concerning ecological validity, I hope I have shown in this article that tests can be designed to map directly on to everyday behaviors and that these tests can provide detailed information that is relevant to the patient, the carer, and the rehabilitation worker in their efforts to achieve the best possible life for the patient returning to the community.
REFERENCES
Albert, M. L. (1973). A simple test of visual neglect. Neurology, 23, 658-664. Alderman, N., Evans, J., Burgess, P., & Wilson, B. A. (1983). Behavioural assessment of the dysexecutive syndrome. Journal of Clinical and Experimental Neuropsychology, 15, 6970. (Abstract) Baddeley, A. D. (1986). Working memory. Oxford: Clarendon Press. Baddeley, A. D., & Hitch, G. (1974). Working l'Aemory. In G.
H. Bower (Ed.), The psychology of learning and motivation (Vol. 8). New York: Academic Press. Baddeley, A. D., & Wilson, B. A. (1986). Amnesia, autobiographical memory and confabulation. In D. Rubin (Ed.), Autobiographical memory (pp. 225-252). Cambridge: Cambridge University Press. Baddeley, A. D., & Wilson, B. A. (1988). Frontal amnesia and the dysexecutive syndrome. Brain and Cognition, 7, 212230.
Ecological Validity o f Neuropsychological Assessment
Banaji, M. R., & Crowder, R. G. (1989). The bankruptcy of everyday memory. American Psychologist, 44, 1185-1193. Caplan, B. (1987). Assessment of unilateral neglect: A new reading test. Journal of Clinical and Experimental Neuropsychology, 9, 359-364. Ceci, S. J., & Bronfenbrenner, U. (1991). On the demise of everyday memory: "The rumors of my death are much exaggerated." American Psychologist, 46(1), 27-31. Costa, L. D., Vaughan, H. G., Horwitz, M., & Ritter, W. (1969). Patterns of behavioural deficit associated with visual spatial neglect. Cortex, 5, 242-263. Denes, F., Semenza, C., Stoppa, E., & Lis, A. (1982). Unilateral spatial neglect and recovery from hemiplegia: A follow up study. Brain, 105, 543-552. Diller, L., & Gordon, W. A. (1981). Rehabilitation and clinical neuropsychology. In S. B. Filskov & T. J. Boll (Eds.), Handbook of clinical neuropsychology (pp. 702-733). New York: Wiley. Evans, J., Wilson, B. A., Wraight, E. P., & Hodges, J. (in press). Journal of Neurology, Neurosurgery and Psychiatry. Halligan, P., Cockburn, J., & Wilson, B. A. (1991). The behavioural assessment of visual neglect. Neuropsychological Rehabilitation, 1, 5-32. Hart, T., & Hayden, M. E. (1986). The ecological validity of neuropsychological assessment and remediation. In B. Uzzell & Y. Gross (Eds.), Clinical neuropsychology of intervention (pp. 21-50). Boston: Martinus Nijhoff. Holland, A. L. (1980). Communicative abilities in daily living. Baltimore, MD: University Park Press. Huppert, F., & Beardsall, L. (1991). A comparison of clinical, psychometric and behavioural memory tests: Findings from a community study of the early detection of dementia. International Journal of Geriatric Psychiatry, 6, 295-306. Huppert, F., & Beardsall, L. (in press). Prospective memory impairment as an early indicator of dementia. Journal of Clinical and Experimental Neuropsychology. Kinsella, G., & Ford, B. (1985). Hemi-inattention and the recovery patterns of stroke patients. International Rehabilitation Medicine, 7, 102-106. Kopelman, M., Wilson, B. A., & Baddeley, A. D. (1989). The autobiographical memory interview: A new assessment of autobiographical and semantic memory in amnesic patients. Journal of Clinical and Experimental Neuropsychology, 11, 724-744. Kopelman, M., Wilson, B. A., & Baddeley, A. D. (1990). The autobiographical memory interview. Flampton Bury St. Edmunds, Suffolk: Thames Valley Test Company. Kotler-Cope, S. (1990). Memory impairment in older adults: The interrelationships between objective and subjective clinical and everyday memory assessment. Paper presented at the annual meeting of the Southern Society for Philosophy and Psychology, Louisville, KY. Morton, J. (1991). The bankruptcy of everyday thinking. American Psychologist, 46(1), 32-33. Newcombe, F. (1987). Psychometric and behavioral evidence: Scope, limitations, and ecological validity. In H. S. Levin, J. Grafman, M. Howard, & H. M. Eisenberg (Eds.), Neurobehavioral recoveryfrom head injury (pp. 129-145). New York: Oxford University Press.
215
Rylander, G. (1939). Personality changes after operations on the frontal lobes. Aeta Psychiatrica et Acta Neurologica Scandinavia, Supplement 20. Schwartz, A. F., & Macmillan, T. (1989). Assessment of everyday memory after severe head injury. Cortex, 25, 665-671. Shallice, T. (1982). Specific impairments of planning. Philosophical Transactions of the Royal Society of London B, 298, 199-209. Shallice, T., & Burgess, P. (1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114, 727741. Shallice, T., & Evans, M. E. (1978). The involvement of the frontal lobes in cognitive estimation. Cortex, 14, 294-303. Shiel, A. (1990). An investigation of the relationship between unilateral neglect and ADL dependency in right hemisphere stroke patients. Unpublished master's thesis, Southampton University Medical School, Southampton. Stuss, D., & Benson, D. F. (1984). Neuropsychological studies of the frontal lobes. Psychological Bulletin, 95, 3-28. Sunderland, A., Harris, J. E., & Baddeley, A. D. (1983). Do laboratory tests predict everyday memory? A neuropsychological study. Journal of Verbal Learning and Verbal Behavior, 22, 341-357. Terman, L. M., & Merrill, M. A. (1961). StanfordBinet Intelligence Scale. London: Harrap & Company, Ltd. Tulving, E. (1991). Memory research is not a zero-sum game. American Psychologist, 46(1), 41-42. Wang, P. L., & Ennis, K. E. (1986). Competency assessment in clinical populations: An introduction to the cognitive competency test. In B. P. Uzzell & Y. Gross (Eds.), Clinical neuropsychology of intervention. Boston: Martinus Nijhoff. Wechsler, D. (1945). A standardised memory scale for clinical use. Journal of Psychology, 19, 87-95. Wilson, B. A. (1991a). Long term prognosis of patients with severe memory disorders. Neuropsychological Rehabilitation, 1, 117-134. Wilson, B. A. (1991b). Theory, assessment and treatment in neuropsychological rehabilitation. Neuropsychology, 5, 281-291. Wilson, B., Cockburn, J., & Baddeley, A. D. (1985). The Rivermead Behavioural Memory Test manual. Flempton, Bury St Edmunds, Suffolk: Thames Valley Test Company. Wilson, B. A., Cockburn, J., & Baddeley, A. D. (1989). Assessment of everyday memory following brain injury. In M. E. Miner & K. A. Wagner (Eds.), Neurotrauma: Treatment, rehabilitation and related issues (No. 3, pp. 83-99). Boston: Butterworths. Wilson, B. A., Cockburn, J., Baddeley, A. D., & Hiorns, R. (t 989). The development and validation of a test battery for detecting and monitoring everyday memory problems. Journal of Clinical and Experimental Neuropsychology, 11, 855870. Wilson, B. A., Cockburn, J., & Halligan, P. (1987). The Behavioural Inattention Test. Flempton, Bury St Edmunds, Suffolk: Thames Valley Test Company. Wilson, B. A., Ivani-Chalian, R., Besag, F., & Bryant, T. (1991). Modification of the Rivermead Behavioural Memory Test for use with 5-10 year old children. Journal of Clinical and Experimental Psychology, 15, 474-486.
Copyright © 1993 AAAPP 0962-1849/93 $5.00 + .00
Ecological validity of neuropsychological assessment: Do neuropsychological indexes predict performance in everyday activities? BARBARA A.,WILSON MRC Applied P~yehology Unit, Cambridge
Abstract
This paper discussesthe nature of neuropsychological assessment within the context of ecological validity. It considers the extent~tO which traditional neuropychological tests can predict performance in everyday life before going on to,describe several new procedures designed to map directly on to real life functioning. Key words: Autobiographical memory, Ecological validity, Neuropsychological tests, Unilateral neglect
difficulties that preclude or hinder attempts to compensate for or bypass her problems. While it can be demonstrated that neuropsychological indexes such as those provided by the range of assessments taken by the two preceding subjects indicate that one of them is likely to have less trouble compensating for problems encountered in everyday life, it remains true that these indexes do not give us sufficient detail to be able to predict what kinds of everyday problems are likely to be faced, nor do they tell us much about the nature and frequency of the problems. Sunderland, Harris, and Baddeley (1983) demonstrated these shortcomings when they studied a group of head-injured patients, their relatives, and a number of orthopedic controls. The authors questioned the usefulness of traditional memory tests for predicting difficulties in everyday life. They discovered that, although some neuropsychological tests such as the prose recall passage from the Wechsler Memory Scale (Wechsler, 1945) were sensitive in detecting organic impairment--that is, their ability to discriminate the head-injured subjects from controls--they were not able to predict the kinds of everyday memory failures reported by either the patients themselves or their relatives. The problem posed, then, for those interested in obtaining as complete a prediction as possible of the future problems likely to be encountered in the everyday lives of brain-injured subjects is that many neuropsychological tests, particularly those regarded as traditional or "laboratory oriented," can only indicate the likelihood of problems and not the specific nature of those problems; The preceding remarks should not be taken as a criti-
L.E. is a young man who sustained a brain hemorrhage when he was a first-year undergraduate at Cambridge University. This left him with a severe memory impairment and all the characteristics of the amnesic syndrome. His scores on all tests of episodic memory and new learning are in the very abnormal range, yet he lives alone, is in paid employment, and remains independent in all activities of daily living. Conversely, V.K., who is a young woman of similar age to L.E., and whose scores on the same tests of memory are in the normal or mildly impaired range, cannot live alone, is unemployed, and needs help with many of her daily activities. Judged simply on data provided by the neuropsychological memory tests taken by both of these subjects, we would have to conclude that these particular tests do not predict performance by the two subjects in their everyday life. However, by looking at evidence from a more complete neuropsychological assessment, we can find broader evidence suggesting that L.E. will be able to cope better in everyday life. We learn that he has focal brain damage and a pure amnesic syndrome so that, apart from memory problems, he has no other intellectual impairment. His attention, perceptual, language, reading, and organizational skills remain intact. V.K., on the other hand, has diffuse brain damage following a severe head injury and, although her memory problems are slight in comparison to L.E., she has numerous cognitive deficits including perceptual, word-finding, reasoning, and organizational Send correspondenceand reprint requests to Barbara A. Wilson, MRC Applied PsychologyUnit, 15 Chaucer Road, Cambridge CB2 2EF, U.K. 209
210
Wilson
cism of certain neuropsychological tests. It would be wrong to expect all neuropsychological assessment procedures to address likely outcomes connected with everyday performance. I argued elsewhere (Wilson, 1991b) that assessments are carried out to answer questions, and the assessment tools used are determined by the nature of the questions asked by the investigator. Thus, a theoretical question, such as whether or not implicit memory is a unitary concept, will require a different form of investigation than a question concerned with teasing out a problem that is likely to be present in the life of a particular patient with a particular injury to the brain. In the former, we would need to assess a large number of subjects on as many implicit memory tasks as possible, looking for double dissociations. In the latter, we would need to carry out neuropsychological tests aimed specifically at revealing problems that are likely to occur in normal everyday living. We would also need, in the latter investigation, to interview the patient and relatives to find out how problems are manifested in real life, observe the patient in a number of settings, and use self-report measures and rating scales. Neuropsychological tests may be used primarily to answer questions about the nature of a subject's intellectual functioning: which particular skills are impaired and which are intact, and how the subject compares to others of the same age. Although these are not direct questions about performance in everyday life, they must nevertheless be asked in order to build up a picture of a subject's cognitive strengths and weaknesses so that impossible demands are not made on the subject when planning treatment or rehabilitation prior to return to some form of everyday living. Other neuropsychological tests may indeed predict everyday performance even though they are not primarily designed for this purpose. For example, stroke patients who make a large number of errors on tests of letter cancellation tend to have more accidents than stroke patients who make fewer errors, presumably because failure on this test reflects the presence of unilateral neglect. Neglect is associated with failure to respond, attend, or orient to one side of space, thus causing omissions in cancellation tasks and, for some patients, accidents in real life. Because not all patients are affected in this way in real life, the letter cancellation test is a useful predictor only at the group level. It is important to differentiate between tests that predict real-life behaviors for individuals and those that predict the same for groups. While it is true that subjects who score poorly on the Wisconsin Card Sorting Test (WCST), for example, are likely to have problems finding and keeping employment, this is not true for all subjects who do badly on this test. Poor performance on block design tests are associated with poor dressing skills and other visuospatial tasks, yet
Newcombe (1987) reported the case of a surgeon who, despite a below-average score on this test following a gun shot wound, was able to continue surgical operations successfully for the rest of his working life. Similar examples of exceptional subjects can be obtained from almost every experienced neuropsychologist. What these neuropsychological tests fail to do of course is take into account such factors as motivation, personality, individual styles, family support, and previous reinforcement history. For this reason, they are unlikely to be successful at predicting real-life performance for each individual patient. One way to enhance their relevance to everyday life is to combine their indexes with information gathered from behavioral observations, interviews, rating scales, self-report measures, and other assessment procedures. For further discussion of these issues, see Hart and Hayden (1986), Wilson (1991b), and Wilson, Cockburn, and Baddeley (1989). Another way of enhancing the ecological validity of neuropsychological indexes is to devise tests that map directly on to everyday behaviors such as the Test of Functional Communication for Aphasic Adults (Holland, 1980) and the Cognitive Competency Test (Wang & Ennis, 1986). This is an approach that colleagues and I have been adopting for a number of years, and the remainder of this article will describe some of the procedures we have developed. The Rivermead Behavioural Memory Test When I began working at Rivermead Rehabilitation Centre in Oxford, England, in 1979, I assessed patients as I had been taught and reported the results at weekly ward rounds. I would say something like "Mrs. Smith is two standard deviations below the mean on the Rey-Osterreith figure" or "Mr. Brown is below the first percentile on the Ravens Matrices." Therapists would typically groan and say, "But is she safe to go home?" or "Can he go back to work?" I realized that I could not answer such questions posed by therapists because the tests I relied on were not representative of problems faced by patients in real life. The reports I wrote were typically filed away and not required in the process of rehabilitation. In short, the priorities of patients and relatives were different from mine. I considered whether or not it was possible to change this mismatch between information I was supplying and information wanted by others involved in work that would eventually lead to some form of rehabilitation that enabled patients to return to their homes and to a life outside Rivermead Rehabilitation Centre. I set about designing a memory test that would predict everyday memory problems and also monitor change over time. This aspect of my work culminated in the development of the Rivermead Behavioural Memory Test (RBMT), a standardized test, administered and scored like any other
Ecological Validity of Neuropsychological Assessment standardized test but including analogs of everyday tasks rather than clinical or experimental material such as paired associates or abstract drawings. Items for inclusion in the test were selected on the basis of observations of memory-impaired people at Rivermead and from the work reported by Sunderland et al. (1983). The 12 subtests of the RBMT include remembering a new name, recalling a new route (immediate and delayed), remembering a newspaper article (immediate and delayed), remembering to ask for an appointment when an alarm sounds, remembering to deliver a message, face and picture recognition tasks, orientation questions, and remembering to ask for a personal belonging at the end of the test. Following the pilot stages, Janet Cockburn from Oxford and Alan Baddeley from Cambridge joined the research project and the test was published in 1985 (Wilson, Cockburn, & Baddeley 1985). There are four parallel versions of the test, and both interrater and testretest reliability are high. Three methods were used to establish the validity of the RBMT (Wilson, Cockburn, Baddeley, & Hiorns 1989), the most important being establishing ecological validity. This was determined by correlating performance on the RBMT with memory failures observed by therapists during patients' daily rehabilitation programs. Therapists observed 80 patients for a mean of 35 hr per patient (range 16-55 hr). The correlation between therapists' observations and RBMT scores was -.75 (p < .001), suggesting that the RBMT is a good and ecologically valid measure of everyday memory performance. The fact that the RBMT has four parallel versions makes it a useful measure for monitoring change over time. Norms exist for all ages from 5 to 96 years. Children between 5 and 10 years of age are given slightly different versions that are more suited to their age (Wilson, Ivani-Chalian, Besag, & Bryant, 1993). Other studies have corroborated the usefulness of the RBMT as an ecologicaly valid test. Wilson (1991a), in a long-term (5-10 years) follow-up study of 54 patients referred for memory therapy between 1979 and 1985, found that the RBMT discriminated between patients who were independent (independence operationally defined as being either in paid employment or in full-time education or living alone) and those who were dependent (Z2 = 24.72, p < .001), whereas the Wechsler Memory Scale--Revised (WMS-R) did not discriminate. Schwartz and Macmillan (1989) found that the RBMT correlated with employment status, whereas a memory questionnaire did not. They concluded that the RBMT was a more objective measure of functional ability. Kotler-Cope (1990) found the WMS-R was not as valid a measure of everyday memory and had less ecological validity than the RBMT. Huppert and Beardsall (1991, in press) have also found the RBMT to be useful for the assessment of early dementia.
211
One of the major advantages of the RBMT is that it has good face validity: It "feels" like a test of memory to the subjects who are being assessed, unlike tests that rely on abstract drawings or pairs of words, regarded by some patients as being remote from everyday life. The Autobiographical Memory Interview Another assessment with good face validity is the Autobiographical Memory Interviews (AMI) (Kopelman, Wilson, & Baddeley, 1989, 1990). Until recently, autobiographical memory had not received a great deal of attention. In 1986, Alan Baddeley and I published a chapter about observations of autobiographical memory in 12 amnesic patients. We noted that some patients appeared to have good access to autobiographical memory at least up until a year or two prior to the onset of the amnesia. When retested a week later, they produced the same details as they had provided originally, even though they had no recall of the earlier assessment. These patients, although severely amnesic, were able to cope with social situations and did not present with management problems for the staff or their relatives. Another subgroup demonstrated "clouded" autobiographical memory in which they were able to recall events on one occasion but not on others. These patients tended to be anxious, frightened, or agitated and created difficulties for those looking after them. A third subgroup provided plenty of autobiographical information but gross confabulation, making errors and inconsistencies and producing bizarre and elaborate stories that were grossly inaccurate. We believed that a measurement of autobiographical memory could be of potential prognostic importance and might help in predicting response to treatment. Our subgroup patients with gross confabulation, for example, were all admitted to long-term psychiatric care on discharge from Rivermead. We were of the opinion that those with clouded autobiographical memory might be agitated or anxious because they had lost a sense of self due possibly to their loss of a remembered past. These remain tentative conclusions. One consequence of the 1986 study was the development of the AMI in collaboration with Michael Kopelman, a neuropsychiatrist working in London. The AMI is a semistructured interview rather than a formal test, although the procedure is standardized and objective scores can be obtained. The AMI has two sections called the personal semantic schedule and the autobiographical incidents schedule. The former assesses subjects' recall of facts from their own past life (e.g., names of school teachers, date and place of wedding); the latter assesses subjects' recall of specific events in their earlier life (e.g., an incident occurring at a subject's first school, first job, or recent holiday).
212
Wilson
Each section assesses memories across three broad chronological bands: childhood, early adult life, and recent past (during the last 12 months). Subjects typically find the test enjoyable and interesting. Is it in any way useful to assess autobiographical memory? After all, most neuropsychologists might argue that they manage without investigating this aspect of memory. We believe such an assessment is valuable for at least three reasons. First, it enables better understanding of the nature of any memory deficit observed--for example, both D.B. and K.J. (reported by Baddeley & Wilson, 1986) had very similar scores on all neuropsychological tests administered but were very different in their ability to access autobiographical memory and also very different in their degree of adaptation to their memory problem. Second, the information gathered by the AMI can help to improve advice and counseling. Third, that same information can provide an individual focus for subsequent management such as reminiscence therapy. The AMI also provides a useful tool for investigating cases in which anterograde and retrograde amnesia may be dissociated and for cases where there may be a dissociation between recall of personal semantic information and recall of personal episodic information for the same time periods. Evans, Wilson, Wraight, and Hodges (in press) reported, for example, the case of a woman assessed while in transient global amnesia. She was able to recall the personal semantic information from all time periods assessed but had very impoverished recall of episodic incidents for all periods. When reassessed 2 days later, she showed normal recall of the episodic incidents. Patients with progressive dysphasia may show the opposite pattern whereby personal episodic information is available for far longer than the personal semantic information (Hodges, personal communication).
The Behavioural Inattention Test The Behavioural Inattention Test (BIT), analagous to the RBMT, is designed to predict everyday problems arising from unilateral visual neglect. Such neglect is a heterogeneous and often transitory phenomenon in which patients fail to report, respond, or orient to stimuli on one side of space. Commonly associated with righthemisphere stroke, it is also seen to a lesser extent after left-hemisphere stroke. Patients with other conditions such as cerebral tumor and head injury may also exhibit unilateral neglect. Several studies have shown tfiat unilateral visual neglect is one of the major factors impeding functional recovery and rehabilitation success (Denes, Semenza, Stoppa, & Lis, 1982; Kinsella & Ford, 1985). Prior to the development of the BIT, there was no single standardized battery for the assessment of unilat-
eral visual neglect. Instead, a variety of procedures were (and still are) used, including the line crossing test (Albert, 1973), indented reading (Caplan, 1987), and position bias on Ravens Coloured Matrices (Costa, Vaughan, Horwitz, & Ritter 1969). Such single tests of neglect are inappropriate because (a) patients often vary in the way neglect presents itself and (b) an individual patient may show great variability in the degree of neglect, depending on such factors as fatigue, position of the tester, and presence or absence of distinctive stimuli in the testing room. For these reasons, a more substantial and varied battery of items is required, and these must allow for testing over periods of time. The immediately aforementioned individual tasks do not map closely on to the everyday demands faced by patients with unilateral visual neglect. Such patients often collide with objects, attend to only one side of the body, ignore food on one side of the plate, and experience difficulty with reading, writing, and drawing. The items in the BIT were selected partly as a result of observing stroke patients who exhibited everyday problems arising from neglect, partly from findings of other studies such as those by Diller and Gordon (1981), and partly from discussions with occupational therapists working with neglect patients. The complete BIT battery (Wilson, Cockburn, & Halligan 1987) comprises nine behavioral subtests that reflect aspects of daily life and six simple paper and pencil measures of neglect. In the former are picture scanning, telephone dialing, reading a menu, reading a newspapertype article, telling and setting the time, coin sorting, address and sentence copying, map navigation, and card sorting. The paper and pencil subtests include line crossing, letter and star cancellation, figure and shape copying, line bisection, and representational drawing. Details of the standardization and validation procedures can be found in Halligan, Cockburn, and Wilson (1991). Like the RBMT, a measure of ecological validity was obtained by using therapists' ratings of problems encountered by our patients: together with scores on a standardized Activities of Daily Living (ADL) scale. On both there were significant correlations suggesting that the BIT is an ecologically valid measure of everyday problems arising from neglect. Another study by Shiel (1990) investigated the relationship between unilateral visual neglect and ADL. She found that patients with neglect, as measured by the BIT, were significantly more likely to have problems with ADL tasks (p < .001) than those without neglect. She also found that (a) the behavioral subtests correlated higher with ADL scores than the paper and pencil subtests; (b) the higher correlations were with indoor mobility, transfer from bed to chair, and using the lavatory; and (c) the results were not due to hemianopia or motor problems.
Ecological Validity of NeuropsychologicalAssessment The Behavioral Assessment of the Dysexecutive Syndrome The Behavioral Assessment of the Dysexecutive Syndrome (BADS) is a test aimed at predicting everyday problems arising from dysexecutive syndrome (DES). Nick Alderman, Paul Burgess, Jon Evans, and I have just completed the pilot stage of this study (Alderman, Evans, Burgess, & Wilson, 1993). DES closely resembles what was once called the frontal lobe syndrome, a term that "is used to refer to an amorphous, varied group of deficits, resulting from diverse etiologies, different locations, and variable extents of abnormalities" (Stuss & Benson, 1984, p. 3). Baddeley and Wilson (1988) claimed that specification of a syndrome in terms of localization is unfortunate and potentially misleading. We do not classify memory, language, reading, or perceptual deficits in this way because it is recognized that to do so would be inadequate. The same inadequacy would seem to apply to frontal lobe damage where a functional definition is more appropriate. Although there is great variability in the extent and degree of cognitive impairment in patients with frontal lobe damage, certain features are highly characteristic. Rylander (1939) described them as "disturbed attention, increased distractability, a difficulty in grasping the whole of a complicated state of a f f a i r s . . , well able to work along routine lines [but] cannot learn to master new types of task" (p. 20). Shallice (1982) believes this pattern of deficits can be described as an impairment in attentional control. He explains this in terms of a model of attention whereby a supervisory system exerts an executive function. Baddeley (1986) believes Shallice's "supervisory system" is analagous to the central executive component of working memory (Baddeley & Hitch, 1974) and suggests the term dysexecutive syndrome as a functional characterization of patients with this particular pattern of deficits. Some of the most widely used neuropsychological tests to assess executive functioning include the WCST, verbal fluency, the Stroop test, and the Reitan Trail Making Test. Yet some patients with obvious impairments of executive functioning can score normally on some or all of these tests. Shallice and Burgess (1991) described three such people in some detail. The patients performed satisfactorily on tests thought sensitive to frontal lobe lesions yet were unable to act effectively on their own initiative as they organised virtually no non-routine activities. Shallice and Burgess made the point that in most neuropsychological tests the patient typically has a single explicit problem to tackle at any one time, the trials tend to be very short (1 minute or so or even less), task initiation is strongly prompted by the examiner and what constitutes suc-
213
cessful trial completion is clearly characterised. (pp. 727-728) Rarely are patients required to organize or plan their behavior over longer time periods or to set priorities in the face of two or more competing tasks. Yet it is these sorts of executive abilities that are a large component of many everyday activities. Shallice and Burgess's solution was to develop the sixelements and the multiple-errands tests, both being ecologically valid and aimed at tapping the more subtle "executive demands." The purpose of the six-elements test was to see how well patients could carry out six open-ended tasks (two sets of three: dictating a route, solving arithmetic problems, and writing down names of pictures) in a 15-min period. Each of the tasks was in itself relatively easy, but two simple rules had to be followed: Subjects were required to attempt (not necessarily complete) some items from each task, and they were not allowed to do two of the same set consecutively. The multiple-errands task was carried out in a pedestrian precinct, previously unknown to the subjects. They had to carry out a number of tasks in which minor, unforeseen events could occur. Errands included buying certain things as well as requiring subjects to be at a certain place at a certain time and to find out certain pieces of information (such as the shop with the most expensive item). Patients with DES both made more errors than controls and showed qualitatively different behavior (such as walking out of a shop without paying). The multipleerrands test has a high ecological validity and captures the nonroutine, problem-solving, planning, organization, and initiative required for everyday functioning. However, it also requires considerable time, planning, and organization on the part of the assessing neuropsychologist and, consequently, is not always practicable in clinical situations. Nick Alderman, Paul Burgess, Jon Evans, and I decided to modify the six-elements test in order to make it simpler for the more severely impaired and less intellectually able brain-injured patients who are more typical of neuropsychological referrals than the patients seen by Shallice and Burgess (1991). We also designed a range of other tests similar to real-life activities that would cause difficulties for some patients with DES. These included (a) a practical problem-solving task, whereby subjects have to extract a cork from a tube with the aid of certain pieces of equipment and by following a prescribed set of rules; (b) a paper and pencil problem-solving task adapted from the plan and search test in the StanfordBinet Test (Terman & Merrill, 1961), whereby the subject has to pretend he or she has lost some keys in a field, and a line is to be drawn to show how the subject would search the field to find the keys; (c) three tasks of increas-
214
Wilson
ing difficulty, demanding attention and the ability to follow rules using playing cards (e.g., say "yes" if the next card is the same color as the previous card, otherwise say "no"; (d) estimating how long it takes to do certain things such as train to be a medical doctor or blow up a party balloon; (e) a test involving the use of a map and planning a route round a zoo, again with certain rules needing to be followed and including two versions varying in level of difficulty; and (f) the simplified six-elements test already described. In addition, a questionnaire about everyday problems is given to each patient and to a carer or relative of the patient. Preliminary results from the pilot study, in which 33 controls and 36 brain-injured people were tested, suggest that (a) the BADS is a better predictor of everyday problems (as measured by the relatives/carers' questionnaire) than the WCST, (b) the modified six elements and the harder version of the zoo map tasks are the most sensitive individual subtests in the battery, (c) using scores from all subtests is best for predicting impaired everyday functioning, and (d) double dissociations are not uncommon. For example, one of our DES patients had no difficulty with the practical problemsolving task but experienced much difficulty with the paper and pencil problem-solving. The results of another patient showed the reverse of this pattern. From these and other examples, it would seem likely that it is better to administer several tasks in order to obtain a true measure of DES. Although we cannot provide a conclusive appraisal of the BADS while research is continuing on its effectiveness, it does hold promise of being a useful assessment battery that provides good prediction of everyday problems arising from impaired executive functioning. Concluding Remarks In 1989 a highly provocative paper was published, entitled "The Bankruptcy of Everyday Memory" (Banaji & Crowder, 1989), in which the authors claimed that the naturalistic study of memory has not been productive. Subsequently, a whole issue of the American Psychologist (1991) was devoted to (mostly very critical) replies to the original article. Ceci and Bronfenbrenner (1991) said, for example, that
by equating mental behaviour with the actions of inner chemical processes, (the authors of the original article) have overlooked a critical difference; namely, living beings are active agents who not only adapt to contexts but also modify, select and create them. (p. 27) A further comment by Morton (1991) argued that "the progress of research in memory is being impeded not by the everyday memory movement, but by an excessively restricted theoretical base" (p. 32). Perhaps the most pertinent message is that from Tulving (1991), who wrote, "there is no reason to believe that there is only one correct way of studying m e m o r y . . . [this] is not a zerosum game in which only one side can win" (p. 41). As argued earlier, we need a complete range of neuropsychological assessments in order to provide the clearest picture o f a brain-injured patient's present neuropsychological condition and possible future progress after returning to everyday life in a community. It is unhelpful for neuropsychologists who ask a particular set of questions to attack other neuropsychologists, asking a different set of questions, because their answers do not seem to apply to the first group. The way forward for the investigator is to be broad minded about testing and recognize that better diagnosis will follow from using tests from a wide range of sources, some of which will be more theoretically based while others will be more practically oriented. Also, as recommended earlier, the tester should be willing to combine information gathered from neuropsychological tests with information gathered from behavioral observations, interviews, rating scales, self-report measures, and other assessment procedures. Doors should not be shut just because a particular investigator has been educated in a particular way. Our education as researchers is never at an end and must involve willingness to open all kinds of doors in the search for a better deal for our patients. Finally, on the specific question concerning ecological validity, I hope I have shown in this article that tests can be designed to map directly on to everyday behaviors and that these tests can provide detailed information that is relevant to the patient, the carer, and the rehabilitation worker in their efforts to achieve the best possible life for the patient returning to the community.
REFERENCES
Albert, M. L. (1973). A simple test of visual neglect. Neurology, 23, 658-664. Alderman, N., Evans, J., Burgess, P., & Wilson, B. A. (1983). Behavioural assessment of the dysexecutive syndrome. Journal of Clinical and Experimental Neuropsychology, 15, 6970. (Abstract) Baddeley, A. D. (1986). Working memory. Oxford: Clarendon Press. Baddeley, A. D., & Hitch, G. (1974). Working l'Aemory. In G.
H. Bower (Ed.), The psychology of learning and motivation (Vol. 8). New York: Academic Press. Baddeley, A. D., & Wilson, B. A. (1986). Amnesia, autobiographical memory and confabulation. In D. Rubin (Ed.), Autobiographical memory (pp. 225-252). Cambridge: Cambridge University Press. Baddeley, A. D., & Wilson, B. A. (1988). Frontal amnesia and the dysexecutive syndrome. Brain and Cognition, 7, 212230.
Ecological Validity o f Neuropsychological Assessment
Banaji, M. R., & Crowder, R. G. (1989). The bankruptcy of everyday memory. American Psychologist, 44, 1185-1193. Caplan, B. (1987). Assessment of unilateral neglect: A new reading test. Journal of Clinical and Experimental Neuropsychology, 9, 359-364. Ceci, S. J., & Bronfenbrenner, U. (1991). On the demise of everyday memory: "The rumors of my death are much exaggerated." American Psychologist, 46(1), 27-31. Costa, L. D., Vaughan, H. G., Horwitz, M., & Ritter, W. (1969). Patterns of behavioural deficit associated with visual spatial neglect. Cortex, 5, 242-263. Denes, F., Semenza, C., Stoppa, E., & Lis, A. (1982). Unilateral spatial neglect and recovery from hemiplegia: A follow up study. Brain, 105, 543-552. Diller, L., & Gordon, W. A. (1981). Rehabilitation and clinical neuropsychology. In S. B. Filskov & T. J. Boll (Eds.), Handbook of clinical neuropsychology (pp. 702-733). New York: Wiley. Evans, J., Wilson, B. A., Wraight, E. P., & Hodges, J. (in press). Journal of Neurology, Neurosurgery and Psychiatry. Halligan, P., Cockburn, J., & Wilson, B. A. (1991). The behavioural assessment of visual neglect. Neuropsychological Rehabilitation, 1, 5-32. Hart, T., & Hayden, M. E. (1986). The ecological validity of neuropsychological assessment and remediation. In B. Uzzell & Y. Gross (Eds.), Clinical neuropsychology of intervention (pp. 21-50). Boston: Martinus Nijhoff. Holland, A. L. (1980). Communicative abilities in daily living. Baltimore, MD: University Park Press. Huppert, F., & Beardsall, L. (1991). A comparison of clinical, psychometric and behavioural memory tests: Findings from a community study of the early detection of dementia. International Journal of Geriatric Psychiatry, 6, 295-306. Huppert, F., & Beardsall, L. (in press). Prospective memory impairment as an early indicator of dementia. Journal of Clinical and Experimental Neuropsychology. Kinsella, G., & Ford, B. (1985). Hemi-inattention and the recovery patterns of stroke patients. International Rehabilitation Medicine, 7, 102-106. Kopelman, M., Wilson, B. A., & Baddeley, A. D. (1989). The autobiographical memory interview: A new assessment of autobiographical and semantic memory in amnesic patients. Journal of Clinical and Experimental Neuropsychology, 11, 724-744. Kopelman, M., Wilson, B. A., & Baddeley, A. D. (1990). The autobiographical memory interview. Flampton Bury St. Edmunds, Suffolk: Thames Valley Test Company. Kotler-Cope, S. (1990). Memory impairment in older adults: The interrelationships between objective and subjective clinical and everyday memory assessment. Paper presented at the annual meeting of the Southern Society for Philosophy and Psychology, Louisville, KY. Morton, J. (1991). The bankruptcy of everyday thinking. American Psychologist, 46(1), 32-33. Newcombe, F. (1987). Psychometric and behavioral evidence: Scope, limitations, and ecological validity. In H. S. Levin, J. Grafman, M. Howard, & H. M. Eisenberg (Eds.), Neurobehavioral recoveryfrom head injury (pp. 129-145). New York: Oxford University Press.
215
Rylander, G. (1939). Personality changes after operations on the frontal lobes. Aeta Psychiatrica et Acta Neurologica Scandinavia, Supplement 20. Schwartz, A. F., & Macmillan, T. (1989). Assessment of everyday memory after severe head injury. Cortex, 25, 665-671. Shallice, T. (1982). Specific impairments of planning. Philosophical Transactions of the Royal Society of London B, 298, 199-209. Shallice, T., & Burgess, P. (1991). Deficits in strategy application following frontal lobe damage in man. Brain, 114, 727741. Shallice, T., & Evans, M. E. (1978). The involvement of the frontal lobes in cognitive estimation. Cortex, 14, 294-303. Shiel, A. (1990). An investigation of the relationship between unilateral neglect and ADL dependency in right hemisphere stroke patients. Unpublished master's thesis, Southampton University Medical School, Southampton. Stuss, D., & Benson, D. F. (1984). Neuropsychological studies of the frontal lobes. Psychological Bulletin, 95, 3-28. Sunderland, A., Harris, J. E., & Baddeley, A. D. (1983). Do laboratory tests predict everyday memory? A neuropsychological study. Journal of Verbal Learning and Verbal Behavior, 22, 341-357. Terman, L. M., & Merrill, M. A. (1961). StanfordBinet Intelligence Scale. London: Harrap & Company, Ltd. Tulving, E. (1991). Memory research is not a zero-sum game. American Psychologist, 46(1), 41-42. Wang, P. L., & Ennis, K. E. (1986). Competency assessment in clinical populations: An introduction to the cognitive competency test. In B. P. Uzzell & Y. Gross (Eds.), Clinical neuropsychology of intervention. Boston: Martinus Nijhoff. Wechsler, D. (1945). A standardised memory scale for clinical use. Journal of Psychology, 19, 87-95. Wilson, B. A. (1991a). Long term prognosis of patients with severe memory disorders. Neuropsychological Rehabilitation, 1, 117-134. Wilson, B. A. (1991b). Theory, assessment and treatment in neuropsychological rehabilitation. Neuropsychology, 5, 281-291. Wilson, B., Cockburn, J., & Baddeley, A. D. (1985). The Rivermead Behavioural Memory Test manual. Flempton, Bury St Edmunds, Suffolk: Thames Valley Test Company. Wilson, B. A., Cockburn, J., & Baddeley, A. D. (1989). Assessment of everyday memory following brain injury. In M. E. Miner & K. A. Wagner (Eds.), Neurotrauma: Treatment, rehabilitation and related issues (No. 3, pp. 83-99). Boston: Butterworths. Wilson, B. A., Cockburn, J., Baddeley, A. D., & Hiorns, R. (t 989). The development and validation of a test battery for detecting and monitoring everyday memory problems. Journal of Clinical and Experimental Neuropsychology, 11, 855870. Wilson, B. A., Cockburn, J., & Halligan, P. (1987). The Behavioural Inattention Test. Flempton, Bury St Edmunds, Suffolk: Thames Valley Test Company. Wilson, B. A., Ivani-Chalian, R., Besag, F., & Bryant, T. (1991). Modification of the Rivermead Behavioural Memory Test for use with 5-10 year old children. Journal of Clinical and Experimental Psychology, 15, 474-486.