Validation of a neurobehavioral functioning inventory for adults with traumatic brain injury

116

Validation of a Neurobehavioral Functioning Adults With Traumatic Brain Injury

Inventory

for

Jeffrey S. Kreutzer, PhD, Jennifer H. Marwitz, MA, Ron Seel, BA, Catherine Devany Serio, PhD ABSTRACT. Kreutzer JS, Marwitz JH, See1R, Serio CD. Validation of a neurobehavioral functioning inventory for adults with traumatic brain injury. Arch Phys Med Rehabil 1996; 77:116-24. Objective: To examine the validity of a neurobehavioral inventory for adults with traumatic brain injury. Design: Construct validity was examined by applying principal components and confirmatory factor analytic procedures. Internal consistency of factors was examined using Cronbach’s alpha. Criterion-related validity was examined by comparing inventory scale scores with neuropsychological and personality test variables. Setting: Medical center outpatient clinic Participants: 520 consecutive adult patients with traumatic brain injury were administered psychological and neuropsychological tests; 520 informants, primarily family members, completed the neurobehavioral inventory. Main Outcome Measure: Neurobehavioral inventory with items describing symptoms and daily living problems; scale scores were compared with a set of neuropsychological measures and Minnesota Multiphasic Personality Inventory scores. Results: Principal components and confirmatory factor analytic procedures identified six scales with a total of 70 items. The primary scaleinclusion criterion required a minimum factor loading of .40. Chronbach’s alpha analysis revealed acceptably high internal reliability for all scales ranging from 36 to .95. Scale scores were statistically compared with patients’ scores on neuropsychological and personality tests. Poor neuropsychological test performance and greater levels of psychopathology were associated with greater frequency of perceived neurobehavioral problems. Conclusions: Findings suggest that the neurobehavioral inventory is a promising means of investigating informants’ perceptions of outpatients’ everyday problems. Such information can serve asa complement to historical information, test results, and information from standardized interviews to develop a holistic perspective of patients. Differential weighting of items, association of scales with other measures, and validation with other neurological patient populations are recommended avenues for future research. 0 1996 by the American Congress of Rehabilitation Medicine and the American Academy of Physical Medicine and Rehabilitation From the Department of Physical Medicine and Rehabilitation, Medical College of Virginia, Richmond. Submitted for publication April 18, 1995. Accepted in revised form October 20, 1995. Supponed in part by grant HI33880029 from the National Institute on Disability and Rehabilitation Research, United States Department of Education. No commercial party having a direct financial interest in the results of the research supporting this article has or will confer a benefit upon the authors or upon any organization with which the authors are associated. Reprint requests to Jeffrey S. Kreutzer, PhD, Department of Physical Medicine and Rehabilitation, Medical College of Virginia, MCV Box 980542, Richmond, Virginia 23298.0542. 0 I996 by the American Congress of Rehabilitation Medicine and the American Academy of Physical Medicine and Rehabilitation OOW9993/96/7702-3488$3.00/O

Arch

Phys Med Rehabil

Vol77,

February

1666

HERE IS ever-increasing evidence that severe traumatic T brain injury often results in long-term psychosocial difficulties.le6High rates of unemployment, family disturbance, and dependence on others for assistancewith daily living activities have routinely been reported by researchersinvestigating outcomes beyond the first few years after injury.‘-” Poor long-term outcomes have been attributed to a broad category of problems often termed “neurobehavioral.“3.“.‘2 Specific neurobehavioral problems often relate to adverse changes in physical well being, personality, mood, initiative, social skills, and cognitive abilities.2.4.6.13.14

Early postinjury, physical problems are the focus of rehabilitation as primary obstacles to independent living. Fortunately, many physical problems arising are nearly resolved in the later stages of injury, especially beyond 2 years.7,‘5Behavioral and psychosocial problems appear to be more persistent, and are often cited as primary obstacles to social integration later postinjury. Professionals have struggled to develop reliable and valid methods to describe neurobehavioral functioning.‘6-‘9 Early measurement attempts relied upon semi-structured interviews, checklists, and questionnaires.20-22Information about patients’ day-to-day problems was typically solicited from family members and other knowledgeable informants.2’~2’~24 Reliance is now placed increasingly on standardized tests including neuropsychological, functional status, and communication measures.25-27 Levin and colleagues*’ developed the Neurobehavioral Rating Scale, based on the Brief Psychiatric Rating Scale.” The scale, standardized on acutely injured populations, provides a means for clinicians to measure behavior based on a brief structured assessment. Unfortunately, current brain injury assessmentprotocols have received widespread criticism.‘7,29~”The questionnaires, checklists, and interviews used early on were not subjected to rigorous psychometric evaluation,‘2~‘7and their reliability and validity remain uncertain. The validity of standardized measures, including the Neurobehavioral Rating Scale,*’ has been questioned. There is concern that behavior, skills, and symptoms measured in a structured clinical setting do not necessarily correspond to what would be observed in natural world settings.4.24.34 Several researchers have suggested that skills required to perform well on tests are distinct from skills required for daily living. Also, many clients can suppress undesirable behaviors when clinicians are present. For example, patients intent on creating a favorable impression may demonstrate a meek demeanor before an examiner, but show serious temper control problems with their families at home.6.3s-‘8 An accurate impression of brain injury consequencesrequires blending information from several assessmentmethods.39Ideally, data from tests, interviews, and observations is combined to yield reliable and valid outcome information.‘2.40.4’Much has been done to improve the psychometric properties of standardized tests and clinical observation, but relatively little progress has been made in collecting information from relatives and other informants. The reports of relatives may be biased by their emotional status, the severity of the patient’s problems,

VALIDATION

OF A NEUROBEHAVIORAL

and their familiarity with the patient. Relatives, however, often have a unique opportunity to observe day-to-day behavior and symptoms in natural settings.“.“-“’ This report describes efforts to validate the Ncurobehavioral Functioning Inventory (NFI), which was dcvclopcd to improve on the methodologies widely used by European researchers who routinely collect information about patients’ postinjury behavior and symptoms from relatives. The inventory, which developed from interests in ecological and content validity, describes a wide spectrum of postinjury behaviors and symptoms commonly encountered in daily life. For the investigation. factor analytic techniques were conducted to develop unidimcnsional scales reflecting major content domains. Criterion-related validity was supported by correlational analyses relating inventory responses to ncuropsychological and personality measures of patient status.

METHOD Participants The original sample consisted of 581 informants and 58 I patients referred for outpatient services at a regional Level I trauma center. Patients were referred for comprehensive outpatient assessment primarily by physicians and other rehabilitation professionals. A small number of referrals were made by community mental health agencies and attorneys. Some were selfreferred. Neurobehavioral data regarding patients’ functioning were primarily obtained from parents (34%) and spouses (34%). A smaller proportion of data was obtained from boy/girlfriends (IO%), siblings (7%). children (6%), friends (4%). and others (5%). These informants were selected by the patients who were asked to identify a single individual who was either a primary caretaker or who knew them well. The neurobchavioral inventory was mailed to informants in advance of the patient’s neuropsychological evaluation and responses were reviewed for compliance with instructions when the patient came for the evaluation. Forms were corrected or missing information was obtained by interviewing the informant in person or by phone. In cases where patients were reevaluated, informants completed the NFI again. To avoid bias or rcdundancy, only one protocol was selected and analyzed for each patient and informant. Where the NFI was completed more than once, the questionnaire with the least number of omitted responses was chosen. For the many cases where inventories had an identical number of completed items, a second criteria was applied. The most recent questionnaire was selected for analysis to enhance variability of time postinjury data. Several criteria were established to help eliminate data from unreliable informants. Sixty-one patients were eliminated from the study for the following reasons: 48 because 10% or more of their item responses were missing and/or marked as “does not apply”; 12 because of a 95% or more response bias (eg. 95% of items marked in the “never occurs” category); and I was eliminated after reporting that responses were “not at all accurate.” The final sample consisted of 520 patients and informants. The following demographic and medical information for patients was identified. Two thirds were male, 77% were Caucasian, 2 I % were African-American, and 2% were of other ethnic backgrounds. Eighty percent of patients were involved in vehicular accidents. Patients’ mean age at the time of injury was 3 I .3 years (SD = 12.7). The mean length of time in coma was 14.6 days (SD = 33.1), and the median length of time in coma was .25 days. Average time postinjury for evaluation was 2.9 years (SD = 4.1) with a median value of 1.3 years.

INVENlORV,

117

Kreutzer

Measures Neuropsychological and psychological evaluations were administered to patients. Informants completed the Neurobehavioral Functioning Inventory. Measures are summarized according to the broad areas assessed. Neurobehaviaral. The Neurobrhavioral Functioning In~cntory (NFI) is contained within a larger questionnaire. the General Health and History Questionnaire (GHHQ).“’ The GHHQ solicits information on prcinjury and postinjury demographics, work history, substance abuse, and criminal history. The NFI is composed of 105 items grouped into 5 categories: somatic, cognitive, behavior, communication, and social problems. To maintain content and face validity, checklist items were originally formulated and compiled from interviews with patients with brain injury and family members, and from thorough reviews of the brain injury litcrature.h.J7 Frequency of problem occurrence is rated on a Cpoint scale of never(l), sometimes (2), o&n (3) or alw*ay.r (4). For a subset of items, readers also have an opportunity to rate problems as not applicable. Neuropsychological. Neuropsychological tests were used to provide information about the diverse cognitive, perceptual, and motor sequclac of generalized brain dysfunction. Test inclusion in the evaluation protocol was based on evidence of reliability and validity, as well as the availability of normative data for persons with brain injury. Also, a number of the measures have been used effectively in multicenter outcome invcstigations.” To organize tests for statistical analysis, measures were assigned to one of three groups: (I) attention, memory, and learning; (2) language and communication; and (3) visuopcrception and motor functioning. Included were measures from the Rcy Auditory Verbal Learning Test, Wechslcr Memory Scale, Gray Oral Reading Test, Controlled Oral Word Association Test, Symbol Digit Modalities Test, and the Grooved Pegboard. Detailed descriptions of instruments and psychometric propertics arc provided in Lezak” and Spreen and Strauss.” Personality and Behavior Assessment. The MMPI is perhaps the most widely used behavior and personality measure described in the general psychology and psychiatric literature. Research reports have provided evidence of diagnostic utility for evaluation of persons with traumatic brain injury.‘.‘” Five of the ten MMPI clinical scales, Hypochondriasis (scale I), Depression (scale 2), Hysteria (scale 3), Psychasthenia (scale 7). and Schizophrenia (scale 8), were chosen for study. These scales have demonstrated sensitivity to the unique effects 01 brain injury.J7~Jx In addition, content areas addressed by the scales arc thematically related to item content on the Neurobchavioral Functioning Inventory.

STATISTICAL

PROCEDURES AND RESULTS

This investigation consisted of 2 statistical studies: (I) factor analytic investigation; and (2) investigation of criterion-related validity. Statistical procedures and corresponding results for each study are presented below. Study I: Factor Analytic Procedures A “model generating” confirmatory factor analytic procedure was utilized. This procedure enables a priori specification of a theoretically derived model consisting of hypothesized factors and corresponding items.‘” Relations between variables in the model can be “tixed” (set to zero) or “free” (calculated by an algorithm). The free option allows the investigator to specify: (1) intercorrelations among latent factors; and (2) correlations between observed variables and latent factors. The model is then examined for “goodness of lit” to the data. If the model

Arch

Phys

Med Rehabil

Vol77,

February

1996

118

VALIDATION

OF A NEUROBEHAVlORAL

fails to meet criteria, a new theory or data driven model may be generated and evaluated. Three goodness-of-fit methods were used to test the models. Criteria for each method was established a priori based on established theoretical guidelines.49-s5 First, t tests were used to determine whether each variable loaded on the predicted factor as measured by a significant t value with p < .05. Second, a chisquare value was calculated, testing the null hypothesis that the model inadequately characterizes the population. A 2:1 chisquare to degrees of freedom ratio for adequacy of fit was used to compensate for the relatively large sample size.49.51.55 Last, a Comparative Fit Index (CFI) was used to evaluate the theoretically derived model. CFI values of .90 or greater are considered indicative of a good model fit to the data with values close to 1.O considered excellent.4y-55 Four confirmatory factor models were developed. First, a confirmatory factor analysis was used to evaluate the original grouping of 105 NFI items into 5 categories. Results of the analysis indicated that the original model inadequately characterized the data. Second, principal components factor analytic procedures were employed, yielding potentially valuable fiveand six-factor solutions. Third, confirmatory factor analyses were conducted suggesting that the six-factor solution was superior. Fourth, minor respecifications were made to the six-factor structure and a final confirmatory factor analysis was conducted. Following are detailed descriptions of the four models. I. Conjirmatory Analysis of Original NFI Groupings. A confirmatory factor analysis was conducted on the original 105 item, five-scale Neurobehavioral Functioning Inventory. Factors 1 and 2 consisted of 30 items representing somatic and cognitive difficulties, respectively. Factor 3 contained 29 items describing behavioral concerns. Factor 4 was comprised of 10 items representing communication deficits, and Factor 5 contained 6 items relevant to social functioning. All 105 items in the model were set free. The latent factor variances were set to 1.0 to eliminate scale indeterminacy.5’-55Considering the diffuse nature of brain injury and interrelatedness of postinjury problems, the latent factor covariances were set free. Last, error variances for each item were set free while error covariances were set to zero. ZZ. Generated Six-Factor Model. A confirmatory factor analysiswas conducted on a 70-item, six-scaleNFI model specified as follows: Factor 1 consisted of 13 items representing symptoms of Depression; Factor 2 represented 11 items expressing Somatic complaints; Factor 3 was comprised of 19 items related to Memory/Attention difficulties; Factor 4 contained 10 items describing Communication deficits; Factor 5 contained 9 items endorsing Aggressive behaviors; and Factor 6 contained 8 items that pertained to Motor impairment. All 70 items in the model were set free and the latent factor variances were set to 1.0 to eliminate scale indeterminacy. The latent factor covariantes were set free, error variances for each item were set free, and the error covariances were set to zero. Table Statistical

1: Goodness-of-Fit

Criteria for goodness of fit: (1) zero variables Comparative Fit Index (CFI) > .90.

Phys Med Rehabil

Vol77,

February

1996

loading

on a factor

Study I: Factor Analytic Results Results of the confirmatory factor analysis on the original 105-item, five-factor NFI are presented in table I. A review of goodness of fit criteria indicated that the model failed to meet any of the three guidelines. One of the five scales had an item with a nonsignificant f value (p > .05). The chi-square to degrees of freedom ratio exceeded the criterion value of 2.0. Also, the Comparative Fit Index (CFI) was considerably below the .90 criterion. For these reasons,the 105-item model was rejected as an adequate characterization of the data. With rejection of the 105-item model, a principal components54.5h factor analytic procedure was conducted to help identify an acceptable alternative. An orthogonal rotation and scree plot analysis of eigenvalueP revealed that the optimal solution would involve between five and seven factors. To justify inclusion of items on individual scales,a two-step decision rule was established.56First, only items with a factor loading of .40 or greater would be retained. A second rule was established for items loading on more than one scale. The item was retained on the scale if the factor loading was at least .lO greater than the loading on any other scales. An examination of the seven-factor solution revealed a seventh factor with an unacceptably small (3) number of items. An examination of the item content of the five- and six-factor solutions revealed two models that were theoretically and clinically viable. A confirmatory factor analysis was then conducted on each model to compare adequacy of fit to the data. Results of the confirmatory factor analyses on the five- and six-factor models are presented in table 1. The two models were evaluated according to the goodness of fit criteria. Both models met the criterion requiring that all items load significantly on the predicted factor with t values of p 5 .05. A review of the criteria for the chi-square to degrees of freedom ratio and the CFI suggested that the six-factor model was superior. The six-factor solution was comprised of 70 items, and the factor loadings of each item are presented in table 2. Among the original 105 items, 35 did not meet one or both of the selection criteria. A listing of excluded items is presented in Confirmatory

Generated Factor

1 14158 5345 2.65 NA NA NA .72

Kreutzer

ZZZ.Generated Five-Factor Model. A confirmatory factor analysis was conducted on a 72-item, five-scale NFI model. Factors I, 4, and 5 exactly replicated the Depression, Communication, and Aggression scales specified in the six-factor structure. Factor 3, the Memory/Attention scale, contained one additional item compared to the six-factor structure. Factors 2 and 6 from the six-factor structure collapsed into a single Sensorimotor scale (Factor 2) with 20 items, 7 of which differed from the content of the six-factor model. All 72 items in the model were set free and the latent factor variances were set to 1.0. The latent factor covariances and error variances for each item were set free and the error covariances were set to zero.

for the Four

105.Item 5 Scale

Analyses

Nonsignificant t values Chi square (x2) Degrees of freedom (dfl Chi-squareldf Satorra-Bentler x2 Satorra-Bentler df Satorra-Bentler $/df Comparative Fit Index (CFI)

Arch

Indices

INVENTORY,

Factor

5

0 6453 2474 2.61 NA NA NA 82 with

t values

of p > .05; (2) chi-square

Models Generated Factor

6

Respecified Factor

0 5790 2330 2.48 NA NA NA 84 to degrees

6

0 NA NA NA 4156 2320 1.79 39 of freedom

ratio

< 2.0; and (3)

VALIDATION

OF A NEUROBEHAVIOBAL

Table Items

Bored Difficulty enjoying activities Feels hopeless Feels worthless Frustrated Lonely No confidence Restless Sad, blue Scared or frightened Sits with nothing to do Can’t get mind off certain thoughts Uncomfortable around others Dizzy Food doesn’t taste right Headaches Nauseous Nightmares Poor appetite Ringing in ears Stomach bloated Stomach hurts Trouble falling asleep Trouble hearing Concentration poor Confused Easily distracted Forgets or misses appointments Forgets peoples names Forgets phone numbers Forgets to do chores or work Forgets to take medication Forgets yesterday’s events Forgets what he or she reads Late for appointments Loses track of time, day, or date Loses way, gets lost Misplaces things Trouble following instructions Loses train of thought Forgets if he or she has done things Forgets to turn off appliances Disorganized Reads slowly Difficulty thinking of the right word Difficulty pronouncing words Difficulty making conversation Makes spellrng mistakes Speech doesn’t make sense Talks too fast or slow Trouble understanding conversation Writes slowly Writing is hard to read Hits or pushes others Inappropriate comments or behavior Screams or yells Threatens to hurt others Breaks or throws things Curses at others Curses at self Argues Rude to others Difficulty lifting heavy objects Loses balance Moves slowly Muscles tingle or twitch Drops things Trips over things Weak Difficulty performing chores l

Indicates

the item

met the requirements

Factor 1: Depression

Factor 2: Somatic

,519. ,497' 686’ ,683’ .581 l .615’ .615* .470* .674* .543* ,551 l .477* ,487, ,109 ,173 ,078 ,139 ,217 .199 ,039 ,083 ,117 ,300 .016 ,316 ,254 ,255 ,152 ,080 .lOO ,197 ,177 ,112 ,220 ,191 ,143 ,101 ,174 .346 .314 .206 ,151 ,358 ,192 ,190 ,036 ,318 .159 ,076 .105 ,214 ,178 ,221 ,085 ,323 ,252 ,138 ,145 ,095 ,239 ,246 ,264 ,055 ,106 .163 .114 ,054 .124 .148 .294 for loading

2: Factor

-.003 ,191 .193 .167 .143 ,091 ,173 .272 ,239 .362 ,069 .266 ,368 ,549’ ,403. ,564’ ,572’ ,548’ ,434’ ,520” .549* ,604’ ,433’ .409’ ,228 .298 ,107 ,142 ,168 ,222 ,130 ,256 ,196 ,152 ,194 ,197 ,142 ,220 ,021 ,176 ,216 .179 ,154 607 .231 .211 ,176 .183 ,210 .027 ,265 -.130 - .069 ,091 ,085 ,212 ,078 ,146 ,116 ,193 ,163 .149 ,166 ,201 .077 .411 ,261 ,292 .389 ,137 under

Loadings

INVENTORY,

for 70 Item

NFI

Factor 3: Memory/ Attention

.126 ,224 ,181 ,188 ,258 ,119 ,241 .259 .105 .191 ,146 ,292 ,130 ,197 ,137 ,177 ,199 ,153 666 ,110 .089 .063 ,106 ,235 ,563’ .527* .525* ,628. .617* 554” .592* .481 l 588’

.613* .452* .607* .548* ,586’ .500* .561* .654* ,500’ ,477s .272 ,349 .142 .120 .187 ,163 ,185 ,259 ,078 ,062 ,076 .204 ,141 ,106 ,105 ,146 ,134 ,148 .153 ,116 ,172 ,195 ,046 ,255 ,274 ,160 ,361

119

Kreutzer

Factor 4:

Communication

Factor 5:

Factor 6:

Aggression

Motor

,273 ,216 ,208 ,198 ,300 ,112 ,157 ,323 .263 .129 ,151 ,249 ,144 ,102 ,166 ,060 .099 ,130 ,146 ,011 ,100 ,162 ,104 .085 ,152 ,176 ,276 ,178 ,061 667 ,182 ,181 ,127 ,072 ,226 ,147 ,056 ,222 ,203 ,169 ,153 ,132 ,217 ,070 ,114 ,074 ,090 ,106 ,284 .172 ,200 ,057 ,167 .627* ,620’ ,615’ ,560” .618’ ,662” .482* .624* ,613’ ,002 ,171 ,047 ,085 ,200 ,208 ,080 ,156

,187 ,199 ,134 ,188 ,128 ,278 ,218 ,197 ,158 ,119 .237 ,164 ,317 .079 ,035 ,086 ,019 ,132 -.035 ,144 ,142 ,146 ,106 ,152 ,196 ,227 .251 .085 .301 ,275 ,129 ,171 .189 .306 ,019 ,238 ,218 .206 .385 ,305 ,210 ,185 .213 ,509’ ,545’ ,658. ,541” ,555’ ,527” ,506’ .562* ,583” ,510’ ,141 ,138 ,069 ,163 ,087 .113 ,112 ,182 ,200 ,062 .224 ,214 ,144 ,168 ,236 ,116 .123

,115 ,322 ,187 ,134 ,250 ,080 ,130 ,123 .132 ,087 ,146 ,177 .035 ,323 ,127 ,304 ,273 ,148 ,139 .232 ,165 ,187 ,250 ,254 ,280 ,234 ,236 ,185 ,125 ,192 .204 ,138 ,156 ,172 ,144 .180 ,112 ,267 ,224 ,279 ,193 ,145 ,278 ,290 .114 ,128 ,075 ,145 -.004 ,215 ,117 .336 ,227 ,012 .124 ,169 ,064 ,140 ,125 ,129 ,099 ,096 .611* .589* ,607. ,516’ ,520. .463* .575* ,494’

the factor

Arch

Phys Med

Rehabil

Vol77,

February

1996

120

VALIDATION

Table

3: Rejected

Items

From

Exploratory

Items

OF A NEUROBEHAVIORAL

Factor

loading > .40 on a factor but c.10 independence factors Muscles ache (Motor/Somatic) Muscles numb (Somatic/Motor) Tired (Motor/Somatic) Drives dangerously (Memory/Motor) Learns slowly (Memory/Communication) Thinks slowly (Memory/Communication) Trouble making decisions (Memory/Depression) Complains (Aggression/Depression) Hard to get started on things (Depression/Memory) Impatient (Aggression/Depression) Jumpy, irritable (Aggression/Depression) Nervous (Depression/Somatic) Items loading < .40 on all factors Blackout spells Difficulty smelling things Eats too much Picks nose or skin Seizures Trouble staying awake Trouble waking up Cries Vision blurred Difficulty handling money Laughs for no reason Difficulty attending work or school Forgets to eat Misunderstood by others Friends or relatives are unfamiliar Makes mistakes doing arithmetic Difficulty understanding jokes Double vision Threatens to hurt self Repeats what others say Avoids family members Avoids friends Doesn’t participate in sports

Analysis from

other

table 3. Twelve of the 35 items met the .40 loading criterion but also loaded within .I0 of another factor. The remaining 23 items did not load at .40 on any of the six factors. Among the three factor models tested, the six-factor NFI model was superior and most closely met the goodness of fit criteria. The chi-square to degrees of freedom ratio and the CFI approached but did not meet the minimum guidelines for a good fitting model. To further assesses the model’s goodness of fit, additional data were analyzed to determine: (1) whether relationships between items had been misspecified and (2) whether underlying assumptions regarding the population distribution had been violated. To identify misspecification of item relationships, a multivariate Lagrange Multiplier Test (LM Test) was conducted.4”.5’.54 The test projects the chi-square value reduction if an item or factor relationship originally set to zero was changed and set free. LM Test results revealed significant error covariance between many pairs of items. To address high levels of item error covariance, criteria were established to identify error covariance for pairs of items that Table

4: Descriptive Instrument

Factor/Scale

Depression Somatic Memory/Attention Communication Aggression Motor

Arch

Phys Med Rehabil

Vol77,

Statistics

INVENTORY,

Kreutzer

could be respecified (set free) without “overfitting” the model to the data. Based on Byrne’s recommendations” that respecifications should be guided by theoretical knowledge and judicious use of statistical information, the following criteria were established: (1) item pairs should have relatively large chisquare values; (2) item pairs with error covariance should load under the same factor and have similar content; and (3) the number of respecifications should not exceed .005% of the possible respecifications to avoid “overfitting” the model. Ten NFI item pairs with error covariance were chosen to be set free in the respecified model. All item pairs had chi-square values greater than 50, loaded under the same factor, and had similar content. Examples of respecified item pairs include: “feels hopeless-feels worthless “; “forgets people’s names-fotgets people’s phone numbers”; “writes slowly-writing is hard to read”; and, “forgets or misses appointments-late for appointments.” The 10 respecifications represented less than .002% of the possible respecifications for the model. Next, analyses were conducted to test whether the sample population was normally distributed. An examination of significant, non-zero univariate sample statistics indicated that five items showed unsatisfactory levels (z score > 2.0) of kurtosis.5” A normalized estimate of multivariate kurtosis was also conducted. Bentle?‘54 has stated that large values of the normalized estimate z. score are a reflection of significant positive kurtosis. The normalized estimated z score for out sample was 92.8, indicating high non-normality in the samples’ NFI responses. To accommodate for high non-normality arising from kurtosis, Satorra and Bentler developed a modified chi-square statistic.” The Satorra-Bentler Scaled Statistic has been recommended for evaluating confirmatory factor models with nonnormally distributed populations.” Analyses of the respecified model was conducted using the Satorra-Bentler chi-square statistic.54 The result of the confirmatory factor analysis on the respecilied six-factor model was highly favorable, indicating the overall superiority of the respecified six-factor solution (table 1). The model met the criteria requiring that all items load significantly on the predicted factor with t values of p 5 .05 and a chi-square to degrees of freedom ratio less than 2.0. The CFI value of .89 closely approximated the minimum criterion of .90 for goodness of fit.

Study II. Scale Validation Procedures A series of analyses were conducted to further evaluate the integrity of the final six-factor solution. For each NFI scale, a total score was calculated based on the sum of ratings for all scale items. Descriptive statistics for the NFI scales were computed along with scale intercorrelations and internal reliability coefficients. Finally, bivariate correlations were calculated between the NFI and neuropsychological tests, and the NFI and MMPI scales to evaluate criterion-related validity. Patients in the present study were the 520 participants from the factor analytic study. Because of differing protocols, a subset for 70-Item,

Six-Factor

NFI

Data

Sample Data

No. of Items

Min. Score

Max. Score

MhWl

SD

Range

Skew

13 11 19 10 9 8

13 11 19 10 9 8

52 44 76 40 36 32

25.16 18.46 36.16 17.95 13.70 15.67

8.17 5.90 11.35 5.79 4.50 5.15

13-51 11-38 19-74 IO-40 9-35 8-30

.69 .74 .60 .76 1.22 .46

February

1996

VALlDATlON

OF A NEUROBEHAVIORAL

Table Aggresston

Aggression Depression Memory/Attention Communication Motor Somatic All correlations

1 .oo .62 .55 .48 .44 .45 significant

5: NFI Intercorrelations

Depression

1 .oo .67 60 .54 .56

Six Factors Commun

1.oo .65 .64 .58

cation

1.00 .53 .45

Motor

Somatic

1.oo .63

1.oo

at p c.: ,001.

Study II. Scale Validation Results Descriptive data for the six-factor NFI is presented in table 4. To further examine the properties of scales generated by the six-factor solution, scale intercorrelations and internal reliability were examined. Consistent with expectations, total scores for the six scales were related a$ indicated by Pearson correlations ranging from .44 to .67 (Table 5). To examine the reliability of the six-factor solution, an internal consistency analysis was conducted using Cronbach’s alpha.sx Scale reliability coefficients were high, providing evidence of internal consistency within factors. Alpha coefficients for each scale were: Depression (.93); Somatic (X6); Memory/ Attention (.95); Communication (XX); Aggression (.89); and Motor (.87). The coefficient alpha representing the internal consistency for the entire 70-item scale was .97. The first set of correlations involved measures of attention, Table

6: Summary

Attention, Memory & Learning Instruments Rey Auditory Verbal Learning (Sum Number Correct of 5 Trials) MCV Paragraph Immediate Recall (Number Correct) MCV Paragraph Delayed Recall (Number Correct) Weschler Memory Immediate Recall (Number Correct) Weschler Memory Delayed Recall (Number Correct) Visual & Motor Functioning Instruments Symbol Digit Modalities-Oral (Number Correct) Grooved Pegboard (Completion Time in Seconds) Language & Communication Functioning Instrument Gray Oral Reading Test-Comprehensive (Scaled Score by Age) Controlled Oral Word Association (Number Correct) Gray Oral Reading Test-Accuracy (Scaled Score by Age) with

Among

121

Kreutzer

Memory/Attention

of patients were not administered all neuropsychological and the MMPI personality scales. Some patients were unable 10 complete the neuropsychological tests because of severe disability. Last. MMPI data from two patients was excluded because of probable profile invalidity (t value for F scale > I 16). (The number of patients completing each test is provided in tables 6 and 7.) Correlations were calculated using the pairwise deletion method.” Nonmissing values for each pair of variables were correlated, regardless of whether other variables for the patient were missing. As a result, the number of participants for each correlation differed based on the number of valid variable pairs. Four sets of Pearson correlations were performed relating the six NFI scales and measures of: (I) attention, memory and learning; (2) language and communication; (3) visuoperception and motor skills; and (4) personality and psychopathology. To increase protection against Type I error arising from the use of multiple correlation analyses, a modified Bonfcrroni procedure was employed.‘x Application of the procedure revealed that a correlation value significant at p < .OOl was appropriate.

All correlations

INVENTORY,

Communication

Scale

significant

memory, and learning. Descriptive data for the neuropsychological assessments are presented in table 6. The NFI Communication Scale correlated significantly @ < HI) with scores on all five mcasurcs of attention, memory, and learning (table 6). Pearson r values ranged from -. I8 to -.34 with all correlations in the expected direction. More frequent communication difficulties were associated with less adequate test performance. The NFI Memory/Attention Scale correlated only with performance on the Symbol Digit Modalities Test with a Pearson r value of -. 18. No other NFI Scales correlated significantly with measures of attention, learning, and memory. The second set of correlations examined relationships between NFI scale scores and neuropsychological measures 01 language and communication. Pearson r values for the NFI Communication scale ranged from -.20 to -.26 and all were in the expcctcd direction. Persons who performed poorly on tests of reading accuracy, reading comprehension, and verbal fluency were also rated by informants as having more communication problems. No other relationships were found with the remaining five NFI scales. The third set of correlations involved visual and motor functioning. NFI Communication Scale scores correlated signilicantly with scores for the Symbol Digit Modalities Test Oral (r = -.3l) and with Grooved Pegboard performance (r = .20). NFI Memory/Attention Scale scores also correlated significantly with scores for the Symbol Digit Modalities Test Oral (r = -.18). Significance levels were p < ,001 and all correlations were in the expected direction. Poor performance on tasks involving visuoperception, vigilance, and attention was associated with more frequent communication and memory difficulties. No other NFI Scales correlated significantly with measures of visual and motor functioning. The fourth and final set of correlations considered personality and psychopathology variables measured by the MMPI. Descriptive data for the live selected MMPI scales is presented in table 7. MMPI scale scores and NFI scores were related as indicated by the Pearson r values reported in table 8. The pattern of correlations revealed that patients reporting more psychologi-

of Neuropsychological

Measures

Meall

SD

Range

N

rCOMM

44.05 6.72 7.67 21.55 17.30

12.60 3.02 3.98 7.92 9.52

9-71 O-15 O-16 5-42 o-43

445 351 350 96 96

-.18 -.24 -.21 .34 -. .32

45.48 91.10

16.85 44.82

l-95 45-300

427 428

.31 .20

9.48 33.91 8.43

4.20 11.21 4.49

I-18 7-78 l-18

309 444 307

.20 -.21 -.26

at p c.. ,001.

Arch

Phys Mad Rehabil

Vol77,

February

1996

122

VALIDATION

OF A NEUROBEHAVIORAL

INVENTORY,

Kreutzer

as having more speech, reading, writing, and spelling difficulties. As expected, persons who performed more poorly on tests Scale Meall SD Range N of reading accuracy, reading comprehension, and verbal fluency 21-115 252 MMPI 1 68.94 15.26 were rated as having more communication difficulties. 73.15 14.55 45-120 252 MMPI 2 Scoreson the Memory/Attention scalewere related to perforMMPI 3 67.92 12.28 35-98 252 MMPI 7 65.31 14.01 28-103 252 mance on the Symbol Digit Modalities Test. Research has 36-120 252 MMPI 8 72.40 16.76 shown the test to be a highly sensitive indicator of brain dysfunction.44.45Scores on the Wechsler Memory Immediate (r = -.25) and Delayed Memory (r = -.23) tests were correlated with Memory/Attention scalescores.However, values exceeded cal problems were more frequently described by informants as the stringent p < .OOl criteria used to control for Type I error. having neurobehavioral problems. MMPI Scale 1 scores were There was no indication that informants’ perceptions of pamost related to NFI scores on the Somatic (r = .65) and Motor tients’ depression, aggression, motor, and somatic difficulties (r = 53) scales. MMPI Scale 2 scores were related to all six were related to neuropsychological test performance. NFI scale scores, and the highest correlations involved the NFI Several NFI scalescontained items reflecting behavioral difDepression (r = .47), Motor (r = .47), and Somatic (r = .47) ficulties, emotional distress,and physical symptomatology often Scales. The highest MMPI Scale 3 correlations were with Soassociated with brain injury. To further examine criterion-rematic complaints (r = .50) and Motor difficulties (I = .42). lated validity, NFI scale scores were compared with patients’ Mh4PI Scale 7 scores correlated most significantly with NFI scoreson five MMPI scales(1, 2, 3,7,8). Elevations on MMPI Depression (r = .43) and Somatic (r = .40) scores. Finally, scales 1 and 3 are associatedwith concerns about diverse areas MMPI Scale 8 scores correlated most with Depression (r = of bodily functioning.62.h’Complaints of muscle, back, and head .40) and Memory/Attention (r = .36) scores. pain are common among patients with high scale scores. As expected, the NFl Somatic scale was most related to MMPI DISCUSSION scale 1 and 3 scores. Principal components and confirmatory factor analytic techThe MMPI’s scale 2 (Depression) has been associated with niques delineated six groups of items with similar content repreboth emotional and psychomotor symptoms often associated senting different constructs. Seventy of the original 105 items with depression.62,64 Feelings of hopelessness,irritability, anheloaded on six scales that were labeled: Depression; Somatic; donia, and low self-esteem are common descriptors of persons Memory/Attention; Communication; Aggression; and Motor. with scale 2 elevations. Also, common are psychomotor retardaThe nature of items and overall content of scales provides a tion, preoccupation with physical complaints, fatigue, and lack picture of brain injury sequelae very much consistent with that of initiative. Among the NFI scales, Depression, Motor, and reported in the literature.‘-6.28 Somatic scoreswere highly and equally correlated with MMPI Cronbach’s alpha calculations revealed high levels of consisscale 2. tency within each scale. Alpha values ranged from .86 to .95 Depression and anxiety are often linked together. Studies of with internal reliability for all items comprising the inventory the MMPI involving both brain injury and psychiatric populaexceeding .97. Correlational analysesrevealed that problems on tions reveal a high degree of correlation between scores on one scale were associated with more frequent problems on othscales 2 (Depression) and scale 7 (Psychasthenia). Scale 7 has ers. A degree of overlap between scales was anticipated for been described as a measure of anxiety, indecisiveness,ruminaseveralreasons.First, a vast majority of the patients had suffered tion, and guilt?* As expected, among the NFI scales,Depression a closed head injury and such injuries have routinely been assowas most related to the MMPI 7 score. ciated with diffuse cortical dysfunction.‘* Diffuse pathophysiolFinal analysis focused on MMPI scale 8, which contains ogy would be expected to cause a diversity of neurobehavioral items reflecting flattened affect, apathy, estrangement, deperproblems. Second, brain injury sequelae are typically interresonalization, lack of ego mastery, disorganized thinking, dislated. For example, patients with more severephysical and cogtorted body image, and peculiar sensory experiences. The NFI nitive problems may have greater reactive depression than paDepression scale was most related to scale 8 scores. Scores on tients with less severe problems. Greater impairment of motor the NFI Memory/Attention and Somatic scaleswere also highly skills necessary for verbal and written communication, could related to the 8 scale. readily be linked to greater levels of overall physical disability. The correspondence between the NFl and MMPI scalesmight The pattern of interrelationships among NFI scalesis similar to be considered impressive, especially given differences in develthat of other widely accepted assessment tools including the opment and diagnostic intent. The MMPI was developed more MMPI and the Wechsler Adult Intelligence Scale Revised.33.60.6’ than 40 years ago to characterize patients with psychiatric probTo ascertain criterion-related validity, relationships between lems, many of whom did not have brain injuries. The NFI was NFI scale scores and neuropsychological measureswere examdeveloped to characterize the day to day problems of persons ined. Among the six NFI scales, Communication scores were with brain injury. Researchers have consistently reported that by far, most related to neuropsychological test performance. psychiatric problems are common among persons with brain Personswho performed poorly on standardized tests were rated injury.65.66 Table

7: Summary

of MMPI

Personality

Scales

Table Aggression MMPI MMPI MMPI MMPI MMPI

1 2 3 7 8

All reported

Arch

Depression

.24 .21

.34 .47 .28 .43 .40

.25 correlations

Phys Med Rehabil

significant

Vol77,

at p < ,001

February

1966

8: NFI Correlations

with

Memory/Attention .36 .40 .32 .39 .36

MMPI

Scales Communication .30 .33 .25 .26 .27

Motor

Somatic

.53 .47 .42 .35 .32

.65 .47 .50 .40 .35

VALIDATION

OF A NEUROBEHAVIORAL

Implications for Future Research The present investigation elicited a number of issues that can readily be addressed by further study. First, some might raise concerns that factor analytic procedures indirectly affected content validity. Of the original 105 items, 35 did not meet the criteria for inclusion. Examination of statistically rejected items revealed a number with potential clinical value including, blackout spells, seizures, dangerous driving, difficulty handling money, and threatening self-harm. Failure to solicit information about these problems could seriously reduce the NFI’s clinical utility. Future validation research with the NFI will include a set of these critical items. Second, all NFl items were weighted equally in calculating scale scores. Unfortunately, problems that occur with equal frequency may have a very different impact on outcome. For example, a patient who sometimes forgets to turn off the stove has a far more serious risk of injury than someone who sometimes forgets to do chores. Future research with the NFI should cxamine differential weighting of items to enhance validity. Additional unresolved issues relating to the psychometric properties of the NFI include interrater agreement, test-retest reliability, and further study of criterion-related validity. For the present investigation, informants were selected based on their reportedly superior knowledge of the patient in comparison to others. Additional research on relationships between informants and patients (eg, duration and type of relationship) may help identify and reduce bias. Expanding the four-point scale to five points may help reduce skewness. The present investigation focused on concurrent validity, one aspect of criterion-related validity. Future studies should include comparisons with clinicians’ ratings (eg, Neurobehavioral Rating Scale),“’ functional assessment indices (eg, Disability Rating Scale),“’ and other psychological tests. The predictive validity of the NFI is also worthy of investigation. The present investigation focused on a heterogeneous sample of persons with traumatic brain injury. Although brain injury is unique, many sequelae arc also characteristic of other neurological disorders (eg, cardiovascular disorders, multiple sclerosis, dementia). The utility of the Neurobehavioral Functioning Inventory for other patient populations warrants further investigalion. Many would agree about the importance of developing a method to accurately characterize patients’ day-to-day abilities. In this regard, questions have been raised about the concurrent validity of ncuropsychological tests and clinicians’ ratings of patients based on structured clinical interviews.” ” The present findings suggest that the NFI is a promising means of invcstigating informants’ perceptions of outpatients’ everyday problems. References I

2.

3.

4.

5.

Eames P. Behavior disorders after severe head injury: their nature and causes and strategies for management. J Head Trauma Rehabil 1988:3:1-6. Kinsella G, Moran C, Ford B, Ponsford J. Emotional disorders and its assessment with the severe head injured population. Psycho1 Med 1988; 1857-63. Kreutzcr J, Devany C. Myers S. Marwitz J. Neurobehavioral outcome following traumatic brain injury: Implications for cognitive rehabilitation. In: Kreutzer J, Wehman P, editors. Cognitive rehabilitation for persons with traumatic brain injury: a functional approach. Baltimore: Paul Brookes. 199155-73 Lezak MD. Relationships between personality disorders. social disturbances, and physical disability following traumatic brain injury. J Head Trauma Rehabil 1987;2:57-69. McLean A, Dikmen SS. Temkin NR. Psychosocial recovery after head injury. Arch Phys Med Rehabil 1993;74:1041-6.

INVENTORY,

123

Kreutzer

6. Oddy M, Humphrey M, Uttley D. Subjective impairment and social recovery after head injury. J Neurol Neurosurg Psychiatry lY78;41:61 l-6. 7. Brooks D. Campsie L, Symington C, Beattie A. McKinlay W. The tive year outcome of severe blunt head injury: A relative’s view. J Ncurol Neurosurg Psychiatry lY86;49:76470. I(. Burke JM. lmhoff CL. Kerrigan JM. MMPI correlates among postacute TBI patients. Brain lnj 1990;4:223-3 I. 9. Thomsen IV. The patient with severe head injury and his family. Stand J Rehab Med 1974;6: 180-3. IO. Wilier B. Rosenthal M. Kreutzer J. Gordon W. Rempel R. Assessment of community integration following rehabilitation for traumatic brain injury. J Head Trauma Rehabil 1993:8:75-87. I I. Arcia E, Gualtieri CT. Association between patient report of symptoms after mild head injury and neurobehavioral performance. Brain Inj 1993:7:481-9. 12. lavin H, Benton A. Grossman R. editors. Neurobehavioral consequences of closed head injury. New York: Oxford University Press. 1982. 13. Becker ME, Vakil E. Behavioral psychotherapy of the frontal-lobe injured patient in an outpatient setting. Brain Inj 1993;7:515-23. 14. Eames P. Haffey WJ. Cope DN. Treatment of behavioral disorders. In: Rosenthal M. Griffith E. Bond M, Miller J. editors. Rehabilitation of the adult and child with traumatic brain injury. 2nd ed. Philadelphia: F.A. Davis. 1990:410-32. 15. Rappaport M, Herrero-Backe C. Rappaport ML, Winterfield KM. Head injury outcome up to ten years later. Arch Phys Med Rehabil 1989:70:88S-93. 16. Hall K. Overview of functional assessment scales in brain injury rehabilitation. NcuroRehabil 1992;2:98-I 13. 17. Hart T. Hayden M. The ecological validity of neuropsychological asscssmcnt and remediation. In: Czzell B. Gross Y. editors. Clinical neuropsychology of intervention. Boston: Martinus Nijhoff. 19X6:2 l-50. 18. LeLak MD. Newer contributions to the neuropsychological assessment of executive functions. J Head Trauma Rehabil 1993;8:2431. 19. R&berg B. Ferris SH. Borenstcin J. Sinaiko E. de Leon M. Buttinger C. Assessment of presenting symptoms. In: Poon LW, editor. Clinical memory assessment of older adults. Washington, DC: American Psychological Association, 1986:108-28. 20. Levin H. High WM. Goethe KE, Sisson RA. Overall JE. Rhoades HM. et al. The ncurobehavioral rating scale: assessment of the behavioral sequelae of head injury by the clinician. J Neurol Neurosurg Psychiatry 1987:.50:183-93. 21. Kay T. Cavallo MM. Erachi 0. Administration manual. N.Y.U. head injury family interview. version I .2. New York: NYU Research & Training Center on Head Trauma and Stroke, 198X. 22. Gilewski MJ. Zelinski EM. Questionnaire assessment of memory complaints. In: Poon LW, editor. Clinical memory assessment of older adults. Washington. DC: American Psychological Association. 1986:93-107. between test scores and everyday life func23. Acker MB. Relationship tioning. In: Uzzell B. Gross Y. editors. Clinical neuropsychology of intervention. Boston: Martinus Nijhoff, 1986:85-I 17. 24. Wilson B. Cockbum J, Baddeley A. Hioms R. Development and validation of a test battery for detecting and monitoring everyday memory problems. J Clin Exper Ncuropsychol 1989; ll:855-70. 25. Reitan RM. Wolfson D. The Halstead-Reitan neuropsychological test battery: theory and clinical interpretation, Tucson: Neuropsychology Press. 198.5. 36. Ryan J. Paolo AM, Smith AJ. Wechslcr Adult lntelligencc ScaleRevised-intersuhtest scatter in brain damaged patients: a comparison with the standardization sample. Psycho1 Assess 1992;4:63-6. 27. Tuel SM. Presty SK, Meythalcr JM. Heinemann AW, Katz RT. Functional improvement in severe head injury after readmission for rehabilitation. Brain Inj 1992;6:363-72. 2x. Overall JE. Gorham DR. The Brief Psychiatric Rating Scale. Psychol Rep 1962; 10:799-X12. 29. Heaton RK. Pendleton MC. Use of ncuropsychological tests to predict adult patients’ everyday functioning. J Consult Clin Psycho1 1981:49:807-21.

Arch

Phys

Mod Rehabil

Vol77,

February

1666

124

VALIDATION

OF A NEUROBEHAVIORAL

30. Kaszniak AW, Davis KL. Instrument and data review: the quest for external validators. In: Poon LW, editor. Clinical memory assessment of older adults. Washington, DC: American Psychological Association, 1986:271-84. 31. Kay T, Silver SM. The contribution of neuropsychological evaluation to the vocational rehabilitation of the head-injured adult. J Head Trauma Rehabil 1988;3:65-76. 32. American Psychological Association. Standards for educational and psychological testing. Washington, DC: American Psychological Association, 1985. 33. Anastasi A. Psychological testing. 6th rev. ed. New York: Macmillan, 1988. assessment. In: Rosenthal M, Grif34. Lynch WJ. Neuropsychological fith ER, Bond MR, Miller JD, editors. Rehabilitation of the adult and child with traumatic brain injury. 2nd rev ed. Philadelphia: F.A. Davis, 1990:310-26. WW, Brooks DN, Bond MR, Martinage DP, Marshall 35. McKinlay MM. The short-term outcome of severe blunt head injury as reported by relatives of injured persons. J Neurol Neurosurg Psychiatry 1981;44:527-33. 36. Peters LC, Stambrook M, Moore AD, Esses L. Psychosocial sequelae of closed head injury: effects on the marital relationship. Brain lnj 1990;4:39-47. 37. Prigatano GP, Schacter DL, editors. Awareness of deficit after brain injury: clinical and theoretical issues. New York: Oxford, 1991. 38. Wilier BS. Allen KM, Liss M. Zicht MS. Problems and cooine . strategies of individuals with traumatic brain injury and their spouses. Arch Phys Med Rehabil 1991;72:460-4. 39. Campbell D, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psycho1 Bull 1959;56:81-105. 40. Hartlage LC, Asken M, Homsby JL, editors. Essentials of neuropsychological assessment. New York: Springer, 1987. 41. Spreen 0, Strauss E. A compendium of neuropsychological tests. New York Oxford University Press, 1991. 42. Kreutzer JS, Leininger BE, Doherty K, Waaland PK. General health and history questionnaire. Richmond, VA: Medical College of Virginia, Rehabilitation Research and Training Center on Severe Brain Injury, 1987. 43. Livingston MG, Brooks DN, Bond MR. Three months after severe head injury: psychiatric and social impact on relatives. J Neurol Neurosurg Psychiatry 1985;48:870-5. 44. Kreutzer J, Gordon W, Rosenthal M, Marwitz J. Neuropsychological characteristics of patients with brain injury: Preliminary findings from a multi-center investigation. J Head Trauma Rehabil 1993;8:47-59. 45. Lezak MD. Neuropsychological assessment. 3rd rev ed. New York: Oxford University Press, 1994. 46. Black WA. Unilateral brain lesions and MMPI performance: A preliminary study. Percept Mot Skills 1975;40:87-93.

Arch

Phys Med Rehabil

Vol77,

February

1996

47.

48. 49. 50. 51. 52.

53. 54. 55.

56. 57. 58. 59. 60. 61.

62. 63. 64. 65. 66.

67.

INVENTORY,

Kreutzer

Leininger B, Kreutzer J, Hill M. Comparison of minor and severe head injury emotional sequelae using the MMPI. Brain lnj 1991;5:199-205. Wooten AJ. MMPI profiles among neuropsychology patients. J Clin Psycho1 1983; 39:392-406. Bollen KA, Long JS. Testing structural equation models. London: Sage Publications, 1993. Cole DA. Utility of contirmatoty factor analysis in test validation research. J Consult Clin Psycho1 1987;55:584-94. Fassinger RE. Use of structural equation modeling in counseling psychology research. J Couns Psy 1987;34:425-36. Hoyle RH. Evaluating measurement models in clinical research: covariance structure analysis of latent variable models of self-conception. J Consult Clin Psycho1 1991;59:67-76. Byrne BM. Structural Equation Modeling with EQS and EQS/Windows. London: Sage Publications, 1994. Bentler PM. EQS: Structural equations program manual. Los Angeles: BMDP Statistical Software, 1992. Chartrand JM, Robbins SB, Morrill WH, Boggs K. Development and validation of the Career Factors Inventory. J Couns Psy 1990; 37:491-501. Tabachnik BG, Fidel1 LS. Using multivariate statistics. 2nd ed. Northridge: HarpersCollins Publisher, 1989. Norusis MJ. SPSS for Windows: Base system user’s guide. Release 6.0. Chicago: SPSS, Inc., 1993. Keppel G. Design and analysis: A researcher’s handbook. 3rd ed. Englewood Cliffs: Prentice Hall, 1991. Larzelere RE, Mulaik SA. Single-sample test for many correlations. Psycho1 Bull 1977; 84:557-69. Wechsler D. WAIS-R manual. New York Psychological Corporation, 1981. Dahlstrom WG, Welsh GS, Dahlstrom LE. An MMPI Handbook, Volume 1: Clinical Interpretation. Minneapolis: University of Minnesota Press, 1972. Greene RL. MMPI-2/MMPl: An interpretive manual. Boston, Allyn and Bacon, 1991. Comrey AL. A factor analysis of items on the MMPI Hysteria scale. Educ Psycho1 Meas 1957; 17:586-92. Graham-JR. The MMPI: A practical guide. 2nd rev ed. New York: Oxford University Press, 1987. Lishman WA. Brain damage in relation to psychiatric disability after head injury. Br J Psycho1 1968; 114:373-410. Silver JM, Yudofsky SC. Psychopharmacological approaches to the patient with affective and psychotic features. J Head Trauma Rehabil 1994;9:61-77. Rappaport M, Hall KM, Hopkins K, Belleza T, Cope DN. Disability rating scale for severe head trauma: Coma to community. Arch Phys Med Rehabil 1982;63: 118-23.