Rasch Analysis of the Community Integration Measure in Persons With Traumatic Brain Injury

Archives of Physical Medicine and Rehabilitation journal homepage: www.archives-pmr.org Archives of Physical Medicine and Rehabilitation 2014;95:734-40

ORIGINAL ARTICLE

Rasch Analysis of the Community Integration Measure in Persons With Traumatic Brain Injury Scott R. Millis, PhD,a Sarah-Jane Meachen, PhD,b Julie A. Griffen, PhD,b Robin A. Hanks, PhD,a Lisa J. Rapport, PhDc From the aDepartment of Physical Medicine and Rehabilitation, Wayne State University, Detroit, MI; bRehabilitation Institute of Michigan, Detroit, MI; and cDepartment of Psychology, Wayne State University, Detroit, MI.

Abstract Objective: To examine the measurement properties of the Community Integration Measure (CIM) in persons with traumatic brain injury (TBI). Design: Rasch analysis was used to retrospectively evaluate the CIM. Setting: Rehabilitation hospital. Participants: Persons (NZ279) 1 to 15 years after a TBI. Interventions: None. Main Outcome Measure: CIM. Results: The CIM met Rasch expectations of unidimensionality and reliability (person separation ratioZ2.01, item separation ratioZ4.52). However, item endorsibility was poorly targeted to the participants’ level of community integration. A ceiling effect was found with this sample. Conclusions: The CIM is a relatively reliable and unidimensional scale. Future iterations might benefit from the addition of items that are more difficult to endorse (ie, improved targeting). Archives of Physical Medicine and Rehabilitation 2014;95:734-40 ª 2014 by the American Congress of Rehabilitation Medicine

Many individuals with traumatic brain injury (TBI) encounter lasting impediments to satisfactory reintegration into their communities in terms of return to work,1 recreation,2 mobility,3 and engaging in social relations.4 Community integration is an important facet of recovery to consider because it is intimately connected with autonomy, productivity, and quality of life after injury. Moreover, enhanced community integration is associated with improved physical, cognitive, and psychosocial outcomes.5 As such, reliable and valid tools to measure community integration are valuable for clinicians and researchers interested in assessing outcomes in TBI. Rasch analysis6 is a family of statistical models used to construct and evaluate measures of latent constructs (eg, questionnaires). A number of instruments assessing rehabilitation outcomes have been subject to Rasch analysis, including the FIM Presented to the American Congress of Rehabilitation Medicine, October 7–11, 2009, Denver, CO. Supported by the National Institute for Disability and Rehabilitation Research through the US Department of Education (grant no. H133A080044). No commercial party having a direct financial interest in the results of the research supporting this article has conferred or will confer a benefit on the authors or on any organization with which the authors are associated.

instrument,7 the National Institutes of Health Stroke Scale,8 and the Beck Depression Inventory II.9 The central aim of the current study was to evaluate the measurement properties of the Community Integration Measure (CIM) using Rasch analysis. A comprehensive assessment of community integration may require >1 approach.10 Many commonly used community integration assessment tools focus on measuring observable, objective aspects of community integration, such as the number of hours spent in recreational activities. Although those tools provide important information, they do not capture a person’s subjective sense of integration within their community. Researchers have emphasized that not only should objective indicators of community integration be measured, but also “the satisfaction of persons with their participation levels.”11(p125) There are a growing number of instruments designed to assess perceived community integration. Of these, one of the most common and well-researched instruments is the CIM.10 The CIM was designed to quickly assess an individual’s subjective sense of integration into his or her community. It is thought to capture one’s sense of participating and belonging.10 Interestingly, although the authors of the CIM adopted a 4-factor theoretical

0003-9993/14/$36 - see front matter ª 2014 by the American Congress of Rehabilitation Medicine http://dx.doi.org/10.1016/j.apmr.2013.11.020

Rasch analysis of the Community Integration Measure in TBI model of community integration (assimilation, social support, occupation, independent living),10 their preliminary validation study10 was more supportive of a 1-factor solution. Psychometric properties of the CIM have since been explored using traditional psychometric methods in a variety of health populations, including oncology,10 spinal cord injury,12 and TBI.13,14 A previously published analysis of the CIM in the current sample15 showed that the CIM had adequate reliability, criterion validity, and utility when examined using a traditional psychometric approach. To date and to our knowledge, the psychometric properties of the CIM have not been examined via Rasch analysis. A Rasch model6 is a criterion (or set of criteria), against which data can be evaluated for conformity or fit. Rasch analysis is particularly well suited to examine whether the CIM is unidimensional in nature, as suggested in the preliminary validation study of McColl et al10 (as opposed to the 4-dimensional theoretical model that originally guided its development).

735 5 (always agree), 4 (sometimes agree), 3 (neutral), 2 (sometimes disagree), and 1 (always disagree). Item content is provided in full in table 1. The CIM has a Flesch Reading Ease Score16 of 76.2, which corresponds to a Flesch-Kincaid Grade Level17 of 5.6.

Procedures The Southeastern Michigan TBI System assesses a variety of outcomes at routine follow-up points: 1, 2, and 5 years postinjury and every 5 years thereafter. The CIM, along with other outcome measures, was administered at those follow-up points. Hence, some participants completed the CIM on multiple occasions. However, each participant contributed only 1 CIM administration to the dataset: the earliest administration of the CIM was analyzed.

Data analysis

Methods Participants Data for this study were deidentified archival data derived from a larger database. The study protocol was approved by the Wayne State University investigational review board. The sample included 279 individuals from the Southeastern Michigan TBI System research project, which is part of the nationwide TBI Model Systems (TBIMS) research study funded by the National Institute of Disability and Rehabilitation research. TBI is defined by the TBIMS as an injury to brain tissue caused by an external mechanical force. Any of the following indicators were taken as evidence of TBI: loss of consciousness from brain trauma, posttraumatic amnesia, skull fracture, or objective neurologic findings that could reasonably be attributed to TBI by physical or mental status examination. All participants were at least 16 years old at the time of injury, they received acute care at a designated TBIMS site within 72 hours of injury, and they received inpatient rehabilitation at the Rehabilitation Institute of Michigan. Individuals were considered for inclusion if their Glasgow Coma Scale score at admission to the emergency department was between 3 and 12 or if their Glasgow Coma Scale score was higher (13e15) but there was evidence of intracranial pathology such as hemorrhage. TBI severity was defined as days to obtain a motor score of 6 on the motor subscale of the Glasgow Coma Scale twice consecutively within a 24-hour period. The sample excluded individuals with mild or very severe injuries because these individuals either did not require inpatient rehabilitation (mild injuries) or were deemed unlikely to benefit from it (very severe injuries).

Outcome measures Community integration measure10 The 10-item CIM requires participants to appraise the quality of their integration into their community using a 5-point scale:

List of abbreviations: CIM RSM TBI TBIMS

Community Integration Measure rating scale model traumatic brain injury Traumatic Brain Injury Model Systems

www.archives-pmr.org

The Rasch rating scale model (RSM)18 was used to evaluate the CIM. The RSM rather than the partial credit model was used because the CIM items all share the same rating scale structure. The partial credit model is used when each item has a unique rating scale structure. Rasch analysis was implemented using Winsteps 3.80.1 software.a In evaluating the CIM, we examined the unidimensionality, reliability, targeting, and response categories. To assess unidimensionality, we examined infit and outfit mean squares.19,20 These fit statistics have a theoretical range of 0 to infinity. Under good fit conditions of the data to the expectation of unidimensionality, the expected value of these fit statistics is 1. Infit and outfit mean-square ranges of 0.6 to 1.4 are reasonable for rating scales/surveys.20 We also assessed unidimensionality of the CIM using a principal component analysis of Rasch residuals where residuals can be understood as the difference between observed and expected data values. We evaluated the internal consistency of person and item performance on the CIM by examining separation reliability estimates and separation ratios.21 Separation reliability for persons refers to the consistency of person responses across items, whereas the separation reliability for items refers to the consistency of item performances across persons. Much like the Cronbach alpha, separation reliability estimates the ratio of the true score variance to the observed

Table 1

Demographic characteristics of the sample (NZ279)

Characteristic Sex Male Female Ethnicity Black White Latin American Asian Native American Years of education <12y High school diploma Bachelor’s degree Postgraduate degree

n

%

225 54

80.6 19.4

207 64 4 2 2

74.2 22.9 1.4 0.7 0.7

124 137 11 7

44.5 49.0 3.9 2.6

736

S.R. Millis et al

Table 2

categories minus 1; in the case of the CIM, there are 4 thresholds (5 response categories).

Injury characteristics (NZ279)

Characteristic Cause of injury Vehicles Gunshot Blunt assaults Other violence Sports Fall Struck by object Pedestrian Time since injury (y) 1 2 5 10 15

n

%

89 31 91 5 2 26 1 34

32.0 11.1 32.6 1.8 0.8 9.3 0.4 12.2

77 41 45 64 52

27.6 14.7 16.1 22.9 18.6

score variance, and its value ranges from 0 to 1. To overcome the potential for a ceiling effect in separation reliability, each separation reliability estimate has a corresponding separation ratio (theoretical value ranging from 0 to infinity), which is the ratio of the true (adjusted) SD of measures to the average SE of measures. Separation ratios >2 provide evidence of internal consistency.22 Targeting of item endorsibility to a person’s level of community integration was assessed by comparison of the distribution and spread of items and persons along a common logit scale of the latent construct (eg, examination of the item-person map).23 Similar to the Rasch model for dichotomous items, the RSM reports both the item difficulties and person abilities, and it also provides a single set of thresholds for the rating scale that is common to all the items. These Rasch-Andrich thresholds represent the point or boundary at which the likelihood of being observed in a given response category is exceeded by the likelihood of being observed in the next higher category.23 The number of Rasch-Andrich thresholds is equal to the number of response

Table 3

Item

8

I know a number of people in this community well enough to say hello and have them say hello back. I know the rules in my community and I can fit in with them. I know my way around this community. I can be independent in this community. I feel that I am accepted in this community. There are people I feel close to in this community. There are things that I can do in this community for fun in my free time. I like where I’m living now. I have something to do in this community during that main part of my day that is useful and productive. I feel like part of this community, like I belong here.

6 10 1

Participant characteristics Demographic characteristics of the sample are shown in table 1. The average age of the participants was 44.913.6 years (range, 18e90y). Nearly three quarters of the sample (74.2%) identified as black. The ratio of men to women was approximately 4:1. Almost half of the sample reported <12 years of high school education. Participants took an average of 7.9 hours to obtain a motor score of 612.8 on the Glasgow Coma Scale (range, 1e 99). The average Glasgow Coma Scale score on admission to the emergency department was 94 (range, 3e15). Blunt assaults comprised almost a third of the cases in the sample, and motor vehicle collisions comprised an additional third. Gunshot wounds, falls, and pedestrian versus motor vehicle collisions were each responsible for close to 10% of the sample. Injury characteristics of the sample are shown in table 2. The mean raw score on the CIM for this sample was 39.38.1 (range, 14e50).

Rasch analysis of the CIM Infit and outfit mean squares for each item are shown in table 3. Fit values were generally acceptable (range, .70e1.45), suggesting that items fit the RSM expectation of unidimensionality. However, initial examination of the Rascheresidual-based principal component analysis did not provide additional evidence of the unidimensionality of the CIM. Variance explained by the measures was only 44.4%. However, modeled variance was 45.2%, that is, the variance that would be explained if the data exactly fit the Rasch model. What might account for this apparent discrepancy? Some possible explanations are sample ability variance was narrow (person separationZ2.01), the CIM is a short instrument (only 10 items), sample-item targeting was poor, and there was a

CIM items: item level statistics

No.

3 2 5 4 7 9

Results

Item Difficulty (logits)

Item Fit (mean squares)

Response Categories Endorsed

Infit

Outfit

1 (%)

.54

0.94

0.99

2

.32 .25 .23 .07 .09 .18

1.10 1.45 1.04 0.71 0.91 0.83

0.99 1.40 1.05 0.70 0.89 0.84

.19 .45

1.28 1.07

.50

0.88

2 (%)

3 (%)

4 (%)

5 (%)

4

11

28

56

4 7 4 4 7 8

5 6 5 7 6 8

11 9 16 19 16 16

27 24 27 25 30 29

52 54 49 45 39 38

1.25 1.09

10 11

10 8

14 20

20 30

45 30

1.04

10

9

26

26

29

NOTE. Items are shown in order of ascending difficulty.

www.archives-pmr.org

Rasch analysis of the Community Integration Measure in TBI

737 ceiling effect in this sample with 9% of the participants obtaining the maximum score. As noted, the person separation ratio was 2.01 with a corresponding person reliability of .80. The item separation ratio was 4.52 with a corresponding item reliability of .95. These values reflect relatively good reliability for this instrument within this sample; although, a higher person separation would be desirable. The Winsteps item-person map is shown in figure 1. An item-person map with good targeting would be symmetric along the vertical axis with items and persons clustered in a similar fashion with a similar range. This was not the case for the CIM in this sample where item measures ranged from .54 to .50 logits, whereas person measures ranged from 1.66 to 4.47 logits. Inspection of the item-person map illustrates this poor targeting of items. It is easily visually appreciated that the items (shown on the right side of the map) measure a much narrower range of the construct of community integration than would be necessary to adequately characterize the participants (represented by x on the left side of the map). The polytomous distribution map (fig 2) provides additional information. In the middle column, each item is placed at its mean calibration. In the left column, the item is shown at the measure level corresponding to a probability of 0.5 of exceeding the bottom rating scale category. In the right column, the item is shown at the level corresponding to the probability of 0.5 of being endorsed in the top rating scale category. The Rasch expectation is that individuals with higher ability, in this case a higher level of community integraton, will have a higher likelihood of endorsing higher response categories on the CIM. Examination of the average abilities of persons endorsing a given response category in the current study showed this expected monotonicity, that is, persons with greater community integration were more likely to endorse the higher response categories. However, examination of the step calibration estimates for the CIM revealed disordered Rasch-Andrich thresholds. This occurred between the neutral and somewhat disagree rating categories. However, further examination revealed that CIM item categories were not substantively disordered. The category mean-square fit statistics did not markedly exceed 1, and the average measured increased monotonically. In this case, the disordering of the step calibrations reflected the relative infrequency of category 2 (sometimes disagree) (see table 3). Therefore, collapsing categories was ruled out.

Discussion

Fig 1 Item-person map. Abbreviations: frequ, frequent; M, mean; S, 1 standard deviation; T, 2 standard deviations.

www.archives-pmr.org

Rasch analysis provided additional evidence that the CIM is reliable and unidimensional. Although the test authors used a 4factor theoretical model of community integration in developing the CIM, the current finding of unidimensionality is not surprising. A similar 1-factor solution was supported by principal components factor analysis in the test authors’ preliminary validation study.10 Unfortunately, in the current sample, there was evidence of poor targeting of items to persons (ie, items were too easily or frequently endorsed). There was a ceiling effect. The CIM needs additional items reflecting high levels of community integration. The selection of items with a difficulty level that is appropriately targeted at the ability level of the persons on whom the test is

738

S.R. Millis et al

Fig 2

Polytomous distribution map.

www.archives-pmr.org

Rasch analysis of the Community Integration Measure in TBI

739

intended to be used is a key consideration in developing a test or questionnaire. In the case of attitudinal questionnaires, the term difficulty can be understood in terms of the endorsibility of an item, whereas ability can be understood in terms of the level or degree of community integration and depression. On a depression instrument, for example, some items have content that clearly assesses more severe symptoms (eg, suicidal ideation) than others (eg, disturbed sleep). In Rasch terms, the former items would be said to have a higher difficulty level, and highly depressed individuals would be said to have higher ability than less depressed individuals.

Keywords

Study limitations

References

It is possible that the CIM would be well targeted to populations in which individuals have a much lower subjective sense of community integration than was the case in the current sample. For instance, the present sample was composed primarily of urban and suburban-dwelling individuals. Persons in remote rural areas might respond quite differently to CIM items. Similarly, participants in the current sample were all at least 1 year postinjury; in many cases they were considerably further. Individuals who are earlier postinjury might have greater barriers to community integration such that current CIM items could be better targeted to such persons. If the CIM is to be used with populations similar to the population in the current sample, future versions could include a wider range of items, particularly more difficult items (ie, items that require much higher levels of subjective community integration in order to endorse them). For example, mobility and/or transportation within the community are frequently reduced after TBI, often permanently.3 The CIM could include items addressing this barrier to community integration, such as “it is easy for me to travel from place to place within my community.” Resumption of leisure activities is often reported to be incomplete after TBI as well.24 Items focused on participation in leisure activities might improve the targeting of the CIM for use in TBI samples. For instance, “I am able to take full advantage of recreational or leisure activities in my community.”

1. Kreutzer JS, Marwitz JH, Walker W, et al. Moderating factors in return to work and job stability after traumatic brain injury. J Head Trauma Rehabil 2003;18:128-38. 2. Wise EK, Mathews-Dalton C, Dikmen S, et al. Impact of traumatic brain injury on participation in leisure activities. Arch Phys Med Rehabil 2010;91:1357-62. 3. Rapport L, Hanks RA, Bryer RC. Barriers to driving and community integration after traumatic brain injury. J Head Trauma Rehabil 2006; 21:34-44. 4. Prigatano GP. Work, love, and play after traumatic brain injury. Bull Menninger Clin 1989;53:414-31. 5. Bodenheimer CF, Roig RL, Worsowicz GM, Cifu DX. Geriatric rehabilitation. 5. The societal aspects of disability in the older adult. Arch Phys Med Rehabil 2004;85(7 Suppl 3):S23-6. 6. Rasch G. Probabilistic models for some intelligence and attainment tests. Copenhagen: The Danish University of Education; 1960. 7. Hamilton BB, Granger CV, Sherwin FS, Zielezny M, Tashman JS. A uniform national data system for medical rehabilitation. In: Fuhrer M, editor. Rehabilitation outcomes: analysis and measurement. Baltimore: Brookes; 1987. p 137-47. 8. Millis SR, Straube D, Iramaneerat C, Smith EV, Lyden P. Measurement properties of the National Institutes of Health Stroke Scale for people with right- and left-hemisphere lesions: further analysis of the clomethiazole for Acute Stroke Study-Ischemic (class-I) trial. Arch Phys Med Rehabil 2007;88:302-8. 9. Siegert RJ, Tennant A, Turner-Stokes L. Rasch analysis of the Beck Depression Inventory-II in a neurological rehabilitation sample. Disabil Rehabil 2010;32:8-17. 10. McColl MA, Davies D, Carlson P, Johnston J, Minnes P. The community integration measure: development and preliminary validation. Arch Phys Med Rehabil 2001;82:429-34. 11. Sander AM, Clark A, Pappadis MR. What is community integration anyway?: defining meaning following traumatic brain injury. J Head Trauma Rehabil 2010;25:121-7. 12. De Wolf A, Lane-Brown A, Tate RL, Middleton J, Cameron ID. Measuring community integration after spinal cord injury: validation of the Sydney psychosocial reintegration scale and community integration measure. Qual Life Res 2010;19:1185-93. 13. Willer B, Rosenthal M, Kreutzer JS, Gordon WA, Rempel R. Assessment of community integration following rehabilitation for traumatic brain injury. J Head Trauma Rehabil 1993;8:75-87. 14. Reistetter TA, Spencer JC, Trujilo K, Abreu BC. Examining the Community Integration measure (CIM): a replications study with life satisfaction. NeuroRehabilitation 2005;20:139-48. 15. Griffen JA, Hanks RA, Meachen SJ. The reliability and validity of the community integration measure in persons with traumatic brain injury. Rehabil Psychol 2010;55:292-7. 16. Flesch RF. A new readability yardstick. J Appl Psych 1948;32: 221-33.

Conclusions The current study sought to examine the measurement properties of the CIM for use in measuring subjective community integration in persons recovering from TBI. Findings from this study provide support for the CIM to assess community integration after TBI, particularly when the clinician or investigator seeks a brief instrument. Users of the CIM need to be aware that the scale is most useful in assessing lower levels of community integration. Alternative measures should be considering when needing to assess high levels of community integration.

Supplier a. Winsteps. http://www.winsteps.com/

www.archives-pmr.org

Brain injuries; Rehabilitation

Community

integration;

Psychometrics;

Corresponding author Scott R. Millis, PhD, Department of Physical Medicine and Rehabilitation, 261 Mack Blvd, Detroit, MI 48201. E-mail address: [email protected].

740 17. Kincaid JP, Fishburne RP, Rogers RL, Chissom BS. Derivation of new readability formulas (automated readability index, fog count, and flesch reading ease formula) for Navy enlisted personnel. Memphis: Memphis Naval Air Station; 1975. 18. Andrich D. A rating formulation for ordered response categories. Psychometrika 1978;43:561-73. 19. Gustafson JE. Testing and obtaining fit of data to the Rasch model. Br J Math Stat Psychol 1980;33:205-33. 20. Wright BD, Linacre JM. Reasonable mean-square fit values. Rasch Measurement Transactions 1994;8:370.

S.R. Millis et al 21. Wright BD. Reliability and separation. Rasch Measurement Transactions 1996;9:472. 22. Linacre JM. What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions 2002;16:878. 23. Bond TG, Fox CM. Applying the Rasch model: fundamental measurement in the human sciences. Mahwah: Erlbaum; 2001. 24. Bier N, Dutil E, Couture M. Factors affecting leisure participation after a traumatic brain injury: an exploratory study. J Head Trauma Rehabil 2009;24:187-94.

www.archives-pmr.org