- Email: [email protected]
Continued validation of the Symptom Inventory in multiple sclerosis
Journal of the Neurological Sciences 285 (2009) 134–136
Contents lists available at ScienceDirect
Journal of the Neurological Sciences j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / j n s
Continued validation of the Symptom Inventory in multiple sclerosis Robert W. Motl a,⁎, Carolyn E. Schwartz b,c, Timothy Vollmer d a
University of Illinois at Urbana-Champaign, Urbana, IL, USA DeltaQuest Foundation, Inc., Concord, MA, USA c Tufts University Medical School, Boston, MA, USA d University of Colorado Health Sciences Center, Denver, CO, USA b
a r t i c l e
i n f o
Article history: Received 21 April 2009 Accepted 11 June 2009 Available online 9 July 2009 Keywords: Multiple sclerosis Neurological impairment Validity Psychometrics
a b s t r a c t Objective: This study examined the construct, discriminant, and incremental validity of scores from the short (SI-S) and long (SI-L) forms of the Symptom Inventory in persons with MS. Methods: The sample included 133 individuals with MS who completed the SI-L, Performance Scales (PS), EDSS, Multiple Sclerosis Walking Scale-12 (MSWS-12), Multiple Sclerosis Impact Scale-29 (MSIS-29), and Godin Leisure-Time Exercise Questionnaire (GLTEQ) and then wore an accelerometer for 7 days. The data were analyzed using SPSS, version 16.0. Results: There were large correlations between SI-S and SI-L total and subscale scores and between SI-S, SI-L, and PS total scores. The correlations were similar in magnitude between SI-S, SI-L, and PS scores with EDSS and MSIS-29 scores, but not with MSWS-12, accelerometer, and GLTEQ scores. Discriminant function analysis indicated that SI-S scores better differentiated groups with minimal, moderate, and severe disability than did SI-L and PS scores. Regression analysis indicated that SI-S and SI-L scores explained incrementally more variance in EDSS, MSWS-12, and MSIS-29 scores after accounting for PS scores alone. Conclusion: Such findings provide additional support for the validity of both SI-S and SI-L scores in individuals with MS and support the adoption of either the SI-S or SI-L by clinical MS researchers. © 2009 Elsevier B.V. All rights reserved.
1. Introduction
evidence that supports the validity of SI-S scores [4]. The present study further examined the construct, discriminant, and incremental validity of scores from the SI-S and SI-L in persons with MS.
Multiple sclerosis (MS) typically begins with intermittent bursts of focal inflammation [1] that result in the demyelination and transection of axons in the central nervous system (CNS) [2] and the accumulation of neurological impairment across time [3]. The study of neurological impairment in MS has been advanced by the development of the Symptom Inventory (SI) [4]. There are long (99 items; SI-L) and short (29 items; SI-S) forms of the SI, and both forms consist of six subscales that were intended to correlate with the localization of brain lesions (visual, left hemisphere, right hemisphere, brain stem and cerebellum, spinal cord, and nonlocalized symptoms) and that can be combined for generating an overall index of neurological impairment. Psychometric analysis has provided preliminary support for the reliability and validity of SI-L and SI-S scores as measures of neurological impairment in individuals with MS [4]. Nevertheless, validation of scores from a measure is an ongoing and evolving process [5,6] and the provision of additional evidence is warranted that further establishes the validity of inferences from SI scores. This is particularly important given the recommendation that clinical MS researchers adopt the SI-L until there is additional
2.2. Measures
⁎ Corresponding author. University of Illinois at Urbana-Champaign, Department of Kinesiology and Community Health, 350 Freer Hall, 906 South Goodwin Ave., Urbana, IL, 61801, USA. Tel.: +1 217 265 0886; fax: +1 217 244 7322. E-mail address: [email protected] (R.W. Motl).
Neurological impairment was measured by the 99-item version of the SI [4]. Neurological disability was measured by the 8-item Performance Scales (PS) [4] and the 15-item self-reported version of the EDSS [8]. Walking ability was measured with the 12-item Multiple
0022-510X/$ – see front matter © 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.jns.2009.06.015
2. Method 2.1. Participants We have described the recruitment and characteristics of the sample elsewhere [7]. Briefly, the sample was recruited through 13 support groups from Midwestern chapters of the National MS Society. There were 140 individuals who were eligible for participation and seven of those individuals did not participate. The final sample consisted of 133 individuals of whom 62% (n = 82) had a relapsing– remitting clinical course and 38% (n = 51) had a progressive clinical course (i.e., primary or secondary progressive). The mean duration since diagnosis was 12 years (SD = 9) and the median Expanded Disability Status Scale (EDSS) score was 5.5 (range = 1–8.5).
R.W. Motl et al. / Journal of the Neurological Sciences 285 (2009) 134–136 Table 1 Construct validity correlations along with 95% confidence intervals among total scores. Measure
Symptom Inventory short form
Symptom Inventory long form
Performance scales
EDSS MSIS-29 MSWS-12 Accelerometer GLTEQ
.68 .71 .73 −.55 −.41
.61 (.49,.71) .74 (.65,.81) .69 (.59,.77) −.56 (−.43, −.67) −.40 (−.25, −.53)
.59 (.47,.69) .69 (.59,.77) .57 (.44,.67) −.39 (−.24, −.53) −.22 (−.05, −.38)
(.58,.76) (.61,.79) (.64,.80) (−.42, −.66) (−.26, −.54)
Note. Values represent r (95% CI). EDSS = Expanded Disability Status Scale; MSWS-12 = Multiple Sclerosis Walking Scale-12; MSIS-29 = Multiple Sclerosis Impact Scale; GLTEQ = Godin Leisure-Time Exercise Questionnaire. Accelerometer = average counts per day.
Sclerosis Walking Scale-12 (MSWS-12) [9]. Quality of life was measured by the 29-item Multiple Sclerosis Impact Scale-29 (MSIS-29) [10]. Physical activity was measured by the ActiGraph single-axis accelerometer (model 7164 version, Health One Technology, Fort Walton Beach, FL) and Godin Leisure-Time Exercise Questionnaire (GLTEQ) [11]. 2.3. Procedure The procedure was approved by a University Institutional Review Board and all participants provided written informed consent. Participants completed a battery of questionnaires that included the SI-L, PS, EDSS, MSWS-12, MSIS-29, and GLTEQ and then wore an accelerometer for a 7-day period. Participants received $10 remuneration for completing the study procedures.
135
CI = .86,.93], p b .0001), Spinal Cord (r = .93 [95% CI = .90,.95], p b .0001), and Nonlocalized (r = .96 [95% CI = .90,.95], p b .0001). 3.1.2. Correlations among overall SI-S, SI-L, and PS scores There was a large correlation between overall scores from the SI-S and PS (r = .71 [95% CI = .61,.79], p b .0001). There was a similarly large correlation between overall scores from the SI-L and PS (r = .73 [95% CI = .64,.80], p b .0001). 3.1.3. Correlations between overall SI-S, SI-L, and PS scores with other measures The correlations along with 95% CIs are provided in Table 1. The correlations were similar in magnitude between SI-S, SI-L, and PS scores with EDSS and MSIS-29 scores. The correlations were larger in magnitude for SI-S and SI-L than PS with MSWS-12, accelerometer, and GLTEQ scores. 3.2. Discriminant validity The mean overall scores for the SI-S, SI-L, and PS as a function of disability level are provided in Table 2. The discriminant function analysis indicated that SI-S scores (eigenvalue = .84) better differentiated groups with minimal, moderate, and severe disability than did SI-L (eigenvalue = .59) and PS (eigenvalue = .68) scores. The SI-S scores clearly distinguished all three groups, whereas the SI-L and PS scores better distinguished the groups with minimal and moderate disability than the groups of moderate and severe disability.
2.4. Data analysis
3.3. Incremental validity
The construct, discriminant, and incremental validity of SI-S and SI-L scores were examined using Pearson correlation, discriminant function, and multiple linear regression analyses in SPSS, version 16.0 (SPSS, Chicago, IL). Construct validity was initially assessed using Pearson correlations (r) and 95% confidence intervals (CI) between (a) SI-S and SI-L subscale and total scores; (b) SI-S, SI-L, and PS total scores; and (c) SI-S, SI-L, and PS total scores with EDSS, MSWS-12, MSIS-29, accelerometer, and GLTEQ scores. Guidelines of .1, .3, and .5 were used for judging the magnitude of the correlations as small, moderate, and large, respectively [12] and 95% CIs demonstrated differences in correlation coefficients. Discriminant validity was assessed using discriminant function analysis. We examined the ability of SI-S, SI-L, and PS scores to distinguish groups with minimal (EDSS score of 1–2.5), moderate (EDSS score of 3–6), and severe (EDSS score of N6) disability. The degree of discrimination was based on eigenvalues, with a larger eigenvalue indicating better differentiation among groups. Incremental validity was assessed using multiple linear regression analysis. We separately regressed EDSS, MSWS-12, and MSIS-29 scores on PS scores entered in Block 1 and SI-S or SI-L scores entered in Block 2. The significance and magnitude of the change in R2 between Blocks (i.e., ΔR2) provided an indication of the incremental variance explained by overall SI-S and SI-L scores beyond that of PS scores alone.
3.3.1. EDSS scores The multiple regression analysis indicated that PS scores explained a large portion of variance in EDSS scores (R2 = .35) and that SI-S scores explained significantly [F(1,121) = 31.03, p b .0001] and incrementally (ΔR2 = .13) more variance in EDSS scores. The multiple regression analysis similarly indicated SI-L scores explained significantly [F(1,121) = 14.25, p b .0001] and incrementally (ΔR2 = .07) more variance in EDSS scores than PS scores alone (R2 = .35).
3. Results
3.3.2. MSIS-29 scores The multiple regression analysis further indicated that PS scores explained a large portion of variance in MSIS-29 scores (R2 = .49) and that SI-S scores explained significantly [F(1,125) = 26.71, p b .0001] and incrementally (ΔR2 = .09) more variance in MSIS-29 scores. The multiple regression analysis similarly indicated that SI-L scores explained significantly [F(1,125) = 33.69, p b .0001] and incrementally (ΔR2 = .11) more variance in MSIS-29 scores than only PS scores (R2 = .49). 3.3.3. MSWS-12 scores The final multiple regression analysis indicated that PS scores explained a large portion of variance in MSWS-12 scores (R2 = .30) and that SI-S scores explained significantly [F(1,109) = 49.88, p b .0001] and incrementally (ΔR2 = .22) more variance in MSWS-12 scores. The multiple regression analysis similarly indicated that SI-S scores explained significantly [F(1,109) = 32.22, p b .0001] and
3.1. Construct validity 3.1.1. Correlations among overall and subscale SI-S and SI-L scores There was a large correlation between overall scores from the SI-S and SI-L (r = .97 [95% CI = .96,.98], p b .0001). There were similarly large correlations between subscale scores from the SI-S and SI-L: Visual (r = .90 [95% CI = .86,.93], p b .0001), Left Hemisphere (r = .83 [95% CI = .77,.88], p b .0001), Right Hemisphere (r = .97 [95% CI = .96,.98], p b .0001), Brainstem and Cerebellum (r = .90 [95%
Table 2 Mean score (SD) by level of disability. Disability level
Symptom Inventory short form
Symptom Inventory long form
Performance scales
EDSS score 1–2.5, n = 29 EDSS score 3–6, n = 69 EDSS score N6, n = 31
4.9 (3.1) 10.0 (3.2) 12.9 (4.1)
4.5 (2.8) 9.0 (3.3) 11.2 (3.1)
8.7 (4.1) 16.8 (5.3) 20.2 (5.0)
Note. EDSS = Expanded Disability Status Scale.
136
R.W. Motl et al. / Journal of the Neurological Sciences 285 (2009) 134–136
incrementally (ΔR2 = .16) more variance in MSWS-12 scores than PS scores alone (R2 = .30). 4. Discussion Validation of scores from a measure is an ongoing and evolving process [5,6] and is particularly important for the Symptom Inventory based on the recommendation that clinical MS researchers adopt the SI-L until there is additional evidence supporting the validity of SI-S scores [4]. To that end, we conducted analyses that both confirmed and extended the validity of inferences from SI-S and SI-L scores in a community-based sample of individuals with MS. Our analyses indicated that (a) there were large correlations between SI-S and SI-L total and subscale scores and between SI-S, SI-L, and PS total scores; (b) the correlations were similar in magnitude between SI-S, SI-L, and PS scores with EDSS and MSIS-29 scores, but not with MSWS-12, accelerometer, and GLTEQ scores; (c) SI-S scores better differentiated groups with minimal, moderate, and severe disability than did SI-L and PS scores; and (d) SI-S and SI-L scores explained incrementally more variance in EDSS, MSWS-12, and MSIS-29 scores after controlling for PS scores. Such findings provide additional support for the validity of both SI-S and SI-L scores in individuals with MS and support the adoption of either the SI-S or SI-L in clinical MS research. Our findings both replicated and extended those of the initial research on the SI. For example, we reported a correlation of r = .97 between overall SI-S and SI-L scores and previous researchers reported a correlation coefficient of r = .99 [4]. We further note that overall SI-S and SI-L scores similarly correlated with PS scores (rs = .71 and .73, respectively) and EDSS scores (rs = .68 and .61, respectively) and this too is consistent with previous research [4]. Nevertheless, there are several noteworthy and unique findings from our analyses of the validity of SI-S and SI-L scores. One noteworthy finding was that overall SI-S, SI-L, and PS scores correlated similarly with EDSS and MSIS-29 scores, whereas there were stronger correlations between SI-S and SI-L scores than PS scores with MSWS-12, accelerometer, and GLTEQ scores. The pattern of correlations both confirms and extends the previously reported aspects of construct validity of SI-S and SI-L scores as a measure of neurological impairment in individuals with MS. Another noteworthy finding was the incremental validity of SI-S and SI-L scores. The SI-S and SI-L scores explained additional variance in EDSS, MSWS-12, and MSIS-29 scores after accounting for PS scores. This provides evidence that SI and PS scores individually explain unique variance in important clinical outcome variables (i.e., disability, ambulation, and quality of life) thereby providing an indication that the two scales are measuring different constructs when delivered concurrently. This study focused on a major concern of previous research, namely further validation of SI-S scores in persons with MS. To that end, there is now evidence supporting the application of the SI-S by clinical researchers for measuring neurological impairment in persons with MS. Nevertheless, there are several directions for future research efforts directed toward validation of SI-S scores. For example, it would be useful to examine the association between subscale scores and objective impairment on MRI in subscale-specific regions of the brain. Another direction involves testing the structural or factorial validity of SI-S scores. This is important for providing an empirical basis for summing item scores into subscale and total scores. Beyond factorial
validity, another direction involves an assessment of the multi-group and longitudinal invariance of SI-S scores. For example, the measurement-model invariance of SI-S scores should be examined between groups differing in disability or disease course and across time or the course of a therapeutic regimen. Such analyses are instrumental for meaningful and unambiguous comparisons and interpretations of mean scores on the SI-S between groups, across time, or as a consequence of an intervention that involves disease-modifying therapies or rehabilitation regimens. An additional direction involves an analysis of the responsiveness of SI-S scores to treatment effects. Such an analysis is critical for establishing the long-term usefulness of the SI-S and calculating effect sizes for clinical trials. One final direction involves comparing results of modern and classical test theory methods for item analysis. Indeed, confirmation of this short form using modern test theory methods would not only be of interest, but could be useful in generating shorter short forms and perhaps a computerized adaptive test of the SI. Overall, we have provided additional and stronger evidence for the validity of SI-S and SI-L scores as a measure of neurological impairment in persons with MS and this supports the viability of important next steps in the validation of the SI tools. We further note that the amount of clinical trials and research related to MS is expanding at a remarkable rate, and patient reported outcomes are becoming more widely used in MS clinical trials and research. Patient reported outcomes are beginning to be perceived by clinicians working in the field as providing equally valid and complementary assessments of outcomes as clinician driven measures. Unfortunately, the field continues to suffer from limited validated outcome assessment tools, particularly those that are sensitive to change. Such observations further support the importance of developing and evaluating outcome assessment tools such as the SI-S and SI-L in persons with MS.
References [1] Hemmer B, Nessler S, Zhou D, Kieseier B, Hartung H-P. Immunopathogenesis and immunotherapy of multiple sclerosis. Nat Clin Pract Neurol 2006;2:201–11. [2] Trapp BD, Nave K-A. Multiple sclerosis: an immune or neurodegenerative disorder? Annu Rev Neurosci 2008;31:247–69. [3] Bjartmar C, Trapp BD. Axonal and neuronal degeneration in multiple sclerosis: mechanisms and functional consequences. Curr Opin Neurol 2001;14:271–8. [4] Schwartz CE, Vollmer T, Lee H. Reliability and validity of two self-report measures of impairment and disability for MS. Neurology 1999;52:63–70. [5] Cronbach LJ, Meehl PE. Construct validation in psychological tests. Psychol Bull 1955;52:281–302. [6] Messick S. Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. Am Psychol 1995;50:741–9. [7] Motl RW, Snook EM, Schapiro RT. Symptoms and physical activity behavior in individuals with multiple sclerosis. Res Nurs Health 2008;31:466–75. [8] Goodin DS. A questionnaire to assess neurological impairment in multiple sclerosis. Mult Scler 1998;4:444–51. [9] Hobart JC, Lamping DL, Fitzpatrick R, Riazi A, Thompson AJ. The Multiple Sclerosis Impact Scale (MSIS-29): a new patient-based outcome measure. Brain 2001;124: 962–73. [10] Riazi A, Hobart JC, Lamping DL, Fitzpatrick R, Thompson A. Multiple Sclerosis Impact Scale (MSIS-29): reliability and validity in hospital based samples. J Neurol Neurosurg Psychiatry 2002;73:701–4. [11] Gosney JL, Scott JA, Snook EM, Motl RW. Physical activity and multiple sclerosis: validity of self-report and objective measures. Fam Commun Health 2007;30:144–50. [12] Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. New Jersey: Lawrence Erlbaum; 1988.
Contents lists available at ScienceDirect
Journal of the Neurological Sciences j o u r n a l h o m e p a g e : w w w. e l s ev i e r. c o m / l o c a t e / j n s
Continued validation of the Symptom Inventory in multiple sclerosis Robert W. Motl a,⁎, Carolyn E. Schwartz b,c, Timothy Vollmer d a
University of Illinois at Urbana-Champaign, Urbana, IL, USA DeltaQuest Foundation, Inc., Concord, MA, USA c Tufts University Medical School, Boston, MA, USA d University of Colorado Health Sciences Center, Denver, CO, USA b
a r t i c l e
i n f o
Article history: Received 21 April 2009 Accepted 11 June 2009 Available online 9 July 2009 Keywords: Multiple sclerosis Neurological impairment Validity Psychometrics
a b s t r a c t Objective: This study examined the construct, discriminant, and incremental validity of scores from the short (SI-S) and long (SI-L) forms of the Symptom Inventory in persons with MS. Methods: The sample included 133 individuals with MS who completed the SI-L, Performance Scales (PS), EDSS, Multiple Sclerosis Walking Scale-12 (MSWS-12), Multiple Sclerosis Impact Scale-29 (MSIS-29), and Godin Leisure-Time Exercise Questionnaire (GLTEQ) and then wore an accelerometer for 7 days. The data were analyzed using SPSS, version 16.0. Results: There were large correlations between SI-S and SI-L total and subscale scores and between SI-S, SI-L, and PS total scores. The correlations were similar in magnitude between SI-S, SI-L, and PS scores with EDSS and MSIS-29 scores, but not with MSWS-12, accelerometer, and GLTEQ scores. Discriminant function analysis indicated that SI-S scores better differentiated groups with minimal, moderate, and severe disability than did SI-L and PS scores. Regression analysis indicated that SI-S and SI-L scores explained incrementally more variance in EDSS, MSWS-12, and MSIS-29 scores after accounting for PS scores alone. Conclusion: Such findings provide additional support for the validity of both SI-S and SI-L scores in individuals with MS and support the adoption of either the SI-S or SI-L by clinical MS researchers. © 2009 Elsevier B.V. All rights reserved.
1. Introduction
evidence that supports the validity of SI-S scores [4]. The present study further examined the construct, discriminant, and incremental validity of scores from the SI-S and SI-L in persons with MS.
Multiple sclerosis (MS) typically begins with intermittent bursts of focal inflammation [1] that result in the demyelination and transection of axons in the central nervous system (CNS) [2] and the accumulation of neurological impairment across time [3]. The study of neurological impairment in MS has been advanced by the development of the Symptom Inventory (SI) [4]. There are long (99 items; SI-L) and short (29 items; SI-S) forms of the SI, and both forms consist of six subscales that were intended to correlate with the localization of brain lesions (visual, left hemisphere, right hemisphere, brain stem and cerebellum, spinal cord, and nonlocalized symptoms) and that can be combined for generating an overall index of neurological impairment. Psychometric analysis has provided preliminary support for the reliability and validity of SI-L and SI-S scores as measures of neurological impairment in individuals with MS [4]. Nevertheless, validation of scores from a measure is an ongoing and evolving process [5,6] and the provision of additional evidence is warranted that further establishes the validity of inferences from SI scores. This is particularly important given the recommendation that clinical MS researchers adopt the SI-L until there is additional
2.2. Measures
⁎ Corresponding author. University of Illinois at Urbana-Champaign, Department of Kinesiology and Community Health, 350 Freer Hall, 906 South Goodwin Ave., Urbana, IL, 61801, USA. Tel.: +1 217 265 0886; fax: +1 217 244 7322. E-mail address: [email protected] (R.W. Motl).
Neurological impairment was measured by the 99-item version of the SI [4]. Neurological disability was measured by the 8-item Performance Scales (PS) [4] and the 15-item self-reported version of the EDSS [8]. Walking ability was measured with the 12-item Multiple
0022-510X/$ – see front matter © 2009 Elsevier B.V. All rights reserved. doi:10.1016/j.jns.2009.06.015
2. Method 2.1. Participants We have described the recruitment and characteristics of the sample elsewhere [7]. Briefly, the sample was recruited through 13 support groups from Midwestern chapters of the National MS Society. There were 140 individuals who were eligible for participation and seven of those individuals did not participate. The final sample consisted of 133 individuals of whom 62% (n = 82) had a relapsing– remitting clinical course and 38% (n = 51) had a progressive clinical course (i.e., primary or secondary progressive). The mean duration since diagnosis was 12 years (SD = 9) and the median Expanded Disability Status Scale (EDSS) score was 5.5 (range = 1–8.5).
R.W. Motl et al. / Journal of the Neurological Sciences 285 (2009) 134–136 Table 1 Construct validity correlations along with 95% confidence intervals among total scores. Measure
Symptom Inventory short form
Symptom Inventory long form
Performance scales
EDSS MSIS-29 MSWS-12 Accelerometer GLTEQ
.68 .71 .73 −.55 −.41
.61 (.49,.71) .74 (.65,.81) .69 (.59,.77) −.56 (−.43, −.67) −.40 (−.25, −.53)
.59 (.47,.69) .69 (.59,.77) .57 (.44,.67) −.39 (−.24, −.53) −.22 (−.05, −.38)
(.58,.76) (.61,.79) (.64,.80) (−.42, −.66) (−.26, −.54)
Note. Values represent r (95% CI). EDSS = Expanded Disability Status Scale; MSWS-12 = Multiple Sclerosis Walking Scale-12; MSIS-29 = Multiple Sclerosis Impact Scale; GLTEQ = Godin Leisure-Time Exercise Questionnaire. Accelerometer = average counts per day.
Sclerosis Walking Scale-12 (MSWS-12) [9]. Quality of life was measured by the 29-item Multiple Sclerosis Impact Scale-29 (MSIS-29) [10]. Physical activity was measured by the ActiGraph single-axis accelerometer (model 7164 version, Health One Technology, Fort Walton Beach, FL) and Godin Leisure-Time Exercise Questionnaire (GLTEQ) [11]. 2.3. Procedure The procedure was approved by a University Institutional Review Board and all participants provided written informed consent. Participants completed a battery of questionnaires that included the SI-L, PS, EDSS, MSWS-12, MSIS-29, and GLTEQ and then wore an accelerometer for a 7-day period. Participants received $10 remuneration for completing the study procedures.
135
CI = .86,.93], p b .0001), Spinal Cord (r = .93 [95% CI = .90,.95], p b .0001), and Nonlocalized (r = .96 [95% CI = .90,.95], p b .0001). 3.1.2. Correlations among overall SI-S, SI-L, and PS scores There was a large correlation between overall scores from the SI-S and PS (r = .71 [95% CI = .61,.79], p b .0001). There was a similarly large correlation between overall scores from the SI-L and PS (r = .73 [95% CI = .64,.80], p b .0001). 3.1.3. Correlations between overall SI-S, SI-L, and PS scores with other measures The correlations along with 95% CIs are provided in Table 1. The correlations were similar in magnitude between SI-S, SI-L, and PS scores with EDSS and MSIS-29 scores. The correlations were larger in magnitude for SI-S and SI-L than PS with MSWS-12, accelerometer, and GLTEQ scores. 3.2. Discriminant validity The mean overall scores for the SI-S, SI-L, and PS as a function of disability level are provided in Table 2. The discriminant function analysis indicated that SI-S scores (eigenvalue = .84) better differentiated groups with minimal, moderate, and severe disability than did SI-L (eigenvalue = .59) and PS (eigenvalue = .68) scores. The SI-S scores clearly distinguished all three groups, whereas the SI-L and PS scores better distinguished the groups with minimal and moderate disability than the groups of moderate and severe disability.
2.4. Data analysis
3.3. Incremental validity
The construct, discriminant, and incremental validity of SI-S and SI-L scores were examined using Pearson correlation, discriminant function, and multiple linear regression analyses in SPSS, version 16.0 (SPSS, Chicago, IL). Construct validity was initially assessed using Pearson correlations (r) and 95% confidence intervals (CI) between (a) SI-S and SI-L subscale and total scores; (b) SI-S, SI-L, and PS total scores; and (c) SI-S, SI-L, and PS total scores with EDSS, MSWS-12, MSIS-29, accelerometer, and GLTEQ scores. Guidelines of .1, .3, and .5 were used for judging the magnitude of the correlations as small, moderate, and large, respectively [12] and 95% CIs demonstrated differences in correlation coefficients. Discriminant validity was assessed using discriminant function analysis. We examined the ability of SI-S, SI-L, and PS scores to distinguish groups with minimal (EDSS score of 1–2.5), moderate (EDSS score of 3–6), and severe (EDSS score of N6) disability. The degree of discrimination was based on eigenvalues, with a larger eigenvalue indicating better differentiation among groups. Incremental validity was assessed using multiple linear regression analysis. We separately regressed EDSS, MSWS-12, and MSIS-29 scores on PS scores entered in Block 1 and SI-S or SI-L scores entered in Block 2. The significance and magnitude of the change in R2 between Blocks (i.e., ΔR2) provided an indication of the incremental variance explained by overall SI-S and SI-L scores beyond that of PS scores alone.
3.3.1. EDSS scores The multiple regression analysis indicated that PS scores explained a large portion of variance in EDSS scores (R2 = .35) and that SI-S scores explained significantly [F(1,121) = 31.03, p b .0001] and incrementally (ΔR2 = .13) more variance in EDSS scores. The multiple regression analysis similarly indicated SI-L scores explained significantly [F(1,121) = 14.25, p b .0001] and incrementally (ΔR2 = .07) more variance in EDSS scores than PS scores alone (R2 = .35).
3. Results
3.3.2. MSIS-29 scores The multiple regression analysis further indicated that PS scores explained a large portion of variance in MSIS-29 scores (R2 = .49) and that SI-S scores explained significantly [F(1,125) = 26.71, p b .0001] and incrementally (ΔR2 = .09) more variance in MSIS-29 scores. The multiple regression analysis similarly indicated that SI-L scores explained significantly [F(1,125) = 33.69, p b .0001] and incrementally (ΔR2 = .11) more variance in MSIS-29 scores than only PS scores (R2 = .49). 3.3.3. MSWS-12 scores The final multiple regression analysis indicated that PS scores explained a large portion of variance in MSWS-12 scores (R2 = .30) and that SI-S scores explained significantly [F(1,109) = 49.88, p b .0001] and incrementally (ΔR2 = .22) more variance in MSWS-12 scores. The multiple regression analysis similarly indicated that SI-S scores explained significantly [F(1,109) = 32.22, p b .0001] and
3.1. Construct validity 3.1.1. Correlations among overall and subscale SI-S and SI-L scores There was a large correlation between overall scores from the SI-S and SI-L (r = .97 [95% CI = .96,.98], p b .0001). There were similarly large correlations between subscale scores from the SI-S and SI-L: Visual (r = .90 [95% CI = .86,.93], p b .0001), Left Hemisphere (r = .83 [95% CI = .77,.88], p b .0001), Right Hemisphere (r = .97 [95% CI = .96,.98], p b .0001), Brainstem and Cerebellum (r = .90 [95%
Table 2 Mean score (SD) by level of disability. Disability level
Symptom Inventory short form
Symptom Inventory long form
Performance scales
EDSS score 1–2.5, n = 29 EDSS score 3–6, n = 69 EDSS score N6, n = 31
4.9 (3.1) 10.0 (3.2) 12.9 (4.1)
4.5 (2.8) 9.0 (3.3) 11.2 (3.1)
8.7 (4.1) 16.8 (5.3) 20.2 (5.0)
Note. EDSS = Expanded Disability Status Scale.
136
R.W. Motl et al. / Journal of the Neurological Sciences 285 (2009) 134–136
incrementally (ΔR2 = .16) more variance in MSWS-12 scores than PS scores alone (R2 = .30). 4. Discussion Validation of scores from a measure is an ongoing and evolving process [5,6] and is particularly important for the Symptom Inventory based on the recommendation that clinical MS researchers adopt the SI-L until there is additional evidence supporting the validity of SI-S scores [4]. To that end, we conducted analyses that both confirmed and extended the validity of inferences from SI-S and SI-L scores in a community-based sample of individuals with MS. Our analyses indicated that (a) there were large correlations between SI-S and SI-L total and subscale scores and between SI-S, SI-L, and PS total scores; (b) the correlations were similar in magnitude between SI-S, SI-L, and PS scores with EDSS and MSIS-29 scores, but not with MSWS-12, accelerometer, and GLTEQ scores; (c) SI-S scores better differentiated groups with minimal, moderate, and severe disability than did SI-L and PS scores; and (d) SI-S and SI-L scores explained incrementally more variance in EDSS, MSWS-12, and MSIS-29 scores after controlling for PS scores. Such findings provide additional support for the validity of both SI-S and SI-L scores in individuals with MS and support the adoption of either the SI-S or SI-L in clinical MS research. Our findings both replicated and extended those of the initial research on the SI. For example, we reported a correlation of r = .97 between overall SI-S and SI-L scores and previous researchers reported a correlation coefficient of r = .99 [4]. We further note that overall SI-S and SI-L scores similarly correlated with PS scores (rs = .71 and .73, respectively) and EDSS scores (rs = .68 and .61, respectively) and this too is consistent with previous research [4]. Nevertheless, there are several noteworthy and unique findings from our analyses of the validity of SI-S and SI-L scores. One noteworthy finding was that overall SI-S, SI-L, and PS scores correlated similarly with EDSS and MSIS-29 scores, whereas there were stronger correlations between SI-S and SI-L scores than PS scores with MSWS-12, accelerometer, and GLTEQ scores. The pattern of correlations both confirms and extends the previously reported aspects of construct validity of SI-S and SI-L scores as a measure of neurological impairment in individuals with MS. Another noteworthy finding was the incremental validity of SI-S and SI-L scores. The SI-S and SI-L scores explained additional variance in EDSS, MSWS-12, and MSIS-29 scores after accounting for PS scores. This provides evidence that SI and PS scores individually explain unique variance in important clinical outcome variables (i.e., disability, ambulation, and quality of life) thereby providing an indication that the two scales are measuring different constructs when delivered concurrently. This study focused on a major concern of previous research, namely further validation of SI-S scores in persons with MS. To that end, there is now evidence supporting the application of the SI-S by clinical researchers for measuring neurological impairment in persons with MS. Nevertheless, there are several directions for future research efforts directed toward validation of SI-S scores. For example, it would be useful to examine the association between subscale scores and objective impairment on MRI in subscale-specific regions of the brain. Another direction involves testing the structural or factorial validity of SI-S scores. This is important for providing an empirical basis for summing item scores into subscale and total scores. Beyond factorial
validity, another direction involves an assessment of the multi-group and longitudinal invariance of SI-S scores. For example, the measurement-model invariance of SI-S scores should be examined between groups differing in disability or disease course and across time or the course of a therapeutic regimen. Such analyses are instrumental for meaningful and unambiguous comparisons and interpretations of mean scores on the SI-S between groups, across time, or as a consequence of an intervention that involves disease-modifying therapies or rehabilitation regimens. An additional direction involves an analysis of the responsiveness of SI-S scores to treatment effects. Such an analysis is critical for establishing the long-term usefulness of the SI-S and calculating effect sizes for clinical trials. One final direction involves comparing results of modern and classical test theory methods for item analysis. Indeed, confirmation of this short form using modern test theory methods would not only be of interest, but could be useful in generating shorter short forms and perhaps a computerized adaptive test of the SI. Overall, we have provided additional and stronger evidence for the validity of SI-S and SI-L scores as a measure of neurological impairment in persons with MS and this supports the viability of important next steps in the validation of the SI tools. We further note that the amount of clinical trials and research related to MS is expanding at a remarkable rate, and patient reported outcomes are becoming more widely used in MS clinical trials and research. Patient reported outcomes are beginning to be perceived by clinicians working in the field as providing equally valid and complementary assessments of outcomes as clinician driven measures. Unfortunately, the field continues to suffer from limited validated outcome assessment tools, particularly those that are sensitive to change. Such observations further support the importance of developing and evaluating outcome assessment tools such as the SI-S and SI-L in persons with MS.
References [1] Hemmer B, Nessler S, Zhou D, Kieseier B, Hartung H-P. Immunopathogenesis and immunotherapy of multiple sclerosis. Nat Clin Pract Neurol 2006;2:201–11. [2] Trapp BD, Nave K-A. Multiple sclerosis: an immune or neurodegenerative disorder? Annu Rev Neurosci 2008;31:247–69. [3] Bjartmar C, Trapp BD. Axonal and neuronal degeneration in multiple sclerosis: mechanisms and functional consequences. Curr Opin Neurol 2001;14:271–8. [4] Schwartz CE, Vollmer T, Lee H. Reliability and validity of two self-report measures of impairment and disability for MS. Neurology 1999;52:63–70. [5] Cronbach LJ, Meehl PE. Construct validation in psychological tests. Psychol Bull 1955;52:281–302. [6] Messick S. Validity of psychological assessment: validation of inferences from persons' responses and performances as scientific inquiry into score meaning. Am Psychol 1995;50:741–9. [7] Motl RW, Snook EM, Schapiro RT. Symptoms and physical activity behavior in individuals with multiple sclerosis. Res Nurs Health 2008;31:466–75. [8] Goodin DS. A questionnaire to assess neurological impairment in multiple sclerosis. Mult Scler 1998;4:444–51. [9] Hobart JC, Lamping DL, Fitzpatrick R, Riazi A, Thompson AJ. The Multiple Sclerosis Impact Scale (MSIS-29): a new patient-based outcome measure. Brain 2001;124: 962–73. [10] Riazi A, Hobart JC, Lamping DL, Fitzpatrick R, Thompson A. Multiple Sclerosis Impact Scale (MSIS-29): reliability and validity in hospital based samples. J Neurol Neurosurg Psychiatry 2002;73:701–4. [11] Gosney JL, Scott JA, Snook EM, Motl RW. Physical activity and multiple sclerosis: validity of self-report and objective measures. Fam Commun Health 2007;30:144–50. [12] Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. New Jersey: Lawrence Erlbaum; 1988.