- Email: [email protected]
Psychometric validation of the O’leary-Sant interstitial cystitis symptom index in a clinical trial of pentosan polysulfate sodium
PSYCHOMETRIC VALIDATION OF THE O’LEARY-SANT INTERSTITIAL CYSTITIS SYMPTOM INDEX IN A CLINICAL TRIAL OF PENTOSAN POLYSULFATE SODIUM DEBORAH P. LUBECK, KRISTENE WHITMORE, GRANNUM R. SANT, SARAH ALVAREZ-HORINE, AND CHINGLIN LAI
ABSTRACT The O’Leary-Sant Interstitial Cystitis Symptom Index (ICSI) has been proposed as a treatment outcome measure in interstitial cystitis (IC). The psychometric properties of the ICSI were assessed for reliability and validity in a randomized, double-blind clinical study of 300, 600, and 900 mg daily dose of pentosan polysulfate sodium (PPS) in patients with IC. The ICSI contains 4 items that measure urgency and frequency of urination, nighttime urination, and pain or burning. The ICSI index score is the sum of the item scores (range: 0–20). ICSI scores were obtained at baseline, 4, 8, 12, 16, 24, and 32 weeks of treatment. Patients’ overall ratings of improvement of symptoms (PORIS) scores evaluating improvements in pain, urgency, and overall IC symptoms were also collected except at baseline. A total of 376 patients were included in the analysis. Psychometric properties evaluated included variability (range), test–retest reliability (intraclass correlation coefficient [ICC]), internal consistency (the Cronbach ␣), construct validity (convergent, discriminant), responsiveness, and clinically meaningful change. The ICSI items and index score had good variability and test–retest reliability. The ICSI demonstrated internal consistency reliability and was responsive to change. Participants indicating a 75% improvement in PORIS had a 48% mean reduction in the ICSI score, while participants reporting 100% improvement in PORIS had a 77% mean reduction in the ICSI score. The ICSI is a valid, reliable, and responsive measure of change in IC symptoms. This outcome measure should be utilized in future treatment outcomes studies in IC. UROLOGY 57 (Suppl 6A): 62–66, 2001. © 2001, Elsevier Science Inc.
I
nterstitial cystitis (IC) is a chronic, debilitating bladder disease characterized by pelvic pain, urinary frequency, and urgency. The disease has significant impact on patients’ quality-of-life as a result of the severity of irritative voiding symptoms and chronic pelvic or lower abdominal pain. Although urologists are cognizant of the impact IC has on patients, standardized assessment and treatment outcome measures have been lacking. For
From the Department of Urology, University of California, San Francisco, California; Graduate Hospital, Philadelphia, Pennsylvania; the Department of Urology, Tufts University School of Medicine, Boston, Massachusetts; and ALZA Corporation, Mountain View, California, USA This study was funded by ALZA Corporation, Mountain View California. Author D.P. Lubeck is a paid consultant to Alza Pharmaceuticals, the sponsor of this supplement. Reprint requests: Deborah P. Lubeck, PhD, Department of Urology, University of California, San Francisco, Box 1319, San Francisco, California 94143-1319. E-mail: [email protected]
62
© 2001, ELSEVIER SCIENCE INC. ALL RIGHTS RESERVED
research on IC, it is important to have a validated treatment outcome measure. The O’Leary-Sant Symptom Index (ICSI) was proposed in 1997 as a uniform outcome measure in IC. O’Leary et al.1 reported on the development of the ICSI and a corresponding problem index, the Interstitial Cystitis Problem Index (ICPI), as measures of lower urinary tract symptoms and their quality-of-life impact on patients with IC. Initial development included a literature search and pretesting questions with volunteers recruited from the Interstitial Cystitis Association. Initial validation of the instrument was conducted among 45 newly diagnosed patients with IC from 3 urology practices in the United States.1 A comparison group of 67 women presenting for routine gynecologic care and with no voiding symptoms also completed the instrument. Analyses resulted in a questionnaire with 8 items sum0090-4295/01/$20.00 PII S0090-4295(01)01126-8
marized in 2 indices: a symptom index (ICSI) and a problem index (ICPI).1 All questions asked about the respondent’s experience over the prior 4 weeks. Although the ICSI has been tested with a community population and a convenience sample of patients with IC, it has not yet been validated in prospective treatment studies. In this article, we describe the validation of the ICSI in a randomized, double-blind study of 3 different daily doses (300, 600, and 900 mg) of pentosan polysulfate sodium (PPS) in patients with IC. MATERIALS AND METHODS STUDY DESIGN The 4-item ICSI was included in a clinical study of 3 daily doses of PPS, and the psychometric properties of the instrument were assessed.2 The other outcome measures included in the validation analyses were: the Patient Overall Rating of Improvement of Symptoms (PORIS), the Medical Outcomes Study (MOS) Sleep Scale,3 the MOS Sexual Functioning Scale,4 and the SF-12 Physical Component Summary (PCS) and Mental Component Summary scores (MCS).5 The ICSI and the SF-12 MCS and PCS were completed at baseline and at weeks 4, 8, 12, 16, 24, and 32 of the study. Because PORIS was used to assess symptom change from baseline, it was completed at weeks 4, 8, 12, 16, 24, and 32. Study participants completed the MOS Sexual Functioning and Sleep Scales at baseline and at weeks 8, 16, 24, and 32. The study endpoint was defined as the last, non-missing, postbaseline score. For patients who completed the study without missing data, the endpoint was week 32. For patients who completed the study with missing data or who did not complete the study, the endpoint was the last, non-missing ICSI visit. A total of 376 patients were included in the validation analyses and blinding was maintained.
STATISTICAL APPROACH The following properties of the ICSI were evaluated: (1) variability; (2) test–retest reliability; (3) internal consistency reliability; (4) construct validity; (5) responsiveness; and (6) clinically meaningful change.2 The criteria for validation were those recommended by the Medical Outcomes Trust.2 Variability refers to the extent to which the full range of item responses and scale scores is observed in the data. Optimal variability suggests patient responses at both ends of the scale as well as the middle. Scales that are skewed, either positively or negatively, tend to be less responsive to change in disease progression. Test–retest reliability refers to the stability of responses to repeated measures of the same question. The intraclass correlation coefficient (ICC) was used to assess test–retest reliability. The ICC ranges from 0.0 to 1.0, with 1.0 equal to complete agreement. The time frame for test–retest evaluation is usually a period when there is no treatment or intervention, during which no change would be expected to occur.2 The schedule for this clinical study did not permit 2 administrations of the ICSI before start of treatment. Therefore, test–retest reliability was assessed using participant responses ⱕ30 days apart (ie, baseline and 4-week responses) for those individuals who reported that their symptoms were “no better” at week 4 in response to PORIS question 3. This question asks respondents to best describe the overall change in their IC since the start of the study as “worse,” “no better,” “slightly improved,” “moderately improved,” “greatly improved,” or “symptoms gone.” UROLOGY 57 (Supplement 6A), June 2001
Internal consistency reliability measures the homogeneity of items in a scale, and the extent to which the scale is free of random error.6 The Cronbach coefficient ␣ is the statistic for reporting internal-consistency reliability.7 The coefficient may range from a low of 0.0 to a maximum of 1.0, with the desired range of scores between 0.70 and 0.95.8 Construct validity examines correlations between scale scores based on known relations. For example, moderate-tostrong correlations (ie, ⱖ0.5) are expected for similar items or scales. We evaluated the correlation between the individual ICSI items and the ICSI score and each of the following: (1) the MOS Sleep Scale; (2) the MOS Sexual Functioning Scale; (3) the SF-12 PCS; (4) the SF-12 MCS; and (5) a single item on the SF-12 that measures the extent to which pain interfered with usual activities. Responsiveness refers to an instrument’s ability to detect improvement or deterioration that results from therapy or disease progression. If the ICSI is responsive, participants who have an indication of clinical change should have a parallel change in their ICSI scores and participants who show no indication of clinical change should have stable scores. Responsiveness is measured by Guyatt’s statistic, which is calculated as the mean change in score (for each group) divided by the standard deviation for those individuals who remain stable. Study participants were divided into 3 groups: (1) those who improved; (2) those who declined; and (3) those who remained stable based on the response to PORIS question 3 at endpoint. A rating of “worse” on PORIS question 3 classified patients as decliners; a rating of “no better” or “slight improvement” denoted stable patients; and a rating of “moderately improved, ” “greatly improved,” or “symptoms gone” denoted improvers. Evaluation of clinically meaningful change on the ICSI was determined on the basis of minimally important differences in the PORIS. Therefore, the absolute change from baseline to endpoint and percent change from baseline to endpoint for the ICSI were evaluated in 3 groups of individuals. These groups were defined as those respondents who indicated they had a moderate improvement in symptoms (50% improvement), they were greatly improved (75% improvement), or their symptoms were gone (100% improvement). Each increment is considered a beneficial improvement in symptoms.
RESULTS As reported in Table I, the study participants were predominantly female, white, and had been diagnosed with IC for a median of 11 months. The mean (SD) age of the participants was 44.1 (14.2) years (range 17– 84 years). Participants experienced an average of 13.2 (6.3) voids a day at baseline. There were no statistically significant differences among demographic and clinical variables at baseline across the 3 treatment groups. Variability of the 4 item scores and the ICSI index score at baseline and at the study endpoint is shown in Table II. Optimal variability was observed for the 4 items at both time points. At baseline, no patients reported the lowest scores on the ICSI (ie, 0, 1, or 2 of 20); this was expected because low scores would indicate that respondents had no IC symptoms. At endpoint, the full range of responses was observed for the ICSI. Test–retest reliability results as measured by the ICC are reported in Table III for 67 patients who 63
TABLE I. Demographics and baseline characteristics: all randomized patients in the validation analysis* Treatment Group
Characteristic Age (yr) Mean (SD) Median Range Sex, n (%) Male Female Race, n (%) White Other Months from diagnosis of IC Mean (SD) Median Average voids in 24 hr Mean (SD) Median
PPS 300 mg/day (n ⴝ 124)
PPS 600 mg/day (n ⴝ 125)
PPS 900 mg/day (n ⴝ 127)
All Patients (n ⴝ 376)
44.8 (13.9) 43.0 (19–84)
43.5 (14.0) 45.0 (19–80)
44.2 (14.7) 43.0 (17–81)
44.1 (14.2) 43.5 (17–84)
11 (8.9) 113 (91.1)
15 (12.0) 110 (88.0)
11 (8.7) 116 (91.3)
37 (9.8) 339 (90.2)
112 (90.3) 12 (9.6)
116 (92.8) 9 (7.2)
118 (92.9) 9 (7.1)
346 (92.0) 30 (7.9)
37.5 (58.3) 9.5
39.2 (62.0) 8.5
40.9 (53.7) 14.0
39.2 (57.9) 11.0
13.0 (5.3) 11.5
13.4 (5.2) 12.0
13.4 (7.9) 12.0
13.2 (6.3) 12.0
IC ⫽ interstitial cystitis; PPS ⫽ pentosan polysulfate sodium. * Demographics were summarized after study unblinding. Validation analysis was performed before unblinding.
TABLE II. Descriptive statistics for the Interstitial Cystitis Symptom Index (ICSI) items and score at baseline and endpoint Index/Item Baseline ICSI score‡ Urinate with little or no warning Urinate ⬍2 hr after you finished urinating Frequency getting up at night to urinate Experienced pain or burning in your bladder Endpoint ICSI score Urinate with little or no warning Urinate ⬍2 hr after you finished urinating Frequency getting up at night to urinate Experienced pain or burning in your bladder
% Floor*
% Ceiling†
n
Range
Mean
Median
SD
376
3–20
12.9
13
3.5
0.0
2.1
376 376 376 376 367
0–5 0–5 0–5 0–5 0–20
2.9 4.0 2.8 3.1 9.8
3 4 3 3 10
1.5 1.1 1.4 1.4 4.6
6.4 0.3 2.7 8.2 0.3
18.6 44.4 14.9 21.0 2.5
367 367 367 367
0–5 0–5 0–5 0–5
2.1 3.1 2.1 2.4
2 3 2 2
1.5 1.5 1.4 1.5
11.7 2.2 7.6 16.9
9.0 25.9 9.8 11.7
* Percent of subjects who scored the lowest possible score. † Percent of subjects who scored the highest possible score. ‡ Possible values for the ICSI score range from 0 to 20, with a higher score representing worse functioning.
completed the ICSI ⱕ30 days apart and who indicated their symptoms were “no better.” The ICC for the ICSI score was 0.80, indicating excellent reproducibility.6 The individual items of the ICSI also had good-to-excellent reproducibility (ICC range: 0.57– 0.94). The Cronbach coefficient ␣, a measure of internal consistency reliability, is shown in Table III. The coefficient was 0.72 for the ICSI score, which meets the criterion for good internal consistency reliability.2 Construct validity is addressed in Table IV, which lists the correlations between the ICSI individual item and index scores and the MOS Sleep 64
and Sexual Functioning scales, the SF-12 PCS, and MCS, and a single SF-12 item measuring the extent to which pain interfered with usual activities. The strongest correlations were observed among individual ICSI items and the ICSI score, as would be expected. Other strong correlations were observed between the single item measuring pain interference with activities and each of the following: (1) the ICSI score (0.489); (2) the ICSI item that measures bladder pain (0.506); and (3) the SF-12 PCS (⫺0.764). These correlations were also expected to be strong because the items and the PCS measure similar concepts. The relation between the SF-12 pain item and the PCS score was reversed because UROLOGY 57 (Supplement 6A), June 2001
TABLE III. Intraclass correlation coefficient and Cronbach ␣ for the Interstitial Cystitis Symptom Index (ICSI) Psychometric Property Evaluated Test-retest reliability*
Internal consistency
Measure Intraclass correlation coefficient
Sample Size
Index/Domain Assessed ICSI score Urinate with little or no warning Urinate ⬍2 hr after you finished urinating Frequency getting up at night to urinate Experienced pain or burning in your bladder ICSI score
Cronbach ␣†
Test Value
67
0.80
67 67 67 67 743
0.57 0.65 0.94 0.66 0.72
* Test-retest scores are reported for subjects with responses ⱕ30 days between baseline and week 4 who answered “no better” on Patients’ Overall Ratings of Improvement of Symptoms (PORIS) question 3 at week 4. Question 3 asks respondents to indicate overall change in IC compared with before start of study as “worse,” “no better,” “slightly improved,” “moderately improved,” “greatly improved,” or “symptoms gone.” † The Cronbach ␣ is reported for pooled responses at baseline and endpoint.
TABLE IV. Interscale correlations for the Interstitial Cystitis Symptom Index (ICSI) items and score with the SF-12, Medical Outcomes Study (MOS) Sleep, and MOS Sexual Functioning scales Scale/Item ICSI score* ICSI item 1 ICSI item 2 ICSI item 3 ICSI item 4 SF-12 PCS SF-12 MCS SF-12 pain item MOS Sleep Scale MOS Sexual Function Scale
ICSI
Item 1
Item 2
Item 3
Item 4
0.761† 0.793† 0.713† 0.690† ⫺0.395† ⫺0.218† 0.489† ⫺0.411† ⫺0.222†
0.494† 0.319† 0.402† ⫺0.294† ⫺0.246† 0.369† ⫺0.299† ⫺0.197†
0.538† 0.347† ⫺0.258† ⫺0.152† 0.285† ⫺0.295† ⫺0.100
0.275† ⫺0.243† ⫺0.036 0.273† ⫺0.317† ⫺0.070
PCS
MCS
Pain
Sleep
⫺0.367† ⫺0.201† 0.158† † 0.506 ⫺0.764† ⫺0.368† ⫺0.305† 0.402† 0.439† ⫺0.415† † † ⫺0.278 0.289 0.360† ⫺0.329† 0.264†
MCS ⫽ Mental Component Summary; PCS ⫽ Physical Component Summary. * ICSI items measure the following: 1, urinary urgency; 2, urinary frequency; 3, nocturia; 4, bladder pain. † P ⬍0.001, indicates statistically significant correlation.
TABLE V. Change from baseline to endpoint and responsiveness to change as measured by the Guyatt statistic by the Patient Overall Rating of Improvement Symptoms (PORIS) response group Response Group Declining respondents Stable respondents Improving respondents
Sample Size
ICSI Change Score
Guyatt Statistic
12 184 168
2.25 ⫺1.47 ⫺5.38
0.69 ⫺0.45 ⫺1.64
ICSI ⫽ Interstitial Cystitis Symptom Index.
the PCS score was calculated such that a higher score equaled more optimal functioning. Weaker correlations were observed among the ICSI and the SF-12 scales, as would be expected because they measure different domains. Information on responsiveness to change is shown in Table V. Only 12 participants indicated a rating of “worse.” For these individuals, mean change in the ICSI score was 2.25 and the Guyatt statistic was 0.69. A small number of decliners are to be expected in dose-ranging studies. There were 184 patients who indicated no improvement (ie, they remained clinically stable). The mean ICSI UROLOGY 57 (Supplement 6A), June 2001
change score for this group was ⫺1.47 and the Guyatt statistic was ⫺0.45. This value is within the expected range for the Guyatt statistic for clinically stable patients. For patients defined as moderately or greatly improved, mean ICSI change was ⫺5.38 with an associated Guyatt statistic of ⫺1.64. A statistic with an absolute value ⱖ1.00 or greater is considered indicative of a measure highly responsive to change, as observed here. Results are reported in Table VI for clinically meaningful change. There was approximately a 2-point improvement in the ICSI score for each 65
TABLE VI. Change and percent change in Interstitial Cystitis Symptom Index (ICSI) score from baseline to endpoint based on improvement in Patient Overall Rating of Improvement of Symptoms (PORIS) question 3 at endpoint PORIS at Endpoint
n Mean change in ICSI score SD % change from baseline to endpoint
Moderately Improved (50% improvement)
Greatly Improved (75% improvement)
Symptoms Gone (100% improvement)
76 ⫺4.03 3.5 ⫺29.5
81 ⫺6.21 4.1 ⫺48.5
11 ⫺8.55 3.9 ⫺77.0
level of improvement in the PORIS. This rate of change, which indicates that the ICSI is associated with clinically meaningful change, was confirmed by the regression equation for the change in the ICSI score versus the PORIS: Change in ICSI Index Score ⫽ ⫺1.97 ⫻ PORIS ⫹ 3.67. CONCLUSIONS The ICSI has been previously validated in a community population and a convenience sample of patients with IC. In that work, the instrument was found to be stable and to have good internal consistency reliability. Information on responsiveness to change was not reported by the instrument developers. This report evaluated the psychometric properties of the ICSI in a prospective clinical drug trial in patients with IC. A total of 376 patients randomized to 1 of 3 treatment groups were included in the evaluation of the instrument, and blinding was maintained. In the trial, the ICSI individual items and score had good variability. Test–retest reliability over a 4-week period was good, and the scale had good internal consistency reliability. Evaluation of the construct validity demonstrated that the individual ICSI items correlated highly with the ICSI score. The pain item on the ICSI also correlated well with a pain item on the SF-12. Weaker correlations were observed among the ICSI items and score and the general scales measuring physical and emotional functioning, sleep, and sexual functioning. It is important that the ICSI be responsive to clinical improvement so that it may be used as an outcome in clinical trails. Responsiveness to change was evaluated using a global, patient-generated measure (PORIS). Patients who indicated an improvement in symptoms had a Guyatt statistic of
66
⫺1.64, which indicates that the instrument is responsive to change. Correspondingly the patients who indicated no change in symptoms had a Guyatt statistic of ⫺0.45, within the expected range for a stable population. For every 1-point improvement in the PORIS score, an average improvement of 1.97 was observed in the ICSI score, indicating that the ICSI is associated with clinically meaningful change. The ICSI is a valid, reliable, and responsive measure of IC symptom impact. This outcome instrument should have great utility and application in future cross-sectional and longitudinal studies of treatment (pharmacologic and nonpharmacologic) outcomes in patients with IC. REFERENCES 1. O’Leary MP, Sant GR, Fowler FJ, et al: The interstitial cystitis symptom index and problem index. Urology 49(suppl 5A): 58 – 63, 1997. 2. Lohr KN, Aaronson NK, Alonso J, et al: Evaluating quality-of-life and health status instruments: development of scientific review criteria. Clin Ther 18(5): 979 –992, 1996. 3. Hays RD, and Stewart AL: Sleep measures, in Stewart AL, and Ware JE (Eds), Measuring Functioning and Well-Being: The Medical Outcomes Study Approach. Durham, NC, Duke University Press, 1992 pp 235–259. 4. Sherbourne CD: Social functioning: sexual problem measures, in Stewart AL, and Ware JE (Eds), Measuring Functioning and Well-Being: The Medical Outcomes Study Approach. Durham, NC, Duke University Press, 1992, pp 194 –204. 5. Ware JE, Kosinski M, and Keller SD: SF-12: How to Score the SF-12 Physical and Mental Health Summary Scales. Boston, The Health Institute, New England Medical Center, 1995. 6. Rosner B: Fundamentals of Biostatistics, 4th ed. Belmont CA, Duxbury Press, 1994, pp 423– 426. 7. Aday LA: Designing and Conducting Health Surveys. San Francisco, Jossey-Bass, 1989, pp 50 –58. 8. Dunn G: Design and Analysis of Reliability Studies. New York, Oxford University Press, 1989, pp 47–50.
UROLOGY 57 (Supplement 6A), June 2001
ABSTRACT The O’Leary-Sant Interstitial Cystitis Symptom Index (ICSI) has been proposed as a treatment outcome measure in interstitial cystitis (IC). The psychometric properties of the ICSI were assessed for reliability and validity in a randomized, double-blind clinical study of 300, 600, and 900 mg daily dose of pentosan polysulfate sodium (PPS) in patients with IC. The ICSI contains 4 items that measure urgency and frequency of urination, nighttime urination, and pain or burning. The ICSI index score is the sum of the item scores (range: 0–20). ICSI scores were obtained at baseline, 4, 8, 12, 16, 24, and 32 weeks of treatment. Patients’ overall ratings of improvement of symptoms (PORIS) scores evaluating improvements in pain, urgency, and overall IC symptoms were also collected except at baseline. A total of 376 patients were included in the analysis. Psychometric properties evaluated included variability (range), test–retest reliability (intraclass correlation coefficient [ICC]), internal consistency (the Cronbach ␣), construct validity (convergent, discriminant), responsiveness, and clinically meaningful change. The ICSI items and index score had good variability and test–retest reliability. The ICSI demonstrated internal consistency reliability and was responsive to change. Participants indicating a 75% improvement in PORIS had a 48% mean reduction in the ICSI score, while participants reporting 100% improvement in PORIS had a 77% mean reduction in the ICSI score. The ICSI is a valid, reliable, and responsive measure of change in IC symptoms. This outcome measure should be utilized in future treatment outcomes studies in IC. UROLOGY 57 (Suppl 6A): 62–66, 2001. © 2001, Elsevier Science Inc.
I
nterstitial cystitis (IC) is a chronic, debilitating bladder disease characterized by pelvic pain, urinary frequency, and urgency. The disease has significant impact on patients’ quality-of-life as a result of the severity of irritative voiding symptoms and chronic pelvic or lower abdominal pain. Although urologists are cognizant of the impact IC has on patients, standardized assessment and treatment outcome measures have been lacking. For
From the Department of Urology, University of California, San Francisco, California; Graduate Hospital, Philadelphia, Pennsylvania; the Department of Urology, Tufts University School of Medicine, Boston, Massachusetts; and ALZA Corporation, Mountain View, California, USA This study was funded by ALZA Corporation, Mountain View California. Author D.P. Lubeck is a paid consultant to Alza Pharmaceuticals, the sponsor of this supplement. Reprint requests: Deborah P. Lubeck, PhD, Department of Urology, University of California, San Francisco, Box 1319, San Francisco, California 94143-1319. E-mail: [email protected]
62
© 2001, ELSEVIER SCIENCE INC. ALL RIGHTS RESERVED
research on IC, it is important to have a validated treatment outcome measure. The O’Leary-Sant Symptom Index (ICSI) was proposed in 1997 as a uniform outcome measure in IC. O’Leary et al.1 reported on the development of the ICSI and a corresponding problem index, the Interstitial Cystitis Problem Index (ICPI), as measures of lower urinary tract symptoms and their quality-of-life impact on patients with IC. Initial development included a literature search and pretesting questions with volunteers recruited from the Interstitial Cystitis Association. Initial validation of the instrument was conducted among 45 newly diagnosed patients with IC from 3 urology practices in the United States.1 A comparison group of 67 women presenting for routine gynecologic care and with no voiding symptoms also completed the instrument. Analyses resulted in a questionnaire with 8 items sum0090-4295/01/$20.00 PII S0090-4295(01)01126-8
marized in 2 indices: a symptom index (ICSI) and a problem index (ICPI).1 All questions asked about the respondent’s experience over the prior 4 weeks. Although the ICSI has been tested with a community population and a convenience sample of patients with IC, it has not yet been validated in prospective treatment studies. In this article, we describe the validation of the ICSI in a randomized, double-blind study of 3 different daily doses (300, 600, and 900 mg) of pentosan polysulfate sodium (PPS) in patients with IC. MATERIALS AND METHODS STUDY DESIGN The 4-item ICSI was included in a clinical study of 3 daily doses of PPS, and the psychometric properties of the instrument were assessed.2 The other outcome measures included in the validation analyses were: the Patient Overall Rating of Improvement of Symptoms (PORIS), the Medical Outcomes Study (MOS) Sleep Scale,3 the MOS Sexual Functioning Scale,4 and the SF-12 Physical Component Summary (PCS) and Mental Component Summary scores (MCS).5 The ICSI and the SF-12 MCS and PCS were completed at baseline and at weeks 4, 8, 12, 16, 24, and 32 of the study. Because PORIS was used to assess symptom change from baseline, it was completed at weeks 4, 8, 12, 16, 24, and 32. Study participants completed the MOS Sexual Functioning and Sleep Scales at baseline and at weeks 8, 16, 24, and 32. The study endpoint was defined as the last, non-missing, postbaseline score. For patients who completed the study without missing data, the endpoint was week 32. For patients who completed the study with missing data or who did not complete the study, the endpoint was the last, non-missing ICSI visit. A total of 376 patients were included in the validation analyses and blinding was maintained.
STATISTICAL APPROACH The following properties of the ICSI were evaluated: (1) variability; (2) test–retest reliability; (3) internal consistency reliability; (4) construct validity; (5) responsiveness; and (6) clinically meaningful change.2 The criteria for validation were those recommended by the Medical Outcomes Trust.2 Variability refers to the extent to which the full range of item responses and scale scores is observed in the data. Optimal variability suggests patient responses at both ends of the scale as well as the middle. Scales that are skewed, either positively or negatively, tend to be less responsive to change in disease progression. Test–retest reliability refers to the stability of responses to repeated measures of the same question. The intraclass correlation coefficient (ICC) was used to assess test–retest reliability. The ICC ranges from 0.0 to 1.0, with 1.0 equal to complete agreement. The time frame for test–retest evaluation is usually a period when there is no treatment or intervention, during which no change would be expected to occur.2 The schedule for this clinical study did not permit 2 administrations of the ICSI before start of treatment. Therefore, test–retest reliability was assessed using participant responses ⱕ30 days apart (ie, baseline and 4-week responses) for those individuals who reported that their symptoms were “no better” at week 4 in response to PORIS question 3. This question asks respondents to best describe the overall change in their IC since the start of the study as “worse,” “no better,” “slightly improved,” “moderately improved,” “greatly improved,” or “symptoms gone.” UROLOGY 57 (Supplement 6A), June 2001
Internal consistency reliability measures the homogeneity of items in a scale, and the extent to which the scale is free of random error.6 The Cronbach coefficient ␣ is the statistic for reporting internal-consistency reliability.7 The coefficient may range from a low of 0.0 to a maximum of 1.0, with the desired range of scores between 0.70 and 0.95.8 Construct validity examines correlations between scale scores based on known relations. For example, moderate-tostrong correlations (ie, ⱖ0.5) are expected for similar items or scales. We evaluated the correlation between the individual ICSI items and the ICSI score and each of the following: (1) the MOS Sleep Scale; (2) the MOS Sexual Functioning Scale; (3) the SF-12 PCS; (4) the SF-12 MCS; and (5) a single item on the SF-12 that measures the extent to which pain interfered with usual activities. Responsiveness refers to an instrument’s ability to detect improvement or deterioration that results from therapy or disease progression. If the ICSI is responsive, participants who have an indication of clinical change should have a parallel change in their ICSI scores and participants who show no indication of clinical change should have stable scores. Responsiveness is measured by Guyatt’s statistic, which is calculated as the mean change in score (for each group) divided by the standard deviation for those individuals who remain stable. Study participants were divided into 3 groups: (1) those who improved; (2) those who declined; and (3) those who remained stable based on the response to PORIS question 3 at endpoint. A rating of “worse” on PORIS question 3 classified patients as decliners; a rating of “no better” or “slight improvement” denoted stable patients; and a rating of “moderately improved, ” “greatly improved,” or “symptoms gone” denoted improvers. Evaluation of clinically meaningful change on the ICSI was determined on the basis of minimally important differences in the PORIS. Therefore, the absolute change from baseline to endpoint and percent change from baseline to endpoint for the ICSI were evaluated in 3 groups of individuals. These groups were defined as those respondents who indicated they had a moderate improvement in symptoms (50% improvement), they were greatly improved (75% improvement), or their symptoms were gone (100% improvement). Each increment is considered a beneficial improvement in symptoms.
RESULTS As reported in Table I, the study participants were predominantly female, white, and had been diagnosed with IC for a median of 11 months. The mean (SD) age of the participants was 44.1 (14.2) years (range 17– 84 years). Participants experienced an average of 13.2 (6.3) voids a day at baseline. There were no statistically significant differences among demographic and clinical variables at baseline across the 3 treatment groups. Variability of the 4 item scores and the ICSI index score at baseline and at the study endpoint is shown in Table II. Optimal variability was observed for the 4 items at both time points. At baseline, no patients reported the lowest scores on the ICSI (ie, 0, 1, or 2 of 20); this was expected because low scores would indicate that respondents had no IC symptoms. At endpoint, the full range of responses was observed for the ICSI. Test–retest reliability results as measured by the ICC are reported in Table III for 67 patients who 63
TABLE I. Demographics and baseline characteristics: all randomized patients in the validation analysis* Treatment Group
Characteristic Age (yr) Mean (SD) Median Range Sex, n (%) Male Female Race, n (%) White Other Months from diagnosis of IC Mean (SD) Median Average voids in 24 hr Mean (SD) Median
PPS 300 mg/day (n ⴝ 124)
PPS 600 mg/day (n ⴝ 125)
PPS 900 mg/day (n ⴝ 127)
All Patients (n ⴝ 376)
44.8 (13.9) 43.0 (19–84)
43.5 (14.0) 45.0 (19–80)
44.2 (14.7) 43.0 (17–81)
44.1 (14.2) 43.5 (17–84)
11 (8.9) 113 (91.1)
15 (12.0) 110 (88.0)
11 (8.7) 116 (91.3)
37 (9.8) 339 (90.2)
112 (90.3) 12 (9.6)
116 (92.8) 9 (7.2)
118 (92.9) 9 (7.1)
346 (92.0) 30 (7.9)
37.5 (58.3) 9.5
39.2 (62.0) 8.5
40.9 (53.7) 14.0
39.2 (57.9) 11.0
13.0 (5.3) 11.5
13.4 (5.2) 12.0
13.4 (7.9) 12.0
13.2 (6.3) 12.0
IC ⫽ interstitial cystitis; PPS ⫽ pentosan polysulfate sodium. * Demographics were summarized after study unblinding. Validation analysis was performed before unblinding.
TABLE II. Descriptive statistics for the Interstitial Cystitis Symptom Index (ICSI) items and score at baseline and endpoint Index/Item Baseline ICSI score‡ Urinate with little or no warning Urinate ⬍2 hr after you finished urinating Frequency getting up at night to urinate Experienced pain or burning in your bladder Endpoint ICSI score Urinate with little or no warning Urinate ⬍2 hr after you finished urinating Frequency getting up at night to urinate Experienced pain or burning in your bladder
% Floor*
% Ceiling†
n
Range
Mean
Median
SD
376
3–20
12.9
13
3.5
0.0
2.1
376 376 376 376 367
0–5 0–5 0–5 0–5 0–20
2.9 4.0 2.8 3.1 9.8
3 4 3 3 10
1.5 1.1 1.4 1.4 4.6
6.4 0.3 2.7 8.2 0.3
18.6 44.4 14.9 21.0 2.5
367 367 367 367
0–5 0–5 0–5 0–5
2.1 3.1 2.1 2.4
2 3 2 2
1.5 1.5 1.4 1.5
11.7 2.2 7.6 16.9
9.0 25.9 9.8 11.7
* Percent of subjects who scored the lowest possible score. † Percent of subjects who scored the highest possible score. ‡ Possible values for the ICSI score range from 0 to 20, with a higher score representing worse functioning.
completed the ICSI ⱕ30 days apart and who indicated their symptoms were “no better.” The ICC for the ICSI score was 0.80, indicating excellent reproducibility.6 The individual items of the ICSI also had good-to-excellent reproducibility (ICC range: 0.57– 0.94). The Cronbach coefficient ␣, a measure of internal consistency reliability, is shown in Table III. The coefficient was 0.72 for the ICSI score, which meets the criterion for good internal consistency reliability.2 Construct validity is addressed in Table IV, which lists the correlations between the ICSI individual item and index scores and the MOS Sleep 64
and Sexual Functioning scales, the SF-12 PCS, and MCS, and a single SF-12 item measuring the extent to which pain interfered with usual activities. The strongest correlations were observed among individual ICSI items and the ICSI score, as would be expected. Other strong correlations were observed between the single item measuring pain interference with activities and each of the following: (1) the ICSI score (0.489); (2) the ICSI item that measures bladder pain (0.506); and (3) the SF-12 PCS (⫺0.764). These correlations were also expected to be strong because the items and the PCS measure similar concepts. The relation between the SF-12 pain item and the PCS score was reversed because UROLOGY 57 (Supplement 6A), June 2001
TABLE III. Intraclass correlation coefficient and Cronbach ␣ for the Interstitial Cystitis Symptom Index (ICSI) Psychometric Property Evaluated Test-retest reliability*
Internal consistency
Measure Intraclass correlation coefficient
Sample Size
Index/Domain Assessed ICSI score Urinate with little or no warning Urinate ⬍2 hr after you finished urinating Frequency getting up at night to urinate Experienced pain or burning in your bladder ICSI score
Cronbach ␣†
Test Value
67
0.80
67 67 67 67 743
0.57 0.65 0.94 0.66 0.72
* Test-retest scores are reported for subjects with responses ⱕ30 days between baseline and week 4 who answered “no better” on Patients’ Overall Ratings of Improvement of Symptoms (PORIS) question 3 at week 4. Question 3 asks respondents to indicate overall change in IC compared with before start of study as “worse,” “no better,” “slightly improved,” “moderately improved,” “greatly improved,” or “symptoms gone.” † The Cronbach ␣ is reported for pooled responses at baseline and endpoint.
TABLE IV. Interscale correlations for the Interstitial Cystitis Symptom Index (ICSI) items and score with the SF-12, Medical Outcomes Study (MOS) Sleep, and MOS Sexual Functioning scales Scale/Item ICSI score* ICSI item 1 ICSI item 2 ICSI item 3 ICSI item 4 SF-12 PCS SF-12 MCS SF-12 pain item MOS Sleep Scale MOS Sexual Function Scale
ICSI
Item 1
Item 2
Item 3
Item 4
0.761† 0.793† 0.713† 0.690† ⫺0.395† ⫺0.218† 0.489† ⫺0.411† ⫺0.222†
0.494† 0.319† 0.402† ⫺0.294† ⫺0.246† 0.369† ⫺0.299† ⫺0.197†
0.538† 0.347† ⫺0.258† ⫺0.152† 0.285† ⫺0.295† ⫺0.100
0.275† ⫺0.243† ⫺0.036 0.273† ⫺0.317† ⫺0.070
PCS
MCS
Pain
Sleep
⫺0.367† ⫺0.201† 0.158† † 0.506 ⫺0.764† ⫺0.368† ⫺0.305† 0.402† 0.439† ⫺0.415† † † ⫺0.278 0.289 0.360† ⫺0.329† 0.264†
MCS ⫽ Mental Component Summary; PCS ⫽ Physical Component Summary. * ICSI items measure the following: 1, urinary urgency; 2, urinary frequency; 3, nocturia; 4, bladder pain. † P ⬍0.001, indicates statistically significant correlation.
TABLE V. Change from baseline to endpoint and responsiveness to change as measured by the Guyatt statistic by the Patient Overall Rating of Improvement Symptoms (PORIS) response group Response Group Declining respondents Stable respondents Improving respondents
Sample Size
ICSI Change Score
Guyatt Statistic
12 184 168
2.25 ⫺1.47 ⫺5.38
0.69 ⫺0.45 ⫺1.64
ICSI ⫽ Interstitial Cystitis Symptom Index.
the PCS score was calculated such that a higher score equaled more optimal functioning. Weaker correlations were observed among the ICSI and the SF-12 scales, as would be expected because they measure different domains. Information on responsiveness to change is shown in Table V. Only 12 participants indicated a rating of “worse.” For these individuals, mean change in the ICSI score was 2.25 and the Guyatt statistic was 0.69. A small number of decliners are to be expected in dose-ranging studies. There were 184 patients who indicated no improvement (ie, they remained clinically stable). The mean ICSI UROLOGY 57 (Supplement 6A), June 2001
change score for this group was ⫺1.47 and the Guyatt statistic was ⫺0.45. This value is within the expected range for the Guyatt statistic for clinically stable patients. For patients defined as moderately or greatly improved, mean ICSI change was ⫺5.38 with an associated Guyatt statistic of ⫺1.64. A statistic with an absolute value ⱖ1.00 or greater is considered indicative of a measure highly responsive to change, as observed here. Results are reported in Table VI for clinically meaningful change. There was approximately a 2-point improvement in the ICSI score for each 65
TABLE VI. Change and percent change in Interstitial Cystitis Symptom Index (ICSI) score from baseline to endpoint based on improvement in Patient Overall Rating of Improvement of Symptoms (PORIS) question 3 at endpoint PORIS at Endpoint
n Mean change in ICSI score SD % change from baseline to endpoint
Moderately Improved (50% improvement)
Greatly Improved (75% improvement)
Symptoms Gone (100% improvement)
76 ⫺4.03 3.5 ⫺29.5
81 ⫺6.21 4.1 ⫺48.5
11 ⫺8.55 3.9 ⫺77.0
level of improvement in the PORIS. This rate of change, which indicates that the ICSI is associated with clinically meaningful change, was confirmed by the regression equation for the change in the ICSI score versus the PORIS: Change in ICSI Index Score ⫽ ⫺1.97 ⫻ PORIS ⫹ 3.67. CONCLUSIONS The ICSI has been previously validated in a community population and a convenience sample of patients with IC. In that work, the instrument was found to be stable and to have good internal consistency reliability. Information on responsiveness to change was not reported by the instrument developers. This report evaluated the psychometric properties of the ICSI in a prospective clinical drug trial in patients with IC. A total of 376 patients randomized to 1 of 3 treatment groups were included in the evaluation of the instrument, and blinding was maintained. In the trial, the ICSI individual items and score had good variability. Test–retest reliability over a 4-week period was good, and the scale had good internal consistency reliability. Evaluation of the construct validity demonstrated that the individual ICSI items correlated highly with the ICSI score. The pain item on the ICSI also correlated well with a pain item on the SF-12. Weaker correlations were observed among the ICSI items and score and the general scales measuring physical and emotional functioning, sleep, and sexual functioning. It is important that the ICSI be responsive to clinical improvement so that it may be used as an outcome in clinical trails. Responsiveness to change was evaluated using a global, patient-generated measure (PORIS). Patients who indicated an improvement in symptoms had a Guyatt statistic of
66
⫺1.64, which indicates that the instrument is responsive to change. Correspondingly the patients who indicated no change in symptoms had a Guyatt statistic of ⫺0.45, within the expected range for a stable population. For every 1-point improvement in the PORIS score, an average improvement of 1.97 was observed in the ICSI score, indicating that the ICSI is associated with clinically meaningful change. The ICSI is a valid, reliable, and responsive measure of IC symptom impact. This outcome instrument should have great utility and application in future cross-sectional and longitudinal studies of treatment (pharmacologic and nonpharmacologic) outcomes in patients with IC. REFERENCES 1. O’Leary MP, Sant GR, Fowler FJ, et al: The interstitial cystitis symptom index and problem index. Urology 49(suppl 5A): 58 – 63, 1997. 2. Lohr KN, Aaronson NK, Alonso J, et al: Evaluating quality-of-life and health status instruments: development of scientific review criteria. Clin Ther 18(5): 979 –992, 1996. 3. Hays RD, and Stewart AL: Sleep measures, in Stewart AL, and Ware JE (Eds), Measuring Functioning and Well-Being: The Medical Outcomes Study Approach. Durham, NC, Duke University Press, 1992 pp 235–259. 4. Sherbourne CD: Social functioning: sexual problem measures, in Stewart AL, and Ware JE (Eds), Measuring Functioning and Well-Being: The Medical Outcomes Study Approach. Durham, NC, Duke University Press, 1992, pp 194 –204. 5. Ware JE, Kosinski M, and Keller SD: SF-12: How to Score the SF-12 Physical and Mental Health Summary Scales. Boston, The Health Institute, New England Medical Center, 1995. 6. Rosner B: Fundamentals of Biostatistics, 4th ed. Belmont CA, Duxbury Press, 1994, pp 423– 426. 7. Aday LA: Designing and Conducting Health Surveys. San Francisco, Jossey-Bass, 1989, pp 50 –58. 8. Dunn G: Design and Analysis of Reliability Studies. New York, Oxford University Press, 1989, pp 47–50.
UROLOGY 57 (Supplement 6A), June 2001