- Email: [email protected]
Patient-oriented outcome measures: The promise of definition
Editorial
Patient-Oriented Outcome Measures: The Promise of Definition Knox H. Todd, MD, MPH
Patient-Oriented Outcome Measures: The Promise of Definition See related articles, p. 633 and p. 639.
Emory University School of Medicine Atlanta, GA Reprints not available from the author. Address for correspondence: Knox H. Todd, MD, MPH, Emory University School of Medicine, 1101 Juniper Street NE, Suite 913, Atlanta, GA 30309; 404-872-5740, fax 404-872-5747; E-mail [email protected]. Copyright © 2001 by the American College of Emergency Physicians. 0196-0644/2001/$35.00 + 0 47/1/119746 doi:10.1067/mem.2001.119746
6 7 2
[Todd KH. Patient-oriented outcome measures: the promise of definition. Ann Emerg Med. December 2001;38:672-674.] In 1996, Annals published results of our research suggesting that the minimum clinically significant difference (MCSD) in visual analog scale (VAS) pain scores for patients with acute traumatic pain averaged 13 mm (95% confidence interval [CI] 10 to 17 mm).1 The purpose of this work was to hold investigators to credible and defensible end points when clinical analgesic trials are conducted. Defining the clinical significance of change in a commonly used outcome measure encourages investigators to use more appropriate sample sizes, thereby avoiding both sins of commission (reporting statistically significant but clinically meaningless differences) and sins of omission (suggesting no difference in alternative therapies when too few patients are studied).2 Both strategies have been used by investigators who may have been tempted to choose end points that supported their underlying biases. This tendency has been most common in studies comparing newer analgesics with older (and less expensive) standard therapies. At the risk of overgeneralization, it seems that with some regularity, the newer analgesic is administered at full dose, whereas the standard is dosed suboptimally, relatively crude outcome measures are chosen, and small numbers of patients are studied. Predictably, a no-difference finding results and is published. There ensues an extensive discussion of the relative advantages of the new drug that may (or may not) be supported by the analysis of secondary trial outcomes. Alternatively, small statistically significant differences in visual analog scores might be found and stressed, without a determination of the clinical relevance of these differences. Neither scenario provides information that the clinician can use with confidence.
ANNALS OF EMERGENCY MEDICINE
38:6 DECEMBER 2001
EDITORIAL
In this issue of Annals, 2 articles further explore the question of clinically significant differences in visual analog pain scales.3,4 A better understanding of clinically relevant end points for clinical analgesic trials is necessary if we hope to influence the research process for our patients’ and society’s interest. Gallagher et al3 have conducted a prospective validation study of the clinically significant difference in a cohort of emergency department patients presenting with acute pain. Under similar conditions to that of our study, they report strikingly similar findings, namely, that mean changes in VAS pain scores of 13 mm (95% CI 10 to 16 mm) represent the minimum change in acute pain that is clinically significant. Bird and Dickson4 have focused on a related question: Does the MCSD vary across the spectrum of pain severity? It seems reasonable to assume that patients with higher levels of pain might identify greater reductions in VAS pain scores as clinically meaningful than would patients with lower levels of pain. The investigators found that 19 mm represented the MCSD in VAS pain scores when patients across the pain severity spectrum were considered; in addition, they found that, for patients with pain scores of 67 to 100 mm, a minimum difference of 28 mm was needed for a perceptible change in pain severity. These investigators defined their outcome measure as the “absolute value” of differences between VAS scores when pain was felt to increase or decrease. This approach could contribute to an overestimation of the MCSD because, in these studies, patients are blinded to their prior VAS scores. Inevitably, some patients will note that their pain has increased (or decreased) when their VAS score actually decreases (or increases). For the other studies cited, to combine VAS differences for pain contrasts of “a little less” or “a little more” pain, the difference was multiplied by –1. In using absolute values, Bird and Dickson4 may have magnified true VAS differences. This is reflected in their findings of positive VAS differences associated with pain that remains the same. This finding differs from that of the other studies cited in which stable pain states are associated with VAS changes of essentially zero. This decision is unlikely to have had a major impact on their findings, however, because only 4 VAS changes were discordant. It would be interesting to reanalyze their results using the alternate calculation of VAS differences. A recent study by Kelly5 addressed a similar question using a similar study design. This investigator found the MCSD to be 12 mm (95% CI 9 to 15 mm). She then grouped her patients according to pain severity and found no differences in the MCSD for patients with mild, moderate, or severe pain (Table 1).
DECEMBER 2001
38:6
ANNALS OF EMERGENCY MEDICINE
To add to this debate, we have reanalyzed the results of our original study and could not discern a relation between the magnitude of MCSD and pain severity (Table 2). In the Figure, we have provided a scatterplot of the
Table 1.
MCSD in VAS scores categorized according to pain severity (mm). Referent VAS
Mean (SD)
95% CI
N
11 14 10
4 to 18 10 to 18 6 to 14
26 39 23
1 to 30 31 to 69 70 to 100
From Kelly A-M. The minimum clinically significant difference in visual analog scale pain score does not differ with severity of pain. Emerg Med J. 2001;18:205-207. Copyright BMJ Publishing Group. Reprinted with permission.
Table 2.
MCSD in VAS scores categorized according to pain severity (mm).1 Referent VAS
Mean (SD)
95% CI
Median
Range
N
14 13 13
8 to 19 7 to 18 7 to 20
10 11 10
–2 to 45 –3 to 45 –22 to 75
20 21 39
1 to 33 34 to 66 67 to 100
Figure.
MCSD in VAS scores plotted against pain severity for 80 patients. VAS difference 75
50
25
0
–25 0
25
50
75
100
Referent Pain Score
6 7 3
EDITORIAL
MCSD for 80 patients reporting either “a little more” or “a little less” pain. Again, although we note a wider dispersion of VAS differences with more severe pain, there is no trend of increasing MCSD with increasing pain severity. In the Figure , the horizontal line represents the mean MCSD (13 mm). Determination of the MCSD in VAS pain scores is a modest, but necessary, step toward defining more relevant patient-oriented outcomes. The goal of this process is to enhance the credibility of clinical trials by advancing candidate trial end points that address issues of relevance to patients, clinicians, and society. Much more work is necessary in this area to develop outcome measures that encompass the multiple domains we implicitly refer to when pain and its treatment are considered. These include pain severity and a multitude of dimensions associated with pain experience, such as mood, side effects, feelings of hopelessness, sleep quality, and functional status, to name only a few. One approach to a more global expression of pain experience consists of measuring multiple traditional outcomes using separate instruments, striving for end points of the highest quality to reflect multiple dimensions of health status. These end points may overlap similar domains and are by definition noncomprehensive. They require statistical adjustment for inevitable multiple comparisons and may tempt researchers to report only those outcomes that show impressive results.6 Another approach is to develop single, pooled outcome measures, or indices, that better reflect the implicit, intuitive, and global experience of our patients. What, then, is the next step? Members of the Annals editorial board have continually and successfully raised the bar for methodologic rigor in our research. Clinical trials are only as credible as their end points.7 Promoting a consensus conference to examine the relative merits of competing end points and to develop standardized pooled outcome measures or indices that are pertinent to the practice of emergency medicine is in keeping with this journal’s tradition. Convening such a conference is an ambitious, but necessary, effort if our specialty is to influence the future of analgesic clinical trials. 1. Todd KH, Funk KG, Funk JP, et al. Clinical significance of reported changes in pain severity. Ann Emerg Med. 1996;27:485-489. 2. Todd KH. Clinical versus statistical significance in the assessment of pain relief. Ann Emerg Med. 1996;27:439-441. 3. Gallagher EJ, Liebman M, Bijur PE. Prospective validation of clinically important changes in pain severity measured on a visual analog scale. Ann Emerg Med. 2001;38:633-638. 4. Bird SB, Dickson EW. Clinically significant changes in pain along the visual analog scale. Ann Emerg Med. 2001;38:639-643. 5. Kelly A-M. The minimum clinically significant difference in visual analogue scale pain score does not differ with severity of pain. Emerg Med J. 2001;18:205-207. 6. Felson DT, Anderson JJ, Meenan RF. Time for changes in the design, analysis, and reporting of rheumatoid arthritis clinical trials. Arthritis Rheum. 1990;33:140-149. 7. Tugwell P, Boers M. OMERACT Conference on outcome measures in rheumatoid arthritis clinical trials: introduction. J Rheumatol. 1993;20:528-530.
6 7 4
ANNALS OF EMERGENCY MEDICINE
38:6 DECEMBER 2001
Patient-Oriented Outcome Measures: The Promise of Definition Knox H. Todd, MD, MPH
Patient-Oriented Outcome Measures: The Promise of Definition See related articles, p. 633 and p. 639.
Emory University School of Medicine Atlanta, GA Reprints not available from the author. Address for correspondence: Knox H. Todd, MD, MPH, Emory University School of Medicine, 1101 Juniper Street NE, Suite 913, Atlanta, GA 30309; 404-872-5740, fax 404-872-5747; E-mail [email protected]. Copyright © 2001 by the American College of Emergency Physicians. 0196-0644/2001/$35.00 + 0 47/1/119746 doi:10.1067/mem.2001.119746
6 7 2
[Todd KH. Patient-oriented outcome measures: the promise of definition. Ann Emerg Med. December 2001;38:672-674.] In 1996, Annals published results of our research suggesting that the minimum clinically significant difference (MCSD) in visual analog scale (VAS) pain scores for patients with acute traumatic pain averaged 13 mm (95% confidence interval [CI] 10 to 17 mm).1 The purpose of this work was to hold investigators to credible and defensible end points when clinical analgesic trials are conducted. Defining the clinical significance of change in a commonly used outcome measure encourages investigators to use more appropriate sample sizes, thereby avoiding both sins of commission (reporting statistically significant but clinically meaningless differences) and sins of omission (suggesting no difference in alternative therapies when too few patients are studied).2 Both strategies have been used by investigators who may have been tempted to choose end points that supported their underlying biases. This tendency has been most common in studies comparing newer analgesics with older (and less expensive) standard therapies. At the risk of overgeneralization, it seems that with some regularity, the newer analgesic is administered at full dose, whereas the standard is dosed suboptimally, relatively crude outcome measures are chosen, and small numbers of patients are studied. Predictably, a no-difference finding results and is published. There ensues an extensive discussion of the relative advantages of the new drug that may (or may not) be supported by the analysis of secondary trial outcomes. Alternatively, small statistically significant differences in visual analog scores might be found and stressed, without a determination of the clinical relevance of these differences. Neither scenario provides information that the clinician can use with confidence.
ANNALS OF EMERGENCY MEDICINE
38:6 DECEMBER 2001
EDITORIAL
In this issue of Annals, 2 articles further explore the question of clinically significant differences in visual analog pain scales.3,4 A better understanding of clinically relevant end points for clinical analgesic trials is necessary if we hope to influence the research process for our patients’ and society’s interest. Gallagher et al3 have conducted a prospective validation study of the clinically significant difference in a cohort of emergency department patients presenting with acute pain. Under similar conditions to that of our study, they report strikingly similar findings, namely, that mean changes in VAS pain scores of 13 mm (95% CI 10 to 16 mm) represent the minimum change in acute pain that is clinically significant. Bird and Dickson4 have focused on a related question: Does the MCSD vary across the spectrum of pain severity? It seems reasonable to assume that patients with higher levels of pain might identify greater reductions in VAS pain scores as clinically meaningful than would patients with lower levels of pain. The investigators found that 19 mm represented the MCSD in VAS pain scores when patients across the pain severity spectrum were considered; in addition, they found that, for patients with pain scores of 67 to 100 mm, a minimum difference of 28 mm was needed for a perceptible change in pain severity. These investigators defined their outcome measure as the “absolute value” of differences between VAS scores when pain was felt to increase or decrease. This approach could contribute to an overestimation of the MCSD because, in these studies, patients are blinded to their prior VAS scores. Inevitably, some patients will note that their pain has increased (or decreased) when their VAS score actually decreases (or increases). For the other studies cited, to combine VAS differences for pain contrasts of “a little less” or “a little more” pain, the difference was multiplied by –1. In using absolute values, Bird and Dickson4 may have magnified true VAS differences. This is reflected in their findings of positive VAS differences associated with pain that remains the same. This finding differs from that of the other studies cited in which stable pain states are associated with VAS changes of essentially zero. This decision is unlikely to have had a major impact on their findings, however, because only 4 VAS changes were discordant. It would be interesting to reanalyze their results using the alternate calculation of VAS differences. A recent study by Kelly5 addressed a similar question using a similar study design. This investigator found the MCSD to be 12 mm (95% CI 9 to 15 mm). She then grouped her patients according to pain severity and found no differences in the MCSD for patients with mild, moderate, or severe pain (Table 1).
DECEMBER 2001
38:6
ANNALS OF EMERGENCY MEDICINE
To add to this debate, we have reanalyzed the results of our original study and could not discern a relation between the magnitude of MCSD and pain severity (Table 2). In the Figure, we have provided a scatterplot of the
Table 1.
MCSD in VAS scores categorized according to pain severity (mm). Referent VAS
Mean (SD)
95% CI
N
11 14 10
4 to 18 10 to 18 6 to 14
26 39 23
1 to 30 31 to 69 70 to 100
From Kelly A-M. The minimum clinically significant difference in visual analog scale pain score does not differ with severity of pain. Emerg Med J. 2001;18:205-207. Copyright BMJ Publishing Group. Reprinted with permission.
Table 2.
MCSD in VAS scores categorized according to pain severity (mm).1 Referent VAS
Mean (SD)
95% CI
Median
Range
N
14 13 13
8 to 19 7 to 18 7 to 20
10 11 10
–2 to 45 –3 to 45 –22 to 75
20 21 39
1 to 33 34 to 66 67 to 100
Figure.
MCSD in VAS scores plotted against pain severity for 80 patients. VAS difference 75
50
25
0
–25 0
25
50
75
100
Referent Pain Score
6 7 3
EDITORIAL
MCSD for 80 patients reporting either “a little more” or “a little less” pain. Again, although we note a wider dispersion of VAS differences with more severe pain, there is no trend of increasing MCSD with increasing pain severity. In the Figure , the horizontal line represents the mean MCSD (13 mm). Determination of the MCSD in VAS pain scores is a modest, but necessary, step toward defining more relevant patient-oriented outcomes. The goal of this process is to enhance the credibility of clinical trials by advancing candidate trial end points that address issues of relevance to patients, clinicians, and society. Much more work is necessary in this area to develop outcome measures that encompass the multiple domains we implicitly refer to when pain and its treatment are considered. These include pain severity and a multitude of dimensions associated with pain experience, such as mood, side effects, feelings of hopelessness, sleep quality, and functional status, to name only a few. One approach to a more global expression of pain experience consists of measuring multiple traditional outcomes using separate instruments, striving for end points of the highest quality to reflect multiple dimensions of health status. These end points may overlap similar domains and are by definition noncomprehensive. They require statistical adjustment for inevitable multiple comparisons and may tempt researchers to report only those outcomes that show impressive results.6 Another approach is to develop single, pooled outcome measures, or indices, that better reflect the implicit, intuitive, and global experience of our patients. What, then, is the next step? Members of the Annals editorial board have continually and successfully raised the bar for methodologic rigor in our research. Clinical trials are only as credible as their end points.7 Promoting a consensus conference to examine the relative merits of competing end points and to develop standardized pooled outcome measures or indices that are pertinent to the practice of emergency medicine is in keeping with this journal’s tradition. Convening such a conference is an ambitious, but necessary, effort if our specialty is to influence the future of analgesic clinical trials. 1. Todd KH, Funk KG, Funk JP, et al. Clinical significance of reported changes in pain severity. Ann Emerg Med. 1996;27:485-489. 2. Todd KH. Clinical versus statistical significance in the assessment of pain relief. Ann Emerg Med. 1996;27:439-441. 3. Gallagher EJ, Liebman M, Bijur PE. Prospective validation of clinically important changes in pain severity measured on a visual analog scale. Ann Emerg Med. 2001;38:633-638. 4. Bird SB, Dickson EW. Clinically significant changes in pain along the visual analog scale. Ann Emerg Med. 2001;38:639-643. 5. Kelly A-M. The minimum clinically significant difference in visual analogue scale pain score does not differ with severity of pain. Emerg Med J. 2001;18:205-207. 6. Felson DT, Anderson JJ, Meenan RF. Time for changes in the design, analysis, and reporting of rheumatoid arthritis clinical trials. Arthritis Rheum. 1990;33:140-149. 7. Tugwell P, Boers M. OMERACT Conference on outcome measures in rheumatoid arthritis clinical trials: introduction. J Rheumatol. 1993;20:528-530.
6 7 4
ANNALS OF EMERGENCY MEDICINE
38:6 DECEMBER 2001