- Email: [email protected]
The role of meta-analysis in medical decision making
The Spine Journal 3 (2003) 329–330
Editorial
The role of meta-analysis in medical decision making Michael Von Korff, ScD* Center for Health Studies, Group Health Cooperative, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101, USA
John Hampton [1] recently articulated the case against overuse of meta-analysis to guide medical decisions, stating: The strict application of evidence-based medicine implies a computer-like approach to medical care, in which the doctor sees the patient as a statistic rather than an individual; this sort of medicine could be practiced by administrators. Clinical trials will tell us what treatments are effective, but not necessarily which patients should receive them. We must accept that medical practice is—and should be—opinion based as much as it is evidence based. Treatment must always be tailored to the individual patient, and the physician’s judgment, if not his clinical freedom, remains as important as ever. Criticisms of meta-analysis he levels include that 1) different trials are often too heterogeneous in treatments or patients to combine; 2) the underlying trials enroll patients who differ from those seen in clinical practice (who tend to be older and sicker with comorbid conditions); 3) metaanalysis tends to call undue attention to small effects and to poorly designed or executed studies and 4) meta-analysis can allow opinion to masquerade as evidence, carrying authority approaching that of a definitive clinical trial. Criticism of meta-analysis also comes from a review of systematic reviews of conservative therapies for nonspecific back pain. In this review of reviews, Furlan et al. [2] concluded, “The reviews often provided contradictory evidence on the effectiveness of a wide range of commonly used conservative interventions for chronic nonspecific low back pain. These findings illustrate the pitfalls of systematic reviews where there are a number of low-quality trials and underscore the need for high quality primary trials that will allow for more conclusive reviews.” Given these criticisms of Author MVK acknowledges a financial relationship (grant research support from National Institutes of Health P01 DE08773) that may indirectly relate to the subject of this manuscript. * Corresponding author. Center for Health Studies, Group Health Cooperative, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101. Tel.: (206) 287-2874; fax: (206) 287-2871. E-mail address: [email protected] (M. Von Korff) 1529-9430/03/$ – see front matter © 2003 Elsevier Inc. All rights reserved. PII: S1529-9430(02)00 5 9 7 - 1
meta-analysis, would patients benefit if meta-analysis and evidence-based guidelines were regarded as passing fads? Meta-analysis and evidence-based guidelines do not supplant informed medical judgment. Like any scientific endeavor, meta-analysis is not immune from bias, methodological error, limited generalizability and invalid data. Although meta-analysis and evidence-based guidelines are not perfect, they are preferable to what they seek to replace: selective and biased reading of the scientific literature, overreliance on idiosyncratic and limited clinical experience, inadequate scrutiny of the methods of clinical trials and limited accountability of the medical profession for practice based in scientific evidence. Rigorous structured reviews using meta-analytic techniques can improve medical decisions in at least five ways that strengthen, rather than limit, informed medical decision making: 1. Identifying treatments that are not effective. Many treatments the benefits of which are slight are widely used in the management of low back pain. Meta-analysis can be a useful means for distinguishing treatments with modest benefits from treatments with unproven benefits. Although a treatment with modest average benefits may have larger benefits for a particular patient, it is hard to justify frequent use of treatments when scientific studies have not shown benefit to patients. For example, a meta-analysis of the efficacy of transcutaneous electrical nerve stimulation (TENS) for treatment of chronic low back pain found no support for efficacy relative to sham TENS [3]. A guideline indicating that TENS is of unproven efficacy is not an unwarranted intrusion on clinician judgment. 2. Summarizing the likely magnitude of benefits of effective treatments. Individualizing therapy is difficult in the absence of reliable knowledge of the magnitude of benefits of effective treatments. Because patients with back pain tend to improve with time in the absence of treatment, clinical experience is not a reliable means of estimating the magnitude of treatment effects. Because most treatments have risks and all treatments have costs, it is important to understand how large coun-
330
M. Von Korff / The Spine Journal 3 (2003) 329–330
tervailing benefits are likely to be. For example, a meta-analysis of nonsteroidal anti-inflammatory drug treatment for low back pain concluded that there was evidence for a small, short-term benefit of NSAIDs relative to placebo for acute low back pain [4]. A guideline that summarizes this evidence can help clinicians make informed prescribing decisions in light of a patient’s particular circumstances. 3. Identifying unanticipated risks of apparently effective treatments. There are situations in which important adverse effects cannot be identified from single randomized controlled trials but can be identified through meta-analysis of multiple trials. An example, unrelated to back pain, is the identification of a dose–response effect for increased mortality among patients with coronary artery disease treated with nifedipine, a calcium antagonist [5]. Among patients at the highest dosage levels, the risk of mortality was increased almost threefold, an effect not observable in the 16 individual randomized trials that were included in the meta-analysis. Evidence-based guidelines that flag treatment risks identified by meta-analysis can protect clinicians and patients from the consequences of significant, but uncommon, adverse treatment effects. 4. Identifying gaps in knowledge. Structured reviews often identify only a handful of randomized controlled trials relevant to the question being addressed. Metaanalyses of back pain treatments often observe that only short-term outcomes have been measured, studies of chronic back pain patients are lacking or key outcomes (eg, pain or disability) have not been assessed in most trials. For example, the meta-analysis for NSAIDs cited above [4] concluded that evidence on the effectiveness of NSAIDs was insufficient for patients with chronic low back pain. Identifying gaps in knowledge guides future research that can strengthen the evidence base for clinical practice, but understanding where gaps in knowledge exist is also needed to inform treatment decisions for individual patients. 5. Auditing the quality of existing randomized controlled trials. Meta-analysis has been a major force for improving the quality of randomized controlled trials. The increasing quality and number of randomized controlled trials evaluating low back pain treatments is the result, in part, of meta-analytic findings showing that the large majority of studies in the field were of poor quality. For example, a meta-analysis of lumbar supports for prevention and treatment of low back pain concluded that
only 4 of 13 randomized trials had positive scores on 50% or more of the internal validity criteria [6]. Metaanalysis has also contributed to the Journal of the American Medical Association [7] and other leading medical journals adopting standards for the reporting of randomized clinical trials. Such standards will contribute to the quality of both reporting and conduct of clinical trials in the future. Meta-analysis and evidence-based guidelines do not replace informed medical judgment and the need to individualize treatment decisions. However, meta-analysis has made important contributions to systematizing the scientific evidence relevant to informed medical decisions, as well strengthening the evidence base on which those decisions ultimately rest. Clinicians need to be free to exercise informed judgment in making treatment decisions. However, this is not a convincing rationale for ignoring the contributions of meta-analysis to systematizing and improving the scientific evidence on which informed medical decisions rest. Meta-analysis and evidence-based guidelines are imperfect bases for medical decision making, but they are a significant advance over medical decisions based only in expert opinion that too often lacked a discernable basis in scientific evidence. When used appropriately, meta-analysis and evidence-based guidelines should strengthen, rather than limit, informed medical decisions appropriate for individual patients.
References [1] Hampton JR. Evidence-based medicine, opinion-based medicine and real-world medicine. Perspectives Biology Med 2002;45:549–68. [2] Furlan AD, Clarke J, Esmail R, Sinclair S, Irvin E, Bombardier C. A critical review of reviews on the treatment of chronic low back pain. Spine 2001;26:E155––62. [3] Brosseau L, Milne S, Robinson V, et al. Efficacy of the transcutaneous electrical nerve stimulation for the treatment of chronic low back pain: a meta-analysis. Spine 2002;27:596–603. [4] van Tulder MW, Scholten RJ, Koes BW, Deyo RA. Nonsteroidal antiinflammatory drugs for low back pain: a systematic review within the framework of the Cochrane Collaboration Back review Group. Spine 2000;25:2501–13. [5] Furberg CD, Psaty B, Meyer JV. Nifedipine. Done-related increase in mortality in patients with coronary heart disease. Circulation 1995;92: 1326–31. [6] van Tulder MW, Jellema P, van Poppel, Nachemson AL, Bouter LM. Lumbar supports for prevention and treatment of low back pain. Cochrane Database Syst Rev 2000;3:CD001823. [7] Moher M, Schulz KF, Altman D, for the CONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001; 285:1987–91.
Editorial
The role of meta-analysis in medical decision making Michael Von Korff, ScD* Center for Health Studies, Group Health Cooperative, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101, USA
John Hampton [1] recently articulated the case against overuse of meta-analysis to guide medical decisions, stating: The strict application of evidence-based medicine implies a computer-like approach to medical care, in which the doctor sees the patient as a statistic rather than an individual; this sort of medicine could be practiced by administrators. Clinical trials will tell us what treatments are effective, but not necessarily which patients should receive them. We must accept that medical practice is—and should be—opinion based as much as it is evidence based. Treatment must always be tailored to the individual patient, and the physician’s judgment, if not his clinical freedom, remains as important as ever. Criticisms of meta-analysis he levels include that 1) different trials are often too heterogeneous in treatments or patients to combine; 2) the underlying trials enroll patients who differ from those seen in clinical practice (who tend to be older and sicker with comorbid conditions); 3) metaanalysis tends to call undue attention to small effects and to poorly designed or executed studies and 4) meta-analysis can allow opinion to masquerade as evidence, carrying authority approaching that of a definitive clinical trial. Criticism of meta-analysis also comes from a review of systematic reviews of conservative therapies for nonspecific back pain. In this review of reviews, Furlan et al. [2] concluded, “The reviews often provided contradictory evidence on the effectiveness of a wide range of commonly used conservative interventions for chronic nonspecific low back pain. These findings illustrate the pitfalls of systematic reviews where there are a number of low-quality trials and underscore the need for high quality primary trials that will allow for more conclusive reviews.” Given these criticisms of Author MVK acknowledges a financial relationship (grant research support from National Institutes of Health P01 DE08773) that may indirectly relate to the subject of this manuscript. * Corresponding author. Center for Health Studies, Group Health Cooperative, 1730 Minor Avenue, Suite 1600, Seattle, WA 98101. Tel.: (206) 287-2874; fax: (206) 287-2871. E-mail address: [email protected] (M. Von Korff) 1529-9430/03/$ – see front matter © 2003 Elsevier Inc. All rights reserved. PII: S1529-9430(02)00 5 9 7 - 1
meta-analysis, would patients benefit if meta-analysis and evidence-based guidelines were regarded as passing fads? Meta-analysis and evidence-based guidelines do not supplant informed medical judgment. Like any scientific endeavor, meta-analysis is not immune from bias, methodological error, limited generalizability and invalid data. Although meta-analysis and evidence-based guidelines are not perfect, they are preferable to what they seek to replace: selective and biased reading of the scientific literature, overreliance on idiosyncratic and limited clinical experience, inadequate scrutiny of the methods of clinical trials and limited accountability of the medical profession for practice based in scientific evidence. Rigorous structured reviews using meta-analytic techniques can improve medical decisions in at least five ways that strengthen, rather than limit, informed medical decision making: 1. Identifying treatments that are not effective. Many treatments the benefits of which are slight are widely used in the management of low back pain. Meta-analysis can be a useful means for distinguishing treatments with modest benefits from treatments with unproven benefits. Although a treatment with modest average benefits may have larger benefits for a particular patient, it is hard to justify frequent use of treatments when scientific studies have not shown benefit to patients. For example, a meta-analysis of the efficacy of transcutaneous electrical nerve stimulation (TENS) for treatment of chronic low back pain found no support for efficacy relative to sham TENS [3]. A guideline indicating that TENS is of unproven efficacy is not an unwarranted intrusion on clinician judgment. 2. Summarizing the likely magnitude of benefits of effective treatments. Individualizing therapy is difficult in the absence of reliable knowledge of the magnitude of benefits of effective treatments. Because patients with back pain tend to improve with time in the absence of treatment, clinical experience is not a reliable means of estimating the magnitude of treatment effects. Because most treatments have risks and all treatments have costs, it is important to understand how large coun-
330
M. Von Korff / The Spine Journal 3 (2003) 329–330
tervailing benefits are likely to be. For example, a meta-analysis of nonsteroidal anti-inflammatory drug treatment for low back pain concluded that there was evidence for a small, short-term benefit of NSAIDs relative to placebo for acute low back pain [4]. A guideline that summarizes this evidence can help clinicians make informed prescribing decisions in light of a patient’s particular circumstances. 3. Identifying unanticipated risks of apparently effective treatments. There are situations in which important adverse effects cannot be identified from single randomized controlled trials but can be identified through meta-analysis of multiple trials. An example, unrelated to back pain, is the identification of a dose–response effect for increased mortality among patients with coronary artery disease treated with nifedipine, a calcium antagonist [5]. Among patients at the highest dosage levels, the risk of mortality was increased almost threefold, an effect not observable in the 16 individual randomized trials that were included in the meta-analysis. Evidence-based guidelines that flag treatment risks identified by meta-analysis can protect clinicians and patients from the consequences of significant, but uncommon, adverse treatment effects. 4. Identifying gaps in knowledge. Structured reviews often identify only a handful of randomized controlled trials relevant to the question being addressed. Metaanalyses of back pain treatments often observe that only short-term outcomes have been measured, studies of chronic back pain patients are lacking or key outcomes (eg, pain or disability) have not been assessed in most trials. For example, the meta-analysis for NSAIDs cited above [4] concluded that evidence on the effectiveness of NSAIDs was insufficient for patients with chronic low back pain. Identifying gaps in knowledge guides future research that can strengthen the evidence base for clinical practice, but understanding where gaps in knowledge exist is also needed to inform treatment decisions for individual patients. 5. Auditing the quality of existing randomized controlled trials. Meta-analysis has been a major force for improving the quality of randomized controlled trials. The increasing quality and number of randomized controlled trials evaluating low back pain treatments is the result, in part, of meta-analytic findings showing that the large majority of studies in the field were of poor quality. For example, a meta-analysis of lumbar supports for prevention and treatment of low back pain concluded that
only 4 of 13 randomized trials had positive scores on 50% or more of the internal validity criteria [6]. Metaanalysis has also contributed to the Journal of the American Medical Association [7] and other leading medical journals adopting standards for the reporting of randomized clinical trials. Such standards will contribute to the quality of both reporting and conduct of clinical trials in the future. Meta-analysis and evidence-based guidelines do not replace informed medical judgment and the need to individualize treatment decisions. However, meta-analysis has made important contributions to systematizing the scientific evidence relevant to informed medical decisions, as well strengthening the evidence base on which those decisions ultimately rest. Clinicians need to be free to exercise informed judgment in making treatment decisions. However, this is not a convincing rationale for ignoring the contributions of meta-analysis to systematizing and improving the scientific evidence on which informed medical decisions rest. Meta-analysis and evidence-based guidelines are imperfect bases for medical decision making, but they are a significant advance over medical decisions based only in expert opinion that too often lacked a discernable basis in scientific evidence. When used appropriately, meta-analysis and evidence-based guidelines should strengthen, rather than limit, informed medical decisions appropriate for individual patients.
References [1] Hampton JR. Evidence-based medicine, opinion-based medicine and real-world medicine. Perspectives Biology Med 2002;45:549–68. [2] Furlan AD, Clarke J, Esmail R, Sinclair S, Irvin E, Bombardier C. A critical review of reviews on the treatment of chronic low back pain. Spine 2001;26:E155––62. [3] Brosseau L, Milne S, Robinson V, et al. Efficacy of the transcutaneous electrical nerve stimulation for the treatment of chronic low back pain: a meta-analysis. Spine 2002;27:596–603. [4] van Tulder MW, Scholten RJ, Koes BW, Deyo RA. Nonsteroidal antiinflammatory drugs for low back pain: a systematic review within the framework of the Cochrane Collaboration Back review Group. Spine 2000;25:2501–13. [5] Furberg CD, Psaty B, Meyer JV. Nifedipine. Done-related increase in mortality in patients with coronary heart disease. Circulation 1995;92: 1326–31. [6] van Tulder MW, Jellema P, van Poppel, Nachemson AL, Bouter LM. Lumbar supports for prevention and treatment of low back pain. Cochrane Database Syst Rev 2000;3:CD001823. [7] Moher M, Schulz KF, Altman D, for the CONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001; 285:1987–91.