- Email: [email protected]
Would Meta-analys Change Our Clinical Practice?
Would Meta-analysis Change Our Clinical Practice? Reg S. Sauve, MD MPH FRCPC Department of Pediatrics and Community Health Sciences University of Calgary CalgaryAB
Abstract: during the 20 years since Glass coined the term "meta-analysis," there has been remarkable growth both in its popularity and in skepticism about its value. The rationale for meta-analysis, also referred to as quantitative systematic review, seems ideal given the large volume of research with which practitioners are faced on a regular basis and the current broad acceptance of the principles of evidence-based medicine. But if a meta-analysis were published in our field of women's health, would it change our clinical practice? A positive answer assumes that if it were published, we would see it, read it, believe it, and welcome a clinical policy or practice guideline based on the meta-analysis, even if it suggested a practice differing from what we normally do, and we would change our practice accordingly. The assumption of a negative answer is that we might not see it or have time to read it, we might not believe it, or it might be presented in such a way that we would not act on it. Which answer is more likely? We can approach this question by reviewing the definitions and methods used in meta-analysis, the factors determining clinical practice, and evidence about whether physicians regularly change their practices on the basis of meta-analysis or guidelines based on meta-analysis. Resume : depuis que Glass a invente le mot « meta-analyse » il y a 20 ans, on assiste a une croissance remarquable de Ia popularite de cette approche aussi bien qu'a une montee du scepticisme quant a sa valeur. La raison d'etre de Ia meta-analyse, qu'on appelle aussi « revue quantitative systematique », semble bien repondre idealement au besoin qu'ont les praticiens d'absorber un volume considerable de recherches de fac;:on reguliere et au fait qu'on accepte de plus en plus universellement les principes de Ia medecine fondee sur les preuves scientifiques. Mais, si on publiait une meta-analyse sur Ia sante des femmes dans notre domaine, est-ce que cela changerait notre pratique clinique ? Une reponse affirmative suppose que, si une telle analyse etait publiee, nous Ia verrions, nous Ia lirions, nous lui accorderions notre confiance et nous ferions bon accueil a des politiques cliniques ou a des !ignes directrices fondees sur cette meta-analyse, meme si elle recommandait une pratique differente de ce que nous faisons normalement, et que nous modifierions notre pratique en consequence. Et si nous repondons par Ia negative, cela peut vouloir dire que nous ne Ia verrions pas, que nous n'aurions pas le temps de Ia lire, que nous ne lui ferions pas confiance ou qu'il se pourrait qu'elle soit presentee d'une fac;:on telle qu'elle n'influencerait pas notre conduite. Laquelle de ces deux reponses est Ia plus vraisemblable ? Nous essayons de repondre a cette derniere question en examinant les definitions et les methodes appliquees a Ia meta-analyse, les facteurs qui determinent Ia pratique clinique et les donnees qui font voir si les medecins changent regulierement leur fac;:on de pratiquer sous !'influence d'une meta-analyse ou de !ignes directrices basees sur des meta-analyses.
J Soc
Obstet Gynaecol Can 2000;22( 12): I050-4
INTRODUCTION
KeyWords Meta-analysis, critical appraisal Competing interest: none declared Received on May 18th, 2000. Revised and accepted on November 7th, 2000.
During the 20 years since Glass 1 coined the term "meta-analysis," there has been remarkable growth both in its popularity and in skepticism about its value. 1· 7 The rationale for metaanalysis, also referred to as quantitative systematic review, seems ideal given the large volume of research with which practitioners are faced on a regular basis and the current broad acceptance of the principles of evidence-based medicine. 8 But if a meta-analysis were published in our practice area, would it change our practice? A positive answer assumes that if it were published, we would see it, read it, believe it, and welcome a clinical policy or practice guideline JOURNAL S O G C . DECEMBER 2000
based on the meta-analysis, even if it suggested a practice differing from what we normally do, and we would change our practice accordingly. The assumption of a negative answer is that we might not see it or have time to read it, we might not believe it, or it might be presented in such a way that we would not act on it. Which answer is more likely? We can approach this question by reviewing the definitions and methods used in meta-analysis, the factors that determine clinical practice, and evidence about whether physicians regularly change their practices on the basis of meta-analysis or guidelines based on meta-analysis. WHAT IS A META-ANALYSIS?
A meta-analysis is the analysis of a large collection of results from individual srudies for the purpose of integrating the findings. 1•9 In clinical medicine they most often consist of summaries of the results of clinical trials, but they may be summaries of other types of srudies as well. 10 Meta-analyses have the advantage of offering more precise conclusions than individual studies, and can provide more definitive estimates of a particular clinical effect. 10 In some cases, they comprise an effective approach to answering primary research questions or resolving conflicts in the interpretation of results from multiple studies. 10•11 Metaanalyses are often done as components of systematic reviews, but they may also be presented as independent analysis. METHODS USED IN META-ANALYSIS
The methods used in meta-analysis have been described in detail elsewhere. 1•2 •10 •12•13 The first step in performing a metaanalysis is the formulation of a clear and relevant research question that could be answered by that technique. Next, the variables to be used in the meta-analysis are specified, including exposure and outcome variables and any confounding vari-
FIGURE I
FUNNEL PLOT OF 35 HYPOTHETICAL STUDY RESULTS DEMONSTRATING AN ABSENCE OF PUBLICATION BIAS
CD
N
''
10000
0
-1.E ('0
(I)
1000
10 0.1
/
'
.•••
'\
-
\\\
··' .. • • • ••••• . /
100
'
I
.././ / • /
'
• •••
FACTORS DETERMINING CUNICAL PRACTICE
• \\\
EVIDENCE
,., \,\,
1
ables. Subsequently, the types of reports to be collected for the analysis are determined. The authors then proceed with a comprehensive literature search, which may include data from full articles and abstracts, from published and unpublished literature, from reports published in English, French, and perhaps other languages. The results of the included studies are then abstracted, with the investigator ensuring that the patients, interventions, outcome measures, and research methodology are similar before the analysis begins. In the past, there were valid concerns that clinically useful decisions could not be drawn by comparing or combining studies which used different techniques, different definitions, and so on; but as experience has been gained in the methods used in meta-analysis and in the ways they can be presented, these issues are becoming resolved. 6•11 •14 Assessment of potential bias is a part of every analysis. In the case of meta-analysis, two common sources of bias are the failure of investigators to find or include all relevant srudies (publication bias), 10•15· 16 and the uncertain reliability and quality of included srudies. Publication bias can be estimated graphically using a "funnel plot" 17 in which sample size is plotted against effect size. If many srudies have been located that estimate a similar effect, the distribution of the data points should result in an inverted, funnel-shaped symmetrical appearance (figure 1). A gap on one side of the wider part of the funnel indicates that perhaps not all results have been published or located. 17 Quality scores are included in some meta-analysis on the basis of randomization details, sample size, and other fearures, but approaches to quality assessment are not standardized and are not always induded. 18-20 Another analytical question that instigators of a metaanalyses have to decide is whether the data being combined is heterogeneous from a statistical point of view. 5 Heterogeneity can arise in two basic ways. If a population of very similar individuals is subjected to identical treatments, there will still be some individual differences of response. This represents "within study variance." Additionally, each study population is drawn from a different sample of individuals, so no matter how careful one is and no matter how large a collection of studies is located, there are still going to be some differences in the results of individual studies. This represents "between study variance." If either of these types of variance is large, the results of individual studies are not consistent with one another, and the data is considered heterogenous. 5
10
Risk Ratio
JOURNAL SOGC
Patient data, or the history, physical, and laboratory findings that a particular patient exhibits, is a major determinant of all clinical decisions, but the contribution of research evidence to clinical decisions is more difficult to determine. Findings published in the biomedical research world are often exciting and DECEMBER 2000
sometimes have the potential to influence our clinical practices. However, there are currently over 20,000 journals published regularly worldwide. Davidoff et aL suggested that to keep up with current medical literature, a general physician would have to read 17 articles each day. 22 No matter what strategy we choose to try to "keep up" with relevant research and reports, their sheer volume may prevent this. However, there are several approaches designed to help practitioners by summarizing, synthesizing, and disseminating the best available evidence, sometimes in the form of meta-analysis and sometimes in other formats. 21 •23-25 A major stimulus to the identification, appraisal, and synthesis of research evidence is the work of the Cochrane Collaboration.24·26 Obstetricians and gynaecologists have played key pioneering roles in this area. In fact, Archie Cochrane, the instigator of the collaboration, referred to the systematic reviews of randomized controlled trials of care during pregnancy and childbirth as "a real milestone in the history of randomized trials and the evaluation of care." 26 Approaches to critical appraisal of individual studies help practitioners to identifY relevant articles and critically approach their reading and interpretation. 27 However, we may not always have the time, motivation or skills needed to find, critically appraise, and synthesize information de novo. Fortunately, systematic reviews developed through the Cochrane collaboration and other sources, many of which include meta-analysis, enhance our ability to identifY, interpret, and apply research evidence. Valuable sources for systematic reviews are the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effectiveness (DARE), ACP Journal Club, Evidence-Based Medicine, and Clinical Evidence. Information from these and other sources have been important in many new and exciting initiatives in the provision of continuing medical education. 23 PATIENT AND PHYSICIAN FACTORS
Another major determinant of clinical decision-making is the background and daily lives of patients and physicians, including their personal values, experiences, and education. 21 Professional relationships between physicians and patients have changed from an authoritarian to a shared approach to clinical decision making. Reasons for the change include broad societal changes, changes in the education of the physicians and of the public, and easy access to vast amounts of health information through the Internet. As physicians, although we might choose to make most of our clinical decisions primarily on the basis of the objective patient data, the research evidence available, and the patient's input regarding personal needs and preferences: our decisions cannot be made in a void, free from our own or our patients' past experiences, preferences, and background-nor should they be.
JOURNAL SOGC
CONSTRAINTS
Constraints on clinical decision-making comprise a changing target in ways beyond the scope of this paper. However, current constraints are more numerous and may be different than was the case in the recent past. It has been pointed out that even when evidence is strong and clinical policy is sound, there are impediments to incorporating evidence from research into practice. 28 One of these impediments has been described as a mismatch between evidence and clinical circumstances. 28 For example, even if certain diagnostic procedures have been found to be effective, if the local resources and logistics do not support the procedures, a physician may not be able to offer them to patients who could benefit. Another constraint is time pressure in clinical practice, and a third is difficulty in learning new clinical skills. 28 This last may be becoming less of a constraint as evidence continues to accumulate in areas such as optimal delivery of continuing medical education, computerized decision support approaches, and measures to enhance patient compliance. EFFECT OF FACTORS DETERMINING CLINICAL PRACTICE
Evidence, patient and physician factors, and constraints act together to form the clinical decisions that we are called upon to make many times a day. It is difficult to single out the effects of any one of these factors on clinical practice, and even more difficult to single out the effects of only one component of one of the factors: in the case of this paper, evidence that has been presented as a meta-analysis. Indeed, as Davis notes: "Even when a valid systematic review leads to a Clinical Practice Guideline (CPG), there are no guarantees that a change in clinical practice will follow." 29 Three possible considerations in determining such effects are the lag time between publication of a meta-analysis and changes in physician practices, the extent to which clinical practice guidelines based on meta-analysis appear to change the process and outcome of clinical practice, 30 and reports of the use of meta-analysis and changes in practice by obstetricians, gynaecologists, and neonatologists. 31 · 32 A commonly cited example of the lag time between publication of both consistent individual clinical trials and metaanalysis and change in physician practices is the administration of antenatal steroids to women presenting with threatened preterm delivery at 34 weeks or less. 33-35 In the 20 years following publication of the initial clinical trial, 36 18 additional trials and, more recently, meta-analysis34·35 have been reported supporting the effectiveness of antenatal steroids in preventing neonatal deaths associated with respiratory distress syndrome. Despite these results, several studies have confirmed that many women whose infants could benefit from this therapy have not received it. 37·38 For example, in a survey of 17 Canadian neonatal intensive care units in 1997, the rate of antenatal steroid DECEMBER 2000
treatment in infants born at or before 34 weeks gestation varied from 27 percent to more than 95 percent, with an average of 61.5 percent.37 This demonstrates that the simple publication of a metaanalysis or even of a clinical practice guideline alone does not appear to be effective in changing physician practice. Some reports have described self-reported practice changes by neonatologists and obstetricians. 31 Worrall et al, reviewing the effects of clinical practice guidelines in primary care, found little evidence that the use of clinical practice guidelines improved patient outcomes. 39 However, several current studies focus on strategies for the dissemination and implementation of guidelines. 40 • 41 Reports in this area are accumulating through the work of such groups such as the Cochrane Effective Practice and Organization of Care Group,24 DARE, and the Department of Health Care and Promotion of the Canadian Medical Association. 29 The use of clinical practice guidelines varies according to resource availability, institutional or practice specific barriers, the context of introduction of the guideline, as well as other factors. Techniques which have positively influenced the utilization of guidelines include: professional interventions such as unique continuing education approaches, 42 audit and feedback, 43 organizational interventions such as incorporating reminders or standard treatment recommendations in computerized order entry programmes, and financial and regulatory interventions. 44 CONCLUSION
There are many situations where meta-analysis are valuable and deserving of our close attention. Yet publication of a meta-analysis alone, or a clinical practice guideline which includes a metaanalysis, is unlikely to change practice. However, evidence is only one of the determinants of clinical practice. Changing clinical practice may require several innovative and multifaceted approaches particular to each meta-analysis or guideline, as well as specific context within which a change in practice might be desirable. Specific approaches such as systemic reviews are available to help practitioners to obtain and critically appraise research findings and meta-analysis, and to help with the ongoing accumulation and interpretation of up-to-date knowledge. These approaches are helpful but they are only part of the process of creating change in clinical practice. REFERENCES I. 2. 3. 4. 5.
Glass G, McGaw B, Smith M. Meta-analysis in social research. Beverly Hills CA: Sage, 1981. Abramson J. Meta-analysis: a review of the pros and cons. Public Health Reviews 1990/91; 18:1-47. Chalmers T. Problems induced by meta-analysis. Stat Med 1991; I0:971-80. Bailar JC.The problems and promise of meta-analysis. N Eng J Med 1997;337:559-61. Thompson S, Pocock S. Can meta-analysis be trusted? Lancet 1991; 338:1127-30.
6. 7. 8.
9. I0. II. 12.
13. 14. 15. 16. 17. 18.
Naylor C. Two cheers for meta-analysis: problems and opportunities in aggregating results of clinical trials. Can Med Assoc J 1988; 138:891-5. Shapiro S. Meta-analysis/schmeta-analysis. Amer J Epidemiol 1994; 140:771-8. Sackett DL, Richardson WS, RosenbergW, Haynes RB. Evidence-Based Medicine: How To Practice and Teach EBM. New York: Churchill Livingtone, 1997. Greenland S. Quantitative methods in the review of epidemiologic literature. Epidemiol Rev 1987;9: 1-30. Dickerson K, Berlin J. Meta-analysis: state-of-the-science. Epidemiol Rev 1992; 14:154-76. Greenland S. Can meta-analysis be salvaged? Amer J Epidemiol 1994; 140:783-7. Wolf F. Meta-analysis: Quantitative Methods For Research Synthesis. in: Sullivan JL, Niemi RG (eds). Quantitative Applications in the Social Sciences. Vol. 59. Beverly Hills: SAGE 1986:1-65. Yusuf S, Simon R, Ellenberg S. Methodologic issues in overviews of randomized clinical trials. Stat Med 1987;6:217-409. Spitzer W. Meta-meta-analysis: unanswered questions about aggregating data.) Clin Epidemiol 1991;44:103-7. Chalmers T, Frank C, Reitman D. Minimizing the three stages of publication bias. J Amer Med Assoc 1990;2263: 1392-95. Begg C, Berlin J. Publication bias: a problem in interpreting medical data. J Royal Stat Soc 1988; lSI :419-63. Light R, Pillemer D. Summing up: the science of reviewing research. Cambridge MA: Harvard UP 1984. Chalmers T, Smith H, Blackburn B, et al. A method for assessing the quality of a randomized clinical trial. Controlled Clinical Trials 1981;
2:31-49. 19. Felson D. Bias in meta-analytic research. J Clin Epidemiol 1992;45:885-92. 20. Begg C, Cho M, Eastwood S, et al.lmproving the quality of reporting of randomized controlled trials: The consort statement. J Amer Med Assoc 1996;276:637-9. 21. Mulrow CD, Cook DJ, Davidoff F. Systematic Reviews: Critical Links in the Great Chain of Evidence. In: Mulrow M, Cook D, (eds). Systematic Reviews. Synthesis of Best Evidence For Health Care Decisions. Philadelphia:American College of Physicians, 1998: 1-4. 22. Davidoff F, Haynes B, Sackett D, Smith R. Evidence-based medicine. J Brit Med Assoc 1995;31 0: I 085-6. 23. Glanville J, Lefebvre C. Identifying systematic reviews: key resources. Evidence Based Medicine 2000;5:68-9. 24. Bero L, Rennie D. The Cochrane Collaboration. Preparing, maintaining and disseminating systematic reviews of the effects of health care. journal of the J Amer Med Assoc 1995; 274: 1935-8. 25. Hunt DL, McKibbon KA. Locating and appraising systematic reviews. Ann lnt Med 1997; 126:532-8. 26. Cochrane A. Foreword. In: Chalmers I, Enkin M, Keirse M, (eds). Effective care in pregnancy and childbrith. Oxford: Oxford UP, 1989. 27. Guyatt G, Oxman A, Cook D. Users guide to the medical literature. J Amer Med Assoc 1994;272: 1367-71. 28. Haynes R, Sackett D, Guyatt G, Cook D, Gray ).Transferring evidence form research into practice:4. Overcoming barriers to application. Evidence Based Medicine 1997;2:68. 29. Davis D,Taylor-Vaisey A. Translating guidelines into practice. Can Med Assoc J 1997; 157:408-16. 30. Freemantle N,Wood J, Crawford F. Evidence into practice, experimentation and quasi-experimentation: are the methods up to the task? J Epidemiol Commun Health 1998;52:75-81. 31. Jordens C, Hawe P, lrwig L, et al. Use of systematic reviews of randomised trials by Australian neonatologists and obstetricians. Med J Australia 1998; 168:267-70. 32. Paterson-BrownS, Fisk N,Wyattj. Uptake of meta-analytical overviews of effective care in English obstetric units. Brit J Obstet Gynecol 1995; I 02:297-30 I. 33. Wallace E, Chapman ),Wright $.Antenatal corticosteroid prescribing: setting standards of care. Brit J Obstet Gynecol 1997; I 04: 1262-6. 34. Crowley P. Antenatal corticosteroid therapy: a meta-analysis of the randomized trials.Amer J Obstet Gynecol 1995; 175:322-35.
JOURNAL S O G C . DECEMBER 2000
35. Sinclair J. Meta-analysis of randomized controlled trials of antenatal corticosteroid for the prevention of respiratory distress syndrome. Amer J Obstet and Gynecol 1995; 173:335. 36. Liggins G, Howie R.A controlled trial of antepartum glucocorticoid treatment for prevention of the respiratory distress syndrome in premature infants. Pediatr 1972;50:5 I 5-25. 37. LeeS. Canadian NICU Network. Vancouver: The Centre for Health Evaluation Research, 1999:45. 38. Soli R,Andruscavage D. The principles and practice of evidence-based neonatology. Pediatr 1999; I03:215-24. 39. Worrall G, Chaulk P, FreakeD. The effects of clinical practice guidelines on patient outcomes in primary care: a systematic review. Can Med Assoc J 1997; 156: 1705-12. 40. Oxman A, Thomson M, Davis D, Haynes R. No magic bullets: a systematic review of I02 trials of interventions to improve professional practice. Can MedAssocJ 1995;153:1423-31. 41. Wyatt J, Paterson-Brown S, Johanson R,Aitman D, Bradburn M, Fisk N. Randomised trial of educational visits to enhance use of systematic reviews in 25 obstetrical units. Brit Med Assoc J 1998; 317: I041-6. 42. Feder G, Eccles M, Grol R, Griffiths C, Grimshaw J. Using clinical guidelines. Brit Med Assoc J 1999;3 18:728-30. 43. Fidler H, Lockeyer J,Toews J,Violato C. Changing physicians' practices: the effect of individual feedback.Acad Med 1999;74:702-14. 44. Hunt D, Haynes R, Hanna S, Smith K. Effects of computer-based clinical decision support on physician performance and patient outcomes: a systematic review. J Amer Med Assoc 1998; 280: 1339-46.
A Guide for Health Professionals Working with Aboriginal Peoples th
This issue of the Journal SOGC includes a Self-Directed Learning Module on the guide for health professionals working with Aboriginal peoples. This module qualifies for credits under Section 2 of the Maintenance of Certification Programme of the Royal College ofPhysicians and Surgeons of Canada. Don't miss this valuable selfdirected learning module on the guide for health professionals working with Aboriginal peoples.
SOGC 11 WEST/CENTRAL CME PROGRAM
ai\NFF, AB March 22-24, 2001 The Banff Springs Hotel Watch your mail in the upcoming weeks for the Preliminary Program, it will give you all the information to plan your trip.
Supported by an unrestricted educational grant from
)~f WYETH-AYERST ®~W CANADA INC
JOURNAL S O G C . DECEMBER 2000
The Society of Obstetricians and Gynaecologists of Canada Tel.: (613) 730-4192 Email: [email protected] SOGCnet: www.sogc.org
Abstract: during the 20 years since Glass coined the term "meta-analysis," there has been remarkable growth both in its popularity and in skepticism about its value. The rationale for meta-analysis, also referred to as quantitative systematic review, seems ideal given the large volume of research with which practitioners are faced on a regular basis and the current broad acceptance of the principles of evidence-based medicine. But if a meta-analysis were published in our field of women's health, would it change our clinical practice? A positive answer assumes that if it were published, we would see it, read it, believe it, and welcome a clinical policy or practice guideline based on the meta-analysis, even if it suggested a practice differing from what we normally do, and we would change our practice accordingly. The assumption of a negative answer is that we might not see it or have time to read it, we might not believe it, or it might be presented in such a way that we would not act on it. Which answer is more likely? We can approach this question by reviewing the definitions and methods used in meta-analysis, the factors determining clinical practice, and evidence about whether physicians regularly change their practices on the basis of meta-analysis or guidelines based on meta-analysis. Resume : depuis que Glass a invente le mot « meta-analyse » il y a 20 ans, on assiste a une croissance remarquable de Ia popularite de cette approche aussi bien qu'a une montee du scepticisme quant a sa valeur. La raison d'etre de Ia meta-analyse, qu'on appelle aussi « revue quantitative systematique », semble bien repondre idealement au besoin qu'ont les praticiens d'absorber un volume considerable de recherches de fac;:on reguliere et au fait qu'on accepte de plus en plus universellement les principes de Ia medecine fondee sur les preuves scientifiques. Mais, si on publiait une meta-analyse sur Ia sante des femmes dans notre domaine, est-ce que cela changerait notre pratique clinique ? Une reponse affirmative suppose que, si une telle analyse etait publiee, nous Ia verrions, nous Ia lirions, nous lui accorderions notre confiance et nous ferions bon accueil a des politiques cliniques ou a des !ignes directrices fondees sur cette meta-analyse, meme si elle recommandait une pratique differente de ce que nous faisons normalement, et que nous modifierions notre pratique en consequence. Et si nous repondons par Ia negative, cela peut vouloir dire que nous ne Ia verrions pas, que nous n'aurions pas le temps de Ia lire, que nous ne lui ferions pas confiance ou qu'il se pourrait qu'elle soit presentee d'une fac;:on telle qu'elle n'influencerait pas notre conduite. Laquelle de ces deux reponses est Ia plus vraisemblable ? Nous essayons de repondre a cette derniere question en examinant les definitions et les methodes appliquees a Ia meta-analyse, les facteurs qui determinent Ia pratique clinique et les donnees qui font voir si les medecins changent regulierement leur fac;:on de pratiquer sous !'influence d'une meta-analyse ou de !ignes directrices basees sur des meta-analyses.
J Soc
Obstet Gynaecol Can 2000;22( 12): I050-4
INTRODUCTION
KeyWords Meta-analysis, critical appraisal Competing interest: none declared Received on May 18th, 2000. Revised and accepted on November 7th, 2000.
During the 20 years since Glass 1 coined the term "meta-analysis," there has been remarkable growth both in its popularity and in skepticism about its value. 1· 7 The rationale for metaanalysis, also referred to as quantitative systematic review, seems ideal given the large volume of research with which practitioners are faced on a regular basis and the current broad acceptance of the principles of evidence-based medicine. 8 But if a meta-analysis were published in our practice area, would it change our practice? A positive answer assumes that if it were published, we would see it, read it, believe it, and welcome a clinical policy or practice guideline JOURNAL S O G C . DECEMBER 2000
based on the meta-analysis, even if it suggested a practice differing from what we normally do, and we would change our practice accordingly. The assumption of a negative answer is that we might not see it or have time to read it, we might not believe it, or it might be presented in such a way that we would not act on it. Which answer is more likely? We can approach this question by reviewing the definitions and methods used in meta-analysis, the factors that determine clinical practice, and evidence about whether physicians regularly change their practices on the basis of meta-analysis or guidelines based on meta-analysis. WHAT IS A META-ANALYSIS?
A meta-analysis is the analysis of a large collection of results from individual srudies for the purpose of integrating the findings. 1•9 In clinical medicine they most often consist of summaries of the results of clinical trials, but they may be summaries of other types of srudies as well. 10 Meta-analyses have the advantage of offering more precise conclusions than individual studies, and can provide more definitive estimates of a particular clinical effect. 10 In some cases, they comprise an effective approach to answering primary research questions or resolving conflicts in the interpretation of results from multiple studies. 10•11 Metaanalyses are often done as components of systematic reviews, but they may also be presented as independent analysis. METHODS USED IN META-ANALYSIS
The methods used in meta-analysis have been described in detail elsewhere. 1•2 •10 •12•13 The first step in performing a metaanalysis is the formulation of a clear and relevant research question that could be answered by that technique. Next, the variables to be used in the meta-analysis are specified, including exposure and outcome variables and any confounding vari-
FIGURE I
FUNNEL PLOT OF 35 HYPOTHETICAL STUDY RESULTS DEMONSTRATING AN ABSENCE OF PUBLICATION BIAS
CD
N
''
10000
0
-1.E ('0
(I)
1000
10 0.1
/
'
.•••
'\
-
\\\
··' .. • • • ••••• . /
100
'
I
.././ / • /
'
• •••
FACTORS DETERMINING CUNICAL PRACTICE
• \\\
EVIDENCE
,., \,\,
1
ables. Subsequently, the types of reports to be collected for the analysis are determined. The authors then proceed with a comprehensive literature search, which may include data from full articles and abstracts, from published and unpublished literature, from reports published in English, French, and perhaps other languages. The results of the included studies are then abstracted, with the investigator ensuring that the patients, interventions, outcome measures, and research methodology are similar before the analysis begins. In the past, there were valid concerns that clinically useful decisions could not be drawn by comparing or combining studies which used different techniques, different definitions, and so on; but as experience has been gained in the methods used in meta-analysis and in the ways they can be presented, these issues are becoming resolved. 6•11 •14 Assessment of potential bias is a part of every analysis. In the case of meta-analysis, two common sources of bias are the failure of investigators to find or include all relevant srudies (publication bias), 10•15· 16 and the uncertain reliability and quality of included srudies. Publication bias can be estimated graphically using a "funnel plot" 17 in which sample size is plotted against effect size. If many srudies have been located that estimate a similar effect, the distribution of the data points should result in an inverted, funnel-shaped symmetrical appearance (figure 1). A gap on one side of the wider part of the funnel indicates that perhaps not all results have been published or located. 17 Quality scores are included in some meta-analysis on the basis of randomization details, sample size, and other fearures, but approaches to quality assessment are not standardized and are not always induded. 18-20 Another analytical question that instigators of a metaanalyses have to decide is whether the data being combined is heterogeneous from a statistical point of view. 5 Heterogeneity can arise in two basic ways. If a population of very similar individuals is subjected to identical treatments, there will still be some individual differences of response. This represents "within study variance." Additionally, each study population is drawn from a different sample of individuals, so no matter how careful one is and no matter how large a collection of studies is located, there are still going to be some differences in the results of individual studies. This represents "between study variance." If either of these types of variance is large, the results of individual studies are not consistent with one another, and the data is considered heterogenous. 5
10
Risk Ratio
JOURNAL SOGC
Patient data, or the history, physical, and laboratory findings that a particular patient exhibits, is a major determinant of all clinical decisions, but the contribution of research evidence to clinical decisions is more difficult to determine. Findings published in the biomedical research world are often exciting and DECEMBER 2000
sometimes have the potential to influence our clinical practices. However, there are currently over 20,000 journals published regularly worldwide. Davidoff et aL suggested that to keep up with current medical literature, a general physician would have to read 17 articles each day. 22 No matter what strategy we choose to try to "keep up" with relevant research and reports, their sheer volume may prevent this. However, there are several approaches designed to help practitioners by summarizing, synthesizing, and disseminating the best available evidence, sometimes in the form of meta-analysis and sometimes in other formats. 21 •23-25 A major stimulus to the identification, appraisal, and synthesis of research evidence is the work of the Cochrane Collaboration.24·26 Obstetricians and gynaecologists have played key pioneering roles in this area. In fact, Archie Cochrane, the instigator of the collaboration, referred to the systematic reviews of randomized controlled trials of care during pregnancy and childbirth as "a real milestone in the history of randomized trials and the evaluation of care." 26 Approaches to critical appraisal of individual studies help practitioners to identifY relevant articles and critically approach their reading and interpretation. 27 However, we may not always have the time, motivation or skills needed to find, critically appraise, and synthesize information de novo. Fortunately, systematic reviews developed through the Cochrane collaboration and other sources, many of which include meta-analysis, enhance our ability to identifY, interpret, and apply research evidence. Valuable sources for systematic reviews are the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effectiveness (DARE), ACP Journal Club, Evidence-Based Medicine, and Clinical Evidence. Information from these and other sources have been important in many new and exciting initiatives in the provision of continuing medical education. 23 PATIENT AND PHYSICIAN FACTORS
Another major determinant of clinical decision-making is the background and daily lives of patients and physicians, including their personal values, experiences, and education. 21 Professional relationships between physicians and patients have changed from an authoritarian to a shared approach to clinical decision making. Reasons for the change include broad societal changes, changes in the education of the physicians and of the public, and easy access to vast amounts of health information through the Internet. As physicians, although we might choose to make most of our clinical decisions primarily on the basis of the objective patient data, the research evidence available, and the patient's input regarding personal needs and preferences: our decisions cannot be made in a void, free from our own or our patients' past experiences, preferences, and background-nor should they be.
JOURNAL SOGC
CONSTRAINTS
Constraints on clinical decision-making comprise a changing target in ways beyond the scope of this paper. However, current constraints are more numerous and may be different than was the case in the recent past. It has been pointed out that even when evidence is strong and clinical policy is sound, there are impediments to incorporating evidence from research into practice. 28 One of these impediments has been described as a mismatch between evidence and clinical circumstances. 28 For example, even if certain diagnostic procedures have been found to be effective, if the local resources and logistics do not support the procedures, a physician may not be able to offer them to patients who could benefit. Another constraint is time pressure in clinical practice, and a third is difficulty in learning new clinical skills. 28 This last may be becoming less of a constraint as evidence continues to accumulate in areas such as optimal delivery of continuing medical education, computerized decision support approaches, and measures to enhance patient compliance. EFFECT OF FACTORS DETERMINING CLINICAL PRACTICE
Evidence, patient and physician factors, and constraints act together to form the clinical decisions that we are called upon to make many times a day. It is difficult to single out the effects of any one of these factors on clinical practice, and even more difficult to single out the effects of only one component of one of the factors: in the case of this paper, evidence that has been presented as a meta-analysis. Indeed, as Davis notes: "Even when a valid systematic review leads to a Clinical Practice Guideline (CPG), there are no guarantees that a change in clinical practice will follow." 29 Three possible considerations in determining such effects are the lag time between publication of a meta-analysis and changes in physician practices, the extent to which clinical practice guidelines based on meta-analysis appear to change the process and outcome of clinical practice, 30 and reports of the use of meta-analysis and changes in practice by obstetricians, gynaecologists, and neonatologists. 31 · 32 A commonly cited example of the lag time between publication of both consistent individual clinical trials and metaanalysis and change in physician practices is the administration of antenatal steroids to women presenting with threatened preterm delivery at 34 weeks or less. 33-35 In the 20 years following publication of the initial clinical trial, 36 18 additional trials and, more recently, meta-analysis34·35 have been reported supporting the effectiveness of antenatal steroids in preventing neonatal deaths associated with respiratory distress syndrome. Despite these results, several studies have confirmed that many women whose infants could benefit from this therapy have not received it. 37·38 For example, in a survey of 17 Canadian neonatal intensive care units in 1997, the rate of antenatal steroid DECEMBER 2000
treatment in infants born at or before 34 weeks gestation varied from 27 percent to more than 95 percent, with an average of 61.5 percent.37 This demonstrates that the simple publication of a metaanalysis or even of a clinical practice guideline alone does not appear to be effective in changing physician practice. Some reports have described self-reported practice changes by neonatologists and obstetricians. 31 Worrall et al, reviewing the effects of clinical practice guidelines in primary care, found little evidence that the use of clinical practice guidelines improved patient outcomes. 39 However, several current studies focus on strategies for the dissemination and implementation of guidelines. 40 • 41 Reports in this area are accumulating through the work of such groups such as the Cochrane Effective Practice and Organization of Care Group,24 DARE, and the Department of Health Care and Promotion of the Canadian Medical Association. 29 The use of clinical practice guidelines varies according to resource availability, institutional or practice specific barriers, the context of introduction of the guideline, as well as other factors. Techniques which have positively influenced the utilization of guidelines include: professional interventions such as unique continuing education approaches, 42 audit and feedback, 43 organizational interventions such as incorporating reminders or standard treatment recommendations in computerized order entry programmes, and financial and regulatory interventions. 44 CONCLUSION
There are many situations where meta-analysis are valuable and deserving of our close attention. Yet publication of a meta-analysis alone, or a clinical practice guideline which includes a metaanalysis, is unlikely to change practice. However, evidence is only one of the determinants of clinical practice. Changing clinical practice may require several innovative and multifaceted approaches particular to each meta-analysis or guideline, as well as specific context within which a change in practice might be desirable. Specific approaches such as systemic reviews are available to help practitioners to obtain and critically appraise research findings and meta-analysis, and to help with the ongoing accumulation and interpretation of up-to-date knowledge. These approaches are helpful but they are only part of the process of creating change in clinical practice. REFERENCES I. 2. 3. 4. 5.
Glass G, McGaw B, Smith M. Meta-analysis in social research. Beverly Hills CA: Sage, 1981. Abramson J. Meta-analysis: a review of the pros and cons. Public Health Reviews 1990/91; 18:1-47. Chalmers T. Problems induced by meta-analysis. Stat Med 1991; I0:971-80. Bailar JC.The problems and promise of meta-analysis. N Eng J Med 1997;337:559-61. Thompson S, Pocock S. Can meta-analysis be trusted? Lancet 1991; 338:1127-30.
6. 7. 8.
9. I0. II. 12.
13. 14. 15. 16. 17. 18.
Naylor C. Two cheers for meta-analysis: problems and opportunities in aggregating results of clinical trials. Can Med Assoc J 1988; 138:891-5. Shapiro S. Meta-analysis/schmeta-analysis. Amer J Epidemiol 1994; 140:771-8. Sackett DL, Richardson WS, RosenbergW, Haynes RB. Evidence-Based Medicine: How To Practice and Teach EBM. New York: Churchill Livingtone, 1997. Greenland S. Quantitative methods in the review of epidemiologic literature. Epidemiol Rev 1987;9: 1-30. Dickerson K, Berlin J. Meta-analysis: state-of-the-science. Epidemiol Rev 1992; 14:154-76. Greenland S. Can meta-analysis be salvaged? Amer J Epidemiol 1994; 140:783-7. Wolf F. Meta-analysis: Quantitative Methods For Research Synthesis. in: Sullivan JL, Niemi RG (eds). Quantitative Applications in the Social Sciences. Vol. 59. Beverly Hills: SAGE 1986:1-65. Yusuf S, Simon R, Ellenberg S. Methodologic issues in overviews of randomized clinical trials. Stat Med 1987;6:217-409. Spitzer W. Meta-meta-analysis: unanswered questions about aggregating data.) Clin Epidemiol 1991;44:103-7. Chalmers T, Frank C, Reitman D. Minimizing the three stages of publication bias. J Amer Med Assoc 1990;2263: 1392-95. Begg C, Berlin J. Publication bias: a problem in interpreting medical data. J Royal Stat Soc 1988; lSI :419-63. Light R, Pillemer D. Summing up: the science of reviewing research. Cambridge MA: Harvard UP 1984. Chalmers T, Smith H, Blackburn B, et al. A method for assessing the quality of a randomized clinical trial. Controlled Clinical Trials 1981;
2:31-49. 19. Felson D. Bias in meta-analytic research. J Clin Epidemiol 1992;45:885-92. 20. Begg C, Cho M, Eastwood S, et al.lmproving the quality of reporting of randomized controlled trials: The consort statement. J Amer Med Assoc 1996;276:637-9. 21. Mulrow CD, Cook DJ, Davidoff F. Systematic Reviews: Critical Links in the Great Chain of Evidence. In: Mulrow M, Cook D, (eds). Systematic Reviews. Synthesis of Best Evidence For Health Care Decisions. Philadelphia:American College of Physicians, 1998: 1-4. 22. Davidoff F, Haynes B, Sackett D, Smith R. Evidence-based medicine. J Brit Med Assoc 1995;31 0: I 085-6. 23. Glanville J, Lefebvre C. Identifying systematic reviews: key resources. Evidence Based Medicine 2000;5:68-9. 24. Bero L, Rennie D. The Cochrane Collaboration. Preparing, maintaining and disseminating systematic reviews of the effects of health care. journal of the J Amer Med Assoc 1995; 274: 1935-8. 25. Hunt DL, McKibbon KA. Locating and appraising systematic reviews. Ann lnt Med 1997; 126:532-8. 26. Cochrane A. Foreword. In: Chalmers I, Enkin M, Keirse M, (eds). Effective care in pregnancy and childbrith. Oxford: Oxford UP, 1989. 27. Guyatt G, Oxman A, Cook D. Users guide to the medical literature. J Amer Med Assoc 1994;272: 1367-71. 28. Haynes R, Sackett D, Guyatt G, Cook D, Gray ).Transferring evidence form research into practice:4. Overcoming barriers to application. Evidence Based Medicine 1997;2:68. 29. Davis D,Taylor-Vaisey A. Translating guidelines into practice. Can Med Assoc J 1997; 157:408-16. 30. Freemantle N,Wood J, Crawford F. Evidence into practice, experimentation and quasi-experimentation: are the methods up to the task? J Epidemiol Commun Health 1998;52:75-81. 31. Jordens C, Hawe P, lrwig L, et al. Use of systematic reviews of randomised trials by Australian neonatologists and obstetricians. Med J Australia 1998; 168:267-70. 32. Paterson-BrownS, Fisk N,Wyattj. Uptake of meta-analytical overviews of effective care in English obstetric units. Brit J Obstet Gynecol 1995; I 02:297-30 I. 33. Wallace E, Chapman ),Wright $.Antenatal corticosteroid prescribing: setting standards of care. Brit J Obstet Gynecol 1997; I 04: 1262-6. 34. Crowley P. Antenatal corticosteroid therapy: a meta-analysis of the randomized trials.Amer J Obstet Gynecol 1995; 175:322-35.
JOURNAL S O G C . DECEMBER 2000
35. Sinclair J. Meta-analysis of randomized controlled trials of antenatal corticosteroid for the prevention of respiratory distress syndrome. Amer J Obstet and Gynecol 1995; 173:335. 36. Liggins G, Howie R.A controlled trial of antepartum glucocorticoid treatment for prevention of the respiratory distress syndrome in premature infants. Pediatr 1972;50:5 I 5-25. 37. LeeS. Canadian NICU Network. Vancouver: The Centre for Health Evaluation Research, 1999:45. 38. Soli R,Andruscavage D. The principles and practice of evidence-based neonatology. Pediatr 1999; I03:215-24. 39. Worrall G, Chaulk P, FreakeD. The effects of clinical practice guidelines on patient outcomes in primary care: a systematic review. Can Med Assoc J 1997; 156: 1705-12. 40. Oxman A, Thomson M, Davis D, Haynes R. No magic bullets: a systematic review of I02 trials of interventions to improve professional practice. Can MedAssocJ 1995;153:1423-31. 41. Wyatt J, Paterson-Brown S, Johanson R,Aitman D, Bradburn M, Fisk N. Randomised trial of educational visits to enhance use of systematic reviews in 25 obstetrical units. Brit Med Assoc J 1998; 317: I041-6. 42. Feder G, Eccles M, Grol R, Griffiths C, Grimshaw J. Using clinical guidelines. Brit Med Assoc J 1999;3 18:728-30. 43. Fidler H, Lockeyer J,Toews J,Violato C. Changing physicians' practices: the effect of individual feedback.Acad Med 1999;74:702-14. 44. Hunt D, Haynes R, Hanna S, Smith K. Effects of computer-based clinical decision support on physician performance and patient outcomes: a systematic review. J Amer Med Assoc 1998; 280: 1339-46.
A Guide for Health Professionals Working with Aboriginal Peoples th
This issue of the Journal SOGC includes a Self-Directed Learning Module on the guide for health professionals working with Aboriginal peoples. This module qualifies for credits under Section 2 of the Maintenance of Certification Programme of the Royal College ofPhysicians and Surgeons of Canada. Don't miss this valuable selfdirected learning module on the guide for health professionals working with Aboriginal peoples.
SOGC 11 WEST/CENTRAL CME PROGRAM
ai\NFF, AB March 22-24, 2001 The Banff Springs Hotel Watch your mail in the upcoming weeks for the Preliminary Program, it will give you all the information to plan your trip.
Supported by an unrestricted educational grant from
)~f WYETH-AYERST ®~W CANADA INC
JOURNAL S O G C . DECEMBER 2000
The Society of Obstetricians and Gynaecologists of Canada Tel.: (613) 730-4192 Email: [email protected] SOGCnet: www.sogc.org