- Email: [email protected]
Interpreting the Results of Systematic Reviews
Interpreting the Results of Systematic Reviews Mike Clarke Systematic reviews are increasingly common and often provide the most robust research approach to informed decisions about health care. The successful interpretation of systematic reviews for decisions about health care requires consideration of their quality and judgement on whether the participants and the interventions assessed are applicable to a different setting. This chapter includes a table of questions to help the reader consider this issue. If a review satisfies the requirements of the present setting, the benefits and harms of the interventions, along with other information including patient preferences, will help the clinician decide what the review “means” and what to do about this in practice. Semin Hematol 45:176-180 © 2008 Elsevier Inc. All rights reserved.
S
ystematic reviews are increasingly common in the health literature and often provide the most robust research strategy by which to inform decisions about health care. There are now many such reviews, covering the full breadth of medical specialties. For example, in hematology, systematic reviews include projects seeking to collect data on each child in every randomized trial of the treatment of acute lymphoblastic leukemia,1,2 and others that examine supportive care, such as the use of antibiotics to prevent pneumonia in patients immunocompromised by their treatment.3 Systematic reviews attempt to identify, appraise, and summarize research on a particular topic, sometimes including meta-analyses to combine the results of the included studies. An estimated 2,500 systematic reviews are published each year (based on data gathered in 2004), of which the majority relate to questions about the effects of interventions.4 This chapter focuses on the interpretation of these reviews. However, there are systematic reviews of other types of study, some of which are discussed elsewhere in this issue of Seminars in Hematology, and many of the general principles outlined here would apply to such reviews as well. This chapter will guide the reader through some of the steps to consider when interpreting the results of a systematic review. These steps parallel the approach followed by the researchers who did the review when they tried to conclude their work with an answer to the question “what does this review mean?”. Considering how the opinions of the original authors might be used to interpret the results of the systematic review is, therefore, a good place to start.
School of Nursing and Midwifery, Trinity College Dublin, Dublin, Ireland. Address correspondence to Mike Clarke, School of Nursing and Midwifery, Trinity College Dublin, 24 D’Olier St, Dublin 2, Ireland. E-mail: [email protected]
176
0037-1963/08/$-see front matter © 2008 Elsevier Inc. All rights reserved. doi:10.1053/j.seminhematol.2008.04.002
Using the Authors’ Conclusions For many researchers confronted with a systematic review, the first place they might look after deciding that the review is appropriate to their topic of interest is to the conclusions section. In some systematic reviews, such as Cochrane reviews (which make up about a quarter of the new and updated systematic reviews published in 20044), these conclusions are easily identifiable within the review because of its structure. Cochrane reviews are produced by the Cochrane Collaboration, which is the world’s largest organization dedicated to the preparation and maintenance of systematic reviews of the effects of healthcare interventions. The Collaboration was established in the early 1990s, and the growth in systematic reviews through the last decade is in some part due to its work. Cochrane reviews are published in full in the Cochrane Library. From a few dozen in 1995, the number has increased to more than 3,400 in 2008. The reviews encompass all areas of health care, with one of 51 Cochrane Review Groups taking editorial responsibility for each review. Within hematology, there is a Cochrane Hematological Malignancies Group based in Cologne, Germany.5 As of the beginning of 2008, this Group had produced 11 full Cochrane reviews and published protocols for a further 22. All Cochrane reviews include an Authors’ Conclusions section, divided into the Implications for Practice and the Implications for Research. These can provide rich sources of information for anyone wishing to make decisions about health care, but, of course, the degree of certainty they provide depends on the quality of the research that has been identified and included in the review. Although many Cochrane reviews include clear implications concerning the effectiveness or ineffectiveness of interventions,6 the majority
Interpreting the results of systematic reviews find that the effects of interventions remain uncertain and highlight the need for further research.7 Recommendations for further research will be useful to those planning new studies, highlighting the scientific and ethical need for researchers planning such studies to begin with a systematic review.8 Suggestions on how this aspect of the reporting of systematic reviews would be made more helpful were published in 2006 and provide a structure that also can be used if an investigator is faced with decisions as to the direction, need, and design of further trials.9 However, most readers of a systematic review lack the ability to initiate a research project and, therefore, they need to interpret the results of the review in a way that will influence their practice. This effort might mean drawing conclusions beyond those written by the original authors of the review. In doing so, the reader will need to consider both the results of the review and also the context within which the decision is being made.
Considering the Results of the Review An important first step in interpreting the results of the review is to decide whether to believe it is a valid source of evidence for the purposes at hand and the treatment decision being considered. This process also relates to whether the review is of sufficient quality and includes recent data.
Is the Review Good Enough? Several articles have been written over the last couple of decades addressing the quality of reviews. Mulrow highlighted, 20 years ago, that most reviews at that time were far from systematic, were open to substantial risks of bias, and were unclear about their methods.10 Since then, attempts have been made to improve the quality of the conduct and the reporting of systematic reviews. For example, extensive manuals have been published on how to undertake reviews,11,12 and the Quality of Reporting of Meta-analyses (QUOROM) statement has sought to improve the quality of
177 their reporting.13 The latter is currently being updated and its revised version, called Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA), is expected to be published in 2008. Tools have also been developed to allow readers to assess the quality of systematic reviews and these, along with ways to critically appraise other types of research, have been reviewed by Katrak et al.14 Several ad hoc tools have been used for individual projects, but those that are most commonly applied in assessments of the quality of systematic reviews are based on the Overview Quality Assessment Questionnaire (OQAQ) developed by Oxman and Guyatt15,16 and the QUOROM checklist.13 The OQAQ and the QUOROM have been used in a wide range of assessments of systematic reviews and meta-analyses during the last decade, with several such studies being reported recently. One example of such studies compared updated Cochrane reviews published in 2002 with their preceding versions. Both the OQAQ and the QUOROM were employed; the relevant instruments were published on the internet, within their open access article.17 The researchers also compared the outcomes of using the two tools and found that they gave similar results in general. An earlier study by some of the same team compared the OQAQ with another instrument, the 23-item Sacks tool,18 and concluded that there were advantages for using both. The OQAQ provided a numerical score that could be used to benchmark changes over time, while the Sacks scale provided a means of assessing additional, important aspects of each review.19 However, caution should be observed when using a single number to score the quality of a systematic review. It is essential to consider in detail the specific strengths and weaknesses of any review. Specific weaknesses may have multiplicative detrimental effects in some settings but may be less important for answering certain medical questions. Table 1 provides a list of questions adapted from the OQAQ instrument, which a clinician might pose when considering whether or not a systematic review was conducted to
Table 1 Some Questions to Ask When Interpreting the Results of a Systematic Review 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
Do the objective and the inclusion criteria for this review match the question I am seeking an answer for? Were the search methods used to find studies relevant to the main questions of the review stated? Was the search for studies reasonably comprehensive? How likely is it that additional relevant research will have been started, completed, or published since this review was done? Were the criteria used for deciding which studies to include in the review reported? Did the reviewers avoid bias when deciding if studies were eligible? Did the selection of studies (in particular the choice of eligible study designs) minimize the possibility of including studies with a high propensity for bias? Were appropriate criteria used to assess the validity of the included studies? Was the validity of all included studies assessed using these criteria and were the findings reproducible? Were appropriate methods used to combine the findings of the relevant studies? Were the conclusions made by the authors supported by the results of the systematic review? How likely is it that any additional relevant research that had started, been completed, or been published since this review was done would change its conclusions? Is there any reason to believe that the findings of this review are not relevant to the types of patient, intervention, or setting I am interested in?
178 a sufficiently high standard to provide reliable knowledge to help with a treatment decision. However, even if a review was of high quality when it was done, the reader will also need to consider whether it is sufficiently up to date to be reliable.
Is the Review Sufficiently Up to Date? To be truly current, a systematic review would need to have been completed just before it was used and to have had access, at that time, to all the relevant research. This is clearly impractical since it would require reviews to be done in “real time” and be constantly updated whenever new evidence became available. Such a “real time” approach might be straightforward if reviews were simply mathematical analyses of datasets into which new data could be input, leading to new meta-analysis, but reviews are more complex than that. The reviewers need to appraise the quality of the included studies, make decisions about the suitability of the outcome data from these studies, and determine how best to combine these results in their review. They need to have searched for and identified eligible studies. Hence, the process of conducting and, later, updating a systematic review can take considerable time. When determining whether a review is sufficiently recent to help in making a current decision, the reader needs to examine whether the reviewers have omitted any studies that might lead to changes to the review and therefore changes in the interpretation of its conclusons.20 This difficult judgement can be built on information about when the searches were done for the review, the reader’s knowledge of whether any important potentially eligible studies had since become available, and, if necessary, a repeat of the reviewers’ searches of electronic bibliographic databases such as MEDLINE and the Cochrane Central Register of Controlled Trials to check for other publications or reported results. Occasionally, among trials studying the same topic, those that find large, statistically significant benefits are published faster while those with more “negative” results take longer to appear (a “time lag” bias).21 Thus, a comprehensive appreciation of the status of the evidence may be needed to assess also whether there are additional registered trials that are ongoing and their results still pending. Improvements in trial registration (see, for example, http://www.who.int/ictrp/en/) have made this anticipation of the results of future trials increasingly feasible. However, many clinical trials still remain unregistered.
Putting the Results of the Review Into Context as Applies to a Treatment Decision When deciding whether the results of a systematic review of the effects of healthcare interventions are applicable to a treatment decision, the clinician must decide whether these results provide valid information about potential benefits and harms in circumstances relevant to the current setting. If the clinician has decided that the review set out to cover the applicable area of interest and is of adequate quality, he/she needs to
M. Clarke decide whether there are any strong reasons why the participants in the included studies and the interventions assessed in these studies are not sufficiently similar to the people and interventions in the present circumstances to allow the results to apply to them.22 For example, if the reader is interested in treating acute lymphoblastic leukemia in children under 6 years of age, the results from a systematic review of chemotherapy for adults are probably not suitable, but the overall result from a review in which all the participants were aged between 5 and 16 might be. As another example, when interpreting the benefits and harms from the trials in a review of bone marrow transplantation, the reader will need to consider what additional interventions and care were used for the patients. Are these similar to those available in the current setting? Was the additional health care better in the trials than what can be offered in the current situation? Or is it better in the present setting? What effect might this have on the balance of the benefits and harms reported in the review? Some reviews might contain enough information to raise the issue of whether a decision should focus on the overall results of the review or on the results in a particular subgroup, which might be closer to the circumstances of the current case. Readers whose inclination is to focus on the findings from a subgroup need to be cautious. It is important to remember that one of the strengths of a systematic review is that it brings together more data than are available for any specific trial or subgroup. This strategy can lead to a more precise estimate of the effects of the interventions under investigation. If the included studies are sufficiently similar for this average to be sensible, it might be a more accurate estimate of the effect in any subgroup than the specific result from that subgroup because the effects of chance might deflect the estimate in the subgroup more than the overall effect.23-25 In interpreting the results from subgroup analyses in the systematic review, it should be considered whether the rationale for these analyses is to identify different effects among the subgroups or to identify similarities. For example, do the analyses provide reassurance that a treatment is beneficial regardless of specific patient characteristics (such as age) or will cross from beneficial to harmful as a characteristic changes (such as white blood cell count)? In a large individual patient data overview of nearly 7,000 patients receiving different chemotherapy regimens for myeloma, the large number of subgroup analyses that were possible because of the central collection of patient level data on many baseline characteristics revealed consistency across the subgroups.26 Typically, the analyses in systematic reviews are performed using meta-analyses based on relative statistics, such as odds ratios, hazard rates, and relative risks. These statistics are assumed to be more stable for combining the results of the included studies than are absolute statistics such as risk difference or number needed to treat.27 However, when interpreting the results of a systematic review as applied to current circumstances, it may be desirable to convert these relative statistics to absolute values, especially if the balance of benefits and harms for a treatment might be different in the
Interpreting the results of systematic reviews current setting as compared to the settings of the studies in the review.
Converting a Relative Result From a Review Into an Absolute Answer for the Present Circumstance If a systematic review has compared the effects of two interventions and produced a relative statistic as the result of this comparison, the usual assumption is that this relative result will be stable for different populations that are similar to the individuals in the studies in the review. However, the baseline risk of persons in those studies might be very different from that in the current setting. Despite this, the clinician can still move easily from the review’s result to an estimate of what might happen for the current patient or population. As an example, a recent review considered the addition of rituximab to chemotherapy for patients with B-cell nonHodgkin’s lymphoma. It combined the results of seven randomized trials, including nearly 2,000 adult patients, all of whom contributed data to the overall survival analysis. This produced a pooled hazard ratio for mortality of 0.65 (95% confidence interval, 0.54 to 0.78; P ⬍.00001).28 This value can then be applied to a variety of prognostic scenarios to help estimate the survival benefit for individual patients, allowing interpretation of whether the effect is worthwhile in comparison to concerns about harms or other negative aspects of the use of rituximab. If a patient has a 10% chance of dying within 2 years if treated with chemotherapy alone, this would be reduced to 6.5% by the addition of rituximab, equivalent to a number needed to treat (NNT) of 29, for one more survivor at 2 years. If the patient’s chance of dying had been 5%, rituximab might change this to 3.25% (an NNT of 57). But, if the patient’s prognosis were worse, with a 20% risk of dying in the next 2 years, the absolute benefit would be a reduction to 13% and an NNT of 14.
Other Factors to Consider When Interpreting the Results of Systematic Reviews The systematic review should provide the reader with reliable estimates of the relative effects of interventions that are being considered for patients about whom the reader needs to make decisions. However, the interpretation of these results into practice will require the consideration of additional information that was probably not contained in the review. This information includes the preferences of the patient for the different interventions and how he or she values the outcomes that might be made more, or less, likely by the treatment, as well as the clinician’s ability to deliver the preferred intervention in a satisfactory manner.
179
Conclusions Systematic reviews are an increasingly common source of knowledge for healthcare decisions, with at least 2,500 appearing every year. Several such reviews are now available for questions relating to treatment in hematology. The successful interpretation of the results of these reviews into the basis for decisions requires the consideration of the quality of the conduct of the review and whether it is sufficiently current. It also requires a judgement on whether the participants and the interventions assessed in the studies in the review are too different from the reader’s specific circumstances for the results to be applicable. If the results are applicable, the benefits and harms can be considered, along with other information including patient preferences, in determination of a final decision about what the review means and how to best use its conclusions.
References 1. Childhood ALL Collaborative Group: Duration and intensity of maintenance chemotherapy in acute lymphoblastic leukaemia: overview of 42 trials involving 12,000 randomised children. Lancet 347:17831788, 1996 2. Clarke M, Gaynon P, Hann I, Harrison G, Masera G, Peto R, et al: CNS-directed therapy for childhood acute lymphoblastic leukaemia: Childhood ALL Collaborative Group overview of 43 randomized trials. J Clin Oncol 21:1798-1809, 2003 3. Green H, Paul M, Vidal L, Leibovici L: Prophylaxis for pneumocystis pneumonia (PCP) in non-HIV immunocompromised patients. Cochrane Database of Systematic Reviews Issue 3:CD005590, 2007 4. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4:e78, 2007 5. Weingart O, Kober T, Engert A, Diehl V, Djulbegovic B, Glasmacher A, et al: Cochrane Haematological Malignancies Group. About The Cochrane Collaboration (Cochrane Review Groups (CRGs)) Issue 2:Art. No.: HAEMATOL, 2006 6. El Dib RP, Atallah AN, Andriolo RB: Mapping the Cochrane evidence for decision making in health care. J Eval Clin Pract 13:689-692, 2007 7. Clarke L, Clarke M, Clarke T: Implications for research from Cochrane reviews. J Health Services Res Policy 12:101-103, 2007 8. Clarke M: Doing new research? Don’t forget the old: Nobody should do a trial without reviewing what is known. PLoS Med 1:100-102, 2004 9. Brown P, Brunnhuber K, Chalkidou K, Chalmers I, Clarke M, Fenton M, et al: How to formulate research recommendations. BMJ 333:804806, 2006 10. Mulrow C: The medical review article: state of the science. Ann Intern Med 106:485-488, 1987 11. Khan KS, ter Riet G, Glanville J, Sowden AJ, Kleijnen J (eds): Undertaking Systematic Reviews of Research on Effectiveness (ed 2). York, UK, NHS Centre for Reviews and Dissemination, University of York, 2001 12. Higgins JPT, Green S (eds): Cochrane Handbook for Systematic Reviews of Interventions 4.2.6 [updated September 2006]. http://www. cochrane.org/resources/handbook/hbook.htm (accessed December 6, 2007) 13. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Lancet 354:1896-1900, 1999 14. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA: A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 4:22, 2004 15. Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol 44:1271-1278, 1991 16. Oxman AD: Checklists for review articles. BMJ 309:648-651, 1994
180 17. Shea BJ, Boers M, Grimshaw JM, Hamel CD, Bouter LM: Does updating improve the methodological and reporting quality of systematic reviews? BMC Med Res Methodol 6:27, 2006 18. Sacks H, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC: Metaanalyses of randomized controlled trials. N Engl J Med 316:450-455, 1987 19. Shea B, Moher D, Graham I, Pham B, Tugwell P: A comparison of the quality of Cochrane reviews and systematic reviews published in paperbased journals. Evaluation Health Professions 25:116-129, 2002 20. Shojania KG, Sampson M, Ansari MT, Doucette S, Moher D: How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147:224-233, 2007 21. Hopewell S, Clarke M, Stewart L, Tierney J: Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews 2:MR000011, 2007 22. Guyatt GH, Sackett DL, Cook DJ, Evidence-Based Medicine Working Group: Users’ guides to the medical literature, II: How to use an article about therapy or prevention, B: What were the results and will they help me in caring for my patients. JAMA 271:59-63, 1994
M. Clarke 23. Yusuf S, Wittes J, Probstfield J, Tyroler HA: Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials. JAMA 266:93-98, 1991 24. Oxman AD, Guyatt GH: A consumer’s guide to subgroup analyses. Ann Intern Med 116:78-84, 1992 25. Counsell CE, Clarke MJ, Slattery J, Sandercock PAG: The miracle of DICE therapy for acute stroke: Fact or fictional product of subgroup analysis? BMJ 309:1677-1681, 1994 26. Myeloma Trialists Collaborative Group: Combination chemotherapy versus melphalan plus prednisone as treatment for multiple myeloma: An overview of 6,633 patients from 27 randomized trials. J Clin Oncol 16:3832-3842, 1998 27. Deeks JJ, Altman DG: Effect measures for meta-analysis of trials with binary outcomes, in Egger M, Davey Smith G, Altman DG (eds): Systematic Reviews in Health Care (ed 2). London, UK, BMJ Publishing Group, 2001, pp 313-335 28. Schulz H, Bohlius J, Skoetz N, Trelle S, Kober T, Reiser M, et al: Chemotherapy plus Rituximab versus chemotherapy alone for B-cell non-Hodgkin’s lymphoma. Cochrane Database of Systematic Reviews 4:CD003805, 2007
S
ystematic reviews are increasingly common in the health literature and often provide the most robust research strategy by which to inform decisions about health care. There are now many such reviews, covering the full breadth of medical specialties. For example, in hematology, systematic reviews include projects seeking to collect data on each child in every randomized trial of the treatment of acute lymphoblastic leukemia,1,2 and others that examine supportive care, such as the use of antibiotics to prevent pneumonia in patients immunocompromised by their treatment.3 Systematic reviews attempt to identify, appraise, and summarize research on a particular topic, sometimes including meta-analyses to combine the results of the included studies. An estimated 2,500 systematic reviews are published each year (based on data gathered in 2004), of which the majority relate to questions about the effects of interventions.4 This chapter focuses on the interpretation of these reviews. However, there are systematic reviews of other types of study, some of which are discussed elsewhere in this issue of Seminars in Hematology, and many of the general principles outlined here would apply to such reviews as well. This chapter will guide the reader through some of the steps to consider when interpreting the results of a systematic review. These steps parallel the approach followed by the researchers who did the review when they tried to conclude their work with an answer to the question “what does this review mean?”. Considering how the opinions of the original authors might be used to interpret the results of the systematic review is, therefore, a good place to start.
School of Nursing and Midwifery, Trinity College Dublin, Dublin, Ireland. Address correspondence to Mike Clarke, School of Nursing and Midwifery, Trinity College Dublin, 24 D’Olier St, Dublin 2, Ireland. E-mail: [email protected]
176
0037-1963/08/$-see front matter © 2008 Elsevier Inc. All rights reserved. doi:10.1053/j.seminhematol.2008.04.002
Using the Authors’ Conclusions For many researchers confronted with a systematic review, the first place they might look after deciding that the review is appropriate to their topic of interest is to the conclusions section. In some systematic reviews, such as Cochrane reviews (which make up about a quarter of the new and updated systematic reviews published in 20044), these conclusions are easily identifiable within the review because of its structure. Cochrane reviews are produced by the Cochrane Collaboration, which is the world’s largest organization dedicated to the preparation and maintenance of systematic reviews of the effects of healthcare interventions. The Collaboration was established in the early 1990s, and the growth in systematic reviews through the last decade is in some part due to its work. Cochrane reviews are published in full in the Cochrane Library. From a few dozen in 1995, the number has increased to more than 3,400 in 2008. The reviews encompass all areas of health care, with one of 51 Cochrane Review Groups taking editorial responsibility for each review. Within hematology, there is a Cochrane Hematological Malignancies Group based in Cologne, Germany.5 As of the beginning of 2008, this Group had produced 11 full Cochrane reviews and published protocols for a further 22. All Cochrane reviews include an Authors’ Conclusions section, divided into the Implications for Practice and the Implications for Research. These can provide rich sources of information for anyone wishing to make decisions about health care, but, of course, the degree of certainty they provide depends on the quality of the research that has been identified and included in the review. Although many Cochrane reviews include clear implications concerning the effectiveness or ineffectiveness of interventions,6 the majority
Interpreting the results of systematic reviews find that the effects of interventions remain uncertain and highlight the need for further research.7 Recommendations for further research will be useful to those planning new studies, highlighting the scientific and ethical need for researchers planning such studies to begin with a systematic review.8 Suggestions on how this aspect of the reporting of systematic reviews would be made more helpful were published in 2006 and provide a structure that also can be used if an investigator is faced with decisions as to the direction, need, and design of further trials.9 However, most readers of a systematic review lack the ability to initiate a research project and, therefore, they need to interpret the results of the review in a way that will influence their practice. This effort might mean drawing conclusions beyond those written by the original authors of the review. In doing so, the reader will need to consider both the results of the review and also the context within which the decision is being made.
Considering the Results of the Review An important first step in interpreting the results of the review is to decide whether to believe it is a valid source of evidence for the purposes at hand and the treatment decision being considered. This process also relates to whether the review is of sufficient quality and includes recent data.
Is the Review Good Enough? Several articles have been written over the last couple of decades addressing the quality of reviews. Mulrow highlighted, 20 years ago, that most reviews at that time were far from systematic, were open to substantial risks of bias, and were unclear about their methods.10 Since then, attempts have been made to improve the quality of the conduct and the reporting of systematic reviews. For example, extensive manuals have been published on how to undertake reviews,11,12 and the Quality of Reporting of Meta-analyses (QUOROM) statement has sought to improve the quality of
177 their reporting.13 The latter is currently being updated and its revised version, called Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA), is expected to be published in 2008. Tools have also been developed to allow readers to assess the quality of systematic reviews and these, along with ways to critically appraise other types of research, have been reviewed by Katrak et al.14 Several ad hoc tools have been used for individual projects, but those that are most commonly applied in assessments of the quality of systematic reviews are based on the Overview Quality Assessment Questionnaire (OQAQ) developed by Oxman and Guyatt15,16 and the QUOROM checklist.13 The OQAQ and the QUOROM have been used in a wide range of assessments of systematic reviews and meta-analyses during the last decade, with several such studies being reported recently. One example of such studies compared updated Cochrane reviews published in 2002 with their preceding versions. Both the OQAQ and the QUOROM were employed; the relevant instruments were published on the internet, within their open access article.17 The researchers also compared the outcomes of using the two tools and found that they gave similar results in general. An earlier study by some of the same team compared the OQAQ with another instrument, the 23-item Sacks tool,18 and concluded that there were advantages for using both. The OQAQ provided a numerical score that could be used to benchmark changes over time, while the Sacks scale provided a means of assessing additional, important aspects of each review.19 However, caution should be observed when using a single number to score the quality of a systematic review. It is essential to consider in detail the specific strengths and weaknesses of any review. Specific weaknesses may have multiplicative detrimental effects in some settings but may be less important for answering certain medical questions. Table 1 provides a list of questions adapted from the OQAQ instrument, which a clinician might pose when considering whether or not a systematic review was conducted to
Table 1 Some Questions to Ask When Interpreting the Results of a Systematic Review 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
Do the objective and the inclusion criteria for this review match the question I am seeking an answer for? Were the search methods used to find studies relevant to the main questions of the review stated? Was the search for studies reasonably comprehensive? How likely is it that additional relevant research will have been started, completed, or published since this review was done? Were the criteria used for deciding which studies to include in the review reported? Did the reviewers avoid bias when deciding if studies were eligible? Did the selection of studies (in particular the choice of eligible study designs) minimize the possibility of including studies with a high propensity for bias? Were appropriate criteria used to assess the validity of the included studies? Was the validity of all included studies assessed using these criteria and were the findings reproducible? Were appropriate methods used to combine the findings of the relevant studies? Were the conclusions made by the authors supported by the results of the systematic review? How likely is it that any additional relevant research that had started, been completed, or been published since this review was done would change its conclusions? Is there any reason to believe that the findings of this review are not relevant to the types of patient, intervention, or setting I am interested in?
178 a sufficiently high standard to provide reliable knowledge to help with a treatment decision. However, even if a review was of high quality when it was done, the reader will also need to consider whether it is sufficiently up to date to be reliable.
Is the Review Sufficiently Up to Date? To be truly current, a systematic review would need to have been completed just before it was used and to have had access, at that time, to all the relevant research. This is clearly impractical since it would require reviews to be done in “real time” and be constantly updated whenever new evidence became available. Such a “real time” approach might be straightforward if reviews were simply mathematical analyses of datasets into which new data could be input, leading to new meta-analysis, but reviews are more complex than that. The reviewers need to appraise the quality of the included studies, make decisions about the suitability of the outcome data from these studies, and determine how best to combine these results in their review. They need to have searched for and identified eligible studies. Hence, the process of conducting and, later, updating a systematic review can take considerable time. When determining whether a review is sufficiently recent to help in making a current decision, the reader needs to examine whether the reviewers have omitted any studies that might lead to changes to the review and therefore changes in the interpretation of its conclusons.20 This difficult judgement can be built on information about when the searches were done for the review, the reader’s knowledge of whether any important potentially eligible studies had since become available, and, if necessary, a repeat of the reviewers’ searches of electronic bibliographic databases such as MEDLINE and the Cochrane Central Register of Controlled Trials to check for other publications or reported results. Occasionally, among trials studying the same topic, those that find large, statistically significant benefits are published faster while those with more “negative” results take longer to appear (a “time lag” bias).21 Thus, a comprehensive appreciation of the status of the evidence may be needed to assess also whether there are additional registered trials that are ongoing and their results still pending. Improvements in trial registration (see, for example, http://www.who.int/ictrp/en/) have made this anticipation of the results of future trials increasingly feasible. However, many clinical trials still remain unregistered.
Putting the Results of the Review Into Context as Applies to a Treatment Decision When deciding whether the results of a systematic review of the effects of healthcare interventions are applicable to a treatment decision, the clinician must decide whether these results provide valid information about potential benefits and harms in circumstances relevant to the current setting. If the clinician has decided that the review set out to cover the applicable area of interest and is of adequate quality, he/she needs to
M. Clarke decide whether there are any strong reasons why the participants in the included studies and the interventions assessed in these studies are not sufficiently similar to the people and interventions in the present circumstances to allow the results to apply to them.22 For example, if the reader is interested in treating acute lymphoblastic leukemia in children under 6 years of age, the results from a systematic review of chemotherapy for adults are probably not suitable, but the overall result from a review in which all the participants were aged between 5 and 16 might be. As another example, when interpreting the benefits and harms from the trials in a review of bone marrow transplantation, the reader will need to consider what additional interventions and care were used for the patients. Are these similar to those available in the current setting? Was the additional health care better in the trials than what can be offered in the current situation? Or is it better in the present setting? What effect might this have on the balance of the benefits and harms reported in the review? Some reviews might contain enough information to raise the issue of whether a decision should focus on the overall results of the review or on the results in a particular subgroup, which might be closer to the circumstances of the current case. Readers whose inclination is to focus on the findings from a subgroup need to be cautious. It is important to remember that one of the strengths of a systematic review is that it brings together more data than are available for any specific trial or subgroup. This strategy can lead to a more precise estimate of the effects of the interventions under investigation. If the included studies are sufficiently similar for this average to be sensible, it might be a more accurate estimate of the effect in any subgroup than the specific result from that subgroup because the effects of chance might deflect the estimate in the subgroup more than the overall effect.23-25 In interpreting the results from subgroup analyses in the systematic review, it should be considered whether the rationale for these analyses is to identify different effects among the subgroups or to identify similarities. For example, do the analyses provide reassurance that a treatment is beneficial regardless of specific patient characteristics (such as age) or will cross from beneficial to harmful as a characteristic changes (such as white blood cell count)? In a large individual patient data overview of nearly 7,000 patients receiving different chemotherapy regimens for myeloma, the large number of subgroup analyses that were possible because of the central collection of patient level data on many baseline characteristics revealed consistency across the subgroups.26 Typically, the analyses in systematic reviews are performed using meta-analyses based on relative statistics, such as odds ratios, hazard rates, and relative risks. These statistics are assumed to be more stable for combining the results of the included studies than are absolute statistics such as risk difference or number needed to treat.27 However, when interpreting the results of a systematic review as applied to current circumstances, it may be desirable to convert these relative statistics to absolute values, especially if the balance of benefits and harms for a treatment might be different in the
Interpreting the results of systematic reviews current setting as compared to the settings of the studies in the review.
Converting a Relative Result From a Review Into an Absolute Answer for the Present Circumstance If a systematic review has compared the effects of two interventions and produced a relative statistic as the result of this comparison, the usual assumption is that this relative result will be stable for different populations that are similar to the individuals in the studies in the review. However, the baseline risk of persons in those studies might be very different from that in the current setting. Despite this, the clinician can still move easily from the review’s result to an estimate of what might happen for the current patient or population. As an example, a recent review considered the addition of rituximab to chemotherapy for patients with B-cell nonHodgkin’s lymphoma. It combined the results of seven randomized trials, including nearly 2,000 adult patients, all of whom contributed data to the overall survival analysis. This produced a pooled hazard ratio for mortality of 0.65 (95% confidence interval, 0.54 to 0.78; P ⬍.00001).28 This value can then be applied to a variety of prognostic scenarios to help estimate the survival benefit for individual patients, allowing interpretation of whether the effect is worthwhile in comparison to concerns about harms or other negative aspects of the use of rituximab. If a patient has a 10% chance of dying within 2 years if treated with chemotherapy alone, this would be reduced to 6.5% by the addition of rituximab, equivalent to a number needed to treat (NNT) of 29, for one more survivor at 2 years. If the patient’s chance of dying had been 5%, rituximab might change this to 3.25% (an NNT of 57). But, if the patient’s prognosis were worse, with a 20% risk of dying in the next 2 years, the absolute benefit would be a reduction to 13% and an NNT of 14.
Other Factors to Consider When Interpreting the Results of Systematic Reviews The systematic review should provide the reader with reliable estimates of the relative effects of interventions that are being considered for patients about whom the reader needs to make decisions. However, the interpretation of these results into practice will require the consideration of additional information that was probably not contained in the review. This information includes the preferences of the patient for the different interventions and how he or she values the outcomes that might be made more, or less, likely by the treatment, as well as the clinician’s ability to deliver the preferred intervention in a satisfactory manner.
179
Conclusions Systematic reviews are an increasingly common source of knowledge for healthcare decisions, with at least 2,500 appearing every year. Several such reviews are now available for questions relating to treatment in hematology. The successful interpretation of the results of these reviews into the basis for decisions requires the consideration of the quality of the conduct of the review and whether it is sufficiently current. It also requires a judgement on whether the participants and the interventions assessed in the studies in the review are too different from the reader’s specific circumstances for the results to be applicable. If the results are applicable, the benefits and harms can be considered, along with other information including patient preferences, in determination of a final decision about what the review means and how to best use its conclusions.
References 1. Childhood ALL Collaborative Group: Duration and intensity of maintenance chemotherapy in acute lymphoblastic leukaemia: overview of 42 trials involving 12,000 randomised children. Lancet 347:17831788, 1996 2. Clarke M, Gaynon P, Hann I, Harrison G, Masera G, Peto R, et al: CNS-directed therapy for childhood acute lymphoblastic leukaemia: Childhood ALL Collaborative Group overview of 43 randomized trials. J Clin Oncol 21:1798-1809, 2003 3. Green H, Paul M, Vidal L, Leibovici L: Prophylaxis for pneumocystis pneumonia (PCP) in non-HIV immunocompromised patients. Cochrane Database of Systematic Reviews Issue 3:CD005590, 2007 4. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG: Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4:e78, 2007 5. Weingart O, Kober T, Engert A, Diehl V, Djulbegovic B, Glasmacher A, et al: Cochrane Haematological Malignancies Group. About The Cochrane Collaboration (Cochrane Review Groups (CRGs)) Issue 2:Art. No.: HAEMATOL, 2006 6. El Dib RP, Atallah AN, Andriolo RB: Mapping the Cochrane evidence for decision making in health care. J Eval Clin Pract 13:689-692, 2007 7. Clarke L, Clarke M, Clarke T: Implications for research from Cochrane reviews. J Health Services Res Policy 12:101-103, 2007 8. Clarke M: Doing new research? Don’t forget the old: Nobody should do a trial without reviewing what is known. PLoS Med 1:100-102, 2004 9. Brown P, Brunnhuber K, Chalkidou K, Chalmers I, Clarke M, Fenton M, et al: How to formulate research recommendations. BMJ 333:804806, 2006 10. Mulrow C: The medical review article: state of the science. Ann Intern Med 106:485-488, 1987 11. Khan KS, ter Riet G, Glanville J, Sowden AJ, Kleijnen J (eds): Undertaking Systematic Reviews of Research on Effectiveness (ed 2). York, UK, NHS Centre for Reviews and Dissemination, University of York, 2001 12. Higgins JPT, Green S (eds): Cochrane Handbook for Systematic Reviews of Interventions 4.2.6 [updated September 2006]. http://www. cochrane.org/resources/handbook/hbook.htm (accessed December 6, 2007) 13. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Lancet 354:1896-1900, 1999 14. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA: A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 4:22, 2004 15. Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol 44:1271-1278, 1991 16. Oxman AD: Checklists for review articles. BMJ 309:648-651, 1994
180 17. Shea BJ, Boers M, Grimshaw JM, Hamel CD, Bouter LM: Does updating improve the methodological and reporting quality of systematic reviews? BMC Med Res Methodol 6:27, 2006 18. Sacks H, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC: Metaanalyses of randomized controlled trials. N Engl J Med 316:450-455, 1987 19. Shea B, Moher D, Graham I, Pham B, Tugwell P: A comparison of the quality of Cochrane reviews and systematic reviews published in paperbased journals. Evaluation Health Professions 25:116-129, 2002 20. Shojania KG, Sampson M, Ansari MT, Doucette S, Moher D: How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med 147:224-233, 2007 21. Hopewell S, Clarke M, Stewart L, Tierney J: Time to publication for results of clinical trials. Cochrane Database of Systematic Reviews 2:MR000011, 2007 22. Guyatt GH, Sackett DL, Cook DJ, Evidence-Based Medicine Working Group: Users’ guides to the medical literature, II: How to use an article about therapy or prevention, B: What were the results and will they help me in caring for my patients. JAMA 271:59-63, 1994
M. Clarke 23. Yusuf S, Wittes J, Probstfield J, Tyroler HA: Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials. JAMA 266:93-98, 1991 24. Oxman AD, Guyatt GH: A consumer’s guide to subgroup analyses. Ann Intern Med 116:78-84, 1992 25. Counsell CE, Clarke MJ, Slattery J, Sandercock PAG: The miracle of DICE therapy for acute stroke: Fact or fictional product of subgroup analysis? BMJ 309:1677-1681, 1994 26. Myeloma Trialists Collaborative Group: Combination chemotherapy versus melphalan plus prednisone as treatment for multiple myeloma: An overview of 6,633 patients from 27 randomized trials. J Clin Oncol 16:3832-3842, 1998 27. Deeks JJ, Altman DG: Effect measures for meta-analysis of trials with binary outcomes, in Egger M, Davey Smith G, Altman DG (eds): Systematic Reviews in Health Care (ed 2). London, UK, BMJ Publishing Group, 2001, pp 313-335 28. Schulz H, Bohlius J, Skoetz N, Trelle S, Kober T, Reiser M, et al: Chemotherapy plus Rituximab versus chemotherapy alone for B-cell non-Hodgkin’s lymphoma. Cochrane Database of Systematic Reviews 4:CD003805, 2007