- Email: [email protected]
Helping Consumers of Healthcare Knowledge: Improving the Quality of Reports of Randomized Controlled Trials
COMMENTARY
HELPING CONSUMERS OF HEALTHCARE KNOWLEDGE: IMPROVING THE QUALITY OF REPORTS OF RANDOMIZED CONTROLLED TRIALS Joel Gagnier, ND, MSc, PhD (candidate),1# and David Moher, PhD2-3 Within the evidence-based medicine paradigm, the randomized controlled trial (RCT) holds a dominant position: ruler of the pack. CenterWatch estimates that there are approximately 40,000 clinical trials ongoing presently; there are also approximately 23,000 healthcare journals, with slightly more than 10% of these indexed on MEDLINE, acting as repositories for the results of RCTs. Given this colossus, one might expect that the scientific community is pretty good at designing, conducting, and reporting such studies. Sadly, this is not what the evidence suggests, although the only way we can estimate how a study was designed and conducted is through examining its report. There is a large volume of evidence indicating that the quality of reporting of RCTs is less than optimal, regardless of whether the intervention is conventional or herbal. For example, it has been found that RCTs published in major internal medicine journals report less than 70% of the information suggested in the Consolidated Standards of Reporting Trials (CONSORT) checklist.1 The case is no better for RCTs of herbal medicine interventions, which have been found to report less than 50% of the CONSORT information.2 Poor reporting is not just an esoteric point of interest to a few high priests of methodology. There are profound consequences for inadequately reporting RCTs. For example, it is clear that it is difficult to determine whether a specific aspect of methodology was completed or not, if it was not explicitly stated in the published report.3-6 That is, if allocation concealment is not reported in a particular RCT, it is not valid to assume it was or was not done,5 although more recent data suggest underutilization of adequate allocation concealment during the conduct of a trial and its subsequent report.6 Furthermore, it is well-known that the lack of specific methodology “safe guards” in clinical trials biases estimates of treatment effect.7 Therefore, the information must be accurate and complete in published reports for one to judge the influence of methodological flaws on the results in RCTs. The CONSORT statement, an evidence-based guidance, consisting of a 22-item checklist and a four-point flow diagram, for reporting two-group parallel RCTs, was developed to counter the aforementioned problems. As an “intervention,” the CONSORT checklist appears to be having its intended effect: results from a recently completed systematic
1 Faculty of Medicine, University of Toronto, Toronto, Canada 2 Chalmers Research Group, Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada 3 Department of Pediatrics, Department of Epidemiology and Community Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada # Corresponding author. Address: 5955 Ontario St., Unit 307, Windsor, Ontario, Canada e-mail: [email protected]
© 2006 by Elsevier Inc. Printed in the United States. All Rights Reserved ISSN 1550-8307/06/$32.00
review of eight quasiexperimental studies indicate that journal adoption of the CONSORT is associated with improved reporting of RCTs.8 Is it time to rest on our laurels? An emphatic No is the simple answer. Most of our attention has been focused on “typical” pharmacological interventions. Although these interventions are relevant, they do not represent the complete spectrum of interventions for which RCTs are reported. Herbal interventions are one important group of products for which there are many published RCTs. A recently completed project that assessed a sample of 206 RCTs of herbal medicine interventions found that such trials fail to report information regarding important components of methodology, known to influence the estimates of treatment effect.2 Less than one third of trials adequately reported whether those administering the intervention were blind (28%), the methods for implementation (22%), and the generation (21%) of the random allocation sequence or whether there were protocol deviations (18%) or blinding of outcome assessors (14%) or any methods to determine success of blinding (⬍2%). Also, it appears that characteristics of the herbal interventions are also not adequately reported. In a sample of 81 RCTs, Wolsko et al found that only 15% performed and reported a quantitative analysis of the intervention; and less than 50% reported the Latin binomial, the processing, or extraction methods; and only 10% the part of the plant used in the intervention.9 The CONSORT for herbal interventions was developed to help improve the quality of these studies. The fruits of our labors are reported in this issue of the Journal. There are a wide variety of commercially available products containing herbal medicines.10-13 In addition, there is great variability in the content of these products. Often, products do not contain the amount (weight, volume, proportion) of the individual constituents listed on their label or any of the constituents at all. Controlled trials of herbal medicines can tease out the efficacy of specific products and also the relative harm of specific products. Therefore, it is necessary that authors of trials of herbal interventions completely describe the product used. Without a complete description of the intervention, the known variability in products, as described above, undermines our ability to form cause-effect arguments for specific herbal interventions. Reported in this issue and elsewhere are a set of recommendations for the reporting of RCTs of herbal interventions.14,15 In particular, we focus on the need for specifics describing the herbal medicine intervention. Our recommendations build on the accepted work reported in the CONSORT statement. It is our hope that these new guidelines will improve the reporting of RCTs of herbal medicine and be of practical use to authors, editors, and reviewers.
EXPLORE March 2006, Vol. 2, No. 2 141 doi:10.1016/j.explore.2005.12.010
Beyond developing any set of evidence-based reporting guidances is the need to ensure that they are actively disseminated. In the context of CONSORT, editors and their respective journals have important leadership roles: endorsement and enforcement. There is little benefit in developing any reporting standard without the active participation of journal editors; they are one of the natural benefactors of the reporting intervention. Recognizing this point, the CONSORT group has worked hard to include editors from the inception of the CONSORT initiative, including the original statement and any of its extensions, including that of herbal interventions. As a result, an increasingly large number of journals, including ones more specifically interested in publishing RCTs reports of complementary and alternative medicine, such as Explore, has endorsed CONSORT. Although this is gratifying, it is insufficient. Endorsement without enforcement is ineffective; it is like randomizing a participant to an intervention but not ensuring they have taken it. Such behavior is likely to minimize for the journal any potential positive effect of using the CONSORT checklist. Although we continue to encourage editors to endorse the CONSORT, we would like them to couple any endorsement with enforcement.
REFERENCES 1. Moher D, Jones A, Lepage L for the CONSORT Group. Use of the CONSORT statement and quality of reports of randomized trials: a comparative before and after evaluation. JAMA. 2001;285:19921995. 2. Gagnier JJ, DeMelo J, Boon H, Rochon P, Bombardier C. Quality of reporting of randomized controlled trials of herbal medicine interventions. Am J Med. (in press). 3. Huwiler-Muntener K, Juni P, Junker C, et al. Quality of reporting of randomized trials as a measure of methodologic quality. JAMA. 2002;287:2801-2804. 4. Soares HP, Daniels S, Kumar A, et al. Bad reporting does not mean bad methods for randomized trials: observational study of randomized controlled trials performed by the radiation therapy oncology group. BMJ. 2004;328:22-25.
142 EXPLORE March 2006, Vol. 2, No. 2
5. Pildel J, Chan A, Hrobjartsson A, Forfang E, Altman DG, Gotzsche PC. Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005;330(7499):1049. 6. Devereaux PJ, Choi P, El-Dika S, et al. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004;57:1232-1236. 7. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7. 8. Plint A, Moher D, Altman DG, Schulz KF, Morrison A. Does the CONSORT checklist improve the quality of reports of randomized controlled trials? A systematic review. Presented at the 5th International Congress on Peer Review and Biomedical Communication (Chicago, September 2005). 9. Wolsko PM, Solondz DK, Phillips RS, Schachter SC, Eisenberg DM. Lack of herbal supplement characterization in published randomized controlled trials. Am J Med. 2005;118:1087-1093. 10. Harkey MR, Henderson GL, Gershwin ME, Stern JS, Hackman RM. Variability in commercial Ginseng products: an analysis of 25 preparations. Am J Clin Nutr. 2002;75:600-601. 11. Gurley BJ, Gardner SF, Hubbard MA. Content versus label claims in ephedra-containing dietary supplements. Am J Health Syst Pharm. 2000;57:963-969. 12. Liberti LE, Der Marderosian A. Evaluation of commercial ginseng products. J Pharm Sci. 1978;67:1487-1489. 13. Groenewegen WA, Heptinstall S. Amounts of feverfew in commercial preparations of the herb. Lancet. 1986;1:44-45. 14. Gagnier JJ, Boon H, Rochon P, Barnes J, Moher D, Bombardier CB for the CONSORT Group. Reporting randomized controlled trials of herbal interventions: an elaborated CONSORT statement. Ann Int Med. 2006; March (In Press). 15. Gagnier JJ, Boon H, Rochon P, Barnes J, Moher D, Bombardier CB for the CONSORT Group. Improving the quality of reporting of randomized controlled trials evaluating herbal interventions: implementing the CONSORT statement. Explore: J Sci Healing. 2006;143149.
Quality of Reports of Randomized Controlled Trials
HELPING CONSUMERS OF HEALTHCARE KNOWLEDGE: IMPROVING THE QUALITY OF REPORTS OF RANDOMIZED CONTROLLED TRIALS Joel Gagnier, ND, MSc, PhD (candidate),1# and David Moher, PhD2-3 Within the evidence-based medicine paradigm, the randomized controlled trial (RCT) holds a dominant position: ruler of the pack. CenterWatch estimates that there are approximately 40,000 clinical trials ongoing presently; there are also approximately 23,000 healthcare journals, with slightly more than 10% of these indexed on MEDLINE, acting as repositories for the results of RCTs. Given this colossus, one might expect that the scientific community is pretty good at designing, conducting, and reporting such studies. Sadly, this is not what the evidence suggests, although the only way we can estimate how a study was designed and conducted is through examining its report. There is a large volume of evidence indicating that the quality of reporting of RCTs is less than optimal, regardless of whether the intervention is conventional or herbal. For example, it has been found that RCTs published in major internal medicine journals report less than 70% of the information suggested in the Consolidated Standards of Reporting Trials (CONSORT) checklist.1 The case is no better for RCTs of herbal medicine interventions, which have been found to report less than 50% of the CONSORT information.2 Poor reporting is not just an esoteric point of interest to a few high priests of methodology. There are profound consequences for inadequately reporting RCTs. For example, it is clear that it is difficult to determine whether a specific aspect of methodology was completed or not, if it was not explicitly stated in the published report.3-6 That is, if allocation concealment is not reported in a particular RCT, it is not valid to assume it was or was not done,5 although more recent data suggest underutilization of adequate allocation concealment during the conduct of a trial and its subsequent report.6 Furthermore, it is well-known that the lack of specific methodology “safe guards” in clinical trials biases estimates of treatment effect.7 Therefore, the information must be accurate and complete in published reports for one to judge the influence of methodological flaws on the results in RCTs. The CONSORT statement, an evidence-based guidance, consisting of a 22-item checklist and a four-point flow diagram, for reporting two-group parallel RCTs, was developed to counter the aforementioned problems. As an “intervention,” the CONSORT checklist appears to be having its intended effect: results from a recently completed systematic
1 Faculty of Medicine, University of Toronto, Toronto, Canada 2 Chalmers Research Group, Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Canada 3 Department of Pediatrics, Department of Epidemiology and Community Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Canada # Corresponding author. Address: 5955 Ontario St., Unit 307, Windsor, Ontario, Canada e-mail: [email protected]
© 2006 by Elsevier Inc. Printed in the United States. All Rights Reserved ISSN 1550-8307/06/$32.00
review of eight quasiexperimental studies indicate that journal adoption of the CONSORT is associated with improved reporting of RCTs.8 Is it time to rest on our laurels? An emphatic No is the simple answer. Most of our attention has been focused on “typical” pharmacological interventions. Although these interventions are relevant, they do not represent the complete spectrum of interventions for which RCTs are reported. Herbal interventions are one important group of products for which there are many published RCTs. A recently completed project that assessed a sample of 206 RCTs of herbal medicine interventions found that such trials fail to report information regarding important components of methodology, known to influence the estimates of treatment effect.2 Less than one third of trials adequately reported whether those administering the intervention were blind (28%), the methods for implementation (22%), and the generation (21%) of the random allocation sequence or whether there were protocol deviations (18%) or blinding of outcome assessors (14%) or any methods to determine success of blinding (⬍2%). Also, it appears that characteristics of the herbal interventions are also not adequately reported. In a sample of 81 RCTs, Wolsko et al found that only 15% performed and reported a quantitative analysis of the intervention; and less than 50% reported the Latin binomial, the processing, or extraction methods; and only 10% the part of the plant used in the intervention.9 The CONSORT for herbal interventions was developed to help improve the quality of these studies. The fruits of our labors are reported in this issue of the Journal. There are a wide variety of commercially available products containing herbal medicines.10-13 In addition, there is great variability in the content of these products. Often, products do not contain the amount (weight, volume, proportion) of the individual constituents listed on their label or any of the constituents at all. Controlled trials of herbal medicines can tease out the efficacy of specific products and also the relative harm of specific products. Therefore, it is necessary that authors of trials of herbal interventions completely describe the product used. Without a complete description of the intervention, the known variability in products, as described above, undermines our ability to form cause-effect arguments for specific herbal interventions. Reported in this issue and elsewhere are a set of recommendations for the reporting of RCTs of herbal interventions.14,15 In particular, we focus on the need for specifics describing the herbal medicine intervention. Our recommendations build on the accepted work reported in the CONSORT statement. It is our hope that these new guidelines will improve the reporting of RCTs of herbal medicine and be of practical use to authors, editors, and reviewers.
EXPLORE March 2006, Vol. 2, No. 2 141 doi:10.1016/j.explore.2005.12.010
Beyond developing any set of evidence-based reporting guidances is the need to ensure that they are actively disseminated. In the context of CONSORT, editors and their respective journals have important leadership roles: endorsement and enforcement. There is little benefit in developing any reporting standard without the active participation of journal editors; they are one of the natural benefactors of the reporting intervention. Recognizing this point, the CONSORT group has worked hard to include editors from the inception of the CONSORT initiative, including the original statement and any of its extensions, including that of herbal interventions. As a result, an increasingly large number of journals, including ones more specifically interested in publishing RCTs reports of complementary and alternative medicine, such as Explore, has endorsed CONSORT. Although this is gratifying, it is insufficient. Endorsement without enforcement is ineffective; it is like randomizing a participant to an intervention but not ensuring they have taken it. Such behavior is likely to minimize for the journal any potential positive effect of using the CONSORT checklist. Although we continue to encourage editors to endorse the CONSORT, we would like them to couple any endorsement with enforcement.
REFERENCES 1. Moher D, Jones A, Lepage L for the CONSORT Group. Use of the CONSORT statement and quality of reports of randomized trials: a comparative before and after evaluation. JAMA. 2001;285:19921995. 2. Gagnier JJ, DeMelo J, Boon H, Rochon P, Bombardier C. Quality of reporting of randomized controlled trials of herbal medicine interventions. Am J Med. (in press). 3. Huwiler-Muntener K, Juni P, Junker C, et al. Quality of reporting of randomized trials as a measure of methodologic quality. JAMA. 2002;287:2801-2804. 4. Soares HP, Daniels S, Kumar A, et al. Bad reporting does not mean bad methods for randomized trials: observational study of randomized controlled trials performed by the radiation therapy oncology group. BMJ. 2004;328:22-25.
142 EXPLORE March 2006, Vol. 2, No. 2
5. Pildel J, Chan A, Hrobjartsson A, Forfang E, Altman DG, Gotzsche PC. Comparison of descriptions of allocation concealment in trial protocols and the published reports: cohort study. BMJ. 2005;330(7499):1049. 6. Devereaux PJ, Choi P, El-Dika S, et al. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004;57:1232-1236. 7. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J. How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technol Assess. 2003;7. 8. Plint A, Moher D, Altman DG, Schulz KF, Morrison A. Does the CONSORT checklist improve the quality of reports of randomized controlled trials? A systematic review. Presented at the 5th International Congress on Peer Review and Biomedical Communication (Chicago, September 2005). 9. Wolsko PM, Solondz DK, Phillips RS, Schachter SC, Eisenberg DM. Lack of herbal supplement characterization in published randomized controlled trials. Am J Med. 2005;118:1087-1093. 10. Harkey MR, Henderson GL, Gershwin ME, Stern JS, Hackman RM. Variability in commercial Ginseng products: an analysis of 25 preparations. Am J Clin Nutr. 2002;75:600-601. 11. Gurley BJ, Gardner SF, Hubbard MA. Content versus label claims in ephedra-containing dietary supplements. Am J Health Syst Pharm. 2000;57:963-969. 12. Liberti LE, Der Marderosian A. Evaluation of commercial ginseng products. J Pharm Sci. 1978;67:1487-1489. 13. Groenewegen WA, Heptinstall S. Amounts of feverfew in commercial preparations of the herb. Lancet. 1986;1:44-45. 14. Gagnier JJ, Boon H, Rochon P, Barnes J, Moher D, Bombardier CB for the CONSORT Group. Reporting randomized controlled trials of herbal interventions: an elaborated CONSORT statement. Ann Int Med. 2006; March (In Press). 15. Gagnier JJ, Boon H, Rochon P, Barnes J, Moher D, Bombardier CB for the CONSORT Group. Improving the quality of reporting of randomized controlled trials evaluating herbal interventions: implementing the CONSORT statement. Explore: J Sci Healing. 2006;143149.
Quality of Reports of Randomized Controlled Trials