- Email: [email protected]
The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis
ARTICLE IN PRESS
JID: JINJ
[m5G;November 8, 2019;6:59]
Injury xxx (xxxx) xxx
Contents lists available at ScienceDirect
Injury journal homepage: www.elsevier.com/locate/injury
The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis Austin L. Johnson a,∗, Corbin Walters a, Harrison Gray a, Trevor Torgerson a, Jake X. Checketts a, Marshall Boose b, Brent Norris b,c, Matt Vassar a a
Oklahoma State University Center for Health Sciences, Tulsa, OK, United States Oklahoma State University Medical Center, Department of Orthopaedics, Tulsa, OK, United States c Orthopaedic & Trauma Services of Oklahoma, Tulsa, OK, United States b
a r t i c l e
i n f o
Article history: Accepted 4 November 2019 Available online xxx Keywords: Meta-research Systematic review Meta-analysis Randomized controlled trials Clinical trial Research waste Methods Orthopaedics Epidemiology Cross-sectional analysis
a b s t r a c t Objective: Orthopaedic trauma is one of the largest surgical fields in medicine, and as such, requires the latest evidence to ensure the best standard of care. Systematic reviews are an invaluable resource that compiles an exhaustive summary of the most current evidence on a given clinical question. The primary aim of this study is to evaluate the use of systematic reviews as justification in conducting randomized controlled trials published in high impact orthopaedic trauma journals. Methods: We analyzed randomized controlled trials published in the top three high impact orthopaedic trauma journals between 2015 and 2018. We performed data extraction blind, independent, and in duplicative manner to ensure the validity of the findings. For each trial, data was extracted by the number of systematic reviews cited in each clinical trial and whether or not the study used a systematic review as justification for conducting the trial. A subgroup of general orthopaedic clinical trials were included for comparison. Results: Of 144 articles retrieved, 128 were included. Overall, 71.1% (91/128; [95% CI, 63.2–78.9]) of included orthopaedic trauma randomized controlled trials referenced a systematic review and 28.9% (37/128) of studies did not cite a systematic review. Of the 91 trials that referenced a systematic review, 33.0% (30/91; [95% CI, 23.3–42.6]) of RCTs cited a systematic review as trial justification, whether that be “verbatim” or “inferred”. “Verbatim” justifications occurred in 20.0% (6/30; [95% CI, 5.7–34.3]) of included trauma RCTs that cited a systematic review as justification for conducting the trial and “inferred” justifications occurred in 80.0% (24/30; [95% CI, 65.7–94.3]). Conclusion: Systematic reviews are frequently cited in orthopaedic trauma RCTs but are not commonly cited as justification for conducting a clinical trial. Ideally, evidentiary uncertainty regarding a research question should be established by existing literature through a systematic review to reduce research waste. © 2019 Elsevier Ltd. All rights reserved.
Introduction Research waste—such as studies with flawed methodology, studies that lack power, or studies that answer redundant research questions—has been described as ethically, scientifically, and economically untenable. A 2014 viewpoint by Chalmers and Glasziou estimated that as much as 85% of biomedical research is avoidably wasted or of little value; in economic terms, $170 billion of the $200 billion spent annually on research worldwide may potentially be wasted [1]. Reducing research waste requires thoughtful
∗ Corresponding author: Oklahoma State University Center for Health Sciences, 1111 W 17th St., Tulsa, OK 74107, United States. E-mail address: [email protected] (A.L. Johnson).
planning, execution, and prioritization. In part, research that does not translate to clinical practice stems from poorly selected research questions and planning of clinical research beforehand [2,3]. One method for reducing research waste is to rely on systematic reviews—studies that synthesize all available evidence for a clinical question—to guide whether future studies are needed or whether existing evidence is sufficient and conclusive for a given research question. In particular, clinical trial planning may benefit from using systematic reviews as a research prioritization strategy. Systematic reviews have become increasingly important in health care settings owing to the need to stay abreast of an everexpanding body of medical research [4,5]. For context, approximately 40 0,0 0 0 medical citations were indexed in MEDLINE in 20 0 0, whereas in 2017, nearly 90 0,0 0 0 were indexed [6]. Given this
https://doi.org/10.1016/j.injury.2019.11.004 0020-1383/© 2019 Elsevier Ltd. All rights reserved.
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ 2
ARTICLE IN PRESS
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
rapid rate of expansion, systematic reviews allow health care professionals to view the totality of evidence and to stay up to date on current research [7–9]. The American Academy of Orthopaedic Surgeons recognizes Cochrane systematic reviews as Level A evidence when evaluating the quality of underlying evidence for developing clinical practice guidelines [10]. A systematic review of well-designed randomized controlled trials (RCTs) can highlight the level of evidence for the efficacy of various medical treatments by as well as exposing knowledge gaps in current literature. For this reason, it is critical that systematic reviews (or lack thereof) are considered when prioritizing or justifying a future RCT for the appropriation of research funding [11,12]. Additionally, if no systematic review, or few RCTs exist for a specific research question, then conducting a new trial may be warranted. Despite the known benefits of this approach, systematic reviews are infrequently referenced as the basis for trial conduct [13–16]. In 2014, Chalmers and Glasziou noted that many RCTs were not preceded by a relevant systematic review before being conducted. Had these trials been justified by a systematic review, perhaps a significant number of adverse events could have been prevented as well as numerous duplicative RCTs [17]. A 2017 study of surgical trials found that 65% of RCTs referenced a systematic review in the manuscript, yet none indicated that a systematic review was used to inform or influence the design and conduct of the trial [18]. Similarly, a 2018 study examining anesthesia literature found that 66% of RCTs referenced a systematic review in the manuscript, but only 20% mentioned a systematic review as justification for trial initiation [16]. Given the wide variability in the use of systematic reviews to justify RCTs in differing medical specialties, the primary aim of this study was to evaluate the use of systematic reviews to justify RCTs in three high-ranking orthopaedic trauma journals. For comparison, a secondary aim of this study was to analyze the reference of systematic reviews for trial justification in RCTs published in general orthopaedic journals.
Screening and data extraction The search results were exported into the Rayyan [21] screening platform, and two of us (A.J., C.W.) screened the results to ensure that they were RCTs. Data were extracted in duplicate and blinded fashion [22], using a pilot-tested Google form by 2 authors (A.J., C.W.). We extracted the following data from each included RCT: study name, year of publication, journal title, type of intervention (drug, medical device, procedure, surgery method, other), funding source (public, industry, or other), type of trial (parallel groups, crossover, cluster), and number of participants in the trial. For each study, the number of systematic reviews cited in the introduction, methods, and discussion was derived by reviewing each publication’s reference list for the following key words: meta-analysis, systematic review, and Cochrane review. We reviewed each citation to determine if the systematic review was explicitly cited within the RCT as justification for conducting the trial (e.g., “One important aspect highlighted by this systematic review is the lack of long-term data… The present study is a randomized 15-year follow-up study designed to compare the outcome of PT and HS primary ACL reconstruction”) [23], whether the cited systematic review could be inferred as forming the basis for initiating the trial (e.g., “Bloom et al., in a systematic review of the literature, were unable to establish any advantage of ultrasound-guided glucocorticoid injection over landmark-guided injections for shoulder disorders in terms of pain, function, shoulder ROM, or safety… Our purpose was to analyze the efficacy of a single intra-articular corticosteroid injection applied without image control in patients with primary adhesive capsulitis… ”) [24], or whether the systematic review was cited in other ways unrelated to trial justification. Study characteristics were also extracted for each study, such as type of intervention, funding, and significance of the study (p-values were classified as positive, <0.05, or negative, >0.05). All discrepancies were resolved by consensus. A third author (M.V.) was available for adjudication if necessary.
Methods Comparative analysis Search strategy This cross-sectional analysis did not require Institutional Review Board approval as it did not meet the requirements to be classified as human subject research per U.S. regulations [19]. Our methodology is based on a study conducted by Engelking et al. with modifications [16]. We analyzed RCTs published in the 3 highest ranking orthopaedic trauma journals based on metrics reported in Google Scholar, which generates a Hirsch index score for journals using an algorithm based on article citations [20]. We searched PubMed on December 3, 2018 for RCTs published in Journal of Orthopaedic Trauma, Injury, and Archives of Orthopaedic and Trauma Surgery. To ensure an adequate sample size, we searched for trials published in these journals between January 1, 2015, and November 30, 2018. Boolean operators and parenthetical expression (e.g., AND, OR) were used to develop the following search string: ((("Injury"[Journal]) OR "Journal of orthopaedic trauma"[Journal]) OR ("Archives of orthopaedic and trauma surgery"[Journal])) AND "randomized controlled trial"[Publication Type] AND ("2015/01/01”[PDAT]: "2018/11/31”[PDAT]). Inclusion and exclusion criteria All included studies are phase 3 RCTs. We excluded all study types that did not explicitly mention that they were phase 3 RCTs or trials that were not RCTs, including systematic reviews/metaanalyses, observational studies (e.g., case-control studies, cohort studies), case reports/case series, and subgroup analyses of previously published trials. We also excluded letters to the editor.
We performed a secondary data extraction on a subgroup of RCTs published in the 5 highest-ranked journals in general orthopaedics based on the 2018 Google h5-index rankings. The selected journals for the subanalysis were Journal of Bone and Joint Surgery; American Journal of Sports Medicine; Clinical Orthopaedics and Related Research; Spine; and Knee Surgery, Sports Traumatology, Arthroscopy. We used the randomized controlled trial publication type filter to locate clinical trials published between January 1, 2015, and November 30, 2018. This subgroup analysis followed the same inclusion/exclusion criteria as detailed above and two additional authors (H.G., T.T.) followed the same robust method for data screening, extraction, and reconciliation used in the primary analysis. Statistical analysis Descriptive data were presented as frequencies and percentages along with 95% confidence intervals (95% CIs). All proportions were calculated alongside their 95% CIs using the Wilson binomial proportion method. Google Sheets were used for summary statistics and data analysis. Results Characteristics of included clinical trials Of 144 studies retrieved from our search, 128 RCTs published in orthopaedic trauma journals were included. Of the 16 excluded
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
ARTICLE IN PRESS
JID: JINJ
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
3
Fig. 1. Flow diagram of study inclusion and systematic review citations (abbreviations: RCT; randomized controlled trial).
studies, 8 were ongoing RCTs and 8 were not RCTs. Our main analysis included 44 RCTs from the Archives of Orthopaedic Trauma, 49 RCTs from Injury, and 35 RCTs from the Journal of Orthopaedic Trauma (Fig. 1). Of the 390 general orthopaedic studies retrieved from our search, 319 RCTs were included for analysis. A total of 71 studies were excluded because they were not RCTs. For the comparative analysis, we included 65 RCTs from American Journal of Sports Medicine, 35 RCTs from Clinical Orthopaedics and Related Research, 116 RCTs from Knee Surgery, Sports Traumatology, Arthroscopy, 44 RCTs from Spine, and 59 RCTs from Journal of Bone and Joint Surgery. Comparative orthopaedic trial characteristics are outlined in Tables 1 and 2.
Orthopaedic trauma trial justification Overall, 71.1% (91/128; [95% CI, 63.2–78.9]) of included trauma RCTs referenced a systematic review and 28.9% (37/128; [95% CI, 21.1–36.8]) of studies did not cite a systematic review. Of the 91 trials that referenced a systematic review, 33.0% (30/91; [95% CI, 23.3–42.6]) of RCTs cited a systematic review as trial justification, whether that be “verbatim” or “inferred”. “Verbatim” justifications occurred in 20.0% (6/30; [95% CI, 5.7–34.3]) of included trauma RCTs that cited a systematic review as justification for conducting the trial and “inferred” justifications occurred in 80.0% (24/30; [95% CI, 65.7–94.3]).
Table 1 Characteristics of included orthopaedic trauma clinical trials (n = 128). Characteristic Journal Injury Archives of Orthopaedic Trauma Journal of Orthopaedic Trauma Intervention Surgery Other Drug/Medical Device Procedure Funding Source No funding Not reported Government Nonprofit Industry Hospital Sample Size Sample size (median [IQR])
No. (%) 49 (38.3%) 44 (34.4%) 35 (27.3%) 54 34 24 16
(42.2%) (26.6%) (18.8%) (12.5%)
10 (7.8%) 84 (65.6%) 12 (9.4%) 10 (7.8%) 6 (4.7%) 6 (4.7%) 67 [44–107.8]
Of the 37 RCTs that did not cite a systematic review, we analyzed whether they cited using other means to justify conducting the trial. We found that 13.5% (5/37) did not state a purpose for conducting the RCT, 51.4% (19/37) stated there was a lack of ev-
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ 4
ARTICLE IN PRESS
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
Table 2 Subgroup general orthopaedic clinical trial characteristics (n = 319). Characteristic Journal (n = 319) American Journal of Sports Medicine Clinical Orthopaedics and Related Research Knee Surgery, Sports Traumatology, Journal of Arthroscopy Spine Journal of Bone and Joint Research Intervention (n = 319) Drug Procedure Medical Device Surgery Other Funding Source (n = 319) Industry Hospital/University Government Nonprofit Not Mentioned None Sample size Sample size (median [IQR])
Discussion No. (%) 65 (20.4%) 35 (10.9%) 116 (36.4%) 44 (13.8%) 59 (18.5%) 28 (8.8%) 65 (20.5%) 17 (5.4%) 127 (39.8%) 82 (25.7%) 62 (19.4%) 18 (5.6%) 37 (11.6%) 13 (4.1%) 115 (36.1%) 74 (23.2%) 82 [52–123]
idence as justification for conducting the RCT, and 35.1% (13/37) stated they were building on the knowledge of previous studies. Among the 19 RCTs that stated a lack of evidence as justification, we found 11/19 (57.9%) RCTs where one or more systematic review(s) existed that could have been cited as justification for conducting the trial but were not cited in the RCT’s manuscript. The 13 RCTs that stated they were building on the knowledge of previous studies cited studies such as other RCTs (11), case series (6), and literature reviews (3) as a mean to justify conducting the trial. Analysis of systematic review citations We identified 256 systematic review citations from the 128 included orthopaedic trauma clinical trials. Of the 256 systematic review citations, 116 (45.3%) were cited in the introduction, 11 (4.3%) were cited in the methods section, and 129 (50.4%) were cited in the discussion. The number of systematic reviews cited per section ranged from 1 to 8. Total number of systematic reviews per journal were 119 in the Archives of Orthopaedic Trauma, 89 in Injury, and 48 in Journal of Orthopaedic Trauma. The average number of systematic reviews cited per paper was 2.70 for papers published in the Archives of Orthopaedic Trauma, 1.61 in Injury, and 1.77 in Journal of Orthopaedic Trauma. General orthopaedic trial justification Analysis of the 319 general orthopaedic trials showed that 237 (237/319, 74.3%; [95% CI, 69.5–79.1]) referenced a systematic review, while 82 (82/237, 25.7%; [95% CI, 20.9–30.5]) did not explicitly mention a systematic review in the text or as a reference. Of the 237 trials that referenced a systematic review, 115 (115/237, 48.5%; [95% CI, 42.6–54.9]) cited at least 1 systematic review as justification for conducting the trial and 122 (122/237, 51.5%; [95% CI, 45.1–57.8]) made no reference of a systematic review guiding the design of the RCT or serving as justification for conducting the trial. Nineteen of the 115 (19/115,16.5%; [95% CI, 9.7–23.3]) RCTs that referenced a systematic review as justification for conducting the trial cited “verbatim” 1 or more systematic review as the basis for conducting the RCT, while 96 (96/115, 83.5%; [95% CI, 76.7– 90.3]) implied that a systematic review contributed to the rationale for conducting the RCT. A total of 774 systematic reviews were cited in the 319 included studies.
Systematic reviews represent one of the great achievements of evidence-based medicine and as such, should be considered when developing future clinical trials [25]. Approximately 71% of orthopaedic trauma trials cited at least one systematic review, yet only one-third of RCTs referenced a systematic review as the basis for conducting the clinical trial. Our findings for orthopaedic trauma RCTs that cited a systematic review as justification for conducting a trial are only marginally reduced when compared to general orthopaedic RCTs (33.0% and 48.5%, respectively), but are considerably higher than findings in general surgery (0%) and anesthesiology (20%) [16,18]. The overall high rate of systematic review citations in orthopaedic literature indicates that clinicians and researchers are aware of systematic reviews and understand their importance; however, systematic reviews do not routinely appear to be considered prior to initiating a clinical trial. For example, sequential audits of trials published in leading medical journals shows that 24% of reports in 1997, 10% in 2001, 33% in 2005, 46% in 2009, and 39% in 2012 failed to refer to any systematic reviews prior to conducting the trial [13,14,26]. When analyzing trial protocols, Pandis et al. found that only 30% reported the use of systematic reviews during their development [27]. These studies across various fields of medicine, including orthopaedics, ultimately demonstrate that systematic reviews are infrequently cited as justification for conducting a clinical trial [13–16]. Duplicative studies are particularly prevalent in orthopaedics as 3% to 8% of “original” investigations have been found to be redundant. Specifically, 1 in 13 of presumably original studies published in The Journal of Bone and Joint Surgery was found to be a duplicate or fragmented investigation [28]. However, redundancy in clinical research is not unique to orthopaedics. For example, a cumulative meta-analysis evaluating several major fields of clinical medicine found as many as 8 of 13 cases contained potentially unnecessary research, with 3/13 reaching significantly high proportions (69% to 89%) [29]. In cardiology, a cumulative meta-analysis analyzing the efficacy of the pharmaceutical drug aprotinin demonstrated greatly reduced risk of bleeding in patients undergoing cardiac surgery over the course of the first 12 studies. However, due to a lack of effort by trialists to systematically analyze existing literature, 52 additional studies beyond the first 12 addressing the same efficacy endpoint have been conducted [30]. This example calls into question the scientific and ethical justification of performing repetitive or duplicative studies and highlights the importance of performing a thorough literature review prior to beginning a trial. To mitigate the effect of research waste in the field of medicine through duplicative or unnecessary studies, systematic reviews (or the lack thereof) may be used to aid in justifying a new clinical trial [31]. If no systematic review exists, this lack may signal a research gap likely exists. By referencing a systematic review when developing a RCT, the amount of potential research waste in orthopaedics may be reduced [32]. From a clinical perspective, it is not difficult to see how research waste and redundancy negatively affects patient care. As new surgical methods, devices, and bone healing augmentation strategies are developed, many are supported by low quality evidence studies and require further analysis by independent investigators to elucidate their true utility [33]. Without using systematic reviews, or being cognizant all current literature on these topics, it is impossible to know patient benefit or contribute to growing healthcare expenditures. This issue is acknowledged by the orthopaedic community concerning rotator cuff repair, as many authors criticized the AAOS rotator cuff guidelines for lacking proper evidence to make recommendations [34–36]. Had the authors of the guideline instead referred to the lack of research knowledge and a call for a systematic reviews as justification for needed tri-
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ
ARTICLE IN PRESS
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
als and research in this area, the guideline may have been better received, and likely served as a nidus for quality research for rotator cuff management. Similar effects would likely be seen in orthopaedic traumatology. It is our hope that the findings of this study sparks efforts to systematically evaluate orthopaedic traumatology as a whole, address the located deficits, and revert funding to these deficits rather than more well studied areas. A call to action All orthopaedic trauma RCTs should be justifiable. At present, the Consolidated Standards of Reporting Trials (CONSORT) guidelines requires that data from a new trial be interpreted “in the light of the totality of the available evidence”; however further criteria specifics or details are scant [37]. We propose trialists present evidence (such as a systematic review) at the time of trial initiation that supports the need to conduct a new trial and has not already been answered with existing research. This evidence should be presented to regulators such as trial sponsors, Institutional Review Boards, stakeholders, and ethics committees to ensure that each study is scientifically justified. Furthermore, we extend an invitation to peer-reviewers and journal editors to encourage authors to provide evidence of any relevant systematic review(s) conduct their own literature synthesis as justification for their trial. If the knowledge base contains a significant gap, current evidence is insufficient or of low quality, or no previous research has been conducted, then a trial may be warranted [38]. Journal editors and peer reviewers are the gatekeepers for the scientific literature, and as such, they play an influential role in maintaining the quality and applicability of RCTs reaching publication. Without providing an enforceable standardized model, all RCTs may be considered justifiable when little to no documented evidence is provided by trialists [16,39]. By providing justification for each new RCTs, orthopaedic research may avoid repeating the flaws of past research and more forward to achieve ‘less research, better research, and research done for the right reasons’ with the ultimate goal to improve patient care in orthopaedics [40]. Strengths and limitations Although our methodology is strong, it is not without limitations. Regarding strengths, our methodology for data extraction— duplicate and blinded fashion with joint reconciliation—followed the same methodology for data extraction recommended by the Cochrane Handbook for Systematic Reviews of Interventions when conducting a systematic review [41]. Furthermore, systematic review citations were denoted as both being either explicitly mentioned “verbatim” or as being “inferred” by the investigators that the cited systematic review was used as justification for the RCT. This classification was done to limit the subjectivity or any possible bias when extracting data. Regarding our limitations, we recognize that other prioritization methods exist and that not all back work before an RCT is published and citable. It is possible that in the absence of a systematic review there might have been other evidence confirming the rationale for the trial. Additionally, our keyword search within each study may not have captured all systematic reviews cited, as these reviews may not have been named or labeled as such. Lastly, although our pragmatic search on PubMed (which includes Medline) used specific keywords, our search pay not have returned all RCTs published in these selected journals in the given timeframe. Our analysis included only 3 high impact factor orthopaedic trauma medical journals over a 3 year period; thus, the results may not be generalizable to RCTs in all medical journals or in other timeframes. Furthermore, conducting RCTs in orthopaedic surgery is difficulty, and as such our sample size may
5
not be as robust or as generalizable as similar studies in other fields of medicine. Conclusion In conclusion, systematic reviews are frequently cited in orthopaedic trauma RCTs but are not commonly cited as justification for conducting a clinical trial. Our findings merit careful consideration and cautious interpretation of the use of systematic reviews to justify RCTs in orthopaedic traumatology. Until a better solution is in place, emphasizing the importance of using systematic reviews for justifying new trials is necessary and important. Doing so may aid in attaining high-quality, evidence-based data in orthopaedic RCTs and reducing research waste. Role of funding source N/A Data availability statement Protocols, materials, analysis scripts and raw data are available upon reasonable request to the author. Declaration of Competing Interest We declare no conflicts of interest. Acknowledgements None References [1] Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009;374:86–9. [2] Bhandari M, Chan S, Jonsson A. Planning a clinical research study. Indian J Orthop 2007;41:16. doi:10.4103/0019-5413.30520. [3] Sackett DL, Hoey J. Why randomized controlled trials fail but needn’t: a new series is launched. CMAJ 20 0 0;162:1301–2. [4] Bhandari M, Morrow F, Kulkarni AV, Tornetta P III. Meta-Analyses in orthopaedic surgery: a systematic review of their methodologies. J Bone Jt Surg-Am Vol 2001;83:15–24. [5] Phan K, Mobbs RJ. Systematic reviews and meta-analyses in spine surgery, neurosurgery and orthopedics: guidelines for the surgeon scientist. J Spine Surg 2015;1:19–27. [6] Citations Added to MEDLINE® by Fiscal Year 2007. [7] Ried K. Interpreting and understanding meta-analysis graphs–a practical guide. Aust Fam Physician 2006;35:635–8. [8] Green S. Systematic reviews and meta-analysis. Singapore Med J 2005;46:270–3 quiz 274. [9] University of York. Centre for Reviews and Dissemination, Akers J. Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. Cent Rev Dissem 2009. [10] American Academy of Orthopaedic Surgeons. Clinical practice guidelines. Available from: https://www.aaos.org/guidelines/. [Accessed on 2018 Dec 21] n.d. [11] World Medical Association declaration of Helsinki ethical principles for medical research involving human subjects 59th WMA General Assembly, Seoul 2008 Available at www.wma.net/e/policy/b3.htm. Accessed Dec. 20 2018 n.d. [12] Freedman B. Scientific value and validity as ethical requirements for research: a proposed explication. IRB 1987;9:7–10. [13] Clarke L, Clarke M, Clarke T. How useful are Cochrane reviews in identifying research needs? J Health Serv Res Policy 2007;12:101–3. [14] Clarke M, Hopewell S. Many reports of randomised trials still don’t begin or end with a systematic review of the relevant evidence. J Bahrain Med Soc 2013;24:145–8. [15] Clarke M, Hopewell S, Chalmers I. Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report. J R Soc Med 20 07;10 0:187–90. [16] Engelking A, Cavar M, Puljak L. The use of systematic reviews to justify anaesthesiology trials: a meta-epidemiological study. Eur J Pain 2018;22:1844–9. [17] Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Metin Gülmezoglu A, et al. How to increase value and reduce waste when research priorities are set. Lancet 2014;383:156–65. [18] Rosenthal R, Bucher HC, Dwan K. The use of systematic reviews when designing and reporting surgical trials. Ann Surg 2017;265:e35–6.
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ 6
ARTICLE IN PRESS
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
[19] Office for Human Research Protections (OHRP). 45 CFR 46. HHS.gov 2016. https://www.hhs.gov/ohrp/regulations- and- policy/regulations/45- cfr- 46/ index.html (accessed July 10, 2018). [20] Google Scholar Metrics Help n.d. https://scholar.google.com/intl/en/scholar/ metrics.html#overview (accessed August 18, 2017). [21] Rayyan QCRI n.d. https://rayyan.qcri.org/ (accessed April 1, 2019). [22] Guides and handbooks n.d. https://training.cochrane.org/handbooks (accessed February 8, 2019). [23] Webster KE, Feller JA, Hartnett N, Leigh WB, Richmond AK. Comparison of patellar tendon and hamstring tendon anterior cruciate ligament reconstruction: a 15-year follow-up of a randomized controlled trial. Am J Sports Med 2016;44:83–90. [24] Ranalletta M, Rossi LA, Bongiovanni SL, Tanoira I, Elizondo CM, Maignon GD. Corticosteroid injections accelerate pain relief and recovery of function compared with oral NSAIDs in patients with adhesive capsulitis: a randomized controlled trial. Am J Sports Med 2016;44:474–81. [25] Masic I, Miokovic M, Muhamedagic B. Evidence based medicine - new approaches and challenges. Acta Inform Med 2008;16:219–25. [26] Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA 1998;280:280–2. [27] Pandis N, Fleming PS, Koletsi D, Hopewell S. The citation of relevant systematic reviews and randomised trials in published reports of trial protocols. Trials 2016;17:581. [28] Gwilym SE, Swan MC, Giele H. One in 13 “original” articles in the journal of bone and joint surgery are duplicate or fragmented publications. J Bone Joint Surg Br 2004;86:743–5. [29] Storz-Pfennig P. Potentially unnecessary and wasteful clinical trial research detected in cumulative meta-epidemiological and trial sequential analysis. J Clin Epidemiol 2017;82:61–70.
[30] Fergusson D, Glass KC, Hutton B, Shapiro S. Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clin Trials 2005;2:218–29 discussion 229–32. [31] Cooper NJ, Jones DR, Sutton AJ. The use of systematic reviews when designing studies. Clin Trials 2005;2:260–4. [32] Young C, Horton R. Putting clinical trials into context. Lancet 2005;366:107–8. [33] Petrisor BA, Keating J, Schemitsch E. Grading the evidence: levels of evidence and grades of recommendation. Injury 2006;37:321–7. [34] Tashjian RZ. AAOS clinical practice guideline: optimizing the management of rotator cuff problems. J Am Acad Orthop Surg 2011;19:380–3. [35] Lubowitz JH, McIntyre LF, Provencher MT, Poehling GG. AAOS rotator cuff clinical practice guideline misses the mark. Arthroscopy 2012;28:589–92. [36] Checketts JX, Scott J, Gordon J, Jones J, Horn J, Farabough M, et al. An evaluation of the rotator cuff repair research pipeline. Orthop J Sports Med 2018;6:2325967118805731. [37] Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Int J Surg 2012;10:28–55. [38] Clarke M, Brice A, Chalmers I. Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS ONE 2014;9:e102670. [39] Chow JTY, Lam K, Naeem A, Akanda ZZ, Si FF, Hodge W. The pathway to RCTs: how many roads are there? Examining the homogeneity of RCT justification. Trials 2017;18:51. [40] Altman DG. The scandal of poor medical research. BMJ 1994;308:283–4. [41] Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons; 2011.
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ
[m5G;November 8, 2019;6:59]
Injury xxx (xxxx) xxx
Contents lists available at ScienceDirect
Injury journal homepage: www.elsevier.com/locate/injury
The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis Austin L. Johnson a,∗, Corbin Walters a, Harrison Gray a, Trevor Torgerson a, Jake X. Checketts a, Marshall Boose b, Brent Norris b,c, Matt Vassar a a
Oklahoma State University Center for Health Sciences, Tulsa, OK, United States Oklahoma State University Medical Center, Department of Orthopaedics, Tulsa, OK, United States c Orthopaedic & Trauma Services of Oklahoma, Tulsa, OK, United States b
a r t i c l e
i n f o
Article history: Accepted 4 November 2019 Available online xxx Keywords: Meta-research Systematic review Meta-analysis Randomized controlled trials Clinical trial Research waste Methods Orthopaedics Epidemiology Cross-sectional analysis
a b s t r a c t Objective: Orthopaedic trauma is one of the largest surgical fields in medicine, and as such, requires the latest evidence to ensure the best standard of care. Systematic reviews are an invaluable resource that compiles an exhaustive summary of the most current evidence on a given clinical question. The primary aim of this study is to evaluate the use of systematic reviews as justification in conducting randomized controlled trials published in high impact orthopaedic trauma journals. Methods: We analyzed randomized controlled trials published in the top three high impact orthopaedic trauma journals between 2015 and 2018. We performed data extraction blind, independent, and in duplicative manner to ensure the validity of the findings. For each trial, data was extracted by the number of systematic reviews cited in each clinical trial and whether or not the study used a systematic review as justification for conducting the trial. A subgroup of general orthopaedic clinical trials were included for comparison. Results: Of 144 articles retrieved, 128 were included. Overall, 71.1% (91/128; [95% CI, 63.2–78.9]) of included orthopaedic trauma randomized controlled trials referenced a systematic review and 28.9% (37/128) of studies did not cite a systematic review. Of the 91 trials that referenced a systematic review, 33.0% (30/91; [95% CI, 23.3–42.6]) of RCTs cited a systematic review as trial justification, whether that be “verbatim” or “inferred”. “Verbatim” justifications occurred in 20.0% (6/30; [95% CI, 5.7–34.3]) of included trauma RCTs that cited a systematic review as justification for conducting the trial and “inferred” justifications occurred in 80.0% (24/30; [95% CI, 65.7–94.3]). Conclusion: Systematic reviews are frequently cited in orthopaedic trauma RCTs but are not commonly cited as justification for conducting a clinical trial. Ideally, evidentiary uncertainty regarding a research question should be established by existing literature through a systematic review to reduce research waste. © 2019 Elsevier Ltd. All rights reserved.
Introduction Research waste—such as studies with flawed methodology, studies that lack power, or studies that answer redundant research questions—has been described as ethically, scientifically, and economically untenable. A 2014 viewpoint by Chalmers and Glasziou estimated that as much as 85% of biomedical research is avoidably wasted or of little value; in economic terms, $170 billion of the $200 billion spent annually on research worldwide may potentially be wasted [1]. Reducing research waste requires thoughtful
∗ Corresponding author: Oklahoma State University Center for Health Sciences, 1111 W 17th St., Tulsa, OK 74107, United States. E-mail address: [email protected] (A.L. Johnson).
planning, execution, and prioritization. In part, research that does not translate to clinical practice stems from poorly selected research questions and planning of clinical research beforehand [2,3]. One method for reducing research waste is to rely on systematic reviews—studies that synthesize all available evidence for a clinical question—to guide whether future studies are needed or whether existing evidence is sufficient and conclusive for a given research question. In particular, clinical trial planning may benefit from using systematic reviews as a research prioritization strategy. Systematic reviews have become increasingly important in health care settings owing to the need to stay abreast of an everexpanding body of medical research [4,5]. For context, approximately 40 0,0 0 0 medical citations were indexed in MEDLINE in 20 0 0, whereas in 2017, nearly 90 0,0 0 0 were indexed [6]. Given this
https://doi.org/10.1016/j.injury.2019.11.004 0020-1383/© 2019 Elsevier Ltd. All rights reserved.
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ 2
ARTICLE IN PRESS
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
rapid rate of expansion, systematic reviews allow health care professionals to view the totality of evidence and to stay up to date on current research [7–9]. The American Academy of Orthopaedic Surgeons recognizes Cochrane systematic reviews as Level A evidence when evaluating the quality of underlying evidence for developing clinical practice guidelines [10]. A systematic review of well-designed randomized controlled trials (RCTs) can highlight the level of evidence for the efficacy of various medical treatments by as well as exposing knowledge gaps in current literature. For this reason, it is critical that systematic reviews (or lack thereof) are considered when prioritizing or justifying a future RCT for the appropriation of research funding [11,12]. Additionally, if no systematic review, or few RCTs exist for a specific research question, then conducting a new trial may be warranted. Despite the known benefits of this approach, systematic reviews are infrequently referenced as the basis for trial conduct [13–16]. In 2014, Chalmers and Glasziou noted that many RCTs were not preceded by a relevant systematic review before being conducted. Had these trials been justified by a systematic review, perhaps a significant number of adverse events could have been prevented as well as numerous duplicative RCTs [17]. A 2017 study of surgical trials found that 65% of RCTs referenced a systematic review in the manuscript, yet none indicated that a systematic review was used to inform or influence the design and conduct of the trial [18]. Similarly, a 2018 study examining anesthesia literature found that 66% of RCTs referenced a systematic review in the manuscript, but only 20% mentioned a systematic review as justification for trial initiation [16]. Given the wide variability in the use of systematic reviews to justify RCTs in differing medical specialties, the primary aim of this study was to evaluate the use of systematic reviews to justify RCTs in three high-ranking orthopaedic trauma journals. For comparison, a secondary aim of this study was to analyze the reference of systematic reviews for trial justification in RCTs published in general orthopaedic journals.
Screening and data extraction The search results were exported into the Rayyan [21] screening platform, and two of us (A.J., C.W.) screened the results to ensure that they were RCTs. Data were extracted in duplicate and blinded fashion [22], using a pilot-tested Google form by 2 authors (A.J., C.W.). We extracted the following data from each included RCT: study name, year of publication, journal title, type of intervention (drug, medical device, procedure, surgery method, other), funding source (public, industry, or other), type of trial (parallel groups, crossover, cluster), and number of participants in the trial. For each study, the number of systematic reviews cited in the introduction, methods, and discussion was derived by reviewing each publication’s reference list for the following key words: meta-analysis, systematic review, and Cochrane review. We reviewed each citation to determine if the systematic review was explicitly cited within the RCT as justification for conducting the trial (e.g., “One important aspect highlighted by this systematic review is the lack of long-term data… The present study is a randomized 15-year follow-up study designed to compare the outcome of PT and HS primary ACL reconstruction”) [23], whether the cited systematic review could be inferred as forming the basis for initiating the trial (e.g., “Bloom et al., in a systematic review of the literature, were unable to establish any advantage of ultrasound-guided glucocorticoid injection over landmark-guided injections for shoulder disorders in terms of pain, function, shoulder ROM, or safety… Our purpose was to analyze the efficacy of a single intra-articular corticosteroid injection applied without image control in patients with primary adhesive capsulitis… ”) [24], or whether the systematic review was cited in other ways unrelated to trial justification. Study characteristics were also extracted for each study, such as type of intervention, funding, and significance of the study (p-values were classified as positive, <0.05, or negative, >0.05). All discrepancies were resolved by consensus. A third author (M.V.) was available for adjudication if necessary.
Methods Comparative analysis Search strategy This cross-sectional analysis did not require Institutional Review Board approval as it did not meet the requirements to be classified as human subject research per U.S. regulations [19]. Our methodology is based on a study conducted by Engelking et al. with modifications [16]. We analyzed RCTs published in the 3 highest ranking orthopaedic trauma journals based on metrics reported in Google Scholar, which generates a Hirsch index score for journals using an algorithm based on article citations [20]. We searched PubMed on December 3, 2018 for RCTs published in Journal of Orthopaedic Trauma, Injury, and Archives of Orthopaedic and Trauma Surgery. To ensure an adequate sample size, we searched for trials published in these journals between January 1, 2015, and November 30, 2018. Boolean operators and parenthetical expression (e.g., AND, OR) were used to develop the following search string: ((("Injury"[Journal]) OR "Journal of orthopaedic trauma"[Journal]) OR ("Archives of orthopaedic and trauma surgery"[Journal])) AND "randomized controlled trial"[Publication Type] AND ("2015/01/01”[PDAT]: "2018/11/31”[PDAT]). Inclusion and exclusion criteria All included studies are phase 3 RCTs. We excluded all study types that did not explicitly mention that they were phase 3 RCTs or trials that were not RCTs, including systematic reviews/metaanalyses, observational studies (e.g., case-control studies, cohort studies), case reports/case series, and subgroup analyses of previously published trials. We also excluded letters to the editor.
We performed a secondary data extraction on a subgroup of RCTs published in the 5 highest-ranked journals in general orthopaedics based on the 2018 Google h5-index rankings. The selected journals for the subanalysis were Journal of Bone and Joint Surgery; American Journal of Sports Medicine; Clinical Orthopaedics and Related Research; Spine; and Knee Surgery, Sports Traumatology, Arthroscopy. We used the randomized controlled trial publication type filter to locate clinical trials published between January 1, 2015, and November 30, 2018. This subgroup analysis followed the same inclusion/exclusion criteria as detailed above and two additional authors (H.G., T.T.) followed the same robust method for data screening, extraction, and reconciliation used in the primary analysis. Statistical analysis Descriptive data were presented as frequencies and percentages along with 95% confidence intervals (95% CIs). All proportions were calculated alongside their 95% CIs using the Wilson binomial proportion method. Google Sheets were used for summary statistics and data analysis. Results Characteristics of included clinical trials Of 144 studies retrieved from our search, 128 RCTs published in orthopaedic trauma journals were included. Of the 16 excluded
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
ARTICLE IN PRESS
JID: JINJ
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
3
Fig. 1. Flow diagram of study inclusion and systematic review citations (abbreviations: RCT; randomized controlled trial).
studies, 8 were ongoing RCTs and 8 were not RCTs. Our main analysis included 44 RCTs from the Archives of Orthopaedic Trauma, 49 RCTs from Injury, and 35 RCTs from the Journal of Orthopaedic Trauma (Fig. 1). Of the 390 general orthopaedic studies retrieved from our search, 319 RCTs were included for analysis. A total of 71 studies were excluded because they were not RCTs. For the comparative analysis, we included 65 RCTs from American Journal of Sports Medicine, 35 RCTs from Clinical Orthopaedics and Related Research, 116 RCTs from Knee Surgery, Sports Traumatology, Arthroscopy, 44 RCTs from Spine, and 59 RCTs from Journal of Bone and Joint Surgery. Comparative orthopaedic trial characteristics are outlined in Tables 1 and 2.
Orthopaedic trauma trial justification Overall, 71.1% (91/128; [95% CI, 63.2–78.9]) of included trauma RCTs referenced a systematic review and 28.9% (37/128; [95% CI, 21.1–36.8]) of studies did not cite a systematic review. Of the 91 trials that referenced a systematic review, 33.0% (30/91; [95% CI, 23.3–42.6]) of RCTs cited a systematic review as trial justification, whether that be “verbatim” or “inferred”. “Verbatim” justifications occurred in 20.0% (6/30; [95% CI, 5.7–34.3]) of included trauma RCTs that cited a systematic review as justification for conducting the trial and “inferred” justifications occurred in 80.0% (24/30; [95% CI, 65.7–94.3]).
Table 1 Characteristics of included orthopaedic trauma clinical trials (n = 128). Characteristic Journal Injury Archives of Orthopaedic Trauma Journal of Orthopaedic Trauma Intervention Surgery Other Drug/Medical Device Procedure Funding Source No funding Not reported Government Nonprofit Industry Hospital Sample Size Sample size (median [IQR])
No. (%) 49 (38.3%) 44 (34.4%) 35 (27.3%) 54 34 24 16
(42.2%) (26.6%) (18.8%) (12.5%)
10 (7.8%) 84 (65.6%) 12 (9.4%) 10 (7.8%) 6 (4.7%) 6 (4.7%) 67 [44–107.8]
Of the 37 RCTs that did not cite a systematic review, we analyzed whether they cited using other means to justify conducting the trial. We found that 13.5% (5/37) did not state a purpose for conducting the RCT, 51.4% (19/37) stated there was a lack of ev-
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ 4
ARTICLE IN PRESS
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
Table 2 Subgroup general orthopaedic clinical trial characteristics (n = 319). Characteristic Journal (n = 319) American Journal of Sports Medicine Clinical Orthopaedics and Related Research Knee Surgery, Sports Traumatology, Journal of Arthroscopy Spine Journal of Bone and Joint Research Intervention (n = 319) Drug Procedure Medical Device Surgery Other Funding Source (n = 319) Industry Hospital/University Government Nonprofit Not Mentioned None Sample size Sample size (median [IQR])
Discussion No. (%) 65 (20.4%) 35 (10.9%) 116 (36.4%) 44 (13.8%) 59 (18.5%) 28 (8.8%) 65 (20.5%) 17 (5.4%) 127 (39.8%) 82 (25.7%) 62 (19.4%) 18 (5.6%) 37 (11.6%) 13 (4.1%) 115 (36.1%) 74 (23.2%) 82 [52–123]
idence as justification for conducting the RCT, and 35.1% (13/37) stated they were building on the knowledge of previous studies. Among the 19 RCTs that stated a lack of evidence as justification, we found 11/19 (57.9%) RCTs where one or more systematic review(s) existed that could have been cited as justification for conducting the trial but were not cited in the RCT’s manuscript. The 13 RCTs that stated they were building on the knowledge of previous studies cited studies such as other RCTs (11), case series (6), and literature reviews (3) as a mean to justify conducting the trial. Analysis of systematic review citations We identified 256 systematic review citations from the 128 included orthopaedic trauma clinical trials. Of the 256 systematic review citations, 116 (45.3%) were cited in the introduction, 11 (4.3%) were cited in the methods section, and 129 (50.4%) were cited in the discussion. The number of systematic reviews cited per section ranged from 1 to 8. Total number of systematic reviews per journal were 119 in the Archives of Orthopaedic Trauma, 89 in Injury, and 48 in Journal of Orthopaedic Trauma. The average number of systematic reviews cited per paper was 2.70 for papers published in the Archives of Orthopaedic Trauma, 1.61 in Injury, and 1.77 in Journal of Orthopaedic Trauma. General orthopaedic trial justification Analysis of the 319 general orthopaedic trials showed that 237 (237/319, 74.3%; [95% CI, 69.5–79.1]) referenced a systematic review, while 82 (82/237, 25.7%; [95% CI, 20.9–30.5]) did not explicitly mention a systematic review in the text or as a reference. Of the 237 trials that referenced a systematic review, 115 (115/237, 48.5%; [95% CI, 42.6–54.9]) cited at least 1 systematic review as justification for conducting the trial and 122 (122/237, 51.5%; [95% CI, 45.1–57.8]) made no reference of a systematic review guiding the design of the RCT or serving as justification for conducting the trial. Nineteen of the 115 (19/115,16.5%; [95% CI, 9.7–23.3]) RCTs that referenced a systematic review as justification for conducting the trial cited “verbatim” 1 or more systematic review as the basis for conducting the RCT, while 96 (96/115, 83.5%; [95% CI, 76.7– 90.3]) implied that a systematic review contributed to the rationale for conducting the RCT. A total of 774 systematic reviews were cited in the 319 included studies.
Systematic reviews represent one of the great achievements of evidence-based medicine and as such, should be considered when developing future clinical trials [25]. Approximately 71% of orthopaedic trauma trials cited at least one systematic review, yet only one-third of RCTs referenced a systematic review as the basis for conducting the clinical trial. Our findings for orthopaedic trauma RCTs that cited a systematic review as justification for conducting a trial are only marginally reduced when compared to general orthopaedic RCTs (33.0% and 48.5%, respectively), but are considerably higher than findings in general surgery (0%) and anesthesiology (20%) [16,18]. The overall high rate of systematic review citations in orthopaedic literature indicates that clinicians and researchers are aware of systematic reviews and understand their importance; however, systematic reviews do not routinely appear to be considered prior to initiating a clinical trial. For example, sequential audits of trials published in leading medical journals shows that 24% of reports in 1997, 10% in 2001, 33% in 2005, 46% in 2009, and 39% in 2012 failed to refer to any systematic reviews prior to conducting the trial [13,14,26]. When analyzing trial protocols, Pandis et al. found that only 30% reported the use of systematic reviews during their development [27]. These studies across various fields of medicine, including orthopaedics, ultimately demonstrate that systematic reviews are infrequently cited as justification for conducting a clinical trial [13–16]. Duplicative studies are particularly prevalent in orthopaedics as 3% to 8% of “original” investigations have been found to be redundant. Specifically, 1 in 13 of presumably original studies published in The Journal of Bone and Joint Surgery was found to be a duplicate or fragmented investigation [28]. However, redundancy in clinical research is not unique to orthopaedics. For example, a cumulative meta-analysis evaluating several major fields of clinical medicine found as many as 8 of 13 cases contained potentially unnecessary research, with 3/13 reaching significantly high proportions (69% to 89%) [29]. In cardiology, a cumulative meta-analysis analyzing the efficacy of the pharmaceutical drug aprotinin demonstrated greatly reduced risk of bleeding in patients undergoing cardiac surgery over the course of the first 12 studies. However, due to a lack of effort by trialists to systematically analyze existing literature, 52 additional studies beyond the first 12 addressing the same efficacy endpoint have been conducted [30]. This example calls into question the scientific and ethical justification of performing repetitive or duplicative studies and highlights the importance of performing a thorough literature review prior to beginning a trial. To mitigate the effect of research waste in the field of medicine through duplicative or unnecessary studies, systematic reviews (or the lack thereof) may be used to aid in justifying a new clinical trial [31]. If no systematic review exists, this lack may signal a research gap likely exists. By referencing a systematic review when developing a RCT, the amount of potential research waste in orthopaedics may be reduced [32]. From a clinical perspective, it is not difficult to see how research waste and redundancy negatively affects patient care. As new surgical methods, devices, and bone healing augmentation strategies are developed, many are supported by low quality evidence studies and require further analysis by independent investigators to elucidate their true utility [33]. Without using systematic reviews, or being cognizant all current literature on these topics, it is impossible to know patient benefit or contribute to growing healthcare expenditures. This issue is acknowledged by the orthopaedic community concerning rotator cuff repair, as many authors criticized the AAOS rotator cuff guidelines for lacking proper evidence to make recommendations [34–36]. Had the authors of the guideline instead referred to the lack of research knowledge and a call for a systematic reviews as justification for needed tri-
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ
ARTICLE IN PRESS
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
als and research in this area, the guideline may have been better received, and likely served as a nidus for quality research for rotator cuff management. Similar effects would likely be seen in orthopaedic traumatology. It is our hope that the findings of this study sparks efforts to systematically evaluate orthopaedic traumatology as a whole, address the located deficits, and revert funding to these deficits rather than more well studied areas. A call to action All orthopaedic trauma RCTs should be justifiable. At present, the Consolidated Standards of Reporting Trials (CONSORT) guidelines requires that data from a new trial be interpreted “in the light of the totality of the available evidence”; however further criteria specifics or details are scant [37]. We propose trialists present evidence (such as a systematic review) at the time of trial initiation that supports the need to conduct a new trial and has not already been answered with existing research. This evidence should be presented to regulators such as trial sponsors, Institutional Review Boards, stakeholders, and ethics committees to ensure that each study is scientifically justified. Furthermore, we extend an invitation to peer-reviewers and journal editors to encourage authors to provide evidence of any relevant systematic review(s) conduct their own literature synthesis as justification for their trial. If the knowledge base contains a significant gap, current evidence is insufficient or of low quality, or no previous research has been conducted, then a trial may be warranted [38]. Journal editors and peer reviewers are the gatekeepers for the scientific literature, and as such, they play an influential role in maintaining the quality and applicability of RCTs reaching publication. Without providing an enforceable standardized model, all RCTs may be considered justifiable when little to no documented evidence is provided by trialists [16,39]. By providing justification for each new RCTs, orthopaedic research may avoid repeating the flaws of past research and more forward to achieve ‘less research, better research, and research done for the right reasons’ with the ultimate goal to improve patient care in orthopaedics [40]. Strengths and limitations Although our methodology is strong, it is not without limitations. Regarding strengths, our methodology for data extraction— duplicate and blinded fashion with joint reconciliation—followed the same methodology for data extraction recommended by the Cochrane Handbook for Systematic Reviews of Interventions when conducting a systematic review [41]. Furthermore, systematic review citations were denoted as both being either explicitly mentioned “verbatim” or as being “inferred” by the investigators that the cited systematic review was used as justification for the RCT. This classification was done to limit the subjectivity or any possible bias when extracting data. Regarding our limitations, we recognize that other prioritization methods exist and that not all back work before an RCT is published and citable. It is possible that in the absence of a systematic review there might have been other evidence confirming the rationale for the trial. Additionally, our keyword search within each study may not have captured all systematic reviews cited, as these reviews may not have been named or labeled as such. Lastly, although our pragmatic search on PubMed (which includes Medline) used specific keywords, our search pay not have returned all RCTs published in these selected journals in the given timeframe. Our analysis included only 3 high impact factor orthopaedic trauma medical journals over a 3 year period; thus, the results may not be generalizable to RCTs in all medical journals or in other timeframes. Furthermore, conducting RCTs in orthopaedic surgery is difficulty, and as such our sample size may
5
not be as robust or as generalizable as similar studies in other fields of medicine. Conclusion In conclusion, systematic reviews are frequently cited in orthopaedic trauma RCTs but are not commonly cited as justification for conducting a clinical trial. Our findings merit careful consideration and cautious interpretation of the use of systematic reviews to justify RCTs in orthopaedic traumatology. Until a better solution is in place, emphasizing the importance of using systematic reviews for justifying new trials is necessary and important. Doing so may aid in attaining high-quality, evidence-based data in orthopaedic RCTs and reducing research waste. Role of funding source N/A Data availability statement Protocols, materials, analysis scripts and raw data are available upon reasonable request to the author. Declaration of Competing Interest We declare no conflicts of interest. Acknowledgements None References [1] Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009;374:86–9. [2] Bhandari M, Chan S, Jonsson A. Planning a clinical research study. Indian J Orthop 2007;41:16. doi:10.4103/0019-5413.30520. [3] Sackett DL, Hoey J. Why randomized controlled trials fail but needn’t: a new series is launched. CMAJ 20 0 0;162:1301–2. [4] Bhandari M, Morrow F, Kulkarni AV, Tornetta P III. Meta-Analyses in orthopaedic surgery: a systematic review of their methodologies. J Bone Jt Surg-Am Vol 2001;83:15–24. [5] Phan K, Mobbs RJ. Systematic reviews and meta-analyses in spine surgery, neurosurgery and orthopedics: guidelines for the surgeon scientist. J Spine Surg 2015;1:19–27. [6] Citations Added to MEDLINE® by Fiscal Year 2007. [7] Ried K. Interpreting and understanding meta-analysis graphs–a practical guide. Aust Fam Physician 2006;35:635–8. [8] Green S. Systematic reviews and meta-analysis. Singapore Med J 2005;46:270–3 quiz 274. [9] University of York. Centre for Reviews and Dissemination, Akers J. Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health Care. Cent Rev Dissem 2009. [10] American Academy of Orthopaedic Surgeons. Clinical practice guidelines. Available from: https://www.aaos.org/guidelines/. [Accessed on 2018 Dec 21] n.d. [11] World Medical Association declaration of Helsinki ethical principles for medical research involving human subjects 59th WMA General Assembly, Seoul 2008 Available at www.wma.net/e/policy/b3.htm. Accessed Dec. 20 2018 n.d. [12] Freedman B. Scientific value and validity as ethical requirements for research: a proposed explication. IRB 1987;9:7–10. [13] Clarke L, Clarke M, Clarke T. How useful are Cochrane reviews in identifying research needs? J Health Serv Res Policy 2007;12:101–3. [14] Clarke M, Hopewell S. Many reports of randomised trials still don’t begin or end with a systematic review of the relevant evidence. J Bahrain Med Soc 2013;24:145–8. [15] Clarke M, Hopewell S, Chalmers I. Reports of clinical trials should begin and end with up-to-date systematic reviews of other relevant evidence: a status report. J R Soc Med 20 07;10 0:187–90. [16] Engelking A, Cavar M, Puljak L. The use of systematic reviews to justify anaesthesiology trials: a meta-epidemiological study. Eur J Pain 2018;22:1844–9. [17] Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Metin Gülmezoglu A, et al. How to increase value and reduce waste when research priorities are set. Lancet 2014;383:156–65. [18] Rosenthal R, Bucher HC, Dwan K. The use of systematic reviews when designing and reporting surgical trials. Ann Surg 2017;265:e35–6.
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004
JID: JINJ 6
ARTICLE IN PRESS
[m5G;November 8, 2019;6:59]
A.L. Johnson, C. Walters and H. Gray et al. / Injury xxx (xxxx) xxx
[19] Office for Human Research Protections (OHRP). 45 CFR 46. HHS.gov 2016. https://www.hhs.gov/ohrp/regulations- and- policy/regulations/45- cfr- 46/ index.html (accessed July 10, 2018). [20] Google Scholar Metrics Help n.d. https://scholar.google.com/intl/en/scholar/ metrics.html#overview (accessed August 18, 2017). [21] Rayyan QCRI n.d. https://rayyan.qcri.org/ (accessed April 1, 2019). [22] Guides and handbooks n.d. https://training.cochrane.org/handbooks (accessed February 8, 2019). [23] Webster KE, Feller JA, Hartnett N, Leigh WB, Richmond AK. Comparison of patellar tendon and hamstring tendon anterior cruciate ligament reconstruction: a 15-year follow-up of a randomized controlled trial. Am J Sports Med 2016;44:83–90. [24] Ranalletta M, Rossi LA, Bongiovanni SL, Tanoira I, Elizondo CM, Maignon GD. Corticosteroid injections accelerate pain relief and recovery of function compared with oral NSAIDs in patients with adhesive capsulitis: a randomized controlled trial. Am J Sports Med 2016;44:474–81. [25] Masic I, Miokovic M, Muhamedagic B. Evidence based medicine - new approaches and challenges. Acta Inform Med 2008;16:219–25. [26] Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA 1998;280:280–2. [27] Pandis N, Fleming PS, Koletsi D, Hopewell S. The citation of relevant systematic reviews and randomised trials in published reports of trial protocols. Trials 2016;17:581. [28] Gwilym SE, Swan MC, Giele H. One in 13 “original” articles in the journal of bone and joint surgery are duplicate or fragmented publications. J Bone Joint Surg Br 2004;86:743–5. [29] Storz-Pfennig P. Potentially unnecessary and wasteful clinical trial research detected in cumulative meta-epidemiological and trial sequential analysis. J Clin Epidemiol 2017;82:61–70.
[30] Fergusson D, Glass KC, Hutton B, Shapiro S. Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clin Trials 2005;2:218–29 discussion 229–32. [31] Cooper NJ, Jones DR, Sutton AJ. The use of systematic reviews when designing studies. Clin Trials 2005;2:260–4. [32] Young C, Horton R. Putting clinical trials into context. Lancet 2005;366:107–8. [33] Petrisor BA, Keating J, Schemitsch E. Grading the evidence: levels of evidence and grades of recommendation. Injury 2006;37:321–7. [34] Tashjian RZ. AAOS clinical practice guideline: optimizing the management of rotator cuff problems. J Am Acad Orthop Surg 2011;19:380–3. [35] Lubowitz JH, McIntyre LF, Provencher MT, Poehling GG. AAOS rotator cuff clinical practice guideline misses the mark. Arthroscopy 2012;28:589–92. [36] Checketts JX, Scott J, Gordon J, Jones J, Horn J, Farabough M, et al. An evaluation of the rotator cuff repair research pipeline. Orthop J Sports Med 2018;6:2325967118805731. [37] Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. Int J Surg 2012;10:28–55. [38] Clarke M, Brice A, Chalmers I. Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS ONE 2014;9:e102670. [39] Chow JTY, Lam K, Naeem A, Akanda ZZ, Si FF, Hodge W. The pathway to RCTs: how many roads are there? Examining the homogeneity of RCT justification. Trials 2017;18:51. [40] Altman DG. The scandal of poor medical research. BMJ 1994;308:283–4. [41] Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions. John Wiley & Sons; 2011.
Please cite this article as: A.L. Johnson, C. Walters and H. Gray et al., The use of systematic reviews to justify orthopaedic trauma randomized controlled trials: A cross-sectional analysis, Injury, https://doi.org/10.1016/j.injury.2019.11.004