- Email: [email protected]
Quality of meta-analyses in major leading orthopedics journals: A systematic review
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
Available online at
ScienceDirect www.sciencedirect.com
Review article
Quality of meta-analyses in major leading orthopedics journals: A systematic review X. Zhi a,1 , Z. Zhang a,1 , J. Cui b , X. Zhai b , X. Chen b,∗ , J. Su b,∗ a b
Graduate Management Unit, Changhai hospital affiliated to the Second Military Medical University, Changhai road, 200433 Shanghai, PR, China Department of Orthopedics, Changhai hospital affiliated to the Second Military Medical University, Changhai road, 200433 Shanghai, PR, China
a r t i c l e
i n f o
Article history: Received 28 February 2017 Accepted 24 August 2017 Keywords: Meta-Analysis Methodological quality Orthopedics PRISMA AMSTAR
a b s t r a c t Background: Meta-Analyses are the basis of professional and healthcare agencies recommendations and have a growing importance. Quality of meta-analyses has been investigated in some medical fields but to our best knowledge this issue remains under investigated in orthopedics. Therefore, we performed a systematic analysis to: 1) after the introduction of PRISMA statement as a comprehensive guideline and the use of the AMSTAR tool as the standard for sufficient review methodology, has the quality of MAs improved because of that? 2) have some general characteristics influenced the quality of MAs (country, funding source, number of authors)? Material and Methods: We systematically searched the meta-analyses in the top four journals with the impact factor (2015) as following: JBJS, Osteoarthritis Cartilage Arthroscopy and Clin Orthop Relat Res from 2005 to 2008 and from 2012 to 2015. Likewise from 2012–2015, we also analyzed the meta-analyses from OTSR. Characteristics were extracted based on the PRISMA statement and the AMSTAR tool. Country, number of authors, funding source were also extracted. Results: A total of 154 meta-analyses were included in the present study. Score with PRISMA statement and the AMSTAR checklist were 20.86 ± 3.04 out of a maximum of 27 and 7.86 ± 1.55 out of a maximum of 11. The best journal was OTSR according to the PRISMA (23.06 ± 1.92) and AMSTAR (9.13 ± 0.87) scores. And the worst journal was Clin Orthop Relat Res according to the PRISMA score (19.4 ± 2.70) and JBJS according to the AMSTAR score (6.78 ± 1.65). Twelve items showed significant difference in the PRISMA statement, and five items in the AMSTAR checklist. Integral score of PRISMA statement and AMSTAR checklist has a significant difference between 2005-2008 and 2012–2015. The MAs reported from U.S. (56, 36.4%) were more than any other region in the world. And the MAs published by Asia/Oceania increased remarkably between these two period times [from (4, 10.8%) to (45, 38.5%)]. Conclusion: This study showed that methodological reporting quality of meta-analyses in the major orthopedics journals has improved after the publication of the PRISMA statement. Level of evidence: Level III. © 2017 Elsevier Masson SAS. All rights reserved.
Abbreviations: MA, Meta-analysis; SRs, System reviews; PRISMA, Preferred reporting items for systematic reviews and meta-analyses; AMSTAR, Assessing the methodological quality of systematic review; JBJS, The Journal of Bone and Joint Surgery; Osteoarthritis Cartilage, Osteoarthritis and Cartilage; Arthroscopy, The journal of arthroscopic and related surgery; CORR, Clinical Orthopaedics and Related Research; OTSR, Orthopaedics, Traumatology & Surgery Research. ∗ Corresponding authors. Department of Orthopedics, Changhai hospital affiliated to the Second Military Medical University, Shanghai, PR China. E-mail addresses: [email protected] (X. Chen), [email protected] (J. Su). 1 These authors contributed equally to this work.
1. Introduction Systematic reviews (SRs) and meta-analyses (MAs) have become important in health care. Clinicians refer to them to learn the latest clinical methods in these fields [1]. MAs and SRs are indispensable tools for summing up evidence precisely and reliably. MAs and SRs help clinicians keep up to date; supply evidence for policy makers to estimate risks, interests and harms of healthcare behaviors and treatments; collect and sum up related researches for
http://dx.doi.org/10.1016/j.otsr.2017.08.009 1877-0568/© 2017 Elsevier Masson SAS. All rights reserved.
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
2
patients; provide a beginning point for clinical practice guideline developers; summarize of previous researches for funders wishing to support new studies; and help editors select the merits of publishing reports of new researches [2]. It was estimated that at least 2500 new MAs (including systematic reviews, SRs) were indexed in Medline yearly [3]. However, lots of MAs were methodologically poorly reported and could introduce bias and impair the reliability of conclusions although rigorous methodology is a feature of MAs [4,5]. To prevent such incidents, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PRISMA) was developed in 2009 (Supplemental data 1) [1]. It was an evidencebased minimum set of items for reporting in Systematic reviews and meta-analyses assessing randomized control trials (RCT) [6]. The PRISMA statement was developed by a group of 29 review authors, methodologists, clinicians, medical editors, and consumers [1]. The objective of the PRISMA statement was to help authors improve the reporting quality of SRs and MAs. PRISMA could also be advantageous for crucial appraisal of published systematic reviews. It has been extensively used as a criterion to assess the reporting quality of MAs since it was published in 2009 [1]. AMSTAR (Assessing the Methodological Quality of SR, AMSTAR) also emerged in 2007 (Supplemental data 2) [7], and subsequently was simplified to 11 items from 37 items. The aim of the AMSTAR is to evaluate the reporting quality of SRs, by establish upon empirical data on previously reported tools, empirical evidence and utilizing expert opinion [8]. After the PRISMA statement and AMSTAR tool were reported, assessment of the reported quality of MAs have been reported in many fields, including surgery [9,10], pulmonary disease [11], telerehabilitation [12], nursing [13] and gastroenterology and hepatology [6]. Meta-Analysis with its present form of statistically integrating information from several studies all with a common underlying theme has been around for over 25 years [14]. There are growing numbers of MAs being published in orthopedics fields, and guiding the clinical work. However, to our best knowledge, the evaluation of orthopedic journals has never been reported before. To solve this issue, we systematically evaluated the methodological reporting quality of MAs before and after the publication of the developed PRISMA statement (from 2005 to 2008 and from 2012 to 2015), in order to observe the quality of MAs whether improve or not after the standard was published. MAs were selected from the 4 orthopedics journals with the highest impact factor (2015) and Orthopaedics & Traumatology: Surgery & Research which is not a leading journal in this field but also introduced PRISMA before 2012. We introduce the following questions: • has the quality of MAs improved after the introduction of PRISMA statement as a comprehensive guideline and the use of the AMSTAR tool as the standard for sufficient review methodology? • have some general characteristics influenced the quality of MAs (country, funding source, number of authors)? 2. Methods of search strategy and criteria 2.1. Search strategy To appraise the methodological quality of MAs on orthopedics fields, we selected the top four journals with the highest impact factor (2015) as following: the Journal of Bone Joint Surgery (JBJS), Osteoarthritis and Cartilage (Osteoarthritis Cartilage), Arthroscopy – the journal of arthroscopic and related surgery (Arthroscopy) and Clinical Orthopaedics and Related Research (CORR) [15]. In
Table 1 Principals characteristics of the included trials.
Number No. authors 1–3 4–6 >6 Region/Country US Europe Asia/Oceania Others Funding source Industry Public Not specific None
All trials (%)
2005–2008 (%)
2012–2015 (%)
154 (100.0)
37 (24.0)
117 (76.0)
27 (17.5) 87 (56.5) 40 (26.0)
6 (16.2) 25 (67.6) 6 (16.2)
21 (17.9) 62 (53.0) 34 (29.1)
56 (36.4) 31 (20.1) 49 (31.8) 18 (11.7)
18 (48.6) 10 (27.0) 4 (10.8) 5 (13.5)
38 (32.4) 21 (17.9) 45 (38.5) 13 (11.2)
19 (12.3) 38 (24.7) 31 (20.1) 66 (42.9)
2 (5.4) 5 (13.5) 12 (32.4) 18 (48.6)
17 (14.5) 33 (28.2) 19 (16.2) 48 (41.1)
addition, we picked up the MAs published from 2012 to 2015 in Orthopaedics & Traumatology: Surgery & Research (OTSR), another journal, which has also introduced PRISMA before 2012 as comparison. The conformed MAs were conducted on the PubMed/MEDLINE database [16] by two co-first authors on October 1st, 2016. The ISSNs (Print) of five journals were 1535–1386, 1063–4584, 0749–8063, 0009–921X and 1877–0568. To evaluate the PRISMA statement published in 2009, we chose the articles from 2005 to 2008 and from 2012 to 2015. We selected the “Meta-analysis” in the “Article types” project of Web of science, and excluded the following types: randomized controlled trial or review or the article published in 2016. 2.2. Data extraction and criteria We primarily extracted geography, publishing, clinical and epidemiological characteristics based on the previous meta-analysis and systematic reviews, for example region/country, number of authors, publishing year, number of patients and funding source. Then we collected the information of MAs with PRISMA statement [2] and AMSTAR checklist [7]. The PRISMA statement is a 27-item list and the AMSTAR checklist is an 11-item list. The work of extraction information was completed by two co-first authors, and any disagreement was resolved by discussion between the two reviewers or other authors. 2.3. Statistical analyses All statistical analyses were performed with SPSS 21.0 (SPSS Statistical software version 21.0. IBM Corporation, USA). Frequencies and proportions were analyzed, and descriptive statistics (mean, standard deviation) of the country, funding source were employed (Table 1). For the parabole between years and journals, we performed chi-squared test and fisher exact test (Tables 2 and 3). For the determination of variables that associated with PRISMA or AMSTAR score, we firstly accomplished the univariate linear regression and then combined with stepwise backward multiple regression analysis. P < 0.05 was regarded statistically significant. Effect size (Odds ratio, OR) and 95% confidence intervals were reported for significant associations (Table 4). 3. Results 3.1. Search results A total of 167 MAs were retrieved from the five journals, in which 13 studies were excluded (Fig. 1) (all detailed results of the 167
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model
ARTICLE IN PRESS
OTSR-1840; No. of Pages 6
X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
3
Table 2 Proportions of adequately reported PRISMA items in different years and different journals. Items of the PRISMA statement
1. Title 2. Structured summary 3. Rationale 4. Objectives 5. Protocol and registration 6. Eligibility criteria 7. Information sources 8. Search 9. Study selection 10. Data collection process 11. Data items 12. Risk of bias in individual studies 13. Summary measures 14. Synthesis of results 15. Risk of bias across studies 16. Additional analyses 17. Study selection 18. Study characteristics 19. Risk of bias within studies 20. Results of individual studies 21. Synthesis of results 22. Risk of bias across studies 23. Additional analysis 24. Summary of evidence 25. Limitations 26. Conclusions 27. Funding PRISMA score (0 to 27) * a
All meta
2006–2008
2012–2015
P-value
(n = 154)
(n = 37)
(n = 117)
134 (87.0) 138 (89.6) 127 (82.5) 153 (99.4) 148 (96.1) 140 (90.9) 143 (92.9) 144 (93.5) 127 (82.5) 106 (68.8) 119 (77.3) 55 (35.7)
31 (83.8) 28 (75.7) 30 (81.1) 36 (97.3) 33 (89.2) 28 (75.7) 28 (75.7) 30 (81.1) 30 (81.1) 20 (54.1) 21 (56.8) 12 (32.4)
103 (88.0) 110 (94.0) 97 (82.9) 117 (100.0) 115 (98.3) 112 (95.7) 115 (98.3) 114 (97.4) 97 (82.9) 86 (73.5) 98 (83.8) 43 (36.8)
0.575 0.003* 0.807 0.24 0.030* 0.001* < 0.001* 0.002* 0.807 0.041* 0.001* 0.697
129 (83.8) 123 (79.9) 42 (27.3) 35 (22.7) 151 (98.1) 141 (91.6) 56 (36.4) 129 (83.8) 144 (93.5) 45 (29.2) 38 (24.7) 154 (100.0) 115 (74.7) 152 (98.7) 88 (57.1) 20.86 ± 3.04
24 (64.9) 28 (75.7) 12 (32.4) 4 (10.8) 35 (94.6) 24 (64.9) 15 (40.5) 25 (67.6) 31 (83.8) 10 (27.0) 5 (13.5) 37 (100.0) 27 (73.0) 35 (94.6) 19 (51.4) 18.27 ± 3.03
105 (89.7) 95 (81.2) 30 (25.6) 31 (26.5) 116 (99.1) 117 (100.0) 41 (35.0) 104 (88.9) 113 (96.6) 35 (29.9) 33 (28.2) 117 (100.0) 88 (75.2) 117 (100.0) 69 (59.0) 21.50 ± 2.69
0.001* 0.485 0.407 0.007* 0.144 < 0.001* 0.561 0.004* 0.013* 0.837 0.082 / 0.829 0.057 0.45 0.001*
CORR
JBJS
OTSR
(n = 41)
Arthroscopy Osteoarthritis Cartilage (n = 34) (n = 31)
P-value
(n = 32)
(n = 16)
30 (73.2) 32 (78.0) 27 (65.9) 40 (97.6) 40 (97.6) 36 (87.8) 38 (92.7) 34 (82.9) 32 (78.0) 32 (78.0) 31 (75.6) 9 (22.0)
29 (85.3) 34 (100.0)a 28 (82.4) 34 (100.0)a 31 (91.2) 32 (94.1) 33 (97.1) 32 (94.1) 28 (82.4) 26 (76.5) 27 (79.4) 8 (23.5)
27 (87.1) 25 (80.6) 24 (77.4) 31 (100.0)a 29 (93.5) 25 (80.6) 29 (93.5) 30 (96.8) 27 (87.1) 26 (83.9)a 24 (77.4) 8 (25.8)
32 (100.0)a 31 (96.9) 32 (100.0)a 32 (100.0)a 32 (100.0)a 31 (96.9) 27 (84.4) 32 (100.0)a 26 (81.3) 9 (28.1) 23 (71.9) 15 (46.9)
16 (100.0)a 16 (100.0)a 16 (100.0)a 16 (100.0)a 16 (100.0)a 16 (100.0)a 16(100.0)a 16(100.0)a 14 (87.5)a 13 (81.3) 14 (87.5)a 15 (93.8)a
0.007* 0.003* 0.001* 0.596 0.295 0.098 0.228 0.022* 0.857 < 0.001* 0.802 < 0.001*
40 (97.6) 33 (80.5) 9 (22.0) 7 (17.1) 41 (100.0)a 38 (92.7) 10 (24.4) 36 (87.8) 36 (87.8) 10 (24.4) 7 (17.1) 41 (100.0) 26 (63.4) 41 (100.0)a 11 (26.8) 19.4 ± 2.70
32 (94.1) 25 (73.5) 7 (20.6) 11 (32.4) 34 (100.0)a 31 (91.2) 8 (23.5) 29 (85.3) 33 (97.1) 8 (23.5) 10 (29.4) 34 (100.0) 29 (85.3) 34 (100.0)a 17 (50.0) 20.6 ± 3.14
30 (96.8) 22 (71.0) 7 (22.6) 11 (35.5)a 31 (100.0)a 30 (96.8) 9 (29.0) 26 (83.9) 30 (96.8) 9 (29.0) 11 (35.5)a 31 (100.0) 12 (38.7) 31 (100.0)a 17 (54.8) 20.2 ± 2.99
11 (34.4) 31 (96.9)a 14 (43.8)a 5 (15.6) 30 (93.8) 26 (81.3) 14 (43.8) 22 (68.8) 29 (90.6) 10 (31.3) 9 (28.1) 32 (100.0) 32 (100.0)a 30 (93.8)a 31 (96.9)a 20.3 ± 3.15
16 (100.0)a 12 (75.0) 5 (31.3) 1 (6.3) 15 (93.8) 16 (100.0)a 15 (93.8)a 16 (100.0)a 16 (100.0)a 8 (50.0)a 1 (6.3) 16 (100.0) 16 (100.0)a 16 (100.0)a 6 (37.5) 23.06 ± 1.92
< 0.001* 0.079 0.184 0.066 0.149 0.134 < 0.001* 0.062 0.288 0.355 0.149 / < 0.001* / < 0.001* < 0.001*
P < 0.05. The highest score.
Table 3 Proportions of adequately reported AMSTAR items in different years and different journals. Items of the AMSTAR statement
1. “a priori” design 2. Duplicate study selection and data extraction 3. Comprehensive literature search 4. Grey literature search 5. List of studies included and excluded 6. Characteristics of studies 7. Scientific quality assessed 8. Scientific quality in conclusions 9. Methods to combine the studies 10. Publication bias assessed 11. Conflict of interest AMSTAR score (0 to 11s) * a
All meta
2006–2008 2012–2015 P-value
CORR
(n = 154)
(n = 37)
(n = 41)
(n = 117)
Arthroscopy Osteoarthritis Cartilage (n = 34) (n = 31)
41 (100.0) 34 (100.0) 41 (100.0)a 33 (97.1)
JBJS
OTSR
P-value
(n = 32)
(n = 16)
31 (100.0) 30 (96.8)
32 (100.0) 29 (90.6)
16 (100.0) / 16 (100.0)a 0.219
154 (100.0) 37 (100.0) 149 (96.8) 33 (89.2)
117 (100) 116 (99.1)
/ 0.012*
112 (72.7) 28 (18.2) 105 (68.2)
22 (59.5) 7 (18.9) 10 (27.0)
90 (76.9) 21 (17.9) 95 (81.2)
0.055 26 (63.4) 1 9 (22.0) < 0.001* 35 (85.4)
21 (61.8) 8 (23.5)a 30 (88.2)
21 (67.7) 6 (19.4) 23 (74.2)
28 (87.5) 5 (15.6) 1 (3.1)
16 (100.0)a 0.008* 0 (0.0) 0.062 16 (100.0)a 0.312
139 (90.3) 124 (80.5) 123 (79.9) 133 (86.4) 49 (31.8) 70 (45.5) 7.86 ± 1.55
26 (70.3) 29 (78.4) 23 (62.2) 26 (70.3) 9 (24.3) 20 (54.1) 6.54 ± 1.34
113 (96.6) 95 (81.2) 100 (85.5) 107 (91.5) 40 (34.2) 50 (42.7) 8.20 ± 1.38
< 0.001* 0.812 0.004* 0.002* 0.314 0.344 0.001*
29 (85.3) 31 (91.2) 34 (100.0)a 29 (85.3) 9 (26.5) 0 (0.0) 7.6 ± 1.52
31 (100.0)a 26 (83.9) 28 (90.3) 28 (90.3) 11 (35.5) 22 (71.0) 8.3 ± 1.42
26 (81.3) 14 (43.8) 8 (25.0) 27 (84.4) 17 (53.1)a 30 (93.8) 6.78 ± 1.65
16 (100.0)a 15 (93.8)a 14 (87.5) 15 (93.8)a 6 (37.5) 16 (100.0)a 9.13 ± 0.87
37 (90.2) 38 (92.7) 39 (95.1) 34 (82.9) 6 (14.6) 2 (4.9) 7.5 ± 1.17
0.062 < 0.001* < 0.001* 0.79 0.011* < 0.001* 0.001*
P < 0.05. The highest score.
studies investigated are reported in Supplemental data 3). Finally, 154 MAs were accepted in the article, including 32 from the JBJS, 34 from the Arthroscopy, 41 from the CORR, 31 from the Osteoarthritis Cartilage and 16 from the OTSR. Thirty-seven (24.0%) MAs were published between 2005 and 2008, and one hundred and seven (76.0%) MAs were published between 2012 and 2015 (all detailed results of the 154 inclusions are reported in Supplemental data 3).
whole, the MAs reported from US (56, 36.4%) were more than any other region in the world. The Asia/Oceania production increased remarkably between these two periods [from (4, 10.8%) to (45, 38.5%)]. In 2005–2008, 18.9% of the MAs were with funding source, and 32.4% were not specific. And it was 42.7% with the funding source and 16.2% that were not specific in 2012–2015. 3.3. Compliance with the PRISMA checklist
3.2. General characteristics Crucial characteristics of the included MAs were shown in the Table 1. 87 (56.5%) of MAs have 4-6 authors per article. As a
The proportion and percentage of MAs that adequately reported each PRISMA items were showed in Fig. 2 and Table 2. Summary of evidence (item 24) was reported 100%. Most domains were
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
4
Table 4 Univariate and multivariate exploratory linear relationship of changes in PRISMA score and AMSTAR score. Items
Univariate exploratory variable Country (vs. America) Europe Asia/Oceania Others Journal (vs. Osteoarthritis Cartilage) Arthroscopy CORR JBJS OTSR Funding (vs. Industry) Public None Not specified Numbers of authors (vs. 1–3) Year (vs. 2005–2008) Multivariate exploratory variable Year (vs. 2005–2008) *
PRISMA score
AMSTAR score
s (OR)
95% CI
P
s (OR)
95% CI
P
0.54 1.676 1.466
(−0.780, 1.86) (0.523, 2.829) (−0.131, 3.064)
0.420 0.005* 0.072
0.667 1.097 0.948
(0.013, 1.321) (0.526, 1.668) (0.157, 1.740)
0.046* < 0.001* 0.019*
0.433 −0.156 0.769 3.214
(−1.001, 1.866) (−1.523, 1.210) (−0.643, 2.181) (1.454, 4.974)
0.552 0.822 0.283 0.000*
−1.370 −0.562 −0.563 0.973
(−2.060, −0.680) (−1.220, 0.096) (−1.243, 0.116) (0.126, 1.821)
< 0.001* 0.094 0.104 0.025*
0.500 0.273 0.032 0.143 2.807
(−1.204, 2.204) (−1.493, 2.040) (−1.547, 1.611) (−0.091, 0.378) (1.761, 3.853)
0.563 0.760 0.968 0.23 0.001*
0.421 −0.321 0.194 0.027 1.528
(−0.430, 1.272) (−1.203, 0.561) (−0.594, 1.74) (−0.092, 0.147) (1.007, 2.048)
< 0.001* 0.330 0.473 0.651 < 0.001*
14.750
(12.107, 17.393)
< 0.001*
5.660
(4.395, 6.925)
0.001*
P < 0.05.
Fig. 1. Flow of information through the different phases of a systematic review. (JBJS = the Journal of Bone and Joint Surgery; Osteoarthritis Cartilage = Osteoarthritis and Cartilage; Arthroscopy = The journal of arthroscopic and related surgery; CORR = Clinical Orthopaedics and Related Research; OTSR = Orthopaedics & Traumatology: Surgery & Research).
However, there were some items poorly reported, including risk of bias in individual studies (item 12, 55, 35.7%), risk of bias across studies (item 15, 42, 27.3%), additional analyses (item 16, 35, 22.7%), Risk of bias within studies (item 19, 56, 36.4%), risk of bias across studies (item 22, 45, 29.2%) and Additional analyses (item 23, 38, 24.7%). There were significant improvements in 2012–2015 than those in 2005–2008. It was reflected in the following items: Structured summary (item 2, P = 0.003), Protocol and registration (item 5, P = 0.03), Eligibility criteria (item 6, P = 0.001), Information sources (item 7, P < 0.001), Search (item 8, P = 0.002), Data collection process (item 10, P = 0.041), Data items (item 11, P = 0.001), Summary measures (item 13, P = 0.001), Additional analyses (item 16, P = 0.007), Study characteristics (item 18, P < 0.001), Results of individual studies (item 20, P = 0.004) and Synthesis of results (item 21, P = 0.013). The PRISMA score was 21.50 ± 2.69 in 2012–2015 (77% of items adequately reported, on average), has significant improvements compared with the PRISMA score in 2005–2008 (18.27 ± 3.03, P < 0.001). In addition, OTSR performed the best in domains of the average score of PRISMA (23.06 ± 1.92) and risk of bias in individual studies (item 12, 15, 93.8%). JBJS performed the best in domains of Funding (item 27, 31, 96.9%).
3.4. Compliance with the AMSTAR Checklist
Fig. 2. The compliance with the PRISMA Checklist in 2005-2008 and 2012-2015. There are 12 items significantly improvements in 2012–2015 than those in 20052008. The PRISMA score was 21.50 ± 2.69 in 2012–2015 and 18.27 ± 3.03 in 2005–2008 (* P < 0.05).
reported well, including objectives (item 4, 153, 99.4%), Protocol and registration (item 5, 148, 96.1%), information sources (item 7, 143, 92.9%), search (item 8, 144, 93.5%), Study selection (item 17, 151, 98.1%), study characteristics (item 18, 141, 91.6%), synthesis of results (item 21, 144, 93.5%) and Conclusions (item 26, 152, 98.7%).
Fig. 3 and Table 3 showed the compliance of the reporting items with the AMSTAR checklist. The “a priori” design (item 1, 154, 100%) was reported best. Duplicate study selection and data extraction (item 2, 149, 96.8%) and Characteristics of studies (item 6, 139, 90.3%) were reported well. However, Grey literature search (item 4: 28, 18.2%) and Publication bias assessed (item 10, 49, 31.8%) were poorly reported. Significant improvements were found in 2012–2015 compared with it in 2005-2008 in some domains, including Duplicate study selection and data extraction (item 2, P = 0.012), List of studies (included and excluded) (item 5, P < 0.001), Characteristics of studies (item 6, P < 0.001), Scientific quality in conclusions (item 8, P = 0.004) and Methods to combine the studies (item 9, P = 0.002). In addition, Osteoarthritis Cartilage reported best of scientific quality in conclusions (item 8, P < 0.001) and OTSR reported best of Comprehensive literature search (item 3, P = 0.008), Scientific quality assessed (item 7, P < 0.001) and conflict of interest (item 11, P < 0.001). OTSR performed the best in domains of comprehensive
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
Fig. 3. The compliance with the AMSTAR Checklist in 2005–2008 and 2012–2015. There are 5 items significantly improvements in 2012–2015 than those in 2005-2008. The AMSTAR score was 8.20 ± 1.38 in 2012–2015 and 6.54 ± 1.39 in 2005–2008 (* P < 0.05).
Literature search (item 3, 16, 100%), conflict of interest (item 11, 16, 100%) and performed the worst in Grey literature search (item 4, 0, 0.0%). And arthroscopy performed the best in domains of scientific quality in conclusions (item 8, 34, 100%) and performed the worst in conflict of interest (item 11, 0, 0.0%). The AMSTAR score was 8.20 ± 1.38 in 2012–2015 (74% of items adequately reported, on average), which was improved when compared with the AMSTAR score in 2005–2008 (6.54 ± 1.39, P < 0.001). 3.5. Variables associated with PRISMA and AMSTAR reporting Results of univariate exploratory linear regression and multivariate exploratory variable regression were presented in Table 4. Univariate exploratory linear regression showed that publication of articles from the Asia/Oceania increased in 1.676 score of PRISMA score compared with that from the America (95% CI: 0.523 - 2.829, P = 0.005) and increased in 1.097 score of AMSTAR score compared with that from the America (95% CI: 0.526–1.668, P < 0.001). And publication of articles supported by public increased in 0.421 score of AMSTAR score compared with that supported by the industry (95% CI: −0.430–1.272, P < 0.001). Publication of articles from OTSR increased in 3.214 score (95% CI: 1.454–4.974, P < 0.001) of PRISMA score and 0.973 score (95% CI: 0.126–1.821, P = 0.025) of AMSTAR score compared with that from Osteoarthritis Cartilage. Multivariate exploratory variable regression showed that publication of articles between 2012 and 2015 were related with an increase in 14.750 score of PRISMA score compared with that in 2005–2008 (95% CI: 12.107–17.393, P < 0.001). Similarly, articles published in 2012–2015 had a related with an increase in 5.660 score of AMSTAR score compared with 2005–2008 (95% CI: 4.395–6.925, P = 0.001). Points of PRISMA scores and AMSTAR scores of each journal from different years were given in Fig. 4. 4. Discussion To the best of our knowledge, this is the first study to describe the methodological reporting quality of MAs publications in five orthopedics journals from 2005 to 2008 and from 2012 to 2015, including JBJS, CORR, Arthroscopy, Osteoarthritis Cartilage and OTSR. 56.5% of the 154 MAs have been completed by 4–6 authors, and most articles published by the US accounting for 36.4%. The articles from the Asia/Oceania were increased from 10.8% in 2005–2008 to 38.5% in 2012–2015, in which China made a great contribution. Due to the support of a variety of funds from public
5
Fig. 4. Variables Associated with PRISMA and AMSTAR Reporting. In 2012–2015, OTSR showed the highest score in PRISMA and AMSTAR scores. In 2005–2008, Arthroscopy showed the highest score in PRISMA scores and Osteoarthritis Cartilage showed the highest score in AMSTAR scores. (JBJS = the Journal of Bone and Joint Surgery; Osteoarthritis Cartilage = Osteoarthritis and Cartilage; Arthroscopy = The journal of arthroscopic and related surgery; CORR = Clinical Orthopaedics and Related Research; OTSR = Orthopaedics & Traumatology: Surgery & Research).
and industries, increasing number of Chinese scientists have participated in this promising research field [17]. In 2015, the amount of research spent in China was 14,000 hundred million Yuan, 2.10% of the GDP, which was increased by 1148.2 hundred million Yuan in 2012 [18]. There are several limitations in the present study. First, there are numerous journals in the orthopedics field, but this study only selected the five journals to represent the quality of MAs of all journals, and there may be with a little error. Second, only MAs published in eight years were included. The study did not show tendency in the quality of these journals consecutively. Besides, there may be some high-quality MAs were missed. Third, in this study, the methodology was the only criteria, it may result as some well-designed and well-conducted MAs were missed. However, to some extent, this study reflected the reporting quality of MAs in the orthopedic field. In most cases, PRISMA statements are important for authors to improve their article’s quality and transparency [19]. Although there was huge promotion of quality of MAs published in 2012–2015 compared with that in 2005–2008, there were still many shortcomings in some items. Twelve items were significantly improved in 2012–2015 compared with that in 2005–2008. For example, Structured summary (item 2) increased from 75.7% to 94.0%, and Data items (item 11) increased from 56.8% to 83.8%. And there was significant improvement in PRISMA score, from 18.27 to 21.50 in average score. But the difference of PRISMA score in two period times is tiny. We think the MAs published in 2005–2008 also have a high quality and only have shortcomings in some items, and with the publication of PRISMA, in 2012–2015, the quality of MAs has comprehensive progress. It was shown that many authors of MAs paid much more attention to the methodological reporting quality of MAs. But there were also some items unsatisfactory, such as Risk of bias in individual studies (item 12), Risk of bias across studies (item 15), Risk of bias within studies (item 19), Risk of bias across studies (item 22) and Additional analyses (item 23), there was no obvious difference between 2012–2015 and 2005–2008 in these items. Although selection method approached to assess and adjusted for publication biases in meta-analysis have a long history that dates back over 30 years, they have yet to be subject to an extensive simulation study that covers multiple continuously related settings and that evaluated model performance across multiple dimensions, we should continue to improve the quality of items for bias [20]. Interestingly, the items that were associated with “Additional analyses” both in the methods (item 16) were still in lower level, despite the fact that there were significant differences from 2012–2015 to 2005–2008. Additional analyses include
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
6
sensitivity analysis, grouping analyses and meta-regression. It can help understand whether the results of their reviews were robust or dependable [21,22]. It represented that the PRISMA guidelines wasn’t completely used by the authors. So, we should advocate publicizing the PRISMA guidelines through forum, lectures and academic conference etc., to make the quality of MAs gradually increased. Apart from PRISMA guidelines, AMSTAR was also an important and effective measurement tool to appraise the methodological quality of meta-analyses [23,24]. There were 4 items had a significant difference in AMSTAR score from 2012–2015 to 2005–2008. Among them List of studies included and excluded (item 5) has a high degree of promotion. Meanwhile, the AMSTAR score also has a significant improvement, explaining that AMSTAR played an important role in improving the quality of MAs. But, there were a few problems. First, Publication bias assessed (item 10) in 2012–2015 was better than before (24.3% in 2005–2008), but still at a low level (34.2% in 2012–2015). Second, Comprehensive literature search (item 3), Grey literature search (item 4), Publication bias assessed (item 10) and Conflict of interest (item 11) have no significant change between two period times. Including grey literature can broaden the scope to relevant studies, thereby providing a more complete view of available evidence, even though searching for grey literature can be challenging [25]. So, popularize of AMSTAR score was most important. Interestingly, the AMSTAR score of four journals has a statistical difference, and the score of OTSR was better than others. Multivariate regression indicated that MAs published in 2012–2015 had a relation with an additional 2.807 in PRISMA score and 1.528 in AMSTAR score to those published in 2005–2008. We could see the growth trend of the points of PRISMA scores and AMSTAR scores of each journal in two period times. After the PRISMA score was published in 2009, it might take an important role in improving the methodological quality of MAs [9]. 5. Conclusion In summary, the present study showed that the methodology quality of MAs from five journals. Although, it has shown a great trend to improve in some items in PRISMA and AMSTAR, there were still many deficiencies in MAs. We cannot attribute this improvement to the publication of PRISMA score and AMSTAR score, and there were other factors such as increased funding source. We expected that authors, journal editors and readers could pay more attention on the quality of MAs, to make the meta-analysis better serves the clinical work. Role of the funding source None. Disclosure of interest The authors declare that they have no competing interest. Acknowledgments
Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.otsr.2017.08.009. References [1] Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010;8:336–41. [2] Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700, http://dx.doi.org/10.1136/bmj.b2700. [3] Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007;4:447–55. [4] Dawson DV, Pihlstrom BL, Blanchette DR. Understanding and evaluating metaanalysis. J Am Dent Assoc 2016;147:264–70. [5] Wen J, Ren Y, Wang L, Li YP, Liu Y, Zhou M, et al. The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol 2008;61: 770–5. [6] Liu P, Qiu Y, Qian Y, Chen X, Wang Y, Cui J, et al. Quality of meta-analyses in major leading gastroenterology and hepatology journals: a systematic review. J Gastroenterol Hepatol 2017;32:39–44. [7] Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10. [8] Faggion CM. Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Med Res Methodol 2015;15:63. [9] Adie S, Ma D, Harris IA, Naylor JM, Craig JC. Quality of conduct and reporting of meta-analyses of surgical interventions. Ann Surg 2015;261:685–94. [10] Tan WK, Wigley J, Shantikumar S. The reporting quality of systematic reviews and meta-analyses in vascular surgery needs improvement: a systematic review. Int J Surg 2014;12:1262–5. [11] Ho RS, Wu X, Yuan J, Liu S, Lai X, Wong SY, et al. Methodological quality of metaanalyses on treatments for chronic obstructive pulmonary disease: a crosssectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. NPJ Prim Care Respir Med 2015;25:14102. [12] Rogante M, Kairy D, Giacomozzi C, Grigioni M. A quality assessment of systematic reviews on telerehabilitation: what does the evidence tell us? Ann Ist Super Sanita 2015;51:11–8. [13] Shi C, Zhu L, Wang X, Qin CX, Xu Q, Tian J. Epidemiology, methodological and reporting characteristics of systematic reviews of nursing interventions published in China. Int J Nurs Pract 2014;20:689–700. [14] Bartolucci AA, Meta-analysis:. Some clinical and statistical contributions in several medical disciplines. Yonsei Med J 2007;48:157–63. [15] MedSci. http://wwwmedscicn/sci/indexdo?action=search#result. [16] National Library of Medicine National Institutes of Health http://www.ncbi. nlm.nih.gov/pubmed. [17] Qiu G. Development of orthopaedics in China-past and future. Zhonghua Wai Ke Za Zhi 2015;53:22–6. [18] Chinese Science and Technology Network. Available at: http://www.wokeji. com/zhengwu/sqfb/201309/t20130926 305042.shtml. [19] Hutton B, Catala-Lopez F, Moher D. The PRISMA statement extension for systematic reviews incorporating network meta-analysis: PRISMA-NMA. Med Clin (Barc) 2016;147:262–6. [20] McShane BB, Bockenholt U, Hansen KT. Adjusting for publication bias in metaanalysis: an evaluation of selection methods and some cautionary notes. Perspect Psychol Sci 2016;11:730–49. [21] van Lent M, Overbeke J, Out HJ. Recommendations for a uniform assessment of publication bias related to funding source. BMC Med Res Methodol 2013;13:120. [22] Higgins JP, Thompson SG. Controlling the risk of spurious findings from metaregression. Stat Med 2004;23:1663–82. [23] Sharif MO, Janjua-Sharif FN, Ali H, Ahmed F. Systematic reviews explained: AMSTAR-how to tell the good from the bad and the ugly. Oral Health Dent Manag 2013;12:9–16. [24] Wegewitz U, Weikert B, Fishta A, Jacobs A, Pieper D. Resuming the discussion of AMSTAR: What can (should) be made better? BMC Med Res Methodol 2016;16:111. [25] Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods 2014;5:221–34.
We thank to the authors of the primary studies.
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
ARTICLE IN PRESS Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
Available online at
ScienceDirect www.sciencedirect.com
Review article
Quality of meta-analyses in major leading orthopedics journals: A systematic review X. Zhi a,1 , Z. Zhang a,1 , J. Cui b , X. Zhai b , X. Chen b,∗ , J. Su b,∗ a b
Graduate Management Unit, Changhai hospital affiliated to the Second Military Medical University, Changhai road, 200433 Shanghai, PR, China Department of Orthopedics, Changhai hospital affiliated to the Second Military Medical University, Changhai road, 200433 Shanghai, PR, China
a r t i c l e
i n f o
Article history: Received 28 February 2017 Accepted 24 August 2017 Keywords: Meta-Analysis Methodological quality Orthopedics PRISMA AMSTAR
a b s t r a c t Background: Meta-Analyses are the basis of professional and healthcare agencies recommendations and have a growing importance. Quality of meta-analyses has been investigated in some medical fields but to our best knowledge this issue remains under investigated in orthopedics. Therefore, we performed a systematic analysis to: 1) after the introduction of PRISMA statement as a comprehensive guideline and the use of the AMSTAR tool as the standard for sufficient review methodology, has the quality of MAs improved because of that? 2) have some general characteristics influenced the quality of MAs (country, funding source, number of authors)? Material and Methods: We systematically searched the meta-analyses in the top four journals with the impact factor (2015) as following: JBJS, Osteoarthritis Cartilage Arthroscopy and Clin Orthop Relat Res from 2005 to 2008 and from 2012 to 2015. Likewise from 2012–2015, we also analyzed the meta-analyses from OTSR. Characteristics were extracted based on the PRISMA statement and the AMSTAR tool. Country, number of authors, funding source were also extracted. Results: A total of 154 meta-analyses were included in the present study. Score with PRISMA statement and the AMSTAR checklist were 20.86 ± 3.04 out of a maximum of 27 and 7.86 ± 1.55 out of a maximum of 11. The best journal was OTSR according to the PRISMA (23.06 ± 1.92) and AMSTAR (9.13 ± 0.87) scores. And the worst journal was Clin Orthop Relat Res according to the PRISMA score (19.4 ± 2.70) and JBJS according to the AMSTAR score (6.78 ± 1.65). Twelve items showed significant difference in the PRISMA statement, and five items in the AMSTAR checklist. Integral score of PRISMA statement and AMSTAR checklist has a significant difference between 2005-2008 and 2012–2015. The MAs reported from U.S. (56, 36.4%) were more than any other region in the world. And the MAs published by Asia/Oceania increased remarkably between these two period times [from (4, 10.8%) to (45, 38.5%)]. Conclusion: This study showed that methodological reporting quality of meta-analyses in the major orthopedics journals has improved after the publication of the PRISMA statement. Level of evidence: Level III. © 2017 Elsevier Masson SAS. All rights reserved.
Abbreviations: MA, Meta-analysis; SRs, System reviews; PRISMA, Preferred reporting items for systematic reviews and meta-analyses; AMSTAR, Assessing the methodological quality of systematic review; JBJS, The Journal of Bone and Joint Surgery; Osteoarthritis Cartilage, Osteoarthritis and Cartilage; Arthroscopy, The journal of arthroscopic and related surgery; CORR, Clinical Orthopaedics and Related Research; OTSR, Orthopaedics, Traumatology & Surgery Research. ∗ Corresponding authors. Department of Orthopedics, Changhai hospital affiliated to the Second Military Medical University, Shanghai, PR China. E-mail addresses: [email protected] (X. Chen), [email protected] (J. Su). 1 These authors contributed equally to this work.
1. Introduction Systematic reviews (SRs) and meta-analyses (MAs) have become important in health care. Clinicians refer to them to learn the latest clinical methods in these fields [1]. MAs and SRs are indispensable tools for summing up evidence precisely and reliably. MAs and SRs help clinicians keep up to date; supply evidence for policy makers to estimate risks, interests and harms of healthcare behaviors and treatments; collect and sum up related researches for
http://dx.doi.org/10.1016/j.otsr.2017.08.009 1877-0568/© 2017 Elsevier Masson SAS. All rights reserved.
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
2
patients; provide a beginning point for clinical practice guideline developers; summarize of previous researches for funders wishing to support new studies; and help editors select the merits of publishing reports of new researches [2]. It was estimated that at least 2500 new MAs (including systematic reviews, SRs) were indexed in Medline yearly [3]. However, lots of MAs were methodologically poorly reported and could introduce bias and impair the reliability of conclusions although rigorous methodology is a feature of MAs [4,5]. To prevent such incidents, the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses, PRISMA) was developed in 2009 (Supplemental data 1) [1]. It was an evidencebased minimum set of items for reporting in Systematic reviews and meta-analyses assessing randomized control trials (RCT) [6]. The PRISMA statement was developed by a group of 29 review authors, methodologists, clinicians, medical editors, and consumers [1]. The objective of the PRISMA statement was to help authors improve the reporting quality of SRs and MAs. PRISMA could also be advantageous for crucial appraisal of published systematic reviews. It has been extensively used as a criterion to assess the reporting quality of MAs since it was published in 2009 [1]. AMSTAR (Assessing the Methodological Quality of SR, AMSTAR) also emerged in 2007 (Supplemental data 2) [7], and subsequently was simplified to 11 items from 37 items. The aim of the AMSTAR is to evaluate the reporting quality of SRs, by establish upon empirical data on previously reported tools, empirical evidence and utilizing expert opinion [8]. After the PRISMA statement and AMSTAR tool were reported, assessment of the reported quality of MAs have been reported in many fields, including surgery [9,10], pulmonary disease [11], telerehabilitation [12], nursing [13] and gastroenterology and hepatology [6]. Meta-Analysis with its present form of statistically integrating information from several studies all with a common underlying theme has been around for over 25 years [14]. There are growing numbers of MAs being published in orthopedics fields, and guiding the clinical work. However, to our best knowledge, the evaluation of orthopedic journals has never been reported before. To solve this issue, we systematically evaluated the methodological reporting quality of MAs before and after the publication of the developed PRISMA statement (from 2005 to 2008 and from 2012 to 2015), in order to observe the quality of MAs whether improve or not after the standard was published. MAs were selected from the 4 orthopedics journals with the highest impact factor (2015) and Orthopaedics & Traumatology: Surgery & Research which is not a leading journal in this field but also introduced PRISMA before 2012. We introduce the following questions: • has the quality of MAs improved after the introduction of PRISMA statement as a comprehensive guideline and the use of the AMSTAR tool as the standard for sufficient review methodology? • have some general characteristics influenced the quality of MAs (country, funding source, number of authors)? 2. Methods of search strategy and criteria 2.1. Search strategy To appraise the methodological quality of MAs on orthopedics fields, we selected the top four journals with the highest impact factor (2015) as following: the Journal of Bone Joint Surgery (JBJS), Osteoarthritis and Cartilage (Osteoarthritis Cartilage), Arthroscopy – the journal of arthroscopic and related surgery (Arthroscopy) and Clinical Orthopaedics and Related Research (CORR) [15]. In
Table 1 Principals characteristics of the included trials.
Number No. authors 1–3 4–6 >6 Region/Country US Europe Asia/Oceania Others Funding source Industry Public Not specific None
All trials (%)
2005–2008 (%)
2012–2015 (%)
154 (100.0)
37 (24.0)
117 (76.0)
27 (17.5) 87 (56.5) 40 (26.0)
6 (16.2) 25 (67.6) 6 (16.2)
21 (17.9) 62 (53.0) 34 (29.1)
56 (36.4) 31 (20.1) 49 (31.8) 18 (11.7)
18 (48.6) 10 (27.0) 4 (10.8) 5 (13.5)
38 (32.4) 21 (17.9) 45 (38.5) 13 (11.2)
19 (12.3) 38 (24.7) 31 (20.1) 66 (42.9)
2 (5.4) 5 (13.5) 12 (32.4) 18 (48.6)
17 (14.5) 33 (28.2) 19 (16.2) 48 (41.1)
addition, we picked up the MAs published from 2012 to 2015 in Orthopaedics & Traumatology: Surgery & Research (OTSR), another journal, which has also introduced PRISMA before 2012 as comparison. The conformed MAs were conducted on the PubMed/MEDLINE database [16] by two co-first authors on October 1st, 2016. The ISSNs (Print) of five journals were 1535–1386, 1063–4584, 0749–8063, 0009–921X and 1877–0568. To evaluate the PRISMA statement published in 2009, we chose the articles from 2005 to 2008 and from 2012 to 2015. We selected the “Meta-analysis” in the “Article types” project of Web of science, and excluded the following types: randomized controlled trial or review or the article published in 2016. 2.2. Data extraction and criteria We primarily extracted geography, publishing, clinical and epidemiological characteristics based on the previous meta-analysis and systematic reviews, for example region/country, number of authors, publishing year, number of patients and funding source. Then we collected the information of MAs with PRISMA statement [2] and AMSTAR checklist [7]. The PRISMA statement is a 27-item list and the AMSTAR checklist is an 11-item list. The work of extraction information was completed by two co-first authors, and any disagreement was resolved by discussion between the two reviewers or other authors. 2.3. Statistical analyses All statistical analyses were performed with SPSS 21.0 (SPSS Statistical software version 21.0. IBM Corporation, USA). Frequencies and proportions were analyzed, and descriptive statistics (mean, standard deviation) of the country, funding source were employed (Table 1). For the parabole between years and journals, we performed chi-squared test and fisher exact test (Tables 2 and 3). For the determination of variables that associated with PRISMA or AMSTAR score, we firstly accomplished the univariate linear regression and then combined with stepwise backward multiple regression analysis. P < 0.05 was regarded statistically significant. Effect size (Odds ratio, OR) and 95% confidence intervals were reported for significant associations (Table 4). 3. Results 3.1. Search results A total of 167 MAs were retrieved from the five journals, in which 13 studies were excluded (Fig. 1) (all detailed results of the 167
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model
ARTICLE IN PRESS
OTSR-1840; No. of Pages 6
X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
3
Table 2 Proportions of adequately reported PRISMA items in different years and different journals. Items of the PRISMA statement
1. Title 2. Structured summary 3. Rationale 4. Objectives 5. Protocol and registration 6. Eligibility criteria 7. Information sources 8. Search 9. Study selection 10. Data collection process 11. Data items 12. Risk of bias in individual studies 13. Summary measures 14. Synthesis of results 15. Risk of bias across studies 16. Additional analyses 17. Study selection 18. Study characteristics 19. Risk of bias within studies 20. Results of individual studies 21. Synthesis of results 22. Risk of bias across studies 23. Additional analysis 24. Summary of evidence 25. Limitations 26. Conclusions 27. Funding PRISMA score (0 to 27) * a
All meta
2006–2008
2012–2015
P-value
(n = 154)
(n = 37)
(n = 117)
134 (87.0) 138 (89.6) 127 (82.5) 153 (99.4) 148 (96.1) 140 (90.9) 143 (92.9) 144 (93.5) 127 (82.5) 106 (68.8) 119 (77.3) 55 (35.7)
31 (83.8) 28 (75.7) 30 (81.1) 36 (97.3) 33 (89.2) 28 (75.7) 28 (75.7) 30 (81.1) 30 (81.1) 20 (54.1) 21 (56.8) 12 (32.4)
103 (88.0) 110 (94.0) 97 (82.9) 117 (100.0) 115 (98.3) 112 (95.7) 115 (98.3) 114 (97.4) 97 (82.9) 86 (73.5) 98 (83.8) 43 (36.8)
0.575 0.003* 0.807 0.24 0.030* 0.001* < 0.001* 0.002* 0.807 0.041* 0.001* 0.697
129 (83.8) 123 (79.9) 42 (27.3) 35 (22.7) 151 (98.1) 141 (91.6) 56 (36.4) 129 (83.8) 144 (93.5) 45 (29.2) 38 (24.7) 154 (100.0) 115 (74.7) 152 (98.7) 88 (57.1) 20.86 ± 3.04
24 (64.9) 28 (75.7) 12 (32.4) 4 (10.8) 35 (94.6) 24 (64.9) 15 (40.5) 25 (67.6) 31 (83.8) 10 (27.0) 5 (13.5) 37 (100.0) 27 (73.0) 35 (94.6) 19 (51.4) 18.27 ± 3.03
105 (89.7) 95 (81.2) 30 (25.6) 31 (26.5) 116 (99.1) 117 (100.0) 41 (35.0) 104 (88.9) 113 (96.6) 35 (29.9) 33 (28.2) 117 (100.0) 88 (75.2) 117 (100.0) 69 (59.0) 21.50 ± 2.69
0.001* 0.485 0.407 0.007* 0.144 < 0.001* 0.561 0.004* 0.013* 0.837 0.082 / 0.829 0.057 0.45 0.001*
CORR
JBJS
OTSR
(n = 41)
Arthroscopy Osteoarthritis Cartilage (n = 34) (n = 31)
P-value
(n = 32)
(n = 16)
30 (73.2) 32 (78.0) 27 (65.9) 40 (97.6) 40 (97.6) 36 (87.8) 38 (92.7) 34 (82.9) 32 (78.0) 32 (78.0) 31 (75.6) 9 (22.0)
29 (85.3) 34 (100.0)a 28 (82.4) 34 (100.0)a 31 (91.2) 32 (94.1) 33 (97.1) 32 (94.1) 28 (82.4) 26 (76.5) 27 (79.4) 8 (23.5)
27 (87.1) 25 (80.6) 24 (77.4) 31 (100.0)a 29 (93.5) 25 (80.6) 29 (93.5) 30 (96.8) 27 (87.1) 26 (83.9)a 24 (77.4) 8 (25.8)
32 (100.0)a 31 (96.9) 32 (100.0)a 32 (100.0)a 32 (100.0)a 31 (96.9) 27 (84.4) 32 (100.0)a 26 (81.3) 9 (28.1) 23 (71.9) 15 (46.9)
16 (100.0)a 16 (100.0)a 16 (100.0)a 16 (100.0)a 16 (100.0)a 16 (100.0)a 16(100.0)a 16(100.0)a 14 (87.5)a 13 (81.3) 14 (87.5)a 15 (93.8)a
0.007* 0.003* 0.001* 0.596 0.295 0.098 0.228 0.022* 0.857 < 0.001* 0.802 < 0.001*
40 (97.6) 33 (80.5) 9 (22.0) 7 (17.1) 41 (100.0)a 38 (92.7) 10 (24.4) 36 (87.8) 36 (87.8) 10 (24.4) 7 (17.1) 41 (100.0) 26 (63.4) 41 (100.0)a 11 (26.8) 19.4 ± 2.70
32 (94.1) 25 (73.5) 7 (20.6) 11 (32.4) 34 (100.0)a 31 (91.2) 8 (23.5) 29 (85.3) 33 (97.1) 8 (23.5) 10 (29.4) 34 (100.0) 29 (85.3) 34 (100.0)a 17 (50.0) 20.6 ± 3.14
30 (96.8) 22 (71.0) 7 (22.6) 11 (35.5)a 31 (100.0)a 30 (96.8) 9 (29.0) 26 (83.9) 30 (96.8) 9 (29.0) 11 (35.5)a 31 (100.0) 12 (38.7) 31 (100.0)a 17 (54.8) 20.2 ± 2.99
11 (34.4) 31 (96.9)a 14 (43.8)a 5 (15.6) 30 (93.8) 26 (81.3) 14 (43.8) 22 (68.8) 29 (90.6) 10 (31.3) 9 (28.1) 32 (100.0) 32 (100.0)a 30 (93.8)a 31 (96.9)a 20.3 ± 3.15
16 (100.0)a 12 (75.0) 5 (31.3) 1 (6.3) 15 (93.8) 16 (100.0)a 15 (93.8)a 16 (100.0)a 16 (100.0)a 8 (50.0)a 1 (6.3) 16 (100.0) 16 (100.0)a 16 (100.0)a 6 (37.5) 23.06 ± 1.92
< 0.001* 0.079 0.184 0.066 0.149 0.134 < 0.001* 0.062 0.288 0.355 0.149 / < 0.001* / < 0.001* < 0.001*
P < 0.05. The highest score.
Table 3 Proportions of adequately reported AMSTAR items in different years and different journals. Items of the AMSTAR statement
1. “a priori” design 2. Duplicate study selection and data extraction 3. Comprehensive literature search 4. Grey literature search 5. List of studies included and excluded 6. Characteristics of studies 7. Scientific quality assessed 8. Scientific quality in conclusions 9. Methods to combine the studies 10. Publication bias assessed 11. Conflict of interest AMSTAR score (0 to 11s) * a
All meta
2006–2008 2012–2015 P-value
CORR
(n = 154)
(n = 37)
(n = 41)
(n = 117)
Arthroscopy Osteoarthritis Cartilage (n = 34) (n = 31)
41 (100.0) 34 (100.0) 41 (100.0)a 33 (97.1)
JBJS
OTSR
P-value
(n = 32)
(n = 16)
31 (100.0) 30 (96.8)
32 (100.0) 29 (90.6)
16 (100.0) / 16 (100.0)a 0.219
154 (100.0) 37 (100.0) 149 (96.8) 33 (89.2)
117 (100) 116 (99.1)
/ 0.012*
112 (72.7) 28 (18.2) 105 (68.2)
22 (59.5) 7 (18.9) 10 (27.0)
90 (76.9) 21 (17.9) 95 (81.2)
0.055 26 (63.4) 1 9 (22.0) < 0.001* 35 (85.4)
21 (61.8) 8 (23.5)a 30 (88.2)
21 (67.7) 6 (19.4) 23 (74.2)
28 (87.5) 5 (15.6) 1 (3.1)
16 (100.0)a 0.008* 0 (0.0) 0.062 16 (100.0)a 0.312
139 (90.3) 124 (80.5) 123 (79.9) 133 (86.4) 49 (31.8) 70 (45.5) 7.86 ± 1.55
26 (70.3) 29 (78.4) 23 (62.2) 26 (70.3) 9 (24.3) 20 (54.1) 6.54 ± 1.34
113 (96.6) 95 (81.2) 100 (85.5) 107 (91.5) 40 (34.2) 50 (42.7) 8.20 ± 1.38
< 0.001* 0.812 0.004* 0.002* 0.314 0.344 0.001*
29 (85.3) 31 (91.2) 34 (100.0)a 29 (85.3) 9 (26.5) 0 (0.0) 7.6 ± 1.52
31 (100.0)a 26 (83.9) 28 (90.3) 28 (90.3) 11 (35.5) 22 (71.0) 8.3 ± 1.42
26 (81.3) 14 (43.8) 8 (25.0) 27 (84.4) 17 (53.1)a 30 (93.8) 6.78 ± 1.65
16 (100.0)a 15 (93.8)a 14 (87.5) 15 (93.8)a 6 (37.5) 16 (100.0)a 9.13 ± 0.87
37 (90.2) 38 (92.7) 39 (95.1) 34 (82.9) 6 (14.6) 2 (4.9) 7.5 ± 1.17
0.062 < 0.001* < 0.001* 0.79 0.011* < 0.001* 0.001*
P < 0.05. The highest score.
studies investigated are reported in Supplemental data 3). Finally, 154 MAs were accepted in the article, including 32 from the JBJS, 34 from the Arthroscopy, 41 from the CORR, 31 from the Osteoarthritis Cartilage and 16 from the OTSR. Thirty-seven (24.0%) MAs were published between 2005 and 2008, and one hundred and seven (76.0%) MAs were published between 2012 and 2015 (all detailed results of the 154 inclusions are reported in Supplemental data 3).
whole, the MAs reported from US (56, 36.4%) were more than any other region in the world. The Asia/Oceania production increased remarkably between these two periods [from (4, 10.8%) to (45, 38.5%)]. In 2005–2008, 18.9% of the MAs were with funding source, and 32.4% were not specific. And it was 42.7% with the funding source and 16.2% that were not specific in 2012–2015. 3.3. Compliance with the PRISMA checklist
3.2. General characteristics Crucial characteristics of the included MAs were shown in the Table 1. 87 (56.5%) of MAs have 4-6 authors per article. As a
The proportion and percentage of MAs that adequately reported each PRISMA items were showed in Fig. 2 and Table 2. Summary of evidence (item 24) was reported 100%. Most domains were
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
4
Table 4 Univariate and multivariate exploratory linear relationship of changes in PRISMA score and AMSTAR score. Items
Univariate exploratory variable Country (vs. America) Europe Asia/Oceania Others Journal (vs. Osteoarthritis Cartilage) Arthroscopy CORR JBJS OTSR Funding (vs. Industry) Public None Not specified Numbers of authors (vs. 1–3) Year (vs. 2005–2008) Multivariate exploratory variable Year (vs. 2005–2008) *
PRISMA score
AMSTAR score
s (OR)
95% CI
P
s (OR)
95% CI
P
0.54 1.676 1.466
(−0.780, 1.86) (0.523, 2.829) (−0.131, 3.064)
0.420 0.005* 0.072
0.667 1.097 0.948
(0.013, 1.321) (0.526, 1.668) (0.157, 1.740)
0.046* < 0.001* 0.019*
0.433 −0.156 0.769 3.214
(−1.001, 1.866) (−1.523, 1.210) (−0.643, 2.181) (1.454, 4.974)
0.552 0.822 0.283 0.000*
−1.370 −0.562 −0.563 0.973
(−2.060, −0.680) (−1.220, 0.096) (−1.243, 0.116) (0.126, 1.821)
< 0.001* 0.094 0.104 0.025*
0.500 0.273 0.032 0.143 2.807
(−1.204, 2.204) (−1.493, 2.040) (−1.547, 1.611) (−0.091, 0.378) (1.761, 3.853)
0.563 0.760 0.968 0.23 0.001*
0.421 −0.321 0.194 0.027 1.528
(−0.430, 1.272) (−1.203, 0.561) (−0.594, 1.74) (−0.092, 0.147) (1.007, 2.048)
< 0.001* 0.330 0.473 0.651 < 0.001*
14.750
(12.107, 17.393)
< 0.001*
5.660
(4.395, 6.925)
0.001*
P < 0.05.
Fig. 1. Flow of information through the different phases of a systematic review. (JBJS = the Journal of Bone and Joint Surgery; Osteoarthritis Cartilage = Osteoarthritis and Cartilage; Arthroscopy = The journal of arthroscopic and related surgery; CORR = Clinical Orthopaedics and Related Research; OTSR = Orthopaedics & Traumatology: Surgery & Research).
However, there were some items poorly reported, including risk of bias in individual studies (item 12, 55, 35.7%), risk of bias across studies (item 15, 42, 27.3%), additional analyses (item 16, 35, 22.7%), Risk of bias within studies (item 19, 56, 36.4%), risk of bias across studies (item 22, 45, 29.2%) and Additional analyses (item 23, 38, 24.7%). There were significant improvements in 2012–2015 than those in 2005–2008. It was reflected in the following items: Structured summary (item 2, P = 0.003), Protocol and registration (item 5, P = 0.03), Eligibility criteria (item 6, P = 0.001), Information sources (item 7, P < 0.001), Search (item 8, P = 0.002), Data collection process (item 10, P = 0.041), Data items (item 11, P = 0.001), Summary measures (item 13, P = 0.001), Additional analyses (item 16, P = 0.007), Study characteristics (item 18, P < 0.001), Results of individual studies (item 20, P = 0.004) and Synthesis of results (item 21, P = 0.013). The PRISMA score was 21.50 ± 2.69 in 2012–2015 (77% of items adequately reported, on average), has significant improvements compared with the PRISMA score in 2005–2008 (18.27 ± 3.03, P < 0.001). In addition, OTSR performed the best in domains of the average score of PRISMA (23.06 ± 1.92) and risk of bias in individual studies (item 12, 15, 93.8%). JBJS performed the best in domains of Funding (item 27, 31, 96.9%).
3.4. Compliance with the AMSTAR Checklist
Fig. 2. The compliance with the PRISMA Checklist in 2005-2008 and 2012-2015. There are 12 items significantly improvements in 2012–2015 than those in 20052008. The PRISMA score was 21.50 ± 2.69 in 2012–2015 and 18.27 ± 3.03 in 2005–2008 (* P < 0.05).
reported well, including objectives (item 4, 153, 99.4%), Protocol and registration (item 5, 148, 96.1%), information sources (item 7, 143, 92.9%), search (item 8, 144, 93.5%), Study selection (item 17, 151, 98.1%), study characteristics (item 18, 141, 91.6%), synthesis of results (item 21, 144, 93.5%) and Conclusions (item 26, 152, 98.7%).
Fig. 3 and Table 3 showed the compliance of the reporting items with the AMSTAR checklist. The “a priori” design (item 1, 154, 100%) was reported best. Duplicate study selection and data extraction (item 2, 149, 96.8%) and Characteristics of studies (item 6, 139, 90.3%) were reported well. However, Grey literature search (item 4: 28, 18.2%) and Publication bias assessed (item 10, 49, 31.8%) were poorly reported. Significant improvements were found in 2012–2015 compared with it in 2005-2008 in some domains, including Duplicate study selection and data extraction (item 2, P = 0.012), List of studies (included and excluded) (item 5, P < 0.001), Characteristics of studies (item 6, P < 0.001), Scientific quality in conclusions (item 8, P = 0.004) and Methods to combine the studies (item 9, P = 0.002). In addition, Osteoarthritis Cartilage reported best of scientific quality in conclusions (item 8, P < 0.001) and OTSR reported best of Comprehensive literature search (item 3, P = 0.008), Scientific quality assessed (item 7, P < 0.001) and conflict of interest (item 11, P < 0.001). OTSR performed the best in domains of comprehensive
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
Fig. 3. The compliance with the AMSTAR Checklist in 2005–2008 and 2012–2015. There are 5 items significantly improvements in 2012–2015 than those in 2005-2008. The AMSTAR score was 8.20 ± 1.38 in 2012–2015 and 6.54 ± 1.39 in 2005–2008 (* P < 0.05).
Literature search (item 3, 16, 100%), conflict of interest (item 11, 16, 100%) and performed the worst in Grey literature search (item 4, 0, 0.0%). And arthroscopy performed the best in domains of scientific quality in conclusions (item 8, 34, 100%) and performed the worst in conflict of interest (item 11, 0, 0.0%). The AMSTAR score was 8.20 ± 1.38 in 2012–2015 (74% of items adequately reported, on average), which was improved when compared with the AMSTAR score in 2005–2008 (6.54 ± 1.39, P < 0.001). 3.5. Variables associated with PRISMA and AMSTAR reporting Results of univariate exploratory linear regression and multivariate exploratory variable regression were presented in Table 4. Univariate exploratory linear regression showed that publication of articles from the Asia/Oceania increased in 1.676 score of PRISMA score compared with that from the America (95% CI: 0.523 - 2.829, P = 0.005) and increased in 1.097 score of AMSTAR score compared with that from the America (95% CI: 0.526–1.668, P < 0.001). And publication of articles supported by public increased in 0.421 score of AMSTAR score compared with that supported by the industry (95% CI: −0.430–1.272, P < 0.001). Publication of articles from OTSR increased in 3.214 score (95% CI: 1.454–4.974, P < 0.001) of PRISMA score and 0.973 score (95% CI: 0.126–1.821, P = 0.025) of AMSTAR score compared with that from Osteoarthritis Cartilage. Multivariate exploratory variable regression showed that publication of articles between 2012 and 2015 were related with an increase in 14.750 score of PRISMA score compared with that in 2005–2008 (95% CI: 12.107–17.393, P < 0.001). Similarly, articles published in 2012–2015 had a related with an increase in 5.660 score of AMSTAR score compared with 2005–2008 (95% CI: 4.395–6.925, P = 0.001). Points of PRISMA scores and AMSTAR scores of each journal from different years were given in Fig. 4. 4. Discussion To the best of our knowledge, this is the first study to describe the methodological reporting quality of MAs publications in five orthopedics journals from 2005 to 2008 and from 2012 to 2015, including JBJS, CORR, Arthroscopy, Osteoarthritis Cartilage and OTSR. 56.5% of the 154 MAs have been completed by 4–6 authors, and most articles published by the US accounting for 36.4%. The articles from the Asia/Oceania were increased from 10.8% in 2005–2008 to 38.5% in 2012–2015, in which China made a great contribution. Due to the support of a variety of funds from public
5
Fig. 4. Variables Associated with PRISMA and AMSTAR Reporting. In 2012–2015, OTSR showed the highest score in PRISMA and AMSTAR scores. In 2005–2008, Arthroscopy showed the highest score in PRISMA scores and Osteoarthritis Cartilage showed the highest score in AMSTAR scores. (JBJS = the Journal of Bone and Joint Surgery; Osteoarthritis Cartilage = Osteoarthritis and Cartilage; Arthroscopy = The journal of arthroscopic and related surgery; CORR = Clinical Orthopaedics and Related Research; OTSR = Orthopaedics & Traumatology: Surgery & Research).
and industries, increasing number of Chinese scientists have participated in this promising research field [17]. In 2015, the amount of research spent in China was 14,000 hundred million Yuan, 2.10% of the GDP, which was increased by 1148.2 hundred million Yuan in 2012 [18]. There are several limitations in the present study. First, there are numerous journals in the orthopedics field, but this study only selected the five journals to represent the quality of MAs of all journals, and there may be with a little error. Second, only MAs published in eight years were included. The study did not show tendency in the quality of these journals consecutively. Besides, there may be some high-quality MAs were missed. Third, in this study, the methodology was the only criteria, it may result as some well-designed and well-conducted MAs were missed. However, to some extent, this study reflected the reporting quality of MAs in the orthopedic field. In most cases, PRISMA statements are important for authors to improve their article’s quality and transparency [19]. Although there was huge promotion of quality of MAs published in 2012–2015 compared with that in 2005–2008, there were still many shortcomings in some items. Twelve items were significantly improved in 2012–2015 compared with that in 2005–2008. For example, Structured summary (item 2) increased from 75.7% to 94.0%, and Data items (item 11) increased from 56.8% to 83.8%. And there was significant improvement in PRISMA score, from 18.27 to 21.50 in average score. But the difference of PRISMA score in two period times is tiny. We think the MAs published in 2005–2008 also have a high quality and only have shortcomings in some items, and with the publication of PRISMA, in 2012–2015, the quality of MAs has comprehensive progress. It was shown that many authors of MAs paid much more attention to the methodological reporting quality of MAs. But there were also some items unsatisfactory, such as Risk of bias in individual studies (item 12), Risk of bias across studies (item 15), Risk of bias within studies (item 19), Risk of bias across studies (item 22) and Additional analyses (item 23), there was no obvious difference between 2012–2015 and 2005–2008 in these items. Although selection method approached to assess and adjusted for publication biases in meta-analysis have a long history that dates back over 30 years, they have yet to be subject to an extensive simulation study that covers multiple continuously related settings and that evaluated model performance across multiple dimensions, we should continue to improve the quality of items for bias [20]. Interestingly, the items that were associated with “Additional analyses” both in the methods (item 16) were still in lower level, despite the fact that there were significant differences from 2012–2015 to 2005–2008. Additional analyses include
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009
G Model OTSR-1840; No. of Pages 6
ARTICLE IN PRESS X. Zhi et al. / Orthopaedics & Traumatology: Surgery & Research xxx (2017) xxx–xxx
6
sensitivity analysis, grouping analyses and meta-regression. It can help understand whether the results of their reviews were robust or dependable [21,22]. It represented that the PRISMA guidelines wasn’t completely used by the authors. So, we should advocate publicizing the PRISMA guidelines through forum, lectures and academic conference etc., to make the quality of MAs gradually increased. Apart from PRISMA guidelines, AMSTAR was also an important and effective measurement tool to appraise the methodological quality of meta-analyses [23,24]. There were 4 items had a significant difference in AMSTAR score from 2012–2015 to 2005–2008. Among them List of studies included and excluded (item 5) has a high degree of promotion. Meanwhile, the AMSTAR score also has a significant improvement, explaining that AMSTAR played an important role in improving the quality of MAs. But, there were a few problems. First, Publication bias assessed (item 10) in 2012–2015 was better than before (24.3% in 2005–2008), but still at a low level (34.2% in 2012–2015). Second, Comprehensive literature search (item 3), Grey literature search (item 4), Publication bias assessed (item 10) and Conflict of interest (item 11) have no significant change between two period times. Including grey literature can broaden the scope to relevant studies, thereby providing a more complete view of available evidence, even though searching for grey literature can be challenging [25]. So, popularize of AMSTAR score was most important. Interestingly, the AMSTAR score of four journals has a statistical difference, and the score of OTSR was better than others. Multivariate regression indicated that MAs published in 2012–2015 had a relation with an additional 2.807 in PRISMA score and 1.528 in AMSTAR score to those published in 2005–2008. We could see the growth trend of the points of PRISMA scores and AMSTAR scores of each journal in two period times. After the PRISMA score was published in 2009, it might take an important role in improving the methodological quality of MAs [9]. 5. Conclusion In summary, the present study showed that the methodology quality of MAs from five journals. Although, it has shown a great trend to improve in some items in PRISMA and AMSTAR, there were still many deficiencies in MAs. We cannot attribute this improvement to the publication of PRISMA score and AMSTAR score, and there were other factors such as increased funding source. We expected that authors, journal editors and readers could pay more attention on the quality of MAs, to make the meta-analysis better serves the clinical work. Role of the funding source None. Disclosure of interest The authors declare that they have no competing interest. Acknowledgments
Appendix A. Supplementary data Supplementary data associated with this article can be found, in the online version, at http://dx.doi.org/10.1016/j.otsr.2017.08.009. References [1] Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010;8:336–41. [2] Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ 2009;339:b2700, http://dx.doi.org/10.1136/bmj.b2700. [3] Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007;4:447–55. [4] Dawson DV, Pihlstrom BL, Blanchette DR. Understanding and evaluating metaanalysis. J Am Dent Assoc 2016;147:264–70. [5] Wen J, Ren Y, Wang L, Li YP, Liu Y, Zhou M, et al. The reporting quality of meta-analyses improves: a random sampling study. J Clin Epidemiol 2008;61: 770–5. [6] Liu P, Qiu Y, Qian Y, Chen X, Wang Y, Cui J, et al. Quality of meta-analyses in major leading gastroenterology and hepatology journals: a systematic review. J Gastroenterol Hepatol 2017;32:39–44. [7] Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10. [8] Faggion CM. Critical appraisal of AMSTAR: challenges, limitations, and potential solutions from the perspective of an assessor. BMC Med Res Methodol 2015;15:63. [9] Adie S, Ma D, Harris IA, Naylor JM, Craig JC. Quality of conduct and reporting of meta-analyses of surgical interventions. Ann Surg 2015;261:685–94. [10] Tan WK, Wigley J, Shantikumar S. The reporting quality of systematic reviews and meta-analyses in vascular surgery needs improvement: a systematic review. Int J Surg 2014;12:1262–5. [11] Ho RS, Wu X, Yuan J, Liu S, Lai X, Wong SY, et al. Methodological quality of metaanalyses on treatments for chronic obstructive pulmonary disease: a crosssectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. NPJ Prim Care Respir Med 2015;25:14102. [12] Rogante M, Kairy D, Giacomozzi C, Grigioni M. A quality assessment of systematic reviews on telerehabilitation: what does the evidence tell us? Ann Ist Super Sanita 2015;51:11–8. [13] Shi C, Zhu L, Wang X, Qin CX, Xu Q, Tian J. Epidemiology, methodological and reporting characteristics of systematic reviews of nursing interventions published in China. Int J Nurs Pract 2014;20:689–700. [14] Bartolucci AA, Meta-analysis:. Some clinical and statistical contributions in several medical disciplines. Yonsei Med J 2007;48:157–63. [15] MedSci. http://wwwmedscicn/sci/indexdo?action=search#result. [16] National Library of Medicine National Institutes of Health http://www.ncbi. nlm.nih.gov/pubmed. [17] Qiu G. Development of orthopaedics in China-past and future. Zhonghua Wai Ke Za Zhi 2015;53:22–6. [18] Chinese Science and Technology Network. Available at: http://www.wokeji. com/zhengwu/sqfb/201309/t20130926 305042.shtml. [19] Hutton B, Catala-Lopez F, Moher D. The PRISMA statement extension for systematic reviews incorporating network meta-analysis: PRISMA-NMA. Med Clin (Barc) 2016;147:262–6. [20] McShane BB, Bockenholt U, Hansen KT. Adjusting for publication bias in metaanalysis: an evaluation of selection methods and some cautionary notes. Perspect Psychol Sci 2016;11:730–49. [21] van Lent M, Overbeke J, Out HJ. Recommendations for a uniform assessment of publication bias related to funding source. BMC Med Res Methodol 2013;13:120. [22] Higgins JP, Thompson SG. Controlling the risk of spurious findings from metaregression. Stat Med 2004;23:1663–82. [23] Sharif MO, Janjua-Sharif FN, Ali H, Ahmed F. Systematic reviews explained: AMSTAR-how to tell the good from the bad and the ugly. Oral Health Dent Manag 2013;12:9–16. [24] Wegewitz U, Weikert B, Fishta A, Jacobs A, Pieper D. Resuming the discussion of AMSTAR: What can (should) be made better? BMC Med Res Methodol 2016;16:111. [25] Mahood Q, Van Eerd D, Irvin E. Searching for grey literature for systematic reviews: challenges and benefits. Res Synth Methods 2014;5:221–34.
We thank to the authors of the primary studies.
Please cite this article in press as: Zhi X, et al. Quality of meta-analyses in major leading orthopedics journals: A systematic review. Orthop Traumatol Surg Res (2017), http://dx.doi.org/10.1016/j.otsr.2017.08.009