- Email: [email protected]
The reporting characteristics and methodological quality of Cochrane reviews about health policy research
G Model
ARTICLE IN PRESS
HEAP-3276; No. of Pages 8
Health Policy xxx (2014) xxx–xxx
Contents lists available at ScienceDirect
Health Policy journal homepage: www.elsevier.com/locate/healthpol
Review
The reporting characteristics and methodological quality of Cochrane reviews about health policy research Li Xiu-xia a,b , Zheng Ya c , Chen Yao-long a,b , Yang Ke-hu a,b,∗ , Zhang Zong-jiu a,∗ a b c
Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou 730000, China School of Public Health, Lanzhou University, Lanzhou 730000, China
a r t i c l e
i n f o
Article history: Received 19 August 2013 Received in revised form 2 September 2014 Accepted 4 September 2014 Keywords: Health policy research Cochrane reviews Epidemiological characteristics Descriptive characteristics Methodological quality
a b s t r a c t The systematic review has increasingly become a popular tool for researching health policy. However, due to the complexity and diversity in the health policy research, it has also encountered more challenges. We set out the Cochrane reviews on health policy research as a representative to provide the first examination of epidemiological and descriptive characteristics as well as the compliance of methodological quality with the AMSTAR. 99 reviews were included by inclusion criteria, 73% of which were Implementation Strategies, 15% were Financial Arrangements and 12% were Governance Arrangements; involved Public Health (34%), Theoretical Exploration (18%), Hospital Management (17%), Medical Insurance (12%), Pharmaceutical Policy (9%), Community Health (7%) and Rural Health (2%). Only 39% conducted meta-analysis, and 49% reported being updates, and none was rated low methodological quality. Our research reveals that the quantity and quality of the evidence should be improved, especially Financial Arrangements and Governance Arrangements involved Rural Health, Health Care Reform and Health Equity, etc. And the reliability of AMSTAR needs to be tested in larger range in this field. © 2014 Elsevier Ireland Ltd. All rights reserved.
1. Introduction There has been growing international interest in the evidence-based health policy-making that scientific evidence should play an important role in the policy formulation, implementation, and assessment. During the 1990s, evidence-based approaches have become prominent on the national and international agendas for health policy and health systems research [1]. The World Health Organization has been vigorously supporting the process
∗ Corresponding authors at: Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, No. 199, Donggang West Road, Chengguan District, Lanzhou, Gansu 730000, China. Tel.: +86 13993694673. E-mail addresses: [email protected] (Y. Ke-hu), [email protected] (Z. Zong-jiu).
of contextualizing evidence and translating it into policy, especially in the developing countries [2]. The First Global Symposium on Health Systems Research (HSR) – Science to Accelerate Universal Health Coverage has been announced by the World Health Organization and partners, and aiming to improve the scientific evidence needed by health policymakers and practitioners to inform their decisions related to accelerating universal health coverage [3]. Systematic review (SR), the important tool for EvidenceBased Medicine (EBM), can inform healthcare management and policy making levels by providing research-based responses to important questions about health systems [4–6]. In recent years, more and more SRs on healthsystems research have been published in many leading journals such as The Lancet (Lancet) and the World Health Organization Bulletin, etc. [7]. An increasing number of health policy makers and researchers have used SRs to
http://dx.doi.org/10.1016/j.healthpol.2014.09.002 0168-8510/© 2014 Elsevier Ireland Ltd. All rights reserved.
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
2
synthesize research evidence to deal with health issues at global and national levels. World Health Organization Global Policy Recommendations “Increasing access to health workers in remote and rural areas through improved retention” (2010) [8] is a representative example. However, due to the complexity and diversity of the health policy research, the methodology of SRs has also been facing challenges in this field. Cochrane reviews are the SRs of primary research on human health care and health policy, and are internationally recognized as the highest standard in evidence-based health care [9]. So far, over 5000 Cochrane Reviews and 2000 protocols for Cochrane reviews are published online in the Cochrane Database of Systematic Reviews (CDSR), which is the database of Cochrane reviews in The Cochrane Library (www.thecochranelibrary.com) and the 2011 ISI Impact Factor is 5.912 [10]. The Cochrane reviews are classified into 31 topics in the CDSR, and the topic “Effective practice/health systems” follows closely health systems research which involves the evidence on health policy research. Newer assessment tool – Assessment of Multiple Systematic Reviews (AMSTAR) has been to assess the methodological quality of SRs, which is built upon previous tools, empirical evidence and expert consensus [11]. The AMSTAR approach consists of 11 items and is featured by good content validity, wide acceptance, recognized reliability and reproducibility [11–14]. It has been considered the best way to assess methodological quality of SRs by the Canadian Agency for Drugs and Technologies in Health [15]. The purpose of this review is to examine the epidemiological and descriptive characteristics, as well as the methodological quality of Cochrane reviews on health policy research published in the CDSR, which serves as a reference for health policy makers and researchers. 2. Materials and methods 2.1. Inclusion criteria We included in this review all systematic reviews indexed in CDSR that met the following four criteria: First, the systematic review synthesized evidence on a research question about health policy, such as the policy formulation, implementation or evaluation, or the methodology for health policy research, etc. Second, the review synthesized evidence in one of three categories of health policy research: Implementation Strategies, Financial Arrangements and Governance Arrangements. Third, the review had to be available in the database on 31 March 2013, when we downloaded all studies for selection and extraction. Fourth, the review was published in English. But the protocols for Cochrane reviews and Withdrawn Cochrane reviews were excluded. 2.2. Search strategy We searched the CDSR by the whole topic on “Effective practice/health systems”, and retrieved the Advanced Search by “Title, Abstract and Keywords” as a supplement
in March 2013. The search terms included health, policy and strategy. 2.3. Screening According to predetermined inclusion criteria, two reviewers independently screened all of the search results by title and abstract, and subsequently retrieved and screened the full text of potentially included studies (if one or both reviewers thought it was potentially relevant). Disagreements were resolved by discussion. 2.4. Data collection and analysis Two reviewers independently extracted data and assessed methodological quality of included studies. Disagreements were resolved by consultations with the third reviewer. Data extracted included the key reporting characteristics (epidemiological and descriptive Characteristics) as well as the methodological quality assessment items from the AMSTAR checklists. Here we summarized the epidemiological and descriptive characteristics in two subsets (Table 1 and 2). We classified the included Cochrane reviews by their topic categories (e.g., Implementation Strategies) and focused areas (e.g., Hospital Management). Microsoft Excel 2003 (http://office.microsoft.com/zhcn/) was applied to design the data form and performed analyses for the collection data. Data was summarized by descriptive statistics (frequency, median, interquartile range [IQR]). AMSTAR checklists [11] were used to assess the quality of included Cochrane reviews. It assessed the degree to which review methods avoided bias by evaluating the methods against 11 distinct criteria. The 11 criteria and the way they work were described in Table 3. Every AMSTAR criterion could be specified three levels of quality: Yes (clearly done), Unclear (cannot answer or not applicable) and No (clearly not done), based on the published reviews report. Apparently, a review that adequately met all of the 11 criteria was considered to be a review of the highest quality. In this way, if a criterion was rated “Yes”, it scoring 1, the included reviews were rated as follows: • High quality: scores 8–11; • Medium quality: scores 4–7; • Low quality: scores 0–3. 3. Results 3.1. Search result Our search identified 900 publications (Fig. 1). Initial screening by title and abstract excluded 76 duplicate records and 628 records irrelevant to health policy. A further screening by full text excluded 97 articles, including 13 Protocols and 2 Withdrawn Cochrane reviews. Finally, 99 Cochrane reviews about health policy research were included, accounting for about 1.98% of the total Cochrane reviews.
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
Table 1 Epidemiology of Cochrane reviews on health policy.
900 citations retrieved by search :
• Topic “Effective practice/health systems”: n=125
Category
Characteristic
Number (%), of n = 99
Year of publication
2008 2009 2010 2011 2012 2013, March 2–3 4–5 6–7 8–9 10–12 UK Canada Australia USA South Africa Norway Italy Uganda Netherlands Ireland Denmark Switzerland Sri Lanka Spain Slovenia Pakistan Nigeria Lebanon Kenya Japan Germany China Chile Bangladesh Implementation Strategies Financial Arrangements Governance Arrangements Public Health Theoretical Exploration Hospital Management Medical Insurance Pharmaceutical Policy Community Health Rural Health
4 (4.0) 34 (34.3) 15 (15.2) 18 (18.2) 21 (21.2) 7 (7.1) 23 (23.2) 33 (33.4) 24 (14.2) 12 (12.2) 7 (7.0) 24 (24.2) 17 (17.2) 12 (12.1) 8 (8.1) 5 (5.1) 5 (5.1) 5 (5.1) 3 (3.0) 3 (3.0) 2 (2.0) 2 (2.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 72 (72.7) 15 (15.2) 12 (12.1) 34 (34.3) 18 (18.2) 17 (17.2) 12 (12.1) 9 (9.1) 7 (7.1) 2 (2.0)
Number of authors
Country of corresponding author
Topic categoriesa
Researched areasb
3
a According to the browse topics on the home page of Cochrane Library (http://www.thecochranelibrary.com/). b Based on the content of Cochrane Reviews.
3.2. Epidemiological characteristics The 99 Cochrane reviews were published online between 2008 and 2013, of which 4 reviews were published in 2008, 34 in 2009, 15 in 2010, 18 in 2011, 21 in 2012 and 7 in 2013. All reviews included 2 or more authors and their corresponding authors were from 24 countries, and most (88.9%) were from 15 developed countries, the corresponding authors from the four countries (UK, Canada, Australia and USA) accounted for 61.6%, and only 11.1% were from 9 non-developed countries (Uganda, Sri Lanka, Pakistan, Nigeria, Lebanon, Kenya, China, Chile and Bangladesh). According to the “browse topics” of Cochrane Library, the reviews were classified into Implementation Strategies (72.7%), Financial Arrangements (15.2%) and Governance Arrangements (12.1%). Nearly one-third (34.3%) of the
• Advanced Search: n=775 704 excluded : Titles and abstracts screened
• Health policy irrelevant: n=628 • Duplication: n=76
196 potentially included studies and full texts retrieved
Full texts screened
97 excluded : • Health policy irrelevant: n=82 • Protocols : n=13 • Withdrawn : n=2
99 Cochrane Reviews included in our study Data abstraction Fig. 1. Flow diagram illustrating the Cochrane Reviews articles identified, included and excluded.
reviews reported Public Health and only a small percentage (2.0%) reported Rural Health (Table 1). 3.3. Descriptive characteristics All reviews included a median of 5 authors (IQR: 4–7), they were all SRs with 14 (14.1%) being New Searches, 6 (6.1%) Conclusions Changed, 3 (3.0%) Comment, 2 (2.0%) Overview and 1 (1.0%) Methodology also. Nearly half (48, 48.5%) of the reviews reported being the updates of previously completed reviews, but the updates on Medical Insurance were only a quarter (3, 25.0%). Most of the first publications were reported 5 years ago, which accounted for 41.4% (41). The reviews included 1742 studies in total with a median of 9 studies (IQR: 3–24), although the median varied considerably by review category. For topic categories, Financial Arrangements reviews included fewer studies (total = 118; median = 7); for focused areas, reviews on Rural Health were the smallest and included fewer studies (total = 9; median = 5), and those reviews on Public Health were the largest by far, they included 589 studies in total (median = 11). Almost all of the reviews (93, 93.9%) considered the RCTs as included studies, 60 (60.6%) reviews considered the CCTs, 56 (56.6%) considered the CABs and 55 (55.6%) considered the ITSs, and only 13 (13.1%) reviews considered other types of studies. Meta-analysis was conducted in 39 reviews only, accounting for 39.4% (Table 2). 3.4. Methodological Quality For all the included reviews, there was a range from 9.1% to 91.9% accorded with the AMSTAR checklist items clearly (Table 3). Most reviews were compliant with the following checklist items: • Provided an ‘a priori’ design (86.9%); • Provided a list of studies (included and excluded) (70.7%); • Provided the characteristics of the included studies (79.8%);
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
Characteristica Overall, n = 99
Implementation, n = 72
Financial arrangements, n = 15
Governance arrangements, n = 12
Public Health n = 34
Theoretical Exploration n = 18
Hospital Management n = 17
Medical Insurance n = 12
Pharmaceutical Community Rural Health n = 7 Health Policy n = 9 n=2
72 (100) 12 (16.7) 6 (8.3) 3 (4.2) 1 (1.4) 1 (1.4) 5 (4–7) 36 (50.0) 5 (6.9) 16 (22.2) 15 (20.8) 1317 11 (4–30) 68 (94.4) 45 (62.5) 37 (51.4) 35 (48.6) 7 (9.2) 32 (44.4) 20 (27.8) 52 (72.2) 0 (0.0)
15 (100) 2 (13.3) 0 (0.0) 0 (0.0) 1 (6.7) 0 (0.0) 5 (3–7) 7 (46.7) 2 (13.3) 3 (20.0) 2 (13.3) 118 7 (4–11) 14 (93.3) 7 (46.7) 12 (80.0) 13 (86.7) 4 (26.7) 2 (13.3) 4 (26.7) 11 (73.3) 0 (0.0)
12 (100) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 5 (4–6) 5 (41.7) 0 (0.0) 4 (33.3) 1 (8.3) 253 6 (2–25) 11 (91.7) 8 (66.7) 7 (58.3) 7 (58.3) 2 (16.7) 5 (41.7) 3 (25.0) 9 (75.0) 0 (0.0)
34 (100) 5 (14.7) 2 (8.3) 3 (8.8) 0 (0.0) 0 (0.0) 4 (3–7) 14 (41.2) 1 (2.9) 9 (26.5) 4 (11.8) 589 11 (4–20) 31 (91.2) 18 (52.9) 11 (32.4) 10 (29.4) 4 (11.8) 18 (52.9) 8 (23.5) 26 (76.5) 0 (0.0)
18 (100) 3 (16.7) 1 (5.6) 0 (0.0) 0 (0.0) 1 (5.6) 7 (4–9) 10 (55.6) 2 (11.1) 2 (11.1) 6 (33.3) 406 9 (5–34) 17 (94.4) 10 (55.6) 11 (61.1) 10 (55.6) 2 (11.1) 7 (38.9) 5 (27.8) 13 (72.2) 0 (0.0)
17 (100) 1 (5.9) 1 (5.9) 0 (0.0) 0 (0.0) 0 (0.0) 6(4–8) 8 (47.1) 2 (11.8) 2 (11.8) 4 (23.5) 155 3 (2–10) 17 (100) 12 (70.6) 14 (82.4) 15 (88.2) 0 (0.0) 3 (17.6) 7 (41.2) 10 (58.8) 0 (0.0)
12 (100) 3 (25.0) 0 (0.0) 0 (0.0) 1 (8.3) 0 (0.0) 4(3–5) 3 (25.0) 1 (8.3) 1 (8.3) 1 (8.3) 64 5 (2–6) 11 (91.7) 5 (41.7) 9 (75.0) 10 (83.3) 1 (8.3) 3 (25.0) 3 (25.0) 9 (75.0) 0 (0.0)
9 (100) 0 (0.0) 0 (0.0) 0 (0.0) 1 (11.1) 0 (0.0) 6(4–8) 7 (77.8) 1 (11.1) 6 (66.7) 0 (0.0) 305 29 (12–38) 8 (88.9) 8 (88.9) 6 (66.7) 6 (66.7) 6 (66.7) 3 (33.3) 2 (22.2) 7 (77.8) 0 (0.0)
7 (100) 2 (28.6) 2 (28.6) 0 (0.0) 0 (0.0) 0 (0.0) 5(4–6) 5 (71.4) 0 (0.0) 2 (28.6) 3 (42.9) 214 24 (14–41) 7 (100) 5 (71.4) 3 (42.9) 2 (28.6) 0 (0.0) 4 (57.1) 2 (28.6) 5 (71.4) 0 (0.0)
2 (100) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 5(4–6) 1 (50.0) 0 (0.0) 1 (50.0) 0 (0.0) 9 5 (0–9) 2 (100) 2 (100) 2 (100) 2 (100) 0 (0.0) 1 (50.0) 0 (0.0) 2 (100) 0 (0.0)
IQR, interquartile range. RCTs, randomized controlled trials; CCTs, controlled clinical trials, non-randomized and quasi-randomized controlled trials; CBAs, controlled before and after studies; ITSs, interrupted time series analyses. a CSR, Cochrane Review; Ns, new search; Cc, conclusions changed; Cm, comment; Ov, overview; Me, methodology.
ARTICLE IN PRESS
99 (100) 14 (14.1) 6(6.1) 3 (3.0) 2 (3.0) 1 (1.0) 5 (4–7) 48 (48.5) 7 (7.1) 23 (23.2) 18 (18.2) 1742 9 (3–24) 93 (93.9) 60 (60.6) 56 (56.6) 55 (55.6) 13 (13.1) 39 (39.4) 27 (27.3) 72 (72.7) 0 (0.0)
Researched areas
L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
CSR, n (%) Ns, n (%) Cc, n (%) Cm, n (%) Ov, n (%) Me, n (%) Median (IQR) Number of authors Update of a previous review Yes, n (%) 1–5, n (%) Interval between the 6–10, n (%) first publication >10, n (%) n Number of included Median (IQR) studies RCTs, n (%) Types of included CCTs, n (%) studies CBAs, n (%) ITSs, n (%) Others, n (%) Meta-analysis Yes, n (%) High Methodological quality Medium Low Publishing status
Topic categories
G Model
Category
HEAP-3276; No. of Pages 8
4
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
Table 2 Descriptive Characteristics and methodological quality of Included Cochrane Reviews.
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
5
Table 3 AMSTAR assessment of methodological characteristics. AMSTAR criteria
Characteristic and scoring criteria
Number (%), of n = 99
1. Was an ‘a priori’ design provided?
Yes – The protocol was established before the conduct of the review. No – The authors stated that the protocol was not available or applicable. Unclear – There was no information about it. Yes – Both selection and extraction were performed independently by two or more people, or one people done duplication, and the method was reported for reaching consensus if disagreements arose. No – Both selection and extraction were performed by one person. Unclear – The selection or extraction was done by one person, or there was no information about it. Yes – At least two electronic databases were searched. It reported years and databases used, provided key words and/or MESH terms and the search strategies, given the supplement searches (by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found). No – Only one database searched or used no other sources. Unclear – There was partial or no information about it. Yes – The authors stated that their search restricted the publication type or language. No – The authors stated that their search regardless of the publication type and language and so on. Unclear – The search was restricted by publication type or language, or there was no information about it. Yes – A list or a narrative summary about the detailed information of included and excluded studies was provided. No – There was no information about it, or only the final results were presented. Unclear – There was partial information about included or excluded studies. Yes – Provided a table or a narrative summary about characteristics of the included studies, included characteristics on participants, interventions and outcomes. No – There was no information about it. Unclear – There was partial information about it. Yes – ‘A priori’ method of assessment was provided and performed appropriately. No – The scientific quality assessment of the included studies was not available or applicable. Unclear – There was partial information about it. Yes – Results of the scientific quality assessment were used in the analysis, conclusions and recommendations of the review. No – Results of the scientific quality assessment were not used. Unclear – Results of the scientific quality assessment were used in the analysis, conclusions or recommendations. Yes – There was appropriate description about the heterogeneity (e.g. Chi-square test, I2 ) and the combination methods (e.g. fixed-effect model). No – The heterogeneity and the combination methods were not available or applicable. Unclear – There was partial information about it. Yes – Publication bias was explicitly considered and assessed by Funnel plot or other methods. No – Publication bias was not assessed of some reason or no information about it. Unclear – There was partial information about it (e.g. discussed in conclusions). Yes – The conflict of interest and potential sources of support was clearly stated. No – There was no information about it. Can’t answer – The conflict of interest or potential sources of support was clearly stated.
86 (86.9)
2. Was there duplicate study selection and data extraction?
3. Was a comprehensive literature search performed?
4. Was the status of publication used as an inclusion criterion?
5. Was a list of studies (included and excluded) provided?
6. Were the characteristics of the included studies provided?
7. Was the scientific quality of the included studies assessed and documented?
8. Was the scientific quality of the included studies used appropriately in formulating conclusions?
9. Were the methods used to combine the findings of studies appropriate?
10. Was the likelihood of publication bias assessed?
11. Was the conflict of interest stated?
0 (0.0) 13 (13.1) 69 (69.7)
1 (1.0) 29 (29.3) 66 (66.7)
3 (3.0) 30 (30.3) 2 (2.0) 18 (18.2) 79 (79.8) 70 (70.7)
4 (4.0) 25 (25.3) 79 (79.8)
0 (0.0) 20 (20.2) 81 (81.8) 1 (1.0) 17 (17.2) 53 (53.6)
3 (3.0) 43 (43.4) 64 (64.6)
1 (1.0) 34 (34.3) 9 (9.1) 67 (67.7) 23 (23.2) 91 (91.9) 0 (0.0) 8 (8.1)
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
6
• Assessed and documented the scientific quality of the included studies (81.8%); • Stated the conflict of interest (91.9%). More than half of the reviews were compliant with the following checklist items: • Reported that there were duplicated study selection and data extraction (69.7%); • Performed a comprehensive literature search (66.7%); • Appropriately addressed the quality of included studies in formulating conclusions (53.6%); • Used appropriate methods to combine the findings of studies (64.6%). Few reviews reported the status of publication used as an inclusion criterion (2.0%) and assessed the likelihood of publication bias (9.1%). The score range of the reviews was from 4 to 10 examined by AMSTAR. Table 2 provides the distribution of methodology quality of the reviews. Of which, none was rated low methodological quality, and only 27 reviews (27.3%) were rated high methodological quality. Within the topic categories, the high-quality researches about Implementation Strategies accounting for 27.8%, Financial Arrangements 26.7% and Governance Arrangements 25.0%. And within the researched areas, Hospital Management had a larger proportion of high-quality researches, were about 41.2%. Rural Health had a smaller proportion of highquality researches, was about 0.0%. In addition, 72 reviews (72.7%) were rated medium methodological quality. 4. Discussion Our study has identified 99 Cochrane reviews published online in the CDSR. The value may be an underestimate of the total number of Cochrane reviews on health policy, as we examined only one topic and excluded 13 protocols and 2 withdrawn articles. But it breaks the belief that systematic review topics are not relevant to health systems policymaking, and that they cannot be found quickly, and are not available in formats and so on [16]. Similar with the clinical research, SRs of health policy research generally include the following steps also: defining the review question, developing criteria for including studies, searching for studies, selecting studies and collecting data, assessing risk of bias in included studies, analyzing and merging data, addressing reporting biases, presenting results and summary of findings’ tables, interpreting results and drawing conclusions, improving and updating the SRs, etc. [17]. Viewing from the published years and countries of corresponding authors, it can be seen that there are increasing reviews on health policy research published, particularly in the recent five years. Developed countries, which have more regular health system, such as UK, Canada, Australia and so on, have most participants. Especially, the members of Cochrane Review Groups [18] have made more attempts and exploration by SRs to address health policy issues in recent years. Our results show that Cochrane reviews of health policy research have collected thousands of original researches focusing on three subtopics, Implementation Strategies, Financial Arrangements and Governance
Arrangements, and mainly involved in Public Health, Theoretical Exploration, Hospital Management, Medical Insurance, Pharmaceutical Policy, Community Health and Rural Health. However, there are fewer policy researches for Rural Health, and none for Health Care Reform, Health Equity and other areas, although the Alma Ata Declaration of WHO [19] and the United Nations Millennium Development Goals [20] have been proposed and working to improve these issues. Nearly half of the reviews are reported as updates of previously completed reviews, the ratio is higher than nonCochrane Reviews [21], although the frequency varies from the updates. Most of them have updated the number of included studies, some updated methods, and some even changed the conclusions. As the updates can sum up the new research evidence and provide the new findings, and the results from reviews are most useful when they are up to date [22], updating systematic reviews should be given more concern in the coming years. Similar to SRs of clinical studies, our results indicate that Cochrane reviews of health policy research are developing toward diversity. For the types of included studies, they are no longer confined to randomized controlled trials, instead, observational studies are also considered. Rockers et al. had proven this point and noted that most systematic reviews of health systems evidence constrained the designs of studies for synthesis to those defined by the Cochrane Collaboration’s Effective Practice and Organization of Care (EPOC) criteria, or a subset [23]. In spite of this, the meta-analysis is no longer the only way to synthesize the results and the qualitative analysis has been also proposed. Moreover, the reviews are no longer limited to addressing questions about effectiveness. For examples, Welch V, et al. had addressed question about the methodology of interventions [24], Flodgren et al. [25] and Ryan et al. [26] had done overview of some Systematic Reviews. Overall, our research indicates the methodological quality of Cochrane reviews on health policy is generally good evaluating by the AMSTAR. But we also see some mainly methodological bias. Few reviews assess the publication bias, despite some evidence for its existence and potential influence on the results of reviews [27]. Evaluating “publication bias” of the studies included in the SRs is an important preparatory work [28]. There are a number of approaches based on examination of the pattern of data are available to help assess publication bias [29]. The most popular of these is the funnel plot; all, however, have substantial limitations [29]. In such cases, we recommend that researchers fully take into account the factors contributing to or resulting in publication bias as well as taking sufficiently advantage of electronic journals and computer technology. In addition, that most of reviews restrict the publication status of their included studies, which may affect the comprehensiveness of evidence included in the reviews and also have an impact on the results of the reviews. We think the researchers should try to avoid it. On the other hand, the potential limitations of AMSTARbased quality assessments for SRs in health policy research should be more concerned. First, in terms of the quality assessment tool, AMSTAR is still a relatively new instrument. Although the instrument has good validity and
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
reliability, it is difficult to become mature and perfected in a short time. We need to continuously improve in practice. In other words, we need a wide range of usage to further test the validity, reliability and responsiveness of AMSTAR [12]. Second, the AMSTAR has only been tested on systematic reviews of randomized control trials evaluating treatment interventions [13]. In the area of health systems research, there have been many systematic reviews including observational studies [23]. Therefore, we must reconsider the usage of AMSTAR criteria for these SRs, and make some adjustments when it is necessary. At last, but not the least, as Systematic Review is a developing methodology, there are about 2500 new SRs published globally every year [21]. The methodological quality assessment tools are important for researchers to design, implement and evaluate the reviews in science. Therefore, it is necessary to regularly update and continue to improve the AMSTAR criteria. According to the above analysis as well as the experiences and impressions collected during the process, the authors would like to suggest three measures to improve the Cochrane reviews and non-Cochrane reviews on health policy research. Firstly, the important methodology components should be considered in future research, especially the assessment of publication bias which is strongly recommended. Secondly, as far as method allows, try to apply SRs in other aspects of health policy research as much as possible. Thirdly, underdeveloped countries should introduce the SRs into health policy research as soon as possible. Our study acknowledges the following. It is limited in that we included just one browse topic from a single database. However, it is unlikely to affect the representative of our results, as the CDSR is the leading resource for systematic reviews in health care [30] and the topic retrieval and advanced search was used. While the included reviews only relate to health policy research, our results concerning the other areas of health care should be interpreted cautiously. In addition, our analysis relied on what the authors reported. It is possible that the authors conducted more but omitted important details from their reports, or the peerreview process resulted in the removal of key information our review might seek for. Finally, this review has not provided detailed information of several characteristics, such as the methods used to assess scientific quality, and more detailed information will be forthcoming in additional reports. And our study has not assessed the reporting quality, as the Cochrane Collaboration has a strict set of policies and guidance for reporting Cochrane Review [31].
5. Conclusions Our intent here is to provide a broad overview on the reporting characteristics and the methodological quality of the Cochrane reviews of health policy research for health policy researchers and makers. Although many Cochrane reviews on health policy research have been published, the quantity and quality of the evidence should be improved, and we should pay more attention to the blank of this field, especially the Financial Arrangements and the Governance
7
Arrangements involving Rural Health and other health policy areas, like Health Care Reform, Health Equity etc. And the reliability of AMSTAR needs to be tested more widely in this field. Disclosure statement No competing financial interests exist. Acknowledgments We thank the year 2013 Cochrane Colloquium has chosen the abstract of this review as a poster to communicate. We thank the library of Lanzhou University for database accessing and acquiring full texts. References [1] Niessen LW, Grijseels EW, Rutten FF. The evidence-based approach in health policy and health care delivery. Social Science & Medicine 2000;51(September (6)):859–69. [2] Guidelines for WHO guidelines Geneva, Switzerland: World Health Organization Global Programme on Evidence for Health Policy; 2003. Available from: http://apps.who.int/iris/bitstream/10665/ 68925/1/EIP GPE EQC 2003 1.pdf [3] First Global Symposium on Health Systems Research (HSR); 2010. Available from: http://healthsystemsresearch.org/hsr 2010/ [4] Huw TO, Davies SMN, Peter C, Smith. What works? Evidence-based policy and practice in public services. UK: The Policy Press; 2000. [5] Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet 2004;364(October–November (9445)):1615–21. [6] Lavis JN, Davies HT, Gruen RL, Walshe K, Farqubar CM. Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Health Policy 2006;1(January (2)):21–33. [7] Gruen RL, Morris PS, McDonald EL, Bailie RS. Making systematic reviews more useful for policy-makers. Bulletin of the World Health Organization 2005;83(June (6)):480. [8] Increasing access to health workers in remote and rural areas through improved retention: global policy recommendations. Geneva: World Health Organization; 2010. Available from: http:// www.who.int/hrh/retention/guidelines/en/ [9] Cochrane Reviews. The Cochrane Collaboration. Available from: http://www.cochrane.org/cochrane-reviews [10] About The Cochrane Library. The Cochrane Collaboration. Available from: http://www.cochrane.org/cochrane-reviews/aboutcochrane-library [11] Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 2007;7:10. [12] Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2007;2(12):e1350. [13] Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology 2009;62(October (10)):1013–20. [14] Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidencebased health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dentistry Journal 2010;4:84–91. [15] Bessa-Nogueira RV, Vasconcelos BC, Niederman R. The methodological quality of systematic reviews comparing temporomandibular joint disorder surgical and non-surgical treatment. BMC Oral Health 2008;8:27. [16] Moat KA, Lavis JN, Wilson MG, Rottingen JA, Barnighausen T. Twelve myths about systematic reviews for health system policymaking rebutted. Journal of Health Services Research & Policy 2013;18(January (1)):44–50.
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
[17] Tarsilla M. Cochrane handbook for systematic reviews of interventions. Journal of Multidisciplinary Evaluation 2010;6(14):142–8. [18] Allen C, Richmond K. The Cochrane Collaboration: international activity within Cochrane Review Groups in the first decade of the twenty-first century. Journal of Evidence-Based Medicine 2011; 4(February (1)):2–7. [19] WHO, editor. Alma Ata Declaration. Geneva: World Health Organization; 1978. [20] U Nations, editor. United Nations Millennium Declaration. United Nations The General Assembly; 2000. [21] Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007;4(March (3)):e78. [22] Moher D, Tsertsvadze A. Systematic reviews: when is an update an update? Lancet 2006;367(March (9514)):881–3. [23] Rockers PC, Feigl AB, Røttingen J-A, Fretheim A, Ferranti D, Lavis JN, et al. Study-design selection criteria in systematic reviews of effectiveness of health systems interventions and reforms: a meta-review. Health Policy 2012;104(3):206–14. [24] Welch V, Tugwell P, Petticrew M, Montigny J, Ueffing E, Kristjansson B, et al. How effects on health equity are assessed in systematic reviews of interventions. Cochrane Database of Systematic Reviews 2010;12:MR000028.
[25] Flodgren G, Eccles MP, Shepperd S, Scott A, Parmelli E, Beyer FR. An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database of Systematic Reviews 2011;7:CD009255. [26] Ryan R, Santesso N, Hill S, Lowe D, Kaufman C, Grimshaw J. Consumer-oriented interventions for evidence-based prescribing and medicines use: an overview of systematic reviews. Cochrane Database of Systematic Reviews 2011;5:CD007768. [27] Hannah R, Rothstein AJS. Michael Borenstein Publication bias in meta-analysis: prevention, assessment, and adjustments: Wiley.com; 2005. [28] Cooper H, Hedges LV, Valentine JC. The handbook of research synthesis and meta-analysis. Russell Sage Foundation; 2009. [29] Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines. 5. Rating the quality of evidence – publication bias. Journal of Clinical Epidemiology 2011;64(12):1277–82. [30] Cochrane Database of Systematic Reviews. The Cochrane Library. from: http://www.thecochranelibrary.com/view/0/ Available AboutTheCochraneLibrary.html [31] Cochrane handbook for systematic reviews of interventions. Chichester: Wiley-Blackwell; 2011.
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
ARTICLE IN PRESS
HEAP-3276; No. of Pages 8
Health Policy xxx (2014) xxx–xxx
Contents lists available at ScienceDirect
Health Policy journal homepage: www.elsevier.com/locate/healthpol
Review
The reporting characteristics and methodological quality of Cochrane reviews about health policy research Li Xiu-xia a,b , Zheng Ya c , Chen Yao-long a,b , Yang Ke-hu a,b,∗ , Zhang Zong-jiu a,∗ a b c
Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou 730000, China Key Laboratory of Evidence Based Medicine and Knowledge Translation of Gansu Province, Lanzhou 730000, China School of Public Health, Lanzhou University, Lanzhou 730000, China
a r t i c l e
i n f o
Article history: Received 19 August 2013 Received in revised form 2 September 2014 Accepted 4 September 2014 Keywords: Health policy research Cochrane reviews Epidemiological characteristics Descriptive characteristics Methodological quality
a b s t r a c t The systematic review has increasingly become a popular tool for researching health policy. However, due to the complexity and diversity in the health policy research, it has also encountered more challenges. We set out the Cochrane reviews on health policy research as a representative to provide the first examination of epidemiological and descriptive characteristics as well as the compliance of methodological quality with the AMSTAR. 99 reviews were included by inclusion criteria, 73% of which were Implementation Strategies, 15% were Financial Arrangements and 12% were Governance Arrangements; involved Public Health (34%), Theoretical Exploration (18%), Hospital Management (17%), Medical Insurance (12%), Pharmaceutical Policy (9%), Community Health (7%) and Rural Health (2%). Only 39% conducted meta-analysis, and 49% reported being updates, and none was rated low methodological quality. Our research reveals that the quantity and quality of the evidence should be improved, especially Financial Arrangements and Governance Arrangements involved Rural Health, Health Care Reform and Health Equity, etc. And the reliability of AMSTAR needs to be tested in larger range in this field. © 2014 Elsevier Ireland Ltd. All rights reserved.
1. Introduction There has been growing international interest in the evidence-based health policy-making that scientific evidence should play an important role in the policy formulation, implementation, and assessment. During the 1990s, evidence-based approaches have become prominent on the national and international agendas for health policy and health systems research [1]. The World Health Organization has been vigorously supporting the process
∗ Corresponding authors at: Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, No. 199, Donggang West Road, Chengguan District, Lanzhou, Gansu 730000, China. Tel.: +86 13993694673. E-mail addresses: [email protected] (Y. Ke-hu), [email protected] (Z. Zong-jiu).
of contextualizing evidence and translating it into policy, especially in the developing countries [2]. The First Global Symposium on Health Systems Research (HSR) – Science to Accelerate Universal Health Coverage has been announced by the World Health Organization and partners, and aiming to improve the scientific evidence needed by health policymakers and practitioners to inform their decisions related to accelerating universal health coverage [3]. Systematic review (SR), the important tool for EvidenceBased Medicine (EBM), can inform healthcare management and policy making levels by providing research-based responses to important questions about health systems [4–6]. In recent years, more and more SRs on healthsystems research have been published in many leading journals such as The Lancet (Lancet) and the World Health Organization Bulletin, etc. [7]. An increasing number of health policy makers and researchers have used SRs to
http://dx.doi.org/10.1016/j.healthpol.2014.09.002 0168-8510/© 2014 Elsevier Ireland Ltd. All rights reserved.
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
2
synthesize research evidence to deal with health issues at global and national levels. World Health Organization Global Policy Recommendations “Increasing access to health workers in remote and rural areas through improved retention” (2010) [8] is a representative example. However, due to the complexity and diversity of the health policy research, the methodology of SRs has also been facing challenges in this field. Cochrane reviews are the SRs of primary research on human health care and health policy, and are internationally recognized as the highest standard in evidence-based health care [9]. So far, over 5000 Cochrane Reviews and 2000 protocols for Cochrane reviews are published online in the Cochrane Database of Systematic Reviews (CDSR), which is the database of Cochrane reviews in The Cochrane Library (www.thecochranelibrary.com) and the 2011 ISI Impact Factor is 5.912 [10]. The Cochrane reviews are classified into 31 topics in the CDSR, and the topic “Effective practice/health systems” follows closely health systems research which involves the evidence on health policy research. Newer assessment tool – Assessment of Multiple Systematic Reviews (AMSTAR) has been to assess the methodological quality of SRs, which is built upon previous tools, empirical evidence and expert consensus [11]. The AMSTAR approach consists of 11 items and is featured by good content validity, wide acceptance, recognized reliability and reproducibility [11–14]. It has been considered the best way to assess methodological quality of SRs by the Canadian Agency for Drugs and Technologies in Health [15]. The purpose of this review is to examine the epidemiological and descriptive characteristics, as well as the methodological quality of Cochrane reviews on health policy research published in the CDSR, which serves as a reference for health policy makers and researchers. 2. Materials and methods 2.1. Inclusion criteria We included in this review all systematic reviews indexed in CDSR that met the following four criteria: First, the systematic review synthesized evidence on a research question about health policy, such as the policy formulation, implementation or evaluation, or the methodology for health policy research, etc. Second, the review synthesized evidence in one of three categories of health policy research: Implementation Strategies, Financial Arrangements and Governance Arrangements. Third, the review had to be available in the database on 31 March 2013, when we downloaded all studies for selection and extraction. Fourth, the review was published in English. But the protocols for Cochrane reviews and Withdrawn Cochrane reviews were excluded. 2.2. Search strategy We searched the CDSR by the whole topic on “Effective practice/health systems”, and retrieved the Advanced Search by “Title, Abstract and Keywords” as a supplement
in March 2013. The search terms included health, policy and strategy. 2.3. Screening According to predetermined inclusion criteria, two reviewers independently screened all of the search results by title and abstract, and subsequently retrieved and screened the full text of potentially included studies (if one or both reviewers thought it was potentially relevant). Disagreements were resolved by discussion. 2.4. Data collection and analysis Two reviewers independently extracted data and assessed methodological quality of included studies. Disagreements were resolved by consultations with the third reviewer. Data extracted included the key reporting characteristics (epidemiological and descriptive Characteristics) as well as the methodological quality assessment items from the AMSTAR checklists. Here we summarized the epidemiological and descriptive characteristics in two subsets (Table 1 and 2). We classified the included Cochrane reviews by their topic categories (e.g., Implementation Strategies) and focused areas (e.g., Hospital Management). Microsoft Excel 2003 (http://office.microsoft.com/zhcn/) was applied to design the data form and performed analyses for the collection data. Data was summarized by descriptive statistics (frequency, median, interquartile range [IQR]). AMSTAR checklists [11] were used to assess the quality of included Cochrane reviews. It assessed the degree to which review methods avoided bias by evaluating the methods against 11 distinct criteria. The 11 criteria and the way they work were described in Table 3. Every AMSTAR criterion could be specified three levels of quality: Yes (clearly done), Unclear (cannot answer or not applicable) and No (clearly not done), based on the published reviews report. Apparently, a review that adequately met all of the 11 criteria was considered to be a review of the highest quality. In this way, if a criterion was rated “Yes”, it scoring 1, the included reviews were rated as follows: • High quality: scores 8–11; • Medium quality: scores 4–7; • Low quality: scores 0–3. 3. Results 3.1. Search result Our search identified 900 publications (Fig. 1). Initial screening by title and abstract excluded 76 duplicate records and 628 records irrelevant to health policy. A further screening by full text excluded 97 articles, including 13 Protocols and 2 Withdrawn Cochrane reviews. Finally, 99 Cochrane reviews about health policy research were included, accounting for about 1.98% of the total Cochrane reviews.
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
Table 1 Epidemiology of Cochrane reviews on health policy.
900 citations retrieved by search :
• Topic “Effective practice/health systems”: n=125
Category
Characteristic
Number (%), of n = 99
Year of publication
2008 2009 2010 2011 2012 2013, March 2–3 4–5 6–7 8–9 10–12 UK Canada Australia USA South Africa Norway Italy Uganda Netherlands Ireland Denmark Switzerland Sri Lanka Spain Slovenia Pakistan Nigeria Lebanon Kenya Japan Germany China Chile Bangladesh Implementation Strategies Financial Arrangements Governance Arrangements Public Health Theoretical Exploration Hospital Management Medical Insurance Pharmaceutical Policy Community Health Rural Health
4 (4.0) 34 (34.3) 15 (15.2) 18 (18.2) 21 (21.2) 7 (7.1) 23 (23.2) 33 (33.4) 24 (14.2) 12 (12.2) 7 (7.0) 24 (24.2) 17 (17.2) 12 (12.1) 8 (8.1) 5 (5.1) 5 (5.1) 5 (5.1) 3 (3.0) 3 (3.0) 2 (2.0) 2 (2.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 1 (1.0) 72 (72.7) 15 (15.2) 12 (12.1) 34 (34.3) 18 (18.2) 17 (17.2) 12 (12.1) 9 (9.1) 7 (7.1) 2 (2.0)
Number of authors
Country of corresponding author
Topic categoriesa
Researched areasb
3
a According to the browse topics on the home page of Cochrane Library (http://www.thecochranelibrary.com/). b Based on the content of Cochrane Reviews.
3.2. Epidemiological characteristics The 99 Cochrane reviews were published online between 2008 and 2013, of which 4 reviews were published in 2008, 34 in 2009, 15 in 2010, 18 in 2011, 21 in 2012 and 7 in 2013. All reviews included 2 or more authors and their corresponding authors were from 24 countries, and most (88.9%) were from 15 developed countries, the corresponding authors from the four countries (UK, Canada, Australia and USA) accounted for 61.6%, and only 11.1% were from 9 non-developed countries (Uganda, Sri Lanka, Pakistan, Nigeria, Lebanon, Kenya, China, Chile and Bangladesh). According to the “browse topics” of Cochrane Library, the reviews were classified into Implementation Strategies (72.7%), Financial Arrangements (15.2%) and Governance Arrangements (12.1%). Nearly one-third (34.3%) of the
• Advanced Search: n=775 704 excluded : Titles and abstracts screened
• Health policy irrelevant: n=628 • Duplication: n=76
196 potentially included studies and full texts retrieved
Full texts screened
97 excluded : • Health policy irrelevant: n=82 • Protocols : n=13 • Withdrawn : n=2
99 Cochrane Reviews included in our study Data abstraction Fig. 1. Flow diagram illustrating the Cochrane Reviews articles identified, included and excluded.
reviews reported Public Health and only a small percentage (2.0%) reported Rural Health (Table 1). 3.3. Descriptive characteristics All reviews included a median of 5 authors (IQR: 4–7), they were all SRs with 14 (14.1%) being New Searches, 6 (6.1%) Conclusions Changed, 3 (3.0%) Comment, 2 (2.0%) Overview and 1 (1.0%) Methodology also. Nearly half (48, 48.5%) of the reviews reported being the updates of previously completed reviews, but the updates on Medical Insurance were only a quarter (3, 25.0%). Most of the first publications were reported 5 years ago, which accounted for 41.4% (41). The reviews included 1742 studies in total with a median of 9 studies (IQR: 3–24), although the median varied considerably by review category. For topic categories, Financial Arrangements reviews included fewer studies (total = 118; median = 7); for focused areas, reviews on Rural Health were the smallest and included fewer studies (total = 9; median = 5), and those reviews on Public Health were the largest by far, they included 589 studies in total (median = 11). Almost all of the reviews (93, 93.9%) considered the RCTs as included studies, 60 (60.6%) reviews considered the CCTs, 56 (56.6%) considered the CABs and 55 (55.6%) considered the ITSs, and only 13 (13.1%) reviews considered other types of studies. Meta-analysis was conducted in 39 reviews only, accounting for 39.4% (Table 2). 3.4. Methodological Quality For all the included reviews, there was a range from 9.1% to 91.9% accorded with the AMSTAR checklist items clearly (Table 3). Most reviews were compliant with the following checklist items: • Provided an ‘a priori’ design (86.9%); • Provided a list of studies (included and excluded) (70.7%); • Provided the characteristics of the included studies (79.8%);
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
Characteristica Overall, n = 99
Implementation, n = 72
Financial arrangements, n = 15
Governance arrangements, n = 12
Public Health n = 34
Theoretical Exploration n = 18
Hospital Management n = 17
Medical Insurance n = 12
Pharmaceutical Community Rural Health n = 7 Health Policy n = 9 n=2
72 (100) 12 (16.7) 6 (8.3) 3 (4.2) 1 (1.4) 1 (1.4) 5 (4–7) 36 (50.0) 5 (6.9) 16 (22.2) 15 (20.8) 1317 11 (4–30) 68 (94.4) 45 (62.5) 37 (51.4) 35 (48.6) 7 (9.2) 32 (44.4) 20 (27.8) 52 (72.2) 0 (0.0)
15 (100) 2 (13.3) 0 (0.0) 0 (0.0) 1 (6.7) 0 (0.0) 5 (3–7) 7 (46.7) 2 (13.3) 3 (20.0) 2 (13.3) 118 7 (4–11) 14 (93.3) 7 (46.7) 12 (80.0) 13 (86.7) 4 (26.7) 2 (13.3) 4 (26.7) 11 (73.3) 0 (0.0)
12 (100) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 5 (4–6) 5 (41.7) 0 (0.0) 4 (33.3) 1 (8.3) 253 6 (2–25) 11 (91.7) 8 (66.7) 7 (58.3) 7 (58.3) 2 (16.7) 5 (41.7) 3 (25.0) 9 (75.0) 0 (0.0)
34 (100) 5 (14.7) 2 (8.3) 3 (8.8) 0 (0.0) 0 (0.0) 4 (3–7) 14 (41.2) 1 (2.9) 9 (26.5) 4 (11.8) 589 11 (4–20) 31 (91.2) 18 (52.9) 11 (32.4) 10 (29.4) 4 (11.8) 18 (52.9) 8 (23.5) 26 (76.5) 0 (0.0)
18 (100) 3 (16.7) 1 (5.6) 0 (0.0) 0 (0.0) 1 (5.6) 7 (4–9) 10 (55.6) 2 (11.1) 2 (11.1) 6 (33.3) 406 9 (5–34) 17 (94.4) 10 (55.6) 11 (61.1) 10 (55.6) 2 (11.1) 7 (38.9) 5 (27.8) 13 (72.2) 0 (0.0)
17 (100) 1 (5.9) 1 (5.9) 0 (0.0) 0 (0.0) 0 (0.0) 6(4–8) 8 (47.1) 2 (11.8) 2 (11.8) 4 (23.5) 155 3 (2–10) 17 (100) 12 (70.6) 14 (82.4) 15 (88.2) 0 (0.0) 3 (17.6) 7 (41.2) 10 (58.8) 0 (0.0)
12 (100) 3 (25.0) 0 (0.0) 0 (0.0) 1 (8.3) 0 (0.0) 4(3–5) 3 (25.0) 1 (8.3) 1 (8.3) 1 (8.3) 64 5 (2–6) 11 (91.7) 5 (41.7) 9 (75.0) 10 (83.3) 1 (8.3) 3 (25.0) 3 (25.0) 9 (75.0) 0 (0.0)
9 (100) 0 (0.0) 0 (0.0) 0 (0.0) 1 (11.1) 0 (0.0) 6(4–8) 7 (77.8) 1 (11.1) 6 (66.7) 0 (0.0) 305 29 (12–38) 8 (88.9) 8 (88.9) 6 (66.7) 6 (66.7) 6 (66.7) 3 (33.3) 2 (22.2) 7 (77.8) 0 (0.0)
7 (100) 2 (28.6) 2 (28.6) 0 (0.0) 0 (0.0) 0 (0.0) 5(4–6) 5 (71.4) 0 (0.0) 2 (28.6) 3 (42.9) 214 24 (14–41) 7 (100) 5 (71.4) 3 (42.9) 2 (28.6) 0 (0.0) 4 (57.1) 2 (28.6) 5 (71.4) 0 (0.0)
2 (100) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 0 (0.0) 5(4–6) 1 (50.0) 0 (0.0) 1 (50.0) 0 (0.0) 9 5 (0–9) 2 (100) 2 (100) 2 (100) 2 (100) 0 (0.0) 1 (50.0) 0 (0.0) 2 (100) 0 (0.0)
IQR, interquartile range. RCTs, randomized controlled trials; CCTs, controlled clinical trials, non-randomized and quasi-randomized controlled trials; CBAs, controlled before and after studies; ITSs, interrupted time series analyses. a CSR, Cochrane Review; Ns, new search; Cc, conclusions changed; Cm, comment; Ov, overview; Me, methodology.
ARTICLE IN PRESS
99 (100) 14 (14.1) 6(6.1) 3 (3.0) 2 (3.0) 1 (1.0) 5 (4–7) 48 (48.5) 7 (7.1) 23 (23.2) 18 (18.2) 1742 9 (3–24) 93 (93.9) 60 (60.6) 56 (56.6) 55 (55.6) 13 (13.1) 39 (39.4) 27 (27.3) 72 (72.7) 0 (0.0)
Researched areas
L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
CSR, n (%) Ns, n (%) Cc, n (%) Cm, n (%) Ov, n (%) Me, n (%) Median (IQR) Number of authors Update of a previous review Yes, n (%) 1–5, n (%) Interval between the 6–10, n (%) first publication >10, n (%) n Number of included Median (IQR) studies RCTs, n (%) Types of included CCTs, n (%) studies CBAs, n (%) ITSs, n (%) Others, n (%) Meta-analysis Yes, n (%) High Methodological quality Medium Low Publishing status
Topic categories
G Model
Category
HEAP-3276; No. of Pages 8
4
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
Table 2 Descriptive Characteristics and methodological quality of Included Cochrane Reviews.
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
5
Table 3 AMSTAR assessment of methodological characteristics. AMSTAR criteria
Characteristic and scoring criteria
Number (%), of n = 99
1. Was an ‘a priori’ design provided?
Yes – The protocol was established before the conduct of the review. No – The authors stated that the protocol was not available or applicable. Unclear – There was no information about it. Yes – Both selection and extraction were performed independently by two or more people, or one people done duplication, and the method was reported for reaching consensus if disagreements arose. No – Both selection and extraction were performed by one person. Unclear – The selection or extraction was done by one person, or there was no information about it. Yes – At least two electronic databases were searched. It reported years and databases used, provided key words and/or MESH terms and the search strategies, given the supplement searches (by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found). No – Only one database searched or used no other sources. Unclear – There was partial or no information about it. Yes – The authors stated that their search restricted the publication type or language. No – The authors stated that their search regardless of the publication type and language and so on. Unclear – The search was restricted by publication type or language, or there was no information about it. Yes – A list or a narrative summary about the detailed information of included and excluded studies was provided. No – There was no information about it, or only the final results were presented. Unclear – There was partial information about included or excluded studies. Yes – Provided a table or a narrative summary about characteristics of the included studies, included characteristics on participants, interventions and outcomes. No – There was no information about it. Unclear – There was partial information about it. Yes – ‘A priori’ method of assessment was provided and performed appropriately. No – The scientific quality assessment of the included studies was not available or applicable. Unclear – There was partial information about it. Yes – Results of the scientific quality assessment were used in the analysis, conclusions and recommendations of the review. No – Results of the scientific quality assessment were not used. Unclear – Results of the scientific quality assessment were used in the analysis, conclusions or recommendations. Yes – There was appropriate description about the heterogeneity (e.g. Chi-square test, I2 ) and the combination methods (e.g. fixed-effect model). No – The heterogeneity and the combination methods were not available or applicable. Unclear – There was partial information about it. Yes – Publication bias was explicitly considered and assessed by Funnel plot or other methods. No – Publication bias was not assessed of some reason or no information about it. Unclear – There was partial information about it (e.g. discussed in conclusions). Yes – The conflict of interest and potential sources of support was clearly stated. No – There was no information about it. Can’t answer – The conflict of interest or potential sources of support was clearly stated.
86 (86.9)
2. Was there duplicate study selection and data extraction?
3. Was a comprehensive literature search performed?
4. Was the status of publication used as an inclusion criterion?
5. Was a list of studies (included and excluded) provided?
6. Were the characteristics of the included studies provided?
7. Was the scientific quality of the included studies assessed and documented?
8. Was the scientific quality of the included studies used appropriately in formulating conclusions?
9. Were the methods used to combine the findings of studies appropriate?
10. Was the likelihood of publication bias assessed?
11. Was the conflict of interest stated?
0 (0.0) 13 (13.1) 69 (69.7)
1 (1.0) 29 (29.3) 66 (66.7)
3 (3.0) 30 (30.3) 2 (2.0) 18 (18.2) 79 (79.8) 70 (70.7)
4 (4.0) 25 (25.3) 79 (79.8)
0 (0.0) 20 (20.2) 81 (81.8) 1 (1.0) 17 (17.2) 53 (53.6)
3 (3.0) 43 (43.4) 64 (64.6)
1 (1.0) 34 (34.3) 9 (9.1) 67 (67.7) 23 (23.2) 91 (91.9) 0 (0.0) 8 (8.1)
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
6
• Assessed and documented the scientific quality of the included studies (81.8%); • Stated the conflict of interest (91.9%). More than half of the reviews were compliant with the following checklist items: • Reported that there were duplicated study selection and data extraction (69.7%); • Performed a comprehensive literature search (66.7%); • Appropriately addressed the quality of included studies in formulating conclusions (53.6%); • Used appropriate methods to combine the findings of studies (64.6%). Few reviews reported the status of publication used as an inclusion criterion (2.0%) and assessed the likelihood of publication bias (9.1%). The score range of the reviews was from 4 to 10 examined by AMSTAR. Table 2 provides the distribution of methodology quality of the reviews. Of which, none was rated low methodological quality, and only 27 reviews (27.3%) were rated high methodological quality. Within the topic categories, the high-quality researches about Implementation Strategies accounting for 27.8%, Financial Arrangements 26.7% and Governance Arrangements 25.0%. And within the researched areas, Hospital Management had a larger proportion of high-quality researches, were about 41.2%. Rural Health had a smaller proportion of highquality researches, was about 0.0%. In addition, 72 reviews (72.7%) were rated medium methodological quality. 4. Discussion Our study has identified 99 Cochrane reviews published online in the CDSR. The value may be an underestimate of the total number of Cochrane reviews on health policy, as we examined only one topic and excluded 13 protocols and 2 withdrawn articles. But it breaks the belief that systematic review topics are not relevant to health systems policymaking, and that they cannot be found quickly, and are not available in formats and so on [16]. Similar with the clinical research, SRs of health policy research generally include the following steps also: defining the review question, developing criteria for including studies, searching for studies, selecting studies and collecting data, assessing risk of bias in included studies, analyzing and merging data, addressing reporting biases, presenting results and summary of findings’ tables, interpreting results and drawing conclusions, improving and updating the SRs, etc. [17]. Viewing from the published years and countries of corresponding authors, it can be seen that there are increasing reviews on health policy research published, particularly in the recent five years. Developed countries, which have more regular health system, such as UK, Canada, Australia and so on, have most participants. Especially, the members of Cochrane Review Groups [18] have made more attempts and exploration by SRs to address health policy issues in recent years. Our results show that Cochrane reviews of health policy research have collected thousands of original researches focusing on three subtopics, Implementation Strategies, Financial Arrangements and Governance
Arrangements, and mainly involved in Public Health, Theoretical Exploration, Hospital Management, Medical Insurance, Pharmaceutical Policy, Community Health and Rural Health. However, there are fewer policy researches for Rural Health, and none for Health Care Reform, Health Equity and other areas, although the Alma Ata Declaration of WHO [19] and the United Nations Millennium Development Goals [20] have been proposed and working to improve these issues. Nearly half of the reviews are reported as updates of previously completed reviews, the ratio is higher than nonCochrane Reviews [21], although the frequency varies from the updates. Most of them have updated the number of included studies, some updated methods, and some even changed the conclusions. As the updates can sum up the new research evidence and provide the new findings, and the results from reviews are most useful when they are up to date [22], updating systematic reviews should be given more concern in the coming years. Similar to SRs of clinical studies, our results indicate that Cochrane reviews of health policy research are developing toward diversity. For the types of included studies, they are no longer confined to randomized controlled trials, instead, observational studies are also considered. Rockers et al. had proven this point and noted that most systematic reviews of health systems evidence constrained the designs of studies for synthesis to those defined by the Cochrane Collaboration’s Effective Practice and Organization of Care (EPOC) criteria, or a subset [23]. In spite of this, the meta-analysis is no longer the only way to synthesize the results and the qualitative analysis has been also proposed. Moreover, the reviews are no longer limited to addressing questions about effectiveness. For examples, Welch V, et al. had addressed question about the methodology of interventions [24], Flodgren et al. [25] and Ryan et al. [26] had done overview of some Systematic Reviews. Overall, our research indicates the methodological quality of Cochrane reviews on health policy is generally good evaluating by the AMSTAR. But we also see some mainly methodological bias. Few reviews assess the publication bias, despite some evidence for its existence and potential influence on the results of reviews [27]. Evaluating “publication bias” of the studies included in the SRs is an important preparatory work [28]. There are a number of approaches based on examination of the pattern of data are available to help assess publication bias [29]. The most popular of these is the funnel plot; all, however, have substantial limitations [29]. In such cases, we recommend that researchers fully take into account the factors contributing to or resulting in publication bias as well as taking sufficiently advantage of electronic journals and computer technology. In addition, that most of reviews restrict the publication status of their included studies, which may affect the comprehensiveness of evidence included in the reviews and also have an impact on the results of the reviews. We think the researchers should try to avoid it. On the other hand, the potential limitations of AMSTARbased quality assessments for SRs in health policy research should be more concerned. First, in terms of the quality assessment tool, AMSTAR is still a relatively new instrument. Although the instrument has good validity and
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
reliability, it is difficult to become mature and perfected in a short time. We need to continuously improve in practice. In other words, we need a wide range of usage to further test the validity, reliability and responsiveness of AMSTAR [12]. Second, the AMSTAR has only been tested on systematic reviews of randomized control trials evaluating treatment interventions [13]. In the area of health systems research, there have been many systematic reviews including observational studies [23]. Therefore, we must reconsider the usage of AMSTAR criteria for these SRs, and make some adjustments when it is necessary. At last, but not the least, as Systematic Review is a developing methodology, there are about 2500 new SRs published globally every year [21]. The methodological quality assessment tools are important for researchers to design, implement and evaluate the reviews in science. Therefore, it is necessary to regularly update and continue to improve the AMSTAR criteria. According to the above analysis as well as the experiences and impressions collected during the process, the authors would like to suggest three measures to improve the Cochrane reviews and non-Cochrane reviews on health policy research. Firstly, the important methodology components should be considered in future research, especially the assessment of publication bias which is strongly recommended. Secondly, as far as method allows, try to apply SRs in other aspects of health policy research as much as possible. Thirdly, underdeveloped countries should introduce the SRs into health policy research as soon as possible. Our study acknowledges the following. It is limited in that we included just one browse topic from a single database. However, it is unlikely to affect the representative of our results, as the CDSR is the leading resource for systematic reviews in health care [30] and the topic retrieval and advanced search was used. While the included reviews only relate to health policy research, our results concerning the other areas of health care should be interpreted cautiously. In addition, our analysis relied on what the authors reported. It is possible that the authors conducted more but omitted important details from their reports, or the peerreview process resulted in the removal of key information our review might seek for. Finally, this review has not provided detailed information of several characteristics, such as the methods used to assess scientific quality, and more detailed information will be forthcoming in additional reports. And our study has not assessed the reporting quality, as the Cochrane Collaboration has a strict set of policies and guidance for reporting Cochrane Review [31].
5. Conclusions Our intent here is to provide a broad overview on the reporting characteristics and the methodological quality of the Cochrane reviews of health policy research for health policy researchers and makers. Although many Cochrane reviews on health policy research have been published, the quantity and quality of the evidence should be improved, and we should pay more attention to the blank of this field, especially the Financial Arrangements and the Governance
7
Arrangements involving Rural Health and other health policy areas, like Health Care Reform, Health Equity etc. And the reliability of AMSTAR needs to be tested more widely in this field. Disclosure statement No competing financial interests exist. Acknowledgments We thank the year 2013 Cochrane Colloquium has chosen the abstract of this review as a poster to communicate. We thank the library of Lanzhou University for database accessing and acquiring full texts. References [1] Niessen LW, Grijseels EW, Rutten FF. The evidence-based approach in health policy and health care delivery. Social Science & Medicine 2000;51(September (6)):859–69. [2] Guidelines for WHO guidelines Geneva, Switzerland: World Health Organization Global Programme on Evidence for Health Policy; 2003. Available from: http://apps.who.int/iris/bitstream/10665/ 68925/1/EIP GPE EQC 2003 1.pdf [3] First Global Symposium on Health Systems Research (HSR); 2010. Available from: http://healthsystemsresearch.org/hsr 2010/ [4] Huw TO, Davies SMN, Peter C, Smith. What works? Evidence-based policy and practice in public services. UK: The Policy Press; 2000. [5] Lavis JN, Posada FB, Haines A, Osei E. Use of research to inform public policymaking. Lancet 2004;364(October–November (9445)):1615–21. [6] Lavis JN, Davies HT, Gruen RL, Walshe K, Farqubar CM. Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Health Policy 2006;1(January (2)):21–33. [7] Gruen RL, Morris PS, McDonald EL, Bailie RS. Making systematic reviews more useful for policy-makers. Bulletin of the World Health Organization 2005;83(June (6)):480. [8] Increasing access to health workers in remote and rural areas through improved retention: global policy recommendations. Geneva: World Health Organization; 2010. Available from: http:// www.who.int/hrh/retention/guidelines/en/ [9] Cochrane Reviews. The Cochrane Collaboration. Available from: http://www.cochrane.org/cochrane-reviews [10] About The Cochrane Library. The Cochrane Collaboration. Available from: http://www.cochrane.org/cochrane-reviews/aboutcochrane-library [11] Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 2007;7:10. [12] Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE 2007;2(12):e1350. [13] Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology 2009;62(October (10)):1013–20. [14] Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidencebased health care: validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dentistry Journal 2010;4:84–91. [15] Bessa-Nogueira RV, Vasconcelos BC, Niederman R. The methodological quality of systematic reviews comparing temporomandibular joint disorder surgical and non-surgical treatment. BMC Oral Health 2008;8:27. [16] Moat KA, Lavis JN, Wilson MG, Rottingen JA, Barnighausen T. Twelve myths about systematic reviews for health system policymaking rebutted. Journal of Health Services Research & Policy 2013;18(January (1)):44–50.
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002
G Model HEAP-3276; No. of Pages 8
8
ARTICLE IN PRESS L. Xiu-xia et al. / Health Policy xxx (2014) xxx–xxx
[17] Tarsilla M. Cochrane handbook for systematic reviews of interventions. Journal of Multidisciplinary Evaluation 2010;6(14):142–8. [18] Allen C, Richmond K. The Cochrane Collaboration: international activity within Cochrane Review Groups in the first decade of the twenty-first century. Journal of Evidence-Based Medicine 2011; 4(February (1)):2–7. [19] WHO, editor. Alma Ata Declaration. Geneva: World Health Organization; 1978. [20] U Nations, editor. United Nations Millennium Declaration. United Nations The General Assembly; 2000. [21] Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007;4(March (3)):e78. [22] Moher D, Tsertsvadze A. Systematic reviews: when is an update an update? Lancet 2006;367(March (9514)):881–3. [23] Rockers PC, Feigl AB, Røttingen J-A, Fretheim A, Ferranti D, Lavis JN, et al. Study-design selection criteria in systematic reviews of effectiveness of health systems interventions and reforms: a meta-review. Health Policy 2012;104(3):206–14. [24] Welch V, Tugwell P, Petticrew M, Montigny J, Ueffing E, Kristjansson B, et al. How effects on health equity are assessed in systematic reviews of interventions. Cochrane Database of Systematic Reviews 2010;12:MR000028.
[25] Flodgren G, Eccles MP, Shepperd S, Scott A, Parmelli E, Beyer FR. An overview of reviews evaluating the effectiveness of financial incentives in changing healthcare professional behaviours and patient outcomes. Cochrane Database of Systematic Reviews 2011;7:CD009255. [26] Ryan R, Santesso N, Hill S, Lowe D, Kaufman C, Grimshaw J. Consumer-oriented interventions for evidence-based prescribing and medicines use: an overview of systematic reviews. Cochrane Database of Systematic Reviews 2011;5:CD007768. [27] Hannah R, Rothstein AJS. Michael Borenstein Publication bias in meta-analysis: prevention, assessment, and adjustments: Wiley.com; 2005. [28] Cooper H, Hedges LV, Valentine JC. The handbook of research synthesis and meta-analysis. Russell Sage Foundation; 2009. [29] Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines. 5. Rating the quality of evidence – publication bias. Journal of Clinical Epidemiology 2011;64(12):1277–82. [30] Cochrane Database of Systematic Reviews. The Cochrane Library. from: http://www.thecochranelibrary.com/view/0/ Available AboutTheCochraneLibrary.html [31] Cochrane handbook for systematic reviews of interventions. Chichester: Wiley-Blackwell; 2011.
Please cite this article in press as: Xiu-xia L, et al. The reporting characteristics and methodological quality of Cochrane reviews about health policy research. Health Policy (2014), http://dx.doi.org/10.1016/j.healthpol.2014.09.002