- Email: [email protected]
Clinical practice guidelines for treating headache with Traditional Chinese Medicine: quality assessment with the appraisal of guidelines for research and evaluation II instrument
Online Submissions: http://www.journaltcm.com [email protected]
J Tradit Chin Med 2018 June 15; 38(3): 339-350 ISSN 0255-2922 © 2018 JTCM. This is an open access article under the CC BY-NC-ND license.
SYSTEMATIC REVIEW TOPIC
Clinical practice guidelines for treating headache with Traditional Chinese Medicine: quality assessment with the appraisal of guidelines for research and evaluation Ⅱ instrument Luo Hao, Li Hui, Wang Yangyang, Yao Sha, Xu Wenjie aa Luo Hao, Yao Sha, School of Basic Medical Sciences, Guangzhou University of Chinese Medicine, Guangzhou 510000, China Li Hui, Wang Yangyang, Research Office of Standardization, Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Engineering and Technology Research Center of Standardization of Traditional Chinese Medicine, Guangzhou 510000, China Xu Wenjie, Scientific Research Office, Beijing Hospital of Traditional Chinese Medicine Affiliated to Capital University of Medicine Sciences, Beijing 100000, China Supported by the Special Research Project of Traditional Chinese Medicine of Guangdong Hospital of Chinese Medicine (No. YN2015MS22) and the Science planning project of Guangzhou (No. 2014Y2-00040) Correspondence to: Prof. Li Hui, Research Office of Standardization, Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Engineering and Technology Research Center of Standardization of Traditional Chinese Medicine, Guangzhou 510000, China. [email protected] Telephone: +86-20-81887233-35945 Accepted: May 29, 2017
Science and Technology Journal Database, and Wanfang database, three guideline-related databases [Guideline-International Network, National Guideline Clearinghouse, and Medlive], and the records of organizations that develop guidelines. The publication date was limited to the period from January 1996 to June 2015. The search terms "headache", "headache disorders", "cephalalgia", "migraine", "tension-type headache", "practice guideline", "consensus ", "statement", "regulation", and "recommendation" were used in the "MeSH" and "Free-text" fields. The guidelines were independently appraised by four researchers using the Appraisal of Guidelines for Research and Evaluation Ⅱ instrument. RESULTS: A total of 23 guidelines published between 1998 and 2014 were reviewed. The overall consistency of the four appraisers was good [interclass correlation coefficient 0.84; 95% confidence interval (CI) 0.82-0.86]. The mean (standard deviation) scores for scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence were 52.1 (18.0), 39.5 (17.1), 33.4 (21.0), 49.8 (21.9), 23.8 (19.3), and 24.2 (23.7). Only two guidelines were recommended, 12 were recommended with modification, and nine were not recommended.
Abstract OBJECTIVE: To critically appraise the methodological quality of clinical practice guidelines for headache produced over the last two decades, including those covering specific interventions using Traditional Chinese Medicine.
CONCLUSION: Physical Traditional Chinese Medicine therapies were recommended to treat headache. The overall quality of headache guidelines was low in China, but evidence-based guidelines are gradually becoming mainstream. Guideline developers should carefully consider, in particular,
METHODS: The guidelines on headache disorders were obtained by searching a number of databases, including PubMed, EMBASE, Web of Science, Chinese Biomedical Literature Database, China National Knowledge Infrastructure Database, China JTCM | www. journaltcm. com
339
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
three domains: rigor of development, applicability, and editorial independence.
edge Infrastructure Database (CNKI), China Science and Technology Journal Database (VIP), and Wanfang database. We also searched two international guideline databases, the Guideline International Network (GIN) and National Guideline Clearinghouse (NGC), as well as a Chinese guideline database (Medlive). We used "headache" as a search term on the websites of several well-known organizations concerned with guideline development: World Health Organization (WHO), American Academy of Neurology (AAN), National Institute for Health and Care Excellence (NICE), Scottish Intercollegiate Guidelines Network (SIGN), and European Academy of Neurology (EAN). We carried out a manual search for gray literature to supplement the electronic searches. The date of publication was restricted to between January 1996 and June 5, 2015. The terms "headache", "headache disorders", "cephalalgia", "migraine", "tension-type headache", "practice guideline", "consensus", "statement", "regulation", and "recommendation" were used in both the "MeSH" and "Free-text" fields. We searched Chinese guidelines for headache disorders manually, as these were published in books and government documents. Using PubMed and EMBASE as examples, the search process was: For PubMed: #1 "Headache"[Mesh] OR "Headache Disorders" [Mesh] #2 "cephalalgia" OR "headache" OR "headache disorders" OR "migraine" OR "tension-type headache" #3 "Practice Guideline" [Publication Type] OR "Practice Guidelines as Topic"[Mesh] #4 "guideline" OR "consensus" OR "statement" OR "recommendation" #5 #1 OR #2 #6 #3 OR #4 #7 #5 AND #6 For EMBASE: #1 'headache'/exp OR 'headache' OR 'cephalalgia' OR 'migraine' OR 'headache disorders' OR 'tension-type headache' #2 'practice guideline'/exp OR 'practice guideline' OR 'consensus' OR 'statement' OR 'recommendation' #3 #1 AND #2
© 2018 JTCM. This is an open access article under the CC BY-NC-ND license.
Keywords: Practice guideline; Headache; Medicine, Chinese Traditional; Appraisal of Guidelines for Research and Evaluation Ⅱ instrument
INTRODUCTION Headache is one of the most common neurological disorders encountered by clinicians.1 According to the World Health Organization (WHO), headaches are among the five most common clinical disorders worldwide. Evidence suggests that active headaches affect more than 46% of the global population, and the lifetime prevalence of headache is more than 90%. Higher prevalence of headaches is reportedly associated with greater social, economic, and family burden.2-5 Traditional Chinese Medicine (TCM) has a long history in treating headache disorders. Many treatments, such as herbal TCM treatments and physical TCM therapy, are effective in treating headaches and have therefore been the focus of studies in modern medicine.6 Several clinical practice guidelines (CPGs) for treating headache with TCM have been developed. Many studies have suggested that CPGs can improve clinical practice and reduce healthcare costs,7,8 but several different CPGs may be published on the same topic. If these guidelines are conflicting, this may affect the confidence of clinicians.9 To our knowledge, no critical appraisal has been performed on the guidelines covering TCM treatment for headaches. There are various guideline assessment tools available, but the Appraisal of Guidelines for Research and Evaluation (AGREE) Ⅱ instrument is the most appropriate for guideline appraisal.10 This study aimed to appraise the methodological quality of guidelines for the treatment of headache produced over the last two decades, and covering interventions using TCM. We hoped to determine the acceptance level worldwide of the different interventions using TCM, and also obtain realistic efficacy data about TCM treatments. By appraising and summarizing the current CPGs for TCM headache treatment, we can determine whether a further guideline for the treatment of headache with TCM is needed.
Inclusion and exclusion criteria We included all Chinese and English language CPGs that provided recommendations on the diagnosis, treatment, and management of headache disorders, and met the definition of guidelines, as described by the Institute of Medicine.9 Guidelines were excluded if they met any of the following criteria: (a) did not include TCM therapies for headaches, including both herbal (Chinese herbal formulas, Chinese patent medicine, and herbal extracts) and physical TCM therapies (acupuncture, moxibustion, massage, manipulation, and osteopathy); (b) translations of international guidelines into Chinese, adaptations of international guidelines for China, ab-
METHODS Information sources We searched for guidelines on headache disorders in PubMed, EMBASE, Web of Science, and four Chinese academic electronic databases, the Chinese Biomedical Literature Database (CBM), China National KnowlJTCM | www. journaltcm. com
340
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
lines. The intra-class correlation coefficients (ICCs) were examined to assess the inter-rater reliability of the four appraisers within each domain. The ICCs ranged between 0 (completely unreliable) and 1 (completely reliable). The reliability was considered poor where the ICC was less than 0.40, and excellent where it was more than 0.75.
stracts or summary reports of international guidelines, and other explanatory or evaluation reports of guidelines; and (c) beta versions or older versions of guidelines from the same source. Literature screening and review data extraction Using the inclusion and exclusion criteria, all records were classified using Endnote (Version X7, Thomson Reuters, USA), and duplicate studies were discarded. Next, two of the authors (Luo Hao and Yao Sha) independently screened all the search records by browsing the title and abstract using the predefined inclusion and exclusion criteria. We then obtained the full text of possible guidelines to determine whether they were eligible, and extracted the general characteristics of each CPG included. Disagreements were resolved by consensus between the two authors or by a third expert (Li Hui).
Statistical analysis We used mean and standard deviation (SD) values to show the proportion of standardized scores for each domain in each guideline. We also calculated the overall mean scores for each domain for all guidelines and the Chinese guidelines. The consistency of the four appraisers for each guideline and across all the guidelines was measured using ICCs and 95% confidence intervals (CIs). 13 Subgroup analyses were performed by year of publication, country, type (consensus-based or evidencebased), grade of evidence, level of recommendation, and economic assessment, using an independent t-test or one-way analysis of variance. All the tests were two-sided, and a P value of < 0.05 was considered statistically significant. All the statistical analyses used SPSS version 19.0 (IBM Inc, New York).
Quality evaluation Each eligible guideline was assessed using the AGREE II instrument, which consists of 23 items across six domains, and two overall assessment items. Each domain assesses a different dimension of guideline quality. Each item was scored from 1 (strong disagreement) to 7 (strong agreement). The standardized scores for each domain were calculated as percentages using the following formulae: Maximum possible score = 7 (strongly agree) × No. of items within a domain × No. of appraisers; Minimum possible score = 1 (strongly disagree) × No. of items within a domain × No. of appraisers; and Observed score = Overall scores of all the appraisers:11,12
RESULTS Figure 1 shows a flowchart of the search and selection process. A total of 13 630 articles were checked across all possible sources. After removing duplicate articles, the remaining 4427 articles were screened by assessing the title and abstract, to give a shortlist of 96 articles for screening by reviewing the entire text. A total of 24 guidelines met our inclusion and exclusion criteria, of which 23 were evaluated using the AGREE Ⅱ instrument, because NAPNP 200725,26 was in two parts. The general characteristics of the eligible guidelines are shown in Table 1. Seven guidelines (30.4%) were developed in China, six (26.1% ) in the USA, and five (21.7% ) in the UK. Only four (17.4% ) were TCM guidelines; the others were modern medicine guidelines touching on TCM treatments. In total, 18 guidelines (78.3% ) were evidence-based and the other five (21.7% ) were consensus-based. Fourteen (60.9% ) guidelines focused on migraine. Only 12 (52.2% ) guidelines included the level of evidence and rated the recommendations. Economic assessment data were only available in seven guidelines (30.4%). The proportions of AGREE Ⅱ standard scores for the eligible guidelines are shown in Table 2. The overall consistency of the four appraisers was good [overall ICC (95% CI), 0.84 (0.82-0.86); data not shown]. After estimation of the scores for each domain and the calculation of the global scores for each guideline, two guidelines were "recommended", 12 were "recommended with modifications", and nine were "not recom-
The overall guideline recommendation considered all the domain scores. Using the standard criteria, a CPG was "recommended" if the scores of five domains were at least 60%; "recommended with modification" if the scores of four domains were at least 30% and the score of one other domain was at least 50%; and "not recommended" if the scores of three domains were 30% or less. Data collection process A pre-designed table with two components (general characteristics and scores of AGREE Ⅱ assessment) was used to extract data from eligible headache guidelines. The general characteristics included title, year of publication, country, organization, nature (Traditional Chinese Medicine or modern medicine), type (evidence-based or consensus-based guidelines), topic covered (syndromes of headache), grade of evidence, level of recommendation, and economic assessment. Synthesis of the results Each eligible guideline was independently assessed by four appraisers (Luo Hao, Wang Yangyang, Yao Sha, and Xu WenJie). Prior to the formal assessment, we conducted a pre-assessment by choosing five guideJTCM | www. journaltcm. com
341
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
Records identified through electronic database searching (n = 3627)
Search in website and other resource (n = 68)
Records after duplicates removed (n = 2737)
Records excluded by checking titles and abstract (n = 2641)
Records after screened (n = 96)
Guidelines excluded by checking full-text articles (n = 72)
Eligible guidelines after screened (n = 24) Figure 1 Flow diagram showing the search and selection process in the study
mended". The overall mean score showed that domain 1 was the highest of the six domains, and domains 5 and 6 were the lowest; the scores for the other domains ranged from 39% to 50%. The mean score for the four TCM guidelines was lower than the overall mean score in each domain, although domains 5 and 6 were again the lowest. Table 3 shows the results of the subgroup analysis of the eligible guidelines. The year of publication was not statistically significant, although the mean scores for the guidelines published in and after 2011 were better than those for guidelines published before 2011. The guidelines developed in China had significantly lower scores than those developed in the USA for domains 1 and 6 (P < 0.05), and those developed in the UK for all the domains except domain 2 (P < 0.01). Evidence-based guidelines had higher quality scores than consensus-based guidelines for all the domains (P < 0.05). The scores for guidelines with an evidence grade were better, particularly in domains 1 to 4 (P < 0.05). Recommended guidelines were significantly likely to have higher scores in domains 4 (P < 0.05). Guidelines with an economic assessment were significantly likely to be better in all domains except domain 6. Table 4 shows the recommendations on the use of JTCM | www. journaltcm. com
TCM in the two "recommended" modern medicine guidelines and two "recommended with modification" TCM guidelines. Physical therapies were more likely to be accepted by modern medicine. Herbal therapies were not recommended in modern medicine guidelines because of insufficient evidence, adverse effects, or poor efficacy. Herbal formulas, Chinese patent medicine, acupuncture, moxibustion, massage and manipulation are, however, recommended in TCM guidelines. We compared the guideline quality in this study with previous related studies. Figure 2 shows comparisons with the studies of Alonso-Coello et al,38 Chen et al,39 Wei et al,40 and Li et al,41 who assessed the quality of international and Chinese guidelines in the last two decades. The quality of the headache guidelines was better than that of other domestic guidelines, but the mean scores for the domestic headache guidelines were poorer than international guidelines for domains 3, 5, and 6. However, the mean domain scores in international guidelines for domains 5 and 6 were still less than 30%. We also compared the quality assessment results for the four TCM guidelines with those from previous studies (Figure 3). The scores for the TCM guide342
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review Table 1 General characteristics of the eligible guidelines Organization year PMDP 200614 PTDT 200715 CACM1 2008
16
CACM2 2008
17
CACMS/CAAM 2011
18
CASP 201119 CACMS 2011
20
CHS 1998
Country
Nature
Type
Topic
Grade of evidence
Level of recommendation
Economic assessment
China
MM
CB
M
No
No
No
China
MM
CB
TTH
No
No
No
China
TCM
CB
M
No
No
No
China
TCM
CB
PH
No
No
No
China
TCM
EB
M
Yes
Yes
No
China
MM
EB
M
Yes
Yes
No
China
TCM
EB
M
Yes
Yes
No
Canada
MM
EB
M
Yes
Yes
Yes
22
USA
MM
EB
M
Yes
Yes
No
AAN2 200023
USA
MM
EB
M
Yes
No
No
USA
MM
EB
M
Yes
Yes
Yes
USA
MM
EB
M
No
No
No
USA
MM
EB
H
Yes
Yes
Yes
ACP/ASIM 201228
USA
MM
EB
M
No
No
Yes
SIGN 2008
UK
MM
EB
H
Yes
Yes
Yes
21
AAN1 2000
AAN3 2000
24
NAPNP 2007 ICSI 2011
25,26
27
29
EFNS 2009
UK
MM
EB
M
No
Yes
No
31
BASH 2010
UK
MM
EB
H
No
No
Yes
EFNS 201032
UK
MM
EB
TTH
Yes
Yes
No
NICE 2012
UK
MM
EB
H
Yes
No
Yes
34
ISSH 2012
Italy
MM
EB
PH
Yes
Yes
No
35
DHS 2012
Denmark
MM
EB
H
No
No
No
France
MM
EB
M
No
Yes
No
Latin America
MM
CB
M
No
No
No
30
33
FSSMH 201436 LAPCMT 2013
37
Notes: MM: modern medicine; TCM: Traditional Chinese Medicine; CB: consensus-based; EB: evidence-based; H: headache; PH; primary headache; M: migraine; TTH: tension-type headache; PMDP: Panel of Migraine Diagnosis and Prevention; PTDT: Panel of TTH Diagnosis and Treatment; CACM1/2: China Association of Chinese Medicine; CACMS/CAAM: China Academy of Chinese Medical Sciences/ China Association of Acupuncture-Moxibustion; CASP: Chinese Association for the Study of Pain; CACMS: China Academy of Chinese Medical Sciences; CHS: Canadian Headache Society; AAN1/2/3: American Academy of Neurology; NAPNP: National Association of Pediatric Nurse Practitioners; ICSI: Institute for Clinical Systems Improvement; ACP/ASIM: American College of Physicians/American Society of Internal Medicine; SIGN: Scottish Intercollegiate Guidelines Network; EFNS: European Federation of Neurological Societies; BASH: British Association for the Study of Headache; NICE: National Institute for Health and Clinical Excellence; ISSH: Italian Society for the Study of Headaches; DHS: Danish Headache Society; FSSMH: French Society for the Study of Migraine Headache; LAPCMT: Latin American Panel of Chronic Migraine Treatment.
DISCUSSION
lines in this study were lower than those reported by Choi et al 42 for all the domains but better than those reported by Chen et al 43 in domains 2 and 3. They were, however, lower than 30% in domains 3, 5, and 6. There are therefore no high quality TCM headache guidelines in China, as the scores for all the TCM guidelines were less than 40% in all domains. As Figure 3 shows, the scores found in two studies were higher than the others. After a thorough investigation, we found that the authors of the two studies had participated in the process of developing some of the guidelines included but had not reported this as a conflict of interest.44,45 JTCM | www. journaltcm. com
Results of the appraisal of high quality CPGs The ICC (95% CI) of the NICE headache guideline33 (0.59, 0.23-0.81) — one of the two recommended guidelines — was the lowest of the eligible guidelines. The four appraisers had different professional academic backgrounds, and we believe that the structures of the reporting guidelines and the miscellaneous content may have led to diversity. We suggest that it is increasingly important to have a uniform format for the reporting guidelines, and also to develop alternative versions for target users from different backgrounds, such as patients, clinicians, methodologists, and policy makers. 343
June 15, 2018 | Volume 38 | Issue 3 |
JTCM | www. journaltcm. com
344 32.6 (21.2)
31.9 (15.8)
NA
23.8 (21.7)
4.7 2.1 4.7 5.7 46.4 24.5 38.5 41.7 40.6 40.6 9.4 45.3 27.6 27.6 71.9 47.9 21.9 46.4 83.3 45.8 19.3 35.4 21.9 33.4 (21.0)
Domain 3: rigor of development
39.6 (19.7)
18.1 18.1 20.8 25.0 61.1 54.2 51.4 47.2 48.6 63.9 3.8 65.3 63.9 63.9 86.1 59.2 56.9 59.7 91.7 69.4 50.0 52.8 29.2 49.8 (21.9)
Domain 4: clarity of presentation
Scores of each domain (%)
7.3 (8.1)
5.2 2.1 2.1 1.0 18.8 29.2 7.3 16.7 18.8 20.8 20.8 47.9 21.9 21.9 67.7 12.5 36.5 17.7 78.1 25.0 34.9 26.0 14.6 23.8 (19.3)
Domain 5: applicability
1.6 (3.1)
Domain 6: editorial independence 0.0 0.0 0.0 0.0 0.0 0.0 6.3 18.8 22.9 22.9 0.0 52.1 16.6 16.6 60.4 60.4 10.4 52.1 77.1 37.5 31.3 43.8 18.8 24.2 (23.7) NA
Overall guideline assessment NR NR NR NR RM NR RM NR RM RM NR RM NR NR Recommended RM RM RM Recommended RM RM RM NR NA
Notes: SD: standard deviation; NA: not applicable; NR: not recommended; RM: recommended with modification; MM: modern medicine; TCM: Traditional Chinese Medicine; CB: consensus-based; EB: evidence-based; H: headache; PH: primary headache; M: migraine; TTH: tension-type headache; PMDP: panel of migraine diagnosis and prevention; PTDT: panel of TTH diagnosis and treatment; CACM1/2: China Association of Chinese Medicine; CACMS/CAAM: China Academy of Chinese Medical Sciences/China Association of Acupuncture-Moxibustion; CASP: Chinese Association for the Study of Pain; CACMS: China Academy of Chinese Medical Sciences; CHS: Canadian Headache Society; AAN1/2/3: American Academy of Neurology; NAPNP: National Association of Pediatric Nurse Practitioners; ICSI: Institute for Clinical Systems Improvement; ACP/ASIM: American College of Physicians/American Society of Internal Medicine; SIGN: Scottish Intercollegiate Guidelines Network; EFNS: European Federation of Neurological Societies; BASH: British Association for the Study of Headache; NICE: National Institute for Health and Clinical Excellence; ISSH: Italian Society for the Study of Headaches; DHS: Danish Headache Society; FSSMH: French Society for the Study of Migraine Headache; LAPCMT: Latin American Panel of Chronic Migraine Treatment.
33.3 11.1 15.3 13.9 55.6 38.9 45.8 41.7 37.5 43.1 15.3 65.3 43.1 43.1 69.4 30.6 45.8 33.3 76.4 29.2 44.4 51.4 33.3 39.5 (17.1)
34.7 23.6 19.4 18.1 50.0 43.1 40.3 48.6 65.3 70.8 38.9 76.4 51.4 51.4 77.8 55.6 61.1 58.3 88.9 55.6 52.8 61.1 48.6 52.1 (18.0)
0.83 (0.68, 0.92) 0.84 (0.70, 0.93) 0.81 (0.64, 0.91) 0.80 (0.61, 0.90) 0.89 (0.78, 0.95) 0.89 (0.79, 0.95) 0.89 (0.80, 0.95) 0.83 (0.67, 0.92) 0.77 (0.57, 0.89) 0.81 (0.63, 0.91) 0.78 (0.59, 0.90) 0.62 (0.29, 0.82) 0.81 (0.64, 0.91) 0.81 (0.64, 0.91) 0.65 (0.35, 0.84) 0.84 (0.70, 0.93) 0.86 (0.73, 0.93) 0.77 (0.56, 0.89) 0.59 (0.23, 0.81) 0.79 (0.61, 0.90) 0.72 (0.47, 0.87) 0.63 (0.31, 0.83) 0.67 (0.37, 0.84) NA
PMDP 200614 PTDT 200715 CACM1 200816 CACM2 200817 CACMS/CAAM 201118 CASP 201119 CACMS 201120 CHS 199821 AAN1 200022 AAN2 200023 AAN3 200024 NAPNP 200725,26 ICSI 201127 ACP/ASIM 201228 SIGN 200829 EFNS 200930 BASH 201031 EFNS 201032 NICE 201233 ISSH 201234 DHS 201235 FSSMH 201436 LAPCMT 201337 Overall mean (SD) score Mean (SD) score of the Chinese guidelines
Domain 2: stakeholder involvement
Domain 1: scope and purpose
ICC (95% CI)
Organization year
Table 2 AGREE Ⅱ domain scores and ICCs (95% CI) of the eligible guidelines
Luo H et al. / Systematic Review
June 15, 2018 | Volume 38 | Issue 3 |
JTCM | www. journaltcm. com
345
5 (21.7) 18 (78.3) -
CB
EB
P value
7 (30.4) 16 (69.6)
Yes
-
No
46.7 (21.7)
11 (47.8)
No
P value 45.2 (15.0)
67.9 (14.8)
0.100
58.0 (12.1)
0.009
42.3 (16.3)
61.1 (15.0)
<0.001
58.6 (13.4)
28.8 (12.8)
0.010
<0.001
12 (52.2)
Yes
-
-
MM
P value
P value
19 (82.6)
TCM
11 (47.8)
56.4 (15.7)
4 (17.4)
P value (China vs UK)
12 (52.2)
31.9 (15.8)
-
Yes
68.3 (14.4)
5 (21.7)
UK
P value (China vs USA)
No
57.3 (14.6)
6 (26.1)
USA 0.002
32.7 (12.5)
7 (30.4)
China
58.0 (15.1)
10 (43.8) 0.281
48.5 (19.8)
13 (56.5)
Scope and purpose
P value
≥2011
<2011
n (%)
32.7 (13.4)
55.0 (14.8)
27.3 (17.6)
47.5 (22.4)
0.008
21.9 (23.6)
33.6 (19.4) 0.111
44.0 (10.9)
<0.001
18.2 (14.7)
47.4 (15.5)
0.001
40.5 (17.5)
7.8 (8.0)
0.333
35.5 (20.1)
23.8 (21.7)
0.005
0.140
54.3 (24.0)
35.0 (12.7)
18.1 (18.4)
0.292
38.8 (18.7)
29.3 (22.4)
Rigor of development
44.9 (13.2)
0.014
30.7 (14.7)
47.6 (15.5)
0.004
44.5 (15.2)
21.4 (11.0)
0.392
40.9 (16.4)
32.6 (21.2)
0.700
0.371
51.1 (20.9)
40.7 (14.9)
30.6 (17.4)
0.260
48.3 (14.4)
32.7 (16.3)
Stakeholder involvement
Scores of each domain [% (SD)]
42.0 (19.6)
67.9 (15.7)
0.024
38.7 (25.8)
60.0 (10.6)
0.003
36.2 (20.8)
62.4 (14.3)
<0.001
57.5 (18.1)
22.2 (4.8)
0.310
52.0 (22.2)
39.6 (19.7)
0.009
0.241
70.7 (16.7)
49.5 (21.5)
35.5 (19.1)
0.081
58.9 (16.1)
42.9 (23.7)
Clarity of presentation
16.1 (10.3)
41.4 (24.2)
0.560
21.4 (22.6)
26.0 (16.6)
0.067
16.1 (12.9)
30.8 (22.0)
0.010
29.0 (18.6)
5.0 (5.6)
0.059
27.3 (19.3)
7.3 (8.1)
0.006
0.130
42.5 (29.4)
22.8 (12.2)
9.4 (10.6)
0.160
30.3 (20.1)
18.8 (17.9)
Applicability
18.6 (21.1)
36.9 (25.9)
0.121
16.1 (23.1)
31.6 (22.6)
0.140
16.5 (20.7)
31.3 (24.8)
0.001
29.9 (23.5)
3.8 (8.4)
<0.001
29.0 (23.3)
1.6 (3.1)
<0.001
0.026
52.1 (25.0)
20.8 (16.7)
0.9 (2.4)
0.471
28.3 (24.9)
21.0 (23.1)
Editorial independence
P value 0.002 0.033 0.030 0.088 0.003 0.006 Notes: SD: standard deviation; AGREE: Appraisal of Guidelines for Research and Evaluation Ⅱ instrument; TCM: Traditional Chinese Medicine; MM: modern medicine; CB: consensus-based; EB: evidence-based.
Economic assessment
Recommendation level
Evidence grade
Type
Nature
Country
Year
Variable
Table 3 AGREE Ⅱ domain scores for guidelines by subgroups
Luo H et al. / Systematic Review
Domains with low scores As shown in Table 2, the overall mean (SD) scores of the headache guidelines were poor for domains 3, 5, and 6. The rigor of development domain (domain 3) is considered to be the most essential domain in the development of guidelines. It consists of eight items and appraises the methods used for evidence collection, grading, and summary, as well as the formulation of recommendations.11 The evidencebased guidelines had better scores than the consensus-based guidelines for all the domains (P < 0.05). Only one non-Chinese guideline was consensus-based. This trend suggests that evidence-based guidelines are becoming more mainstream. In Table 4, the four "recommended" or "recommended with modification" guidelines were all evidence-based, although the evidence grading and formulation of recommendations differed. These findings suggest that there is no common,
June 15, 2018 | Volume 38 | Issue 3 |
CACMS 2011
20
CACMS/CAAM 2011
Notes: NA: not applicable; NR: not recommended; SIGN: Scottish Intercollegiate Guidelines Network; NICE: National Institute for Health and Clinical Excellence; CACMS/CAAM: China Academy of Chinese Medical Sciences/China Association of Acupuncture-Moxibustion; CACMS: China Academy of Chinese Medical Sciences.
Recommended Recommended Recommended Recommended
NA
Recommended
NA Recommended NA NA
NA
Recommended
Recommended NA NA NA
SIGN 2008
18
NICE 201233
NR
Recommended
Insufficient Evidence NA Recommended NA NA
NR
Moxibustion Acupuncture Herbal extract Chinese patent medicine Herbal formula
29
Herbal therapy
Organization year
Table 4 Recommendations of two "recommended" modern medicine guidelines and two "recommended with modification" TCM guidelines
Physical therapy
Massage or manipulation
Luo H et al. / Systematic Review
JTCM | www. journaltcm. com
sensible, and transparent approach to grading the quality (or certainty) of evidence and the strength of the recommendations. The applicability domain (domain 5) evaluates whether the guideline provides advice on implementing recommendations. It considers the potential resources, facilitators, and barriers and provides monitoring or auditing criteria derived from the key recommendations.11 However, most of the guidelines included in this study did not discuss the applicability, or highlight the tools or methods required to facilitate or promote the guidelines. The mean score (SD) for domain 5 was therefore the lowest of the domains [23.8 (19.3); Figure 2], which is consistent with the results of previous studies.38-40,42 The editorial independence domain (domain 6) assesses the bias related to conflict of interests in the development of guidelines. 11 In our study, the mean score (SD) for domain 6 was 24.2 (23.7), and most of the guidelines included did not report any conflicts of interest or financial conflicts.
In a previous study, 87% of guideline developers were found to have some form of affiliation with the pharmaceutical industry, 58% had received financial support, and 38% had served as an employee or consultant for a pharmaceutical company.46 Conflicts of interest are often hard to detect, but evidence suggests that they can influence the recommendations in a guideline.49 The results suggest that developers should focus on the "rigor of development", "applicability", and "editorial independence" when developing TCM headache guidelines in the future. They should also use widely-recognized methods, such as the GRADE approach, to rate the quality of the best available evidence and develop healthcare recommendations for the establishment of CPGs.50 The potential advantages and disadvantages of treatments, such as patient and social values or preferences, should be carefully considered and combined with the evidence to formulate the final recommendations.51 Before developing a new guideline, it is important to list the funding sources, or provide formal conflict of interest statements for all the authors for discussion and public disclosure.47,52,53 Discussion of the two TCM guidelines With the support of the World Health Organization/ Western Pacific Regional Office (WHO/WPRO), the CACMS/CAAM 2011 and CACMS 2011 were the first evidence-based guidelines on TCM to be adopted in China. However, these two guidelines had poor scores for domains 5 and 6. Table 4 shows that they recommended the use of herbal therapies to treat headache. They differed from the two "recommended" guidelines, which described the method for the extraction of herbs, acupuncture, massage, and manipulation in the prophylaxis of headache. We believe that there are two main reasons for the difficulty in forming recommendations for traditional Chinese treatments in the guidelines. First, it is difficult to conduct high-quality randomized clinical trials because of the personalized nature of treatment with TCM.54,55 Second, randomized clinical trials are one of the major sources of clinical evidence, but their current quality — although better than before — remains unsatisfactory.56 The AGREE Ⅱ instrument is a tool for assessing guideline development, reporting, and evaluation.11 However, if a TCM guideline is fully developed by this instrument, we think that the clinical practice characteristics of TCM may not be appropriately reflected. Different methods should therefore be considered when developing a guideline, and the existing guidelines on headache should be critically appraised to develop high-quality evidence-based TCM guidelines for headache, suitable for use with individuals.57-60 We hope that ongoing work will lead to the development of high-quality TCM guidelines and contribute to the incorporation of effective traditional therapies into clinical practice. 346
June 15, 2018 | Volume 38 | Issue 3 |
Mean scores of each domain (%)
Luo H et al. / Systematic Review 70 60 50 40 30 20 10 0
e cop
S
and
ent
ent
ose
rp pu
er old
em olv inv
igo
f ro
pm elo v e d
of
t
nta
se pre
lop
ve ion
Mean scores of each domain (%)
R keh ty Sta lari In our study Alonso 2010 C Chen 2012 Figure 2 Comparison of assessed quality between headache guidelines and others
p Ap
e enc
y
ilit
b lica
Ed Wei 2013
nd
al i
ri ito
nd epe
Li 2013
120 100 80 60 40 20 0
t y ce bilit t men op nden elop nvel men plica e v e o p i e p v t l e A d a o d t r of r inv al in resen Scop Rigo olde itori of p d h e y E t k i Sta Clar TCM Guildline Choi 2015 Chen 2014 Fang 2014 Yu 2011 Figure 3 Comparison of the assessment of quality of the TCM guidelines between this study and other TM studies se
urpo
dp e an
In conclusion, we found that modern medicine guidelines did not recommend using Chinese herbal therapies, but the two recommended guidelines, supported by clinical evidence, suggest the use of physical TCM therapies for treating headache. The two TCM guidelines, which were "recommended with modification" following assessment, recommended the use of Chinese herbal and physical therapies to treat headaches. Healthcare professionals should choose suitable and high-quality CPGs to guide their clinical practice. The overall quality of TCM headache guidelines was low in China, but evidence-based guidelines are gradually becoming mainstream. Guideline developers should carefully consider the three most important domains, covering the rigor of development, applicability, and editorial independence.
Strengths and limitations Our study has several strengths. First, the systematic review of guideline quality covered a wide range of guidelines on headache disorders, from around the world. Second, our team included a methodologist and neurologist who had prior experience of developing guidelines. The study also had some limitations, however. First, we focused on guidelines that covered TCM therapies for headache and were published in Chinese or English. Most of the guidelines included were therefore published in Europe, America, and China. These may not be representative of all the available guidelines on headache disorders. Second, the supplementary materials and background information on the guidelines could not be searched and assessed thoroughly, because different organizations used different measures to report the supporting information. This could lead to an underestimation of guideline quality in certain domains. Third, the AGREE Ⅱ instrument used to assess the overall quality of the guidelines did not provide any criteria or instructions. This makes it difficult for assessors to reach a consensus on the overall quality of the guidelines.10 JTCM | www. journaltcm. com
ACKNOWLEDGMENTS We thank Yao Liang, Wei Dang, Wang Qi, Wang Xiaoqin, Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, Gansu, China, for their general assistance and help in the revision of the manuscript. 347
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
REFERENCES 1
2
3
4
5
6
7
8
9
10
11
12
13
14 15 16
Headache Classification Committee of the International Headache Society. The International Classification of Headache Disorders. 3rd ed (beta version). Cephalalgia 2013; 33(9): 629-808. Vos T, Flaxman AD, Naghavi M, et al. Years lived with disability (YLDS) for 160 sequelae of 289 diseases and injuries: 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. The Lancet 2014; 384(9943): 582-582. Stovner LJ, Hagen K, Jensen R, et al. The global burden of headache: a documentation of headache prevalence and disability worldwide. Cephalalgia 2007; 27: 193-210. Rasmussen BK, Jensen R, Schroll M, et al. Epidemiology of headache in a general population — a prevalence study. J Clin Epidemiol 1991; 44: 1147-1157. Latinovic R, Gulliford M, Ridsdale L. Headache and migraine in primary care: consultation, prescription, and referral rates in a large population. J Neurol Neurosurg Psychiatry 2006; 77(3): 385-387. Wang YY, Zhang YL. A review of the clinical research on the treatment of headache in Chinese medicine in the past 10 years. Beijing Zhong Yi Yao Da Xue Xue Bao 1992; (1): 5-12. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. The Lancet 1993; 342(8883): 1317-1322. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ 1999; 318(7182): 527-530. Graham R, Mancher M, Miller Wolman D, et al. Clinical practice guidelines we can trust. Washington, DC: National Academy of Sciences, 2011: 20, 50. Vlayen J, Aertgeerts B, Hannes,K, Sermeus W, Ramaekers D. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care 2005; 17(3): 235-242. Brouwers MC, Kho ME, Browman GP, et al. AGREE Ⅱ: advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J 2010; 182(18): E839-E842. AGREE. AGREE Next Steps Consortium (2009). The
17
18
19
20
21
22
23
24
25
26
AGREE Ⅱ Instrument UPDATE September, 2013. Available from URL: http://www.agreetrust.org/wp-content/uploads/2013/10/AGREE-II-Users-Manual-and-23-item-Instrument_2009_UPDATE_2013.pdf. Pan X, Ni Z. Application of intra-class correlation coefficient to reliability assessment. Hua Xi Yi Ke Da Xue Xue Bao 1999; 30(1): 62-63, 67. Li YS. The diagnosis and management of migraine: consensus. Chin J Intern Med 2006; 45(8): 694-696. Yu SY. The diagnosis and management of tension-type headache: consensus. Chin J Neuro 2007; 40(7): 496-497. China Association of Chinese Medicine. Guidelines for di-
JTCM | www. journaltcm. com
27
28
29
30 348
agnosis and treatment of common internal diseases in chinese medicine: disease of modern medicine (migraine). Beijing: China Press of Traditional Chinese Medicine 2008: 275-278. China Association of Chinese Medicine. guidelines for diagnosis and treatment of common internal diseases in chinese medicine: symptoms in chinese medicine (headache). Beijing: China Press of Traditional Chinese Medicine 2008: 126-128. Wu ZC. Evidence-based Guidelines of Clinical Practice in Chinese Medicine: Acupuncture (Migraine), Beijing: China Press of Traditional Chinese Medicine 2014, 144-162. Yu SY. Guidelines for Diagnosis and Treatment of Migraine. Zhong Guo Teng Tong Yi Xue Za Zhi 2011; 17 (2): 65-86. China Academy of Chinese Medical Sciences, Evidence-based Guidelines of Clinical Practice in Chinese Medicine: Internal Medicine (Migraine), Beijing: China Press of Traditional Chinese Medicine, 2011: 210-226. Pryse-Phillips WE, Dodick DW, Edmeads JG, et al. Guidelines for the non-pharmacologic management of migraine in clinical practice. CMAJ 1998; 159(1): 47-54. Ramadan NM, Silberstein SD, Freitag FG, et al. Evidence-based guidelines for migraine headache in the primary care setting: pharmacological management for prevention of migraine. Headache 2000; 46(7): 11511160. Campbell JK, Penzien DB. Evidenced-based guidelines for migraine headache: behavioral and physical treatments. American Academy of Neurology. Available from URL: http://tools.aan.com/professionals/practice/pdfs/gl 0089.pdf. Silberstein SD. Practice Parameter: Evidence-based guidelines for migraine headache. Neurology 2000; 55(6): 754-762. gunner kb, smith hd, ferguson le, et al. practice guideline for diagnosis and management of migraine headaches in children and adolescents: part one. J Pediatr Health Care 2007; 21(1): 327-332. Gunner KB, Smith HD, Ferguson LE, et al. Practice guideline for diagnosis and management of migraine headaches in children and adolescents: Part two. J Pediatr Health Care 2007; 22(1): 52-59. Institute for Clinical Systems Improvement. health care guideline: diagnosis and treatment of headache 10th edition 2011. Available from URL: https://www.icsi.org/_asset/qwrznq/Headache.pdf. Snow V, Weiss K, Wall EM, et al. Pharmacologic management of acute attacks of migraine and prevention of migraine headache. Ann Intern Med 2002; 137(10): 840-849. Scottish Intercollegiate Guidelines Network. Diagnosis and management of headache in adults 2008. Available from URL: http://www.sign.ac.uk/pdf/sign107. Evers S, Afra J, Frese A, et al. EFNS guideline on the drug June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
treatment of migraine-revised report of an EFNS task force. Eur J Neurol 2009; 16(9): 968-981. MacGregor EA, Steiner TJ, Davies PTG. Guidelines for all healthcare professionals in the diagnosis and management of migraine, tension-type headache, cluster headache and medication-overuse headache. Available from URL: http://www.bash.org.uk/wp-content/uploads/2012/07/10102-BASH-Guidelines-update-2_v5-1indd.pdf. Bendtsen L, Evers S, Linde M, et al. EFNS guideline on the treatment of tension-type headache-Report of an EFNS task force. Eur J Neurol 2010; 17(11); 1318-1325. National Institute for Health and Care Excellence. Diagnosis and management of headache in young people and adults 2012. Available from URL: https://www.nice.org. uk/guidance/cg150/update/CG150/documents/headaches-full-guideline-for-consultation2. Sarchielli P, Granella F, Prudenzano M, et al. Italian guidelines for primary headaches: 2012 revised version. J Headache Pain 2012; 13(2): 31-70. Bendtsen L, Birk S, Kasch H, et al. Reference programme: diagnosis and treatment of headache disorders and facial pain (2nd Edition). J Headache Pain 2012; 13 (suppl 1): 1-29. Géraud G, Lantéri-Minet M, Lucas C, et al. French guidelines for the diagnosis and management of migraine in adults and children. Clinical Therapeutics 2004; 26(8): 1305-1318. Giacomozzi AR, Vindas AP, Jr SA, et al. Latin American consensus on guidelines for chronic migraine treatment. Arq Neuro-psiquiatr 2013; 71(7): 478-486. Alonso-Coello P, Irfan A, SolàI, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. QualSaf Health Care 2010; 19(6): e58. Chen YL, Yao L, Xiao XJ, et al. Quality assessment of clinical guidelines in China: 1993-2010. Chin Med J 2012; 125(20): 3660-3664. Wei D, Wang XQ, Wu QF, et al. Quality evaluation on chinese clinical practice guidelines in 2011. Chin J Evid-based Med 2013; 13(6): 760-763. Li N, Yao L, Wu QF, et al. Quality evaluation of clinical practice guidelines published in journals of mainland China during 2012-2013. Chin J Evid-based Med 2015; 15 (3): 259-263. Choi TY, Choi J, Lee JA, Ji HJ, Bongki P, Myeong SL. The quality of clinical practice guidelines in traditional medicine in Korea: appraisal using the AGREE Ⅱ instrument. Implement Sci 2015; 10(1): 104. Chen H, Li GL, Xu WT, Xu B. Quality Assessment of Clinical Practice Guidelines of Acupuncture in China. Chin J Evid-based Med 2014; 14(6): 772-775. Fang YG, Bai Y, Liu BY, Wang F, Xue W. Quality assessment on guidelines of clinical practice in acupuncture and moxibustion: a study based on AGREE Ⅱ. Zhong Guo Zhen Jiu 2014; 34(6): 599-601. Yu WY, Han XJ, Zhang H, et al. Study on quality assess-
JTCM | www. journaltcm. com
46
47
48 49
50
51
52
53
54
55
56
57
58
59
349
ment of TCM clinical practice guidelines based on AGREE instrument. Shi Jie Ke Xue Ji Shu-Zhong Yi Yao Xian Dai Hua 2011; 13(4): 596-600. Papanikolaou GN, Baltogianni MS, Contopoulos-Ioannidis DG, et al. Reporting of conflicts of interest in guidelines of preventive and therapeutic interventions. BMC Med Res Methodol 2011; 1(1): 3. Choudhry NK, Stelfox HT, Detsky AS. Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA 2002; 287(5): 612-617. Thompson DF. Understanding financial conflicts of interest. N Engl J Med 1993; 329(8): 573-576. Norris SL, Burda BU, Holmer HK, et al. Author's specialty and conflicts of interest contribute to conflicting guidelines for screening mammography. J Clin Epidemiol 2012; 65(7): 725-733. Brozek JL, Akl EA, Alonso-Coello P, Lang D, Jaeschke R, Williams JW; GRADE Working Group. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions. Allergy 2009; 64(5): 669-677. Guyatt G, Oxman AD, Akl EA, et al. (2011) GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 64(4): 383-394. Schünemann HJ, Al-Ansary LA, Forland F, et al. Guidelines international network: principles for disclosure of interests and management of conflicts in guidelines. Ann Intern Med 2015; 163(7): 548-553. Campbell EG (2010) Public disclosure of conflicts of interest: moving the policy debate forward. Arch Intern Med 170(8): 667. Shekelle PG, Morton SC, Suttorp MJ, Buscemi N, Friesen C; Agency for Healthcare Research and Quality. Challenges in systematic reviews of complementary and alternative medicine topics. Ann Intern Med 2005; 142(12 Pt 2): 1042-1047. Practice and Policy Guidelines Panel, National Institutes of Health Office of Alternative Medicine. Clinical practice guidelines in complementary and alternative medicine. An analysis of opportunities and obstacles. Arch Fam Med 1997; 6(2): 149-154. He J, Du L, Liu G, et al. Quality assessment of reporting of randomization, allocation concealment, and blinding in Traditional Chinese Medicine RCTs: a review of 3159 RCTs identified from 260 systematic reviews. Trials 2011; 12(1): 286-298. Chen KJ, Jiang YR. Current status and problems in developing clinical guidelines for Chinese medicine and integrative medicine. Zhong Xi Yi Jie He Xue Bao 2009; 7(4): 301-305. Liu M, Yang J, Wang YP. Thoughts on the development and Implementing of Evidence-Based Guideline. Chin J Evid-based Med 2009; 9(2): 127-128. Wang B, Zhan SY, Liu BY. Problems and strategies in developing chinese medicine evidence-based clinical pracJune 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
60
egies 2014-2023. Available from URL: http://www. who.int/medicines/publications/traditional/trm_strategy 14_23/en/.
tice guidelines. Chin J Integr Med 2011; 31(11): 1565-1569. World Health Organization. Traditional Medicine Strat-
JTCM | www. journaltcm. com
350
June 15, 2018 | Volume 38 | Issue 3 |
J Tradit Chin Med 2018 June 15; 38(3): 339-350 ISSN 0255-2922 © 2018 JTCM. This is an open access article under the CC BY-NC-ND license.
SYSTEMATIC REVIEW TOPIC
Clinical practice guidelines for treating headache with Traditional Chinese Medicine: quality assessment with the appraisal of guidelines for research and evaluation Ⅱ instrument Luo Hao, Li Hui, Wang Yangyang, Yao Sha, Xu Wenjie aa Luo Hao, Yao Sha, School of Basic Medical Sciences, Guangzhou University of Chinese Medicine, Guangzhou 510000, China Li Hui, Wang Yangyang, Research Office of Standardization, Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Engineering and Technology Research Center of Standardization of Traditional Chinese Medicine, Guangzhou 510000, China Xu Wenjie, Scientific Research Office, Beijing Hospital of Traditional Chinese Medicine Affiliated to Capital University of Medicine Sciences, Beijing 100000, China Supported by the Special Research Project of Traditional Chinese Medicine of Guangdong Hospital of Chinese Medicine (No. YN2015MS22) and the Science planning project of Guangzhou (No. 2014Y2-00040) Correspondence to: Prof. Li Hui, Research Office of Standardization, Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Engineering and Technology Research Center of Standardization of Traditional Chinese Medicine, Guangzhou 510000, China. [email protected] Telephone: +86-20-81887233-35945 Accepted: May 29, 2017
Science and Technology Journal Database, and Wanfang database, three guideline-related databases [Guideline-International Network, National Guideline Clearinghouse, and Medlive], and the records of organizations that develop guidelines. The publication date was limited to the period from January 1996 to June 2015. The search terms "headache", "headache disorders", "cephalalgia", "migraine", "tension-type headache", "practice guideline", "consensus ", "statement", "regulation", and "recommendation" were used in the "MeSH" and "Free-text" fields. The guidelines were independently appraised by four researchers using the Appraisal of Guidelines for Research and Evaluation Ⅱ instrument. RESULTS: A total of 23 guidelines published between 1998 and 2014 were reviewed. The overall consistency of the four appraisers was good [interclass correlation coefficient 0.84; 95% confidence interval (CI) 0.82-0.86]. The mean (standard deviation) scores for scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence were 52.1 (18.0), 39.5 (17.1), 33.4 (21.0), 49.8 (21.9), 23.8 (19.3), and 24.2 (23.7). Only two guidelines were recommended, 12 were recommended with modification, and nine were not recommended.
Abstract OBJECTIVE: To critically appraise the methodological quality of clinical practice guidelines for headache produced over the last two decades, including those covering specific interventions using Traditional Chinese Medicine.
CONCLUSION: Physical Traditional Chinese Medicine therapies were recommended to treat headache. The overall quality of headache guidelines was low in China, but evidence-based guidelines are gradually becoming mainstream. Guideline developers should carefully consider, in particular,
METHODS: The guidelines on headache disorders were obtained by searching a number of databases, including PubMed, EMBASE, Web of Science, Chinese Biomedical Literature Database, China National Knowledge Infrastructure Database, China JTCM | www. journaltcm. com
339
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
three domains: rigor of development, applicability, and editorial independence.
edge Infrastructure Database (CNKI), China Science and Technology Journal Database (VIP), and Wanfang database. We also searched two international guideline databases, the Guideline International Network (GIN) and National Guideline Clearinghouse (NGC), as well as a Chinese guideline database (Medlive). We used "headache" as a search term on the websites of several well-known organizations concerned with guideline development: World Health Organization (WHO), American Academy of Neurology (AAN), National Institute for Health and Care Excellence (NICE), Scottish Intercollegiate Guidelines Network (SIGN), and European Academy of Neurology (EAN). We carried out a manual search for gray literature to supplement the electronic searches. The date of publication was restricted to between January 1996 and June 5, 2015. The terms "headache", "headache disorders", "cephalalgia", "migraine", "tension-type headache", "practice guideline", "consensus", "statement", "regulation", and "recommendation" were used in both the "MeSH" and "Free-text" fields. We searched Chinese guidelines for headache disorders manually, as these were published in books and government documents. Using PubMed and EMBASE as examples, the search process was: For PubMed: #1 "Headache"[Mesh] OR "Headache Disorders" [Mesh] #2 "cephalalgia" OR "headache" OR "headache disorders" OR "migraine" OR "tension-type headache" #3 "Practice Guideline" [Publication Type] OR "Practice Guidelines as Topic"[Mesh] #4 "guideline" OR "consensus" OR "statement" OR "recommendation" #5 #1 OR #2 #6 #3 OR #4 #7 #5 AND #6 For EMBASE: #1 'headache'/exp OR 'headache' OR 'cephalalgia' OR 'migraine' OR 'headache disorders' OR 'tension-type headache' #2 'practice guideline'/exp OR 'practice guideline' OR 'consensus' OR 'statement' OR 'recommendation' #3 #1 AND #2
© 2018 JTCM. This is an open access article under the CC BY-NC-ND license.
Keywords: Practice guideline; Headache; Medicine, Chinese Traditional; Appraisal of Guidelines for Research and Evaluation Ⅱ instrument
INTRODUCTION Headache is one of the most common neurological disorders encountered by clinicians.1 According to the World Health Organization (WHO), headaches are among the five most common clinical disorders worldwide. Evidence suggests that active headaches affect more than 46% of the global population, and the lifetime prevalence of headache is more than 90%. Higher prevalence of headaches is reportedly associated with greater social, economic, and family burden.2-5 Traditional Chinese Medicine (TCM) has a long history in treating headache disorders. Many treatments, such as herbal TCM treatments and physical TCM therapy, are effective in treating headaches and have therefore been the focus of studies in modern medicine.6 Several clinical practice guidelines (CPGs) for treating headache with TCM have been developed. Many studies have suggested that CPGs can improve clinical practice and reduce healthcare costs,7,8 but several different CPGs may be published on the same topic. If these guidelines are conflicting, this may affect the confidence of clinicians.9 To our knowledge, no critical appraisal has been performed on the guidelines covering TCM treatment for headaches. There are various guideline assessment tools available, but the Appraisal of Guidelines for Research and Evaluation (AGREE) Ⅱ instrument is the most appropriate for guideline appraisal.10 This study aimed to appraise the methodological quality of guidelines for the treatment of headache produced over the last two decades, and covering interventions using TCM. We hoped to determine the acceptance level worldwide of the different interventions using TCM, and also obtain realistic efficacy data about TCM treatments. By appraising and summarizing the current CPGs for TCM headache treatment, we can determine whether a further guideline for the treatment of headache with TCM is needed.
Inclusion and exclusion criteria We included all Chinese and English language CPGs that provided recommendations on the diagnosis, treatment, and management of headache disorders, and met the definition of guidelines, as described by the Institute of Medicine.9 Guidelines were excluded if they met any of the following criteria: (a) did not include TCM therapies for headaches, including both herbal (Chinese herbal formulas, Chinese patent medicine, and herbal extracts) and physical TCM therapies (acupuncture, moxibustion, massage, manipulation, and osteopathy); (b) translations of international guidelines into Chinese, adaptations of international guidelines for China, ab-
METHODS Information sources We searched for guidelines on headache disorders in PubMed, EMBASE, Web of Science, and four Chinese academic electronic databases, the Chinese Biomedical Literature Database (CBM), China National KnowlJTCM | www. journaltcm. com
340
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
lines. The intra-class correlation coefficients (ICCs) were examined to assess the inter-rater reliability of the four appraisers within each domain. The ICCs ranged between 0 (completely unreliable) and 1 (completely reliable). The reliability was considered poor where the ICC was less than 0.40, and excellent where it was more than 0.75.
stracts or summary reports of international guidelines, and other explanatory or evaluation reports of guidelines; and (c) beta versions or older versions of guidelines from the same source. Literature screening and review data extraction Using the inclusion and exclusion criteria, all records were classified using Endnote (Version X7, Thomson Reuters, USA), and duplicate studies were discarded. Next, two of the authors (Luo Hao and Yao Sha) independently screened all the search records by browsing the title and abstract using the predefined inclusion and exclusion criteria. We then obtained the full text of possible guidelines to determine whether they were eligible, and extracted the general characteristics of each CPG included. Disagreements were resolved by consensus between the two authors or by a third expert (Li Hui).
Statistical analysis We used mean and standard deviation (SD) values to show the proportion of standardized scores for each domain in each guideline. We also calculated the overall mean scores for each domain for all guidelines and the Chinese guidelines. The consistency of the four appraisers for each guideline and across all the guidelines was measured using ICCs and 95% confidence intervals (CIs). 13 Subgroup analyses were performed by year of publication, country, type (consensus-based or evidencebased), grade of evidence, level of recommendation, and economic assessment, using an independent t-test or one-way analysis of variance. All the tests were two-sided, and a P value of < 0.05 was considered statistically significant. All the statistical analyses used SPSS version 19.0 (IBM Inc, New York).
Quality evaluation Each eligible guideline was assessed using the AGREE II instrument, which consists of 23 items across six domains, and two overall assessment items. Each domain assesses a different dimension of guideline quality. Each item was scored from 1 (strong disagreement) to 7 (strong agreement). The standardized scores for each domain were calculated as percentages using the following formulae: Maximum possible score = 7 (strongly agree) × No. of items within a domain × No. of appraisers; Minimum possible score = 1 (strongly disagree) × No. of items within a domain × No. of appraisers; and Observed score = Overall scores of all the appraisers:11,12
RESULTS Figure 1 shows a flowchart of the search and selection process. A total of 13 630 articles were checked across all possible sources. After removing duplicate articles, the remaining 4427 articles were screened by assessing the title and abstract, to give a shortlist of 96 articles for screening by reviewing the entire text. A total of 24 guidelines met our inclusion and exclusion criteria, of which 23 were evaluated using the AGREE Ⅱ instrument, because NAPNP 200725,26 was in two parts. The general characteristics of the eligible guidelines are shown in Table 1. Seven guidelines (30.4%) were developed in China, six (26.1% ) in the USA, and five (21.7% ) in the UK. Only four (17.4% ) were TCM guidelines; the others were modern medicine guidelines touching on TCM treatments. In total, 18 guidelines (78.3% ) were evidence-based and the other five (21.7% ) were consensus-based. Fourteen (60.9% ) guidelines focused on migraine. Only 12 (52.2% ) guidelines included the level of evidence and rated the recommendations. Economic assessment data were only available in seven guidelines (30.4%). The proportions of AGREE Ⅱ standard scores for the eligible guidelines are shown in Table 2. The overall consistency of the four appraisers was good [overall ICC (95% CI), 0.84 (0.82-0.86); data not shown]. After estimation of the scores for each domain and the calculation of the global scores for each guideline, two guidelines were "recommended", 12 were "recommended with modifications", and nine were "not recom-
The overall guideline recommendation considered all the domain scores. Using the standard criteria, a CPG was "recommended" if the scores of five domains were at least 60%; "recommended with modification" if the scores of four domains were at least 30% and the score of one other domain was at least 50%; and "not recommended" if the scores of three domains were 30% or less. Data collection process A pre-designed table with two components (general characteristics and scores of AGREE Ⅱ assessment) was used to extract data from eligible headache guidelines. The general characteristics included title, year of publication, country, organization, nature (Traditional Chinese Medicine or modern medicine), type (evidence-based or consensus-based guidelines), topic covered (syndromes of headache), grade of evidence, level of recommendation, and economic assessment. Synthesis of the results Each eligible guideline was independently assessed by four appraisers (Luo Hao, Wang Yangyang, Yao Sha, and Xu WenJie). Prior to the formal assessment, we conducted a pre-assessment by choosing five guideJTCM | www. journaltcm. com
341
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
Records identified through electronic database searching (n = 3627)
Search in website and other resource (n = 68)
Records after duplicates removed (n = 2737)
Records excluded by checking titles and abstract (n = 2641)
Records after screened (n = 96)
Guidelines excluded by checking full-text articles (n = 72)
Eligible guidelines after screened (n = 24) Figure 1 Flow diagram showing the search and selection process in the study
mended". The overall mean score showed that domain 1 was the highest of the six domains, and domains 5 and 6 were the lowest; the scores for the other domains ranged from 39% to 50%. The mean score for the four TCM guidelines was lower than the overall mean score in each domain, although domains 5 and 6 were again the lowest. Table 3 shows the results of the subgroup analysis of the eligible guidelines. The year of publication was not statistically significant, although the mean scores for the guidelines published in and after 2011 were better than those for guidelines published before 2011. The guidelines developed in China had significantly lower scores than those developed in the USA for domains 1 and 6 (P < 0.05), and those developed in the UK for all the domains except domain 2 (P < 0.01). Evidence-based guidelines had higher quality scores than consensus-based guidelines for all the domains (P < 0.05). The scores for guidelines with an evidence grade were better, particularly in domains 1 to 4 (P < 0.05). Recommended guidelines were significantly likely to have higher scores in domains 4 (P < 0.05). Guidelines with an economic assessment were significantly likely to be better in all domains except domain 6. Table 4 shows the recommendations on the use of JTCM | www. journaltcm. com
TCM in the two "recommended" modern medicine guidelines and two "recommended with modification" TCM guidelines. Physical therapies were more likely to be accepted by modern medicine. Herbal therapies were not recommended in modern medicine guidelines because of insufficient evidence, adverse effects, or poor efficacy. Herbal formulas, Chinese patent medicine, acupuncture, moxibustion, massage and manipulation are, however, recommended in TCM guidelines. We compared the guideline quality in this study with previous related studies. Figure 2 shows comparisons with the studies of Alonso-Coello et al,38 Chen et al,39 Wei et al,40 and Li et al,41 who assessed the quality of international and Chinese guidelines in the last two decades. The quality of the headache guidelines was better than that of other domestic guidelines, but the mean scores for the domestic headache guidelines were poorer than international guidelines for domains 3, 5, and 6. However, the mean domain scores in international guidelines for domains 5 and 6 were still less than 30%. We also compared the quality assessment results for the four TCM guidelines with those from previous studies (Figure 3). The scores for the TCM guide342
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review Table 1 General characteristics of the eligible guidelines Organization year PMDP 200614 PTDT 200715 CACM1 2008
16
CACM2 2008
17
CACMS/CAAM 2011
18
CASP 201119 CACMS 2011
20
CHS 1998
Country
Nature
Type
Topic
Grade of evidence
Level of recommendation
Economic assessment
China
MM
CB
M
No
No
No
China
MM
CB
TTH
No
No
No
China
TCM
CB
M
No
No
No
China
TCM
CB
PH
No
No
No
China
TCM
EB
M
Yes
Yes
No
China
MM
EB
M
Yes
Yes
No
China
TCM
EB
M
Yes
Yes
No
Canada
MM
EB
M
Yes
Yes
Yes
22
USA
MM
EB
M
Yes
Yes
No
AAN2 200023
USA
MM
EB
M
Yes
No
No
USA
MM
EB
M
Yes
Yes
Yes
USA
MM
EB
M
No
No
No
USA
MM
EB
H
Yes
Yes
Yes
ACP/ASIM 201228
USA
MM
EB
M
No
No
Yes
SIGN 2008
UK
MM
EB
H
Yes
Yes
Yes
21
AAN1 2000
AAN3 2000
24
NAPNP 2007 ICSI 2011
25,26
27
29
EFNS 2009
UK
MM
EB
M
No
Yes
No
31
BASH 2010
UK
MM
EB
H
No
No
Yes
EFNS 201032
UK
MM
EB
TTH
Yes
Yes
No
NICE 2012
UK
MM
EB
H
Yes
No
Yes
34
ISSH 2012
Italy
MM
EB
PH
Yes
Yes
No
35
DHS 2012
Denmark
MM
EB
H
No
No
No
France
MM
EB
M
No
Yes
No
Latin America
MM
CB
M
No
No
No
30
33
FSSMH 201436 LAPCMT 2013
37
Notes: MM: modern medicine; TCM: Traditional Chinese Medicine; CB: consensus-based; EB: evidence-based; H: headache; PH; primary headache; M: migraine; TTH: tension-type headache; PMDP: Panel of Migraine Diagnosis and Prevention; PTDT: Panel of TTH Diagnosis and Treatment; CACM1/2: China Association of Chinese Medicine; CACMS/CAAM: China Academy of Chinese Medical Sciences/ China Association of Acupuncture-Moxibustion; CASP: Chinese Association for the Study of Pain; CACMS: China Academy of Chinese Medical Sciences; CHS: Canadian Headache Society; AAN1/2/3: American Academy of Neurology; NAPNP: National Association of Pediatric Nurse Practitioners; ICSI: Institute for Clinical Systems Improvement; ACP/ASIM: American College of Physicians/American Society of Internal Medicine; SIGN: Scottish Intercollegiate Guidelines Network; EFNS: European Federation of Neurological Societies; BASH: British Association for the Study of Headache; NICE: National Institute for Health and Clinical Excellence; ISSH: Italian Society for the Study of Headaches; DHS: Danish Headache Society; FSSMH: French Society for the Study of Migraine Headache; LAPCMT: Latin American Panel of Chronic Migraine Treatment.
DISCUSSION
lines in this study were lower than those reported by Choi et al 42 for all the domains but better than those reported by Chen et al 43 in domains 2 and 3. They were, however, lower than 30% in domains 3, 5, and 6. There are therefore no high quality TCM headache guidelines in China, as the scores for all the TCM guidelines were less than 40% in all domains. As Figure 3 shows, the scores found in two studies were higher than the others. After a thorough investigation, we found that the authors of the two studies had participated in the process of developing some of the guidelines included but had not reported this as a conflict of interest.44,45 JTCM | www. journaltcm. com
Results of the appraisal of high quality CPGs The ICC (95% CI) of the NICE headache guideline33 (0.59, 0.23-0.81) — one of the two recommended guidelines — was the lowest of the eligible guidelines. The four appraisers had different professional academic backgrounds, and we believe that the structures of the reporting guidelines and the miscellaneous content may have led to diversity. We suggest that it is increasingly important to have a uniform format for the reporting guidelines, and also to develop alternative versions for target users from different backgrounds, such as patients, clinicians, methodologists, and policy makers. 343
June 15, 2018 | Volume 38 | Issue 3 |
JTCM | www. journaltcm. com
344 32.6 (21.2)
31.9 (15.8)
NA
23.8 (21.7)
4.7 2.1 4.7 5.7 46.4 24.5 38.5 41.7 40.6 40.6 9.4 45.3 27.6 27.6 71.9 47.9 21.9 46.4 83.3 45.8 19.3 35.4 21.9 33.4 (21.0)
Domain 3: rigor of development
39.6 (19.7)
18.1 18.1 20.8 25.0 61.1 54.2 51.4 47.2 48.6 63.9 3.8 65.3 63.9 63.9 86.1 59.2 56.9 59.7 91.7 69.4 50.0 52.8 29.2 49.8 (21.9)
Domain 4: clarity of presentation
Scores of each domain (%)
7.3 (8.1)
5.2 2.1 2.1 1.0 18.8 29.2 7.3 16.7 18.8 20.8 20.8 47.9 21.9 21.9 67.7 12.5 36.5 17.7 78.1 25.0 34.9 26.0 14.6 23.8 (19.3)
Domain 5: applicability
1.6 (3.1)
Domain 6: editorial independence 0.0 0.0 0.0 0.0 0.0 0.0 6.3 18.8 22.9 22.9 0.0 52.1 16.6 16.6 60.4 60.4 10.4 52.1 77.1 37.5 31.3 43.8 18.8 24.2 (23.7) NA
Overall guideline assessment NR NR NR NR RM NR RM NR RM RM NR RM NR NR Recommended RM RM RM Recommended RM RM RM NR NA
Notes: SD: standard deviation; NA: not applicable; NR: not recommended; RM: recommended with modification; MM: modern medicine; TCM: Traditional Chinese Medicine; CB: consensus-based; EB: evidence-based; H: headache; PH: primary headache; M: migraine; TTH: tension-type headache; PMDP: panel of migraine diagnosis and prevention; PTDT: panel of TTH diagnosis and treatment; CACM1/2: China Association of Chinese Medicine; CACMS/CAAM: China Academy of Chinese Medical Sciences/China Association of Acupuncture-Moxibustion; CASP: Chinese Association for the Study of Pain; CACMS: China Academy of Chinese Medical Sciences; CHS: Canadian Headache Society; AAN1/2/3: American Academy of Neurology; NAPNP: National Association of Pediatric Nurse Practitioners; ICSI: Institute for Clinical Systems Improvement; ACP/ASIM: American College of Physicians/American Society of Internal Medicine; SIGN: Scottish Intercollegiate Guidelines Network; EFNS: European Federation of Neurological Societies; BASH: British Association for the Study of Headache; NICE: National Institute for Health and Clinical Excellence; ISSH: Italian Society for the Study of Headaches; DHS: Danish Headache Society; FSSMH: French Society for the Study of Migraine Headache; LAPCMT: Latin American Panel of Chronic Migraine Treatment.
33.3 11.1 15.3 13.9 55.6 38.9 45.8 41.7 37.5 43.1 15.3 65.3 43.1 43.1 69.4 30.6 45.8 33.3 76.4 29.2 44.4 51.4 33.3 39.5 (17.1)
34.7 23.6 19.4 18.1 50.0 43.1 40.3 48.6 65.3 70.8 38.9 76.4 51.4 51.4 77.8 55.6 61.1 58.3 88.9 55.6 52.8 61.1 48.6 52.1 (18.0)
0.83 (0.68, 0.92) 0.84 (0.70, 0.93) 0.81 (0.64, 0.91) 0.80 (0.61, 0.90) 0.89 (0.78, 0.95) 0.89 (0.79, 0.95) 0.89 (0.80, 0.95) 0.83 (0.67, 0.92) 0.77 (0.57, 0.89) 0.81 (0.63, 0.91) 0.78 (0.59, 0.90) 0.62 (0.29, 0.82) 0.81 (0.64, 0.91) 0.81 (0.64, 0.91) 0.65 (0.35, 0.84) 0.84 (0.70, 0.93) 0.86 (0.73, 0.93) 0.77 (0.56, 0.89) 0.59 (0.23, 0.81) 0.79 (0.61, 0.90) 0.72 (0.47, 0.87) 0.63 (0.31, 0.83) 0.67 (0.37, 0.84) NA
PMDP 200614 PTDT 200715 CACM1 200816 CACM2 200817 CACMS/CAAM 201118 CASP 201119 CACMS 201120 CHS 199821 AAN1 200022 AAN2 200023 AAN3 200024 NAPNP 200725,26 ICSI 201127 ACP/ASIM 201228 SIGN 200829 EFNS 200930 BASH 201031 EFNS 201032 NICE 201233 ISSH 201234 DHS 201235 FSSMH 201436 LAPCMT 201337 Overall mean (SD) score Mean (SD) score of the Chinese guidelines
Domain 2: stakeholder involvement
Domain 1: scope and purpose
ICC (95% CI)
Organization year
Table 2 AGREE Ⅱ domain scores and ICCs (95% CI) of the eligible guidelines
Luo H et al. / Systematic Review
June 15, 2018 | Volume 38 | Issue 3 |
JTCM | www. journaltcm. com
345
5 (21.7) 18 (78.3) -
CB
EB
P value
7 (30.4) 16 (69.6)
Yes
-
No
46.7 (21.7)
11 (47.8)
No
P value 45.2 (15.0)
67.9 (14.8)
0.100
58.0 (12.1)
0.009
42.3 (16.3)
61.1 (15.0)
<0.001
58.6 (13.4)
28.8 (12.8)
0.010
<0.001
12 (52.2)
Yes
-
-
MM
P value
P value
19 (82.6)
TCM
11 (47.8)
56.4 (15.7)
4 (17.4)
P value (China vs UK)
12 (52.2)
31.9 (15.8)
-
Yes
68.3 (14.4)
5 (21.7)
UK
P value (China vs USA)
No
57.3 (14.6)
6 (26.1)
USA 0.002
32.7 (12.5)
7 (30.4)
China
58.0 (15.1)
10 (43.8) 0.281
48.5 (19.8)
13 (56.5)
Scope and purpose
P value
≥2011
<2011
n (%)
32.7 (13.4)
55.0 (14.8)
27.3 (17.6)
47.5 (22.4)
0.008
21.9 (23.6)
33.6 (19.4) 0.111
44.0 (10.9)
<0.001
18.2 (14.7)
47.4 (15.5)
0.001
40.5 (17.5)
7.8 (8.0)
0.333
35.5 (20.1)
23.8 (21.7)
0.005
0.140
54.3 (24.0)
35.0 (12.7)
18.1 (18.4)
0.292
38.8 (18.7)
29.3 (22.4)
Rigor of development
44.9 (13.2)
0.014
30.7 (14.7)
47.6 (15.5)
0.004
44.5 (15.2)
21.4 (11.0)
0.392
40.9 (16.4)
32.6 (21.2)
0.700
0.371
51.1 (20.9)
40.7 (14.9)
30.6 (17.4)
0.260
48.3 (14.4)
32.7 (16.3)
Stakeholder involvement
Scores of each domain [% (SD)]
42.0 (19.6)
67.9 (15.7)
0.024
38.7 (25.8)
60.0 (10.6)
0.003
36.2 (20.8)
62.4 (14.3)
<0.001
57.5 (18.1)
22.2 (4.8)
0.310
52.0 (22.2)
39.6 (19.7)
0.009
0.241
70.7 (16.7)
49.5 (21.5)
35.5 (19.1)
0.081
58.9 (16.1)
42.9 (23.7)
Clarity of presentation
16.1 (10.3)
41.4 (24.2)
0.560
21.4 (22.6)
26.0 (16.6)
0.067
16.1 (12.9)
30.8 (22.0)
0.010
29.0 (18.6)
5.0 (5.6)
0.059
27.3 (19.3)
7.3 (8.1)
0.006
0.130
42.5 (29.4)
22.8 (12.2)
9.4 (10.6)
0.160
30.3 (20.1)
18.8 (17.9)
Applicability
18.6 (21.1)
36.9 (25.9)
0.121
16.1 (23.1)
31.6 (22.6)
0.140
16.5 (20.7)
31.3 (24.8)
0.001
29.9 (23.5)
3.8 (8.4)
<0.001
29.0 (23.3)
1.6 (3.1)
<0.001
0.026
52.1 (25.0)
20.8 (16.7)
0.9 (2.4)
0.471
28.3 (24.9)
21.0 (23.1)
Editorial independence
P value 0.002 0.033 0.030 0.088 0.003 0.006 Notes: SD: standard deviation; AGREE: Appraisal of Guidelines for Research and Evaluation Ⅱ instrument; TCM: Traditional Chinese Medicine; MM: modern medicine; CB: consensus-based; EB: evidence-based.
Economic assessment
Recommendation level
Evidence grade
Type
Nature
Country
Year
Variable
Table 3 AGREE Ⅱ domain scores for guidelines by subgroups
Luo H et al. / Systematic Review
Domains with low scores As shown in Table 2, the overall mean (SD) scores of the headache guidelines were poor for domains 3, 5, and 6. The rigor of development domain (domain 3) is considered to be the most essential domain in the development of guidelines. It consists of eight items and appraises the methods used for evidence collection, grading, and summary, as well as the formulation of recommendations.11 The evidencebased guidelines had better scores than the consensus-based guidelines for all the domains (P < 0.05). Only one non-Chinese guideline was consensus-based. This trend suggests that evidence-based guidelines are becoming more mainstream. In Table 4, the four "recommended" or "recommended with modification" guidelines were all evidence-based, although the evidence grading and formulation of recommendations differed. These findings suggest that there is no common,
June 15, 2018 | Volume 38 | Issue 3 |
CACMS 2011
20
CACMS/CAAM 2011
Notes: NA: not applicable; NR: not recommended; SIGN: Scottish Intercollegiate Guidelines Network; NICE: National Institute for Health and Clinical Excellence; CACMS/CAAM: China Academy of Chinese Medical Sciences/China Association of Acupuncture-Moxibustion; CACMS: China Academy of Chinese Medical Sciences.
Recommended Recommended Recommended Recommended
NA
Recommended
NA Recommended NA NA
NA
Recommended
Recommended NA NA NA
SIGN 2008
18
NICE 201233
NR
Recommended
Insufficient Evidence NA Recommended NA NA
NR
Moxibustion Acupuncture Herbal extract Chinese patent medicine Herbal formula
29
Herbal therapy
Organization year
Table 4 Recommendations of two "recommended" modern medicine guidelines and two "recommended with modification" TCM guidelines
Physical therapy
Massage or manipulation
Luo H et al. / Systematic Review
JTCM | www. journaltcm. com
sensible, and transparent approach to grading the quality (or certainty) of evidence and the strength of the recommendations. The applicability domain (domain 5) evaluates whether the guideline provides advice on implementing recommendations. It considers the potential resources, facilitators, and barriers and provides monitoring or auditing criteria derived from the key recommendations.11 However, most of the guidelines included in this study did not discuss the applicability, or highlight the tools or methods required to facilitate or promote the guidelines. The mean score (SD) for domain 5 was therefore the lowest of the domains [23.8 (19.3); Figure 2], which is consistent with the results of previous studies.38-40,42 The editorial independence domain (domain 6) assesses the bias related to conflict of interests in the development of guidelines. 11 In our study, the mean score (SD) for domain 6 was 24.2 (23.7), and most of the guidelines included did not report any conflicts of interest or financial conflicts.
In a previous study, 87% of guideline developers were found to have some form of affiliation with the pharmaceutical industry, 58% had received financial support, and 38% had served as an employee or consultant for a pharmaceutical company.46 Conflicts of interest are often hard to detect, but evidence suggests that they can influence the recommendations in a guideline.49 The results suggest that developers should focus on the "rigor of development", "applicability", and "editorial independence" when developing TCM headache guidelines in the future. They should also use widely-recognized methods, such as the GRADE approach, to rate the quality of the best available evidence and develop healthcare recommendations for the establishment of CPGs.50 The potential advantages and disadvantages of treatments, such as patient and social values or preferences, should be carefully considered and combined with the evidence to formulate the final recommendations.51 Before developing a new guideline, it is important to list the funding sources, or provide formal conflict of interest statements for all the authors for discussion and public disclosure.47,52,53 Discussion of the two TCM guidelines With the support of the World Health Organization/ Western Pacific Regional Office (WHO/WPRO), the CACMS/CAAM 2011 and CACMS 2011 were the first evidence-based guidelines on TCM to be adopted in China. However, these two guidelines had poor scores for domains 5 and 6. Table 4 shows that they recommended the use of herbal therapies to treat headache. They differed from the two "recommended" guidelines, which described the method for the extraction of herbs, acupuncture, massage, and manipulation in the prophylaxis of headache. We believe that there are two main reasons for the difficulty in forming recommendations for traditional Chinese treatments in the guidelines. First, it is difficult to conduct high-quality randomized clinical trials because of the personalized nature of treatment with TCM.54,55 Second, randomized clinical trials are one of the major sources of clinical evidence, but their current quality — although better than before — remains unsatisfactory.56 The AGREE Ⅱ instrument is a tool for assessing guideline development, reporting, and evaluation.11 However, if a TCM guideline is fully developed by this instrument, we think that the clinical practice characteristics of TCM may not be appropriately reflected. Different methods should therefore be considered when developing a guideline, and the existing guidelines on headache should be critically appraised to develop high-quality evidence-based TCM guidelines for headache, suitable for use with individuals.57-60 We hope that ongoing work will lead to the development of high-quality TCM guidelines and contribute to the incorporation of effective traditional therapies into clinical practice. 346
June 15, 2018 | Volume 38 | Issue 3 |
Mean scores of each domain (%)
Luo H et al. / Systematic Review 70 60 50 40 30 20 10 0
e cop
S
and
ent
ent
ose
rp pu
er old
em olv inv
igo
f ro
pm elo v e d
of
t
nta
se pre
lop
ve ion
Mean scores of each domain (%)
R keh ty Sta lari In our study Alonso 2010 C Chen 2012 Figure 2 Comparison of assessed quality between headache guidelines and others
p Ap
e enc
y
ilit
b lica
Ed Wei 2013
nd
al i
ri ito
nd epe
Li 2013
120 100 80 60 40 20 0
t y ce bilit t men op nden elop nvel men plica e v e o p i e p v t l e A d a o d t r of r inv al in resen Scop Rigo olde itori of p d h e y E t k i Sta Clar TCM Guildline Choi 2015 Chen 2014 Fang 2014 Yu 2011 Figure 3 Comparison of the assessment of quality of the TCM guidelines between this study and other TM studies se
urpo
dp e an
In conclusion, we found that modern medicine guidelines did not recommend using Chinese herbal therapies, but the two recommended guidelines, supported by clinical evidence, suggest the use of physical TCM therapies for treating headache. The two TCM guidelines, which were "recommended with modification" following assessment, recommended the use of Chinese herbal and physical therapies to treat headaches. Healthcare professionals should choose suitable and high-quality CPGs to guide their clinical practice. The overall quality of TCM headache guidelines was low in China, but evidence-based guidelines are gradually becoming mainstream. Guideline developers should carefully consider the three most important domains, covering the rigor of development, applicability, and editorial independence.
Strengths and limitations Our study has several strengths. First, the systematic review of guideline quality covered a wide range of guidelines on headache disorders, from around the world. Second, our team included a methodologist and neurologist who had prior experience of developing guidelines. The study also had some limitations, however. First, we focused on guidelines that covered TCM therapies for headache and were published in Chinese or English. Most of the guidelines included were therefore published in Europe, America, and China. These may not be representative of all the available guidelines on headache disorders. Second, the supplementary materials and background information on the guidelines could not be searched and assessed thoroughly, because different organizations used different measures to report the supporting information. This could lead to an underestimation of guideline quality in certain domains. Third, the AGREE Ⅱ instrument used to assess the overall quality of the guidelines did not provide any criteria or instructions. This makes it difficult for assessors to reach a consensus on the overall quality of the guidelines.10 JTCM | www. journaltcm. com
ACKNOWLEDGMENTS We thank Yao Liang, Wei Dang, Wang Qi, Wang Xiaoqin, Evidence-Based Medicine Center, School of Basic Medical Sciences, Lanzhou University, Lanzhou, Gansu, China, for their general assistance and help in the revision of the manuscript. 347
June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
REFERENCES 1
2
3
4
5
6
7
8
9
10
11
12
13
14 15 16
Headache Classification Committee of the International Headache Society. The International Classification of Headache Disorders. 3rd ed (beta version). Cephalalgia 2013; 33(9): 629-808. Vos T, Flaxman AD, Naghavi M, et al. Years lived with disability (YLDS) for 160 sequelae of 289 diseases and injuries: 1990-2010: a systematic analysis for the Global Burden of Disease Study 2010. The Lancet 2014; 384(9943): 582-582. Stovner LJ, Hagen K, Jensen R, et al. The global burden of headache: a documentation of headache prevalence and disability worldwide. Cephalalgia 2007; 27: 193-210. Rasmussen BK, Jensen R, Schroll M, et al. Epidemiology of headache in a general population — a prevalence study. J Clin Epidemiol 1991; 44: 1147-1157. Latinovic R, Gulliford M, Ridsdale L. Headache and migraine in primary care: consultation, prescription, and referral rates in a large population. J Neurol Neurosurg Psychiatry 2006; 77(3): 385-387. Wang YY, Zhang YL. A review of the clinical research on the treatment of headache in Chinese medicine in the past 10 years. Beijing Zhong Yi Yao Da Xue Xue Bao 1992; (1): 5-12. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. The Lancet 1993; 342(8883): 1317-1322. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ 1999; 318(7182): 527-530. Graham R, Mancher M, Miller Wolman D, et al. Clinical practice guidelines we can trust. Washington, DC: National Academy of Sciences, 2011: 20, 50. Vlayen J, Aertgeerts B, Hannes,K, Sermeus W, Ramaekers D. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care 2005; 17(3): 235-242. Brouwers MC, Kho ME, Browman GP, et al. AGREE Ⅱ: advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J 2010; 182(18): E839-E842. AGREE. AGREE Next Steps Consortium (2009). The
17
18
19
20
21
22
23
24
25
26
AGREE Ⅱ Instrument UPDATE September, 2013. Available from URL: http://www.agreetrust.org/wp-content/uploads/2013/10/AGREE-II-Users-Manual-and-23-item-Instrument_2009_UPDATE_2013.pdf. Pan X, Ni Z. Application of intra-class correlation coefficient to reliability assessment. Hua Xi Yi Ke Da Xue Xue Bao 1999; 30(1): 62-63, 67. Li YS. The diagnosis and management of migraine: consensus. Chin J Intern Med 2006; 45(8): 694-696. Yu SY. The diagnosis and management of tension-type headache: consensus. Chin J Neuro 2007; 40(7): 496-497. China Association of Chinese Medicine. Guidelines for di-
JTCM | www. journaltcm. com
27
28
29
30 348
agnosis and treatment of common internal diseases in chinese medicine: disease of modern medicine (migraine). Beijing: China Press of Traditional Chinese Medicine 2008: 275-278. China Association of Chinese Medicine. guidelines for diagnosis and treatment of common internal diseases in chinese medicine: symptoms in chinese medicine (headache). Beijing: China Press of Traditional Chinese Medicine 2008: 126-128. Wu ZC. Evidence-based Guidelines of Clinical Practice in Chinese Medicine: Acupuncture (Migraine), Beijing: China Press of Traditional Chinese Medicine 2014, 144-162. Yu SY. Guidelines for Diagnosis and Treatment of Migraine. Zhong Guo Teng Tong Yi Xue Za Zhi 2011; 17 (2): 65-86. China Academy of Chinese Medical Sciences, Evidence-based Guidelines of Clinical Practice in Chinese Medicine: Internal Medicine (Migraine), Beijing: China Press of Traditional Chinese Medicine, 2011: 210-226. Pryse-Phillips WE, Dodick DW, Edmeads JG, et al. Guidelines for the non-pharmacologic management of migraine in clinical practice. CMAJ 1998; 159(1): 47-54. Ramadan NM, Silberstein SD, Freitag FG, et al. Evidence-based guidelines for migraine headache in the primary care setting: pharmacological management for prevention of migraine. Headache 2000; 46(7): 11511160. Campbell JK, Penzien DB. Evidenced-based guidelines for migraine headache: behavioral and physical treatments. American Academy of Neurology. Available from URL: http://tools.aan.com/professionals/practice/pdfs/gl 0089.pdf. Silberstein SD. Practice Parameter: Evidence-based guidelines for migraine headache. Neurology 2000; 55(6): 754-762. gunner kb, smith hd, ferguson le, et al. practice guideline for diagnosis and management of migraine headaches in children and adolescents: part one. J Pediatr Health Care 2007; 21(1): 327-332. Gunner KB, Smith HD, Ferguson LE, et al. Practice guideline for diagnosis and management of migraine headaches in children and adolescents: Part two. J Pediatr Health Care 2007; 22(1): 52-59. Institute for Clinical Systems Improvement. health care guideline: diagnosis and treatment of headache 10th edition 2011. Available from URL: https://www.icsi.org/_asset/qwrznq/Headache.pdf. Snow V, Weiss K, Wall EM, et al. Pharmacologic management of acute attacks of migraine and prevention of migraine headache. Ann Intern Med 2002; 137(10): 840-849. Scottish Intercollegiate Guidelines Network. Diagnosis and management of headache in adults 2008. Available from URL: http://www.sign.ac.uk/pdf/sign107. Evers S, Afra J, Frese A, et al. EFNS guideline on the drug June 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
treatment of migraine-revised report of an EFNS task force. Eur J Neurol 2009; 16(9): 968-981. MacGregor EA, Steiner TJ, Davies PTG. Guidelines for all healthcare professionals in the diagnosis and management of migraine, tension-type headache, cluster headache and medication-overuse headache. Available from URL: http://www.bash.org.uk/wp-content/uploads/2012/07/10102-BASH-Guidelines-update-2_v5-1indd.pdf. Bendtsen L, Evers S, Linde M, et al. EFNS guideline on the treatment of tension-type headache-Report of an EFNS task force. Eur J Neurol 2010; 17(11); 1318-1325. National Institute for Health and Care Excellence. Diagnosis and management of headache in young people and adults 2012. Available from URL: https://www.nice.org. uk/guidance/cg150/update/CG150/documents/headaches-full-guideline-for-consultation2. Sarchielli P, Granella F, Prudenzano M, et al. Italian guidelines for primary headaches: 2012 revised version. J Headache Pain 2012; 13(2): 31-70. Bendtsen L, Birk S, Kasch H, et al. Reference programme: diagnosis and treatment of headache disorders and facial pain (2nd Edition). J Headache Pain 2012; 13 (suppl 1): 1-29. Géraud G, Lantéri-Minet M, Lucas C, et al. French guidelines for the diagnosis and management of migraine in adults and children. Clinical Therapeutics 2004; 26(8): 1305-1318. Giacomozzi AR, Vindas AP, Jr SA, et al. Latin American consensus on guidelines for chronic migraine treatment. Arq Neuro-psiquiatr 2013; 71(7): 478-486. Alonso-Coello P, Irfan A, SolàI, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. QualSaf Health Care 2010; 19(6): e58. Chen YL, Yao L, Xiao XJ, et al. Quality assessment of clinical guidelines in China: 1993-2010. Chin Med J 2012; 125(20): 3660-3664. Wei D, Wang XQ, Wu QF, et al. Quality evaluation on chinese clinical practice guidelines in 2011. Chin J Evid-based Med 2013; 13(6): 760-763. Li N, Yao L, Wu QF, et al. Quality evaluation of clinical practice guidelines published in journals of mainland China during 2012-2013. Chin J Evid-based Med 2015; 15 (3): 259-263. Choi TY, Choi J, Lee JA, Ji HJ, Bongki P, Myeong SL. The quality of clinical practice guidelines in traditional medicine in Korea: appraisal using the AGREE Ⅱ instrument. Implement Sci 2015; 10(1): 104. Chen H, Li GL, Xu WT, Xu B. Quality Assessment of Clinical Practice Guidelines of Acupuncture in China. Chin J Evid-based Med 2014; 14(6): 772-775. Fang YG, Bai Y, Liu BY, Wang F, Xue W. Quality assessment on guidelines of clinical practice in acupuncture and moxibustion: a study based on AGREE Ⅱ. Zhong Guo Zhen Jiu 2014; 34(6): 599-601. Yu WY, Han XJ, Zhang H, et al. Study on quality assess-
JTCM | www. journaltcm. com
46
47
48 49
50
51
52
53
54
55
56
57
58
59
349
ment of TCM clinical practice guidelines based on AGREE instrument. Shi Jie Ke Xue Ji Shu-Zhong Yi Yao Xian Dai Hua 2011; 13(4): 596-600. Papanikolaou GN, Baltogianni MS, Contopoulos-Ioannidis DG, et al. Reporting of conflicts of interest in guidelines of preventive and therapeutic interventions. BMC Med Res Methodol 2011; 1(1): 3. Choudhry NK, Stelfox HT, Detsky AS. Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA 2002; 287(5): 612-617. Thompson DF. Understanding financial conflicts of interest. N Engl J Med 1993; 329(8): 573-576. Norris SL, Burda BU, Holmer HK, et al. Author's specialty and conflicts of interest contribute to conflicting guidelines for screening mammography. J Clin Epidemiol 2012; 65(7): 725-733. Brozek JL, Akl EA, Alonso-Coello P, Lang D, Jaeschke R, Williams JW; GRADE Working Group. Grading quality of evidence and strength of recommendations in clinical practice guidelines. Part 1 of 3. An overview of the GRADE approach and grading quality of evidence about interventions. Allergy 2009; 64(5): 669-677. Guyatt G, Oxman AD, Akl EA, et al. (2011) GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 64(4): 383-394. Schünemann HJ, Al-Ansary LA, Forland F, et al. Guidelines international network: principles for disclosure of interests and management of conflicts in guidelines. Ann Intern Med 2015; 163(7): 548-553. Campbell EG (2010) Public disclosure of conflicts of interest: moving the policy debate forward. Arch Intern Med 170(8): 667. Shekelle PG, Morton SC, Suttorp MJ, Buscemi N, Friesen C; Agency for Healthcare Research and Quality. Challenges in systematic reviews of complementary and alternative medicine topics. Ann Intern Med 2005; 142(12 Pt 2): 1042-1047. Practice and Policy Guidelines Panel, National Institutes of Health Office of Alternative Medicine. Clinical practice guidelines in complementary and alternative medicine. An analysis of opportunities and obstacles. Arch Fam Med 1997; 6(2): 149-154. He J, Du L, Liu G, et al. Quality assessment of reporting of randomization, allocation concealment, and blinding in Traditional Chinese Medicine RCTs: a review of 3159 RCTs identified from 260 systematic reviews. Trials 2011; 12(1): 286-298. Chen KJ, Jiang YR. Current status and problems in developing clinical guidelines for Chinese medicine and integrative medicine. Zhong Xi Yi Jie He Xue Bao 2009; 7(4): 301-305. Liu M, Yang J, Wang YP. Thoughts on the development and Implementing of Evidence-Based Guideline. Chin J Evid-based Med 2009; 9(2): 127-128. Wang B, Zhan SY, Liu BY. Problems and strategies in developing chinese medicine evidence-based clinical pracJune 15, 2018 | Volume 38 | Issue 3 |
Luo H et al. / Systematic Review
60
egies 2014-2023. Available from URL: http://www. who.int/medicines/publications/traditional/trm_strategy 14_23/en/.
tice guidelines. Chin J Integr Med 2011; 31(11): 1565-1569. World Health Organization. Traditional Medicine Strat-
JTCM | www. journaltcm. com
350
June 15, 2018 | Volume 38 | Issue 3 |