Should systematic reviews include non-randomized and uncontrolled studies? The case of acupuncture for chronic headache

Journal of Clinical Epidemiology 55 (2002) 77–85

Should systematic reviews include non-randomized and uncontrolled studies? The case of acupuncture for chronic headache Klaus Lindea,c,*, Michael Scholzb, Dieter Melchartc, Stefan N. Willicha a Institute for Social Medicine & Epidemiology, Charité Hospital, Humboldt-Universität, 10098 Berlin, Germany Institute for Medical Statistics & Epidemiology, Technische Universität, Ismaningerstr. 22, 81675 Munich, Germany c Centre for Complementary Medicine Research, Department of Internal Medicine II, Technische Universität, Kaiserstr. 9, 80801 Munich, Germany Received 1 December 2000; received in revised form 28 June 2001; accepted 6 July 2001 b

Abstract We aimed to investigate: (1) whether patient and intervention characteristics, design-independent quality aspects, and response rates differ between randomized and non-randomized trials of acupuncture for chronic headache; (2) whether non-randomized studies provide useful additional information (regarding long-term effects, prognostic factors, adverse effects, and generalizability); (3) reasons for potential differences in response rates. Studies including at least five patients and reporting clinical outcome data were identified through searches in Medline, Embase, the Cochrane Controlled Trials Register, other databases and checking of bibliographies. Twenty-four randomized trials and 35 non-randomized studies (five nonrandomized controlled cohort studies, 10 prospective uncontrolled studies, 10 case series, and 10 cross-sectional surveys) met the inclusion criteria. Studies were heterogeneous regarding patients, interventions, outcome measurements and results. On average, randomized trials had smaller sample sizes, met more quality criteria, and had lower response rates (0.59 [95% confidence interval 0.48–0.69] vs. 0.78 [0.72–0.83]). Whether randomized or not, studies meeting more quality criteria had lower response rates. Non-randomized studies did not have significantly longer follow-up periods, three included an analysis of prognostic variables, only one reported on adverse effects, and the degree of generalizability was unclear. In the case of acupuncture for chronic headache, non-randomized studies confirmed the finding of a systematic review of randomized trials that the treatment is likely to be effective but provided little relevant additional information on long-term effects, prognostic factors, and adverse effects. © 2002 Elsevier Science Inc. All rights reserved. Keywords: Meta-analysis; Acupuncture; Headache; Migraine; Randomized controlled trial

1. Introduction Systematic reviews play an increasingly important role in assessing the existing evidence for health care interventions. Most systematic reviews of prophylactic and therapeutic interventions include only randomized controlled trials (RCTs) which are generally considered as the gold standard for evaluation. However, the medical literature is full of non-randomized and uncontrolled studies. Should the results of these studies be dismissed completely? Or should they be considered as useful additional information for systematic reviews? It is obvious that it is highly problematic to draw conclusions on causal effects from non-randomized or even uncontrolled studies. However, in other aspects such studies can—if planned and performed well—provide valid data. * Corresponding author. Tel.: 089-33041048; fax: 089-393484. E-mail address: [email protected] (K. Linde).

For example, one problem of RCTs is that they often recruit only a small and non-representative sample of patients who are then treated and monitored under conditions which do not necessarily reflect everyday practice [1,2]. As a consequence, the degree to which the results are generalizable and externally valid is unclear. Therefore, it would be desirable to compare patient characteristics, treatments and course of disease in RCTs and pragmatic non-randomized studies. Another problem of RCTs (particularly placebocontrolled) in chronic conditions is the often limited observation period due to ethical reasons, feasibility or resources [3,4]. Cohort studies in which patients receive their preferred treatment can follow patients more easily over longer periods. Large-scale non-randomized trials could also be useful to identify potentially relevant prognostic factors (as well as factors determining why certain people choose certain treatments) and to evaluate safety. Furthermore, if the number or quality of available randomized trials is insufficient to draw conclusions, non-ran-

0895-4356/02/$ – see front matter © 2002 Elsevier Science Inc. All rights reserved. PII: S0895-4356(01)00 4 2 2 - X

78

K. Linde et al. / Journal of Clinical Epidemiology 55 (2002) 77–85

domized studies might be useful to get a better overview of what is known so far and to inform future research. This last point might be particularly relevant in systematic reviews of complementary therapies. For example, there are two systematic reviews (one on acupuncture for induction of labor [5] and one for cranberries to treat urinary tract infections [6]) limited to RCTs in the Cochrane Database of Systematic Reviews which did not identify a single trial meeting the inclusion criteria. Many other reviews restricted to RCTs contain very few trials. Would these reviews be more informative if they included non-randomized studies? There is a need for empirical studies investigating under what circumstances the inclusion of nonrandomized trials is adequate and which problems have to be taken into account. Parallel to a “conventional” systematic review of the RCTs of acupuncture for chronic headache [7] we performed a more inclusive analysis to investigate the following questions: (1) Do randomized and non-randomized studies of acupuncture for chronic headache differ in regard to patients, interventions, design-independent quality aspects, and response rates? (2) Do non-randomized studies provide relevant additional information (results on longterm outcomes, prognostic factors, adverse effects or complications, response rates in representative or well-defined groups of patients)? (3) In case response rates in randomized and non-randomized patients differ, what are possible explanations? 2. Methods 2.1. Selection criteria We included reports providing clinical outcome data in at least five patients with recurrent or chronic headaches (migraine, tension headache, cluster headache, not precisely classified chronic or recurrent headaches) who had been treated with acupuncture (needle insertion at acupuncture points, pain points, trigger points or other methods of stimulating acupuncture points such as laser acupuncture or electroacupuncture) for prophylaxis of their headaches. We excluded studies focussing on acute headaches only or solely reporting physiological outcomes. Studies only available in Chinese language were not included due to the lack of resources for translation. 2.2. Search strategy To identify published and unpublished eligible studies we searched: (1) Medline 1966 to December 1998 (search terms: acupuncture/acupressure/electroacupuncture and headache/ migraine); (2) Embase 1989 to December 1998 (acupuncture/acupressure and headache/migraine); (3) the Cochrane Controlled Trials Register (Issue 1, 1999); (4) individual trial collections and private databases; and (5) bibliographies of articles obtained. All references identified by the literature search, and if available their abstracts, were screened

by two reviewers. Full reports of potentially relevant publications were obtained and checked for eligibility. 2.3. Classification Included studies were classified into five categories: (1) Randomized controlled trials; (2) non-randomized controlled cohort studies; (3) uncontrolled prospective studies (studies apparently based on a preplanned study protocol and meeting at least one of the following criteria: explicit in- and exclusion criteria, at least two predefined clinical outcome measures, predefined treatment intervention); (4) case series (uncontrolled studies without evidence of a predefined protocol); (5) cross-sectional studies/surveys. The classification of non-randomized studies was done twice with a time lag of 5 months by the first author. 2.4. Data extraction We extracted source details (bibliographic information, language, country of the first author, whether published in a Medline-listed journal), information on patients (number, diagnosis, in- and exclusion criteria, use of headache classification guidelines, setting, mean age, percentage females, years since onset of disease, resistance to previous treatment), interventions (strategy to choose acupuncture points, number and duration of sessions), length of follow-up, and data on response to treatment (definition of response, number of patients classified as responders after treatment, at early follow-up, and follow-up of at least 6 months after treatment; number of patients treated with acupuncture, time of the earliest available response measurement). Response rates for meta-analysis were calculated as number of responders divided by the number of patients analyzed for the earliest available time point after treatment. Data from groups without acupuncture (control groups) were not analyzed. To assess whether non-randomized studies add relevant information we checked the number of studies: (1) with at least 6 months follow-up after the last acupuncture session; (2) which provided outcome data for at least 80% of treated patients after at least 6 months follow-up; (3) which systematically investigated prognostic factors; (4) with data on side effects or complications; (5) with at least three study centers and a transparent sampling strategy. Selection, extraction and assessments were performed by at least two reviewers for randomized and non-randomized controlled trials, and by one reviewer for the other studies. Disagreements were resolved by discussion. In addition, using a list of 16 items proposed by White and Ernst [8] we investigated how detailed the acupuncture treatment was described (results presented in [9]). 2.5. Assessment of quality aspects relevant for headache The aim of the quality rating was to assess the accuracy of observation and reporting, as far as possible independently from the specific design chosen. Whether randomized or not a study with good quality in this respect should

K. Linde et al. / Journal of Clinical Epidemiology 55 (2002) 77–85

allow a health care professional treating headache patients to have an idea of the patients in the study and the course of symptoms. The criteria were partly derived from the guidelines for clinical trials in migraine and tension-type headache from the International Headache Society [10,11]. The following 10 items could be answered yes, no or unclear: (1) Sampling method clear and clearly described? (2) Clear headache diagnosis? (3) Patients characterized (at least age, sex, duration, severity of symptoms)? (4) At least 4 weeks baseline period? (5) At least two clinical headache outcomes? (6) Use of a headache diary? (7) Cointerventions described? (8) At least 90% of patients included analyzed after treatment? (9) At least 80% of patients treated analyzed at early (6 months) follow-up? (10) At least 80% of patients treated analyzed at late (6 months) follow-up? In case of cross-sectional studies items 9 and 10 were scored when the survey took place at least 6 months after acupuncture treatment and the response rate was greater than 80%. 2.6. Analyses Characteristics and fulfillment of the quality criteria among RCTs and other studies were compared with the chisquare test and the Mann-Whitney test (no adjustment for multiple comparisons). Responder proportions were estimated by a generalized linear mixed model (random effects model) as implemented in the SAS® Macro GLIMMIX [12]. In particular, we used a linear logistic regression model with an additional random intercept term to reflect the heterogeneity between the studies. GLIMMIX uses restricted pseudo-likelihood (REPL) to find the parameter estimates of the linear mixed logistic model we specified. The approach allows for trial-specific and treatment-averaged inference. This estimation procedure was utilized for univariate as well as for multivariate analysis. 3. Results From a total of 97 acupuncture studies on headache identified, 59 met the inclusion criteria. The most frequent reason for exclusion was the mixing of data of a variety of chronic pain patients without separate reporting of results in those with chronic headache. Twenty-four of the 59 included studies were RCTs and 35 studies were non-randomized (5 non-randomized controlled cohort studies, 10 prospective uncontrolled studies, 10 case series, and 10 cross-sectional studies). Table 1 provides an overview of basic features of the included studies which were heterogeneous regarding patients, interventions, type and timing of outcome measurements. The median number of patients treated with acupuncture was significantly smaller in RCTs than in other studies (see Table 2). A total of 535 patients received acupuncture treatment in the 24 RCTs compared to 2695 in the 35 non-randomized studies. RCTs were performed more often in migraine patients and slightly more often in specialized

79

outpatient pain departments. Age, sex distribution and the duration of headache complaints were similar. Only 14 studies (seven RCTs and seven other studies) made statements whether patients had been resistant to previous therapy; in four RCTs and four other studies this had been the case. Acupuncture sessions lasted significantly longer in the non-randomized studies while the number of treatment sessions and the proportion of studies using an individualized treatment strategy was similar. RCTs met criteria for accuracy of observation and reporting much more often than non-randomized studies but in both groups the number of criteria met covered the full range indicating that there were very low and good quality studies in both groups (see Table 3). Ten of the 24 RCTs and 26 of the 35 non-randomized studies met less than five quality criteria. RCTs significantly more often described a clear headache diagnosis, a pre-treatment baseline observation period, at least two clinically relevant outcome measures, and the use of headache diaries. Sampling strategies and cointerventions (such as treatment for headache attacks) were rarely well described regardless of the type of study. Follow-up was either short or incomplete in most studies as well. A total of 50 studies presented some sort of responder data for at least one of the three predefined time-points. RCTs more often used a clearly defined response criterion. The pooled response rate was 59% in RCTs [95% confidence interval (CI) 48–69%) and 78% in non-randomized studies (72–83%; P  0.001). Ten (42%) RCTs and 14 (40%) non-randomized studies measured outcome data at least 6 months after completion of the treatment (see Table 4. However, only two (8%) RCTs and six non-randomized (17%) studies actually provided data for more than 80% of treated patients. In the two RCTs and in two non-randomized studies reliable data on frequency of headache was available from patient diaries. All these studies suggest that improvements achieved after treatment last in the majority of patients. The role of prognostic factors was investigated in two (8%) RCTs and three (9%) non-randomized studies. Due to the small sample sizes and the exploratory character of the analyses the results can at best be considered hypothesis generating. Only nine nonrandomized studies (and eight RCTs) gave information (number of centers, selection process) on how they recruited their patients. In all these studies acupuncture was provided within a single practice or unit. Data on adverse effects (number of patients reporting adverse effects and/or explicit description of the events observed) was reported in four (17%) RCTs and one (3%) non-randomized study. In the univariate analysis diagnosis, sex distribution, mean age, number and duration of treatment sessions, type and timing of response measurement as well as completeness of follow-up did not have a significant impact on the response rates observed (see Table 5). Instead, studies describing their sampling strategy, a clear headache diagnosis, having a baseline period and using a headache diary had significantly lower response rates. Fig. 1 shows that both a

80

K. Linde et al. / Journal of Clinical Epidemiology 55 (2002) 77–85

Table 1 Overview of studies of acupuncture treatment for recurrent headaches Response rates (%) Follow-up First author (ref)

Type of headache

RCTs Ahonen [25] Tension Baust [26] Migraine Carlsson [27] Tension Ceccherelli [28] Migraine Doerr-Proske [29] Migraine Dowson [30] Migraine Hansen [31] Tension Henry [32] Migraine Hesse [33] Migraine Heydenreich [34] Migraine Heydenreich [35] Migraine Johansson [36] Tension Kubiena [37] Migraine Lehmann [38] Migraine Lenhard [39] Migraine Loh [40] Various Okazaki [41] Migraine Pintov [42] Migraine Tavola [43] Tension Vincent [44] Migraine Vincent [45] Tension Weinschütz [46] Migraine Weinschütz [47] Migraine White [48] Tension Non-randomized controlled cohort studies Borglum-J. [49] Unclear Guccione [50] Migraine Sold-Darseff [51] Migraine Turk [52] Migraine Zibin [53] Migraine Uncontrolled prospective studies Baischer [54] Migraine Boivie [55] Migraine Börglum-J. [56] Tension Börglum-J. [57] Tension Ceccherelli [58] Various Ceccherelli [59] Various Kim [60] Migraine Major [61] Migraine Tolksdorf [62] Various Zwölfer [63] Tension Case series Batra [64] Migraine Cheng [65] Various Fischer [66] Unclear Heydenreich [67] Migraine Heydenreich [68] Migraine Laitinen [69] Migraine Lawrence [70] Various Spoerel [71] Various Stux [72] Migraine Yamauchi [73] Various Surveys Carlsson [74] Various Chen [75] Unclear Haug [76] Unclear Junilla [77] Unclear Junilla [78] Various Kubiena [79] Migraine Maxion [80] Migraine McKibbin [81] Various Pöntinen [82] Migraine Shifman [83] Various

Quality criteria met

Response definition

4 4 5 5 7 8 2 5 8 3 3 0 4 3 7 3 1 7 10 9 9 5 5 5

Improved at least 2 mo* 50% index reduction — 50% index reduction — 50% frequency reduct. 33% index reduction Patient assessment — Unclear Unclear — 33% index reduction — 33% frequency reduct. Patient assessment Unclear — 50% index reduction 33% frequency reduct. 50% index reduction Single case statistics Single case statistics —

8 1 1 1 0

Unclear — Unclear Unclear Unclear

After treatment

Early

6 mo.

12 23 30 15 10 25 25 20 42 20 50 17 15 21 16 23 20 11 15 16 14 20 20 5

n.a. 61 n.a. 87 n.a. 32 28 65 n.a. n.a. n.a. n.a. 75 n.a. n.a. 35 75 n.a. 53 47 64 50 45 n.a.

58 n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. 56 n.a. n.a. n.a. n.a. n.a. 43 65 75 n.a.

n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. n.a. 80 80 n.a. n.a. n.a. n.a. n.a. n.a. n.a. 40 n.a. n.a. n.a. n.a. n.a.

32 24 30 100 340

79 n.a. 93 86 99

n.a. n.a. n.a. n.a. n.a.

n.a. n.a. n.a. n.a. n.a.

n treated with acupuncture

10 10 6 5 4 6 3 8 5 3

33% index reduction Unclear Unclear Unclear 50% index reduction 40% index reduction Meeting defined criteria* 50% frequency reduct. 50% index reduction Patient assessment

31 30 21 16 12 92 25 33 70 20

70 n.a. 67 63 100 76 n.a. 57 59 80

n.a. 56 57 n.a. n.a. n.a. 80 n.a. n.a. n.a.

58 56 n.a. n.a. n.a. n.a. n.a. n.a. 38 n.a.

3 2 2 0 3 5 1 2 0 2

At least 50% improvem.* No headache at all 50% symptom reduct.* At least 50% improvem.* At least 50% improvem.* Meeting defined criteria* Alleviation symptoms* Severity & freq. reduct.* Unclear Unclear

20 33 189 320 296 39 96 38 50 7

n.a. 91 75 n.a. n.a. 92 92 n.a. 88 71

65 n.a. n.a. n.a. n.a. n.a. n.a. 84 n.a. n.a.

n.a. n.a. n.a. 88 76 46 n.a. n.a. n.a. n.a.

4 1 1 2 2 1 1 4 2 0

Patient assessment Patient assessment Patient assessment 30% symptom reduct.* Patient assessment Patient assessment Patient assessment — Patient assessment Patient assessment

79 44 49 115 69 118 13 107 107 33

57 n.a. n.a. n.a. 81 n.a. 15 n.a. n.a. 58

n.a. n.a. n.a. n.a. n.a. 62 n.a. n.a. n.a. n.a.

23 68 84 77 n.a. n.a. n.a. n.a. 57 n.a.

n.a.  not available; *operationalization unclear. Patient assessments performed with various categorizations (without use of headache diaries).

K. Linde et al. / Journal of Clinical Epidemiology 55 (2002) 77–85

81

Table 2 Comparison of randomized and non-randomized studies of acupuncture for chronic headache: Patient and intervention characteristics

Patient characteristics N treated with acupuncture (median, range) Diagnosis Migraine Tension-type headache various/unclear Mean age (median, range) % proportion female (median, range) Onset of headache  5 years ago Setting Specialized outpatient dpt. etc. Practice Intervention characteristics Number of acupuncture sessions (median; range) Duration of sessions in minutes (median; range) Intervention individualized

RCTs (n  24)

Other studies (n  35)

P-value

20 (5–100)

39 (7–340)

0.001

16 (67%) 7 (29%) 1 (4%) 40 (32–46; mi: 7) 80 (57–100, mi: 3) 18 (94%, mi: 6) mi: 5 17 (89%) 2 (11%)

16 (46%) 3 (8%) 16 (46%) 40.5 (22–53, mi: 19) 78 (33–100, mi: 10) 18 (100%, mi: 17) mi: 6 20 (69%) 9 (31%)

0.161

8 (4–14; mi: 3) 15 (1–30; mi: 7) 9 (43%, mi: 3)

10 (1–30; mi: 11) 22.5 (10–45; mi: 10) 14 (48%, mi: 6)

0.731 0.001 0.631

0.002 0.515 0.667 0.866

mi  number of studies with missing information.

randomized trial design and a higher number of quality criteria met were associated with lower response rates. We were unable to specify an adequate model for multiple regression analysis. Apart from design (randomized or not) other factors had little consistent effects. 4. Discussion Our results show that in the case of acupuncture for chronic headache: (1) RCTs and non-randomized studies differ in regard to patients included, interventions provided, and response rates reported; (2) RCTs have in average better accuracy of observation and reporting, but a relevant proportion of RCTs has very low and some prospective un-

controlled trials have acceptable accuracy of observation and reporting; (3) the available non-randomized studies provide only limited useful additional information in regard to long-term response and prognostic factors, and no additional information regarding adverse effects and generalizability; (4) better study design (randomized vs. not randomized) and better accuracy of observation and reporting (regardless whether a study is randomized or not) are associated with lower response rates. When interpreting our results, a number of limitations have to be kept in mind. While extractions and assessments for the RCTs were performed by at least two reviewers the non-randomized studies were analyzed by a single reviewer. Although each study was read at least two times it cannot be

Table 3 Comparison of randomized and non-randomized studies of acupuncture for chronic headache: accuracy of observation and reporting, outcome characteristics and response rates

Accuracy of observation and reporting Sampling described Clear headache diagnosis Patients characterized At least 4 wks baseline Cointerventions described At least two outcomes Headache diary used Complete after treatment Complete at early follow-up Complete at late follow-up Median number (range) of items met At least 5 quality crit. met Subjective quality weak Outcome characteristics Response data available Measurement within 3 months after treatment Complete at response measurement Response rates (95% confidence intervals) mi  number of trials with missing information.

RCTs (n  24)

Other studies (n  35)

P-value

8 (33%) 16 (67%) 17 (71%) 15 (63%) 7 (29%) 20 (83%) 18 (75%) 11 (46%) 6 (25%) 2 ( 8%) 5 (0–10) 14 (58%) 6 (25%)

9 (26%) 8 (23%) 17 (49%) 5 (14%) 10 (29%) 18 (51%) 8 (23%) 19 (54%) 10 (29%) 6 (17%) 2 (0–10) 9 (26%) 29 (83%)

0.559 0.001 0.112 0.001 1.000 0.014 0.001 0.601 1.000 0.453 0.008 0.016 0.001

16 (67%) 16 (89%) 10 (56%) 0.59 (0.48–0.69)

33 (94%) 21 (64%) 21 (64%) 0.78 (0.72–0.83)

0.024 0.634 0.765 0.001

82

K. Linde et al. / Journal of Clinical Epidemiology 55 (2002) 77–85

Table 4 Comparison of randomized and non-randomized studies of acupuncture for chronic headache: information on long-term results, prognostic factors, generalizability, and safety RCTs (n  24) At least 6 months follow-up after treatment Outcome data for at least 80% of included patients at follow-up (at least 6 months) Analysis of prognostic factors At least three study centers and transparent sampling strategy Data on adverse effects

Other studies (n  35)

Table 5 Pooled response rates (random effects model) in all studies (both RCTs and non-randomized) meeting or not meeting defined criteria (univariate mixed logistic regression model)

P-value Variable

10 (42%)

14 (40%)

2 (8%) 2 (8%) –

1.000

6 (17%) 3 (9%)

0.453 1.000

1 (3%)

0.148

– 4 (17%)

ruled out that coding errors occurred or that some details would have been interpreted differently by a second reviewer. The list for assessing the accuracy of observation and reporting was not formally validated. It can only be considered as a trial to operationalize quality aspects which go beyond methodological criteria covered in commonly used scales [13–15]. We did multiple statistical testing; therefore, the significant P-values reported have to be interpreted with caution. A number of empirical analyses have addressed the question whether effect size estimates from RCTs and non-randomized controlled studies differ (e.g., [16–18] for newer original work and [19,20] for reviews of older studies). The findings of these studies are variable. Among randomized trials there is evidence that trials with better quality yield less optimistic results. Particularly, adequate concealment of randomization and double-blinding seem to be associated with less optimistic outcomes [21–23]. We are not aware of studies comparing characteristics, design-independent quality aspects and outcomes of RCTs and non-randomized (including uncontrolled) studies. We believe that the current discussion of quality of clinical trials over-emphasizes methodological aspects and disregards clinical aspects. This might have pragmatic reasons as methodological criteria of good quality are valid in most instances while clinical quality criteria strongly differ among conditions and interventions. Nevertheless, in our opinion the quality of a trial on a complex treatment for chronic headache cannot be assessed in a valid manner by checking only whether there are the words randomized, double-blind and a description of drop-outs and withdrawals in the text. Ten of the 24 RCTs met less than half of our criteria for accuracy of observation and reporting which seem to us quite basic. This indicates that a relevant proportion of the RCTs reviewed are unlikely to provide clinically useful results. The conclusion of our systematic review of RCTs [7] was that the available evidence suggests that acupuncture is effective in migraine and tension-type headache but that further trials with better methodology and larger sample sizes are urgently needed. The non-randomized studies included in the analysis presented here reinforce the impres-

Clinical feature Only migraine patients Intervention feature 10 sessions of 15 minutes Accuracy of observation and reporting Sampling described Clear headache diagnosis Patients characterized At least 4 wks baseline Cointerventions described At least two outcomes Headache diary used Complete after treatment Complete at early follow-up Complete at late follow-up At least 5 quality crit. met Outcome features Response criterion  50% reduction Early response measurement Complete at response measurement

Response rates (95% confidence intervals) Criterion not met vs. criterion met

P-value

0.72 (0.63–0.79) vs. 0.73 (0.65–0.81) 0.7543

0.76 (0.63–0.86) vs. 0.69 (0.60–0.78) 0.3414

0.76 (0.71–0.82) vs. 0.62 (0.50–0.72) 0.0138 0.78 (0.72–0.83) vs. 0.62 (0.52–0.72) 0.0037 0.78 (0.70–0.84) vs. 0.68 (0.60–0.75) 0.0581 0.77 (0.72–0.82) vs. 0.61 (0.49–0.72) 0.0099 0.74 (0.67–0.80) vs. 0.69 (0.57–0.79) 0.4695 0.77 (0.68–0.83) vs. 0.70 (0.62–0.77) 0.2256 0.78 (0.73–0.83) vs. 0.61 (0.51–0.70) 0.0015 0.75 (0.68–0.82) vs. 0.69 (0.60–0.77) 0.2208 0.75 (0.68–0.80) vs. 0.66 (0.54–0.77) 0.1735 0.73 (0.66–0.79) vs. 0.72 (0.58–0.82) 0.8816 0.77 (0.71–0.82) vs. 0.63 (0.53–0.73) 0.0186

0.74 (0.68–0.80) vs. 0.70 (0.57–0.80) 0.4966 0.71 (0.64–0.77) vs. 0.79 (0.66–0.88) 0.2383 0.75 (0.66–0.82) vs. 0.71 (0.63–0.78) 0.4425

sion of effectiveness. However, the amount of relevant new information they add is disappointing. The differences between RCTs and non-randomized studies suggest that patients and interventions in RCTs are not representative for routine acupuncture practice. But the sampling strategies in non-randomized studies—when described at all—did not allow assessment of generalizability either. Both RCTs and non-randomized studies almost completely neglected the question of safety. The majority of non-randomized studies had very low quality and the potential advantages of using a non-randomized design were rarely ever used. If we would have included the non-randomized studies in our original systematic review [7] the conclusions regarding implications for practice would not have been different. The conclusions regarding future research are obvious: apart from better RCTs better non-randomized studies with long-term follow-up and data about safety are needed. Including non-randomized studies increases the workload of a systematic review considerably. As efficient search strategies to identify non-randomized studies have not been developed yet the search has to be very broad. Both during screening for eligibility and when coding included studies

K. Linde et al. / Journal of Clinical Epidemiology 55 (2002) 77–85

Fig. 1. Proportions (in percent with 95% confidence intervals) of patients classified as improved after acupuncture treatment in randomized trials and non-randomized studies stratified by the number of criteria for accuracy of observations and reporting met.

much more material has to be obtained and read than in a systematic review limited to RCTs. Usual methods for quality assessment typically focus on issues related to bias affecting the difference between compared groups such as the allocation to groups, allocation concealment, and blinding. Therefore, other aspects of quality have to be addressed in addition. Extraction, analysis and—last but not least—the report of results have to take into account the additional questions addressed and problems encountered. Consequently, a report of an “all-inclusive” review is likely to be longer and more complex than that of a review restricted to RCTs. Considering all these issues, the increase in knowledge by including non-randomized studies does not seem to outweigh the additional workload, at least in the specific case of acupuncture for headache. However, this conclusion would be too negative. Confirmation of data from randomized trials by clinical observations in actual care has not invariably been seen [24]. The finding that non-randomized studies of good quality yielded similar response rates as randomized trials suggests that their inclusion may present a useful extension to systematic reviews to investigate generalizability. Further empirical research investigating whether and how non-randomized studies should be included into systematic reviews on prophylactic and treatment interventions is warranted. Acknowledgments This study was in part supported by the Karl und Veronica Carstens Foundation and the NIH Grant No 5 U24AR-43346-02. We thank Wolfgang Weidenhammer for his help with the analysis of the data, Patricia Fischer for help in identifying and screening the studies, and Regina Kunz for commenting on earlier drafts of the manuscript. We also thank the peer reviewers whose valuable comments greatly contributed in the preparation of the final manuscript. References [1] Charlson ME, Horwitz RI. Applying the results of randomised trials to clinical practice: impact of losses before randomisation. BMJ 1984;289:1281–4.

83

[2] Mant D. Can randomised trials inform clinical decisions about individual patients? Lancet 1999;353:743–6. [3] Feinstein AR. An additional basic science for clinical medicine: II. The limitations of randomized trials. Ann Intern Med 1983;99:544–50. [4] Pincus T. Limitations of randomized clinical trials to recognize possible advantages of combination therapies in rheumatic diseases. Seminar Arthrit Rheumatism 1993;23:2–10. [5] Smith CA, Crowther CA. Acupuncture for the induction of labour (Cochrane Review). In: The Cochrane Library, Issue 1. Oxford: Update Software, 2001. [6] Jepson RG, Mihaljevic L, Craig J. Cranberries for urinary tract infection (Cochrane Review). In: The Cochrane Library, Issue 1. Oxford: Update Software, 2001. [7] Melchart D, Linde K, Fischer P, White A, Allois G, Vickers A, Berman B. Acupuncture for recurrent headaches: a systematic review of randomized controlled trials. Cephalalgia 1999;19:779–86. [8] White AR, Ernst E. A trial method for assessing the adequacy of acupuncture treatments. Altern Ther Health Med 1998;4:66–71. [9] Linde K, Melchart D, Willich SN. Die Beschreibung der Therapie in klinischen Studien zur Akupunktur bei chronischen Kopfschmerzen. Deutsche Zeitschrift für Akupunktur 2001;44:8–14. [10] International Headache Society Committee on Clinical Trials. Guidelines for controlled trials of drugs in migraine. Cephalalgia 1991;11: 1–12. [11] International Headache Society Committee on Clinical Trials. Guidelines for trials of drug treatments in tension-type headache. Cephalalgia 1995;15:165–79. [12] Wolfinger R, O’Connell M. Generalized linear mixed models: a pseudo-likelihood approach. J Stat Comput Simul 1992;48:233–44. [13] Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary. Control Clin Trials 1996;17:1–12. [14] Verhagen AP, de Vet HC, de Bie RA, et al. The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting. J Clin Epidemiol 1998;51:1235–41. [15] Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Int J Technol Assessm Health Care 1996;12:195–208. [16] Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. New Engl J Med 2000;342:1878–86. [17] Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research design. N Engl J Med 2000;342:1887–92. [18] Guyatt GH, DiCenso A, Farewell V, Willan A, Griffith L. Randomized trials versus observational studies in adolescent pregnancy prevention. J Clin Epidemiol 2000;53:167–74. [19] Britton A, McKee M, Black N, McPherson K, Sanderson C, Bain C. Choosing between randomised and non-randomised studies: a systematic review. Health Technol Assessment 1998;2(13). [20] Kunz R, Oxman A. The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998;317:1185–90. [21] Schulz KF, Chalmers I, Hayes RJ, et al. Empirical evidence of bias. JAMA 1995;273:408–12. [22] Moher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609–13. [23] Linde K, Scholz M, Ramirez G, et al. Impact of study quality on outcome in placebo-controlled trials of homeopathy. J Clin Epidemiol 1999;52:631–6. [24] Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996;312:1215–8. [25] Ahonen E, Hakumäki M, Mahlamäki S, Partanen J, Riekkinen P, Sivenius J. Effectiveness of acupuncture and physiotherapy on myogenic headache: a comparative study. Acupuncture & Electro-Therapeutics Res Int J 1984;9:141–50. [26] Baust W, Stürtzbecher H. Akupunkturbehandlung der Migräne im Doppelblindversuch. Med Welt 1978;29:669–73.

84

K. Linde et al. / Journal of Clinical Epidemiology 55 (2002) 77–85

[27] Carlsson J, Fahlcrantz A, Augustinsson LE. Muscle tenderness in tension headache treated with acupuncture or physiotherapy. Cephalalgia 1990;10:131–41. [28] Ceccherelli F, Altafini L, Rossato M, et al. Trattamento agopunturale dell’emicrania senz’aura. Studio in doppio cieco vs placebo. Proceedings XV Congresso Nazionale, AISD Associazione Italiana per lo Studio del Dolore, S. Margherita Ligure, 1992. pp. 310–8. [29] Doerr-Proske H, Wittchen HU. Ein muskel- und gefä·orientiertes Entspannungsprogramm (SEP) zur Behandlung chronischer Migränepatienten: Eine randomisierte klinische Vergleichsstudie. Ztschr Psychosom Med 1985;31:2472–66. [30] Dowson DI, Lewith GT, Machin D. The effects of acupuncture versus placebo in the treatment of headache. Pain 1985;21:35–42. [31] Hansen E, Hansen JH. Acupuncture treatment of chronic tension headache: a controlled cross-over trial. Cephalalgia 1985;5:137–42. [32] Henry P, Baille H, Dartigues F, Jogeix M. Traitement de la maladie par acupuncture: étude controlée. Premières Rencontres Médecines Alternatives 1986;12/13:209–16. [33] Hesse J, Mögelvang B, Simonsen H. Acupuncture versus metoproplol in migraine prophylaxis: a randomized trial of trigger point inactivation. J Intern Med 1994;235:451–6. [34] Heydenreich A, Thiessen M. Effektivitätsvergleich zwischen medikamentöser Therapie, invasiver und nichtinvasiver Akupunktur bei der Migräne (Eine prospektive randomisierte Studie mit Fremdauswertung). Ztschr ärztl Fortbild 1989;83:877–9. [35] Heydenreich A. Punktförmige transkutane elektrische Nervenstimulation (PuTENS) im einfachen Placebovergleich bei der Migräne (Eine prospektive randomisierte Studie). Ztschr ärztl Fortbild 1989;83:881–3. [36] Johansson V, Kosic S, Lindahl O, Lindwall L, Tibbling L. Effect of acupuncture in tension headache and brainstem reflexes. In: Bonica JJ, Albe-Fessard D, editors. Advances in pain research and therapy, vol. 1. New York: Raven Press, 1976. pp. 839–41. [37] Kubiena G, Nissel H, Porenta G, Veitl M, Wessely P. Akupunktur bei Migräne. Follow-up Studie. Dtsche Ztschr Akup 1992;35:140–8. [38] Lehmann V, Banzhaf E, Kunze E, et al. Randomisierte klinisch kontrollierte Studie der Effizienz der Akupunktur im Vergleich zur Elektroakupunktur sowie zur Pharmakotherapie mit Propranolol bei Patienten mit häufiger Migräne. Dtsche Ztschr Akup 1991;34:27–30. [39] Lenhard L, Walte PME. Acupuncure in the prophylactic treatment of migraine headaches: pilot study. N Z Med J 1983;96:663–6. [40] Loh L, Nathan PW, Schott GD, Zilkha KJ. Acupuncture versus medical treatment for migraine and muscle tension headaches. J Neurol Neurosurg Psychiat 1984;47:333–7. [41] Okazaki K, Sadove MS, Kim SI, Lee MH, Cheng D. Ryodoraku therapy for migraine headache. Am J Chin Med 1975;3;61–70. [42] Pintov S, Lahat E, Alstein M, Vogel Z, Barg J. Acupuncture and the opioid system: implications in management of migraine. Pediatr Neurol 1997;17:129–33. [43] Tavola T, Gala C, Conte G, Invernizzi G. Traditional Chinese acupuncture in tension-type headache: a controlled study. Pain 1992;48:325–9. [44] Vincent CA. A controlled trial of the treatment of migraine by acupuncture. Clin J Pain 1989;5:305–12. [45] Vincent CA. The treatment of tension headache by acupuncture: a controlled single case design with time series analysis. J Psychosom Res 1990;34:553–61. [46] Weinschütz T, Lindner V, Niederberger U, Schreiber J, Soyka D. Möglichkeiten und Grenzen der Akupunktur in einer neurologischen Schmerzambulanz: Reevaluierung der klinischen Wirkung am Beispiel der Migränebehandlung. Schimrigk K: Verhandlungen der Deutschen Gesellschaft für Neurologie, vol. 7. Heidelberg: Springer Verlag, 1993. pp. 533–4. [47] Weinschütz T, Niederberger U, Johnsen S, Schreiber J, Kropp P. Zur neuroregulativen Wirkung der Akupunktur bei Kopfschmerzpatienten. Dtsche Ztschr Akup 1994;37:106–17. [48] White AR, Eddleston C, Hardie R, Resch KL, Ernst E. A pilot study of acupuncture for tension headache, using a novel placebo. Acup Med 1996;14:11–5.

[49] Börglum-Jensen L, Melsen B, Jensen SB. Effect of acupuncture on headache measured by reduction in number of attacks and use of drugs. Scand J Dent Res 1979;87:373–80. [50] Guccione G, Gentile F, Pullara C, Sparacia B, Liguori A, Sparacia A. Trattamento dell’emicrania senza aura con agopuntura basata sulla differenziazione delle sindromi vs terapia farmacologica e agopunutra standardizzata. In: Gruppo italiano di ricerca in agopuntura. La ricerca in agopuntura in Italia. Roma: Edizioni Paracelso, 1996. pp. 175–7. [51] Sold-Darseff J, Leydhecker W. Akupunktur bei Schmerzen im Kopfbereich sowie bei Blepharospasmus ohne organische Ursache. Klin Monatsbl Augenheilkd 1986;189:167–9. [52] Turk Z, Moser I. Heilung des chronischen Kopfschmerzes durch Akupunktur. Erfahrungsheilkunde 1990;39:724–6. [53] Zibin D. Treating vascular migraine with deep puncture at fengchi: a report of 240 cases. Int J Clin Acup 1994;5:455–8. [54] Baischer W. Acupuncture in migraine: long-term outcome and predicting factors. Headache 1995;35:472–4. [55] Boivie J, Brattberg G. Are there long lasting effects on migraine headache after one series of acupuncture treatments? Am J Chin Med 1987;15:69–75. [56] Börglum-Jensen L, Jensen SB. Effect of acupuncture on tension headache and urinary catecholamine excretion. Scand J Dent Res 1982;90:397–403. [57] Börglum-Jensen L, Tallgren A, Troest T, Jensen SB. Effect of acupuncture on myogenic headache. Scand J Dent Res 1977;85: 456–70. [58] Ceccherelli F, Cavallo C, Ambrosio F, Altopiedi A, Moro-Boscolo E, Giron GP. Psychologische Untersuchung bei Patienten mit chronischem Schmerz, die mit Akupunkturreflextherapie behandelt wurden. Dtsch Ztschr Akup 1983;5:105–12. [59] Ceccherelli F, Ambrosio F, Altopiedi A, et al. Trattamento agopunturale delle cefalee. Min Riflessoter Laserter 1985;2:45–59. [60] Kim KC, Yount RA. The effect of acupuncture on migraine headache. Am J Chin Med 1974;2:407–11. [61] Major P, Standnes B. Akupunkturbehandling ved migrene. Tidskr Nor Laegeforen 1983;103:686–9. [62] Tolksdorf W, Klimm J, Klimczyk K, Penninger M. Die Akupunktur zur Behandlung chronischer Kopfschmerzen. Eine prospektive Studie zum Einfluss der Depression auf den Behandlungserfolg. Dtsche Ztschr Akup 1988;31:104–9. [63] Zwölfer W, Kaiser B, Cartellieri M, Spacek A. Behandlung des Spannungskopfschmerzes mit Akupunktur. Dtsche Ztschr Akup 1994;37:129–32. [64] Batra YK. Acupuncture in the treatment of migraine. Am J Acup 1986;14:135–7. [65] Cheng ACK. The treatment of headaches employing acupuncture. Am J Chin Med 1975;3:181–5. [66] Fischer MV, Behr A, von Reumont J. Acupuncture: a therapeutic concept in the treatment of painful conditions and functional disorders. Report on 971 cases. Acupuncture & Electro-Therapeutics Res Int J 1984;9:11–29. [67] Heydenreich A. Die punktförmige transkutane elektrische Nervenstimulation in der Migränetherapie. Psychiat Neurol med Psychol 1988;40:717–23. [68] Heydenreich A. Die punktförmige transkutane elektrische Nervenstimulation bei funktionellen vertebrogenen Störungen und bei der Migräne. Ztschr Ges Inn Med 1988;43:651–3. [69] Laitinen J. Acupuncture for migraine prophylaxis: a prospective clinical study with six months follow-up. Am J Chin Med 1975;3:271–4. [70] Lawrence RM. Acupuncture for the treatment of headache. A preliminary report. Am J Acup 1974;2:105–8. [71] Spoerel WE, Verkey M, Leung CY. Acupuncture in chronic pain. Am J Chin Med 1976;4:267–79. [72] Stux G. Treatment of migraine with acupuncture and moxibustion. Acup Med 1987;4(2):6. [73] Yamauchi N. The results of therapeutic acupuncture in a pain clinic. Can Anaesth Soc J 1976;23:196–206.

K. Linde et al. / Journal of Clinical Epidemiology 55 (2002) 77–85 [74] Carlsson CPO, Sjölund BH. Acupuncture and subtypes of chronic pain: assessment of long-term results. Clin J Pain 1994;10:290–5. [75] Chen GS, Hwang YC. Therapeutic effect of acupuncture for chronic pain. Am J Chin Med 1977;5:45–61. [76] Haug HU, Robben H. Die Akupunktur als Objekt allgemeinmedizinischer Forschung. Ztschr Allg Med 1986;62:607–12. [77] Junnila SYT. Acupuncture for long-term treatment of headache in a national health center. Am J Acup 1986;14:351–3. [78] Junnila SYT. Long-term treatment of chronic pain with acupuncture. Part I. Acupuncture & Electro-Therapeutics Res Int J 1987;12:23–36. [79] Kubiena G. Akupunktur bei Migräne – eine Dreijahresstudie mit den

[80] [81] [82]

[83]

85

Migränepatient(inn)en der Ambulanz des Ludwig-Boltzmann-Institutes für Akupunktur in Wien 1976-1978. Dtsch Ztschr Akup 1985;1:4–14. Maxion H, Klein HJ. Zur Pathogenese und Behandlung der Migräne. Münch med Wschr 1978;120:315–8. McKibbin LS, Downie R. Treatment of migraine headaches using auricular acupuncture techniques. Laser Ther 1993;5:23–8. Pöntinen PJ, Salmela TM. Acupuncture treatment of migraine. In: Sicuteri F, editor. Headache new vistas. Florence: Biomedical Press, 1977. pp. 251–6. Shifman AC. The clinical response of 328 private patients to acupuncture therapy. Am J Chin Med 1975;3:165–79.