- Email: [email protected]
Meta-analysis of randomised controlled trials
of the consultation, 130 patients were randomly sent an unmodified copy of the clinic letter mailed to their family doctor, with a covering note to explain the letter, and 130 patients followed usual practice (controls). 1 week later, we sent all the patients a questionnaire about their perception of the consultation, their knowledge of their diagnosis, and the perceived veracity of the clinic doctor; they also received a standard hospital anxiety and depression questionnaire.5 Those individuals who had received the clinic letter had an extended questionnaire about their response to the letter. 169 (65%) patients responded to the questionnaire. There were slightly fewer respondents (49%) among patients who had cerebrovascular disease, epilepsy, or demyelination and received the clinic letter. 79% of patients who received clinic letters versus 65% of controls believed that the clinic doctor was telling “the whole truth” or “the truth as far as he went”. When we compared patients’ understanding of their diagnosis with the neurologist’s diagnosis under masked conditions, more patients who received clinic letters knew their diagnosis than controls (56% vs 46%). Overall, 91% of patients who received a letter thought it “a good idea”. Compared with the controls, patients who had letters felt better informed (p<0·05, one-tail) and more thought they knew their diagnosis (p<0·05). However, patients in the letter group felt more “anxious” or “very anxious” compared with controls (p<0·05, two-tail). 48% of those who received a clinic letter versus 35% of controls were moderately or severely anxious according to the hospital anxiety and depression questionnaire. This increased anxiety is a cause of concern. A common assumption is that the more informed a patient the better, and the days of a paternalistic medical profession are over. Better communication can improve patient compliance and satisfaction, and also helps to reduce complaints.1 Our survey shows that patients like the idea of receiving a copy of the clinic letter, but that it may increase their anxiety. Perhaps a copy of the clinic letter written between two medical professionals is not the best way to inform patients, though 76% of patients found the letter “clear and understandable”. Ideally, written information should be tailored to the individual patient's needs and written in plain English. This occurs in the genetics clinic in North Staffordshire, but in busy neurological clinics where there is one neurologist per 200 000
Vol 350 • October 18, 1997
people is logistically impractical. The belief that the more information given to a patient the better may also be naive. A fully informed patient may be a fearful patient, and not without justification. We thank the Neurosciences Trust for financial support, Brendan Davis for reviewing the manuscript, Janet Harrison for her assistance, and the citizens of North Staffordshire for their forthright views in difficult times.
*Simon J Ellis, Caroline Matthews Department of Neurology, Keele University, Staffordshire ST4 7LN, UK 1
2
3
4
5
Audit Commission. What seems to be the matter: communication between hospitals and patients. London: HM Stationery Office, 1993. Ley P. Towards better doctor-patient communications. In: Bennett AE, ed. Communications between doctors and patients. Oxford: Oxford University Press, 1976: 77–96. Ellis DA, Hopkin JM, Leitch AG, Crofton J. “Doctors’ orders”: controlled trial of supplementary, written information for patients. BMJ 1979; i: 456. Simpson M, Buckman R, Stewart M, et al. Doctor-patient communication: the Toronto consensus statement. BMJ 1991; 303: 1385–87. Snaith RP, Zigmond AS. The hospital anxiety and depression scale. Windsor: NFER-Nelson, 1994.
Meta-analyses of randomised controlled trials SIR—Clinical recommendations are often based on evidence derived from meta-analysis of randomised controlled trials (RCTs). In your Sept 6 editorial1 you ask “can meta-analyses be trusted?”. There is increasing recognition of the shortcomings of meta-analysis and the potential for bias. In response to clinically important discrepancies between the results of meta-analyses and the results of subsequent megatrials, there have been several attempts to reassess the validity of the technique.2–4 However, these studies merely assessed the agreement, within individual metaanalyses, between the pooled results of small trials and the results of large trials (variously defined), and assumed that good agreement would exclude significant bias. The expectation that any systematic bias would be directly related to trial size is also the basis of the use of funnel plots to screen for bias in meta-analysis.5 These approaches ignore other possible sources of bias. For example, meta-analyses often contain trials that were done many years apart. Diagnostic criteria, concurrent treatments, and patterns of pathology may all change with time, and the direction of publication bias may also vary.
Meta-analysis of early trials (OR, 95% CI)
THE LANCET
2·0 1·8 1·6 1·4 1·2 1·0 0·8 0·6 0·4 0·2 0 0 0·2 0·4 0·6 0·8 1·0 1·2 1·4 1·6 Meta-analysis of subsequent trials (OR)
Comparison of estimate of relative treatment effect derived from metaanalysis of early trials (years 1 and 2) with point estimate of treatment effect derived from meta-analysis of subsequent trials in random selection of 26 published meta-analyses Points lying below diagonal line represent overestimation of relative treatment effect by early trials compared with subsequent trials. OR=odds ratio.
To test the hypothesis that treatment effect in RCTs included in metaanalyses might be related to year of publication, one of us (GR) identified meta-analyses from a search of Cambridge Medline (1985–95) with the search terms meta-analysis, review, and overview. 100 of the papers identified were randomly selected for analysis. Of these, 34 contained formal metaanalyses. Exclusion of meta-analyses of non-trial data, analyses containing trials published in fewer than 3 different years, analyses in which outcome was based on continuous variables, and duplicate analyses, left 21 papers containing 26 meta-analyses of 241 trials* In common with previous studies,2–4 the relative treatment effect in the largest trial in each meta-analysis was, on average, only slightly less than the effect suggested by a meta-analysis of the remaining trials (mean difference in relative odds 6·5%, 95% CI ⫺16 to 19). However, when the trials within each meta-analysis were ordered according to year of publication (the year in which the first trial/s were published was designated year 1, the next year was year 2, and so on), there was significant variation in the proportion of trials in which treatment was better than control (2-test for heterogeneity, p<0·001), with a significant excess of positive trials in years 1 and 2. The variation remained significant after correction for differences in trial size in different years in a multiple regression analysis. Early trials (years 1 and 2) overestimated the treatment effect compared with a meta-analysis of the *Details available from the authors or The Lancet, on request.
1181
*Peter M Rothwell, Gary Robertson Department of Clinical Neurology, Radcliffe Infirmary, Oxford OX2 6HE, UK email: peter.rothwell.clneuro.ox.ac.uk 1 2
3
4
5
Editorial. Meta-analysis under scrutiny. Lancet 1997; 350: 675. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomised controlled trials. N Engl J Med 1997; 337: 536–42. Villar J, Carroli G, Belizan JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet 1995; 345: 772–76. Cappelleri JC, Ionnidis JPA, Schmid CH, de Ferranti SD, Aubert M, Chalmers TC, Lau J. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA 1996; 276: 1332–38. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–34.
SIR—Your editorial,1 inspired by LeLorier and colleagues’2 report of serious discrepancies between metaanalyses of small trials and subsequent large trials, raised the question of whether meta-analyses can be trusted. Meta-analysis cannot be trusted when carried out mechanically and with no broader understanding of the issues under examination. For example, LeLorier and colleagues consider that meta-analyses of the influence of cholesterol lowering drugs on mortality were not supported by the outcome of later definitive studies. The drugs used in the earlier trials produced only small reductions in cholesterol compared with the substantial reductions produced by
1182
32 31 Large trial 30 10 Meta8 analysis
ISIS–4
6 4 Small trials 2 0
Precision
subsequent trials in 20 of the 26 metaanalyses studied (figure), the average difference in relative odds being 35% (95% CI 15–55). There seems to be a systematic bias in reported treatment effect related to the order of publication of trials. This bias is independent of trial size, and is probably due, at least in part, to temporal trends in publication bias. If an initial trial of a treatment is positive, the trial would be more likely to be published than if it is negative. However, once a treatment is established, a negative trial showing that the treatment may, in fact, be ineffective is of more interest. Moreover, if initial trials are especially positive, further trials, and subsequent meta-analyses, are more likely to follow than if the initial trials are negative. Thus meta-analyses done early in the evolution of published trial data could overestimate the efficacy of treatment and should be interpreted with caution. Meta-analysists should consider the possibility that heterogeneity of treatment effect within meta-analyses might be related to the order of publication of trials as well as trial size.
Precision
THE LANCET
21 19 Large trial 17 15 Meta13 analysis 6
ISIS–2 GISS–1
4 Small 2 trials 0 0·1
0·5 1·0 2·0 Odds ratio
10·0
Funnel plot of small trials, and metaanalyses of small trials and individual large trials of magnesium (upper) and streptokinase (lower) in acute myocardial infarction Circles=odds ratios (ORs) from trials included in meta-analysis; diamonds=combined odds ratios with 95% CIs from a meta-analysis; squares=ORs with 95% CIs from large trials appearing after the meta-analysis.
the statins. Meta-analyses had shown that the treatment effect was significantly related to the percentage reduction in cholesterol concentration3 and therefore one would not predict that the same effect would be seen in the statin trials as were seen in earlier trials. Do the earlier trials provide a basis for predicting the outcome of a current treatment regimen? There is an approach to the data included in a metaanalyses which can help here: inspection and statistical analysis of the funnel plot.4 This plot allows examination of the association between the outcomes seen in trials (often odds ratios) and the statistical information (precision) contained within the trial, which is closely related to sample size. If an association is seen, with smaller trials producing larger beneficial effects, then the plot becomes asymmetrical and the meta-analyses may be seriously biased. The funnel plot of intravenous magnesium in the treatment of myocardial infarction is shown in the figure along with the meta-analysis and the latter large ISIS-4 trial, which failed to demonstrate the benefit seen in metaanalyses. The funnel plot is clearly asymmetrical, and a statistical test for asymmetry that we have developed4 demonstrates significant (p=0·005) asymmetry. Conversely, in the case of streptokinase the funnel plot of trials published before the appearance of the large GISSI-1 and ISIS-2 trial was clearly symmetrical, with no statistical evidence of asymmetry in formal
analysis. In this case the outcome of the large trials was almost identical to the result of the meta-analysis. There are several possible causes of asymmetry in funnel plots,4 including publication bias, location bias due to negative findings being preferentially published in non-English language journals or receiving fewer citations than positive trials, and data irregularities. True heterogeneity may also lead to funnel plot asymmetry, if the size of the effect really differs according to sample size because, for example, of a greater intensity of intervention occurring in smaller trials or smaller trials being done in patients at higher initial level of risk, who receive greater benefit. In all these cases of asymmetry the pooled effect from a meta-analysis will be misleading, with the degree of asymmetry indicating the likelihood that bias is substantial. We suggest that funnel plots and formal statistical testing for asymmetry is routinely included in the performance and the reporting of meta-analyses. *George Davey Smith, Matthias Egger Department of Social Medicine, University of Bristol, Bristol BS8 2PR, UK 1 2
3
4
Editorial. Meta-analysis under scrutiny. Lancet 1997; 350: 675. LeLorier J, Grègoire G, Benhaddad A, Kaouerre J, Derderian F. Discrepancies between meta-analysis and subsequent large randomized controlled trials. N Engl J Med 1997; 337: 536–42. Davey Smith G, Song F, Sheldon T. Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ 1993; 306: 1367–73. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–34.
SIR—The following limerick may be of interest and amusement to your readers, especially those who share my skepticism about the arm-chair research done by meta-analysers:1
Meta-analysis An ambitious physician in Boston Wished to publish quickly and often So he re-searched the lit P’ed and R’ed it a bit And first-authored a meta-concoction
Ciarán P Kelly Beth Israel Deaconess Medical Center, Division of Gastroenterology, Boston, MA 02215, USA 1
Editorial. Meta-analysis under scrutiny. Lancet 1997; 350: 675.
DEPARTMENT OF ERROR Poppy Tea and the baker’s first seizure—In this Research Letter by Mark A King and colleagues (Sept 6, p 716) the first sentence should have begun “A 26-year-old baker had a witnessed first tonic-clonic seizure” and the figure caption should have been “Morphine concentrations in blood”.
Vol 350 • October 18, 1997
Vol 350 • October 18, 1997
people is logistically impractical. The belief that the more information given to a patient the better may also be naive. A fully informed patient may be a fearful patient, and not without justification. We thank the Neurosciences Trust for financial support, Brendan Davis for reviewing the manuscript, Janet Harrison for her assistance, and the citizens of North Staffordshire for their forthright views in difficult times.
*Simon J Ellis, Caroline Matthews Department of Neurology, Keele University, Staffordshire ST4 7LN, UK 1
2
3
4
5
Audit Commission. What seems to be the matter: communication between hospitals and patients. London: HM Stationery Office, 1993. Ley P. Towards better doctor-patient communications. In: Bennett AE, ed. Communications between doctors and patients. Oxford: Oxford University Press, 1976: 77–96. Ellis DA, Hopkin JM, Leitch AG, Crofton J. “Doctors’ orders”: controlled trial of supplementary, written information for patients. BMJ 1979; i: 456. Simpson M, Buckman R, Stewart M, et al. Doctor-patient communication: the Toronto consensus statement. BMJ 1991; 303: 1385–87. Snaith RP, Zigmond AS. The hospital anxiety and depression scale. Windsor: NFER-Nelson, 1994.
Meta-analyses of randomised controlled trials SIR—Clinical recommendations are often based on evidence derived from meta-analysis of randomised controlled trials (RCTs). In your Sept 6 editorial1 you ask “can meta-analyses be trusted?”. There is increasing recognition of the shortcomings of meta-analysis and the potential for bias. In response to clinically important discrepancies between the results of meta-analyses and the results of subsequent megatrials, there have been several attempts to reassess the validity of the technique.2–4 However, these studies merely assessed the agreement, within individual metaanalyses, between the pooled results of small trials and the results of large trials (variously defined), and assumed that good agreement would exclude significant bias. The expectation that any systematic bias would be directly related to trial size is also the basis of the use of funnel plots to screen for bias in meta-analysis.5 These approaches ignore other possible sources of bias. For example, meta-analyses often contain trials that were done many years apart. Diagnostic criteria, concurrent treatments, and patterns of pathology may all change with time, and the direction of publication bias may also vary.
Meta-analysis of early trials (OR, 95% CI)
THE LANCET
2·0 1·8 1·6 1·4 1·2 1·0 0·8 0·6 0·4 0·2 0 0 0·2 0·4 0·6 0·8 1·0 1·2 1·4 1·6 Meta-analysis of subsequent trials (OR)
Comparison of estimate of relative treatment effect derived from metaanalysis of early trials (years 1 and 2) with point estimate of treatment effect derived from meta-analysis of subsequent trials in random selection of 26 published meta-analyses Points lying below diagonal line represent overestimation of relative treatment effect by early trials compared with subsequent trials. OR=odds ratio.
To test the hypothesis that treatment effect in RCTs included in metaanalyses might be related to year of publication, one of us (GR) identified meta-analyses from a search of Cambridge Medline (1985–95) with the search terms meta-analysis, review, and overview. 100 of the papers identified were randomly selected for analysis. Of these, 34 contained formal metaanalyses. Exclusion of meta-analyses of non-trial data, analyses containing trials published in fewer than 3 different years, analyses in which outcome was based on continuous variables, and duplicate analyses, left 21 papers containing 26 meta-analyses of 241 trials* In common with previous studies,2–4 the relative treatment effect in the largest trial in each meta-analysis was, on average, only slightly less than the effect suggested by a meta-analysis of the remaining trials (mean difference in relative odds 6·5%, 95% CI ⫺16 to 19). However, when the trials within each meta-analysis were ordered according to year of publication (the year in which the first trial/s were published was designated year 1, the next year was year 2, and so on), there was significant variation in the proportion of trials in which treatment was better than control (2-test for heterogeneity, p<0·001), with a significant excess of positive trials in years 1 and 2. The variation remained significant after correction for differences in trial size in different years in a multiple regression analysis. Early trials (years 1 and 2) overestimated the treatment effect compared with a meta-analysis of the *Details available from the authors or The Lancet, on request.
1181
*Peter M Rothwell, Gary Robertson Department of Clinical Neurology, Radcliffe Infirmary, Oxford OX2 6HE, UK email: peter.rothwell.clneuro.ox.ac.uk 1 2
3
4
5
Editorial. Meta-analysis under scrutiny. Lancet 1997; 350: 675. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomised controlled trials. N Engl J Med 1997; 337: 536–42. Villar J, Carroli G, Belizan JM. Predictive ability of meta-analyses of randomised controlled trials. Lancet 1995; 345: 772–76. Cappelleri JC, Ionnidis JPA, Schmid CH, de Ferranti SD, Aubert M, Chalmers TC, Lau J. Large trials vs meta-analysis of smaller trials: how do their results compare? JAMA 1996; 276: 1332–38. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–34.
SIR—Your editorial,1 inspired by LeLorier and colleagues’2 report of serious discrepancies between metaanalyses of small trials and subsequent large trials, raised the question of whether meta-analyses can be trusted. Meta-analysis cannot be trusted when carried out mechanically and with no broader understanding of the issues under examination. For example, LeLorier and colleagues consider that meta-analyses of the influence of cholesterol lowering drugs on mortality were not supported by the outcome of later definitive studies. The drugs used in the earlier trials produced only small reductions in cholesterol compared with the substantial reductions produced by
1182
32 31 Large trial 30 10 Meta8 analysis
ISIS–4
6 4 Small trials 2 0
Precision
subsequent trials in 20 of the 26 metaanalyses studied (figure), the average difference in relative odds being 35% (95% CI 15–55). There seems to be a systematic bias in reported treatment effect related to the order of publication of trials. This bias is independent of trial size, and is probably due, at least in part, to temporal trends in publication bias. If an initial trial of a treatment is positive, the trial would be more likely to be published than if it is negative. However, once a treatment is established, a negative trial showing that the treatment may, in fact, be ineffective is of more interest. Moreover, if initial trials are especially positive, further trials, and subsequent meta-analyses, are more likely to follow than if the initial trials are negative. Thus meta-analyses done early in the evolution of published trial data could overestimate the efficacy of treatment and should be interpreted with caution. Meta-analysists should consider the possibility that heterogeneity of treatment effect within meta-analyses might be related to the order of publication of trials as well as trial size.
Precision
THE LANCET
21 19 Large trial 17 15 Meta13 analysis 6
ISIS–2 GISS–1
4 Small 2 trials 0 0·1
0·5 1·0 2·0 Odds ratio
10·0
Funnel plot of small trials, and metaanalyses of small trials and individual large trials of magnesium (upper) and streptokinase (lower) in acute myocardial infarction Circles=odds ratios (ORs) from trials included in meta-analysis; diamonds=combined odds ratios with 95% CIs from a meta-analysis; squares=ORs with 95% CIs from large trials appearing after the meta-analysis.
the statins. Meta-analyses had shown that the treatment effect was significantly related to the percentage reduction in cholesterol concentration3 and therefore one would not predict that the same effect would be seen in the statin trials as were seen in earlier trials. Do the earlier trials provide a basis for predicting the outcome of a current treatment regimen? There is an approach to the data included in a metaanalyses which can help here: inspection and statistical analysis of the funnel plot.4 This plot allows examination of the association between the outcomes seen in trials (often odds ratios) and the statistical information (precision) contained within the trial, which is closely related to sample size. If an association is seen, with smaller trials producing larger beneficial effects, then the plot becomes asymmetrical and the meta-analyses may be seriously biased. The funnel plot of intravenous magnesium in the treatment of myocardial infarction is shown in the figure along with the meta-analysis and the latter large ISIS-4 trial, which failed to demonstrate the benefit seen in metaanalyses. The funnel plot is clearly asymmetrical, and a statistical test for asymmetry that we have developed4 demonstrates significant (p=0·005) asymmetry. Conversely, in the case of streptokinase the funnel plot of trials published before the appearance of the large GISSI-1 and ISIS-2 trial was clearly symmetrical, with no statistical evidence of asymmetry in formal
analysis. In this case the outcome of the large trials was almost identical to the result of the meta-analysis. There are several possible causes of asymmetry in funnel plots,4 including publication bias, location bias due to negative findings being preferentially published in non-English language journals or receiving fewer citations than positive trials, and data irregularities. True heterogeneity may also lead to funnel plot asymmetry, if the size of the effect really differs according to sample size because, for example, of a greater intensity of intervention occurring in smaller trials or smaller trials being done in patients at higher initial level of risk, who receive greater benefit. In all these cases of asymmetry the pooled effect from a meta-analysis will be misleading, with the degree of asymmetry indicating the likelihood that bias is substantial. We suggest that funnel plots and formal statistical testing for asymmetry is routinely included in the performance and the reporting of meta-analyses. *George Davey Smith, Matthias Egger Department of Social Medicine, University of Bristol, Bristol BS8 2PR, UK 1 2
3
4
Editorial. Meta-analysis under scrutiny. Lancet 1997; 350: 675. LeLorier J, Grègoire G, Benhaddad A, Kaouerre J, Derderian F. Discrepancies between meta-analysis and subsequent large randomized controlled trials. N Engl J Med 1997; 337: 536–42. Davey Smith G, Song F, Sheldon T. Cholesterol lowering and mortality: the importance of considering initial level of risk. BMJ 1993; 306: 1367–73. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997; 315: 629–34.
SIR—The following limerick may be of interest and amusement to your readers, especially those who share my skepticism about the arm-chair research done by meta-analysers:1
Meta-analysis An ambitious physician in Boston Wished to publish quickly and often So he re-searched the lit P’ed and R’ed it a bit And first-authored a meta-concoction
Ciarán P Kelly Beth Israel Deaconess Medical Center, Division of Gastroenterology, Boston, MA 02215, USA 1
Editorial. Meta-analysis under scrutiny. Lancet 1997; 350: 675.
DEPARTMENT OF ERROR Poppy Tea and the baker’s first seizure—In this Research Letter by Mark A King and colleagues (Sept 6, p 716) the first sentence should have begun “A 26-year-old baker had a witnessed first tonic-clonic seizure” and the figure caption should have been “Morphine concentrations in blood”.
Vol 350 • October 18, 1997