- Email: [email protected]
On meta-analyses of meta-analyses
THE LANCET
A UK Academy of Medicine? Sir—A Working Group, which I chair, has been established by MERCC (Medical Education and Research Coordinating Committee) and asked to report on “all aspects of the desirability and feasibility of establishing an academy of medicine for the UK”. A brief consultation paper was produced by the Working Group and widely distributed earlier in the year. This elicited around 400 responses from a great variety of individuals and these are now being analysed. Inevitably, on such a complex matter, we could hardly expect unanimity. There was substantial support for the idea of an Academy but much less agreement on its nature and scope. There was also significant opposition from those who felt that there were already too many bodies in the medical field. Some supporters of an Academy focused in particular on its broad interdisciplinary role, while others emphasised the interests of academic medicine. The relation of any Academy to existing bodies, particularly to the new Academy of Medical Royal Colleges, was a matter of particular concern to many respondents. The aim of the Working Group is to explore possible models of an Academy, including ones that might grow out of existing bodies, and to gauge the measure of support that might be forthcoming. Our initial consultation exercise was the first step in this process and we are grateful to all those who took the trouble to respond. We shall now digest the responses, undertake further consultations and produce a more focused document that outlines possible alternatives. This can then be used as a basis for further public discussion. In its early stages our Working Group had a secretariat based at the Royal Society of Medicine, but this has now, by mutual agreement, been transferred to the Royal Society.
likely to have less power to detect statistically significant differences compared with one that includes many more patients irrespective of age and neutrophil count. Sensitivity analyses should be encouraged. This applies also to measures of the treatment effect but, in many circumstances, it may be vain to ask whether the risk difference or risk ratio is more appropriate. An 84·9% vs 81·4% absolute difference in efficacy is equivalent to a 19% relative reduction in failures. No statistical rules, numbers, or algorithms can replace clinical judgment in the interpretation of the results of clinical research. Physicians should always try to evaluate whether the protocol and the generalisability of a study are clinically valid and applicable to their patients. A major strength of metaanalysis is that it can promote critical clinical thinking rather than offer a ready-to-use, automated “comfortable reassurance” sanctified by the authority of a p value. What differentiates meta-analysis from dogma, expert opinion, traditional reviews, anecdotal experience, makebelieve, and hear-say is its systematic approach and use of objective, quantitative methods. Readers need to understand what a random or fixed effects model means before they can make an informed judgment on their application. We also need to improve our ability to evaluate the quality of randomised clinical trials and of meta-analyses. We have a long way to go in this area. For example, Prins and Büller list an otherwise excellent traditional review5 as a meta-analysis, although it simply adds up the number of events across studies. *John P A Ioannidis, Joseph Lau New England Medical Center Hospitals, Boston MA 02111, USA
1 2
Michael Atiyah The Royal Society, 6 Carlton House Terrace, London SW1Y 5AG, UK
3
On meta-analyses of meta-analyses SIR—Prins and Büller (July 20, p 199)1 claim that discrepancies between overviews on the same topic may confuse physicians. Since we have long been interested in the strengths and weaknesses of meta-analyses and since we took part in one of the meta-analyses on aminoglycoside dosing that Prins and Büller address we would like to offer a different perspective. Comparing different studies on the basis of p values is methodologically unsound. Medicine has often been misled by p values, but statistical significance is attractive to investigators and editors. For example, among the aminoglycoside trials, the one to reach The Lancet’s pages detected an unrealistically extreme, but statistically significant, difference for nephrotoxicity.2 The magnitude of the treatment effect is more important than its exact p value, and in that respect the results of the meta-analyses cited by Prins and Büller are very consistent and all support the use of single doses. The magnitude of the estimated treatment effect is about a 20% reduction in toxicity and clinical failures. The sensitivity analyses in one meta-analysis3 encompass the others. All the systematic overviews detected heterogeneity in the analysis of efficacy. Beyond that, we should not expect meta-analyses—or trials of 2000 patients—to give us the final answer at the level of the second decimal point. As with any research study, meta-analyses have a protocol, and the results have to be read in the context of that specific protocol and the adherence to it. It is inevitable that the selection criteria or measures of effect in protocols formulated by independent teams will not be the same. For example, a meta-analysis limited to adult non-neutropenic patients4 is
756
4
5
Prins JM, Büller HR. Meta-analysis: the final answer or even more confusion? Lancet 1996; 348: 199. Prins JM, Büller HR, Kuijper EJ, Tange RA, Spleeman P. Once versus thrice daily gentamicin in patients with serious infections. Lancet 1993; 341: 335–39. Barza M, Ioannidis JPA, Cappelleri JC, Lau J. Single or multiple daily doses of aminoglycosides: a meta-analysis. BMJ 1996; 312: 338–45. Hatala R, Dinh T, Cook DJ. Once-daily aminoglycoside dosing in immunocompetent adults: a meta-analysis. Ann Intern Med 1996; 124: 717–25. Blaser J, Konig C. Once-daily dosing of aminoglycosides. Eur J Clin Microbiol Inf Dis 1995; 14: 1029–38.
CONSORT SIR—The CONSORT initiative (Aug 31, p 562)1 aimed at improving the conduct and reporting of randomised trials is, as you said, overdue and most welcome. It is therefore unfortunate that the example you publish (p 564)2 does not meet the stringent requirements that you applaud. According to your editorial, a “published report will contain a trial profile of the number of participants eligible . . .”. In the example cited, 339 patients were recruited from 38 centres (on average, three patients per year at each centre) but there was no mention of how many patients were eligible during the recruitment period. The importance of the external validity (or generalisability) of trial results has been largely ignored in the past on the grounds that it is of little consequence compared with the issue of internal validity (confounding and bias). Although this belief may be warranted, it is an assumption that requires investigation before it is accepted. Systematic reviews of the literature on this topic, as part of the current National Health Service research and development methodology programme in the UK, should help clarify this neglected issue. Nick Black Health Services Research Unit, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London WC1E 7HT, UK
Vol 348 • September 14, 1996
A UK Academy of Medicine? Sir—A Working Group, which I chair, has been established by MERCC (Medical Education and Research Coordinating Committee) and asked to report on “all aspects of the desirability and feasibility of establishing an academy of medicine for the UK”. A brief consultation paper was produced by the Working Group and widely distributed earlier in the year. This elicited around 400 responses from a great variety of individuals and these are now being analysed. Inevitably, on such a complex matter, we could hardly expect unanimity. There was substantial support for the idea of an Academy but much less agreement on its nature and scope. There was also significant opposition from those who felt that there were already too many bodies in the medical field. Some supporters of an Academy focused in particular on its broad interdisciplinary role, while others emphasised the interests of academic medicine. The relation of any Academy to existing bodies, particularly to the new Academy of Medical Royal Colleges, was a matter of particular concern to many respondents. The aim of the Working Group is to explore possible models of an Academy, including ones that might grow out of existing bodies, and to gauge the measure of support that might be forthcoming. Our initial consultation exercise was the first step in this process and we are grateful to all those who took the trouble to respond. We shall now digest the responses, undertake further consultations and produce a more focused document that outlines possible alternatives. This can then be used as a basis for further public discussion. In its early stages our Working Group had a secretariat based at the Royal Society of Medicine, but this has now, by mutual agreement, been transferred to the Royal Society.
likely to have less power to detect statistically significant differences compared with one that includes many more patients irrespective of age and neutrophil count. Sensitivity analyses should be encouraged. This applies also to measures of the treatment effect but, in many circumstances, it may be vain to ask whether the risk difference or risk ratio is more appropriate. An 84·9% vs 81·4% absolute difference in efficacy is equivalent to a 19% relative reduction in failures. No statistical rules, numbers, or algorithms can replace clinical judgment in the interpretation of the results of clinical research. Physicians should always try to evaluate whether the protocol and the generalisability of a study are clinically valid and applicable to their patients. A major strength of metaanalysis is that it can promote critical clinical thinking rather than offer a ready-to-use, automated “comfortable reassurance” sanctified by the authority of a p value. What differentiates meta-analysis from dogma, expert opinion, traditional reviews, anecdotal experience, makebelieve, and hear-say is its systematic approach and use of objective, quantitative methods. Readers need to understand what a random or fixed effects model means before they can make an informed judgment on their application. We also need to improve our ability to evaluate the quality of randomised clinical trials and of meta-analyses. We have a long way to go in this area. For example, Prins and Büller list an otherwise excellent traditional review5 as a meta-analysis, although it simply adds up the number of events across studies. *John P A Ioannidis, Joseph Lau New England Medical Center Hospitals, Boston MA 02111, USA
1 2
Michael Atiyah The Royal Society, 6 Carlton House Terrace, London SW1Y 5AG, UK
3
On meta-analyses of meta-analyses SIR—Prins and Büller (July 20, p 199)1 claim that discrepancies between overviews on the same topic may confuse physicians. Since we have long been interested in the strengths and weaknesses of meta-analyses and since we took part in one of the meta-analyses on aminoglycoside dosing that Prins and Büller address we would like to offer a different perspective. Comparing different studies on the basis of p values is methodologically unsound. Medicine has often been misled by p values, but statistical significance is attractive to investigators and editors. For example, among the aminoglycoside trials, the one to reach The Lancet’s pages detected an unrealistically extreme, but statistically significant, difference for nephrotoxicity.2 The magnitude of the treatment effect is more important than its exact p value, and in that respect the results of the meta-analyses cited by Prins and Büller are very consistent and all support the use of single doses. The magnitude of the estimated treatment effect is about a 20% reduction in toxicity and clinical failures. The sensitivity analyses in one meta-analysis3 encompass the others. All the systematic overviews detected heterogeneity in the analysis of efficacy. Beyond that, we should not expect meta-analyses—or trials of 2000 patients—to give us the final answer at the level of the second decimal point. As with any research study, meta-analyses have a protocol, and the results have to be read in the context of that specific protocol and the adherence to it. It is inevitable that the selection criteria or measures of effect in protocols formulated by independent teams will not be the same. For example, a meta-analysis limited to adult non-neutropenic patients4 is
756
4
5
Prins JM, Büller HR. Meta-analysis: the final answer or even more confusion? Lancet 1996; 348: 199. Prins JM, Büller HR, Kuijper EJ, Tange RA, Spleeman P. Once versus thrice daily gentamicin in patients with serious infections. Lancet 1993; 341: 335–39. Barza M, Ioannidis JPA, Cappelleri JC, Lau J. Single or multiple daily doses of aminoglycosides: a meta-analysis. BMJ 1996; 312: 338–45. Hatala R, Dinh T, Cook DJ. Once-daily aminoglycoside dosing in immunocompetent adults: a meta-analysis. Ann Intern Med 1996; 124: 717–25. Blaser J, Konig C. Once-daily dosing of aminoglycosides. Eur J Clin Microbiol Inf Dis 1995; 14: 1029–38.
CONSORT SIR—The CONSORT initiative (Aug 31, p 562)1 aimed at improving the conduct and reporting of randomised trials is, as you said, overdue and most welcome. It is therefore unfortunate that the example you publish (p 564)2 does not meet the stringent requirements that you applaud. According to your editorial, a “published report will contain a trial profile of the number of participants eligible . . .”. In the example cited, 339 patients were recruited from 38 centres (on average, three patients per year at each centre) but there was no mention of how many patients were eligible during the recruitment period. The importance of the external validity (or generalisability) of trial results has been largely ignored in the past on the grounds that it is of little consequence compared with the issue of internal validity (confounding and bias). Although this belief may be warranted, it is an assumption that requires investigation before it is accepted. Systematic reviews of the literature on this topic, as part of the current National Health Service research and development methodology programme in the UK, should help clarify this neglected issue. Nick Black Health Services Research Unit, Department of Public Health and Policy, London School of Hygiene and Tropical Medicine, London WC1E 7HT, UK
Vol 348 • September 14, 1996