- Email: [email protected]
Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting
Correspondence
See Comment page 10
Submissions should be made via our electronic submission system at http://ees.elsevier.com/ thelancet/
Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting If a new clinical trial is to be justifiable both scientifically and ethically it should be designed in the light of an assessment of relevant previous research, ideally a systematic review.1 When its findings are reported, these should be set in the context of updated reviews of other, similar research.2 In 1997,3 2001,4 and 2005,5 we assessed reports of randomised trials published in the month of May in five medical journals: Annals of Internal Medicine, BMJ, JAMA, The Lancet, and the New England Journal of Medicine. Only a small proportion of trial reports provided sufficient information to assess the contribution of the new results to the totality of the available evidence. We repeated our study in May, 2009. As previously, we assessed the Discussion sections of the trial reports and, as in 2005, we also investigated the extent to which reports referred to systematic reviews used in the design of the new research in their Introduction sections. Our criteria for including a report as a “trial” were as before: it was a randomised or quasi-randomised trial; it was mainly concerned with the outcomes studied in the trial; and it was published as a full report or paper in one of the five journals in 1997 (n=26)
2001 2005 2009 (n=33) (n=18) (n=29)
First trial addressing the question
1
3
3
5
Contained an updated systematic review integrating the new results
2
0
0
1
Discussed a previous review but did not attempt to integrate their results
4
3
5
10
No apparent systematic attempt to set the results in the context of other trials
19
27
10
13
Table: Classification of Discussion sections in reports of randomised controlled trials published in the month of May in 1997, 2001, 2005, and 2009 in five general medical journals
20
May, 2009. One of us (SH) searched the relevant issues of the journals to identify eligible reports. The Introduction and Discussion sections of each eligible report were then assessed independently by at least two of us to decide whether they referred to a systematic review or included an updated systematic review. If a trial claimed to be the only trial of a topic, we searched for trials in the Cochrane Central Register of Controlled Trials (CENTRAL), which might be considered for inclusion in a systematic review of the topic. We resolved disagreements by discussion between all three of us. 28 reports of randomised trials were identified. Systematic reviews were referred to in the Introduction section of 11 reports. In five of the 28 reports, the authors claimed that their study was the first to have addressed the question concerned. Some of these five reports cited other systematic reviews as proof of this claim,6,7 or gave details of the search they had done.8 Our searches of CENTRAL did not identify apparently similar trials for any of the five claiming to be the “first trial”. One of the 24 reports of trials that did not claim to be the first trial placed the results of the new trial in the context of an updated systematic review of other research in the Discussion section.9 Reference was made to relevant systematic reviews in ten other reports, without any integration of the results of the new trials into an update of these reviews. In the remaining 13 reports, there was no evidence that any systematic attempt had been made in the Discussion section to set the new results in the context of previous trials (table). There is no evidence of progress between 1997 and 2009 in the use of updated systematic reviews in discussing the findings of trials published in these five general medical journals. Although the proportion of trials referring to systematic reviews has increased, most reports still fail to do this. Similarly, most researchers do not seem to have considered systematic reviews when designing their trial.
The CONSORT statement,10 first published in 1996, required that data from a new trial should be interpreted “in the light of the totality of the available evidence”. More than 12 years later, most trials still do not do this. The expectation that a new trial be reported in the context of an up-to-date systematic review does not imply that their Discussion section should contain a full account of the materials, methods, and findings of such a review. The technology has existed for some time to enable a brief review of the evidence to be included in the Discussion section, and for links to relevant, up-to-date systematic reviews published elsewhere. With several thousand systematic reviews published each year11 and 4000 full Cochrane reviews now published, the availability and accessibility of systematic reviews has never been greater. As was shown by one research group in our sample, it is possible to integrate the results of a new trial into a systematic review within the report of a trial.9 People who make decisions about health care should be able to be confident in the use of randomised trials to inform their decision. Such confidence requires that these trials be designed and reported in the light of other similar research. In our four consecutive studies, over 12 years, we have shown that this is not the case for trials published in these five journals. We are not aware of other empirical research assessing the extent to which this issue has been addressed and we would welcome such research. In the absence of other evidence, therefore, our findings have shown that editors and authors—in these five high-impact journals at least—continue to fail to serve the needs of those who wish to use the results of randomised trials to make decisions about health care. Work such as that reported in this letter relating to the quality of reporting research is relevant to the employment of all authors.
*Mike Clarke, Sally Hopewell, Iain Chalmers [email protected]
www.thelancet.com Vol 376 July 3, 2010
Correspondence
1
2
3
4
5
6
7
8
9
10
11
Clarke M. Doing new research? Don’t forget the old. Nobody should do a trial without reviewing what is known. PLoS Med 2004; 1: 100–02. Chalmers I, Altman DG. How can medical journals help prevent poor medical research? Some opportunities presented by electronic publishing. Lancet 1999; 353: 490–93. Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA 1998; 280: 280–82. Clarke M, Alderson P, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals. JAMA 2002; 287: 2799–801. Clarke M, Hopewell S, Chalmers I. Reports of clinical trials should begin and end with up-todate systematic reviews of other relevant evidence: a status report. J R Soc Med 2007; 100: 187–90. Ndekha MJ, van Oosterhout JJG, Zijlstra EE, Manary M, Saloojee H, Manary MJ. Supplementary feeding with either ready-to-use fortified spread or corn-soy blend in wasted adults starting antiretroviral therapy in Malawi: randomized, investigator blinded, controlled trial. BMJ 2009; 338: b1867. Mahlungulu S, Grobler LA, Visser ME, Volmink J. Nutritional interventions for reducing morbidity and mortality in people with HIV. Cochrane Database Syst Rev 2007; 3: CD004536. Rannou F, Dimet J, Boutron I, et al. Splint for base-of-thumb osteoarthritis: a randomized trial. Ann Intern Med 2009; 150: 661–69. Ellis P, Barrett-Lee P, Johnson L, et al, for the TACT Trial Management Group and the TACT Trialists. Sequential docetaxel as adjuvant chemotherapy for early breast cancer (TACT): an open-label, phase III, randomised controlled trial. Lancet 2009; 373: 1681–92. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 1996; 276: 637–39. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007; 4: e78.
WHO Intergovernmental Working Group on Public Health, Innovation and Intellectual Property It saddens me when a publication as illustrious as The Lancet slips the bonds of rational, reasonable argument. I refer to your Editorial of Jan 2,1 which started “The pharmaceutical industry’s latest attempt to sabotage the work of the WHO Intergovernmental Working Group on Public Health, www.thelancet.com Vol 376 July 3, 2010
Innovation and Intellectual Property [IGWG]...” I did not respond earlier, since I had hoped that this storm-in-ateacup would have quietly died away. However, the same activists who stirred it up then continue to do so, and I now feel obliged to point out the shortcomings in your contribution. Having served as a member of the WHO’s Commission on Intellectual Property Rights, Innovation and Public Health (CIPIH), the work of which preceded and directly led to the establishment of the WHO IGWG, which in turn led to the Expert Working Group (EWG), I am quite familiar with the modus operandi of ad-hoc WHO bodies looking at issues related to improving global health. WHO takes great pains to ensure that the membership of such bodies is balanced—and therefore encompasses a wide range of conflicting views. Under such circumstances, it is impossible for any individual interest group to impose its views. This was certainly the case in the CIPIH and it also seems to have been the case in the EWG, as the Chair of which, the estimable Sir George Alleyne from Barbados, clearly told the WHO Executive Board in January, 2010, that no external party had influenced the EWG deliberations inappropriately. Your assertion that the work of this WHO group was improperly influenced was perhaps ill judged. It contradicted the Chair of that group, who is surely in a better position than you to know how the group’s decisions were made. Your assertion was also based on spurious logic. The possession and analysis of a working document by an external party only demonstrates an interest in the Group’s work—it does not, by any stretch of the imagination, prove that the Group’s deliberations were interfered with in an inappropriate manner. I should add that all kinds of external bodies follow closely the deliberations of such working groups, and the members of such groups consult frequently with stakeholders representing the whole spectrum of views.
The EWG came to a reasoned and reasonable conclusion, which did not match the more extreme vision of a few extremists. They—with your support—have therefore raised an allegation of improper influence, so as to create an opening for getting the group’s agreed consensus position reviewed, and therefore provide another opportunity to derail a broadly supported pragmatic position in pursuit of their narrow ideological goals. Such tactics are not new. They have been used since the beginning of the CIPIH, and they continue to this day, needlessly complicating and prolonging discussions, and continually pushing back the day when broadly supported, practical measures can be implemented and so contribute to the goal of actually helping to improve the health of people in developing countries. The self-styled activists are actually the opponents of action and their obstructionist tactics do a great disservice to the people they purport to represent. I am employed by several pharmaceutical and biotechnology companies as an executive or non-executive director and am a former Director General of the Association of the British Pharmaceutical Industry (ABPI).
Trevor M Jones [email protected] Woodhyrst House, 18 Friths Drive, Reigate RH2 0DS, UK 1
The Lancet. Drug development for neglected diseases: pharma’s influence. Lancet 2010; 375: 2.
Two-component generalised HIV epidemics James Shelton (March 20, p 964)1 comments that he finds it mystifying how generalised HIV epidemics materialise, given the low infectivity rate of HIV—ie, well below one in 1000 per act of intercourse during the latent phase among heterosexuals. However, this figure ignores one of the main differences between high-prevalence
Getty Images
UK Cochrane Centre, National Institute for Health Research, Oxford OX2 7LG, UK (MC, SH); and James Lind Library, Oxford, UK (IC)
21
See Comment page 10
Submissions should be made via our electronic submission system at http://ees.elsevier.com/ thelancet/
Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting If a new clinical trial is to be justifiable both scientifically and ethically it should be designed in the light of an assessment of relevant previous research, ideally a systematic review.1 When its findings are reported, these should be set in the context of updated reviews of other, similar research.2 In 1997,3 2001,4 and 2005,5 we assessed reports of randomised trials published in the month of May in five medical journals: Annals of Internal Medicine, BMJ, JAMA, The Lancet, and the New England Journal of Medicine. Only a small proportion of trial reports provided sufficient information to assess the contribution of the new results to the totality of the available evidence. We repeated our study in May, 2009. As previously, we assessed the Discussion sections of the trial reports and, as in 2005, we also investigated the extent to which reports referred to systematic reviews used in the design of the new research in their Introduction sections. Our criteria for including a report as a “trial” were as before: it was a randomised or quasi-randomised trial; it was mainly concerned with the outcomes studied in the trial; and it was published as a full report or paper in one of the five journals in 1997 (n=26)
2001 2005 2009 (n=33) (n=18) (n=29)
First trial addressing the question
1
3
3
5
Contained an updated systematic review integrating the new results
2
0
0
1
Discussed a previous review but did not attempt to integrate their results
4
3
5
10
No apparent systematic attempt to set the results in the context of other trials
19
27
10
13
Table: Classification of Discussion sections in reports of randomised controlled trials published in the month of May in 1997, 2001, 2005, and 2009 in five general medical journals
20
May, 2009. One of us (SH) searched the relevant issues of the journals to identify eligible reports. The Introduction and Discussion sections of each eligible report were then assessed independently by at least two of us to decide whether they referred to a systematic review or included an updated systematic review. If a trial claimed to be the only trial of a topic, we searched for trials in the Cochrane Central Register of Controlled Trials (CENTRAL), which might be considered for inclusion in a systematic review of the topic. We resolved disagreements by discussion between all three of us. 28 reports of randomised trials were identified. Systematic reviews were referred to in the Introduction section of 11 reports. In five of the 28 reports, the authors claimed that their study was the first to have addressed the question concerned. Some of these five reports cited other systematic reviews as proof of this claim,6,7 or gave details of the search they had done.8 Our searches of CENTRAL did not identify apparently similar trials for any of the five claiming to be the “first trial”. One of the 24 reports of trials that did not claim to be the first trial placed the results of the new trial in the context of an updated systematic review of other research in the Discussion section.9 Reference was made to relevant systematic reviews in ten other reports, without any integration of the results of the new trials into an update of these reviews. In the remaining 13 reports, there was no evidence that any systematic attempt had been made in the Discussion section to set the new results in the context of previous trials (table). There is no evidence of progress between 1997 and 2009 in the use of updated systematic reviews in discussing the findings of trials published in these five general medical journals. Although the proportion of trials referring to systematic reviews has increased, most reports still fail to do this. Similarly, most researchers do not seem to have considered systematic reviews when designing their trial.
The CONSORT statement,10 first published in 1996, required that data from a new trial should be interpreted “in the light of the totality of the available evidence”. More than 12 years later, most trials still do not do this. The expectation that a new trial be reported in the context of an up-to-date systematic review does not imply that their Discussion section should contain a full account of the materials, methods, and findings of such a review. The technology has existed for some time to enable a brief review of the evidence to be included in the Discussion section, and for links to relevant, up-to-date systematic reviews published elsewhere. With several thousand systematic reviews published each year11 and 4000 full Cochrane reviews now published, the availability and accessibility of systematic reviews has never been greater. As was shown by one research group in our sample, it is possible to integrate the results of a new trial into a systematic review within the report of a trial.9 People who make decisions about health care should be able to be confident in the use of randomised trials to inform their decision. Such confidence requires that these trials be designed and reported in the light of other similar research. In our four consecutive studies, over 12 years, we have shown that this is not the case for trials published in these five journals. We are not aware of other empirical research assessing the extent to which this issue has been addressed and we would welcome such research. In the absence of other evidence, therefore, our findings have shown that editors and authors—in these five high-impact journals at least—continue to fail to serve the needs of those who wish to use the results of randomised trials to make decisions about health care. Work such as that reported in this letter relating to the quality of reporting research is relevant to the employment of all authors.
*Mike Clarke, Sally Hopewell, Iain Chalmers [email protected]
www.thelancet.com Vol 376 July 3, 2010
Correspondence
1
2
3
4
5
6
7
8
9
10
11
Clarke M. Doing new research? Don’t forget the old. Nobody should do a trial without reviewing what is known. PLoS Med 2004; 1: 100–02. Chalmers I, Altman DG. How can medical journals help prevent poor medical research? Some opportunities presented by electronic publishing. Lancet 1999; 353: 490–93. Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals: islands in search of continents? JAMA 1998; 280: 280–82. Clarke M, Alderson P, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals. JAMA 2002; 287: 2799–801. Clarke M, Hopewell S, Chalmers I. Reports of clinical trials should begin and end with up-todate systematic reviews of other relevant evidence: a status report. J R Soc Med 2007; 100: 187–90. Ndekha MJ, van Oosterhout JJG, Zijlstra EE, Manary M, Saloojee H, Manary MJ. Supplementary feeding with either ready-to-use fortified spread or corn-soy blend in wasted adults starting antiretroviral therapy in Malawi: randomized, investigator blinded, controlled trial. BMJ 2009; 338: b1867. Mahlungulu S, Grobler LA, Visser ME, Volmink J. Nutritional interventions for reducing morbidity and mortality in people with HIV. Cochrane Database Syst Rev 2007; 3: CD004536. Rannou F, Dimet J, Boutron I, et al. Splint for base-of-thumb osteoarthritis: a randomized trial. Ann Intern Med 2009; 150: 661–69. Ellis P, Barrett-Lee P, Johnson L, et al, for the TACT Trial Management Group and the TACT Trialists. Sequential docetaxel as adjuvant chemotherapy for early breast cancer (TACT): an open-label, phase III, randomised controlled trial. Lancet 2009; 373: 1681–92. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 1996; 276: 637–39. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007; 4: e78.
WHO Intergovernmental Working Group on Public Health, Innovation and Intellectual Property It saddens me when a publication as illustrious as The Lancet slips the bonds of rational, reasonable argument. I refer to your Editorial of Jan 2,1 which started “The pharmaceutical industry’s latest attempt to sabotage the work of the WHO Intergovernmental Working Group on Public Health, www.thelancet.com Vol 376 July 3, 2010
Innovation and Intellectual Property [IGWG]...” I did not respond earlier, since I had hoped that this storm-in-ateacup would have quietly died away. However, the same activists who stirred it up then continue to do so, and I now feel obliged to point out the shortcomings in your contribution. Having served as a member of the WHO’s Commission on Intellectual Property Rights, Innovation and Public Health (CIPIH), the work of which preceded and directly led to the establishment of the WHO IGWG, which in turn led to the Expert Working Group (EWG), I am quite familiar with the modus operandi of ad-hoc WHO bodies looking at issues related to improving global health. WHO takes great pains to ensure that the membership of such bodies is balanced—and therefore encompasses a wide range of conflicting views. Under such circumstances, it is impossible for any individual interest group to impose its views. This was certainly the case in the CIPIH and it also seems to have been the case in the EWG, as the Chair of which, the estimable Sir George Alleyne from Barbados, clearly told the WHO Executive Board in January, 2010, that no external party had influenced the EWG deliberations inappropriately. Your assertion that the work of this WHO group was improperly influenced was perhaps ill judged. It contradicted the Chair of that group, who is surely in a better position than you to know how the group’s decisions were made. Your assertion was also based on spurious logic. The possession and analysis of a working document by an external party only demonstrates an interest in the Group’s work—it does not, by any stretch of the imagination, prove that the Group’s deliberations were interfered with in an inappropriate manner. I should add that all kinds of external bodies follow closely the deliberations of such working groups, and the members of such groups consult frequently with stakeholders representing the whole spectrum of views.
The EWG came to a reasoned and reasonable conclusion, which did not match the more extreme vision of a few extremists. They—with your support—have therefore raised an allegation of improper influence, so as to create an opening for getting the group’s agreed consensus position reviewed, and therefore provide another opportunity to derail a broadly supported pragmatic position in pursuit of their narrow ideological goals. Such tactics are not new. They have been used since the beginning of the CIPIH, and they continue to this day, needlessly complicating and prolonging discussions, and continually pushing back the day when broadly supported, practical measures can be implemented and so contribute to the goal of actually helping to improve the health of people in developing countries. The self-styled activists are actually the opponents of action and their obstructionist tactics do a great disservice to the people they purport to represent. I am employed by several pharmaceutical and biotechnology companies as an executive or non-executive director and am a former Director General of the Association of the British Pharmaceutical Industry (ABPI).
Trevor M Jones [email protected] Woodhyrst House, 18 Friths Drive, Reigate RH2 0DS, UK 1
The Lancet. Drug development for neglected diseases: pharma’s influence. Lancet 2010; 375: 2.
Two-component generalised HIV epidemics James Shelton (March 20, p 964)1 comments that he finds it mystifying how generalised HIV epidemics materialise, given the low infectivity rate of HIV—ie, well below one in 1000 per act of intercourse during the latent phase among heterosexuals. However, this figure ignores one of the main differences between high-prevalence
Getty Images
UK Cochrane Centre, National Institute for Health Research, Oxford OX2 7LG, UK (MC, SH); and James Lind Library, Oxford, UK (IC)
21