- Email: [email protected]
Validity of randomized clinical trials in gastroenterology from 1964–2000
GASTROENTEROLOGY2002;122:1157-1160
Validity of Randomized Clinical Trials in GASTROENTEROLOGYFrom 1964 - 2 0 0 0 LISE L. KJAERGARD, SARAH L. FREDERIKSEN, and CHRISTIAN GLUUD The Cochrane Hepato-Biliary Group, Copenhagen Trial Unit, Center for Clinical Intervention Research, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark
Background & Aims: The internal validity of clinical trials
depends on the adequacy of the reported methodological quality. We assessed the methodological quality of all 383 randomized clinical trials published in GASTROENTEROLOGY as original articles from 1964 to 2000. Methods: The methodological quality (randomization and blinding), sample size, publication year, and disease area were extracted from each trial. Changes during the study period were analyzed by analysis of variance with adjustments for potential confounders. Results: Fortytwo percent of all trials reported adequate generation of the allocation sequence, 39% reported adequate allocation concealment, and 62% were double blind. The reported methodological quality improved significantly in the mid-1990s. Conclusions: The present study shows a positive development, but the reported methodological quality of trials can still be improved.
he reliability (internal validity) of clinical trials depends on the methodological quality and sample size, First, r a n d o m i z e d clinical trials that do not rep o r t adequate r a n d o m i z a t i o n m e t h o d s exaggerate int e r v e n t i o n benefits by a b o u t 30% c o m p a r e d w i t h trials w i t h adequately p e r f o r m e d and clearly reported rand o m i z a t i o n . 1-4 Trials that are not d o u b l e b l i n d exaggerate i n t e r v e n t i o n benefits by a b o u t 14% c o m p a r e d w i t h d o u b l e - b l i n d trials. 1,3,4 Second, despite adequate
T
m e t h o d o l o g i c a l quality, small trials run a considerable risk of p r o d u c i n g false-positive or false-negative resuits. <5 Few gastroenterological randomized trials were p u b lished before the 1970s6; however, since then, the n u m ber of trials has increased considerably, It is the impression that, with this diffusion, research standards were not maintained.V 9 Several initiatives were launched in the 1990s to improve the quality of trials, but whether these have had an effect on gastroenterological trial research remains to be established. To address these questions, we reviewed the methodological quality and sample size of randomized trials published in GASTROENTEROLOGY from 1964 to 2000.
Materials and Methods The present study includes all randomized clinical trials published as original articles in GASTROENTEROLOGY from 1964 to 2000. We chose to focus on trials published in GASTROENTEROLOGYbecause it is a leading specialist journal and has published randomized trials for several decades. Because few trials were published before 1964, 6 we excluded earlier years. Trials were considered randomized if some form of the word "random" was used to describe the allocation method. However, trials published as abstracts or editorials, trials including healthy volunteers or animals, quasi-randomized trials, and articles referring to subgroups of patients from randomized trials were excluded. Eligible trials were identified through The Cochrane Library m and MEDLINE. The Cochrane Inflammatory Bowel Diseases Group ~° hand-searched all issues of GASTROENTEROLOGYfrom 1964 to 1998, and one of the authors (S.L.F.) hand-searched the 1999 issues. We assessed the methodological quality and extracted the sample size (number of patients/intervention arm), disease area, and year of publication of each trial. Two contributors (S.L.F. and L.L.K.) performed the data abstraction. Consensus was reached before analyses. The methodological quality was assessed by the following components. (1) Generation of the allocation sequence: adequate (e.g., computer-generated random numbers), unclear (not described), or inadequate (quasirandomized trials, which were excluded). (2) Allocation concealment: adequate (e.g., central independent unit, sealed envelopes [irrespective of whether authors reported that the envelopes were opaque and sequentially numbered]), unclear (not described), or inadequate (e.g., open table of random numbers). (3) Double blinding: adequate (e.g., outcome assessors and/or caretakers and patients were kept unaware of treatment allocation by identical placebo), unclear (not described), inadequate (e.g., tablets vs. injections), or not double blind. The quality assessments were unmasked because we previously found no significant difference between masked and unmasked assessments) The intraclass correlation coefficients of the quality assessment (generation of the allocation seAbbreviation used in this paper: CONSORT,ConsolidatedStandards of ReportingTrials. © 2002 by the American GastroenterologicalAssociation 0016-5085/02/$35.00 doi:lO.1053/gast.2002.32390
1158
~AERGARDETAL.
quence, 0.86 [95% confidence interval, 0.70-0.93]; allocation concealment, 0.84 [95% confidence interval, 0.68-0.93]; double blinding, 0.74 [95% confidence interval, 0.50-0.87]) showed high interobserver reliability. Statistical Analyses
The data were analyzed for discrepancies between trials stratified according to publication year. The primary outcome was the methodological quality (generation of the allocation sequence and allocation concealment [adequate vs. unclear/ inadequate] plus double blinding [adequate vs. unclear/inadequate/not performed]). Categorical variables were assessed by Pearson Xe test and continuous data by Kendall "r-b. Discrepancies between groups of trials were also assessed by analysis of variance with adjustments for potential confounders. The sample size was log transformed to obtain homogeneity of variance. The level of significance was set to 5 %. The analyses were performed in SPSS version 10.0 for Windows (SPSS Inc., Chicago, IL). Results
W e identified 1009 references through MEDLINE, 794 references through The Cochrane Library, 1° and 7 trials through the hand-search of 1999 issues. Of these references, 1229 were duplicates, abstracts, editorials, or reviews and 30 were quasi-randomized trials. The remaining 551 references were retrieved. O f these references, we excluded 75 studies on healthy volunteers; 2 quasi-randomized trials; 33 articles referring to subgroups from randomized trials, pathophysiologic studies, or animal studies; and 64 abstracts, editorials, and reviews. Accordingly, the present study includes 377 original articles reporting 383 randomized trials (available on request). Fifty percent of all trials were published after 1988. The generation of the allocation sequence was adequate in 160 trials and unclear in the remaining trials. The allocation concealment was adequate in 150 trials, unclear in 228 trials, and inadequate in 5 trials. Adequate double blinding was reported in 239 trials. O f these trials, 192 used placebo, 40 assessed different drugs, 5 dealt with surgery, and 2 dealt with diagnostic procedures or nursing interventions. The method of blinding was unclear in 8 trials, and 136 trials were not double blind. O f trials without adequate double blinding, 95 dealt with drugs, 34 with surgery, and 15 with diagnostic procedures or nursing interventions. On average, the trials included 43 patients per intervention arm (standard error of the mean, 4). The included trials were grouped into 15 disease areas. A total of 53 trials dealt with duodenal ulcer, 51 with inflammatory bowel diseases, 29 with upper gastrointestinal diseases, 26 with esophageal diseases, 26 with chronic hepatitis, 25 with
GASTROENTEROLOGYVol. 122, No. 4
diarrhea, 25 with cirrhosis, 23 with colorectal diseases, 20 with miscellaneous hepatobiliary diseases, 19 with miscellaneous gastrointestinal diseases, 19 with primary biliary cirrhosis, 19 with portal hypertension, 18 with biliary diseases, 16 with alcoholic liver diseases, and 14 with extra-abdominal diseases. The proportion of trials with adequate generation of the allocation sequence increased significantly during the study period (P = 0.007), as did the proportion of trials with adequate allocation concealment (P = 0.011, Figure 1). The improvements primarily occurred after 1994. After adjusting for confounding (sample size and disease area), the year of publication was a significant predictor of the adequacy of the generation of the allocation sequence (r 2 = 0.12; P = 0.048) and allocation concealment (r 2 = 0.14; P = 0.007). The proportion of doubleblind trials did not increase significantly during the study period before (P = 0.11) or after (r e = 0.14; P = 0.18) adjusting for confounding. The year of publication was significantly associated with the sample size without adjustments (P = 0.001, Figure 2) but not after adjusting for confounding (methodological quality and disease area) (r 2 = 0 . 0 0 5 ; P = 0.17). The adjusted analyses also showed a significant association between the disease area and the methodological quality (P = 0.030) and sample size (P = 0.002).
Discussion
The reported randomization methods of trials in GASTROENTEROLOGY have improved considerably since
1964, especially from 1995 onward. Nevertheless, several trials still lack this information. From 1991 to 2000, 54% of the trials did not report how the allocation sequence was generated and 56% did not report the method of allocation concealment. The present study has limitations. First, it is uncertain whether the results reflect general trends in gastroenterological research. Previous evidence on this issue is equivocal. Trials published in the Journal of Hepatology from 1985 to 1997 showed similar improvements, v but trials published in Hepatology from 1981 to 1998 did not. 9 Second, the design does not allow for an assessment of the association between methodological quality and the estimated intervention effects in the included trials. The implications therefore rest on previous studies on trials from other disease areas) 4 These studies have shown a significant association between unclear randomization methods and exaggerated estimates of intervention effects.
April 2002
RANDOMIZED TRIALS IN GASTROENTEROLOGY 1159
Proportion of trials with adequate allocation concealment
Proportion of lrials with adequate allocation concealment
0.60 0.80 0.50
0.60
I
0.40
0.30
+
0.40
0.20' 0.20 I
I
1964-72
1973-81
I
tttt
I
1982-90
1991-99
0.00'
!
1991
I
I
1993
I
I
1995
I
l
1997
I
I
1999
Figure 1. Mean proportion of trials (standard error of the mean) with adequate allocation concealment from 1964 to 2000 in 9-year intervals and from 1991 to 2000.
We were unable to assess whether the improvements reflect changes at the trial level., the review process, or editorial policies. Inadequate or unclear reporting may conceal methodological flaws. We did not write to the
Number of patients per intervention arm
,f
60
50
40
30
20
10 I
1964-72
|
1973-81
I
1982-90
I
1991-99
Figure 2. Mean number of patients per intervention arm (standard error of the mean) in randomized trials from 1964 to 2000 in 9-year intervals.
authors for additional information, because authors of trials published before the 1990s are difficult to locate. Furthermore, previous evidence indicates that communication with authors provides little additional information about methodological quality. 11 It is likely that the improvements are a combined action by researchers, peer reviewers, and editors. The rate of acceptance of submissions to GASTROENTEROLOGYdeclined considerably during the 1990s, le increasing the demands for originality and quality. During the 1990s, several international initiatives focused on the need for improving the quality of clinical trials. The Cochrane Collaboration l° and International Conference on Harmonisation Expert Working Group, 13 studies showing the importance of methodological quality,U14 and reporting guidelines 15,16 may have had a positive effect. A recent study ~v indicates a positive effect of the Consolidated Standards of Reporting Trials (CONSORT) statement, 18 which provides guidelines for the reporting of the assignment, masking, follow-up, and statistical analysis. However, the improvements in GASTROENTEROLOGYoccurred before the publication of the CONSORT statement, ~8 and the changes must therefore reflect other initiatives. Several methods have been suggested for the assessment of methodological quality, but few are validated. .9 Because the conclusions of most methods are questionable,/9 we focused on components supported by empirical evidence) -4 However, other aspects may be important (e.g., external validity). The quality of a trial
1160
KJAERGARD ET AL.
therefore also depends on clear reporting of the characteristics of the included patients, setting, treatment regimens, and outcomes assessed. 2o Nevertheless, the question of external validity becomes irrelevant if the results are biased. In conclusion, the reporting of randomization methods has improved significantly in GASTROENTEROLOGY. It is likely that the recent decision of GASTROENTEROLOGY21 to adopt the C O N S O R T statement 22.e3 will support this positive development.
References 1. Schulz KF, Chalmers i, Hayes RJ, AItman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273:408-412. 2. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1 9 9 8 ; 3 5 2 : 6 0 9 - 6 1 3 . 3. Kjaergard LL, Villumsen J, Gluud C. Reported methodological quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982-989. 4. J~ni P, Altman D, Egger M. Assessing the quality of controlled clinical trials. BMJ 2 0 0 1 ; 3 2 3 : 4 2 - 4 6 . 5. Peto R, Baigent C. Trials: the next 50 years. Large scale randomised evidence of moderate benefits. BMJ 1998;317:1170-1171. 6. Juhl E, Christensen E, Tygstrup N. The epidemiology of the gastrointestinal randomized clinical trial. N Engl J Med 1977;296: 20-22. 7. Gluud C, Nikolova D. Quality assessment of reports on clinical trials in the Journal of Hepatology. J Hepatol 1 9 9 8 ; 2 9 : 3 2 1 - 3 2 7 . 8. Gluud C. Evidence based medicine in Liver. Liver 1999;19:1-2. 9. Kjaergard LL, Nikolova D, Gluud C. Randomized clinical trials in Hepatology: predictors of quality. Hepatology 1 9 9 9 ; 3 0 : 1 1 3 4 1138. 10. The Cochrane Library, Issue 2, 2000. Oxford: Update Software. 11. Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 1986;4:942-951. 12. LaMont TJ, Link AM, Podolsky DK. We are pleased to inform y o u . . . Gastroenterology 1 9 9 8 ; 1 1 4 : 1 1 2 3 - 1 1 2 4 . 13. International Conference on Harmonisation Expert Working Group. Code of federal regulations & international conference on harmonization guidelines. Philadelphia: Parexel Barnett, 1997.
GASTROENTEROLOGY Vol. 122, No. 4
14. Chalmers TC, Celano P, Sacks HS, Smith H. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983;309: 1358-1361. 15. A proposal for structured reporting of randomized controlled trials. The Standards of Reporting Trials Group. JAMA 1994;272: 1926 -1931. 16. Call for comments on a proposal to improve reporting of clinical trials in the biomedical literature. Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature. Ann Intern Med 1 9 9 4 ; 1 2 1 : 8 9 4 - 8 9 5 . 17. Moher D, Jones A, Lepage L, for the CONSORT Group. Use of the CONSORT statement and quality of reports of randomized trials. JAMA 2001;285:1992-1995. 18. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1 9 9 6 ; 2 7 6 : 6 3 7 - 6 3 9 . 19. JL~ni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:10541060. 20. Altman DG, Bland JM. Generalisation and extrapolation. BMJ 1998;317:409-410. 21. Sandier RS. Reporting randomised controlled trials in Gastroenterology: the Consort Statement. Gastroenterology 2001;121: 755. 22. Moher D, Schulz KF, Altman D, CONSORT GROUP (Consolidated Standards of Reporting Trials). The CONSORT statement: revised recommendations for improving the quality of reports of parallelgroup randomized trials. Ann Intern Med 2 0 0 1 ; 1 3 4 : 6 5 7 - 6 6 2 . 23. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, G#tzsche PC. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134:663-694.
Received May 10, 2001. Accepted November 29, 2001. Address requests for reprints to: Christian Gluud, M.D., The Cochrane Hepato-Biliary Group, Copenhagen Trial Unit, Center for Clinical Intervention Research, Dept. 7102, Blegdamsvej 9, H:S Rigshospitalet, DK-2100 Copenhagen, Denmark. e-maih [email protected]; fax: (45) 3545-7101. Supported by grants from the Danish Medical Research Council and the 1991 Pharmacy Foundation in Denmark. The authors thank Ninna Frydendall and Dimitrinka Nikolova for secretarial assistance, Nader Salasshahri for computer assistance, and The Cochrane Inflammatory Bowel Diseases Group for their handsearches of GASTROENTEROLOGY.
Validity of Randomized Clinical Trials in GASTROENTEROLOGYFrom 1964 - 2 0 0 0 LISE L. KJAERGARD, SARAH L. FREDERIKSEN, and CHRISTIAN GLUUD The Cochrane Hepato-Biliary Group, Copenhagen Trial Unit, Center for Clinical Intervention Research, Copenhagen University Hospital, Rigshospitalet, Copenhagen, Denmark
Background & Aims: The internal validity of clinical trials
depends on the adequacy of the reported methodological quality. We assessed the methodological quality of all 383 randomized clinical trials published in GASTROENTEROLOGY as original articles from 1964 to 2000. Methods: The methodological quality (randomization and blinding), sample size, publication year, and disease area were extracted from each trial. Changes during the study period were analyzed by analysis of variance with adjustments for potential confounders. Results: Fortytwo percent of all trials reported adequate generation of the allocation sequence, 39% reported adequate allocation concealment, and 62% were double blind. The reported methodological quality improved significantly in the mid-1990s. Conclusions: The present study shows a positive development, but the reported methodological quality of trials can still be improved.
he reliability (internal validity) of clinical trials depends on the methodological quality and sample size, First, r a n d o m i z e d clinical trials that do not rep o r t adequate r a n d o m i z a t i o n m e t h o d s exaggerate int e r v e n t i o n benefits by a b o u t 30% c o m p a r e d w i t h trials w i t h adequately p e r f o r m e d and clearly reported rand o m i z a t i o n . 1-4 Trials that are not d o u b l e b l i n d exaggerate i n t e r v e n t i o n benefits by a b o u t 14% c o m p a r e d w i t h d o u b l e - b l i n d trials. 1,3,4 Second, despite adequate
T
m e t h o d o l o g i c a l quality, small trials run a considerable risk of p r o d u c i n g false-positive or false-negative resuits. <5 Few gastroenterological randomized trials were p u b lished before the 1970s6; however, since then, the n u m ber of trials has increased considerably, It is the impression that, with this diffusion, research standards were not maintained.V 9 Several initiatives were launched in the 1990s to improve the quality of trials, but whether these have had an effect on gastroenterological trial research remains to be established. To address these questions, we reviewed the methodological quality and sample size of randomized trials published in GASTROENTEROLOGY from 1964 to 2000.
Materials and Methods The present study includes all randomized clinical trials published as original articles in GASTROENTEROLOGY from 1964 to 2000. We chose to focus on trials published in GASTROENTEROLOGYbecause it is a leading specialist journal and has published randomized trials for several decades. Because few trials were published before 1964, 6 we excluded earlier years. Trials were considered randomized if some form of the word "random" was used to describe the allocation method. However, trials published as abstracts or editorials, trials including healthy volunteers or animals, quasi-randomized trials, and articles referring to subgroups of patients from randomized trials were excluded. Eligible trials were identified through The Cochrane Library m and MEDLINE. The Cochrane Inflammatory Bowel Diseases Group ~° hand-searched all issues of GASTROENTEROLOGYfrom 1964 to 1998, and one of the authors (S.L.F.) hand-searched the 1999 issues. We assessed the methodological quality and extracted the sample size (number of patients/intervention arm), disease area, and year of publication of each trial. Two contributors (S.L.F. and L.L.K.) performed the data abstraction. Consensus was reached before analyses. The methodological quality was assessed by the following components. (1) Generation of the allocation sequence: adequate (e.g., computer-generated random numbers), unclear (not described), or inadequate (quasirandomized trials, which were excluded). (2) Allocation concealment: adequate (e.g., central independent unit, sealed envelopes [irrespective of whether authors reported that the envelopes were opaque and sequentially numbered]), unclear (not described), or inadequate (e.g., open table of random numbers). (3) Double blinding: adequate (e.g., outcome assessors and/or caretakers and patients were kept unaware of treatment allocation by identical placebo), unclear (not described), inadequate (e.g., tablets vs. injections), or not double blind. The quality assessments were unmasked because we previously found no significant difference between masked and unmasked assessments) The intraclass correlation coefficients of the quality assessment (generation of the allocation seAbbreviation used in this paper: CONSORT,ConsolidatedStandards of ReportingTrials. © 2002 by the American GastroenterologicalAssociation 0016-5085/02/$35.00 doi:lO.1053/gast.2002.32390
1158
~AERGARDETAL.
quence, 0.86 [95% confidence interval, 0.70-0.93]; allocation concealment, 0.84 [95% confidence interval, 0.68-0.93]; double blinding, 0.74 [95% confidence interval, 0.50-0.87]) showed high interobserver reliability. Statistical Analyses
The data were analyzed for discrepancies between trials stratified according to publication year. The primary outcome was the methodological quality (generation of the allocation sequence and allocation concealment [adequate vs. unclear/ inadequate] plus double blinding [adequate vs. unclear/inadequate/not performed]). Categorical variables were assessed by Pearson Xe test and continuous data by Kendall "r-b. Discrepancies between groups of trials were also assessed by analysis of variance with adjustments for potential confounders. The sample size was log transformed to obtain homogeneity of variance. The level of significance was set to 5 %. The analyses were performed in SPSS version 10.0 for Windows (SPSS Inc., Chicago, IL). Results
W e identified 1009 references through MEDLINE, 794 references through The Cochrane Library, 1° and 7 trials through the hand-search of 1999 issues. Of these references, 1229 were duplicates, abstracts, editorials, or reviews and 30 were quasi-randomized trials. The remaining 551 references were retrieved. O f these references, we excluded 75 studies on healthy volunteers; 2 quasi-randomized trials; 33 articles referring to subgroups from randomized trials, pathophysiologic studies, or animal studies; and 64 abstracts, editorials, and reviews. Accordingly, the present study includes 377 original articles reporting 383 randomized trials (available on request). Fifty percent of all trials were published after 1988. The generation of the allocation sequence was adequate in 160 trials and unclear in the remaining trials. The allocation concealment was adequate in 150 trials, unclear in 228 trials, and inadequate in 5 trials. Adequate double blinding was reported in 239 trials. O f these trials, 192 used placebo, 40 assessed different drugs, 5 dealt with surgery, and 2 dealt with diagnostic procedures or nursing interventions. The method of blinding was unclear in 8 trials, and 136 trials were not double blind. O f trials without adequate double blinding, 95 dealt with drugs, 34 with surgery, and 15 with diagnostic procedures or nursing interventions. On average, the trials included 43 patients per intervention arm (standard error of the mean, 4). The included trials were grouped into 15 disease areas. A total of 53 trials dealt with duodenal ulcer, 51 with inflammatory bowel diseases, 29 with upper gastrointestinal diseases, 26 with esophageal diseases, 26 with chronic hepatitis, 25 with
GASTROENTEROLOGYVol. 122, No. 4
diarrhea, 25 with cirrhosis, 23 with colorectal diseases, 20 with miscellaneous hepatobiliary diseases, 19 with miscellaneous gastrointestinal diseases, 19 with primary biliary cirrhosis, 19 with portal hypertension, 18 with biliary diseases, 16 with alcoholic liver diseases, and 14 with extra-abdominal diseases. The proportion of trials with adequate generation of the allocation sequence increased significantly during the study period (P = 0.007), as did the proportion of trials with adequate allocation concealment (P = 0.011, Figure 1). The improvements primarily occurred after 1994. After adjusting for confounding (sample size and disease area), the year of publication was a significant predictor of the adequacy of the generation of the allocation sequence (r 2 = 0.12; P = 0.048) and allocation concealment (r 2 = 0.14; P = 0.007). The proportion of doubleblind trials did not increase significantly during the study period before (P = 0.11) or after (r e = 0.14; P = 0.18) adjusting for confounding. The year of publication was significantly associated with the sample size without adjustments (P = 0.001, Figure 2) but not after adjusting for confounding (methodological quality and disease area) (r 2 = 0 . 0 0 5 ; P = 0.17). The adjusted analyses also showed a significant association between the disease area and the methodological quality (P = 0.030) and sample size (P = 0.002).
Discussion
The reported randomization methods of trials in GASTROENTEROLOGY have improved considerably since
1964, especially from 1995 onward. Nevertheless, several trials still lack this information. From 1991 to 2000, 54% of the trials did not report how the allocation sequence was generated and 56% did not report the method of allocation concealment. The present study has limitations. First, it is uncertain whether the results reflect general trends in gastroenterological research. Previous evidence on this issue is equivocal. Trials published in the Journal of Hepatology from 1985 to 1997 showed similar improvements, v but trials published in Hepatology from 1981 to 1998 did not. 9 Second, the design does not allow for an assessment of the association between methodological quality and the estimated intervention effects in the included trials. The implications therefore rest on previous studies on trials from other disease areas) 4 These studies have shown a significant association between unclear randomization methods and exaggerated estimates of intervention effects.
April 2002
RANDOMIZED TRIALS IN GASTROENTEROLOGY 1159
Proportion of trials with adequate allocation concealment
Proportion of lrials with adequate allocation concealment
0.60 0.80 0.50
0.60
I
0.40
0.30
+
0.40
0.20' 0.20 I
I
1964-72
1973-81
I
tttt
I
1982-90
1991-99
0.00'
!
1991
I
I
1993
I
I
1995
I
l
1997
I
I
1999
Figure 1. Mean proportion of trials (standard error of the mean) with adequate allocation concealment from 1964 to 2000 in 9-year intervals and from 1991 to 2000.
We were unable to assess whether the improvements reflect changes at the trial level., the review process, or editorial policies. Inadequate or unclear reporting may conceal methodological flaws. We did not write to the
Number of patients per intervention arm
,f
60
50
40
30
20
10 I
1964-72
|
1973-81
I
1982-90
I
1991-99
Figure 2. Mean number of patients per intervention arm (standard error of the mean) in randomized trials from 1964 to 2000 in 9-year intervals.
authors for additional information, because authors of trials published before the 1990s are difficult to locate. Furthermore, previous evidence indicates that communication with authors provides little additional information about methodological quality. 11 It is likely that the improvements are a combined action by researchers, peer reviewers, and editors. The rate of acceptance of submissions to GASTROENTEROLOGYdeclined considerably during the 1990s, le increasing the demands for originality and quality. During the 1990s, several international initiatives focused on the need for improving the quality of clinical trials. The Cochrane Collaboration l° and International Conference on Harmonisation Expert Working Group, 13 studies showing the importance of methodological quality,U14 and reporting guidelines 15,16 may have had a positive effect. A recent study ~v indicates a positive effect of the Consolidated Standards of Reporting Trials (CONSORT) statement, 18 which provides guidelines for the reporting of the assignment, masking, follow-up, and statistical analysis. However, the improvements in GASTROENTEROLOGYoccurred before the publication of the CONSORT statement, ~8 and the changes must therefore reflect other initiatives. Several methods have been suggested for the assessment of methodological quality, but few are validated. .9 Because the conclusions of most methods are questionable,/9 we focused on components supported by empirical evidence) -4 However, other aspects may be important (e.g., external validity). The quality of a trial
1160
KJAERGARD ET AL.
therefore also depends on clear reporting of the characteristics of the included patients, setting, treatment regimens, and outcomes assessed. 2o Nevertheless, the question of external validity becomes irrelevant if the results are biased. In conclusion, the reporting of randomization methods has improved significantly in GASTROENTEROLOGY. It is likely that the recent decision of GASTROENTEROLOGY21 to adopt the C O N S O R T statement 22.e3 will support this positive development.
References 1. Schulz KF, Chalmers i, Hayes RJ, AItman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273:408-412. 2. Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, Tugwell P, Klassen TP. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1 9 9 8 ; 3 5 2 : 6 0 9 - 6 1 3 . 3. Kjaergard LL, Villumsen J, Gluud C. Reported methodological quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982-989. 4. J~ni P, Altman D, Egger M. Assessing the quality of controlled clinical trials. BMJ 2 0 0 1 ; 3 2 3 : 4 2 - 4 6 . 5. Peto R, Baigent C. Trials: the next 50 years. Large scale randomised evidence of moderate benefits. BMJ 1998;317:1170-1171. 6. Juhl E, Christensen E, Tygstrup N. The epidemiology of the gastrointestinal randomized clinical trial. N Engl J Med 1977;296: 20-22. 7. Gluud C, Nikolova D. Quality assessment of reports on clinical trials in the Journal of Hepatology. J Hepatol 1 9 9 8 ; 2 9 : 3 2 1 - 3 2 7 . 8. Gluud C. Evidence based medicine in Liver. Liver 1999;19:1-2. 9. Kjaergard LL, Nikolova D, Gluud C. Randomized clinical trials in Hepatology: predictors of quality. Hepatology 1 9 9 9 ; 3 0 : 1 1 3 4 1138. 10. The Cochrane Library, Issue 2, 2000. Oxford: Update Software. 11. Liberati A, Himel HN, Chalmers TC. A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 1986;4:942-951. 12. LaMont TJ, Link AM, Podolsky DK. We are pleased to inform y o u . . . Gastroenterology 1 9 9 8 ; 1 1 4 : 1 1 2 3 - 1 1 2 4 . 13. International Conference on Harmonisation Expert Working Group. Code of federal regulations & international conference on harmonization guidelines. Philadelphia: Parexel Barnett, 1997.
GASTROENTEROLOGY Vol. 122, No. 4
14. Chalmers TC, Celano P, Sacks HS, Smith H. Bias in treatment assignment in controlled clinical trials. N Engl J Med 1983;309: 1358-1361. 15. A proposal for structured reporting of randomized controlled trials. The Standards of Reporting Trials Group. JAMA 1994;272: 1926 -1931. 16. Call for comments on a proposal to improve reporting of clinical trials in the biomedical literature. Working Group on Recommendations for Reporting of Clinical Trials in the Biomedical Literature. Ann Intern Med 1 9 9 4 ; 1 2 1 : 8 9 4 - 8 9 5 . 17. Moher D, Jones A, Lepage L, for the CONSORT Group. Use of the CONSORT statement and quality of reports of randomized trials. JAMA 2001;285:1992-1995. 18. Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1 9 9 6 ; 2 7 6 : 6 3 7 - 6 3 9 . 19. JL~ni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999;282:10541060. 20. Altman DG, Bland JM. Generalisation and extrapolation. BMJ 1998;317:409-410. 21. Sandier RS. Reporting randomised controlled trials in Gastroenterology: the Consort Statement. Gastroenterology 2001;121: 755. 22. Moher D, Schulz KF, Altman D, CONSORT GROUP (Consolidated Standards of Reporting Trials). The CONSORT statement: revised recommendations for improving the quality of reports of parallelgroup randomized trials. Ann Intern Med 2 0 0 1 ; 1 3 4 : 6 5 7 - 6 6 2 . 23. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, G#tzsche PC. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 2001;134:663-694.
Received May 10, 2001. Accepted November 29, 2001. Address requests for reprints to: Christian Gluud, M.D., The Cochrane Hepato-Biliary Group, Copenhagen Trial Unit, Center for Clinical Intervention Research, Dept. 7102, Blegdamsvej 9, H:S Rigshospitalet, DK-2100 Copenhagen, Denmark. e-maih [email protected]; fax: (45) 3545-7101. Supported by grants from the Danish Medical Research Council and the 1991 Pharmacy Foundation in Denmark. The authors thank Ninna Frydendall and Dimitrinka Nikolova for secretarial assistance, Nader Salasshahri for computer assistance, and The Cochrane Inflammatory Bowel Diseases Group for their handsearches of GASTROENTEROLOGY.