- Email: [email protected]
Funding, disease area, and internal validity of hepatobiliary randomized clinical trials
THE AMERICAN JOURNAL OF GASTROENTEROLOGY © 2002 by Am. Coll. of Gastroenterology Published by Elsevier Science Inc.
Vol. 97, No. 11, 2002 ISSN 0002-9270/02/$22.00 PII S0002-9270(02)05484-9
SPECIAL ARTICLE
Funding, Disease Area, and Internal Validity of Hepatobiliary Randomized Clinical Trials Lise Lotte Kjaergard, M.D., and Christian Gluud, M.D., Dr.Med.Sci. The Cochrane Hepatobiliary Group, The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Rigshospitalet, Denmark
OBJECTIVE: The aim of this study was to assess whether funding and the disease area are related to the internal validity of hepatobiliary randomized clinical trials. METHODS: We gathered data on funding, disease area, methodological quality (randomization and double blinding), and sample size from 616 hepatobiliary randomized clinical trials published from 1985 to 1996 in 12 MEDLINE indexed journals. RESULTS: The internal validity (methodological quality and sample size) of trials funded by profit or nonprofit organizations was not significantly different. Compared with these trials, trials without funding were significantly less likely to report adequate generation of the allocation sequence (55% vs 41%, p ⫽ 0.001) and to be double blind (42% vs 25%, p ⬍ 0.001), but the proportion with adequate allocation concealment and the sample size were not significantly different. The trials covered 12 disease areas. The proportion of funded trials did not differ significantly in different disease areas. The disease area was significantly associated with the proportion of trials with adequate generation of the allocation sequence (p ⬍ 0.001), allocation concealment (p ⫽ 0.003), and double blinding (p ⬍ 0.001) as well as the sample size (p ⬍ 0.001). This association was not explained by the proportion of trials with funding. CONCLUSIONS: External funding was significantly associated with adequate methodological quality, but not with the sample size. Irrespective of funding, the disease area was significantly associated with the methodological quality and sample size. Accordingly, external funding and the disease area are significant predictors of the internal validity of hepatobiliary randomized clinical trials. (Am J Gastroenterol 2002;97:2708 –2713. © 2002 by Am. Coll. of Gastroenterology)
INTRODUCTION Traditionally, clinical decisions have been made on the basis of animal models, expert knowledge, clinical experience, or similar (1). Evidence-based medicine may, therefore, be viewed as a paradigm shift as it involves the use of the best patient-centered research evidence in clinical decision making (2). Based on the estimated risk of bias, a hierarchy of
evidence has been proposed (1). The evidence-based practitioner must seek to identify and use evidence from the highest possible level in this hierarchy. The quality of the evidence should also be graded within each level. The lowest rank has been assigned to uncontrolled clinical observations and physiological experiments based on human or animal models (1). This ranking is based on the lack of reliable comparison groups, unsystematic collection of data, and the necessity for extrapolation. Clinical controlled studies have a higher ranking. Such studies can be randomized or observational. Several studies have found that observational studies have a considerable risk of generating exaggerated estimates of intervention effects compared with randomized clinical trials (3). Observational studies, therefore, have a lower ranking than randomized clinical trials. Randomized clinical trials and systematic reviews have the highest ranking in the hierarchy of clinical intervention research (4, 5). However, such evidence may also be biased (5). Inadequate sample size or inadequate methodological quality can lead to false-negative or false-positive conclusions (5– 8). Compared with large “gold-standard” randomized clinical trials, intervention effects were significantly more positive in small trials without adequate allocation sequence generation, allocation concealment, or double blinding (5). This indicates that small randomized trials with inadequate methodological quality have a considerable risk of bias and therefore of producing exaggerated estimates of intervention effects. On the other hand, the risk of generating false-negative results is also considerable in small trials because of inadequate statistical power. The internal validity of randomized clinical trials, therefore, depends on the methodological quality and sample size. In four major hepatobiliary specialist journals, several hepatobiliary randomized clinical trials were small and had inadequate methodological quality (9 –12). Previous evidence indicates that the extent of these problems varies significantly in different disease areas (11, 12). This variation could reflect financial interests, as external funding has also been associated with adequate methodological quality (11). Funding may also be associated with larger sample sizes. Trials funded by profit organizations more often indicate a significant benefit of innovative interventions than trials funded by nonprofit organizations (13). Inadequate
AJG – November, 2002
methodological quality or greater statistical power of trials funded by for-profit organizations could explain this association (14). In the present study, we have examined whether funding and the disease area are associated with the internal validity of hepatobiliary randomized clinical trials.
MATERIALS AND METHODS The present study includes all hepatobiliary randomized clinical trials that were published during 1985 to 1996 in the following 12 English-language, MEDLINE-indexed journals, which had been handsearched by The Cochrane Collaboration (15): The New England Journal of Medicine, Annals of Internal Medicine, Lancet, Gastroenterology, Gut, Hepatology, Journal of Hepatology, Digestive Diseases and Sciences, Digestion, Liver, Scandinavian Journal of Gastroenterology, and Journal of Clinical Gastroenterology. This study period chosen as 1985 was the first publication year for Journal of Hepatology and as our searches for eligible trials were initiated in 1997. Trials were considered randomized if some form of the word random was used to describe the allocation of patients. Quasirandomized trials and articles referring to subgroups of patients from randomized trials were excluded. The Cochrane Hepatobiliary Group (CHBG) Controlled Trials Register includes all trials identified by the CHBG (15). We, therefore, also searched The CHBG Controlled Trials Register to identify eligible trials for the present study. The CHBG Controlled Trials Register was also searched to estimate the total number of original article hepatobiliary randomized clinical trials published during 1985 to 1996. Data Extraction From each of the included trials, we gathered data on the methodological quality, sample size, disease area (areas with fewer than 10 trials were classified as miscellaneous), and funding. The data extraction was performed unblinded. Funding was classified as for-profit, nonprofit, or no funding. Funding was extracted from the authors’ acknowledgments. The text was scanned for potential sources of funding, in case funding was not reported in the acknowledgments. We defined for-profit organizations as any organization that may incur financial gain or loss depending on the outcome of a trial. Medical research councils, national institutes of health, and any organization that would not have financial gain or loss depending on the outcome of the trial were classified as nonprofit. Some trials were funded by both profit and nonprofit organizations. Based on previous evidence, these trials were grouped with trials funded by nonprofit organizations alone (14). We estimated the interobserver reliability of the classification of funding using 50 arbitrarily selected trials included in the present study. The observers disagreed on the classification of funding in only five of the 50 trials ( ⫽ 0.79), indicating high interobserver agreement.
Internal Validity of Hepatobiliary Trials
2709
According to previous evidence (5– 8), methodological quality was assessed by the generation of the allocation sequence, allocation concealment, and double blinding. The purpose of randomization is to avoid selection bias (5– 8). Adequate randomization does not mean the same as hazard but that each patient has a known (usually equal) chance of receiving the interventions assessed and that the allocation of the next patient cannot be predicted (16). Adequate randomization, therefore, involves both adequate allocation sequence generation and adequate allocation concealment. The allocation sequence should be based on random numbers generated by a table, computer, tossing of a coin, or shuffling of cards. This sequence should be concealed until the moment of randomization by serially numbered opaque sealed envelopes, identically appearing containers, or by a central independent unit. Using these definitions, we classified the generation of the allocation sequence as adequate, unclear (not described), or inadequate (quasirandomized trials, which were excluded). The allocation concealment was classified as adequate, unclear (not described), or inadequate. The purpose of double blinding is to avoid performance and detection bias (8). For a trial to be adequately double blinded, both health care providers/assessors and patients must be kept unaware of the intervention allocated to the individual patient. Double blinding can be adequate if the interventions to be compared are identical with regard to taste, smell, and mode of administration. If an intervention is to be compared with no intervention, double blinding required the use of an inactive placebo. Using these definitions, we classified double blinding as adequate, unclear (not described), inadequate, or not performed. We assessed the interobserver reliability for the methodological quality assessment using 100 arbitrarily selected trials included in the present study. The observers disagreed on the classification of the allocation sequence generation in seven trials ( ⫽ 0.82), allocation concealment in eight trials ( ⫽ 0.83), and double blinding in 11 trials ( ⫽ 0.78). Accordingly, there was a high interobserver agreement in the assessment of methodological quality. The data extraction was based on standardized extraction sheets generated in Lotus Notes (227) version 5.05 (IBMLotus; IBM Corporation, Armonk, New York). According to these sheets, methodological quality was extracted before funding. However, knowledge of funding could influence the assessment of methodological quality as the source of funding was not blinded. To estimate whether this was the case, we compared the methodological quality assessment of two observers with and without blinding of funding. In an arbitrarily selected sample of 50 trials included in the present study, the observers disagreed on the classification of the allocation sequence generation in four trials ( ⫽ 0.73), allocation concealment in four trials ( ⫽ 0.83), and double blinding in five trials ( ⫽ 0.78). These results indicate that knowledge of funding did not bias the assessment of methodological quality significantly.
2710
Kjaergard and Gluud
AJG – Vol. 97, No. 11, 2002
Table 1. Number and Proportion of Included Randomized Clinical Trials in Different Hepatobiliary Disease Areas
Disease Area
Number of Trials
Proportion of Total (%)
Portal hypertension Hepatitis B Biliary tract diseases Hepatitis C Cirrhosis Hepatic encephalopathy Miscellaneous Primary biliary cirrhosis Alcoholic liver diseases Malignant diseases Surgical procedures* Autoimmune liver diseases
150 105 78 68 39 39 35 34 21 21 15 11
24 17 13 11 6 6 6 6 3 3 2 2
* Trials from several disease areas.
Statistical Analysis Categorical data are summarized as proportions and continuous data as medians with 5–95 percentiles. Proportions are compared using 2 and continuous data using Mann-Whitney U test or Kruskal-Wallis test. To assess whether funding can explain discrepancies between disease areas, we performed sensitivity analyses in which trials were stratified according to funding. Significance was accepted at the 0.05 level. The analyses were performed in SPSS version 10.0 for Windows (SPSS Inc, Chicago, Illinois).
RESULTS According to The CHBG Controlled Trials Register, 2134 hepatobiliary randomized clinical trials were published as original articles during 1985 to 1996. This study includes 616 (29%) of these trials (references available on request). The generation of the allocation sequence was adequate in 48% and unclear in 52% of the included trials. The allocation concealment was adequate in 38% and unclear in 62% of the trials. Adequate double blinding was reported in 34%, and 66% were not double blind. The median sample size was 23 patients per intervention arm (percentiles ⫽ 7–102 patients per intervention arm). Forty-seven percent of the trials received no external funding, 30% were funded by nonprofit organizations, and 23% were funded by for-profit organizations. The trials were classified into 12 disease areas (Table 1). Trials on portal hypertension primarily dealt with endoscopic or pharmacological interventions for the prevention and treatment of bleeding esophageal varices. Trials on acute chronic hepatitis B dealt with vaccines or antiviral drugs. Trials on hepatitis C dealt with antiviral drugs. Trials on biliary tract diseases primarily dealt with medical or surgical interventions for cholecystitis or gallbladder stones. The majority of trials on cirrhosis, primary biliary cirrhosis, hepatic encephalopathy, alcoholic liver diseases, fulminant hepatic failure, and liver transplantation assessed medical interventions.
Table 2. Proportion of Randomized Clinical Trials With Adequate Methodological Quality in Different Hepatobiliary Disease Areas Proportion of Trials With Adequate
Disease Area
Allocation Sequence Allocation Double Generation* Concealment† Blinding* (%) (%) (%)
Alcoholic liver diseases Autoimmune liver diseases Biliary tract diseases Cirrhosis Hepatic encephalopathy Hepatitis B Hepatitis C Malignant diseases Miscellaneous Portal hypertension Primary biliary cirrhosis Surgical procedures‡
52 64
43 46
67 55
24 44 59 44 34 62 34 68 44 60
27 18 49 30 35 48 40 50 38 53
47 33 36 22 13 19 31 29 88 40
* p ⬍ 0.001. † p ⫽ 0.003. ‡ Trials from several disease areas.
Trials on malignant liver diseases assessed chemotherapy or transcatheter embolization of hepatocellular carcinoma or metastases. The group of trials that were classified as miscellaneous dealt with primary sclerosing cholangitis, hepatitis A, congenital liver diseases, porphyria, or fatty liver. Funding and Internal Validity Trials funded by nonprofit organizations did not differ significantly from trials funded by for-profit organizations regarding the proportion with adequate generation of the allocation sequence (58% vs 50%, p ⫽ 0.13), adequate allocation concealment (37% vs 36%, p ⫽ 0.83), adequate double blinding (38% vs 46%, p ⫽ 0.11), or sample size (median number of patients per intervention arm ⫽ 23 [percentiles ⫽ 7–140] vs 23 [percentiles ⫽ 6 –118], p ⫽ 0.25). In the analyses to follow, we compared all trials with external funding (i.e., trials funded by nonprofit and forprofit organizations) with trials without funding. Compared with funded trials, trials without funding were significantly less likely to report adequate generation of the allocation sequence (55% vs 41%, p ⫽ 0.001) and to be double blind (42% vs 25%, p ⬍ 0.001). There were no significant differences between funded trials and trials without funding regarding the proportion with adequate allocation concealment (37% vs 40%, p ⫽ 0.40) or sample size (median number of patients per intervention arm 23 [percentiles ⫽ 7–132] vs 24 [percentiles ⫽ 7–90], p ⫽ 0.51). Internal Validity and Funding in Different Disease Areas As shown in Table 2, the 12 disease areas differed significantly regarding the proportion of trials with adequate generation of the allocation sequence (p ⬍ 0.001), allocation concealment (p ⫽ 0.003), and double blinding (p ⬍ 0.001).
AJG – November, 2002
Internal Validity of Hepatobiliary Trials
Table 3. Sample Size (Number of Patients Per Intervention Arm) of Randomized Clinical Trials in Different Hepatobiliary Disease Areas Disease Area
Median*
(5–95 Percentiles)
Alcoholic liver diseases Portal hypertension Primary biliary cirrhosis Malignant diseases Hepatitis C Surgical procedures† Biliary tract diseases Hepatitis B Cirrhosis Hepatic encephalopathy Miscellaneous Autoimmune liver diseases
36 34 29 28 27 26 20 18 16 15 15 12
(6–149) (9–88) (6–112) (9–132) (9–122) (10–265) (5–90) (6–96) (6–58) (6–33) (3–510) (6–41)
* p ⬍ 0.001. † Trials from several disease areas.
The median number of patients per intervention arm in different disease areas also differed significantly (p ⬍ 0.001, Table 3). The proportion of trials with funding was not significantly different in different disease areas (p ⫽ 0.11). According to the sensitivity analyses including funded trials only, the disease areas differed significantly regarding the proportion of trials with adequate generation of the allocation sequence (p ⬍ 0.001), allocation concealment (p ⫽ 0.004), double blinding (p ⬍ 0.001), and sample size (p ⬍ 0.001). According to the sensitivity analyses including only trials without funding, the disease areas differed significantly regarding the proportion of trials with adequate generation of the allocation sequence (p ⬍ 0.001), double blinding (p ⬍ 0.001), and sample size (p ⬍ 0.001), but not the proportion of trials with adequate allocation concealment (p ⫽ 0.36).
DISCUSSION This study includes almost one third of all hepatobiliary randomized trials published as original articles during an 11-yr period. Several of these trials had a considerable risk of producing false-positive or false-negative results because of inadequate methodological quality (5– 8) or sample size (5, 17, 18). The internal validity (methodological quality and sample size) of trials funded by nonprofit organizations and/or for-profit organizations was not significantly different. All trials with funding were significantly more likely to report adequate generation of the allocation sequence and to be double blind than trials without funding. Trials with funding did not differ significantly from trials without funding with regard to the adequacy of the allocation concealment or sample size. The proportion of funded trials did not differ significantly between disease areas, but the internal validity did. The variations in internal validity in different disease areas were not explained by funding. We included a heterogeneous group of trials from several hepatobiliary disease areas. The variety of the included trials
2711
increases the external validity but excludes the possibility of analyzing the direct association between methodological quality and intervention effects in the included trials. This association, therefore, has to be inferred from previous evidence (5– 8). Three empirical studies have assessed the association between methodological quality and intervention effects in randomized clinical trials included in metaanalyses from several different disease areas including cardiology, gynecology, obstetrics, psychiatry, and hematology. Methodological quality, i.e., the confidence that a trial has adequate bias control, can be assessed by the risk of selection bias, performance bias, and detection bias. Such bias may be prevented through adequate randomization and double blinding. On average, the estimated intervention effects generated by small trials with adequate randomization or double blinding and large “gold-standard” trials were not significantly different (5). Conversely, small trials without adequate randomization or double blinding exaggerated intervention effects significantly by about 50% (5). There is no similar evidence documenting the association between methodological quality and intervention effects in hepatobiliary randomized clinical trials. However, we cannot find any reason why such trials should be immune to bias. Furthermore, systematic reviews point to a similar tendency of exaggerated estimates of intervention effects in low-quality hepatobiliary randomized clinical trials (19, 20). In accordance with previous studies on the methodological quality of randomized clinical trials, we based our quality assessments on the published reports. We were, therefore, not able to distinguish between what was reported and how the trials were designed and conducted. In theory, inadequate (i.e., unclear) reporting may conceal methodological flaws. It may, therefore, be considered a weakness that we did not write to authors for additional information. On the other hand, previous evidence (5– 8) indicates that inadequate reporting reflects inadequate trial design and conduct. Furthermore, even when including only trials published during the 1990s, only about 20% of authors reply to requests for additional information about their trials (21, 22). Accordingly, although methodological quality and funding may be inconsistently reported, correspondence with authors would provide only little additional information. Several methods for assessment of quality have been suggested, but few have been validated (23, 24). It has been shown that the conclusions of different quality scales can be equivocal (24). We, therefore, limited the quality assessment to components, which have been supported by empirical evidence (5– 8). However, the term quality can refer to several aspects. For example, the clinical value of a trial also depends on the external validity, and the quality of a trial, therefore, also depends on the reporting of patient characteristics, the setting, the interventions, and the outcomes. However, the question of the external validity becomes irrelevant if a trial lacks internal validity. We assessed the methodological quality of trials by the
2712
Kjaergard and Gluud
reported randomization methods and double blinding. It may be argued that double blinding cannot be used as a measure of quality in larger samples of trials as the nature of some interventions precludes adequate double blinding of caretakers and patients. Single blinding (e.g., of outcome assessors or data analyses) is always possible, but is rarely reported and has not been associated with intervention effects. We did not perform analyses that were adjusted for the feasibility of blinding. We did, however, compare the proportion of double-blind trials in different disease areas and found a significant variation. In some disease areas, e.g., hepatitis C, all known interventions are drugs and, therefore, possible to assess in a double-blind manner. In other areas, e.g., treatment of biliary tract diseases, surgical interventions are more common. This may influence the proportion of double-blind trials in different disease areas and could explain why we found a significant association between the disease area and the proportion of double-blind trials. Likewise, if funded trials more often dealt with drugs than trials without funding, this could influence the association between funding and double blinding. On the other hand, adequate randomization is always possible, irrespective of the intervention assessed. We found a significant association between funding, disease area, and the adequacy of the reported randomization methods. This supports the assumption that the methodological quality is associated with the disease area and funding, irrespective of the type of intervention assessed. Two previous studies indicate that trials funded by forprofit organizations are significantly more likely to indicate a significant benefit of experimental interventions compared with trials funded by nonprofit organizations (13, 14). This discrepancy could reflect variations in methodological quality or statistical power. We found that the methodological quality and sample size of trials funded by for-profit organizations was not significantly different from nonprofit organizations. Accordingly, the present study supports previous findings indicating that the association between funding and authors’ conclusions in randomized clinical trials does not reflect variations in the internal validity (14). This study confirms that several hepatobiliary trials are small and of inadequate methodological quality (9 –12). As the results are based on a large proportion of the total number of published trials, it is likely that the problem is general. The present study also supports previous findings indicating that there is a positive association between funding and methodological quality (11) and shows that this association does not explain the significant variation of internal validity in different disease areas. The internal validity seemed to be relatively high in rare diseases. As trials on rare diseases often have to be performed as multicenter trials, this finding could indicate that cooperation of researchers may not only increase the possibility of including more patients, but also increase the methodological quality (11). There was no apparent association between the prev-
AJG – Vol. 97, No. 11, 2002
alence of the diseases and the average sample size of the trials. The present study demonstrates a significant positive association between external funding and adequate methodological quality. This association may reflect better resources. Furthermore, in most cases, obtainment of funding usually requires the preparation of a thorough protocol, which is reviewed by persons not involved in the trial. The association between funding and adequate quality could, therefore, also be caused by the positive effect generated by a critical mass. The reasons for the significant association between the disease area and internal validity are less obvious. Possibly, this association could reflect initiatives, such as methodological workshops (25, 26). Despite the considerable variation, the present study demonstrates several weaknesses within hepatobiliary intervention research. Low-quality evidence may lead to the implementation of useless or even harmful interventions, and the results, therefore, suggest that several hepatobiliary health care interventions need to be reexamined (27). We suggest that more attention is being paid to the requirements for adequate statistical power and methodological quality, e.g., following the recommendations given in the Consolidated Standard of Reporting Trials (CONSORT) statement (www.consortstatement.org).
ACKNOWLEDGMENTS This work was supported by grants from The Danish Medical Research Council and The 1991 Pharmacy Foundation in Denmark. We thank M. Kjaer, D. Nikolova, S. Frederiksen, and N. Salasshahri for their assistance with the identification and retrieval of trials. Reprint requests and correspondence: Lise Lotte Kjaergard, M.D., The Cochrane Hepatobiliary Group, The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Department 7102, H:S Rigshospitalet, DK-2100, Copenhagen, Denmark. Received Mar. 12, 2002; accepted June 24, 2002.
REFERENCES 1. Guyatt GH, Sackett DL, Cook DJ. Users’ guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? EvidenceBased Medicine Working Group. JAMA 1993;270:2598 – 601. 2. Sackett D, Rosenberg WM, Gray JA, et al. Evidence based medicine: What it is and what it isn’t. BMJ 1996;312:71–2. 3. Kunz R, Oxman A. The unpredictability paradox: Review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998;317:1185–90. 4. Peto R, Baigent C. Trials: The next 50 years. Large scale randomised evidence of moderate benefits. BMJ 1998;317: 1170 –1. 5. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982–9. 6. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality asso-
AJG – November, 2002
7. 8. 9. 10. 11. 12. 13. 14.
15. 16. 17.
18.
ciated with estimates of treatment effects in controlled trials. JAMA 1995;273:408 –12. Moher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609 –13. Ju¨ ni P, Altman D, Egger M. Assessing the quality of controlled clinical trials. BMJ 2001;323:42– 6. Gluud C, Nikolova D. Quality assessment of reports on clinical trials in the Journal of Hepatology. J Hepatol 1998;29: 321–7. Gluud C. Evidence based medicine in Liver. Liver 1999;19: 1–2. Kjaergard LL, Nikolova D, Gluud C. Randomized clinical trials in Hepatology: Predictors of quality. Hepatology 1999; 30:1134 – 8. Kjaergard LL, Frederiksen SL, Gluud C. Validity of randomized clinical trials in Gastroenterology from 1964 –2000. Gastroenterology 2002;122:1157– 60. Djulbegovic B, Lacevic M, Cantor A, et al. The uncertainty principle and industry-sponsored research. Lancet 2000;356: 635– 8. Kjaergard LL, Als-Nielsen B. The association between competing interests and authors’ conclusions: Epidemiological study of randomised clinical trials published in the BMJ. BMJ 2002;325:249 –52. Clarke M, Oxman AD, eds. Cochrane reviewers’ handbook 4.1.5. In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software. Altman DG. Randomisation. BMJ 1991;302:1481–2. Collins R, Gray R, Godwin J, Peto R. Avoidance of large biases and large random errors in the assessment of moderate treatment effects: The need for systematic overviews. Stat Med 1987;6:245–54. Peto R, Collins R, Gray R. Large-scale randomized evidence: Large, simple trials and overviews of trials. Ann N Y Acad Sci 1993;703:314 – 40.
Internal Validity of Hepatobiliary Trials
2713
19. Rambaldi A, Gluud C. Colchicine for alcoholic and nonalcoholic liver fibrosis and cirrhosis (Cochrane review). In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software. 20. D’Amico G, Pietrosi G, Tarantino I, Pagliaro L. Emergency sclerotherapy versus medical interventions for bleeding oesophageal varices in cirrhotic patients (Cochrane review). In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software. 21. Kjaergard LL, Krogsgaard K, Gluud C. Interferon alfa with or without ribavirin for chronic hepatitis C: Systematic review of randomised trials. BMJ 2001;323:1151–5. 22. Kjaergard LL, Krogsgaard K, Gluud C. Ribavirin with or without alpha interferon versus no intervention, placebo or alpha interferon for chronic hepatitis C (Cochrane review). In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software. 23. Moher D, Jadad AR, Nichol G, et al. Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Control Clin Trials 1995;16:62–73. 24. Ju¨ ni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999; 282:1054 – 60. 25. Burroughs AK. Methodology and reviews of clinical trials in portal hypertension. Amsterdam: Excerpta Medica, 1987. 26. De Franchis R. What have we accomplished? In: De Franchis R, ed. Portal hypertension III. Proceedings of the Third Baveno International Consensus Workshop on Definitions, Methodology and Therapeutic Strategies. Oxford: Blackwell Science, 2002:1–12. 27. Gluud C, on behalf of The Cochrane Hepatobiliary Group Editorial Team. The Cochrane Hepatobiliary Group. In: Bircher J, Benhamou JP, McIntyre N, Rizzetto M, Rodes J, eds. Oxford textbook of clinical hepatology. Oxford: Oxford University Press, 1999:2145– 8.
Vol. 97, No. 11, 2002 ISSN 0002-9270/02/$22.00 PII S0002-9270(02)05484-9
SPECIAL ARTICLE
Funding, Disease Area, and Internal Validity of Hepatobiliary Randomized Clinical Trials Lise Lotte Kjaergard, M.D., and Christian Gluud, M.D., Dr.Med.Sci. The Cochrane Hepatobiliary Group, The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Rigshospitalet, Denmark
OBJECTIVE: The aim of this study was to assess whether funding and the disease area are related to the internal validity of hepatobiliary randomized clinical trials. METHODS: We gathered data on funding, disease area, methodological quality (randomization and double blinding), and sample size from 616 hepatobiliary randomized clinical trials published from 1985 to 1996 in 12 MEDLINE indexed journals. RESULTS: The internal validity (methodological quality and sample size) of trials funded by profit or nonprofit organizations was not significantly different. Compared with these trials, trials without funding were significantly less likely to report adequate generation of the allocation sequence (55% vs 41%, p ⫽ 0.001) and to be double blind (42% vs 25%, p ⬍ 0.001), but the proportion with adequate allocation concealment and the sample size were not significantly different. The trials covered 12 disease areas. The proportion of funded trials did not differ significantly in different disease areas. The disease area was significantly associated with the proportion of trials with adequate generation of the allocation sequence (p ⬍ 0.001), allocation concealment (p ⫽ 0.003), and double blinding (p ⬍ 0.001) as well as the sample size (p ⬍ 0.001). This association was not explained by the proportion of trials with funding. CONCLUSIONS: External funding was significantly associated with adequate methodological quality, but not with the sample size. Irrespective of funding, the disease area was significantly associated with the methodological quality and sample size. Accordingly, external funding and the disease area are significant predictors of the internal validity of hepatobiliary randomized clinical trials. (Am J Gastroenterol 2002;97:2708 –2713. © 2002 by Am. Coll. of Gastroenterology)
INTRODUCTION Traditionally, clinical decisions have been made on the basis of animal models, expert knowledge, clinical experience, or similar (1). Evidence-based medicine may, therefore, be viewed as a paradigm shift as it involves the use of the best patient-centered research evidence in clinical decision making (2). Based on the estimated risk of bias, a hierarchy of
evidence has been proposed (1). The evidence-based practitioner must seek to identify and use evidence from the highest possible level in this hierarchy. The quality of the evidence should also be graded within each level. The lowest rank has been assigned to uncontrolled clinical observations and physiological experiments based on human or animal models (1). This ranking is based on the lack of reliable comparison groups, unsystematic collection of data, and the necessity for extrapolation. Clinical controlled studies have a higher ranking. Such studies can be randomized or observational. Several studies have found that observational studies have a considerable risk of generating exaggerated estimates of intervention effects compared with randomized clinical trials (3). Observational studies, therefore, have a lower ranking than randomized clinical trials. Randomized clinical trials and systematic reviews have the highest ranking in the hierarchy of clinical intervention research (4, 5). However, such evidence may also be biased (5). Inadequate sample size or inadequate methodological quality can lead to false-negative or false-positive conclusions (5– 8). Compared with large “gold-standard” randomized clinical trials, intervention effects were significantly more positive in small trials without adequate allocation sequence generation, allocation concealment, or double blinding (5). This indicates that small randomized trials with inadequate methodological quality have a considerable risk of bias and therefore of producing exaggerated estimates of intervention effects. On the other hand, the risk of generating false-negative results is also considerable in small trials because of inadequate statistical power. The internal validity of randomized clinical trials, therefore, depends on the methodological quality and sample size. In four major hepatobiliary specialist journals, several hepatobiliary randomized clinical trials were small and had inadequate methodological quality (9 –12). Previous evidence indicates that the extent of these problems varies significantly in different disease areas (11, 12). This variation could reflect financial interests, as external funding has also been associated with adequate methodological quality (11). Funding may also be associated with larger sample sizes. Trials funded by profit organizations more often indicate a significant benefit of innovative interventions than trials funded by nonprofit organizations (13). Inadequate
AJG – November, 2002
methodological quality or greater statistical power of trials funded by for-profit organizations could explain this association (14). In the present study, we have examined whether funding and the disease area are associated with the internal validity of hepatobiliary randomized clinical trials.
MATERIALS AND METHODS The present study includes all hepatobiliary randomized clinical trials that were published during 1985 to 1996 in the following 12 English-language, MEDLINE-indexed journals, which had been handsearched by The Cochrane Collaboration (15): The New England Journal of Medicine, Annals of Internal Medicine, Lancet, Gastroenterology, Gut, Hepatology, Journal of Hepatology, Digestive Diseases and Sciences, Digestion, Liver, Scandinavian Journal of Gastroenterology, and Journal of Clinical Gastroenterology. This study period chosen as 1985 was the first publication year for Journal of Hepatology and as our searches for eligible trials were initiated in 1997. Trials were considered randomized if some form of the word random was used to describe the allocation of patients. Quasirandomized trials and articles referring to subgroups of patients from randomized trials were excluded. The Cochrane Hepatobiliary Group (CHBG) Controlled Trials Register includes all trials identified by the CHBG (15). We, therefore, also searched The CHBG Controlled Trials Register to identify eligible trials for the present study. The CHBG Controlled Trials Register was also searched to estimate the total number of original article hepatobiliary randomized clinical trials published during 1985 to 1996. Data Extraction From each of the included trials, we gathered data on the methodological quality, sample size, disease area (areas with fewer than 10 trials were classified as miscellaneous), and funding. The data extraction was performed unblinded. Funding was classified as for-profit, nonprofit, or no funding. Funding was extracted from the authors’ acknowledgments. The text was scanned for potential sources of funding, in case funding was not reported in the acknowledgments. We defined for-profit organizations as any organization that may incur financial gain or loss depending on the outcome of a trial. Medical research councils, national institutes of health, and any organization that would not have financial gain or loss depending on the outcome of the trial were classified as nonprofit. Some trials were funded by both profit and nonprofit organizations. Based on previous evidence, these trials were grouped with trials funded by nonprofit organizations alone (14). We estimated the interobserver reliability of the classification of funding using 50 arbitrarily selected trials included in the present study. The observers disagreed on the classification of funding in only five of the 50 trials ( ⫽ 0.79), indicating high interobserver agreement.
Internal Validity of Hepatobiliary Trials
2709
According to previous evidence (5– 8), methodological quality was assessed by the generation of the allocation sequence, allocation concealment, and double blinding. The purpose of randomization is to avoid selection bias (5– 8). Adequate randomization does not mean the same as hazard but that each patient has a known (usually equal) chance of receiving the interventions assessed and that the allocation of the next patient cannot be predicted (16). Adequate randomization, therefore, involves both adequate allocation sequence generation and adequate allocation concealment. The allocation sequence should be based on random numbers generated by a table, computer, tossing of a coin, or shuffling of cards. This sequence should be concealed until the moment of randomization by serially numbered opaque sealed envelopes, identically appearing containers, or by a central independent unit. Using these definitions, we classified the generation of the allocation sequence as adequate, unclear (not described), or inadequate (quasirandomized trials, which were excluded). The allocation concealment was classified as adequate, unclear (not described), or inadequate. The purpose of double blinding is to avoid performance and detection bias (8). For a trial to be adequately double blinded, both health care providers/assessors and patients must be kept unaware of the intervention allocated to the individual patient. Double blinding can be adequate if the interventions to be compared are identical with regard to taste, smell, and mode of administration. If an intervention is to be compared with no intervention, double blinding required the use of an inactive placebo. Using these definitions, we classified double blinding as adequate, unclear (not described), inadequate, or not performed. We assessed the interobserver reliability for the methodological quality assessment using 100 arbitrarily selected trials included in the present study. The observers disagreed on the classification of the allocation sequence generation in seven trials ( ⫽ 0.82), allocation concealment in eight trials ( ⫽ 0.83), and double blinding in 11 trials ( ⫽ 0.78). Accordingly, there was a high interobserver agreement in the assessment of methodological quality. The data extraction was based on standardized extraction sheets generated in Lotus Notes (227) version 5.05 (IBMLotus; IBM Corporation, Armonk, New York). According to these sheets, methodological quality was extracted before funding. However, knowledge of funding could influence the assessment of methodological quality as the source of funding was not blinded. To estimate whether this was the case, we compared the methodological quality assessment of two observers with and without blinding of funding. In an arbitrarily selected sample of 50 trials included in the present study, the observers disagreed on the classification of the allocation sequence generation in four trials ( ⫽ 0.73), allocation concealment in four trials ( ⫽ 0.83), and double blinding in five trials ( ⫽ 0.78). These results indicate that knowledge of funding did not bias the assessment of methodological quality significantly.
2710
Kjaergard and Gluud
AJG – Vol. 97, No. 11, 2002
Table 1. Number and Proportion of Included Randomized Clinical Trials in Different Hepatobiliary Disease Areas
Disease Area
Number of Trials
Proportion of Total (%)
Portal hypertension Hepatitis B Biliary tract diseases Hepatitis C Cirrhosis Hepatic encephalopathy Miscellaneous Primary biliary cirrhosis Alcoholic liver diseases Malignant diseases Surgical procedures* Autoimmune liver diseases
150 105 78 68 39 39 35 34 21 21 15 11
24 17 13 11 6 6 6 6 3 3 2 2
* Trials from several disease areas.
Statistical Analysis Categorical data are summarized as proportions and continuous data as medians with 5–95 percentiles. Proportions are compared using 2 and continuous data using Mann-Whitney U test or Kruskal-Wallis test. To assess whether funding can explain discrepancies between disease areas, we performed sensitivity analyses in which trials were stratified according to funding. Significance was accepted at the 0.05 level. The analyses were performed in SPSS version 10.0 for Windows (SPSS Inc, Chicago, Illinois).
RESULTS According to The CHBG Controlled Trials Register, 2134 hepatobiliary randomized clinical trials were published as original articles during 1985 to 1996. This study includes 616 (29%) of these trials (references available on request). The generation of the allocation sequence was adequate in 48% and unclear in 52% of the included trials. The allocation concealment was adequate in 38% and unclear in 62% of the trials. Adequate double blinding was reported in 34%, and 66% were not double blind. The median sample size was 23 patients per intervention arm (percentiles ⫽ 7–102 patients per intervention arm). Forty-seven percent of the trials received no external funding, 30% were funded by nonprofit organizations, and 23% were funded by for-profit organizations. The trials were classified into 12 disease areas (Table 1). Trials on portal hypertension primarily dealt with endoscopic or pharmacological interventions for the prevention and treatment of bleeding esophageal varices. Trials on acute chronic hepatitis B dealt with vaccines or antiviral drugs. Trials on hepatitis C dealt with antiviral drugs. Trials on biliary tract diseases primarily dealt with medical or surgical interventions for cholecystitis or gallbladder stones. The majority of trials on cirrhosis, primary biliary cirrhosis, hepatic encephalopathy, alcoholic liver diseases, fulminant hepatic failure, and liver transplantation assessed medical interventions.
Table 2. Proportion of Randomized Clinical Trials With Adequate Methodological Quality in Different Hepatobiliary Disease Areas Proportion of Trials With Adequate
Disease Area
Allocation Sequence Allocation Double Generation* Concealment† Blinding* (%) (%) (%)
Alcoholic liver diseases Autoimmune liver diseases Biliary tract diseases Cirrhosis Hepatic encephalopathy Hepatitis B Hepatitis C Malignant diseases Miscellaneous Portal hypertension Primary biliary cirrhosis Surgical procedures‡
52 64
43 46
67 55
24 44 59 44 34 62 34 68 44 60
27 18 49 30 35 48 40 50 38 53
47 33 36 22 13 19 31 29 88 40
* p ⬍ 0.001. † p ⫽ 0.003. ‡ Trials from several disease areas.
Trials on malignant liver diseases assessed chemotherapy or transcatheter embolization of hepatocellular carcinoma or metastases. The group of trials that were classified as miscellaneous dealt with primary sclerosing cholangitis, hepatitis A, congenital liver diseases, porphyria, or fatty liver. Funding and Internal Validity Trials funded by nonprofit organizations did not differ significantly from trials funded by for-profit organizations regarding the proportion with adequate generation of the allocation sequence (58% vs 50%, p ⫽ 0.13), adequate allocation concealment (37% vs 36%, p ⫽ 0.83), adequate double blinding (38% vs 46%, p ⫽ 0.11), or sample size (median number of patients per intervention arm ⫽ 23 [percentiles ⫽ 7–140] vs 23 [percentiles ⫽ 6 –118], p ⫽ 0.25). In the analyses to follow, we compared all trials with external funding (i.e., trials funded by nonprofit and forprofit organizations) with trials without funding. Compared with funded trials, trials without funding were significantly less likely to report adequate generation of the allocation sequence (55% vs 41%, p ⫽ 0.001) and to be double blind (42% vs 25%, p ⬍ 0.001). There were no significant differences between funded trials and trials without funding regarding the proportion with adequate allocation concealment (37% vs 40%, p ⫽ 0.40) or sample size (median number of patients per intervention arm 23 [percentiles ⫽ 7–132] vs 24 [percentiles ⫽ 7–90], p ⫽ 0.51). Internal Validity and Funding in Different Disease Areas As shown in Table 2, the 12 disease areas differed significantly regarding the proportion of trials with adequate generation of the allocation sequence (p ⬍ 0.001), allocation concealment (p ⫽ 0.003), and double blinding (p ⬍ 0.001).
AJG – November, 2002
Internal Validity of Hepatobiliary Trials
Table 3. Sample Size (Number of Patients Per Intervention Arm) of Randomized Clinical Trials in Different Hepatobiliary Disease Areas Disease Area
Median*
(5–95 Percentiles)
Alcoholic liver diseases Portal hypertension Primary biliary cirrhosis Malignant diseases Hepatitis C Surgical procedures† Biliary tract diseases Hepatitis B Cirrhosis Hepatic encephalopathy Miscellaneous Autoimmune liver diseases
36 34 29 28 27 26 20 18 16 15 15 12
(6–149) (9–88) (6–112) (9–132) (9–122) (10–265) (5–90) (6–96) (6–58) (6–33) (3–510) (6–41)
* p ⬍ 0.001. † Trials from several disease areas.
The median number of patients per intervention arm in different disease areas also differed significantly (p ⬍ 0.001, Table 3). The proportion of trials with funding was not significantly different in different disease areas (p ⫽ 0.11). According to the sensitivity analyses including funded trials only, the disease areas differed significantly regarding the proportion of trials with adequate generation of the allocation sequence (p ⬍ 0.001), allocation concealment (p ⫽ 0.004), double blinding (p ⬍ 0.001), and sample size (p ⬍ 0.001). According to the sensitivity analyses including only trials without funding, the disease areas differed significantly regarding the proportion of trials with adequate generation of the allocation sequence (p ⬍ 0.001), double blinding (p ⬍ 0.001), and sample size (p ⬍ 0.001), but not the proportion of trials with adequate allocation concealment (p ⫽ 0.36).
DISCUSSION This study includes almost one third of all hepatobiliary randomized trials published as original articles during an 11-yr period. Several of these trials had a considerable risk of producing false-positive or false-negative results because of inadequate methodological quality (5– 8) or sample size (5, 17, 18). The internal validity (methodological quality and sample size) of trials funded by nonprofit organizations and/or for-profit organizations was not significantly different. All trials with funding were significantly more likely to report adequate generation of the allocation sequence and to be double blind than trials without funding. Trials with funding did not differ significantly from trials without funding with regard to the adequacy of the allocation concealment or sample size. The proportion of funded trials did not differ significantly between disease areas, but the internal validity did. The variations in internal validity in different disease areas were not explained by funding. We included a heterogeneous group of trials from several hepatobiliary disease areas. The variety of the included trials
2711
increases the external validity but excludes the possibility of analyzing the direct association between methodological quality and intervention effects in the included trials. This association, therefore, has to be inferred from previous evidence (5– 8). Three empirical studies have assessed the association between methodological quality and intervention effects in randomized clinical trials included in metaanalyses from several different disease areas including cardiology, gynecology, obstetrics, psychiatry, and hematology. Methodological quality, i.e., the confidence that a trial has adequate bias control, can be assessed by the risk of selection bias, performance bias, and detection bias. Such bias may be prevented through adequate randomization and double blinding. On average, the estimated intervention effects generated by small trials with adequate randomization or double blinding and large “gold-standard” trials were not significantly different (5). Conversely, small trials without adequate randomization or double blinding exaggerated intervention effects significantly by about 50% (5). There is no similar evidence documenting the association between methodological quality and intervention effects in hepatobiliary randomized clinical trials. However, we cannot find any reason why such trials should be immune to bias. Furthermore, systematic reviews point to a similar tendency of exaggerated estimates of intervention effects in low-quality hepatobiliary randomized clinical trials (19, 20). In accordance with previous studies on the methodological quality of randomized clinical trials, we based our quality assessments on the published reports. We were, therefore, not able to distinguish between what was reported and how the trials were designed and conducted. In theory, inadequate (i.e., unclear) reporting may conceal methodological flaws. It may, therefore, be considered a weakness that we did not write to authors for additional information. On the other hand, previous evidence (5– 8) indicates that inadequate reporting reflects inadequate trial design and conduct. Furthermore, even when including only trials published during the 1990s, only about 20% of authors reply to requests for additional information about their trials (21, 22). Accordingly, although methodological quality and funding may be inconsistently reported, correspondence with authors would provide only little additional information. Several methods for assessment of quality have been suggested, but few have been validated (23, 24). It has been shown that the conclusions of different quality scales can be equivocal (24). We, therefore, limited the quality assessment to components, which have been supported by empirical evidence (5– 8). However, the term quality can refer to several aspects. For example, the clinical value of a trial also depends on the external validity, and the quality of a trial, therefore, also depends on the reporting of patient characteristics, the setting, the interventions, and the outcomes. However, the question of the external validity becomes irrelevant if a trial lacks internal validity. We assessed the methodological quality of trials by the
2712
Kjaergard and Gluud
reported randomization methods and double blinding. It may be argued that double blinding cannot be used as a measure of quality in larger samples of trials as the nature of some interventions precludes adequate double blinding of caretakers and patients. Single blinding (e.g., of outcome assessors or data analyses) is always possible, but is rarely reported and has not been associated with intervention effects. We did not perform analyses that were adjusted for the feasibility of blinding. We did, however, compare the proportion of double-blind trials in different disease areas and found a significant variation. In some disease areas, e.g., hepatitis C, all known interventions are drugs and, therefore, possible to assess in a double-blind manner. In other areas, e.g., treatment of biliary tract diseases, surgical interventions are more common. This may influence the proportion of double-blind trials in different disease areas and could explain why we found a significant association between the disease area and the proportion of double-blind trials. Likewise, if funded trials more often dealt with drugs than trials without funding, this could influence the association between funding and double blinding. On the other hand, adequate randomization is always possible, irrespective of the intervention assessed. We found a significant association between funding, disease area, and the adequacy of the reported randomization methods. This supports the assumption that the methodological quality is associated with the disease area and funding, irrespective of the type of intervention assessed. Two previous studies indicate that trials funded by forprofit organizations are significantly more likely to indicate a significant benefit of experimental interventions compared with trials funded by nonprofit organizations (13, 14). This discrepancy could reflect variations in methodological quality or statistical power. We found that the methodological quality and sample size of trials funded by for-profit organizations was not significantly different from nonprofit organizations. Accordingly, the present study supports previous findings indicating that the association between funding and authors’ conclusions in randomized clinical trials does not reflect variations in the internal validity (14). This study confirms that several hepatobiliary trials are small and of inadequate methodological quality (9 –12). As the results are based on a large proportion of the total number of published trials, it is likely that the problem is general. The present study also supports previous findings indicating that there is a positive association between funding and methodological quality (11) and shows that this association does not explain the significant variation of internal validity in different disease areas. The internal validity seemed to be relatively high in rare diseases. As trials on rare diseases often have to be performed as multicenter trials, this finding could indicate that cooperation of researchers may not only increase the possibility of including more patients, but also increase the methodological quality (11). There was no apparent association between the prev-
AJG – Vol. 97, No. 11, 2002
alence of the diseases and the average sample size of the trials. The present study demonstrates a significant positive association between external funding and adequate methodological quality. This association may reflect better resources. Furthermore, in most cases, obtainment of funding usually requires the preparation of a thorough protocol, which is reviewed by persons not involved in the trial. The association between funding and adequate quality could, therefore, also be caused by the positive effect generated by a critical mass. The reasons for the significant association between the disease area and internal validity are less obvious. Possibly, this association could reflect initiatives, such as methodological workshops (25, 26). Despite the considerable variation, the present study demonstrates several weaknesses within hepatobiliary intervention research. Low-quality evidence may lead to the implementation of useless or even harmful interventions, and the results, therefore, suggest that several hepatobiliary health care interventions need to be reexamined (27). We suggest that more attention is being paid to the requirements for adequate statistical power and methodological quality, e.g., following the recommendations given in the Consolidated Standard of Reporting Trials (CONSORT) statement (www.consortstatement.org).
ACKNOWLEDGMENTS This work was supported by grants from The Danish Medical Research Council and The 1991 Pharmacy Foundation in Denmark. We thank M. Kjaer, D. Nikolova, S. Frederiksen, and N. Salasshahri for their assistance with the identification and retrieval of trials. Reprint requests and correspondence: Lise Lotte Kjaergard, M.D., The Cochrane Hepatobiliary Group, The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Department 7102, H:S Rigshospitalet, DK-2100, Copenhagen, Denmark. Received Mar. 12, 2002; accepted June 24, 2002.
REFERENCES 1. Guyatt GH, Sackett DL, Cook DJ. Users’ guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? EvidenceBased Medicine Working Group. JAMA 1993;270:2598 – 601. 2. Sackett D, Rosenberg WM, Gray JA, et al. Evidence based medicine: What it is and what it isn’t. BMJ 1996;312:71–2. 3. Kunz R, Oxman A. The unpredictability paradox: Review of empirical comparisons of randomised and non-randomised clinical trials. BMJ 1998;317:1185–90. 4. Peto R, Baigent C. Trials: The next 50 years. Large scale randomised evidence of moderate benefits. BMJ 1998;317: 1170 –1. 5. Kjaergard LL, Villumsen J, Gluud C. Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med 2001;135:982–9. 6. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality asso-
AJG – November, 2002
7. 8. 9. 10. 11. 12. 13. 14.
15. 16. 17.
18.
ciated with estimates of treatment effects in controlled trials. JAMA 1995;273:408 –12. Moher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609 –13. Ju¨ ni P, Altman D, Egger M. Assessing the quality of controlled clinical trials. BMJ 2001;323:42– 6. Gluud C, Nikolova D. Quality assessment of reports on clinical trials in the Journal of Hepatology. J Hepatol 1998;29: 321–7. Gluud C. Evidence based medicine in Liver. Liver 1999;19: 1–2. Kjaergard LL, Nikolova D, Gluud C. Randomized clinical trials in Hepatology: Predictors of quality. Hepatology 1999; 30:1134 – 8. Kjaergard LL, Frederiksen SL, Gluud C. Validity of randomized clinical trials in Gastroenterology from 1964 –2000. Gastroenterology 2002;122:1157– 60. Djulbegovic B, Lacevic M, Cantor A, et al. The uncertainty principle and industry-sponsored research. Lancet 2000;356: 635– 8. Kjaergard LL, Als-Nielsen B. The association between competing interests and authors’ conclusions: Epidemiological study of randomised clinical trials published in the BMJ. BMJ 2002;325:249 –52. Clarke M, Oxman AD, eds. Cochrane reviewers’ handbook 4.1.5. In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software. Altman DG. Randomisation. BMJ 1991;302:1481–2. Collins R, Gray R, Godwin J, Peto R. Avoidance of large biases and large random errors in the assessment of moderate treatment effects: The need for systematic overviews. Stat Med 1987;6:245–54. Peto R, Collins R, Gray R. Large-scale randomized evidence: Large, simple trials and overviews of trials. Ann N Y Acad Sci 1993;703:314 – 40.
Internal Validity of Hepatobiliary Trials
2713
19. Rambaldi A, Gluud C. Colchicine for alcoholic and nonalcoholic liver fibrosis and cirrhosis (Cochrane review). In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software. 20. D’Amico G, Pietrosi G, Tarantino I, Pagliaro L. Emergency sclerotherapy versus medical interventions for bleeding oesophageal varices in cirrhotic patients (Cochrane review). In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software. 21. Kjaergard LL, Krogsgaard K, Gluud C. Interferon alfa with or without ribavirin for chronic hepatitis C: Systematic review of randomised trials. BMJ 2001;323:1151–5. 22. Kjaergard LL, Krogsgaard K, Gluud C. Ribavirin with or without alpha interferon versus no intervention, placebo or alpha interferon for chronic hepatitis C (Cochrane review). In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software. 23. Moher D, Jadad AR, Nichol G, et al. Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Control Clin Trials 1995;16:62–73. 24. Ju¨ ni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA 1999; 282:1054 – 60. 25. Burroughs AK. Methodology and reviews of clinical trials in portal hypertension. Amsterdam: Excerpta Medica, 1987. 26. De Franchis R. What have we accomplished? In: De Franchis R, ed. Portal hypertension III. Proceedings of the Third Baveno International Consensus Workshop on Definitions, Methodology and Therapeutic Strategies. Oxford: Blackwell Science, 2002:1–12. 27. Gluud C, on behalf of The Cochrane Hepatobiliary Group Editorial Team. The Cochrane Hepatobiliary Group. In: Bircher J, Benhamou JP, McIntyre N, Rizzetto M, Rodes J, eds. Oxford textbook of clinical hepatology. Oxford: Oxford University Press, 1999:2145– 8.