- Email: [email protected]
Citation bias of hepato-biliary randomized clinical trials
Journal of Clinical Epidemiology 55 (2002) 407–410
Citation bias of hepato-biliary randomized clinical trials Lise L. Kjaergard*, Christian Gluud The Cochrane Hepato-Biliary Group, The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Copenhagen, Denmark Received 22 May 2001; received in revised form 12 November 2001; accepted 15 November 2001
Abstract The objective of this study was to assess whether trials with a positive (i.e., statistically significant) outcome are cited more often than negative trials. We reviewed 530 randomized clinical trials on hepato-biliary diseases published in 11 English-language journals indexed in MEDLINE from 1985–1996. From each trial, we extracted the statistical significance of the primary study outcome (positive or negative), the disease area, and methodological quality (randomization and double blinding). The number of citations during two calendar years after publication was obtained from Science Citation Index. There was a significant positive association between a statistically significant study outcome and the citation frequency (, 0.55, 95% confidence interval, 0.39–0.72). The disease area and adequate generation of the allocation sequence were also significant predictors of the citation frequency. We concluded that positive trials are cited significantly more often than negative trials. The association was not explained by disease area or methodological quality. © 2002 Elsevier Science Inc. All rights reserved. Keywords: Randomized clinical trials; Bias; Systematic reviews; Meta-analysis; Journal citation reports; Science citation index
1. Introduction Metaanalyses are increasingly being recognized as a reliable tool for summarizing research evidence [1], but the results can be misleading if based on a biased sample of the available evidence [2]. For example, metaanalyses will provide an exaggerated estimate of the effect of an intervention if omitting negative trials (i.e., trials showing no significant effect). There are several ways by which such bias can be introduced [3]. First, positive studies (i.e., studies with statistically significant results) are more likely to be published and have a significantly shorter time to publication than negative trials [4,5]. Second, among published studies, those with positive results are more likely to be published in English [6], to be published in journals indexed in electronic databases [3], and to be published several times [4]. Bias in the identification and inclusion of trials for metaanalyses may also be affected by the number of times they are cited [7]. However, the evidence concerning the extent and direction of citation bias is equivocal. Two studies found that positive trials (i.e., trials showing a significant intervention benefit) are cited more often than negative trials [8–10]. However, two other studies showed the direct oppo-
* Corresponding author. The Cochrane Hepato-Biliary Group, The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Blegdamsvej 9, H:S Rigshospitalet, DK-2100, Copenhagen, Denmark. Tel: 45-35457165; fax: 45-3545-7101. E-mail address: [email protected] (L.L. Kjaergard).
site (i.e., that negative trials are cited more often than positive trials) [11,12]. The discrepancy between these findings may reflect different citation habits in different areas [13], or confounding (e.g., language bias [6], database bias [3], or variations in the methodological quality [14–16]). In the present study, we assessed the extent and direction of citation bias among trials on several hepato-biliary diseases published in English-language journals indexed in MEDLINE. 2. Materials and methods 2.1. Selection of studies We included all randomized trials from the following arbitrarily selected general and specialist English-language journals that were indexed in MEDLINE and were likely to publish hepato-biliary randomized clinical trials: The New England Journal of Medicine, Annals of Internal Medicine, Lancet, Gastroenterology, Hepatology, Journal of Hepatology, Digestive Diseases and Sciences, Digestion, Liver, Scandinavian Journal of Gastroenterology, and Journal of Clinical Gastroenterology. Trials were included in the present study if they were published as an original article during 1985– 1996, dealt with any hepato-biliary disease, and used some form of the word “random” to describe the allocation of patients. Quasi-randomized trials, trials published as abstracts, and articles referring to subgroups of patients from randomized trials were excluded. Eligible trials were identified through manual searches [17–19] and electronic searches of
0895-4356/02/$ – see front matter © 2002 Elsevier Science Inc. All rights reserved. PII: S0895-4356(01)00 5 1 3 - 3
408
L.L. Kjaergard and C. Gluud / Journal of Clinical Epidemiology 55 (2002) 407–410
The Cochrane Hepato-Biliary Group Controlled Trials Register [20], The Cochrane Library [20], and MEDLINE.
3. Results
2.2. Data extraction
In total, 530 trials (available on request) fulfilled the inclusion and none of the exclusion criteria. According to the Science Citation Index, the trials received a total of 2,327 citations during the first two calendar years after the year of publication. The median citation frequency of all trials was 2 (interquartile range, 0–6; range, 0–50). One hundred sixty trials (30%) were not cited. The study outcome was classified as positive in 374 trials (71%). The trials covered 12 disease areas (alcoholic liver diseases, cirrhosis, fulminant hepatic failure, gall bladder stones, hepatic encephalopathy, hepatitis B, hepatitis C, liver transplantation, malignant diseases, portal hypertension, primary biliary cirrhosis, and miscellaneous). Adequate generation of the allocation sequence was reported in 273 trials (52%) and adequate allocation concealment in 178 trials (34%). The remaining trials had unclear generation of the allocation sequence and unclear allocation concealment. One hundred ninety-four trials (37%) were adequately double blinded. The remaining trials were not double blind.
The number of citations to each trial during the first two calendar years following the year of publication was obtained though the Science Citation Index. The included trials were classified as positive (2P 0.05) or negative according to the statistical significance of the primary outcome measure. The outcome measure mentioned first in the abstract was used when a primary outcome was not specified. The disease area (areas covered by fewer than 10 trials were classified as miscellaneous) and the reported methodological quality of the included trials were extracted and registered. According to previous evidence [14–16], the reported methodological quality was classified by the following components: Generation of the allocation sequence: (I) adequate (e.g., table of random numbers or similar) or (II) inadequate/unclear (i.e., quasi randomized trials [excluded from the present study] and trials in which the generation of the allocation sequence was not described). Allocation concealment: (I) adequate (e.g., central independent unit or sealed envelopes) or (II) inadequate/unclear (e.g., open table of random numbers and trials in which the allocation concealment was not described). Double-blinding: (I) adequate (e.g., identical placebo tablets) or (II) inadequate/not performed/unclear (e.g., trials comparing tablets versus injections, trials without double blinding, and trials in which the method of blinding was not described). The data extraction was performed unblinded by LLK and verified by CG. We have previously examined the reliability of the quality assessment and found high interobserver agreement [19] and no significant difference between blinded and unblinded assessments [16]. 2.3 Data analyses The primary outcome measure of the present study was the number of citations during the first two calendar years following the year of publication. Continuous variables were summarized by the median and interquartile range. The association between the statistical significance of the outcome of the included trials (positive or negative) and the citation frequency was analyzed by linear regression without and with adjustments for the disease area and methodological quality. The citation frequency was log-transformed to approximate a normal distribution. The outcome of the included trials and the methodological quality (generation of the allocation sequence, allocation concealment, and double blinding) were entered as categorical variables. The disease area was transformed to a dummy variable using the group of trials classified as miscellaneous as reference. The results of the linear regression analyses were summarize by the determinants () and 95% confidence intervals (CI). The analyses were performed in SPSS version 10.0 for Windows.
3.1. Description of trials
3.2. Association between the outcome of the included trials and the citation frequency As shown in Fig. 1, the median citation frequency of positive trials was 2 (interquartile range, 0–7), and the median citation frequency of negative trials was 1 (interquartile range, 1–5). According to the unadjusted analyses, there was a significant positive association between a statistically significant study outcome and the citation frequency (, 1.21; 95% CI, 1.10–1.33). The disease area was also a significant predictor of the citation frequency (, 0.15; 95% CI, 0.14–0.16). Adequate allocation concealment was not significantly associated with the citation frequency (, 0.04; 95% CI, 0.34–0.26), but there was a significant positive association between adequate generation of the allocation sequence and the citation frequency (, 1.05; 95% CI, 0.81–1.30). Accordingly, trials with adequate generation of the allocation sequence were cited significantly more often than trials with unclear generation of the allocation sequence. There was also a significant positive association between double blinding and the citation frequency (, 0.60; 95% CI, 0.40–0.79), indicating that doubleblind trials were cited significantly more often than trials without double blinding. According to the adjusted analyses, a statistically significant study outcome was the strongest independent predictor of the citation frequency (, 0.55; 95% CI, 0.39–0.72). The disease area and adequate generation of the allocation sequence were also significant predictors of the citation frequency (, 0.07; 95% CI, 0.05–0.10 and , 0.43; 95% CI, 0.19–0.67, respectively). Neither allocation concealment nor double blinding were significantly associated with the citation frequency (, 0.05; 95% CI, 0.32–0.21 and , 0.24; 95% CI, 0.22–0.17, respectively).
L.L. Kjaergard and C. Gluud / Journal of Clinical Epidemiology 55 (2002) 407–410
Fig. 1. Summary plot of the citation frequency of trials with a positive or a negative study outcome. The line across the box indicates the median, the box represents the interquartile range, and the whiskers represent the highest and lowest values, excluding outliers () and extremes (*). Outliers represent cases with values between 1.5 and 3 times the interquartile range. Extremes represent cases with values of more than 3 times the interquartile range.
4. Discussion In the present study, we found significant citation bias of hepato-biliary randomized clinical trials. Trials with a positive outcome were cited significantly more often than trials with a negative outcome. The disease area and the generation of the allocation sequence were significant independent predictors of the citation frequency but did not explain the association between the study outcome and citation frequency. The present study includes trials from several disease areas within one specialty. This may be viewed as the strength and weakness of the study. On one hand, including a more homogenous sample of trials would have provided a more focused answer. On the other hand, the variety of the included trials increases the external validity of the results, which indicate that citation bias may be a general problem. We included trials published in English-language journals indexed in MEDLINE to avoid language bias and database bias [3]. This approach may, however, also affect the external validity of the findings. Whether there is citation bias among trials published in non-English language journals not indexed in MEDLINE remains to be established. It is possible that confounders not included in the present study affect the findings. First, the accessibility and popularity of a journal may be associated with the citation rates. However, Ravnskov [9] found that the place of publication
409
had no significant effect on citation rates. Second, citation rates may be affected by the sample size of the trials. However, statistical significance may be considered an intermediate variable in the assessment of the association between sample size and citation counts, and, accordingly, we did not adjust for this potential confounder. Furthermore, it may be more important to establish whether there is a bias toward preferential citation of positive randomized clinical trials than to establish why such bias exist. Due to the design of the present study, the results cannot directly show the effect of citation bias on the results of metaanalyses. However, the effect of citation bias on the conclusions of systematic reviews may be inferred from the results. In theory, preferential citing of positive trials may lead to the implementation of ineffective or even harmful interventions. The results therefore support recommendations that the identification of randomized clinical trials for systematic reviews should be based on thorough electronic and manual searches [21]. Trials that are not indexed in an electronic database have to be identified through manual searches. Citation bias can be an important problem among such trials. Several initiatives are being performed to improve the registration of trials including the ongoing handsearches within The Cochrane Collaboration [21] and campaigns for searchable archives of ongoing and completed randomized clinical trials [22–24]. The magnitude of the problem of citation bias among trials available through manual searches only may therefore be decreasing. However, registration of randomized clinical trials in electronic databases does not eliminate the problem of citation bias. Several trials may be missed through electronic searches due to inadequate indexing in the electronic databases and/or inadequate search strategies. We therefore performed manual and electronic searches to ensure the completeness of the sample. The results indicate that citation bias can be an important problem among trials registered in electronic databases and thereby stresses the importance of comprehensive electronic search strategies. The present study shows that among randomized clinical trials on several hepato-biliary disease areas, there was a significant positive association between a statistically significant study outcome and citation frequency. This confirms two previous studies showing that nonsteroid antiinflammatory drug trials and trials on lowering cholesterol levels were cited significantly more often if the result was statistically significant than if the result was nonsignificant [8–10]. Two other studies have found the opposite trend among trials in the judgment and decision literature and on pneumococcal vaccines [11,12]. The inconsistency may reflect different citation habits in different areas (e.g., depending on financial interests and prior beliefs). The present study included a relatively large sample of trials compared with previous studies and, unlike previous studies, covered several disease areas. This may increase the external validity of the conclusions suggesting that citation bias is a general problem.
410
L.L. Kjaergard and C. Gluud / Journal of Clinical Epidemiology 55 (2002) 407–410
Researchers should above all cite good-quality research. We have previously found that trials with inadequate or unclear randomization and trials without double blinding exaggerate intervention effects significantly compared with large, gold-standard, randomized clinical trials [16]. We found a positive association between adequate generation of the allocation sequence and citation frequency. To some extent, the results oppose the results of a survey showing that in general researchers do not consider the quality of the evidence when deciding which evidence to cite [7]. Even retracted articles continue to be cited as valid work after publication of the retraction [25]. Ravnskov concluded that “fundamental parts of the dietheart idea are based on biased citations” [10]. Our results confirm that the search for eligible trials for inclusion in systematic reviews should be as thorough as possible. We suggest that manual searches should not only include bibliographies but also relevant journals and conference proceedings. Furthermore, communication with pharmaceutical companies and other researchers within the field may also provide important information about additional trials not identified through manual or electronic searches. Above all, extensive efforts are needed to ensure that the conclusions of a meta-analysis are not based on a biased sample of trials. Acknowledgments We thank Peter C. Gøtzsche, Thorkild I.A. Sørensen, and two peer referees at Journal of Clinical Epidemiology for valuable comments to an earlier draft of this manuscript. We also thank Ninna Frydendal, Dimitrinka Nikolova, Sarah L. Frederiksen, and Nader Salasshahri for their help with the identification and retrieval of trials. This work was supported by grants from The Danish Medical Research Council and The 1991 Pharmacy Foundation in Denmark. References [1] Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med 1997;127:37–42. [2] Gluud C. “Negative trials” are positive! J Hepatol 1998;28:731–3.
[3] Egger M, Smith GD. Bias in location and selection of studies. BMJ 1998;316:61–6. [4] Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867–72. [5] Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640–5. [6] Egger M, Zellweger-Zahner T, Schneider M, Juncker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997;350:326–9. [7] Song F, Eastwood S, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess 2000;4:1–115. [8] Gøtzsche PC. Reference bias in reports of drug trials. BMJ 1987;195: 654–56. [9] Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 1992;305:15–9. [10] Ravnskov U. Quotation bias in reviews of the diet-heart idea. J Clin Epidemiol 1995;48:713–9. [11] Christensen-Szalanski JJJ, Beach LR. The citation bias: fad and fashion in the judgement and descision literature. J Am Psychol 1984;39:75–8. [12] Hutchison BG, Lloyd S. Comprehensiveness and bias in reporting clinical trials: study of reviews of pneumococcal vaccine effectiveness. Can Fam Physician 1995;41:1356–60. [13] Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:498–502. [14] Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408–12. [15] Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609–13. [16] Kjaergard LL, Villumsen J, Gluud C. Reported methodological quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med December 2001;135:982–9. [17] Gluud C, Nikolova D. Quality assessment of reports on clinical trials in the Journal of Hepatology. J Hepatol 1998;29:321–7. [18] Gluud C. Evidence based medicine in Liver. Liver 1999;19:1–2. [19] Kjaergard LL, Nikolova D, Gluud C. Randomized clinical trials in Hepatology: predictors of quality. Hepatol 1999;30:1134–8. [20] The Cochrane Library. Issue 2, 2001. Oxford: Update Software. [21] Clarke M, Oxman A, Ed. Cochrane reviewers’ handbook 4.1.4. In: The Cochrane Library, Issue 4, 2001; Oxford: Update Software. [22] Roberts I. Pharmaceutical industry is invited to respond to amnesty for unreported trials. BMJ 1997;315:683. [23] Blunt J. Research ethics committees must ensure registration of research and accessibility of results. BMJ 1998;316:312. [24] Tonks A. Registering clinical trials. BMJ 1999;319:1565–8. [25] Budd JM, Sievert M, Schultz TR. Phenomena of retraction: reasons for retraction and citations to the publications. JAMA 2001;280:296–7.
Citation bias of hepato-biliary randomized clinical trials Lise L. Kjaergard*, Christian Gluud The Cochrane Hepato-Biliary Group, The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Copenhagen University Hospital, Copenhagen, Denmark Received 22 May 2001; received in revised form 12 November 2001; accepted 15 November 2001
Abstract The objective of this study was to assess whether trials with a positive (i.e., statistically significant) outcome are cited more often than negative trials. We reviewed 530 randomized clinical trials on hepato-biliary diseases published in 11 English-language journals indexed in MEDLINE from 1985–1996. From each trial, we extracted the statistical significance of the primary study outcome (positive or negative), the disease area, and methodological quality (randomization and double blinding). The number of citations during two calendar years after publication was obtained from Science Citation Index. There was a significant positive association between a statistically significant study outcome and the citation frequency (, 0.55, 95% confidence interval, 0.39–0.72). The disease area and adequate generation of the allocation sequence were also significant predictors of the citation frequency. We concluded that positive trials are cited significantly more often than negative trials. The association was not explained by disease area or methodological quality. © 2002 Elsevier Science Inc. All rights reserved. Keywords: Randomized clinical trials; Bias; Systematic reviews; Meta-analysis; Journal citation reports; Science citation index
1. Introduction Metaanalyses are increasingly being recognized as a reliable tool for summarizing research evidence [1], but the results can be misleading if based on a biased sample of the available evidence [2]. For example, metaanalyses will provide an exaggerated estimate of the effect of an intervention if omitting negative trials (i.e., trials showing no significant effect). There are several ways by which such bias can be introduced [3]. First, positive studies (i.e., studies with statistically significant results) are more likely to be published and have a significantly shorter time to publication than negative trials [4,5]. Second, among published studies, those with positive results are more likely to be published in English [6], to be published in journals indexed in electronic databases [3], and to be published several times [4]. Bias in the identification and inclusion of trials for metaanalyses may also be affected by the number of times they are cited [7]. However, the evidence concerning the extent and direction of citation bias is equivocal. Two studies found that positive trials (i.e., trials showing a significant intervention benefit) are cited more often than negative trials [8–10]. However, two other studies showed the direct oppo-
* Corresponding author. The Cochrane Hepato-Biliary Group, The Copenhagen Trial Unit, Centre for Clinical Intervention Research, Blegdamsvej 9, H:S Rigshospitalet, DK-2100, Copenhagen, Denmark. Tel: 45-35457165; fax: 45-3545-7101. E-mail address: [email protected] (L.L. Kjaergard).
site (i.e., that negative trials are cited more often than positive trials) [11,12]. The discrepancy between these findings may reflect different citation habits in different areas [13], or confounding (e.g., language bias [6], database bias [3], or variations in the methodological quality [14–16]). In the present study, we assessed the extent and direction of citation bias among trials on several hepato-biliary diseases published in English-language journals indexed in MEDLINE. 2. Materials and methods 2.1. Selection of studies We included all randomized trials from the following arbitrarily selected general and specialist English-language journals that were indexed in MEDLINE and were likely to publish hepato-biliary randomized clinical trials: The New England Journal of Medicine, Annals of Internal Medicine, Lancet, Gastroenterology, Hepatology, Journal of Hepatology, Digestive Diseases and Sciences, Digestion, Liver, Scandinavian Journal of Gastroenterology, and Journal of Clinical Gastroenterology. Trials were included in the present study if they were published as an original article during 1985– 1996, dealt with any hepato-biliary disease, and used some form of the word “random” to describe the allocation of patients. Quasi-randomized trials, trials published as abstracts, and articles referring to subgroups of patients from randomized trials were excluded. Eligible trials were identified through manual searches [17–19] and electronic searches of
0895-4356/02/$ – see front matter © 2002 Elsevier Science Inc. All rights reserved. PII: S0895-4356(01)00 5 1 3 - 3
408
L.L. Kjaergard and C. Gluud / Journal of Clinical Epidemiology 55 (2002) 407–410
The Cochrane Hepato-Biliary Group Controlled Trials Register [20], The Cochrane Library [20], and MEDLINE.
3. Results
2.2. Data extraction
In total, 530 trials (available on request) fulfilled the inclusion and none of the exclusion criteria. According to the Science Citation Index, the trials received a total of 2,327 citations during the first two calendar years after the year of publication. The median citation frequency of all trials was 2 (interquartile range, 0–6; range, 0–50). One hundred sixty trials (30%) were not cited. The study outcome was classified as positive in 374 trials (71%). The trials covered 12 disease areas (alcoholic liver diseases, cirrhosis, fulminant hepatic failure, gall bladder stones, hepatic encephalopathy, hepatitis B, hepatitis C, liver transplantation, malignant diseases, portal hypertension, primary biliary cirrhosis, and miscellaneous). Adequate generation of the allocation sequence was reported in 273 trials (52%) and adequate allocation concealment in 178 trials (34%). The remaining trials had unclear generation of the allocation sequence and unclear allocation concealment. One hundred ninety-four trials (37%) were adequately double blinded. The remaining trials were not double blind.
The number of citations to each trial during the first two calendar years following the year of publication was obtained though the Science Citation Index. The included trials were classified as positive (2P 0.05) or negative according to the statistical significance of the primary outcome measure. The outcome measure mentioned first in the abstract was used when a primary outcome was not specified. The disease area (areas covered by fewer than 10 trials were classified as miscellaneous) and the reported methodological quality of the included trials were extracted and registered. According to previous evidence [14–16], the reported methodological quality was classified by the following components: Generation of the allocation sequence: (I) adequate (e.g., table of random numbers or similar) or (II) inadequate/unclear (i.e., quasi randomized trials [excluded from the present study] and trials in which the generation of the allocation sequence was not described). Allocation concealment: (I) adequate (e.g., central independent unit or sealed envelopes) or (II) inadequate/unclear (e.g., open table of random numbers and trials in which the allocation concealment was not described). Double-blinding: (I) adequate (e.g., identical placebo tablets) or (II) inadequate/not performed/unclear (e.g., trials comparing tablets versus injections, trials without double blinding, and trials in which the method of blinding was not described). The data extraction was performed unblinded by LLK and verified by CG. We have previously examined the reliability of the quality assessment and found high interobserver agreement [19] and no significant difference between blinded and unblinded assessments [16]. 2.3 Data analyses The primary outcome measure of the present study was the number of citations during the first two calendar years following the year of publication. Continuous variables were summarized by the median and interquartile range. The association between the statistical significance of the outcome of the included trials (positive or negative) and the citation frequency was analyzed by linear regression without and with adjustments for the disease area and methodological quality. The citation frequency was log-transformed to approximate a normal distribution. The outcome of the included trials and the methodological quality (generation of the allocation sequence, allocation concealment, and double blinding) were entered as categorical variables. The disease area was transformed to a dummy variable using the group of trials classified as miscellaneous as reference. The results of the linear regression analyses were summarize by the determinants () and 95% confidence intervals (CI). The analyses were performed in SPSS version 10.0 for Windows.
3.1. Description of trials
3.2. Association between the outcome of the included trials and the citation frequency As shown in Fig. 1, the median citation frequency of positive trials was 2 (interquartile range, 0–7), and the median citation frequency of negative trials was 1 (interquartile range, 1–5). According to the unadjusted analyses, there was a significant positive association between a statistically significant study outcome and the citation frequency (, 1.21; 95% CI, 1.10–1.33). The disease area was also a significant predictor of the citation frequency (, 0.15; 95% CI, 0.14–0.16). Adequate allocation concealment was not significantly associated with the citation frequency (, 0.04; 95% CI, 0.34–0.26), but there was a significant positive association between adequate generation of the allocation sequence and the citation frequency (, 1.05; 95% CI, 0.81–1.30). Accordingly, trials with adequate generation of the allocation sequence were cited significantly more often than trials with unclear generation of the allocation sequence. There was also a significant positive association between double blinding and the citation frequency (, 0.60; 95% CI, 0.40–0.79), indicating that doubleblind trials were cited significantly more often than trials without double blinding. According to the adjusted analyses, a statistically significant study outcome was the strongest independent predictor of the citation frequency (, 0.55; 95% CI, 0.39–0.72). The disease area and adequate generation of the allocation sequence were also significant predictors of the citation frequency (, 0.07; 95% CI, 0.05–0.10 and , 0.43; 95% CI, 0.19–0.67, respectively). Neither allocation concealment nor double blinding were significantly associated with the citation frequency (, 0.05; 95% CI, 0.32–0.21 and , 0.24; 95% CI, 0.22–0.17, respectively).
L.L. Kjaergard and C. Gluud / Journal of Clinical Epidemiology 55 (2002) 407–410
Fig. 1. Summary plot of the citation frequency of trials with a positive or a negative study outcome. The line across the box indicates the median, the box represents the interquartile range, and the whiskers represent the highest and lowest values, excluding outliers () and extremes (*). Outliers represent cases with values between 1.5 and 3 times the interquartile range. Extremes represent cases with values of more than 3 times the interquartile range.
4. Discussion In the present study, we found significant citation bias of hepato-biliary randomized clinical trials. Trials with a positive outcome were cited significantly more often than trials with a negative outcome. The disease area and the generation of the allocation sequence were significant independent predictors of the citation frequency but did not explain the association between the study outcome and citation frequency. The present study includes trials from several disease areas within one specialty. This may be viewed as the strength and weakness of the study. On one hand, including a more homogenous sample of trials would have provided a more focused answer. On the other hand, the variety of the included trials increases the external validity of the results, which indicate that citation bias may be a general problem. We included trials published in English-language journals indexed in MEDLINE to avoid language bias and database bias [3]. This approach may, however, also affect the external validity of the findings. Whether there is citation bias among trials published in non-English language journals not indexed in MEDLINE remains to be established. It is possible that confounders not included in the present study affect the findings. First, the accessibility and popularity of a journal may be associated with the citation rates. However, Ravnskov [9] found that the place of publication
409
had no significant effect on citation rates. Second, citation rates may be affected by the sample size of the trials. However, statistical significance may be considered an intermediate variable in the assessment of the association between sample size and citation counts, and, accordingly, we did not adjust for this potential confounder. Furthermore, it may be more important to establish whether there is a bias toward preferential citation of positive randomized clinical trials than to establish why such bias exist. Due to the design of the present study, the results cannot directly show the effect of citation bias on the results of metaanalyses. However, the effect of citation bias on the conclusions of systematic reviews may be inferred from the results. In theory, preferential citing of positive trials may lead to the implementation of ineffective or even harmful interventions. The results therefore support recommendations that the identification of randomized clinical trials for systematic reviews should be based on thorough electronic and manual searches [21]. Trials that are not indexed in an electronic database have to be identified through manual searches. Citation bias can be an important problem among such trials. Several initiatives are being performed to improve the registration of trials including the ongoing handsearches within The Cochrane Collaboration [21] and campaigns for searchable archives of ongoing and completed randomized clinical trials [22–24]. The magnitude of the problem of citation bias among trials available through manual searches only may therefore be decreasing. However, registration of randomized clinical trials in electronic databases does not eliminate the problem of citation bias. Several trials may be missed through electronic searches due to inadequate indexing in the electronic databases and/or inadequate search strategies. We therefore performed manual and electronic searches to ensure the completeness of the sample. The results indicate that citation bias can be an important problem among trials registered in electronic databases and thereby stresses the importance of comprehensive electronic search strategies. The present study shows that among randomized clinical trials on several hepato-biliary disease areas, there was a significant positive association between a statistically significant study outcome and citation frequency. This confirms two previous studies showing that nonsteroid antiinflammatory drug trials and trials on lowering cholesterol levels were cited significantly more often if the result was statistically significant than if the result was nonsignificant [8–10]. Two other studies have found the opposite trend among trials in the judgment and decision literature and on pneumococcal vaccines [11,12]. The inconsistency may reflect different citation habits in different areas (e.g., depending on financial interests and prior beliefs). The present study included a relatively large sample of trials compared with previous studies and, unlike previous studies, covered several disease areas. This may increase the external validity of the conclusions suggesting that citation bias is a general problem.
410
L.L. Kjaergard and C. Gluud / Journal of Clinical Epidemiology 55 (2002) 407–410
Researchers should above all cite good-quality research. We have previously found that trials with inadequate or unclear randomization and trials without double blinding exaggerate intervention effects significantly compared with large, gold-standard, randomized clinical trials [16]. We found a positive association between adequate generation of the allocation sequence and citation frequency. To some extent, the results oppose the results of a survey showing that in general researchers do not consider the quality of the evidence when deciding which evidence to cite [7]. Even retracted articles continue to be cited as valid work after publication of the retraction [25]. Ravnskov concluded that “fundamental parts of the dietheart idea are based on biased citations” [10]. Our results confirm that the search for eligible trials for inclusion in systematic reviews should be as thorough as possible. We suggest that manual searches should not only include bibliographies but also relevant journals and conference proceedings. Furthermore, communication with pharmaceutical companies and other researchers within the field may also provide important information about additional trials not identified through manual or electronic searches. Above all, extensive efforts are needed to ensure that the conclusions of a meta-analysis are not based on a biased sample of trials. Acknowledgments We thank Peter C. Gøtzsche, Thorkild I.A. Sørensen, and two peer referees at Journal of Clinical Epidemiology for valuable comments to an earlier draft of this manuscript. We also thank Ninna Frydendal, Dimitrinka Nikolova, Sarah L. Frederiksen, and Nader Salasshahri for their help with the identification and retrieval of trials. This work was supported by grants from The Danish Medical Research Council and The 1991 Pharmacy Foundation in Denmark. References [1] Bero LA, Jadad AR. How consumers and policymakers can use systematic reviews for decision making. Ann Intern Med 1997;127:37–42. [2] Gluud C. “Negative trials” are positive! J Hepatol 1998;28:731–3.
[3] Egger M, Smith GD. Bias in location and selection of studies. BMJ 1998;316:61–6. [4] Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867–72. [5] Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315:640–5. [6] Egger M, Zellweger-Zahner T, Schneider M, Juncker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997;350:326–9. [7] Song F, Eastwood S, Gilbody S, Duley L, Sutton AJ. Publication and related biases. Health Technol Assess 2000;4:1–115. [8] Gøtzsche PC. Reference bias in reports of drug trials. BMJ 1987;195: 654–56. [9] Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 1992;305:15–9. [10] Ravnskov U. Quotation bias in reviews of the diet-heart idea. J Clin Epidemiol 1995;48:713–9. [11] Christensen-Szalanski JJJ, Beach LR. The citation bias: fad and fashion in the judgement and descision literature. J Am Psychol 1984;39:75–8. [12] Hutchison BG, Lloyd S. Comprehensiveness and bias in reporting clinical trials: study of reviews of pneumococcal vaccine effectiveness. Can Fam Physician 1995;41:1356–60. [13] Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314:498–502. [14] Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273:408–12. [15] Moher D, Pham B, Jones A, Cook DJ, Jadad AR, Moher M, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609–13. [16] Kjaergard LL, Villumsen J, Gluud C. Reported methodological quality and discrepancies between large and small randomized trials in meta-analyses. Ann Intern Med December 2001;135:982–9. [17] Gluud C, Nikolova D. Quality assessment of reports on clinical trials in the Journal of Hepatology. J Hepatol 1998;29:321–7. [18] Gluud C. Evidence based medicine in Liver. Liver 1999;19:1–2. [19] Kjaergard LL, Nikolova D, Gluud C. Randomized clinical trials in Hepatology: predictors of quality. Hepatol 1999;30:1134–8. [20] The Cochrane Library. Issue 2, 2001. Oxford: Update Software. [21] Clarke M, Oxman A, Ed. Cochrane reviewers’ handbook 4.1.4. In: The Cochrane Library, Issue 4, 2001; Oxford: Update Software. [22] Roberts I. Pharmaceutical industry is invited to respond to amnesty for unreported trials. BMJ 1997;315:683. [23] Blunt J. Research ethics committees must ensure registration of research and accessibility of results. BMJ 1998;316:312. [24] Tonks A. Registering clinical trials. BMJ 1999;319:1565–8. [25] Budd JM, Sievert M, Schultz TR. Phenomena of retraction: reasons for retraction and citations to the publications. JAMA 2001;280:296–7.