- Email: [email protected]
Adjudication-related processes are underreported and lack standardization in clinical trials of venous thromboembolism: a systematic review
Journal of Clinical Epidemiology 67 (2014) 278e284
Adjudication-related processes are underreported and lack standardization in clinical trials of venous thromboembolism: a systematic review Anna K. Stucka,*, Evelyn Fuhrera, Andreas Limacherb,c, Marie Meana, Drahomir Aujeskya a Division of General Internal Medicine, University Hospital of Bern, 3010 Bern, Switzerland CTU Bern, Department of Clinical Research, University of Bern, Finkenhubelweg 11, 3012 Bern, Switzerland c Institute of Social and Preventive Medicine (ISPM), University of Bern, Finkenhubelweg 11, 3012 Bern, Switzerland b
Accepted 18 September 2013; Published online 28 November 2013
Abstract Objectives: Although the use of an adjudication committee (AC) for outcomes is recommended in randomized controlled trials, there are limited data on the process of adjudication. We therefore aimed to assess whether the reporting of the adjudication process in venous thromboembolism (VTE) trials meets existing quality standards and which characteristics of trials influence the use of an AC. Study Design and Setting: We systematically searched MEDLINE and the Cochrane Library from January 1, 2003, to June 1, 2012, for randomized controlled trials on VTE. We abstracted information about characteristics and quality of trials and reporting of adjudication processes. We used stepwise backward logistic regression model to identify trial characteristics independently associated with the use of an AC. Results: We included 161 trials. Of these, 68.9% (111 of 161) reported the use of an AC. Overall, 99.1% (110 of 111) of trials with an AC used independent or blinded ACs, 14.4% (16 of 111) reported how the adjudication decision was reached within the AC, and 4.5% (5 of 111) reported on whether the reliability of adjudication was assessed. In multivariate analyses, multicenter trials [odds ratio (OR), 8.6; 95% confidence interval (CI): 2.7, 27.8], use of a data safetyemonitoring board (OR, 3.7; 95% CI: 1.2, 11.6), and VTE as the primary outcome (OR, 5.7; 95% CI: 1.7, 19.4) were associated with the use of an AC. Trials without random allocation concealment (OR, 0.3; 95% CI: 0.1, 0.8) and open-label trials (OR, 0.3; 95% CI: 0.1, 1.0) were less likely to report an AC. Conclusion: Recommended processes of adjudication are underreported and lack standardization in VTE-related clinical trials. The use of an AC varies substantially by trial characteristics. Ó 2014 Elsevier Inc. All rights reserved. Keywords: Systematic review; Outcome adjudication; Venous thromboembolism; Outcome assessment; Data quality; Data reporting
1. Introduction The quality of the design and execution of randomized clinical trials (RCTs) has a direct impact on study results and internal validity [1]. Although an inadequate or unclear allocation concealment and a lack of blinding were shown to overestimate treatment effects [1,2], the effect of the method of outcome adjudication on study results has received little attention. Outcome adjudication is usually defined as the process to determine whether a patient has reached one or several predefined study end points. Limited evidence suggests that the process of outcome adjudication
Funding: This work was partially supported by a grant from the Swiss National Science Foundation (no. 33CSCO-122659/139 470). Conflict of interest: The authors have no conflict of interest. * Corresponding author. Tel.: 31-632-2111; fax: 31-632-0638. E-mail address: [email protected] (A.K. Stuck). 0895-4356/$ - see front matter Ó 2014 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jclinepi.2013.09.011
has a direct impact on study results in RCTs of heart diseases [3e6]. Moreover, in a systematic review, nonblinded outcome assessors generated substantially biased effect estimates in RCTs, overestimating odds ratios (ORs) by 36% [7]. To standardize the assessment of outcomes and reduce the risk of ascertainment bias, the Food and Drug Administration and the European Medicines Agency recommend assessment of events by adjudication committees (ACs) [8,9]. Based on the observation that the reporting and quality of ACs may often be insufficient to minimize the risk of ascertainment bias, Dechartres et al. [10] established quality criteria for planning and reporting ACs in RCTs. These include a description of the method for selecting cases to adjudicate, type of information provided to the AC, number and composition of the ACs involved, blinding/independence of the AC members, method of reaching a final decision, and assessment of the reliability of adjudication [10].
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
What is new? Recommended processes of outcome adjudication are underreported and lack standardization in randomized clinical trials of venous thromboembolism (VTE). Our results demonstrate that the use of adjudication committees varies substantially by trial characteristics. Adequate reporting of adjudication processes and the use of standardized adjudication methods should be encouraged in future randomized clinical trials of VTE.
The assessment of clinical end points may be particularly challenging in clinical trials of venous thromboembolism (VTE). Commonly used outcomes such as the first or recurrent VTE, major bleeding, and disease-specific mortality have complex definitions and, therefore, may be particularly prone to bias [10]. We therefore aimed to assess in a systematic review whether the reporting of the adjudication process in clinical VTE trials meets existing quality standards and which characteristics of trials influence the quality of the adjudication reporting.
2. Methods 2.1. Data sources and search strategy We developed and followed a protocol for this review based on the PRISMA statement for conducting and reporting systematic reviews [11]. We conducted a systematic search in MEDLINE and the Cochrane Central Register of Controlled Trials from January 1, 2003, to June 1, 2012, with no language restrictions. We identified additional articles by searching cited references of relevant articles. The detailed search strategy is shown in the Appendix at www.jclinepi.com. 2.2. Study selection We included original RCTs of VTE prevention, diagnosis, or treatment with any of the following outcomes: VTE (index event or recurrence), bleeding, or mortality. We excluded method articles of RCTs, articles reporting subgroup analyses of a previous trial, and follow-up trials. The other reason for exclusion was a language other than English, German, French, Italian, or Spanish. Two investigators (A.K.S. and E.F.) independently screened the titles and abstracts of the citations identified by the search strategy and evaluated the full text of the original articles for study
279
inclusion. Disagreements on inclusion were resolved through discussion with a third reviewer (M.M.). The agreement on inclusion of the trials between the two reviewers was substantial (k value 0.84 for screening the abstracts and k value 0.83 for screening the full texts). 2.3. Data extraction We abstracted information about characteristics of trials and the reporting of the use of an AC and other adjudication processes using a standardized data extraction tool adapted from Dechartres et al. [10]. If there was a reference to a methodological article or an appendix in the article, these were consulted as well. Two investigators (A.K.S. and E.F.) independently abstracted data on the reporting of the use of an AC. Disagreements on abstraction were resolved through discussion with a third reviewer (M.M.). 2.4. Characteristics and quality of trials We abstracted characteristics of trials including the study topic (prevention, diagnosis, or treatment), type of intervention (pharmacological or nonpharmacological), number of participating centers and enrolled patients, funding source (industrial or nonindustrial), year of publication, 2011 journal impact factor, use of a data safetyemonitoring board, and study outcomes (index VTE event or recurrence, bleeding, or mortality). Trials that did not provide data about the number of involved centers were considered single center. Trials that had mixed funding from industrial and nonindustrial sources were considered as industry-funded. Based on the 2011 journal impact factor, we classified journals into two groups: impact factor !10 vs. 10, as suggested by Wangge et al. [12]. Because Dechartres et al. published their recommendations about outcome adjudication in 2009, we stratified trials according to the year of publication (2003e2008 vs. 2009e2012) [10]. We assessed the methodological quality of trials based on three domains that are known to minimize bias in randomized controlled trials: adequate reporting of the random sequence generation and allocation concealment and the presence of double blinding (blinding of participants and personnel) [1]. If the random sequence generation and allocation methods were adequate and clearly reported, the trial was categorized as being at low risk for bias. All other trials were considered as being at high risk. We also documented whether trials were open-label or single- or double-blinded. A trial was considered as having an overall low risk of bias if it was double-blinded and had adequate random sequence generation and allocation concealment. All other trials were considered at overall high risk of bias. 2.5. Recommended processes of adjudication We assessed whether the use of an AC was reported in each trial, searching for all possible terms related to adjudication (eg, AC, end point committee, or clinical event
280
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
committee). If no term related to adjudication was reported, studies with at least two investigators in charge of reviewing potential cases were also considered having an AC. To characterize the quality of adjudication reporting according to Dechartres et al. [10], we recorded information about the methods for selecting cases to be adjudicated, type of information provided to the AC, number and composition of the AC (number, identification, and expertise of members), independence and blinding of the AC, methods of reaching a decision, and methods to assess the reliability of adjudication. 2.6. Data synthesis and analysis We described the general characteristics of trials and quality criteria, stratified by reported AC use. The proportions of reported AC use were compared across trial characteristics and quality criteria using chi-square tests, with P-values !0.05 indicating statistical significance. We used a stepwise backward logistic regression model to identify predefined trial characteristics and quality criteria that are independently associated with the reported use of an AC, including the following variables: study topic, type of intervention, multicenter trial, year of publication, funding source, 2011 journal impact factor, use of a data safetyemonitoring board, VTE as the primary outcome, description of the random sequence generation and allocation concealment, and blinding of participants and personnel. We used a P-value of 0.1 for variable elimination. Data analysis was performed using Stata statistical software (Release 11; StataCorp LP, College Station, TX).
3. Results 3.1. General characteristics and quality of trials We initially identified 775 studies; of which, 161 were included in our analysis (Fig. 1). Of these, most (80.1%) 769 abstracts identified through database searching: 767 from Medline 2 from the Cochrane Library
examined pharmacological interventions, and 77.0% were multicenter trials (Table 1). The median number of enrolled study patients was 523. Overall, 35.4% and 52.8% of trials did not report adequate random sequence generation and random allocation concealment, respectively, and 59.6% were open-label (Table 2). The interrater agreement between the two reviewers for reporting the use of an AC was substantial (k value 0.91). 3.2. Processes of adjudication Overall, 68.9% (111 of 161) of trials reported the use of an AC (Table 3). In trials with reported AC use, the method to select cases for adjudication was described in 86.5% (96 of 111), the type of information provided to the AC such as results of tests and procedures in 55.0% (61 of 111), and the number of AC members in 77.5% (86 of 111). Most trials reported a single AC consisting of three adjudicators. With the exception of one trial, all ACs (99.1%) were described as either independent or blinded. However, only 14.4% (16 of 111) of trials reported how a decision was reached within the committee (eg, on the basis of a full consensus of the AC members). Information on the expertise of the AC members and whether the reliability of adjudication was assessed was given only in 7.2% (8 of 111) and 4.5% (5 of 111) of trials, respectively. 3.3. Predictors of adjudication reporting In univariate analyses, the following characteristics were significantly associated with the use of an AC: use of a pharmacological intervention, multicenter design, larger sample size, industrial funding, publication in a journal with an impact factor 10, the use of a data safetyemonitoring board, and the following types of outcome: VTE, bleeding, and overall mortality (Table 1). In contrast, trials without adequately reported random sequence generation and
6 additional abstracts identified by hand search
775 abstracts screened
190 full-text articles assessed for eligibility
161 trials included in qualitative synthesis
585 abstracts excluded: 360 published before January 1, 2003 107 no venous thromboembolism study 64 no randomized controlled trial 34 follow up or subgroup study 6 no venous thromboembolism, bleeding or mortality as an outcome 14 other * 29 full-text articles excluded: 8 no venous thromboembolism study 8 follow up or subgroup trial 3 no randomized controlled trial 1 no venous thromboembolism, bleeding or mortality as an outcome 9 other *
Fig. 1. Flow chart. *Methodological articles in languages other than English, German, French, Italian, or Spanish were excluded.
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284 Table 1. General characteristics of trials and reporting of an adjudication committee (n 5 161)
Characteristic Study topic Diagnosis Prevention Treatment Type of interventiona Pharmacological Nonpharmacological Number of participating centersc Multicenter Single center Number of patients enrolled Median size of the trials !523 Median size of the trials 523 Funding sourced Industrial Nonindustrial Year of publicatione 2003e2008 2009e2012 2011 Impact factor !10 10 Use of a data safetyemonitoring board Yes No Primary outcomef Venous thromboembolism Yes No Bleeding Yes No Overall mortality Yes No Disease-specific mortality Yes No
No. of trials
Adjudication committee reported, N (%)
6 90 65
6 (100.0) 63 (70.0) 42 (64.6)
129 32
98 (76.0)b 13 (40.6)
124 37
b
105 (84.7) 6 (16.2)
80 81
36 (45.0)b 75 (92.6)
115 46
91 (79.1)b 20 (43.5)
107 54
71 (66.4) 40 (74.1)
104 57
59 (56.7)b 52 (91.2)
82 79
74 (90.2)b 37 (46.8)
132 29
98 (74.2)b 13 (44.8)
99 62
76 (76.8)b 35 (56.5)
61 100
51 (83.6)b 60 (60.0)
29 132
24 (82.8) 87 (65.9)
a Trials reporting both a pharmacological and a nonpharmacological intervention were considered nonpharmacological trials if the study focus was on the nonpharmacological intervention. b We used chi-square tests to examine associations between trial characteristics and the reporting of the use of an adjudication committee. Statistical significance at P-value !0.05. c Trials were classified as multicenter vs. single-center trials. Two trials that did not report the number of involved centers were classified as single center. d Trials with both industrial and nonindustrial funding were considered industry-funded. e We classified trials according to whether they were published before (2003e2008) or after (2009e2012) the recommendations by Dechartres et al. were established. f Multiple primary outcomes per trial were possible.
concealment and open-label trials were significantly less likely to report the use of an AC (Table 2). In a multivariate analysis, multicenter design (OR, 8.6; 95% CI [confidence interval]: 2.7, 27.8), use of a data safetyemonitoring board (OR, 3.7; 95% CI: 1.2, 11.6), and
281
Table 2. Methodological quality of trials and reporting of an adjudication committee (n 5 161)
Characteristic Random sequence generationa Yes (low risk of bias) No (high risk of bias) Random allocation concealmentc Yes (low risk of bias) No (high risk of bias) Blinding of participants and personnel Double-blind Open-labeld Overall risk of biase Low risk High risk
No. of trials
Adjudication committee reported, N (%)
104 57
81 (77.9)b 30 (52.6)
76 85
70 (92.1)b 41 (48.2)
65 96
59 (90.8)b 52 (54.2)
31 130
29 (93.6)b 82 (63.1)
a If the random sequence generation was adequate and clearly reported, the trial was categorized as being at low risk for bias. All other trials were considered as being at high risk. b We used chi-square tests to examine associations between quality criteria of trials and the reporting of the use of an adjudication committee. Statistical significance at P-value !0.05. c If the random allocation concealment was adequate and clearly reported, the trial was categorized as being at low risk for bias. All other trials were considered as being at high risk. d We classified two single-blinded trials as open-label. Trials with both an open-label and a double-blind component were also classified as open-label. e We defined a trial to be at low overall risk of bias if it was doubleblinded and had a low risk of bias for random sequence generation and allocation concealment.
VTE as the primary outcome (OR, 5.7; 95% CI: 1.7, 19.4) were independently associated with the use of an AC (Table 4). Trials without adequately reported random allocation concealment (OR, 0.3; 95% CI: 0.1, 0.8) and open-label trials (OR, 0.3; 95% CI: 0.1, 1.0) were less likely to report an AC. In contrast, the study topic, type of intervention, year of publication, funding source, journal impact factor, and blinding of participants and personnel were not independently associated with the use of an AC.
4. Discussion The results of our systematic review show that the use of an AC was reported only in two-thirds of VTE-related clinical trials and that the reporting of adjudication processes varied substantially by characteristics and quality criteria of trials. In trials reporting an AC, the description of adjudication processes was variable, ranging from 4.5% for the method used to assess the reliability of adjudication to 99.1% for the independency or blinding of AC members. Moreover, the reporting not only differed in quantity but also showed substantial methodological heterogeneity, as demonstrated by the example of seven different methods used to reach a final decision on adjudication. Overall, the use of an AC and many recommended adjudication-
282
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
Table 3. Reporting of adjudication processes (n 5 111) Reporting of Method of selecting casesa Suspected events identified by local investigators Otherb Information provided to the ACc Results of tests and procedures performed Otherd Number of ACs involved per trial 1 2 3 Number of members per AC 1 2 3 4 5 6 Number of AC members reviewing one case 1 2 3 4 Name of AC members mentioned Expertise of AC memberse Independence or blinding of the AC Blinding Independence Method of reaching a decision Full consensus within the whole AC Two independent raters, final decision by an arbiter in case of disagreement Two independent raters, final decision based on the consensus of two physicians in case of disagreement Otherf Method to assess reliability of adjudication
No. of trials (%) 96 (86.5) 80 (72.1) 53 (47.7) 61 (55.0) 60 (54.1) 15 86 57 16 13 86 11 23 55 20 13 12 27
(13.5) (77.5) (51.4) (14.4) (11.7) (77.5) (9.9) (20.7) (49.5) (18.0) (11.7) (10.8) (24.3)
3 24 4 3 76 8 110 103 83 16 6 3
(2.7) (21.6) (3.6) (2.7) (68.5) (7.2) (99.1) (92.8) (74.8) (14.4) (5.4) (2.7)
3 (2.7)
4 (3.6) 5 (4.5)
Abbreviation: AC, adjudication committee. a Multiple methods for selecting cases to adjudicate may apply. b Patients with pathologic imaging results (eg, ventilationeperfusion lung scan, venography, or ultrasound) and patients who died were selected for review. c Multiple types of information may have been provided to the AC. d Autopsy reports (n 5 8), standardized case report forms (n 5 4), results of laboratory tests and clinical notes (n 5 1), and clinical details (n 5 1). e Including thrombosis and bleeding experts, radiologists, surgeons, and other clinicians. f Two independent raters, final decision based on the consensus of the three physicians in case of disagreement (n 5 1); a local radiologist and a central AC member independently, in case of disagreement, the whole AC made the final decision (n 5 1); local and central AC independently, in case of disagreement, the central AC made the final decision (n 5 1); and one central rater (n 5 1).
related processes appear to be underreported and lack standardization in clinical trials of VTE. There are several potential reasons for the incomplete reporting of AC and adjudication processes. First, in contrast
Table 4. Independent predictors of reporting an adjudication committee (n 5 161) Predictor Multicenter design (vs. single center) Use of a data safetyemonitoring board Venous thromboembolism as the primary outcome No random allocation concealment Open-label design (vs. double-blind)
Adjusted odds ratioa (95% confidence interval) 8.6 (2.7, 27.8) 3.7 (1.2, 11.6) 5.7 (1.7, 19.4) 0.3 (0.1, 0.8) 0.3 (0.1, 1.0)
a The stepwise backward selection logistic regression model (Pvalue 0.1 for variable elimination) included the following variables: study topic, type of intervention, multicenter design, funding source, year of publication, 2011 journal impact factor, use of a data safetyemonitoring board, venous thromboembolism as the primary outcome, description of random sequence generation and allocation concealment, and blinding of participants and personnel.
to the less widely known detailed recommendations by Dechartres et al., the original and updated CONSORT guidelines of reporting RCTs give the somewhat unspecific recommendation to describe ‘‘how outcomes were assessed’’ and whether the assessors were blinded [13e15]. Thus, although authors almost always described whether the adjudicators were blinded or independent in our work (99.1% of trials), they may have felt less compelled to report other adjudication processes, given the space limitations in medical journals. Second, the establishment and running of an AC is a time- and resource-intensive task and may potentially represent a logistical barrier. Finally, although a recent systematic review clearly demonstrated significantly exaggerated treatment effects in trials with nonblinded vs. blinded assessors for binary and scale measurement outcomes [7,16], whether the use of an AC improves the ability to determine treatment effects remains controversial [17]. Evidence from heart trials suggests that compared with a central AC, local site investigators both under- and overreported protocoldefined end points such as myocardial infarctions, refractory angina, and cardiac death [3e6]. In contrast, a meta-analysis of RCTs did not find any benefit of event adjudication of cardiovascular outcomes such as myocardial infarction, stroke, or cardiovascular death [18]. Whether the quality of adjudication reporting has an impact on the effect size of an intervention in VTE trials and which individual adjudication processes are potentially outcome-relevant must be further explored before adjudication standards can be defined with confidence. Our results are not entirely consistent with the review of 314 RCTs by Dechartres et al. [10] who found a much lower proportion of trials (33.4%) reporting an AC than our study (68.9%). This difference could be explained by the fact that Dechartres et al. included articles from various medical fields, with AC reporting ranging from 8.8% for infectious disease trials to 81.3% for trials of the cardiac and cardiovascular system. After adjustment for other trial characteristics, multicenter trials and trials using a data safetyemonitoring board
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
were significantly associated with AC reporting. It is unclear why VTE as the primary outcome was significantly associated with AC reporting. Given the complexity of VTE diagnosis, which is based on various imaging techniques, investigators may more frequently rely on ACs for diagnosing VTE than for more explicit outcomes such as overall mortality and bleeding. Obviously, the need for an AC also depends on the outcome studied. Although a study examining overall mortality may not need an AC at all, an AC is all the more important when subjective or complex outcomes are used such as cause-specific mortality. Not surprisingly, we found that other markers of a lower methodological study quality, that is, missing random allocation concealment and an open-label design, were negatively associated with AC reporting. In contrast, neither the funding source nor the journal impact factor was related with AC reporting, and we did not observe an improvement of AC reporting over time. There are several potential limitations to our study. First, our systematic review focused on RCTs related to VTE. Thus, our findings cannot be extrapolated to other clinical fields and study designs. Second, our search was restricted to articles published between January 1, 2003, and June 1, 2012, and may not be representative of all VTE-related clinical trials. Third, we possibly did not identify all trials during this time period, and we do not know whether they would report substantially different results. However, publication bias is usually driven by the effect of a treatment [19] and may have less impact on our analysis of adjudication processes. Fourth, our findings are based on reported information only. This is justified because faulty reporting often reflects faulty methods [1]. However, a wellconducted but badly reported trial would have been misclassified [20,21]. Finally, our study focused on the reporting of recommended adjudication processes, and we could not evaluate the impact of the quality of adjudication reporting on the effect sizes of interventions, which must be done in future studies. In conclusion, our results demonstrate that in clinical trials of VTE, the reporting of the use of an AC and other recommended processes of adjudication varies considerably based on characteristics of trials. Many adjudicationrelated processes are underreported and lack standardization. Whether the quality of adjudication reporting has an impact on the magnitude of the effect size of interventions in VTE trials must be further examined.
Acknowledgments A.K.S.: planning of the study, data abstraction, statistical analyses, and drafting of the manuscript. E.F.: data abstraction and critical review of the manuscript. A.L.: statistical consulting and critical review of the manuscript. M.M.: data abstraction and critical review of the manuscript. D.A.: planning of the study and drafting of the manuscript.
283
Appendix A Supplementary data The supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.jclinepi.2013.09.011.
References [1] Juni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001;323: 42e6. [2] Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012;157:429e38. [3] Mahaffey KW, Harrington RA, Akkerhuis M, Kleiman NS, Berdan LG, Crenshaw BS, et al. Disagreements between central clinical events committee and site investigator assessments of myocardial infarction endpoints in an international clinical trial: review of the PURSUIT study. Curr Control Trials Cardiovasc Med 2001;2(4): 187e94. [4] Naslund U, Grip L, Fischer-Hansen J, Gundersen T, Lehto S, Wallentin L. The impact of an end-point committee in a large multicentre, randomized, placebo-controlled clinical trial: results with and without the end-point committee’s final decision on end-points. Eur Heart J 1999;20:771e7. [5] Petersen JL, Haque G, Hellkamp AS, Flaker GC, Mark Estes NA 3rd, Marchlinski FE, et al. Comparing classifications of death in the Mode Selection Trial: agreement and disagreement among site investigators and a clinical events committee. Contemp Clin Trials 2006;27: 260e8. [6] Mahaffey KW, Roe MT, Dyke CK, Newby LK, Kleiman NS, Connolly P, et al. Misreporting of myocardial infarction end points: results of adjudication by a central clinical events committee in the PARAGON-B trial. Second Platelet IIb/IIIa Antagonist for the Reduction of Acute Coronary Syndrome Events in a Global Organization Network Trial. Am Heart J 2002;143:242e8. [7] Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ 2012;344:e1119. [8] Guideline on data monitoring committees [Internet]. European Medicines Agency. Committee for medicinal products for human use (CHMP) 2005 [cited 2013 Apr 19]. Available from: http://www. emea.europa.eu/docs/en_GB/document_library/Scientific_guideline/ 2009/09/WC500003635.pdf. [9] Guidance for clinical trial sponsors. Establishment and operation of clinical trial data monitoring committees [Internet]. U.S. Department of Health and Human Services Food and Drug Administration 2006 [cited 2013 Apr 19]. Available from: http://www.fda.gov/downloads/ Regulatoryinformation/Guidances/ucm127073.pdf. [10] Dechartres A, Boutron I, Roy C, Ravaud P. Inadequate planning and reporting of adjudication committees in clinical trials: recommendation proposal. J Clin Epidemiol 2009;62:695e702. [11] Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535. [12] Wangge G, Klungel OH, Roes KC, de Boer A, Hoes AW, Knol MJ. Room for improvement in conducting and reporting non-inferiority randomized controlled trials on drugs: a systematic review. PLoS One 2010;5(10):e13550. [13] Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637e9.
284
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
[14] Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallelgroup randomized trials. JAMA 2001;285:1987e91. [15] Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869. [16] Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. CMAJ 2013;185(4):E201e11. [17] Granger CB, Vogel V, Cummings SR, Held P, Fiedorek F, Lawrence M, et al. Do we need to adjudicate major clinical events? Clin Trials 2008;5:56e60.
[18] Pogue J, Walter SD, Yusuf S. Evaluating the benefit of event adjudication of cardiovascular outcomes in large simple RCTs. Clin Trials 2009;6:239e51. [19] Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315: 640e5. [20] Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004; 328:22e4. [21] Huwiler-Muntener K, Juni P, Junker C, Egger M. Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 2002;287:2801e4.
Adjudication-related processes are underreported and lack standardization in clinical trials of venous thromboembolism: a systematic review Anna K. Stucka,*, Evelyn Fuhrera, Andreas Limacherb,c, Marie Meana, Drahomir Aujeskya a Division of General Internal Medicine, University Hospital of Bern, 3010 Bern, Switzerland CTU Bern, Department of Clinical Research, University of Bern, Finkenhubelweg 11, 3012 Bern, Switzerland c Institute of Social and Preventive Medicine (ISPM), University of Bern, Finkenhubelweg 11, 3012 Bern, Switzerland b
Accepted 18 September 2013; Published online 28 November 2013
Abstract Objectives: Although the use of an adjudication committee (AC) for outcomes is recommended in randomized controlled trials, there are limited data on the process of adjudication. We therefore aimed to assess whether the reporting of the adjudication process in venous thromboembolism (VTE) trials meets existing quality standards and which characteristics of trials influence the use of an AC. Study Design and Setting: We systematically searched MEDLINE and the Cochrane Library from January 1, 2003, to June 1, 2012, for randomized controlled trials on VTE. We abstracted information about characteristics and quality of trials and reporting of adjudication processes. We used stepwise backward logistic regression model to identify trial characteristics independently associated with the use of an AC. Results: We included 161 trials. Of these, 68.9% (111 of 161) reported the use of an AC. Overall, 99.1% (110 of 111) of trials with an AC used independent or blinded ACs, 14.4% (16 of 111) reported how the adjudication decision was reached within the AC, and 4.5% (5 of 111) reported on whether the reliability of adjudication was assessed. In multivariate analyses, multicenter trials [odds ratio (OR), 8.6; 95% confidence interval (CI): 2.7, 27.8], use of a data safetyemonitoring board (OR, 3.7; 95% CI: 1.2, 11.6), and VTE as the primary outcome (OR, 5.7; 95% CI: 1.7, 19.4) were associated with the use of an AC. Trials without random allocation concealment (OR, 0.3; 95% CI: 0.1, 0.8) and open-label trials (OR, 0.3; 95% CI: 0.1, 1.0) were less likely to report an AC. Conclusion: Recommended processes of adjudication are underreported and lack standardization in VTE-related clinical trials. The use of an AC varies substantially by trial characteristics. Ó 2014 Elsevier Inc. All rights reserved. Keywords: Systematic review; Outcome adjudication; Venous thromboembolism; Outcome assessment; Data quality; Data reporting
1. Introduction The quality of the design and execution of randomized clinical trials (RCTs) has a direct impact on study results and internal validity [1]. Although an inadequate or unclear allocation concealment and a lack of blinding were shown to overestimate treatment effects [1,2], the effect of the method of outcome adjudication on study results has received little attention. Outcome adjudication is usually defined as the process to determine whether a patient has reached one or several predefined study end points. Limited evidence suggests that the process of outcome adjudication
Funding: This work was partially supported by a grant from the Swiss National Science Foundation (no. 33CSCO-122659/139 470). Conflict of interest: The authors have no conflict of interest. * Corresponding author. Tel.: 31-632-2111; fax: 31-632-0638. E-mail address: [email protected] (A.K. Stuck). 0895-4356/$ - see front matter Ó 2014 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jclinepi.2013.09.011
has a direct impact on study results in RCTs of heart diseases [3e6]. Moreover, in a systematic review, nonblinded outcome assessors generated substantially biased effect estimates in RCTs, overestimating odds ratios (ORs) by 36% [7]. To standardize the assessment of outcomes and reduce the risk of ascertainment bias, the Food and Drug Administration and the European Medicines Agency recommend assessment of events by adjudication committees (ACs) [8,9]. Based on the observation that the reporting and quality of ACs may often be insufficient to minimize the risk of ascertainment bias, Dechartres et al. [10] established quality criteria for planning and reporting ACs in RCTs. These include a description of the method for selecting cases to adjudicate, type of information provided to the AC, number and composition of the ACs involved, blinding/independence of the AC members, method of reaching a final decision, and assessment of the reliability of adjudication [10].
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
What is new? Recommended processes of outcome adjudication are underreported and lack standardization in randomized clinical trials of venous thromboembolism (VTE). Our results demonstrate that the use of adjudication committees varies substantially by trial characteristics. Adequate reporting of adjudication processes and the use of standardized adjudication methods should be encouraged in future randomized clinical trials of VTE.
The assessment of clinical end points may be particularly challenging in clinical trials of venous thromboembolism (VTE). Commonly used outcomes such as the first or recurrent VTE, major bleeding, and disease-specific mortality have complex definitions and, therefore, may be particularly prone to bias [10]. We therefore aimed to assess in a systematic review whether the reporting of the adjudication process in clinical VTE trials meets existing quality standards and which characteristics of trials influence the quality of the adjudication reporting.
2. Methods 2.1. Data sources and search strategy We developed and followed a protocol for this review based on the PRISMA statement for conducting and reporting systematic reviews [11]. We conducted a systematic search in MEDLINE and the Cochrane Central Register of Controlled Trials from January 1, 2003, to June 1, 2012, with no language restrictions. We identified additional articles by searching cited references of relevant articles. The detailed search strategy is shown in the Appendix at www.jclinepi.com. 2.2. Study selection We included original RCTs of VTE prevention, diagnosis, or treatment with any of the following outcomes: VTE (index event or recurrence), bleeding, or mortality. We excluded method articles of RCTs, articles reporting subgroup analyses of a previous trial, and follow-up trials. The other reason for exclusion was a language other than English, German, French, Italian, or Spanish. Two investigators (A.K.S. and E.F.) independently screened the titles and abstracts of the citations identified by the search strategy and evaluated the full text of the original articles for study
279
inclusion. Disagreements on inclusion were resolved through discussion with a third reviewer (M.M.). The agreement on inclusion of the trials between the two reviewers was substantial (k value 0.84 for screening the abstracts and k value 0.83 for screening the full texts). 2.3. Data extraction We abstracted information about characteristics of trials and the reporting of the use of an AC and other adjudication processes using a standardized data extraction tool adapted from Dechartres et al. [10]. If there was a reference to a methodological article or an appendix in the article, these were consulted as well. Two investigators (A.K.S. and E.F.) independently abstracted data on the reporting of the use of an AC. Disagreements on abstraction were resolved through discussion with a third reviewer (M.M.). 2.4. Characteristics and quality of trials We abstracted characteristics of trials including the study topic (prevention, diagnosis, or treatment), type of intervention (pharmacological or nonpharmacological), number of participating centers and enrolled patients, funding source (industrial or nonindustrial), year of publication, 2011 journal impact factor, use of a data safetyemonitoring board, and study outcomes (index VTE event or recurrence, bleeding, or mortality). Trials that did not provide data about the number of involved centers were considered single center. Trials that had mixed funding from industrial and nonindustrial sources were considered as industry-funded. Based on the 2011 journal impact factor, we classified journals into two groups: impact factor !10 vs. 10, as suggested by Wangge et al. [12]. Because Dechartres et al. published their recommendations about outcome adjudication in 2009, we stratified trials according to the year of publication (2003e2008 vs. 2009e2012) [10]. We assessed the methodological quality of trials based on three domains that are known to minimize bias in randomized controlled trials: adequate reporting of the random sequence generation and allocation concealment and the presence of double blinding (blinding of participants and personnel) [1]. If the random sequence generation and allocation methods were adequate and clearly reported, the trial was categorized as being at low risk for bias. All other trials were considered as being at high risk. We also documented whether trials were open-label or single- or double-blinded. A trial was considered as having an overall low risk of bias if it was double-blinded and had adequate random sequence generation and allocation concealment. All other trials were considered at overall high risk of bias. 2.5. Recommended processes of adjudication We assessed whether the use of an AC was reported in each trial, searching for all possible terms related to adjudication (eg, AC, end point committee, or clinical event
280
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
committee). If no term related to adjudication was reported, studies with at least two investigators in charge of reviewing potential cases were also considered having an AC. To characterize the quality of adjudication reporting according to Dechartres et al. [10], we recorded information about the methods for selecting cases to be adjudicated, type of information provided to the AC, number and composition of the AC (number, identification, and expertise of members), independence and blinding of the AC, methods of reaching a decision, and methods to assess the reliability of adjudication. 2.6. Data synthesis and analysis We described the general characteristics of trials and quality criteria, stratified by reported AC use. The proportions of reported AC use were compared across trial characteristics and quality criteria using chi-square tests, with P-values !0.05 indicating statistical significance. We used a stepwise backward logistic regression model to identify predefined trial characteristics and quality criteria that are independently associated with the reported use of an AC, including the following variables: study topic, type of intervention, multicenter trial, year of publication, funding source, 2011 journal impact factor, use of a data safetyemonitoring board, VTE as the primary outcome, description of the random sequence generation and allocation concealment, and blinding of participants and personnel. We used a P-value of 0.1 for variable elimination. Data analysis was performed using Stata statistical software (Release 11; StataCorp LP, College Station, TX).
3. Results 3.1. General characteristics and quality of trials We initially identified 775 studies; of which, 161 were included in our analysis (Fig. 1). Of these, most (80.1%) 769 abstracts identified through database searching: 767 from Medline 2 from the Cochrane Library
examined pharmacological interventions, and 77.0% were multicenter trials (Table 1). The median number of enrolled study patients was 523. Overall, 35.4% and 52.8% of trials did not report adequate random sequence generation and random allocation concealment, respectively, and 59.6% were open-label (Table 2). The interrater agreement between the two reviewers for reporting the use of an AC was substantial (k value 0.91). 3.2. Processes of adjudication Overall, 68.9% (111 of 161) of trials reported the use of an AC (Table 3). In trials with reported AC use, the method to select cases for adjudication was described in 86.5% (96 of 111), the type of information provided to the AC such as results of tests and procedures in 55.0% (61 of 111), and the number of AC members in 77.5% (86 of 111). Most trials reported a single AC consisting of three adjudicators. With the exception of one trial, all ACs (99.1%) were described as either independent or blinded. However, only 14.4% (16 of 111) of trials reported how a decision was reached within the committee (eg, on the basis of a full consensus of the AC members). Information on the expertise of the AC members and whether the reliability of adjudication was assessed was given only in 7.2% (8 of 111) and 4.5% (5 of 111) of trials, respectively. 3.3. Predictors of adjudication reporting In univariate analyses, the following characteristics were significantly associated with the use of an AC: use of a pharmacological intervention, multicenter design, larger sample size, industrial funding, publication in a journal with an impact factor 10, the use of a data safetyemonitoring board, and the following types of outcome: VTE, bleeding, and overall mortality (Table 1). In contrast, trials without adequately reported random sequence generation and
6 additional abstracts identified by hand search
775 abstracts screened
190 full-text articles assessed for eligibility
161 trials included in qualitative synthesis
585 abstracts excluded: 360 published before January 1, 2003 107 no venous thromboembolism study 64 no randomized controlled trial 34 follow up or subgroup study 6 no venous thromboembolism, bleeding or mortality as an outcome 14 other * 29 full-text articles excluded: 8 no venous thromboembolism study 8 follow up or subgroup trial 3 no randomized controlled trial 1 no venous thromboembolism, bleeding or mortality as an outcome 9 other *
Fig. 1. Flow chart. *Methodological articles in languages other than English, German, French, Italian, or Spanish were excluded.
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284 Table 1. General characteristics of trials and reporting of an adjudication committee (n 5 161)
Characteristic Study topic Diagnosis Prevention Treatment Type of interventiona Pharmacological Nonpharmacological Number of participating centersc Multicenter Single center Number of patients enrolled Median size of the trials !523 Median size of the trials 523 Funding sourced Industrial Nonindustrial Year of publicatione 2003e2008 2009e2012 2011 Impact factor !10 10 Use of a data safetyemonitoring board Yes No Primary outcomef Venous thromboembolism Yes No Bleeding Yes No Overall mortality Yes No Disease-specific mortality Yes No
No. of trials
Adjudication committee reported, N (%)
6 90 65
6 (100.0) 63 (70.0) 42 (64.6)
129 32
98 (76.0)b 13 (40.6)
124 37
b
105 (84.7) 6 (16.2)
80 81
36 (45.0)b 75 (92.6)
115 46
91 (79.1)b 20 (43.5)
107 54
71 (66.4) 40 (74.1)
104 57
59 (56.7)b 52 (91.2)
82 79
74 (90.2)b 37 (46.8)
132 29
98 (74.2)b 13 (44.8)
99 62
76 (76.8)b 35 (56.5)
61 100
51 (83.6)b 60 (60.0)
29 132
24 (82.8) 87 (65.9)
a Trials reporting both a pharmacological and a nonpharmacological intervention were considered nonpharmacological trials if the study focus was on the nonpharmacological intervention. b We used chi-square tests to examine associations between trial characteristics and the reporting of the use of an adjudication committee. Statistical significance at P-value !0.05. c Trials were classified as multicenter vs. single-center trials. Two trials that did not report the number of involved centers were classified as single center. d Trials with both industrial and nonindustrial funding were considered industry-funded. e We classified trials according to whether they were published before (2003e2008) or after (2009e2012) the recommendations by Dechartres et al. were established. f Multiple primary outcomes per trial were possible.
concealment and open-label trials were significantly less likely to report the use of an AC (Table 2). In a multivariate analysis, multicenter design (OR, 8.6; 95% CI [confidence interval]: 2.7, 27.8), use of a data safetyemonitoring board (OR, 3.7; 95% CI: 1.2, 11.6), and
281
Table 2. Methodological quality of trials and reporting of an adjudication committee (n 5 161)
Characteristic Random sequence generationa Yes (low risk of bias) No (high risk of bias) Random allocation concealmentc Yes (low risk of bias) No (high risk of bias) Blinding of participants and personnel Double-blind Open-labeld Overall risk of biase Low risk High risk
No. of trials
Adjudication committee reported, N (%)
104 57
81 (77.9)b 30 (52.6)
76 85
70 (92.1)b 41 (48.2)
65 96
59 (90.8)b 52 (54.2)
31 130
29 (93.6)b 82 (63.1)
a If the random sequence generation was adequate and clearly reported, the trial was categorized as being at low risk for bias. All other trials were considered as being at high risk. b We used chi-square tests to examine associations between quality criteria of trials and the reporting of the use of an adjudication committee. Statistical significance at P-value !0.05. c If the random allocation concealment was adequate and clearly reported, the trial was categorized as being at low risk for bias. All other trials were considered as being at high risk. d We classified two single-blinded trials as open-label. Trials with both an open-label and a double-blind component were also classified as open-label. e We defined a trial to be at low overall risk of bias if it was doubleblinded and had a low risk of bias for random sequence generation and allocation concealment.
VTE as the primary outcome (OR, 5.7; 95% CI: 1.7, 19.4) were independently associated with the use of an AC (Table 4). Trials without adequately reported random allocation concealment (OR, 0.3; 95% CI: 0.1, 0.8) and open-label trials (OR, 0.3; 95% CI: 0.1, 1.0) were less likely to report an AC. In contrast, the study topic, type of intervention, year of publication, funding source, journal impact factor, and blinding of participants and personnel were not independently associated with the use of an AC.
4. Discussion The results of our systematic review show that the use of an AC was reported only in two-thirds of VTE-related clinical trials and that the reporting of adjudication processes varied substantially by characteristics and quality criteria of trials. In trials reporting an AC, the description of adjudication processes was variable, ranging from 4.5% for the method used to assess the reliability of adjudication to 99.1% for the independency or blinding of AC members. Moreover, the reporting not only differed in quantity but also showed substantial methodological heterogeneity, as demonstrated by the example of seven different methods used to reach a final decision on adjudication. Overall, the use of an AC and many recommended adjudication-
282
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
Table 3. Reporting of adjudication processes (n 5 111) Reporting of Method of selecting casesa Suspected events identified by local investigators Otherb Information provided to the ACc Results of tests and procedures performed Otherd Number of ACs involved per trial 1 2 3 Number of members per AC 1 2 3 4 5 6 Number of AC members reviewing one case 1 2 3 4 Name of AC members mentioned Expertise of AC memberse Independence or blinding of the AC Blinding Independence Method of reaching a decision Full consensus within the whole AC Two independent raters, final decision by an arbiter in case of disagreement Two independent raters, final decision based on the consensus of two physicians in case of disagreement Otherf Method to assess reliability of adjudication
No. of trials (%) 96 (86.5) 80 (72.1) 53 (47.7) 61 (55.0) 60 (54.1) 15 86 57 16 13 86 11 23 55 20 13 12 27
(13.5) (77.5) (51.4) (14.4) (11.7) (77.5) (9.9) (20.7) (49.5) (18.0) (11.7) (10.8) (24.3)
3 24 4 3 76 8 110 103 83 16 6 3
(2.7) (21.6) (3.6) (2.7) (68.5) (7.2) (99.1) (92.8) (74.8) (14.4) (5.4) (2.7)
3 (2.7)
4 (3.6) 5 (4.5)
Abbreviation: AC, adjudication committee. a Multiple methods for selecting cases to adjudicate may apply. b Patients with pathologic imaging results (eg, ventilationeperfusion lung scan, venography, or ultrasound) and patients who died were selected for review. c Multiple types of information may have been provided to the AC. d Autopsy reports (n 5 8), standardized case report forms (n 5 4), results of laboratory tests and clinical notes (n 5 1), and clinical details (n 5 1). e Including thrombosis and bleeding experts, radiologists, surgeons, and other clinicians. f Two independent raters, final decision based on the consensus of the three physicians in case of disagreement (n 5 1); a local radiologist and a central AC member independently, in case of disagreement, the whole AC made the final decision (n 5 1); local and central AC independently, in case of disagreement, the central AC made the final decision (n 5 1); and one central rater (n 5 1).
related processes appear to be underreported and lack standardization in clinical trials of VTE. There are several potential reasons for the incomplete reporting of AC and adjudication processes. First, in contrast
Table 4. Independent predictors of reporting an adjudication committee (n 5 161) Predictor Multicenter design (vs. single center) Use of a data safetyemonitoring board Venous thromboembolism as the primary outcome No random allocation concealment Open-label design (vs. double-blind)
Adjusted odds ratioa (95% confidence interval) 8.6 (2.7, 27.8) 3.7 (1.2, 11.6) 5.7 (1.7, 19.4) 0.3 (0.1, 0.8) 0.3 (0.1, 1.0)
a The stepwise backward selection logistic regression model (Pvalue 0.1 for variable elimination) included the following variables: study topic, type of intervention, multicenter design, funding source, year of publication, 2011 journal impact factor, use of a data safetyemonitoring board, venous thromboembolism as the primary outcome, description of random sequence generation and allocation concealment, and blinding of participants and personnel.
to the less widely known detailed recommendations by Dechartres et al., the original and updated CONSORT guidelines of reporting RCTs give the somewhat unspecific recommendation to describe ‘‘how outcomes were assessed’’ and whether the assessors were blinded [13e15]. Thus, although authors almost always described whether the adjudicators were blinded or independent in our work (99.1% of trials), they may have felt less compelled to report other adjudication processes, given the space limitations in medical journals. Second, the establishment and running of an AC is a time- and resource-intensive task and may potentially represent a logistical barrier. Finally, although a recent systematic review clearly demonstrated significantly exaggerated treatment effects in trials with nonblinded vs. blinded assessors for binary and scale measurement outcomes [7,16], whether the use of an AC improves the ability to determine treatment effects remains controversial [17]. Evidence from heart trials suggests that compared with a central AC, local site investigators both under- and overreported protocoldefined end points such as myocardial infarctions, refractory angina, and cardiac death [3e6]. In contrast, a meta-analysis of RCTs did not find any benefit of event adjudication of cardiovascular outcomes such as myocardial infarction, stroke, or cardiovascular death [18]. Whether the quality of adjudication reporting has an impact on the effect size of an intervention in VTE trials and which individual adjudication processes are potentially outcome-relevant must be further explored before adjudication standards can be defined with confidence. Our results are not entirely consistent with the review of 314 RCTs by Dechartres et al. [10] who found a much lower proportion of trials (33.4%) reporting an AC than our study (68.9%). This difference could be explained by the fact that Dechartres et al. included articles from various medical fields, with AC reporting ranging from 8.8% for infectious disease trials to 81.3% for trials of the cardiac and cardiovascular system. After adjustment for other trial characteristics, multicenter trials and trials using a data safetyemonitoring board
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
were significantly associated with AC reporting. It is unclear why VTE as the primary outcome was significantly associated with AC reporting. Given the complexity of VTE diagnosis, which is based on various imaging techniques, investigators may more frequently rely on ACs for diagnosing VTE than for more explicit outcomes such as overall mortality and bleeding. Obviously, the need for an AC also depends on the outcome studied. Although a study examining overall mortality may not need an AC at all, an AC is all the more important when subjective or complex outcomes are used such as cause-specific mortality. Not surprisingly, we found that other markers of a lower methodological study quality, that is, missing random allocation concealment and an open-label design, were negatively associated with AC reporting. In contrast, neither the funding source nor the journal impact factor was related with AC reporting, and we did not observe an improvement of AC reporting over time. There are several potential limitations to our study. First, our systematic review focused on RCTs related to VTE. Thus, our findings cannot be extrapolated to other clinical fields and study designs. Second, our search was restricted to articles published between January 1, 2003, and June 1, 2012, and may not be representative of all VTE-related clinical trials. Third, we possibly did not identify all trials during this time period, and we do not know whether they would report substantially different results. However, publication bias is usually driven by the effect of a treatment [19] and may have less impact on our analysis of adjudication processes. Fourth, our findings are based on reported information only. This is justified because faulty reporting often reflects faulty methods [1]. However, a wellconducted but badly reported trial would have been misclassified [20,21]. Finally, our study focused on the reporting of recommended adjudication processes, and we could not evaluate the impact of the quality of adjudication reporting on the effect sizes of interventions, which must be done in future studies. In conclusion, our results demonstrate that in clinical trials of VTE, the reporting of the use of an AC and other recommended processes of adjudication varies considerably based on characteristics of trials. Many adjudicationrelated processes are underreported and lack standardization. Whether the quality of adjudication reporting has an impact on the magnitude of the effect size of interventions in VTE trials must be further examined.
Acknowledgments A.K.S.: planning of the study, data abstraction, statistical analyses, and drafting of the manuscript. E.F.: data abstraction and critical review of the manuscript. A.L.: statistical consulting and critical review of the manuscript. M.M.: data abstraction and critical review of the manuscript. D.A.: planning of the study and drafting of the manuscript.
283
Appendix A Supplementary data The supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.jclinepi.2013.09.011.
References [1] Juni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001;323: 42e6. [2] Savovic J, Jones HE, Altman DG, Harris RJ, Juni P, Pildal J, et al. Influence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Ann Intern Med 2012;157:429e38. [3] Mahaffey KW, Harrington RA, Akkerhuis M, Kleiman NS, Berdan LG, Crenshaw BS, et al. Disagreements between central clinical events committee and site investigator assessments of myocardial infarction endpoints in an international clinical trial: review of the PURSUIT study. Curr Control Trials Cardiovasc Med 2001;2(4): 187e94. [4] Naslund U, Grip L, Fischer-Hansen J, Gundersen T, Lehto S, Wallentin L. The impact of an end-point committee in a large multicentre, randomized, placebo-controlled clinical trial: results with and without the end-point committee’s final decision on end-points. Eur Heart J 1999;20:771e7. [5] Petersen JL, Haque G, Hellkamp AS, Flaker GC, Mark Estes NA 3rd, Marchlinski FE, et al. Comparing classifications of death in the Mode Selection Trial: agreement and disagreement among site investigators and a clinical events committee. Contemp Clin Trials 2006;27: 260e8. [6] Mahaffey KW, Roe MT, Dyke CK, Newby LK, Kleiman NS, Connolly P, et al. Misreporting of myocardial infarction end points: results of adjudication by a central clinical events committee in the PARAGON-B trial. Second Platelet IIb/IIIa Antagonist for the Reduction of Acute Coronary Syndrome Events in a Global Organization Network Trial. Am Heart J 2002;143:242e8. [7] Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. BMJ 2012;344:e1119. [8] Guideline on data monitoring committees [Internet]. European Medicines Agency. Committee for medicinal products for human use (CHMP) 2005 [cited 2013 Apr 19]. Available from: http://www. emea.europa.eu/docs/en_GB/document_library/Scientific_guideline/ 2009/09/WC500003635.pdf. [9] Guidance for clinical trial sponsors. Establishment and operation of clinical trial data monitoring committees [Internet]. U.S. Department of Health and Human Services Food and Drug Administration 2006 [cited 2013 Apr 19]. Available from: http://www.fda.gov/downloads/ Regulatoryinformation/Guidances/ucm127073.pdf. [10] Dechartres A, Boutron I, Roy C, Ravaud P. Inadequate planning and reporting of adjudication committees in clinical trials: recommendation proposal. J Clin Epidemiol 2009;62:695e702. [11] Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 2009;339:b2535. [12] Wangge G, Klungel OH, Roes KC, de Boer A, Hoes AW, Knol MJ. Room for improvement in conducting and reporting non-inferiority randomized controlled trials on drugs: a systematic review. PLoS One 2010;5(10):e13550. [13] Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276:637e9.
284
A.K. Stuck et al. / Journal of Clinical Epidemiology 67 (2014) 278e284
[14] Moher D, Schulz KF, Altman D. The CONSORT statement: revised recommendations for improving the quality of reports of parallelgroup randomized trials. JAMA 2001;285:1987e91. [15] Moher D, Hopewell S, Schulz KF, Montori V, Gotzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 2010;340:c869. [16] Hrobjartsson A, Thomsen AS, Emanuelsson F, Tendal B, Hilden J, Boutron I, et al. Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. CMAJ 2013;185(4):E201e11. [17] Granger CB, Vogel V, Cummings SR, Held P, Fiedorek F, Lawrence M, et al. Do we need to adjudicate major clinical events? Clin Trials 2008;5:56e60.
[18] Pogue J, Walter SD, Yusuf S. Evaluating the benefit of event adjudication of cardiovascular outcomes in large simple RCTs. Clin Trials 2009;6:239e51. [19] Stern JM, Simes RJ. Publication bias: evidence of delayed publication in a cohort study of clinical research projects. BMJ 1997;315: 640e5. [20] Soares HP, Daniels S, Kumar A, Clarke M, Scott C, Swann S, et al. Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group. BMJ 2004; 328:22e4. [21] Huwiler-Muntener K, Juni P, Junker C, Egger M. Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 2002;287:2801e4.