A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews

Accepted Manuscript A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews Iosief Abraha, Francesco Cozzolino, Massimiliano Orso, Mauro Marchesi, Antonella Germani, Guido Lombardo, Paolo Eusebi, Rita De Florio, Maria Laura Luchetta, Alfonso Iorio, MD, Alessandro Montedori PII:

S0895-4356(16)30756-9

DOI:

10.1016/j.jclinepi.2016.11.012

Reference:

JCE 9281

To appear in:

Journal of Clinical Epidemiology

Received Date: 12 February 2016 Revised Date:

18 November 2016

Accepted Date: 28 November 2016

Please cite this article as: Abraha I, Cozzolino F, Orso M, Marchesi M, Germani A, Lombardo G, Eusebi P, De Florio R, Luchetta ML, Iorio A, Montedori A, A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews, Journal of Clinical Epidemiology (2017), doi: 10.1016/j.jclinepi.2016.11.012. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT

A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews

RI PT

Iosief Abrahaa, Francesco Cozzolinoa, Massimiliano Orsoa, Mauro Marchesib, Antonella Germanib, Guido Lombardoc, Paolo Eusebia, Rita De Floriod, Maria Laura Luchettae, Alfonso Iorio MDf, Alessandro Montedoria

a

Health Planning Service, Regional Health Authority of Umbria, Perugia, Italy Transfusion Medicine Service, Azienda Ospedaliera di Perugia, Perugia, Italy c Surgical and Biomedical Sciences, University of Perugia, Perugia, Italy d General Medicine, Azienda USL Umbria 1, Perugia, Italy e General Medicine, Azienda USL Umbria 2, Terni, Italy f Department of Clinical Epidemiology and Biostatistics, McMaster, Hamilton, ON, Canada

M AN U

AC C

EP

TE D

Correspondence to: Dr. Iosief Abraha Health Planning Service Regional Health Authority of Umbria Via Mario Angeloni, 61 06124 Perugia (Italy) tel +39 075 504 5251 cell. +39349 077 0910 fax +39 075 504 5569 e-mail: [email protected] [email protected]

SC

b

1

ACCEPTED MANUSCRIPT

Abstract Objectives To describe the characteristics, and estimate the incidence, of trials included in systematic reviews deviating from the intention-to-treat (ITT) principle. Study Design and Setting A 5% random sample of reviews (Medline 2006 to 2010). Trials from

RI PT

reviews were classified based on the intention-to-treat: 1) ITT trials (trials reporting standard ITT analyses); 2) mITT trials (modified ITT; trials deviating from standard ITT); or 3) no ITT trials. Results Of 222 reviews, 81 (36%) included at least one mITT trial. Reviews with mITT trials were

SC

more likely to contain trials that used placebo, that investigated drugs, and that reported favourable results.

M AN U

The incidence of reviews with mITT trial ranged from 29% (17/58) to 48% (23/48). Of the 2349 trials, 597 (25.4%) were classified as ITT trials, 323 (13.8%) as mITT trials and 1429 (60.8%) as no ITT trials. The mITT trials were more likely to have reported exclusions compared to

Conclusion

TE D

studies classified as ITT trials and to have received funding.

The reporting of the type of intention-to-treat may differ according to the clinical area and the type of intervention. Deviation from intention-to-treat in randomised controlled trials is a widespread

EP

phenomenon that significantly affects systematic reviews.

AC C

Keywords: deviation from intention-to-treat; modified intention-to-treat; bias; epidemiology; randomised trials; meta-analysis

2

ACCEPTED MANUSCRIPT

Introduction Intention-to-treat is the recommended standard approach to analyse data from randomised controlled trials. This method requires that patients are analysed according to their original random allocation to preserve the prognostic balance, thereby minimising selection bias and confounding

RI PT

[1-3].

Trials that use a so-called “modified intention-to-treat” approach are appearing in increasing numbers in the medical literature. The features of the modified intention-to-treat analysis are highly

SC

variable, with authors often using more than one criterion to describe this alternative approach[4]. For example, in a trial that evaluated the efficacy of topiramate, compared to placebo, to reduce the

M AN U

mean percentage change in body weight, the authors stated that their modified intention-to-treat population comprised “randomized individuals who had at least one dose of study medication, at least one post-baseline efficacy assessment, and the opportunity to complete at least 44 weeks of medication”. Despite the use of “modified intention-to-treat”, the analysis performed was

TE D

substantially a “per-protocol analysis” since the real reason for excluding 48% of the participants was withdrawal. [5]. In a trial where rofecoxib and celecoxib were compared to acetaminophen, to reduce pain in patients with osteoarthritis, the authors declared that the efficacy analysis was

EP

“conducted using a modified intent-to-treat approach, whereby all patients who took at least 1 dose

AC C

of study medication were included in the analysis”[6]. Carefully assessing this last paper, there is no apparent patient exclusion from analysis and the deviation from the intention-to-treat remains unclear.

The frequency of trials using a modified intention-to-treat approach (475 trials, mostly published between 2000 and 2006) may well have been underestimated, since the inquiry was limited to the trials that explicitly used the word “modified” in the description of the deviation from the intentionto-treat analysis [4]. Indeed, in a subsequent study [7], a new selection criterion was used which also included studies that deviated from an intention-to-treat analysis without specifically using the

3

ACCEPTED MANUSCRIPT word “modified” (e.g., a study that compared cyclooxygenase-2 inhibitors with acetaminophen declared its intention-to-treat population as “all patients who received at least 1 dose of assigned study medication”). This generated a higher than expected number of trials (32 trials that deviated from the intention-to-treat analysis, without using the word “modified” in addition to 24 trials that

RI PT

used the word “modified”), underlining that the trials that deviated from the intention-to-treat analysis should, to a large extent, be considered as modified intention-to-treat reporting trials. In that cross-sectional study[7], trials classified as modified intention-to-treat trials were more likely to

SC

report post-randomisation exclusions and to have received funding from for-profit enterprises than trials that reported standard intention-to-treat. This analysis was, however, limited to trials

M AN U

published in three general and three specialty journals that were more likely to publish trials with modified intention-to-treat approach. Thus, there is a need to understand how widespread the phenomenon of the deviation from intention-to-treat is in the medical literature especially within systematic reviews.

TE D

Systematic reviews are generally considered the best source of evidence for clinical decision making and are often used as a baseline tool for guideline developers. To be a reliable tool, reviews need to be well conducted, and any bias present in the included trials should be made explicit, in

EP

order not to influence the conclusions of the review. Hence, readers of systematic reviews should be aware of the characteristics and reporting of systematic reviews including the type and

AC C

characteristics of the trials included in these reviews[8]. In a meta-epidemiological evaluation [9], we have shown that the reported increase in frequency of the trials that deviated from intention-totreat, in the medical literature[4], influenced the estimate of the treatment effect reported in systematic reviews and meta-analyses. Using the initial sample of reviews for which we performed the meta-epidemiological evaluation[9], we performed a further investigation regarding the incidence of trials that deviated from intention-to-treat in systematic reviews, as well as how these reviews differ in terms of several characteristics and reporting from reviews that do not contain trials reported deviation from intention-to-treat. 4

ACCEPTED MANUSCRIPT The primary aim of this study was to assess the epidemiology of intention-to-treat reporting of randomised trials included in systematic reviews; to assess the incidence of trials that deviated from intention-to-treat in meta-analyses; to compare the difference in characteristics between systematic reviews, with at least one randomised trial that deviated from intention-to-treat included in meta-

RI PT

analyses, with systematic reviews with randomised trials that did not deviate; and to compare the difference in characteristics among trials based on intention-to-treat reporting.

We believe that the results of the present study will be important for various professionals engaged

SC

in the design, conduct and reporting of clinical trials and meta-analyses, as well as for those involved in guideline development and clinical decision making. Knowledge of potential bias

M AN U

associated with the type of ITT reporting, and of the incidence of trials that deviated from an ITT analysis, shall prompt clinical researchers to adopt adequate randomisation modalities, minimise attrition bias, as well as use adequate methods of data analysis and stimulate clinicians to carefully

AC C

EP

TE D

consider whether to adopt interventions supported solely by papers reporting a mITT approach.

5

ACCEPTED MANUSCRIPT

Material and Methods Data source and search strategy Our articles of interest were systematic reviews that were published between 2006 and 2010. To identify the reviews, we launched Montori’s highly specific search strategy in PubMed (MEDLINE

RI PT

[Title/Abstract] OR ((systematic[Title/Abstract] AND review[Title/Abstract] OR meta-

each year, for abstract screening. Study selection

M AN U

Selection of systematic reviews or meta-analyses

SC

analysis[Publication Type]))[10]. Subsequently, we randomly selected 5% of the records, from

We were interested only in reviews of therapeutic or preventive interventions that included at least two randomised trials. Diagnostic, prognostic, epidemiological, cost-effectiveness and animal studies were excluded. Network meta-analyses and reviews with cluster or crossover trials were

TE D

excluded. Pairs of investigators independently assessed relevant reviews for eligibility. If two investigators disagreed, they attempted to reach consensus by discussion. Where necessary, a third investigator was consulted.

EP

Data Extraction and Quality Assessment

AC C

Two separate data collection forms for systematic reviews and primary studies were developed. Systematic reviews or meta-analyses The following information was retrieved from each systematic review: the journal name, the publication year, the clinical area of interest, the type of the intervention in the experimental and control groups, the type of primary outcome, the funding source and author conflict of interest. In addition, the number of events and the population, the outcome measure of the primary outcome (dichotomous or continuous), the results of the meta-analysis with its method of pooling (random or

6

ACCEPTED MANUSCRIPT fixed effect model) were recorded. When necessary, to clarify uncertainties, authors of the systematic reviews were contacted. Randomised trials

RI PT

The following general characteristics were collected from each randomised trial: the journal name, the publication year, the type of the intervention, the number of patients included, together with the number of events, the funding source and author conflict of interest. Trials were subsequently recorded according to the type of interventions (pharmacological or non-pharmacological, such as

SC

the use of devices, surgery, technical procedures, behavioural interventions, and complementary

M AN U

and alternative medicine). Categorisation of intention-to-treat reporting

To calculate the incidence of systematic reviews that reported deviation from intention-to-treat and to assess the methodological and reporting differences, the following classification was performed

TE D

based on the type of the intention-to-treat reported in each included trial. This classification was used in our earlier assessments, which was independent of any reported post-randomisation exclusions that may have occurred in the trials [7, 9]

EP

a) ITT trials – were those trials that correctly defined the ITT principle or did not give any type of definition; if a trial did not use the phrase ‘intention-to-treat’, but the authors

AC C

expressed the intention of analysing the trial data according to the original patient allocation, then the trial was classified in this category; b) mITT trials – were those trials that deviated from the standard intention-to-treat or explicitly used the phrase ‘modified intention-to-treat’; based on the types of deviation proposed in our previous assessment [4, 7] we sub-classified the mITT trials as follows: (i) mITT trials with deviation from protocol (i.e., trials with treatment-related, baseline assessment-related or target condition-related deviations; (ii) mITT trials with missing data (i.e., trials with

7

ACCEPTED MANUSCRIPT baseline assessment-related deviations); and (iii) mITT trials with mixed components (i.e., trials that encompass deviation from protocol and missing data). c) no ITT trials – were those trials that did not refer to intention-to-treat or that did not fall into the previous two categories.

RI PT

Once the trials were categorised, the systematic reviews were also classified, respectively, as mITT reviews and no-mITT reviews, based on the presence or absence of at least one mITT trial. Funding Sources

SC

Information regarding the sources of funding for each trial was extracted from the systematic

M AN U

reviews and randomised trials by reviewing the article text, the conflict of interest section, information on funding (if present) and the acknowledgements section. Funding sources of trials were abstracted and categorised as follows: (1) unfunded studies or studies that received funding from not-for-profit institutions; (2) studies that were funded or co-funded by for-profit agencies;

Author Conflict of Interest

TE D

and (3) studies that did not contain any funding information.

In each included review and randomised trial, the presence or absence of author conflict of interest

EP

was determined by reviewing the authors’ institutional affiliations, the conflicts of interest section, information on funding (if present) and the acknowledgements section. The institutional affiliation

AC C

of the first author (e.g., university, not-for-profit institution or for-profit agency) was retrieved from each article. The affiliation of the other authors was also recorded when a financial tie with a forprofit agency was present. Subsequently, the studies were classified for analysis as follows: (1) studies with authors declaring no competing interests; (2) studies with authors declaring financial ties with the sponsor of the study, indicating that at least one author participated on behalf of a forprofit agency (e.g., as an employee or consultant) or any other financial ties disclosed, indicating that an author did not participate on behalf of the study sponsor, but did report a conflict of interest

8

ACCEPTED MANUSCRIPT such as a past financial tie with for-profit agency; and (3) studies that did not mention any conflict of interest. Statistical analysis

RI PT

Categorical covariates were described by frequency distribution, while continuous covariates were expressed in terms of their mean + standard deviation (SD) or median and interquartile range as appropriate.

The incidence of the trials categorised as mITT were calculated for each year, with 95% confidence

SC

intervals, and a χ2 test was calculated to evaluate the trend over time after inspection of a raw data

M AN U

plot. To evaluate the trend over time in the proportion of reviews that included at least one mITT trial, we used a χ2 test for linear trend with one degree of freedom.

In systematic reviews, we evaluated the association between the presence or absence of a mITT trial and the type of intervention (pharmacological, non-pharmacological), the results of the primary analysis (significant or not-significant at P=0.05), the use of placebo, the funding source, and the

TE D

presence of authors' conflicts of interest using logistic regression. These variables were also used to adjust for potential confounding variables

EP

In primary studies (randomised trials), we evaluated the association between type of intention-totreat analysis (as a dependent, multinomial 3-level variable: ITT, mITT, or no ITT) and the type of

AC C

intervention (pharmacological, non-pharmacological), the results of the primary analysis (significant or not-significant at P=0.05), the use of placebo, the funding source, and the presence of authors' conflicts of interest using multinomial logistic regression with an adjustment for the cluster effect of the systematic review. Results were considered statistically significant with a two-tailed P-value less than 0.05. The software STATA/SE, version 13 for Windows (StataCorp, College Station, Texas), was used for statistical analysis.

9

ACCEPTED MANUSCRIPT

Results Our electronic search in Medline identified 44,900 reviews published between 2006 and 2010 from which 2,245 abstracts (5%) were randomly selected and screened. We excluded 1,830 records based

RI PT

on the title and abstract. The resulting 415 full-text reviews were evaluated and, after excluding 193 articles with reasons, 222 reviews remained for further assessment. Of the 2,701 potentially eligible trials that were included in the 222 reviews, we were unable to obtain 26 full-texts(1%), and 326 studies were excluded with reasons. Overall, 2,349 randomised controlled trials remained for final

M AN U

reviews and trials are shown in the Supplemental file.

SC

evaluation. Figure 1 summarises the study screening process; the list of included and excluded

Incidence of randomised controlled trials that deviated from intention-to-treat in systematic reviews

The incidence of randomised controlled trials that deviated from intention-to-treat in systematic

TE D

reviews each year was 32.0% (16/50) in 2006, 38.7% (12/31) in 2007, 37.1% (13/35) in 2008, 47.9% (23/48) in 2009, and 29.3% (17/58) in 2010 (Pfor trend = 0.883). When we limited the calculation to the trials that evaluated pharmacological interventions, the incidence of reviews with

AC C

Systematic reviews

EP

at least one mITT trial rose to 47%.

Of the 222 included systematic reviews, 81 (36%) had at least one trial with mITT, whereas the remaining 141 (64%) trials did not contain a mITT trial. The two groups of reviews differed significantly in terms of several variables. The use of pharmacological interventions was higher in the reviews with mITT trials 74 (n = 74, 91%) than in the reviews without mITT trials (n = 83, 59%), P<0.001. Similarly, the use of placebo as a comparator was higher in the reviews with mITT trials than in the reviews without mITT trials (n=34, 42%) vs (n=34, 24%) respectively, P = 0.005. Despite the presence of few trials receiving funding from for-profit agencies in both groups of 10

ACCEPTED MANUSCRIPT reviews, there was a higher proportion of authors with a conflict of interest in the review with mITT trials compared to the reviews without mITT trials 25/81 (n=25, 31%) vs 20/141 (n= 20, 14%), P = 0.04. Table 1 displays the basic characteristics and differences between the two groups of reviews.

RI PT

Characteristics of Systematic Reviews and the Type of Intention-to-treat Reporting

Logistic regression analyses showed that reviews with mITT trials were more likely to use placebo as a comparator (OR 2.28, 95% CI: 1.27, 4.09) to evaluate a pharmacological intervention (OR

SC

10.26, 95% CI: 3.81, 27.59), and to report favourable results (OR 2.44, 95% CI: 1.27, 4.70). No association was found between the inclusion of mITT trials in a review and funding or authors’

M AN U

conflict of interest. Randomised trials

Of the 2,349 included trials, 597 (25.4%) were classified as ITT, 323 (13.8%) as mITT and 1,429 (60.8%) as no ITT trials. The number of trials with exclusions were higher in the mITT trials

TE D

(n=246, 76%) than in the ITT trials (n=252, 42%) and no ITT trials (n=672, 47%). Studies classified as mITT trials were more likely to be funded or to have authors with a conflict of interest

EP

that studies classified as ITT or no ITT trials. In addition, while mITT trials increased from 1.4% in 1970-1994 to 19.7% in 2005-2009, the no ITT trials decreased from 92.2% to 44.8% during the

AC C

same time periods. The characteristics of the included trials are described in Table 2. Characteristics of Randomised Trials and the Type of Intention-to-treat Reporting In the cluster-adjusted multivariate analysis, compared to studies classified as ITT trials, the mITT trials were more likely to report post-randomisation exclusions [OR 4.20 (95% CI: 3.07, 5.76), P < 0.001], to have received funding from a for-profit agency [OR 5.29 (95% CI: 2.72, 10.27), P < 0.001], and to have had at least one author with a conflict of interest [OR 3.96 (95% CI: 1.91, 8.19), P < 0.001]. In addition, mITT trials were more likely to be published after 1995 than ITT trials.

11

ACCEPTED MANUSCRIPT Interestingly, compared to ITT trials, studies classified as no ITT trials reported more postrandomisation exclusions [OR 1.34 (95% CI: 1.06 to 1.70), P = 0.01] and were less likely to be published in recent years.

RI PT

Subtypes of mITT trials based on the type of deviation. Based on the types of deviation, of the 323 mITT trials, 114 (35%) were classified as trials with deviation from protocol, 52 (16%) as trials with missing data, and 157 (49%) as trials with both

AC C

EP

TE D

M AN U

SC

types of components.

12

ACCEPTED MANUSCRIPT

Discussion Summary of findings After randomly selecting 5% of the intervention reviews published in Medline, we compared the characteristics of systematic reviews according to the type of intention-to-treat reported in the

RI PT

included randomised trials. We found that there was a substantial difference between the two groups of reviews (reviews with mITT trials and reviews without mITT trials). The clinical areas of most relevance for reviews with mITT trials were neuropsychiatric and rheumatic diseases, whereas

SC

gastrointestinal and gynaecological diseases were the areas of higher interest for reviews without mITT trials. Consequently, the type of intervention investigated differed. Most mITT trial

M AN U

containing reviews addressed pharmacological interventions (n=74, 91%) of which (n=23 31%) constituted neuropsychiatric drugs followed by monoclonal antibodies (n=10, 14%) and antiinfective agents (n=10, 14%). In addition, the use of placebo was significantly associated with mITT trial containing reviews indicating that deviation from intention-to-treat is a specific characteristic

TE D

of new drugs under investigation.

In systematic reviews, only a few studies received funding from for-profit agencies. However, a significant number of authors with a conflict of interest was found in the reviews with mITT trials.

EP

In the primary studies, both funding and authors’ conflict of interest were specific to the trials that

AC C

deviated from intention-to-treat.

The present study confirms the association between these characteristics, namely the use of pharmacological interventions as an experimental drug, use of placebo, funding from industry and the presence of authors’ conflict of interest, and the trials classified as mITT trials that was shown in our previous assessment [7]. In that study, however, the investigation was limited to three general medical journals and three specialty journals that were more likely to publish mITT trials[7].

13

ACCEPTED MANUSCRIPT The incidence of reviews with mITT trials Another objective of this study was to evaluate the incidence and characteristics of systematic reviews that included trials reporting deviation from the intention-to-treat principle with respect to all the reviews that assessed preventive and therapeutic interventions - published in the medical

RI PT

literature - irrespective of the type of target condition. We found that between 2006 and 2010, the incidence of systematic reviews of interventions with mITT trials was relatively high with an average of 36% (81/222), which remained constant across the five years. The incidence of mITT

SC

trials reached an average of 47% (47/157) when analysis was restricted to reviews that evaluated pharmacological interventions. These findings demonstrate that the phenomenon of deviation from

M AN U

intention-to-treat is as widespread as previously suspected [4].

The intention-to-treat reporting, deviation from protocol and missing data In the last 20 years, the variability in reporting and modification of the intention-to-treat analysis have been cause for debate and the subject of investigation in several studies [1, 11-17]. A recent

TE D

survey revealed the presence of variability in the description of the intention-to-treat method, even in papers of methodological studies[18]. This variability has been reported also in several

EP

investigations of published randomised trials [11, 12] including specific clinical areas[19]. In another study, Anna Døssing and colleagues evaluated 72 trials that tested biological or targeted

AC C

agents in patients with rheumatoid arthritis and identified a substantial proportion (51%) of trials that used the modified intention-to-treat approach[20]. According to the Cochrane handbook, in order to fulfil the principle of intention-to-treat, (a) participants in a trial need to be retained in the groups in which they were originally allocated, (b) the outcome data need to be measured on all participants, and (c) all participants need to be included in the analysis. Strict intention-to-treat analysis is often hard to achieve, either because of deviation from protocol, or because of missing data [21-23]. Deviation from protocol occurs, for example, when patients do 14

ACCEPTED MANUSCRIPT not take the desired treatment, when patients were mistakenly enrolled or when patients were enrolled before information on eligibility was obtained and were thus subsequently excluded, when patients developed the outcome of interest before commencing the assigned treatment. A crucial point in the case of deviation from protocol is whether it is legitimate to exclude patients and

RI PT

whether such exclusions may compromise the results. Ferguson et al[2] suggested that exclusions of mistakenly, or prematurely randomised patients, or of patients who did not receive the intended treatment, can be legitimate. In our assessment, we performed a second classification of the mITT

SC

trials based on the type of deviation. Thirty-five percent of the mITT trials reported descriptions that were related to deviation from protocol and another 49% reported both types of descriptions.

M AN U

Whatever the definition used in the intention-to-treat approach, it is clear that the analysis performed in the mITT trials is a “a per-protocol analysis”. It is important to underline that the authors of mITT studies inappropriately used the term intention-to-treat, because in reality, the analyses they performed were substantially “per-protocol analyses”, which can confuse the reader.

TE D

In some mITT trials, further confusion is generated, when in addition to providing an intention-totreat analysis with a deviation, authors provided a second analysis, this time defined as a “perprotocol” analysis. For example, in a randomised trial that assessed the efficacy, safety, and

EP

tolerability of caspofungin with respect to amphotericin B, the authors reported the main analysis as a “modified ITT (mITT) analysis population” as a “population that presented with a surgically

AC C

proven complicated intra-abdominal infection”. The authors also proposed a secondary analysis, this time defined as a “per protocol (PP) population included all patients in the mITT population with no major protocol violation”[24]. This raises concerns with the inconsistency or lack of clarity in the proposed descriptions of the type of analysis employed. Trial authors should be warned that a “per-protocol analysis” renders a trial to become a non-randomised, observational comparison and may introduce bias[25].

15

ACCEPTED MANUSCRIPT Missing data is a common problem in clinical trials[11], which affects as much as 89% of the trials published in highly cited general medical journals[22, 26]. Other reports show that at least in half of the trials, more than 10% of randomised patients may have missing outcomes, exposing the trials to bias and/or reducing their power and precision[21]. In the present study we found that of the 323

RI PT

mITT trials, the deviation from ITT was related to missing data in at least 65% of the trials. Whatever the description provided to report the missing outcome data, the performance of

intention-to-treat is impractical and the appropriate term of the analysis should “available cases”.

SC

Methodologists suggest different approaches to handle missing data, such as the use of complete case analysis in which participants with missing data are simply excluded, the use of simple

M AN U

imputation methods in which a single value is provided in place of a missing data or the use of estimated-equation methods[23]. Others have proposed the use of multiple imputation as a general approach to deal with incomplete data[27], a method which could conform to an intention-to-treat approach[28]. The CONSORT group strongly advocates the use of sensitivity analyses, although

TE D

pitfalls and limitations must be accounted for[28]. However, the best method to handle missing data remains prevention. Little and colleagues provide interesting suggestions on how to limit missing data, including the provision of incentives to participants to remain in the study, the conduct of a

EP

run-in period that serves to identify patients with the best compliance and good treatment tolerance, shortening the follow-up period for the primary outcome, consideration of a randomised withdrawal

AC C

design in the case of long term efficacy assessment, and anticipating and avoiding outcome measures that are likely to lead to significant missing data[23]. While the exclusion of a participant subsequently found not to have the eligibility criteria is clear cut, there can be various reasons for participants not to have received the intended treatment (at least one dose of drug). For example, a participant may withdraw consent, or may develop the outcome of interest, or an event may occur, that hinders him/her from starting the treatment, or the participant can be lost to follow-up. The latter situation can be a case of missing data, which may

16

ACCEPTED MANUSCRIPT introduce a misclassification in a hypothetical assumption. Thus, authors should provide the exact reason for which the participant did not start or complete the intended treatment. Positive findings

RI PT

Another important point that emerged from the present assessment is that statistically significant findings were more preponderant in the reviews with mITT trials (n=60, 74%) than in the reviews without mITT trials (n=79, 56%). This effect seems to be driven by the presence of the mITT trials. The preponderance of significant findings in mITT trials has previously been observed[4] and our

SC

recent meta-epidemiological study showed that deviation from intention-to-treat affects the estimate

M AN U

of the treatment effect[9]. The meta-epidemiological study, conducted by Anna Døssing and colleagues, of trials regarding rheumatoid arthritis did not find any bias related to the use of mITT. This difference can be due to the methodological approach. While in our assessment, the categorisation was based on the intention-to-treat reporting without taking into account the occurrence of post-randomisation exclusions[7, 9, 29], in Døssing’s assessment, trial classification

TE D

was based on the occurrence of post-randomisation exclusions independently of the reporting of the intention-to-treat. Our rational was to provide an alternative approach to that based on exclusions,

EP

given that trial authors may exclude patients without clearly reporting those exclusions[30]. Since ITT trials were more conservative, compared to mITT trials, it is possible to hypothesise that

AC C

exclusions may have a different impact on the treatment effect[9]. Further research is needed to confirm this hypothesis.

Strengths and limitations

The strength of our investigation includes the use of a systematic approach to identify intervention reviews in Medline published during five consecutive years. In addition, we acknowledge the use of pilot tested forms with specific instructions, the use of detailed and explicit eligibility criteria, and duplicate and independent processes for study selection, data abstraction, and data interpretation.

17

ACCEPTED MANUSCRIPT Our study has several limitations. First, our classification of trials was based on the authors’ reporting of intention-to-treat and, in particular, the categorisation of ITT trials was independent of the presence or absence of post-randomisation exclusions. Therefore, it is not possible to claim that all the ITT trials actually followed a strict intention-to-treat analysis. In addition, mITT trials have a

RI PT

varying and unpredictable description, and their categorisation was again based on the reported description. Hence, the intention-to-treat reporting on which we provided our classification is specific to this study. Second, we did not address the issue of methodological quality of either the

SC

systematic reviews or randomised controlled trials. Systematic differences between trials adopting the intention-to-treat, modified intention-to-treat and no intention-to-treat approaches, described in

M AN U

this manuscript, could be affected by attrition bias of the ITT studies. Indeed, the main focus of the present paper was to investigate the epidemiology of mITT trials, while the bias originating from the mITT approach, was documented in a previous paper [9]”. Another important aspect not addressed in this paper, and potentially object of a future work, is exploring the impact of different

TE D

scenarios and approaches[31].

Third, the reviews we included could be outdated given that the time frame was between 2006 and

EP

2010. However, the case of deviation from intention-to-treat in randomised trial is still a contemporary issue given that the incidence of these trials is still increasing as reported in the last

AC C

Cochrane Colloquium[32]. Conclusions

In this five-year sample of reviews, we showed that the reporting of the type of intention-to-treat may differ according to the clinical area and the type of intervention. The use of a modified intention-to-treat approach, or deviation from intention-to-treat, in randomised controlled trials, is a widespread phenomenon that significantly affects systematic reviews. Such trials are specific to new experimental drugs, are characterised by post-randomisation exclusions and significant results and are dominated by industrial funding. 18

ACCEPTED MANUSCRIPT Acknowledgement We thank Joseph M Rimland for editing and commenting the manuscript Author Contributions: IA and AM conceived and designed the study; IA, RDF, MLL, MO, FC and AG were involved in

RI PT

the data acquisition; IA, AM, AI, MM, GL and PE analysed and interpreted the data. IA, AM, RDF, MLL, MO, FC, AG, AI, MM, GL and PE contributed in the drafting and revising the paper and have approved submission of the final version of the manuscript.

SC

Funding

The study was funded by the Ministry of Health, Italy (RF-2009-1549561). The study sponsor had

M AN U

no role in the in study design; in the collection, analysis and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

Data Sharing Statement

AC C

None.

EP

Competing interests

TE D

No additional data available.

19

ACCEPTED MANUSCRIPT Title of figure and tables Figure 1. Study screening process Table 1. Characteristics of Systematic Reviews

AC C

EP

TE D

M AN U

SC

RI PT

Table 2. Characteristics of Randomized Controlled Trials included in Systematic Review

20

ACCEPTED MANUSCRIPT

References

7.

8. 9. 10. 11.

12. 13. 14.

15. 16.

17. 18.

19.

20. 21.

RI PT

6.

SC

5.

M AN U

4.

TE D

3.

EP

2.

Newell, D.J., Intention-to-Treat Analysis: Implications for Quantitative and Qualitative Research. Int. J. Epidemiol., 1992. 21(5): p. 837-841. Fergusson, D., et al., Post-randomisation exclusions: the intention to treat principle and excluding patients from analysis. BMJ, 2002. 325(7365): p. 652-4. Lewis, J.A. and D. Machin, Intention to treat--who should use ITT? Br J Cancer, 1993. 68(4): p. 647-50. Abraha and Montedori, Modified Intention to Treat Reporting in Randomised Controlled Trials: A Systematic Review. BMJ, 2010. 340: p. c2697. Astrup, A., et al., Topiramate: Long-Term Maintenance of Weight Loss Induced by a LowCalorie Diet in Obese Subjects. Obesity Research, 2004. 12(10): p. 1658-1669. Geba, G.P., et al., Efficacy of rofecoxib, celecoxib, and acetaminophen in osteoarthritis of the knee: a randomized trial. JAMA, 2002. 287(1): p. 64-71. Montedori, A., et al., Modified versus standard intention-to-treat reporting: are there differences in methodological quality, sponsorship, and findings in randomized trials? A cross-sectional study. Trials, 2011. 12: p. 58. Moher, D., et al., Epidemiology and reporting characteristics of systematic reviews. PLoS Med, 2007. 4(3): p. e78. Abraha, I., et al., Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study. BMJ, 2015. 350: p. h2445. Montori, V.M., et al., Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ, 2005. 330(7482): p. 68. Gravel, J., L. Opatrny, and S. Shapiro, The intention-to-treat approach in randomized controlled trials: are authors saying what they do and doing what they say? Clin Trials, 2007. 4(4): p. 350-6. Hollis, S. and F. Campbell, What is meant by intention to treat analysis? Survey of published randomised controlled trials. BMJ, 1999. 319(7211): p. 670-674. Kruse, R.L., et al., Intention-to-treat analysis: who is in? Who is out? J Fam Pract, 2002. 51(11): p. 969-71. Baron, G., et al., Violation of the intent-to-treat principle and rate of missing data in superiority trials assessing structural outcomes in rheumatic diseases. Arthritis Rheum, 2005. 52(6): p. 1858-65. Porta, N., C. Bonet, and E. Cobo, Discordance between reported intention-to-treat and per protocol analyses. J Clin Epidemiol, 2007. 60(7): p. 663-9. Miladinovic, B., et al., Instrumental variable meta-analysis of individual patient data: application to adjust for treatment non-compliance. BMC Med Res Methodol, 2011. 11: p. 55. Shrier, I., et al., Beyond intention to treat: what is the right question? Clin Trials, 2014. 11(1): p. 28-37. Alshurafa, M., et al., Inconsistent definitions for intention-to-treat in relation to missing outcome data: systematic review of the methods literature. PLoS One, 2012. 7(11): p. e49163. Joseph, R., et al., A systematic review finds variable use of the intention-to-treat principle in musculoskeletal randomized controlled trials with missing data. J Clin Epidemiol, 2015. 68(1): p. 15-24. Dossing, A., et al., Modified intention-to-treat analysis did not bias trial results. Journal of Clinical Epidemiology. Altman, D.G., Missing outcomes in randomized trials: addressing the dilemma. Open Med, 2009. 3(2): p. e51-3.

AC C

1.

21

ACCEPTED MANUSCRIPT

28. 29. 30.

31.

32.

RI PT

27.

SC

26.

M AN U

25.

TE D

24.

EP

23.

Wood, A.M., I.R. White, and S.G. Thompson, Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials, 2004. 1(4): p. 368-76. Little, R.J., et al., The prevention and treatment of missing data in clinical trials. N Engl J Med, 2012. 367(14): p. 1355-60. Villanueva, A., et al., A randomized double-blind study of caspofungin versus amphotericin for the treatment of candidal esophagitis. Clin Infect Dis, 2001. 33(9): p. 1529-35. Moher, D., et al., CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. Vol. 340. 2010. Bell, M., et al., Handling missing data in RCTs; a review of the top medical journals. BMC Medical Research Methodology, 2014. 14(1): p. 118. Sterne, J.A., et al., Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ, 2009. 338: p. b2393. Moher, D., et al., CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ, 2010. 340: p. c869. Abraha and Montedori, Modified Intention to Treat Reporting in Randomised Controlled Trials: A Systematic Review. BMJ, 2010: p. in press. Schulz, K.F., et al., Blinding and exclusions after allocation in randomised controlled trials: survey of published parallel group trials in obstetrics and gynaecology. BMJ, 1996. 312(7033): p. 742-744. Akl, E.A., et al., Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study. Syst Rev, 2014. 3: p. 137. Orso. M, et al. Incidence of randomised trials with modified intention-to-treat analysis in the medical literature: a systematic review update. Cochrane Database of Systematic Reviews, Supplement, 2015. Suppl.

AC C

22.

22

ACCEPTED MANUSCRIPT

Table 1. Characteristics of Systematic Reviews Review without mITT trial

81

141

11 (7-18)

7 (5-13)

2261 (1224-5418)

RI PT

Included systematic reviews Trials included in systematic reviews, median (IQR) Patients included, median (IQR) Clinical areas of interest, n (%)

Review with at least one mITT trial

921 (476-1740)

− Gastrointestinal − Gynobs − Onco-hematology − Cardiovascular − Infection − Neuro-psychiatric − Metabolic diseases (7) − Others (34)

M AN U

SC

− Neuro-psychiatric 19 (23) − Rheumatic disease 9 (11) − Onco-hematology 8 (10) − Cardiovascular 7 (9) − Infection 7 (9) − Gastrointestinal 6 (7) − Metabolic diseases 6 (7) − Others 19 (24)

Pharmacological Intervention, n (%) Flow-chart, n (%) Favourable results, n (%)

Not funded, For-profit agency

AC C

Not reported

EP

Funding, n (%)

34 (42)

34 (24)

74 (91)

83 (59)

TE D

Placebo use, n (%)

32 (40)

57 (41)

60 (74)

79 (56)

35 (43)

58 (41)

11 (14)

9 (6)

35 (43)

74 (52)

24 (30)

49 (35)

25 (31)

20 (14)

32 (40)

72 (51)

17 (12) 14 (10) 14 (10) 13 (9) 12 (9) 12 (9) 10 49

Author’s conflict of interest, n (%) Author declaring no competing interest Author on behalf of a forprofit agency Competing interest not reported IQR, interquartile range;

1

ACCEPTED MANUSCRIPT

Table 2. Characteristics of Randomised Controlled Trials included in Systematic Review mITT (n=323)

no ITT (n=1429)

198 (90-400)

290 (150-476)

70 (38-142)

1970 – 1994

31 (6.4)

7 (1.4)

1995 – 2000

139 (23.9)

77 (13.2)

366 (62.9)

2001 – 2004

205 (31.3)

116 (17.7)

335 (51.1)

2005 - 2009

222 (35.6)

123 (19.7)

Post-randomisation exclusion Number of studies with exclusions, n (%) Number of exclusions, median (IQR)

252 (42.2)

246 (76.2)

672 (47.0)

8 (3-19)

9 (3-21)

6 (3-15)

Not-for-profit organization

168 (28.1)

19 (5.9)

316 (22.1)

For-profit agency

291 (48.7)

244 (75.5)

390 (27.3)

138 (23.1)

60 (18.5)

723 (50.6)

72 (12)

12 (3.7)

75 (5.2)

165 (27.6)

168 (52)

130 (9.1)

360 (60.3)

143 (44.3)

1224 (85.6)

Patients included, median (IQR)

M AN U

TE D

Funding, n (%)

Not reported

AC C

EP

Author’s conflict of interest, n (%) Author declaring no competing interest Author on behalf of a forprofit agency Competing interest not reported

SC

Year of publication, n (%)§

RI PT

ITT (n=597)

448 (92.2)

280 (44.8)

§ Percentages calculation based on the total number of publication in each period of time (row percentage)

2

AC C

EP

TE D

M AN U

SC

RI PT

ACCEPTED MANUSCRIPT