- Email: [email protected]
An observational study of missing data inquiry from randomized trial authors showed a poor response
Journal Pre-proof An observational study of missing data enquiry from randomised trial authors showed a poor response Sofia Tsokani, MSc, Dimitrios Mavridis, MSc, PhD, Shahab Hadjibandeh, MD, Shahin Hadjibandeh, MD, George A. Antoniou, MD MSc PhD FEBVS, Alexandros Andreou, MD, FEBS, Stavros A. Antoniou, MD PhD FEBS PII:
S0895-4356(19)30643-2
DOI:
https://doi.org/10.1016/j.jclinepi.2019.11.014
Reference:
JCE 10020
To appear in:
Journal of Clinical Epidemiology
Received Date: 22 July 2019 Revised Date:
12 November 2019
Accepted Date: 20 November 2019
Please cite this article as: Tsokani S, Mavridis D, Hadjibandeh S, Hadjibandeh S, Antoniou GA, Andreou A, Antoniou SA, An observational study of missing data enquiry from randomised trial authors showed a poor response, Journal of Clinical Epidemiology (2019), doi: https://doi.org/10.1016/ j.jclinepi.2019.11.014. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Elsevier Inc. All rights reserved.
Original study
An observational study of missing data enquiry from randomised trial authors showed a poor response
Running title: Information supplementation by original authors for meta-analysis
Authors 1
Sofia Tsokani, MSc
2,3
Dimitrios Mavridis, MSc, PhD
4
Shahab Hadjibandeh, MD
5
Shahin Hadjibandeh, MD
6
George A. Antoniou, MD MSc PhD FEBVS
7
Alexandros Andreou, MD, FEBS
8
Stavros A. Antoniou, MD PhD FEBS
Affiliations 1. Department of Mathematics, School of Sciences, University of Ioannina, Ioannina, Greece 2. Department of Primary Education, School of Education, University of Ioannina, Ioannina, Greece 3. Paris Descartes University, Sorbonne Paris Cité, Faculté de Médecine, Paris, France 4. Department of General Surgery, Royal Bolton Hospital, Bolton, UK 5. Department of General Surgery, Stepping Hill Hospital, Stockport, UK 6. Department of Vascular and Endovascular Surgery, The Royal Oldham Hospital, Pennine Acute Hospitals NHS Trust, Manchester, United Kingdom 7. Upper GI Department, Castle Hill Hospital, Hull and East Yorkshire Hospitals NHS Foundation Trust, Hull, UK 8. School of Medicine, European University Cyprus, Nicosia, Cyprus
Conflict of interest: none Funding: none 1
ABSTRACT Background The quality of evidence from a meta-analysis might be affected by poor or inadequate reporting of individual randomized controlled trials (RCTs). The results of data request from authors has not been investigated to date. Methods After completion of a feasibility trial, the authors of 116 RCTs selected for inclusion in 4 meta-analyses and network meta-analyses were contacted via email to check abstracted data from their studies and/or provide missing data. Univariable and multivariable analyses were performed to investigate the association of response rate with predefined characteristics, including time interval between publication of the RCT and the date of contact, role of the contact author as corresponding author, the number of participating centers, the source of contact email address and risk of bias assessments. Results The response rate to the primary email invitation was 31.1% (median response time 0 days, interquartile range (IQR) 0-1). Out of the authors who did not respond to the primary invitation, 31.2% responded to the reminder (median response time 1 day, IQR 0-3). Despite an overall response rate of 41%, both confirmation of data accuracy and supplementation of missing data were provided for 9 RCTs only (7.8%). No association was found between lack of response and predefined characteristics on univariable and multivariable analyses. Conclusion Despite a fair response rate, contacting authors of RCTs to check abstracted data and complete missing items yielded a very low rate of task completion. These findings highlight the importance of endorsing a data sharing culture among researchers. The role of alternative incentives is yet to be investigated. Keywords: randomized controlled trial, meta-analysis, data acquisition, data sharing 2
INTRODUCTION Randomized controlled trials (RCTs), when properly planned and conducted, are considered to provide the highest quality evidence to inform clinical practice and decision making amongst clinical trials.1,2 Participant randomization is the optimum method to control for known and unknown effect modifiers and confounders.3 For instance, a case-matched cohort study on laparoscopic versus open groin hernia repair might control for confounders which investigators are aware of, such as patient age, comorbidities or hernia size, but other factors such as systemic connective tissue disorder or hiatal hernia,4,5 whose confounding or modifying effect may be as important, as well as other unknown confounders, remain uncontrolled. Furthermore, blinding of participants, personnel and outcome assessors aims to reduce several sources of bias, which may affect certainty on evidence in observational studies.6 Meta-analysis of RCTs has the potential to increase precision of effect sizes, assess heterogeneity (differences in true effects across trials) and small study effects (systematic differences in effect sizes across small and large trials), explore subgroup differences and/or analyze trends over time.7-10 Nevertheless, the evidence provided by meta-analyses is based on the quality of the included RCTs. Indeed, the GRADE approach for evidence appraisal suggests that high or unknown risk of bias across RCTs may require downgrading or even double-downgrading the quality of evidence.11 Poor reporting of RCTs, however, does not necessarily mean poor conduct.12 Transparency in trial reporting is a low-hanging fruit in clinical research.13 Poor reporting of methodological and outcome parameters undoubtedly has a negative impact on the quality of meta-analyses. Organizations such as Cochrane and the National Institute for Health and Care Excellence (NICE) suggest that, in the absence of adequate and valid data, evidence synthesis might not be appropriate,14,15 thus leaving a large evidence base unexploited. In order to ensure reliability of the data collection process, the PRISMA (Preferred Reporting 3
Items for Systematic Reviews and Meta-analyses) initiative recommends for evidence synthesis researchers to obtain and confirm data from authors of RCTs.16 In our experience, authors’ feedback to this quest is poor. Aims of this prospective observational study were to quantify the response of authors of RCTs to the request for confirmation of original investigators’ data and provide missing information; to evaluate the adequacy of information provided; and to investigate whether predefined factors are associated with response rate. The objective was to inform communication practices and propose actions for improvement of data retrieval from RCTs for the purposes of evidence synthesis.
METHODS Protocol The study protocol was reviewed and approved by all authors. A repository for bibliometric studies does not exist, so the protocol was maintained by the scientific committee of the University Hospital of Heraklion. Following commencement of the study, there were no amendments and no deviations from the study protocol. Institutional review board approval or written consent was not required as no interventions on subjects were undertaken. Feasibility study Prior to embarking on this work, we performed a feasibility study with the objective to refine the methodology and adjust the data collection form. In the context of two meta-analyses conducted by our evidence synthesis group, we contacted the corresponding authors of 33 RCTs included in these meta-analyses through email to confirm data/provide missing information on their trials.17,18 We documented the response times and the information provided by original authors to define the time interval between the primary invitation and the reminder email, and to optimize the data collection form.
4
Criteria for inclusion of RCTs in the present study The first 100 eligible surgical RCTs for inclusion in systematic reviews performed by our evidence synthesis group were planned to be considered. No language restrictions were applied. The protocols of these reviews were submitted for registration in the International Prospective Register of Systematic Reviews (PROSPERO). The studies were selected upon second level screening of records identified in the study screening process, i.e. after title and abstract screening and assessment of full text reports of potentially eligible studies as per standard guidelines.19 Criteria for exclusion of RCTs from the present study RCTs identified through protocols of intervention systematic reviews not eligible for registration in the PROSPERO database and Cochrane reviews were excluded. Νo further exclusion criteria were applied. Data collection Data from RCTs were collected by (SH, SH, or AA) author and independently cross-checked by a second author (SH, SH, or AA). A third author (SAA) acted as an arbitrator in case of disagreement. Data items were pre-specified in study-specific protocols, which were submitted for registration in the PROSPERO database.20-23 These items included (i) studyrelated information (name of the primary author, name and contact details of the corresponding author, country of the primary institution, number of participating institutions), (ii) clinical/demographic information of the study population, (iii) outcome data, and (iv) risk of bias assessment data using the Cochrane Risk of Bias Tool for Randomized Trials. For each review, ad hoc Microsoft Excel spreadsheets based on the Cochrane Consumers and Communication Review Group’s data extraction template were developed, which were pilot-tested on the three most recent studies and refined accordingly.
5
Contact information of the corresponding author of each trial was recorded. After completion of data collection, the corresponding author of each trial was contacted via email by the senior author, using a standard invitation letter (Appendix), and was asked to check the abstracted data and provide missing items (predefined items to be collected, including patient demographics, inclusion/exclusion criteria, outcome data and risk of bias parameters). The spreadsheet containing abstracted data on their study was sent as attachment to the email and embedded in the body of the email. The date of invitation and the date of response, if any, were recorded. If the author did not respond after 14 days, a reminder email was sent (Appendix). The completeness of data provided by the authors was assessed and the date of response, if any, was recorded. Data analysis Data are presented using descriptive statistics. Categorical variables are presented as % percentage; continuous variables are presented as median, range and IQR range. Associations between author response and independent variables were explored using Pearson’s chi-square of independence for categorical data and t-tests for continuous data. The following independent variables were considered: 1. Time interval between publication of the RCT and email invitation (more than five versus less than five years; 0 = 0 to 5 years, 1 = 6 or more years) 2. Number of institutions (one versus more than one; 0 = 1 institution, 1 = 2 or more institutions) 3. The corresponding author is the first author (yes versus no; 1 = yes, 0 = no) 4. The source of the email address was the article versus any other source (0 = other, 1 = the article) 5. Risk of bias in the domain ‘selective reporting’ was considered high (yes versus no, 1 = yes, 0 = no)
6
6. Risk of bias was considered high in more than three risk of bias domains (1 = yes, 0 = no). The ‘author’s response’ variable is binary (0/1), with value 0 implying that the author did not respond to our request and value 1 otherwise. We applied logistic regression to explore potential associations between author response and the aforementioned characteristics. In total, there are six explanatory variables. We applied univariable logistic regression to investigate a potential association between authors’ response and each of these characteristics (‘time interval between publication of the RCT and email invitation’, ‘number of institutions’, ‘corresponding author is first author’, ‘source of email address’, ‘selective reporting’, ‘high or unclear risk of bias in more than 3 risk of bias components’. In addition, a multivariable logistic regression model was fitted to the data. Multivariable logistic regression analysis is an extension of bivariable logistic regression in which two or more independent variables are taken into consideration simultaneously to predict a dependent variable for each subject.24 The multivariable model fitted in this study used all six variables as predictors. The logistic regression analyses were carried out by the Logistic procedure in RStudio Version 1.1.423 (2009-2018 RStudio, Inc) in the Windows 10 environment, using the glm2 R-package.
RESULTS The overall response rate of the feasibility trial was 11 out of 33 (33%) and the median time from first contact to return of reports was 7 days (range 0-35, interquartile range, IQR 4-16). A total of 116 RCTs contributing to 4 systematic reviews/meta-analyses in the field of general and laparoscopic surgery were included: 3 network meta-analyses on the management of complicated diverticulitis,22
laparoscopic antireflux surgery23 and
laparoscopic appendectomy,25 respectively; and a meta-analysis and trial sequential analysis 7
on the risk of trocar site hernia after single-incision laparoscopic surgery.26 Sixteen (13.8%) were published between 1992-1999, 40 (34.5%) between 2000-2010 and 60 (51.7%) between 2011-2017. The years lapsed between publication and date of contact ranged from 1 to 24 years (median 6 years, IQR 4-13). The number of institutions contributing to the trials ranged from 1 to 20 (median 1 center, IQR 1-1). The first author was corresponding author in 61 articles (57.0%). Contact information could not be located for 7 corresponding authors (6.0%). The source of contact information was the published article (n=94, 86.2%), other publications of the same author (n=12, 11.0%) and the Web (n=3, 2.8%). Seven email addresses were not valid (6.4%) and 11 email messages were undeliverable (10.1%). Thirty six RCTs (33.6%) were judged to be at high or unclear risk of bias in the domain ‘selective reporting’ and 80 RCTs (74.8%) were judged to be at high or unclear risk of bias in more than 3 risk of bias domains. Twenty-eight authors responded to the primary invitation (response rate 31.1%, median response time 0 days, IQR 0-1, range 0-8). Out of the authors who responded, 19 provided some information (67.9%). Three authors reported that they were no longer staff at the institution where the work was conducted and one author had retired. The remaining 9 authors who responded but did not provide the requested information were sent a reminder. Six of those authors provided some information. Overall, out of the authors who responded to the primary email or the reminder email (n=47), 36 provided some information (76.6%). Typically a confirmation that the provided data were accurate, but missing data were mostly not addressed. Out of authors who did not respond to the primary invitation, 19 responded to the reminder email invitation (response rate 31.2%, median response time 1 day, IQR 0-3, range 0-5) and 17 provided information (27.9%).
8
Overall, both confirmation of data accuracy and missing data (if any) were provided for 9 RCTs (7.8%) (Fig. 1). None of the investigated characteristics were found to be associated with lack of response on univariable and multivariable analyses, albeit interval estimates were generally wide and a type II error cannot be excluded (Table 1).
DISCUSSION In this prospective observational study, we found that contacting authors of RCTs to confirm data and complete missing information for their trials resulted in a very low response rate. More importantly, obtaining requested information was achieved for as low as 7.8% of trials. In the majority of trial reports, contact information of the corresponding author was readily available in the article, however no bibliometric characteristics were found to be associated with author response. Nevertheless, this finding may be due to the small sample size; a true association might be present but not identifiable, thus warranting further research. The observation that 41% of authors responded to the first or second invitation and 39% eventually provided some kind of –usually incomplete– information, suggests that time constraints might be an impediment to effectively respond to such request. A second reminder might have resulted in obtaining further data, although lack of response in two invitations may suggest that this possibility is low. A minority of authors reported objective limitations to provide data, including change in working place and retirement. Interestingly, an author considered our communication as a spam and provided the requested data following cross-checking our contact email after a period of 2 months. In the era of a plethora of communication pathways, including email, social media and researcher communication platforms, one would expect that author feedback for research purposes would be more effective. In view of the results of the present analysis, what is considered as state-of-the-art methodology to acquire missing information from original investigators seems suboptimal. 9
Providing incentives to original investigators to confirm accuracy and supplement missing data for evidence synthesis purposes might be reasonable. The authors of systematic reviews may consider offering collaboration and co-authorship to original investigators. In our experience, such an approach results in a better insight to original data and a multifaceted view of the problem under investigation.27-30 Transparent collaboration of review authors with original investigators might decrease communication barriers and increase the quality of the meta-analysis. Research on obtaining data for the purposes of meta-analyses has focused on individual patient data meta-analyses. A systematic review found that only 61% of datasets included in 122 individual patient data meta-analyses were obtained.31 Financial incentives could probably drive increased contribution by original investigators, but this would require (additional) funding for systematic reviews. A recent RCT suggested that providing a financial incentive to authors of RCTs may not result in a higher rate of individual patient data sharing.32 In another RCT, authors who received a data sharing agreement were more likely to provide their datasets for the purposes of an individual patient data meta-analysis.33 Communication through an institutional email might decrease any doubts about the purposes of the contact. Providing the review registration number and/or the link to the study protocol (so the authors could affirm the originality of the work), might further increase the response rate. There is increasing awareness about data sharing in the research community. The International Committee of Medical Journal Editors (ICMJE) requires that trial authors contain a data sharing statement in their reports, declaring whether trial participant data are available, for how long and for which purposes.34 This policy has been embraced by major funders, such as the Wellcome Trust and the Medical Research Council.35-37 Data repositories dedicated to this policy are now publicly available. 10
The research community is however still far from wide implementation of an obligatory data sharing policy. Scientific data sharing platforms have only recently been launched and only a few (mostly industry-sponsored) clinical trials have agreed to provide data. Anecdotal data suggest that obtaining trial data through platforms is a time-consuming process.38 Due to this early stage of development of such platforms, their use for the purposes of systematic reviews is probably not yet an effective approach and might introduce study selection (publication) bias. It is however not difficult to envision a future with obligatory data sharing for trials submitted to peer-reviewed journals. A more effective approach might be the request of data through data sharing platforms. Indeed, some platforms already provide this option to researchers, which may be more effective and reliable than contacting original investigators directly.39-42 Original investigators may be less reluctant to provide patient-level data to trusted authorities, rather than to other researchers. A bundle of actions that might increase data acquisition from original investigators are summarized in Table 2. More importantly, the development of a data-sharing culture is imperative, to allow full exploit of trusted, deidentified patient data in meta-analyses and individual patient data meta-analyses. The practice of data sharing would have the potential to produce more precise effect estimates in meta-analysis and possibly provide different results. The incentive to share data is rather an ethical obligation to the research community, which calls for transparency, validation of outcomes and re-use of research data. The present work has some limitations that need to be taken into account when interpreting results. The studies included herein deal with general surgery RCTs, which might limit its external validity to this field. The low number of responses has reduced the power of exploratory analyses, therefore a type II error cannot be excluded. Our evidence synthesis group has undertaken a significant number of meta-analyses, its members might therefore be recognizable, hence response rates might be inflated. 11
CONCLUSION Data confirmation and completion of missing items by original investigators was achieved in only 7.8% of requests in a select cohort of surgical trials authors. A culture of data sharing is necessary to fully exploit clinical trial data in evidence synthesis projects.
12
REFERENCES 1.
OCEBM Levels of Evidence Working Group, Durieux N, Pasleau F, Howick J. The Oxford 2011 Levels of Evidence. Group. 2011;1(version):5653. http://www.cebm.net/index.aspx?o=1025.
2.
Guyatt GH. GRADE : an emerging consensus on rating quality of evidence and strength of recommendations. 2008;336(April).
3.
Jager K, Zoccali C, MacLeod A, Dekker F. Confounding: What it is and how to deal with it. Kidney Int. 2008;73(3):256-260. doi:10.1038/sj.ki.5002650
4.
Antoniou SA, Antoniou GA, Granderath FA, Simopoulos C. The role of matrix metalloproteinases in the pathogenesis of abdominal wall hernias. Eur J Clin Invest. 2009;39(11). doi:10.1111/j.1365-2362.2009.02199.x
5.
Antoniou GA, Tentes IK, Antoniou SA, Simopoulos C, Lazarides MK. Matrix metalloproteinase imbalance in inguinal hernia formation. J Investig Surg. 2011;24(4). doi:10.3109/08941939.2011.558610
6.
Schünemann HJ, Brozek J, Guyatt GH, Oxman AD. 5.2 Factors that can reduce the quality of the evidence. GRADE Handbook. https://gdt.gradepro.org/app/handbook/handbook.html#h.m9385o5z3li7. Published 2013.
7.
Egger M, Smith GD, Phillips AN. Meta-analysis: Principles and procedures. BMJ. 2011. doi:10.1136/bmj.315.7121.1533
8.
Nikolakopoulou A, Mavridis D, Salanti G. Demystifying fixed and random effects meta-analysis. Evid Based Ment Health. 2014. doi:10.1136/eb-2014-101795
9.
Mavridis D, Salanti G. Exploring and accounting for publication bias in mental health: 13
a brief overview of methods. Evid Based Ment Health. 2014. doi:10.1136/eb-2013101700 10.
Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat Med. 2002. doi:10.1002/sim.1187
11.
Schünemann HJ, Brozek J, Guyatt GH, Oxman AD. 5.1 Factors determining the quality of evidence. GRADE Handbook. https://gdt.gradepro.org/app/handbook/handbook.html. Published 2013. Accessed May 22, 2019.
12.
Higgins JP, Altman DG, Sterne JA. 8.3.2 Reporting versus conduct. In: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. ; 2011:8.4.
13.
Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. Int J Surg. 2012. doi:10.1016/j.ijsu.2011.10.001
14.
Deeks JJ, Higgins JP, Altman DG. 9.1.4 When not to use meta-analysis in a review. In: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. ; 2011:9.4.
15.
NICE. Reviewing the evidence. The guidelines manual. https://www.nice.org.uk/process/pmg6/chapter/reviewing-the-evidence. Published 2012. Accessed May 22, 2019.
16.
Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Med. 2009;6(7). doi:10.1371/journal.pmed.1000100
17.
Antoniou SA, Köhler G, Antoniou GA, Muysoms FE, Pointner R, Granderath F-A. Meta14
analysis of randomized trials comparing nonpenetrating vs mechanical mesh fixation in laparoscopic inguinal hernia repair. Am J Surg. 2016. doi:10.1016/j.amjsurg.2015.06.008 18.
Antoniou SA, Morales-Conde S, Antoniou GA, Granderath FA, Berrevoet F, Muysoms FE. Single-incision laparoscopic surgery through the umbilicus is associated with a higher incidence of trocar-site hernia than conventional laparoscopy: a meta-analysis of randomized controlled trials. Hernia. 2016. doi:10.1007/s10029-015-1371-8
19.
Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors) Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. In: The Cochrane Collaboration. ; 2011. doi:10.1002/9780470712184
20.
Antoniou S, Mavridis D, Hajibandeh S, et al. Optimal stump management in laparoscopic appendectomy: a network meta-analysis. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=32730. Published 2016.
21.
Antoniou S, Hajibandeh S, Hajibandeh S, et al. Single-incision surgery and trocar-site hernia: an updated systematic review and meta-analysis with trial sequential analysis. http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42016051465. Published 2016. Accessed May 28, 2019.
22.
Andreou A, Hajibandeh S, Hajibandeh S, et al. Laparoscopic lavage, primary anastomosis or Hartmann procedure for acute complicated diverticulitis: network meta-analysis of the LADIES, DILALA and Scandinavian trials by the Minimally Invasive Surgery Synthesis of Interventions Outcomes Network (MISSION. http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42017079842. Published 2017. Accessed May 22, 2019.
15
23.
Andreou A, Antoniou S, Mavridis D, Chitsabesan P. Total and partial fundoplication in patients with reflux disease: a network meta-analysis of effectiveness and adverse outcomes. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=74783. Published 2017. Accessed May 22, 2019.
24.
Liu L. Heart Failure: Epidemiology and Research Methods. Elsevier; 2018. doi:10.1016/C2016-0-01350-5
25.
Antoniou SA, Mavridis D, Hajibandeh S, et al. Optimal stump management in laparoscopic appendectomy: A network meta-analysis by the Minimally Invasive Surgery Synthesis of Interventions and Outcomes Network. Surg (United States). 2017. doi:10.1016/j.surg.2017.07.013
26.
Antoniou SA, García-Alamino JM, Hajibandeh S, et al. Single-incision surgery trocarsite hernia: an updated systematic review meta-analysis with trial sequential analysis by the Minimally Invasive Surgery Synthesis of Interventions Outcomes Network (MISSION). Surg Endosc Other Interv Tech. 2017. doi:10.1007/s00464-017-5717-4
27.
Antoniou SA, Antoniou GA, Koch OO, Pointner R, Granderath FA. Lower recurrence rates after mesh-reinforced versus simple hiatal hernia repair: a meta-analysis of randomized trials. Surg Laparosc Endosc Percutan Tech 2012; 22(6):498-502
28.
Andreou A, Watson DI, Mavridis D, Francis NK, Antoniou SA. Assessing the efficacy and safety of laparoscopic antireflux procedures for the management of gastroesophageal reflux disease: A systematic review with network meta-analysis. Surg Endosc 2019. doi: 10.1007/s00464-019-07208-9 [Epub ahead of print]
29.
Kontopodis N, Galanakis N, Antoniou SA, Tsetis D, Ioannou CV, Veith FJ, Powell JT, Antoniou GA. Meta-analysis and meta-regression analysis of outcomes of
16
endovascular and open repair for ruptured abdominal aortic aneurysm. Eur J Vasc Endovasc Surg (in press). 30.
Antoniou GA, Juszczak MT, Nasr H, Narlawar R, Antoniou SA, Matsagkas M, Donas KP, de Vries JPM. Prognosis review and time-to-event data meta-analysis of endovascular aneurysm repair outside versus within instructions for use of aortic endograft devices. J Vasc Surg (in press).
31.
Polanin JR. Efforts to retrieve individual participant data sets for use in a metaanalysis result in moderate data sharing but many data sets remain missing. J Clin Epidemiol. 2018 Jun;98:157-159. doi: 10.1016/j.jclinepi.2017.12.014. Epub 2017 Dec 27.
32.
Taichman DB, Sahni P, Pinborg A, et al. Data sharing statements for clinical trials: A requirement of the international committee of medical journal editors. Rev Med Chil. 2017. doi:10.4067/s0034-98872017000600691
33.
Polanin JR, Terzian M. A data-sharing agreement helps to increase researchers' willingness to share primary data: results from a randomized controlled trial. J Clin Epidemiol. 2019 Feb;106:60-69. doi: 10.1016/j.jclinepi.2018.10.006. Epub 2018 Oct 19.
34.
Veroniki AA, Ashoor HM, Le SPC, Rios P, Stewart LA, Clarke M, Mavridis D, Straus SE, Tricco AC. Retrieval of individual patient data depended on study characteristics: A randomised controlled trial. J Clin Epidemiol. 2019 May 30. pii: S08954356(18)31061-8. doi: 10.1016/j.jclinepi.2019.05.031.
35.
WellcomeTrust. Our new policy on sharing research data: what it means for you. https://wellcome.ac.uk/news/our-new-policy-sharing-research-data-what-it-meansyou. Published 2017. Accessed May 22, 2019.
17
36.
Dukes P, Clement-Stoneham G. Data Sharing Policy.; 2016. https://mrc.ukri.org/documents/pdf/mrc-data-sharing-policy/.
37.
Kiley R, Peatfield T, Hansen J, Reddington F. Data Sharing from Clinical Trials — A Research Funder’s Perspective. N Engl J Med. 2017. doi:10.1056/nejmsb1708278
38.
Geifman N, Bollyky J, Bhattacharya S, Butte AJ. Opening clinical trial data: are the voluntary data-sharing portals enough? BMC Med. 2015 Nov 11;13:280. doi: 10.1186/s12916-015-0525-y
39.
ClinicalStudyDataRequest.com. https://clinicalstudydatarequest.com/Default.aspx. Accessed May 22, 2019.
40.
UK Data Service ReShare. https://reshare.ukdataservice.ac.uk. Accessed May 22, 2019.
41.
Vivli Center for Global Clinical Research Data. https://vivli.org. Accessed May 22, 2019.
42.
Grünenthal. Clinical data sharing with researchers. https://www.grunenthal.com/r-dvision-mission/clinical-trials/data-sharing-clinical-trials. Accessed May 22, 2019.
18
FIGURE LEGENDS Fig. 1: Flow chart of author response and providing requested information
19
Table 1. Univariable and multivariable logistic regression analyses for exploring association between various predictors and author response. In the multivariable case, the Odds Ratio for a predictor assume that all other predictors are constant. Univariable analysis
Multivariable analysis
Odds Ratio
95% Confidence Interval
Odds Ratio
95% Confidence Interval
Years between publication and contact with author > 5
0.980
(0.911,1.050)
0.931
(0.829,1.038)
Primary author is corresponding author
0.694
(0.312,1.536)
0.874
(0.355,2.164)
Source of email address is the article
0.830
(0.264,2.604)
0.285
(0.039,1.896)
Single-center study
1.917
(0.764,4.813)
1.912
(0.678,5.463)
Risk of bias for ‘Selective reporting’ is high or unclear
1.399
(0.608,3.194)
1.800
(0.672,4.928)
Risk of bias is high or unclear in > 3 domains
0.637
(0.261,1.567)
0.628
(0.545,1.962)
Variable
Table 2. Proposed bundle of actions to improve response rates from original investigators
Action
Offer collaboration to the review and joint authorship
Communicate through an institutional email
Provide registration number and link to review protocol
Consider financial incentives
Use a data sharing agreement
Request data though a data sharing platform
116 RCTs
Web 3 authors
Article 94 authors
Other publications 12 authors
Contact information not found n=7
Contacted 109 authors
Did not respond 81 authors
Responded 28 authors
Did not provide information 9 authors
Reminder 9 authors
Provided information 19 authors
Second invitation 81 authors
What is new?
Key findings • Response rate by authors of randomized trials to confirm accuracy of abstracted data and provide missing items for the purposes of evidence synthesis is low. • Actual contribution of missing data is as low as 9%. What this adds to what is known? • Standard methodology guidelines for meta-analyses, including MOOSE, PRISMA and the Cochrane Handbook, recommend contacting authors of original studies to confirm data and complete missing items. • Response to the request to confirm data accuracy and provide missing items has not been investigated in the context of meta-analysis. • This study investigates the response to this request and proposes actions for improvement of data retrieval from RCTs for the purposes of evidence synthesis. What is the implication and what should change now? • Data repositories and data sharing platforms should be further endorsed. • Scientific collaboration between authors of original studies and evidence synthesis experts should be emphasized. • Development of a data sharing culture is imperative.
S0895-4356(19)30643-2
DOI:
https://doi.org/10.1016/j.jclinepi.2019.11.014
Reference:
JCE 10020
To appear in:
Journal of Clinical Epidemiology
Received Date: 22 July 2019 Revised Date:
12 November 2019
Accepted Date: 20 November 2019
Please cite this article as: Tsokani S, Mavridis D, Hadjibandeh S, Hadjibandeh S, Antoniou GA, Andreou A, Antoniou SA, An observational study of missing data enquiry from randomised trial authors showed a poor response, Journal of Clinical Epidemiology (2019), doi: https://doi.org/10.1016/ j.jclinepi.2019.11.014. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2019 Elsevier Inc. All rights reserved.
Original study
An observational study of missing data enquiry from randomised trial authors showed a poor response
Running title: Information supplementation by original authors for meta-analysis
Authors 1
Sofia Tsokani, MSc
2,3
Dimitrios Mavridis, MSc, PhD
4
Shahab Hadjibandeh, MD
5
Shahin Hadjibandeh, MD
6
George A. Antoniou, MD MSc PhD FEBVS
7
Alexandros Andreou, MD, FEBS
8
Stavros A. Antoniou, MD PhD FEBS
Affiliations 1. Department of Mathematics, School of Sciences, University of Ioannina, Ioannina, Greece 2. Department of Primary Education, School of Education, University of Ioannina, Ioannina, Greece 3. Paris Descartes University, Sorbonne Paris Cité, Faculté de Médecine, Paris, France 4. Department of General Surgery, Royal Bolton Hospital, Bolton, UK 5. Department of General Surgery, Stepping Hill Hospital, Stockport, UK 6. Department of Vascular and Endovascular Surgery, The Royal Oldham Hospital, Pennine Acute Hospitals NHS Trust, Manchester, United Kingdom 7. Upper GI Department, Castle Hill Hospital, Hull and East Yorkshire Hospitals NHS Foundation Trust, Hull, UK 8. School of Medicine, European University Cyprus, Nicosia, Cyprus
Conflict of interest: none Funding: none 1
ABSTRACT Background The quality of evidence from a meta-analysis might be affected by poor or inadequate reporting of individual randomized controlled trials (RCTs). The results of data request from authors has not been investigated to date. Methods After completion of a feasibility trial, the authors of 116 RCTs selected for inclusion in 4 meta-analyses and network meta-analyses were contacted via email to check abstracted data from their studies and/or provide missing data. Univariable and multivariable analyses were performed to investigate the association of response rate with predefined characteristics, including time interval between publication of the RCT and the date of contact, role of the contact author as corresponding author, the number of participating centers, the source of contact email address and risk of bias assessments. Results The response rate to the primary email invitation was 31.1% (median response time 0 days, interquartile range (IQR) 0-1). Out of the authors who did not respond to the primary invitation, 31.2% responded to the reminder (median response time 1 day, IQR 0-3). Despite an overall response rate of 41%, both confirmation of data accuracy and supplementation of missing data were provided for 9 RCTs only (7.8%). No association was found between lack of response and predefined characteristics on univariable and multivariable analyses. Conclusion Despite a fair response rate, contacting authors of RCTs to check abstracted data and complete missing items yielded a very low rate of task completion. These findings highlight the importance of endorsing a data sharing culture among researchers. The role of alternative incentives is yet to be investigated. Keywords: randomized controlled trial, meta-analysis, data acquisition, data sharing 2
INTRODUCTION Randomized controlled trials (RCTs), when properly planned and conducted, are considered to provide the highest quality evidence to inform clinical practice and decision making amongst clinical trials.1,2 Participant randomization is the optimum method to control for known and unknown effect modifiers and confounders.3 For instance, a case-matched cohort study on laparoscopic versus open groin hernia repair might control for confounders which investigators are aware of, such as patient age, comorbidities or hernia size, but other factors such as systemic connective tissue disorder or hiatal hernia,4,5 whose confounding or modifying effect may be as important, as well as other unknown confounders, remain uncontrolled. Furthermore, blinding of participants, personnel and outcome assessors aims to reduce several sources of bias, which may affect certainty on evidence in observational studies.6 Meta-analysis of RCTs has the potential to increase precision of effect sizes, assess heterogeneity (differences in true effects across trials) and small study effects (systematic differences in effect sizes across small and large trials), explore subgroup differences and/or analyze trends over time.7-10 Nevertheless, the evidence provided by meta-analyses is based on the quality of the included RCTs. Indeed, the GRADE approach for evidence appraisal suggests that high or unknown risk of bias across RCTs may require downgrading or even double-downgrading the quality of evidence.11 Poor reporting of RCTs, however, does not necessarily mean poor conduct.12 Transparency in trial reporting is a low-hanging fruit in clinical research.13 Poor reporting of methodological and outcome parameters undoubtedly has a negative impact on the quality of meta-analyses. Organizations such as Cochrane and the National Institute for Health and Care Excellence (NICE) suggest that, in the absence of adequate and valid data, evidence synthesis might not be appropriate,14,15 thus leaving a large evidence base unexploited. In order to ensure reliability of the data collection process, the PRISMA (Preferred Reporting 3
Items for Systematic Reviews and Meta-analyses) initiative recommends for evidence synthesis researchers to obtain and confirm data from authors of RCTs.16 In our experience, authors’ feedback to this quest is poor. Aims of this prospective observational study were to quantify the response of authors of RCTs to the request for confirmation of original investigators’ data and provide missing information; to evaluate the adequacy of information provided; and to investigate whether predefined factors are associated with response rate. The objective was to inform communication practices and propose actions for improvement of data retrieval from RCTs for the purposes of evidence synthesis.
METHODS Protocol The study protocol was reviewed and approved by all authors. A repository for bibliometric studies does not exist, so the protocol was maintained by the scientific committee of the University Hospital of Heraklion. Following commencement of the study, there were no amendments and no deviations from the study protocol. Institutional review board approval or written consent was not required as no interventions on subjects were undertaken. Feasibility study Prior to embarking on this work, we performed a feasibility study with the objective to refine the methodology and adjust the data collection form. In the context of two meta-analyses conducted by our evidence synthesis group, we contacted the corresponding authors of 33 RCTs included in these meta-analyses through email to confirm data/provide missing information on their trials.17,18 We documented the response times and the information provided by original authors to define the time interval between the primary invitation and the reminder email, and to optimize the data collection form.
4
Criteria for inclusion of RCTs in the present study The first 100 eligible surgical RCTs for inclusion in systematic reviews performed by our evidence synthesis group were planned to be considered. No language restrictions were applied. The protocols of these reviews were submitted for registration in the International Prospective Register of Systematic Reviews (PROSPERO). The studies were selected upon second level screening of records identified in the study screening process, i.e. after title and abstract screening and assessment of full text reports of potentially eligible studies as per standard guidelines.19 Criteria for exclusion of RCTs from the present study RCTs identified through protocols of intervention systematic reviews not eligible for registration in the PROSPERO database and Cochrane reviews were excluded. Νo further exclusion criteria were applied. Data collection Data from RCTs were collected by (SH, SH, or AA) author and independently cross-checked by a second author (SH, SH, or AA). A third author (SAA) acted as an arbitrator in case of disagreement. Data items were pre-specified in study-specific protocols, which were submitted for registration in the PROSPERO database.20-23 These items included (i) studyrelated information (name of the primary author, name and contact details of the corresponding author, country of the primary institution, number of participating institutions), (ii) clinical/demographic information of the study population, (iii) outcome data, and (iv) risk of bias assessment data using the Cochrane Risk of Bias Tool for Randomized Trials. For each review, ad hoc Microsoft Excel spreadsheets based on the Cochrane Consumers and Communication Review Group’s data extraction template were developed, which were pilot-tested on the three most recent studies and refined accordingly.
5
Contact information of the corresponding author of each trial was recorded. After completion of data collection, the corresponding author of each trial was contacted via email by the senior author, using a standard invitation letter (Appendix), and was asked to check the abstracted data and provide missing items (predefined items to be collected, including patient demographics, inclusion/exclusion criteria, outcome data and risk of bias parameters). The spreadsheet containing abstracted data on their study was sent as attachment to the email and embedded in the body of the email. The date of invitation and the date of response, if any, were recorded. If the author did not respond after 14 days, a reminder email was sent (Appendix). The completeness of data provided by the authors was assessed and the date of response, if any, was recorded. Data analysis Data are presented using descriptive statistics. Categorical variables are presented as % percentage; continuous variables are presented as median, range and IQR range. Associations between author response and independent variables were explored using Pearson’s chi-square of independence for categorical data and t-tests for continuous data. The following independent variables were considered: 1. Time interval between publication of the RCT and email invitation (more than five versus less than five years; 0 = 0 to 5 years, 1 = 6 or more years) 2. Number of institutions (one versus more than one; 0 = 1 institution, 1 = 2 or more institutions) 3. The corresponding author is the first author (yes versus no; 1 = yes, 0 = no) 4. The source of the email address was the article versus any other source (0 = other, 1 = the article) 5. Risk of bias in the domain ‘selective reporting’ was considered high (yes versus no, 1 = yes, 0 = no)
6
6. Risk of bias was considered high in more than three risk of bias domains (1 = yes, 0 = no). The ‘author’s response’ variable is binary (0/1), with value 0 implying that the author did not respond to our request and value 1 otherwise. We applied logistic regression to explore potential associations between author response and the aforementioned characteristics. In total, there are six explanatory variables. We applied univariable logistic regression to investigate a potential association between authors’ response and each of these characteristics (‘time interval between publication of the RCT and email invitation’, ‘number of institutions’, ‘corresponding author is first author’, ‘source of email address’, ‘selective reporting’, ‘high or unclear risk of bias in more than 3 risk of bias components’. In addition, a multivariable logistic regression model was fitted to the data. Multivariable logistic regression analysis is an extension of bivariable logistic regression in which two or more independent variables are taken into consideration simultaneously to predict a dependent variable for each subject.24 The multivariable model fitted in this study used all six variables as predictors. The logistic regression analyses were carried out by the Logistic procedure in RStudio Version 1.1.423 (2009-2018 RStudio, Inc) in the Windows 10 environment, using the glm2 R-package.
RESULTS The overall response rate of the feasibility trial was 11 out of 33 (33%) and the median time from first contact to return of reports was 7 days (range 0-35, interquartile range, IQR 4-16). A total of 116 RCTs contributing to 4 systematic reviews/meta-analyses in the field of general and laparoscopic surgery were included: 3 network meta-analyses on the management of complicated diverticulitis,22
laparoscopic antireflux surgery23 and
laparoscopic appendectomy,25 respectively; and a meta-analysis and trial sequential analysis 7
on the risk of trocar site hernia after single-incision laparoscopic surgery.26 Sixteen (13.8%) were published between 1992-1999, 40 (34.5%) between 2000-2010 and 60 (51.7%) between 2011-2017. The years lapsed between publication and date of contact ranged from 1 to 24 years (median 6 years, IQR 4-13). The number of institutions contributing to the trials ranged from 1 to 20 (median 1 center, IQR 1-1). The first author was corresponding author in 61 articles (57.0%). Contact information could not be located for 7 corresponding authors (6.0%). The source of contact information was the published article (n=94, 86.2%), other publications of the same author (n=12, 11.0%) and the Web (n=3, 2.8%). Seven email addresses were not valid (6.4%) and 11 email messages were undeliverable (10.1%). Thirty six RCTs (33.6%) were judged to be at high or unclear risk of bias in the domain ‘selective reporting’ and 80 RCTs (74.8%) were judged to be at high or unclear risk of bias in more than 3 risk of bias domains. Twenty-eight authors responded to the primary invitation (response rate 31.1%, median response time 0 days, IQR 0-1, range 0-8). Out of the authors who responded, 19 provided some information (67.9%). Three authors reported that they were no longer staff at the institution where the work was conducted and one author had retired. The remaining 9 authors who responded but did not provide the requested information were sent a reminder. Six of those authors provided some information. Overall, out of the authors who responded to the primary email or the reminder email (n=47), 36 provided some information (76.6%). Typically a confirmation that the provided data were accurate, but missing data were mostly not addressed. Out of authors who did not respond to the primary invitation, 19 responded to the reminder email invitation (response rate 31.2%, median response time 1 day, IQR 0-3, range 0-5) and 17 provided information (27.9%).
8
Overall, both confirmation of data accuracy and missing data (if any) were provided for 9 RCTs (7.8%) (Fig. 1). None of the investigated characteristics were found to be associated with lack of response on univariable and multivariable analyses, albeit interval estimates were generally wide and a type II error cannot be excluded (Table 1).
DISCUSSION In this prospective observational study, we found that contacting authors of RCTs to confirm data and complete missing information for their trials resulted in a very low response rate. More importantly, obtaining requested information was achieved for as low as 7.8% of trials. In the majority of trial reports, contact information of the corresponding author was readily available in the article, however no bibliometric characteristics were found to be associated with author response. Nevertheless, this finding may be due to the small sample size; a true association might be present but not identifiable, thus warranting further research. The observation that 41% of authors responded to the first or second invitation and 39% eventually provided some kind of –usually incomplete– information, suggests that time constraints might be an impediment to effectively respond to such request. A second reminder might have resulted in obtaining further data, although lack of response in two invitations may suggest that this possibility is low. A minority of authors reported objective limitations to provide data, including change in working place and retirement. Interestingly, an author considered our communication as a spam and provided the requested data following cross-checking our contact email after a period of 2 months. In the era of a plethora of communication pathways, including email, social media and researcher communication platforms, one would expect that author feedback for research purposes would be more effective. In view of the results of the present analysis, what is considered as state-of-the-art methodology to acquire missing information from original investigators seems suboptimal. 9
Providing incentives to original investigators to confirm accuracy and supplement missing data for evidence synthesis purposes might be reasonable. The authors of systematic reviews may consider offering collaboration and co-authorship to original investigators. In our experience, such an approach results in a better insight to original data and a multifaceted view of the problem under investigation.27-30 Transparent collaboration of review authors with original investigators might decrease communication barriers and increase the quality of the meta-analysis. Research on obtaining data for the purposes of meta-analyses has focused on individual patient data meta-analyses. A systematic review found that only 61% of datasets included in 122 individual patient data meta-analyses were obtained.31 Financial incentives could probably drive increased contribution by original investigators, but this would require (additional) funding for systematic reviews. A recent RCT suggested that providing a financial incentive to authors of RCTs may not result in a higher rate of individual patient data sharing.32 In another RCT, authors who received a data sharing agreement were more likely to provide their datasets for the purposes of an individual patient data meta-analysis.33 Communication through an institutional email might decrease any doubts about the purposes of the contact. Providing the review registration number and/or the link to the study protocol (so the authors could affirm the originality of the work), might further increase the response rate. There is increasing awareness about data sharing in the research community. The International Committee of Medical Journal Editors (ICMJE) requires that trial authors contain a data sharing statement in their reports, declaring whether trial participant data are available, for how long and for which purposes.34 This policy has been embraced by major funders, such as the Wellcome Trust and the Medical Research Council.35-37 Data repositories dedicated to this policy are now publicly available. 10
The research community is however still far from wide implementation of an obligatory data sharing policy. Scientific data sharing platforms have only recently been launched and only a few (mostly industry-sponsored) clinical trials have agreed to provide data. Anecdotal data suggest that obtaining trial data through platforms is a time-consuming process.38 Due to this early stage of development of such platforms, their use for the purposes of systematic reviews is probably not yet an effective approach and might introduce study selection (publication) bias. It is however not difficult to envision a future with obligatory data sharing for trials submitted to peer-reviewed journals. A more effective approach might be the request of data through data sharing platforms. Indeed, some platforms already provide this option to researchers, which may be more effective and reliable than contacting original investigators directly.39-42 Original investigators may be less reluctant to provide patient-level data to trusted authorities, rather than to other researchers. A bundle of actions that might increase data acquisition from original investigators are summarized in Table 2. More importantly, the development of a data-sharing culture is imperative, to allow full exploit of trusted, deidentified patient data in meta-analyses and individual patient data meta-analyses. The practice of data sharing would have the potential to produce more precise effect estimates in meta-analysis and possibly provide different results. The incentive to share data is rather an ethical obligation to the research community, which calls for transparency, validation of outcomes and re-use of research data. The present work has some limitations that need to be taken into account when interpreting results. The studies included herein deal with general surgery RCTs, which might limit its external validity to this field. The low number of responses has reduced the power of exploratory analyses, therefore a type II error cannot be excluded. Our evidence synthesis group has undertaken a significant number of meta-analyses, its members might therefore be recognizable, hence response rates might be inflated. 11
CONCLUSION Data confirmation and completion of missing items by original investigators was achieved in only 7.8% of requests in a select cohort of surgical trials authors. A culture of data sharing is necessary to fully exploit clinical trial data in evidence synthesis projects.
12
REFERENCES 1.
OCEBM Levels of Evidence Working Group, Durieux N, Pasleau F, Howick J. The Oxford 2011 Levels of Evidence. Group. 2011;1(version):5653. http://www.cebm.net/index.aspx?o=1025.
2.
Guyatt GH. GRADE : an emerging consensus on rating quality of evidence and strength of recommendations. 2008;336(April).
3.
Jager K, Zoccali C, MacLeod A, Dekker F. Confounding: What it is and how to deal with it. Kidney Int. 2008;73(3):256-260. doi:10.1038/sj.ki.5002650
4.
Antoniou SA, Antoniou GA, Granderath FA, Simopoulos C. The role of matrix metalloproteinases in the pathogenesis of abdominal wall hernias. Eur J Clin Invest. 2009;39(11). doi:10.1111/j.1365-2362.2009.02199.x
5.
Antoniou GA, Tentes IK, Antoniou SA, Simopoulos C, Lazarides MK. Matrix metalloproteinase imbalance in inguinal hernia formation. J Investig Surg. 2011;24(4). doi:10.3109/08941939.2011.558610
6.
Schünemann HJ, Brozek J, Guyatt GH, Oxman AD. 5.2 Factors that can reduce the quality of the evidence. GRADE Handbook. https://gdt.gradepro.org/app/handbook/handbook.html#h.m9385o5z3li7. Published 2013.
7.
Egger M, Smith GD, Phillips AN. Meta-analysis: Principles and procedures. BMJ. 2011. doi:10.1136/bmj.315.7121.1533
8.
Nikolakopoulou A, Mavridis D, Salanti G. Demystifying fixed and random effects meta-analysis. Evid Based Ment Health. 2014. doi:10.1136/eb-2014-101795
9.
Mavridis D, Salanti G. Exploring and accounting for publication bias in mental health: 13
a brief overview of methods. Evid Based Ment Health. 2014. doi:10.1136/eb-2013101700 10.
Thompson SG, Higgins JPT. How should meta-regression analyses be undertaken and interpreted? Stat Med. 2002. doi:10.1002/sim.1187
11.
Schünemann HJ, Brozek J, Guyatt GH, Oxman AD. 5.1 Factors determining the quality of evidence. GRADE Handbook. https://gdt.gradepro.org/app/handbook/handbook.html. Published 2013. Accessed May 22, 2019.
12.
Higgins JP, Altman DG, Sterne JA. 8.3.2 Reporting versus conduct. In: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. ; 2011:8.4.
13.
Moher D, Hopewell S, Schulz KF, et al. CONSORT 2010 explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. Int J Surg. 2012. doi:10.1016/j.ijsu.2011.10.001
14.
Deeks JJ, Higgins JP, Altman DG. 9.1.4 When not to use meta-analysis in a review. In: Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. ; 2011:9.4.
15.
NICE. Reviewing the evidence. The guidelines manual. https://www.nice.org.uk/process/pmg6/chapter/reviewing-the-evidence. Published 2012. Accessed May 22, 2019.
16.
Liberati A, Altman DG, Tetzlaff J, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. PLoS Med. 2009;6(7). doi:10.1371/journal.pmed.1000100
17.
Antoniou SA, Köhler G, Antoniou GA, Muysoms FE, Pointner R, Granderath F-A. Meta14
analysis of randomized trials comparing nonpenetrating vs mechanical mesh fixation in laparoscopic inguinal hernia repair. Am J Surg. 2016. doi:10.1016/j.amjsurg.2015.06.008 18.
Antoniou SA, Morales-Conde S, Antoniou GA, Granderath FA, Berrevoet F, Muysoms FE. Single-incision laparoscopic surgery through the umbilicus is associated with a higher incidence of trocar-site hernia than conventional laparoscopy: a meta-analysis of randomized controlled trials. Hernia. 2016. doi:10.1007/s10029-015-1371-8
19.
Lefebvre C, Manheimer E, Glanville J. Chapter 6: Searching for studies. In: Higgins JPT, Green S (editors) Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0. In: The Cochrane Collaboration. ; 2011. doi:10.1002/9780470712184
20.
Antoniou S, Mavridis D, Hajibandeh S, et al. Optimal stump management in laparoscopic appendectomy: a network meta-analysis. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=32730. Published 2016.
21.
Antoniou S, Hajibandeh S, Hajibandeh S, et al. Single-incision surgery and trocar-site hernia: an updated systematic review and meta-analysis with trial sequential analysis. http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42016051465. Published 2016. Accessed May 28, 2019.
22.
Andreou A, Hajibandeh S, Hajibandeh S, et al. Laparoscopic lavage, primary anastomosis or Hartmann procedure for acute complicated diverticulitis: network meta-analysis of the LADIES, DILALA and Scandinavian trials by the Minimally Invasive Surgery Synthesis of Interventions Outcomes Network (MISSION. http://www.crd.york.ac.uk/PROSPERO/display_record.php?ID=CRD42017079842. Published 2017. Accessed May 22, 2019.
15
23.
Andreou A, Antoniou S, Mavridis D, Chitsabesan P. Total and partial fundoplication in patients with reflux disease: a network meta-analysis of effectiveness and adverse outcomes. https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=74783. Published 2017. Accessed May 22, 2019.
24.
Liu L. Heart Failure: Epidemiology and Research Methods. Elsevier; 2018. doi:10.1016/C2016-0-01350-5
25.
Antoniou SA, Mavridis D, Hajibandeh S, et al. Optimal stump management in laparoscopic appendectomy: A network meta-analysis by the Minimally Invasive Surgery Synthesis of Interventions and Outcomes Network. Surg (United States). 2017. doi:10.1016/j.surg.2017.07.013
26.
Antoniou SA, García-Alamino JM, Hajibandeh S, et al. Single-incision surgery trocarsite hernia: an updated systematic review meta-analysis with trial sequential analysis by the Minimally Invasive Surgery Synthesis of Interventions Outcomes Network (MISSION). Surg Endosc Other Interv Tech. 2017. doi:10.1007/s00464-017-5717-4
27.
Antoniou SA, Antoniou GA, Koch OO, Pointner R, Granderath FA. Lower recurrence rates after mesh-reinforced versus simple hiatal hernia repair: a meta-analysis of randomized trials. Surg Laparosc Endosc Percutan Tech 2012; 22(6):498-502
28.
Andreou A, Watson DI, Mavridis D, Francis NK, Antoniou SA. Assessing the efficacy and safety of laparoscopic antireflux procedures for the management of gastroesophageal reflux disease: A systematic review with network meta-analysis. Surg Endosc 2019. doi: 10.1007/s00464-019-07208-9 [Epub ahead of print]
29.
Kontopodis N, Galanakis N, Antoniou SA, Tsetis D, Ioannou CV, Veith FJ, Powell JT, Antoniou GA. Meta-analysis and meta-regression analysis of outcomes of
16
endovascular and open repair for ruptured abdominal aortic aneurysm. Eur J Vasc Endovasc Surg (in press). 30.
Antoniou GA, Juszczak MT, Nasr H, Narlawar R, Antoniou SA, Matsagkas M, Donas KP, de Vries JPM. Prognosis review and time-to-event data meta-analysis of endovascular aneurysm repair outside versus within instructions for use of aortic endograft devices. J Vasc Surg (in press).
31.
Polanin JR. Efforts to retrieve individual participant data sets for use in a metaanalysis result in moderate data sharing but many data sets remain missing. J Clin Epidemiol. 2018 Jun;98:157-159. doi: 10.1016/j.jclinepi.2017.12.014. Epub 2017 Dec 27.
32.
Taichman DB, Sahni P, Pinborg A, et al. Data sharing statements for clinical trials: A requirement of the international committee of medical journal editors. Rev Med Chil. 2017. doi:10.4067/s0034-98872017000600691
33.
Polanin JR, Terzian M. A data-sharing agreement helps to increase researchers' willingness to share primary data: results from a randomized controlled trial. J Clin Epidemiol. 2019 Feb;106:60-69. doi: 10.1016/j.jclinepi.2018.10.006. Epub 2018 Oct 19.
34.
Veroniki AA, Ashoor HM, Le SPC, Rios P, Stewart LA, Clarke M, Mavridis D, Straus SE, Tricco AC. Retrieval of individual patient data depended on study characteristics: A randomised controlled trial. J Clin Epidemiol. 2019 May 30. pii: S08954356(18)31061-8. doi: 10.1016/j.jclinepi.2019.05.031.
35.
WellcomeTrust. Our new policy on sharing research data: what it means for you. https://wellcome.ac.uk/news/our-new-policy-sharing-research-data-what-it-meansyou. Published 2017. Accessed May 22, 2019.
17
36.
Dukes P, Clement-Stoneham G. Data Sharing Policy.; 2016. https://mrc.ukri.org/documents/pdf/mrc-data-sharing-policy/.
37.
Kiley R, Peatfield T, Hansen J, Reddington F. Data Sharing from Clinical Trials — A Research Funder’s Perspective. N Engl J Med. 2017. doi:10.1056/nejmsb1708278
38.
Geifman N, Bollyky J, Bhattacharya S, Butte AJ. Opening clinical trial data: are the voluntary data-sharing portals enough? BMC Med. 2015 Nov 11;13:280. doi: 10.1186/s12916-015-0525-y
39.
ClinicalStudyDataRequest.com. https://clinicalstudydatarequest.com/Default.aspx. Accessed May 22, 2019.
40.
UK Data Service ReShare. https://reshare.ukdataservice.ac.uk. Accessed May 22, 2019.
41.
Vivli Center for Global Clinical Research Data. https://vivli.org. Accessed May 22, 2019.
42.
Grünenthal. Clinical data sharing with researchers. https://www.grunenthal.com/r-dvision-mission/clinical-trials/data-sharing-clinical-trials. Accessed May 22, 2019.
18
FIGURE LEGENDS Fig. 1: Flow chart of author response and providing requested information
19
Table 1. Univariable and multivariable logistic regression analyses for exploring association between various predictors and author response. In the multivariable case, the Odds Ratio for a predictor assume that all other predictors are constant. Univariable analysis
Multivariable analysis
Odds Ratio
95% Confidence Interval
Odds Ratio
95% Confidence Interval
Years between publication and contact with author > 5
0.980
(0.911,1.050)
0.931
(0.829,1.038)
Primary author is corresponding author
0.694
(0.312,1.536)
0.874
(0.355,2.164)
Source of email address is the article
0.830
(0.264,2.604)
0.285
(0.039,1.896)
Single-center study
1.917
(0.764,4.813)
1.912
(0.678,5.463)
Risk of bias for ‘Selective reporting’ is high or unclear
1.399
(0.608,3.194)
1.800
(0.672,4.928)
Risk of bias is high or unclear in > 3 domains
0.637
(0.261,1.567)
0.628
(0.545,1.962)
Variable
Table 2. Proposed bundle of actions to improve response rates from original investigators
Action
Offer collaboration to the review and joint authorship
Communicate through an institutional email
Provide registration number and link to review protocol
Consider financial incentives
Use a data sharing agreement
Request data though a data sharing platform
116 RCTs
Web 3 authors
Article 94 authors
Other publications 12 authors
Contact information not found n=7
Contacted 109 authors
Did not respond 81 authors
Responded 28 authors
Did not provide information 9 authors
Reminder 9 authors
Provided information 19 authors
Second invitation 81 authors
What is new?
Key findings • Response rate by authors of randomized trials to confirm accuracy of abstracted data and provide missing items for the purposes of evidence synthesis is low. • Actual contribution of missing data is as low as 9%. What this adds to what is known? • Standard methodology guidelines for meta-analyses, including MOOSE, PRISMA and the Cochrane Handbook, recommend contacting authors of original studies to confirm data and complete missing items. • Response to the request to confirm data accuracy and provide missing items has not been investigated in the context of meta-analysis. • This study investigates the response to this request and proposes actions for improvement of data retrieval from RCTs for the purposes of evidence synthesis. What is the implication and what should change now? • Data repositories and data sharing platforms should be further endorsed. • Scientific collaboration between authors of original studies and evidence synthesis experts should be emphasized. • Development of a data sharing culture is imperative.