- Email: [email protected]
A Question of Reproducibility
editorial
© The American Society of Gene & Cell Therapy
doi:10.1038/mt.2014.211
A Question of Reproducibility In all science, error precedes the truth, and it is better it should go first than last. —Hugh Walpole
R
eproducibility is a fundamental tenet of the scientific process, and sharing of both data and methodology in a public forum is key to ensuring reproducibility of scientific findings. However, in recent years there has been increasing concern over the lack of reproducibility of many published scientific findings. This issue has been exacerbated by the rapid expansion of new publishing models, such as open-access journals and preprint servers, not all of which have same requirements with regard to data reproducibility and rigor of the review process. This is not a trivial issue, as irreproducible data can send basic and preclinical researchers down blind alleys, wasting both time and money. This has prompted action by stakeholders in the integrity of the scientific process. Besides scientists, these include funders, academic societies, publishers, and, of course, the public, including patient-advocacy groups. For example, the Prostate Cancer Foundation (PCF) recently announced the Movember Foundation–PCF Scientific Reproducibility Initiative, which aims to accelerate translation of promising discoveries into new tests and treatments for prostate cancer through faster validation of the science underlying new experimental therapies. This past July, the Office of Science and Technology Policy and the National Economic Council asked how the federal government could leverage its role as a significant funder of scientific research to address the reproducibility issue. Publishers and academic societies are also tackling this key issue. Nature Publishing Group (NPG) has taken an active lead in establishing means to fight irreproducibility. NPG introduced measures last year with a view to ensuring good reporting standards and improving the reproducibility of findings published in the pages of its life-science journals. One such measure is requiring completion of a checklist (http://www.nature.com/authors/policies/checklist. pdf). In June, NPG cohosted a meeting with the leadership of the National Institutes of Health (NIH) and colleagues at Science magazine to examine the issues of reproducibility and rigor of research findings. The one-day workshop brought together editors repre-
Molecular Therapy vol. 22 no. 12 december 2014
senting 30 basic- and preclinical-research journals, which had been selected by the NIH based on frequency of publication of their grantees. The objective was to discuss the measures that some journals have already taken and to agree on a common set of minimum measures to which all participants could subscribe that would be presented to the larger community for consideration. The workshop participants came to consensus on a set of principles to facilitate these goals, which has been published on the NIH website (http://www. nih.gov/about/reporting-preclinical-research.htm) along with a list of endorsing journals (http://www. nih.gov/about/endorsing-jounals.htm). NPG has, of course, endorsed these principles, as has the American Society of Gene Therapy’s Molecular Therapy family of journals. A key factor in data reproducibility is effective and comprehensive description of methodology. The advent of online supplements has rendered moot traditional considerations regarding the size of print publications and authors should not feel constrained by space limitations. The aim of the agreed principles is to facilitate interpretation and repetition of experiments as they were conducted in the published studies. They include guidelines for statistical analyses and for transparency in reporting key methodology, analytical information, and reagents, and mandates for data and material sharing. Editors of the MT family of journals will continue to ask reviewers to comment on whether, in their view, there is sufficient description of methodology to allow independent reproduction of the data. Many of our authors already conform to these principals, and, although these additional requirements will impose a small added burden on other authors, we are confident that, overall, the time and effort saved as a consequence of improved reproducibility will more than outweigh any inconvenience.
Robert M Frederickson Editor
Malcolm K Brenner Editor-in-Chief
2015
© The American Society of Gene & Cell Therapy
doi:10.1038/mt.2014.211
A Question of Reproducibility In all science, error precedes the truth, and it is better it should go first than last. —Hugh Walpole
R
eproducibility is a fundamental tenet of the scientific process, and sharing of both data and methodology in a public forum is key to ensuring reproducibility of scientific findings. However, in recent years there has been increasing concern over the lack of reproducibility of many published scientific findings. This issue has been exacerbated by the rapid expansion of new publishing models, such as open-access journals and preprint servers, not all of which have same requirements with regard to data reproducibility and rigor of the review process. This is not a trivial issue, as irreproducible data can send basic and preclinical researchers down blind alleys, wasting both time and money. This has prompted action by stakeholders in the integrity of the scientific process. Besides scientists, these include funders, academic societies, publishers, and, of course, the public, including patient-advocacy groups. For example, the Prostate Cancer Foundation (PCF) recently announced the Movember Foundation–PCF Scientific Reproducibility Initiative, which aims to accelerate translation of promising discoveries into new tests and treatments for prostate cancer through faster validation of the science underlying new experimental therapies. This past July, the Office of Science and Technology Policy and the National Economic Council asked how the federal government could leverage its role as a significant funder of scientific research to address the reproducibility issue. Publishers and academic societies are also tackling this key issue. Nature Publishing Group (NPG) has taken an active lead in establishing means to fight irreproducibility. NPG introduced measures last year with a view to ensuring good reporting standards and improving the reproducibility of findings published in the pages of its life-science journals. One such measure is requiring completion of a checklist (http://www.nature.com/authors/policies/checklist. pdf). In June, NPG cohosted a meeting with the leadership of the National Institutes of Health (NIH) and colleagues at Science magazine to examine the issues of reproducibility and rigor of research findings. The one-day workshop brought together editors repre-
Molecular Therapy vol. 22 no. 12 december 2014
senting 30 basic- and preclinical-research journals, which had been selected by the NIH based on frequency of publication of their grantees. The objective was to discuss the measures that some journals have already taken and to agree on a common set of minimum measures to which all participants could subscribe that would be presented to the larger community for consideration. The workshop participants came to consensus on a set of principles to facilitate these goals, which has been published on the NIH website (http://www. nih.gov/about/reporting-preclinical-research.htm) along with a list of endorsing journals (http://www. nih.gov/about/endorsing-jounals.htm). NPG has, of course, endorsed these principles, as has the American Society of Gene Therapy’s Molecular Therapy family of journals. A key factor in data reproducibility is effective and comprehensive description of methodology. The advent of online supplements has rendered moot traditional considerations regarding the size of print publications and authors should not feel constrained by space limitations. The aim of the agreed principles is to facilitate interpretation and repetition of experiments as they were conducted in the published studies. They include guidelines for statistical analyses and for transparency in reporting key methodology, analytical information, and reagents, and mandates for data and material sharing. Editors of the MT family of journals will continue to ask reviewers to comment on whether, in their view, there is sufficient description of methodology to allow independent reproduction of the data. Many of our authors already conform to these principals, and, although these additional requirements will impose a small added burden on other authors, we are confident that, overall, the time and effort saved as a consequence of improved reproducibility will more than outweigh any inconvenience.
Robert M Frederickson Editor
Malcolm K Brenner Editor-in-Chief
2015