- Email: [email protected]
Peering into Peer-Review
Peering into Peer-Review Monica L. Helton, BA and William F. Balistreri, MD
P
eer-review is defined as the expert assessment of submitted materials.1 Because ‘‘the goal of this process is to ensure that the valid article is accepted, the messy article improved, and the invalid article rejected,’’2 quality reviews are essential to the peer-review process and assist in maintaining the integrity of the medical literature. Grainger believes that ‘‘a critical determinant of any successful journal or technical communication in general is the sound, reliable capability to readily access a talented, adept, accomplished and reliable reviewer pool.’’3 Because The Journal of Pediatrics publishes articles focusing on a wide variety of specialties and subspecialties in the field of pediatrics, The Journal uses an extensive historical database of experts, to which more are added daily. However, because of heavy workloads and time constraints of potential reviewers, it is often challenging for the editors to secure a commitment to review a manuscript from appropriate reviewers. Editors of The Journal usually choose reviewers from these sources: (1) personal knowledge of experts in the field; (2) authors of articles referenced in the manuscript; (3) reviewers suggested by the authors; (4) literature searches; (5) requesting suggestions from the editorial board; and (6) Elsevier Editorial System (EES) report of manuscript/personal classification matches.4 Occasionally, reviewers will be invited because another reviewer who declined or was uninvited suggests an individual as a good alternative candidate. Occasionally editors will choose reviewers from eTBlast, a freely available ‘‘similarity-based search engine,’’ which allows them to find ‘‘authors that are the most published in the topic of [their] query’’ (http://etest.vbi.vt.edu/etblast3/).
Peer-Review Process for The Journal The Journal uses a single-blind review process; reviewers have access to the authors’ names and affiliations, but the identity of reviewers is confidential. Manuscript submission and subsequent peer-review occurs through EES, the online submission and editorial system for The Journal (http://ees.elsevier. com/jpeds). The Journal invites authors to suggest 5 to 7 potential reviewers in their initial letter of submission. Reviewers who accept the invitation are e-mailed specific reviewer instructions (http://jpeds.com/authorinfo#guiderev), which ask them to consider the quality of the manuscript, the novelty and importance of the observation, appropriateness for the readership, and overlap between any other publications. After a thorough assessment of the manuscript,
reviewers submit their confidential comments to the editors and anonymous comments to authors through EES. After all reviews have been received for a manuscript, the editor considers the reviewers’ assessments and the editorial policies and criteria specific to The Journal and makes a decision. The Journal publishes <20% of submitted manuscripts, which are rarely approved for acceptance without at least one revision. Because conflicts of interest for reviewers are just as important as conflicts of interest for authors, when an invited reviewer is at the same department or institution as the authors, has conducted research with the authors, or is aware of anything that may be perceived as a conflict of interest, the reviewer should not accept the invitation to review.9 When reviewers believe that, despite the circumstances, a fair and objective review can be conducted, they must notify the editor of the potential conflict(s) of interest in their confidential comments to the editor. When the reviewer is uncertain whether objectivity can be attained, it is best to err on the side of caution and decline the invitation.
Assessment of Reviewers Recommended by Authors Versus Editors Studies have found that editor-suggested reviewers (ESRs) were less likely than author-suggested reviewers (ASRs) to recommend acceptance.5-7 Rivara et al found, in The Archives of Pediatrics and Adolescent Medicine, that 75% of ESRs recommended accept or revise, and 86% of ASRs recommended accept or revise.7 In a sample of reviewers from 10 journals, Schroter et al found that ASRs tended to review manuscripts more favorably than ESRs.8 Because the habits of ASRs versus ESRs may affect how editors choose reviewers for manuscripts, we wanted to examine whether the same was true for The Journal by evaluating whether ASRs were more likely than ESRs to provide positive reviews. Additionally, we tested the hypothesis that ASRs are more likely than ESRs to accept invitations to review and concur less frequently with the editor’s final decision. Methods The editorial office of The Journal retrospectively evaluated the first 300 manuscripts submitted to The Journal that were assigned consecutive manuscript numbers in one year
From the Cincinnati Children’s Hospital Medical Center, Cincinnati, OH M.H. and W.B. are paid employees of The Journal of Pediatrics. The Publisher, Elsevier, provided funding support for the statistical analysis.
ARS ERS ESS
Author-suggested reviewer Editor-suggested reviewer Elsevier Editorial System
The abstract was presented as a poster at the Sixth International Congress on Peer Review and Biomedical Publication, Sep 10-12, 2009, Vancouver, BC, Canada. 0022-3476/$ - see front matter. Copyright ª 2011 Mosby Inc. All rights reserved. 10.1016/j.jpeds.2011.02.012
150
Vol. 159, No. 1 July 2011 (2007). At the time of our assessment, all revisions had been received and final decisions made for these articles. Manuscripts that did not undergo external peer-review (ie, Editorials, Insights, and manuscripts that were editorially rejected without peer review) were excluded (n = 122). For the 178 peer-reviewed manuscripts, we recorded the following in Microsoft Excel 2003 software (Microsoft Corporation, Redmond, Washington): (1) whether the invited reviewers were suggested by the author or chosen by the editor; (2) the number of ASRs and ESRs who completed reviews; (3) the initial recommendation of the reviewer (‘‘accept,’’ ‘‘accept with revisions,’’ or ‘‘reject’’); and (4) the final decision of the editor (‘‘accept’’ or ‘‘reject’’). For our analyses, reviewer recommendations for ‘‘accept’’ included ‘‘accept’’ and ‘‘accept with revisions’’ recommendations. An independent statistician used the c2 and the McNemar test for correlated proportions. All computations were done with SAS software version 9.2 (SAS, Cary, North Carolina). Approval of the study protocol was not required according to the institutional review board at Cincinnati Children’s Hospital Medical Center. Results Of the total reviewers examined, 39.4% of invited reviewers (n = 873) accepted the invitation. ASRs accepted 37.2% of invitations (167/449) compared with 41.8% of ESRs (177/ 424; P = .17). Reviewers who were suggested by classification lists populated by EES or proposed as alternative candidates by reviewers who did not agree to review were excluded because they were chosen by neither the author nor the editor (n = 106). When evaluating manuscripts, 65.3% of ASRs recommended acceptance (109/167), whereas 54.2% of ESRs recommended acceptance (96/177; P = .04). Editors agreed with 49.5% (54/109) of the accept recommendations of ASRs (P < .0001) and with 55.2% (53/96) of ESRs (P < .0001). Discussion The evidence that ASRs are more likely than ESRs to accept an invitation to review is not statistically significant. However, ASRs are more likely to recommend acceptance of a submitted manuscript, although editors are less likely than both ASRs and ESRs to recommend acceptance of a manuscript. Regardless of whether a reviewer was suggested by an author or recommended by the editor, all reviewers should strive to provide a fair and balanced review, maintain confidentiality of the manuscript and its findings, and disclose any real or perceived biases. The rigorous assessment by the editors of The Journal appears to offset accept-recommendation advantages of ASRs. Editors weigh reviews on the basis of the expertise of the reviewer and thoroughness of the review, among other factors; the overall balance of the reviews is also considered before making a final decision on a manuscript. The study has limitations. The reviewers’ initial recommendations on publication were included in this analysis.
However, reviewers may recommend acceptance while giving a low priority for publication or recommend rejection with a high priority. Additionally, manuscripts are judged on the current medical literature and other manuscripts submitted to/published in The Journal. When reports with similar findings or topics have recently been submitted or published, manuscripts may be rejected because of lack of novelty. Although our findings could be caused by a variety of factors, the results support the peer review motto, ‘‘Reviewers advise; editors decide.’’4 Because editors serve as the gatekeepers to medical literature, the final responsibility resides with the editors to ensure impartiality in the peer-review process.10 n We thank Rebecca Lindeman, BS, for data acquisition, Judy Bean, PhD, for statistical assistance, and the associate editors of The Journal of Pediatrics (Steven H. Abman, M.D., Stephen R. Daniels, MD, PhD, Alan H. Jobe, MD, PhD, Sarah S. Long, MD, Thomas R. Welch, MD, and Robert W. Wilmott, MD) for their valuable input. Finally, we thank the authors, reviewers, and readers of The Journal of Pediatrics for their time, effort, and support. Submitted for publication Oct 28, 2010; last revision received Jan 26, 2011; accepted Feb 4, 2011. Reprint requests: Monica Helton, BA, The Journal of Pediatrics, Cincinnati Children’s Hospital Medical Center, 3333 Burnet Ave., Cincinnati, OH 45229. E-mail: [email protected].
References 1. Bailar JC, Patterson K. Journal peer review: the need for a research agenda. N Engl J Med 1985;312:654-7. 2. Bornmann L, Daniel H-D. The manuscript reviewing process: empirical research on review requests, review sequences, and decision rules in peer review. Libr Information Sci Res 2010;32:5-12. 3. Grainger DW. Peer review as professional responsibility: a quality control system only as good as the participants. Biomaterials 2007;28: 5199-203. 4. Balistreri WF. Landmark, landmine, or landfill? The role of peer review in assessing manuscripts. J Pediatr 2007;151:107-8. 5. Earnshaw JJ, Farndon JR, Guillou PF, Johnson CD, Murie JA, Murray GD. A comparison of reports from referees chosen by authors or journal editors in the peer review process. Ann R Coll Surg Engl 2000;82:133-5. 6. Wager E, Parkin EC, Tamber PS. Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study. BMC 2006;4:13. 7. Rivara FP, Cummings P, Ringold S, Bergman AB, Joffe A, Christakis DA. A comparison of reviewers selected by editors and suggested by authors. J Pediatr 2007;151:107-8. 8. Schroter S, Tite L, Hutchings A, Black N. Differences in review quality and recommendations for publication between peer reviews suggested by authors or by editors. JAMA 2006;295:314-7. 9. Barbour V, Clark J, Peiperly L, Veitch E, Wong M, Yamey G. Making sense of non-financial competing interests. PLoS Medicine 2008;5: 1299-301. 10. Hojat M, Rosenzweig S. Journal peer review in integrative medicine discipline. Semin Integrative Med 2004;2:1-4.
151
P
eer-review is defined as the expert assessment of submitted materials.1 Because ‘‘the goal of this process is to ensure that the valid article is accepted, the messy article improved, and the invalid article rejected,’’2 quality reviews are essential to the peer-review process and assist in maintaining the integrity of the medical literature. Grainger believes that ‘‘a critical determinant of any successful journal or technical communication in general is the sound, reliable capability to readily access a talented, adept, accomplished and reliable reviewer pool.’’3 Because The Journal of Pediatrics publishes articles focusing on a wide variety of specialties and subspecialties in the field of pediatrics, The Journal uses an extensive historical database of experts, to which more are added daily. However, because of heavy workloads and time constraints of potential reviewers, it is often challenging for the editors to secure a commitment to review a manuscript from appropriate reviewers. Editors of The Journal usually choose reviewers from these sources: (1) personal knowledge of experts in the field; (2) authors of articles referenced in the manuscript; (3) reviewers suggested by the authors; (4) literature searches; (5) requesting suggestions from the editorial board; and (6) Elsevier Editorial System (EES) report of manuscript/personal classification matches.4 Occasionally, reviewers will be invited because another reviewer who declined or was uninvited suggests an individual as a good alternative candidate. Occasionally editors will choose reviewers from eTBlast, a freely available ‘‘similarity-based search engine,’’ which allows them to find ‘‘authors that are the most published in the topic of [their] query’’ (http://etest.vbi.vt.edu/etblast3/).
Peer-Review Process for The Journal The Journal uses a single-blind review process; reviewers have access to the authors’ names and affiliations, but the identity of reviewers is confidential. Manuscript submission and subsequent peer-review occurs through EES, the online submission and editorial system for The Journal (http://ees.elsevier. com/jpeds). The Journal invites authors to suggest 5 to 7 potential reviewers in their initial letter of submission. Reviewers who accept the invitation are e-mailed specific reviewer instructions (http://jpeds.com/authorinfo#guiderev), which ask them to consider the quality of the manuscript, the novelty and importance of the observation, appropriateness for the readership, and overlap between any other publications. After a thorough assessment of the manuscript,
reviewers submit their confidential comments to the editors and anonymous comments to authors through EES. After all reviews have been received for a manuscript, the editor considers the reviewers’ assessments and the editorial policies and criteria specific to The Journal and makes a decision. The Journal publishes <20% of submitted manuscripts, which are rarely approved for acceptance without at least one revision. Because conflicts of interest for reviewers are just as important as conflicts of interest for authors, when an invited reviewer is at the same department or institution as the authors, has conducted research with the authors, or is aware of anything that may be perceived as a conflict of interest, the reviewer should not accept the invitation to review.9 When reviewers believe that, despite the circumstances, a fair and objective review can be conducted, they must notify the editor of the potential conflict(s) of interest in their confidential comments to the editor. When the reviewer is uncertain whether objectivity can be attained, it is best to err on the side of caution and decline the invitation.
Assessment of Reviewers Recommended by Authors Versus Editors Studies have found that editor-suggested reviewers (ESRs) were less likely than author-suggested reviewers (ASRs) to recommend acceptance.5-7 Rivara et al found, in The Archives of Pediatrics and Adolescent Medicine, that 75% of ESRs recommended accept or revise, and 86% of ASRs recommended accept or revise.7 In a sample of reviewers from 10 journals, Schroter et al found that ASRs tended to review manuscripts more favorably than ESRs.8 Because the habits of ASRs versus ESRs may affect how editors choose reviewers for manuscripts, we wanted to examine whether the same was true for The Journal by evaluating whether ASRs were more likely than ESRs to provide positive reviews. Additionally, we tested the hypothesis that ASRs are more likely than ESRs to accept invitations to review and concur less frequently with the editor’s final decision. Methods The editorial office of The Journal retrospectively evaluated the first 300 manuscripts submitted to The Journal that were assigned consecutive manuscript numbers in one year
From the Cincinnati Children’s Hospital Medical Center, Cincinnati, OH M.H. and W.B. are paid employees of The Journal of Pediatrics. The Publisher, Elsevier, provided funding support for the statistical analysis.
ARS ERS ESS
Author-suggested reviewer Editor-suggested reviewer Elsevier Editorial System
The abstract was presented as a poster at the Sixth International Congress on Peer Review and Biomedical Publication, Sep 10-12, 2009, Vancouver, BC, Canada. 0022-3476/$ - see front matter. Copyright ª 2011 Mosby Inc. All rights reserved. 10.1016/j.jpeds.2011.02.012
150
Vol. 159, No. 1 July 2011 (2007). At the time of our assessment, all revisions had been received and final decisions made for these articles. Manuscripts that did not undergo external peer-review (ie, Editorials, Insights, and manuscripts that were editorially rejected without peer review) were excluded (n = 122). For the 178 peer-reviewed manuscripts, we recorded the following in Microsoft Excel 2003 software (Microsoft Corporation, Redmond, Washington): (1) whether the invited reviewers were suggested by the author or chosen by the editor; (2) the number of ASRs and ESRs who completed reviews; (3) the initial recommendation of the reviewer (‘‘accept,’’ ‘‘accept with revisions,’’ or ‘‘reject’’); and (4) the final decision of the editor (‘‘accept’’ or ‘‘reject’’). For our analyses, reviewer recommendations for ‘‘accept’’ included ‘‘accept’’ and ‘‘accept with revisions’’ recommendations. An independent statistician used the c2 and the McNemar test for correlated proportions. All computations were done with SAS software version 9.2 (SAS, Cary, North Carolina). Approval of the study protocol was not required according to the institutional review board at Cincinnati Children’s Hospital Medical Center. Results Of the total reviewers examined, 39.4% of invited reviewers (n = 873) accepted the invitation. ASRs accepted 37.2% of invitations (167/449) compared with 41.8% of ESRs (177/ 424; P = .17). Reviewers who were suggested by classification lists populated by EES or proposed as alternative candidates by reviewers who did not agree to review were excluded because they were chosen by neither the author nor the editor (n = 106). When evaluating manuscripts, 65.3% of ASRs recommended acceptance (109/167), whereas 54.2% of ESRs recommended acceptance (96/177; P = .04). Editors agreed with 49.5% (54/109) of the accept recommendations of ASRs (P < .0001) and with 55.2% (53/96) of ESRs (P < .0001). Discussion The evidence that ASRs are more likely than ESRs to accept an invitation to review is not statistically significant. However, ASRs are more likely to recommend acceptance of a submitted manuscript, although editors are less likely than both ASRs and ESRs to recommend acceptance of a manuscript. Regardless of whether a reviewer was suggested by an author or recommended by the editor, all reviewers should strive to provide a fair and balanced review, maintain confidentiality of the manuscript and its findings, and disclose any real or perceived biases. The rigorous assessment by the editors of The Journal appears to offset accept-recommendation advantages of ASRs. Editors weigh reviews on the basis of the expertise of the reviewer and thoroughness of the review, among other factors; the overall balance of the reviews is also considered before making a final decision on a manuscript. The study has limitations. The reviewers’ initial recommendations on publication were included in this analysis.
However, reviewers may recommend acceptance while giving a low priority for publication or recommend rejection with a high priority. Additionally, manuscripts are judged on the current medical literature and other manuscripts submitted to/published in The Journal. When reports with similar findings or topics have recently been submitted or published, manuscripts may be rejected because of lack of novelty. Although our findings could be caused by a variety of factors, the results support the peer review motto, ‘‘Reviewers advise; editors decide.’’4 Because editors serve as the gatekeepers to medical literature, the final responsibility resides with the editors to ensure impartiality in the peer-review process.10 n We thank Rebecca Lindeman, BS, for data acquisition, Judy Bean, PhD, for statistical assistance, and the associate editors of The Journal of Pediatrics (Steven H. Abman, M.D., Stephen R. Daniels, MD, PhD, Alan H. Jobe, MD, PhD, Sarah S. Long, MD, Thomas R. Welch, MD, and Robert W. Wilmott, MD) for their valuable input. Finally, we thank the authors, reviewers, and readers of The Journal of Pediatrics for their time, effort, and support. Submitted for publication Oct 28, 2010; last revision received Jan 26, 2011; accepted Feb 4, 2011. Reprint requests: Monica Helton, BA, The Journal of Pediatrics, Cincinnati Children’s Hospital Medical Center, 3333 Burnet Ave., Cincinnati, OH 45229. E-mail: [email protected].
References 1. Bailar JC, Patterson K. Journal peer review: the need for a research agenda. N Engl J Med 1985;312:654-7. 2. Bornmann L, Daniel H-D. The manuscript reviewing process: empirical research on review requests, review sequences, and decision rules in peer review. Libr Information Sci Res 2010;32:5-12. 3. Grainger DW. Peer review as professional responsibility: a quality control system only as good as the participants. Biomaterials 2007;28: 5199-203. 4. Balistreri WF. Landmark, landmine, or landfill? The role of peer review in assessing manuscripts. J Pediatr 2007;151:107-8. 5. Earnshaw JJ, Farndon JR, Guillou PF, Johnson CD, Murie JA, Murray GD. A comparison of reports from referees chosen by authors or journal editors in the peer review process. Ann R Coll Surg Engl 2000;82:133-5. 6. Wager E, Parkin EC, Tamber PS. Are reviewers suggested by authors as good as those chosen by editors? Results of a rater-blinded, retrospective study. BMC 2006;4:13. 7. Rivara FP, Cummings P, Ringold S, Bergman AB, Joffe A, Christakis DA. A comparison of reviewers selected by editors and suggested by authors. J Pediatr 2007;151:107-8. 8. Schroter S, Tite L, Hutchings A, Black N. Differences in review quality and recommendations for publication between peer reviews suggested by authors or by editors. JAMA 2006;295:314-7. 9. Barbour V, Clark J, Peiperly L, Veitch E, Wong M, Yamey G. Making sense of non-financial competing interests. PLoS Medicine 2008;5: 1299-301. 10. Hojat M, Rosenzweig S. Journal peer review in integrative medicine discipline. Semin Integrative Med 2004;2:1-4.
151