- Email: [email protected]
Characterization and Fate of Unpublished Research Articles Reported by Orthopedic Surgery Residency Applicants
ARTICLE IN PRESS ORIGINAL REPORTS
Characterization and Fate of Unpublished Research Articles Reported by Orthopedic Surgery Residency Applicants Nicholas J. Lemme, MD, Neill Y. Li, MD, John Twomey-Kozak, BA, Steven F. DeFroda, MD, Zachary Silber, BS, Alan H. Daniels, MD, and Craig P. Eberson, MD Department of Orthopaedic Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island OBJECTIVE: As the competitiveness of matching to an
orthopedic residency continues to increase, applicants attempt to bolster their application by participating in research activities. However, due to the brief duration of medical school, applicants’ articles may not be published at the time of applying. The purpose of this study was to identify projects that were listed under “publications-other than published” within Electronic Residency Application System (ERAS) applications of prospective orthopedic surgery residents to determine the rate and time of these projects to future publication in a peer-reviewed journal. Program directors can use this information to help interpret the importance of such articles on the applications of future residency candidates. DESIGN: Retrospective study of prospective residents’
applications to a single orthopedic residency program during the 2014 to 2015 application cycle were reviewed to identify articles designated as “other than published." Articles which advanced to official publication were confirmed using the Embase, PubMed, and Google Scholar databases. Applicant and article characteristics were recorded to identify variables associated with an increased proportion of articles that were able to be confirmed. PARTICIPANTS: Prospective residents to a single ortho-
pedic residency program during the 2014 to 2015 application cycle. Investigation performed at: Department of Orthopedics, Alpert Medical School of Brown University, Rhode Island Hospital. 593 Eddy St, Providence, RI 02903. Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Correspondence: Inquiries to Nicholas Lemme, MD, Department of Orthopaedic Surgery, Warren Alpert Medical School of Brown University, 593 Eddy St, Coop 1st Floor, Providence, RI 02903; e-mail: [email protected]
RESULTS: A total of 1957 article titles were listed amongst 563 applicants, with 48% of applicants (n = 271) having at least one peer-reviewed article listed as “other than published.” Overall, 34.2% (709) of the articles were designated as being unpublished including 208 listed as accepted/in-press and 501 listed as submitted/under review. Of the accepted/in-press articles, 90.7% (n = 189) were able to be confirmed as successfully published papers, compared to 63.4% (n = 318) of articles designated as submitted/under review (p < 0.001). Factors predictive of articles which advanced to official publication were being accepted/in-press at the time of applying, a lower United States Medical Licensing Exam (USMLE) Step 1 score, and articles on orthopedic topics. CONCLUSIONS: Nearly one-half of orthopedic resi-
dency applicants report unpublished research articles on their ERAS application. While 90.7% of the articles listed as being accepted/ in press were eventually published, less than two-thirds of the articles designated as being in submission/under-review progressed to official publication. ( J Surg Ed 000:1 6. Ó 2019 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: ERAS, residency, orthopedics, application,
research, publication COMPETENCIES: Interpersonal and Communication
Skills, Practice-Based Learning and Improvement, Systems-Based Practice
INTRODUCTION Matching into an orthopedic surgery residency program has become increasingly competitive. As the number
Journal of Surgical Education © 2019 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jsurg.2019.11.005
1
ARTICLE IN PRESS and quality of applicants continue to rise, programs are in search of objective methods to successfully select applicants that will excel as residents.1 During application review, medical school grades, AOA status, USMLE scores, and research publications drive interview consideration at many programs.2 In a 2016 survey of orthopedic surgery residency program directors, 65% cited research involvement as being highly important when determining which applicants would receive an invitation to interview.2 Research can be a time-consuming process and many projects may not be formally published in time for residency application deadlines. To combat this, applicants are allowed to list submitted works as “accepted,” “in-press,” or “in submission/under review” in the designated area on the ERAS application under “Peer-Reviewed Articles-Other Than Published.” To date, no previous study has characterized these publications nor determined what proportion advance to be successfully published. This leaves those who review applications of prospective residents skeptical when interpreting and assigning credibility to projects listed as “in submission.”4 The purpose of this study was to identify projects that were listed under “publicationsother than published” within ERAS applications in order to determine the rate and time of these projects to future publication in a peer-reviewed journal. Program directors can use this information to help interpret the importance of such articles on the applications of future residency candidates.
METHODS Data Collection Prospective applicants to a single orthopedic residency program during the 2014 to 2015 application cycle were reviewed following IRB approval. This application year was selected to allow for a minimum of 5 years to capture publications that subsequently progressed to an official publication. The following characteristics were extracted from each application; applicant gender, USMLE Step 1 score, USMLE Step 2 Score, Alpha Omega Alpha Honor Society Status, whether the applicant was from a Top 25 Ranked US News & World Report Medical School (Research) and/or a medical school associated with a Top 25 Ranked Doximity Orthopedic program (website, reputation).16,6 Research numbers included the number of peerreviewed articles published and the number listed in the other than published category. The title of each listed article identified as being “other than published” was recorded. For each article, 2 authors determined if the manuscript topic was related to orthopedics using the
2
title, abstract and full text, if necessary. Two authors (JTK and ZS) identified all of the articles listed as “other than published” that went on to successful publication using Embase, PubMed, and Google Scholar databases. Briefly, the article title was copied and pasted from the respective application into all 3 search engines. If the article was not able to be identified, keywords from the manuscript title and any combination of the authors’ last names were entered into both search engines. The articles that still could not be identified underwent the above protocol by a separate third author (NJL blinded.) to confirm lack of publication. Articles were listed as a confirmed publication under the following circumstances (1) the article title was identical to the one listed on the application (2) if the title was not identical, the abstract and/or full text was read to determine if the research focus was the same as the one listed on the application (3) the applicant was listed as an author. For each article that progressed to successful publication the time to publication was determined from the time of application submission (September 2014) until the time of online publication (in months). Statistical Analysis Shapiro-Wilk testing was performed to determine normality of data. Statistical difference amongst groups was determined using either independent t test or KruskalWallis test for parametric and nonparametric data, respectively. Binary logistic regression was used to which determine applicant and article variables were independently predictive of progression of an article designated as “other than published” to official publication. Statistical analyses were performed using SPSS V.24 (SPSS Inc., Chicago, IL). A p value < 0.05 was considered statistically significant.
RESULTS Applicant Overview A total of 563 ERAS applications were reviewed, applicants were predominantly male population representing 83% of all applicants. The average Step 1 and 2 scores were 243 and 249, respectively. One a third of eligible applicants were inducted to AOA. (Table 1). A total of 1957 article titles were listed amongst all applicants. Of these articles, 63.8% (n = 1248) were published prior to the ERAS submission deadline and 34.2% (709) were listed in other than published (Fig. 1). Published Articles Applicants had an average of 3.5 listed peer-reviewed articles published at the time of ERAS submission. The
Journal of Surgical Education Volume 00 /Number 00 Month 2019
ARTICLE IN PRESS
TABLE 1. Characteristics of Orthopedic Surgery Candidates at a Single Institution During the 2014 to 2015 Application Cycle 2014-2015 Applicant Characteristics Gender Male Female AOA
Yes No Not applicable
142 286 135 Mean Step 1 Mean Step 2
n = 468 n = 95 US News Top 25 Medical School
Doximity Top 25 Home Program
142 421 0
133 430 0 243 249
FIGURE 1. Articles listed as published or “other than published” on the ERAS applications of prospective residents during the 2014 to 2015 application cycle.
mean number of publications was found to be higher for male applicants (5.9 vs 3.0 publications, p = 0.04) and applicants from a medical school ranked in the top 25 by USNWR (research) (5.0 vs 3.0, p < 0.01). The ranking of an applicant’s home orthopedic program, USMLE Step 1 score, USMLE Step 2 score, or AOA status had no significant effect on an applicant’s overall number of publications listed. “Other Than Published” Articles Of the 563 applicants, 48% (n = 271) were found to have at least one peer-reviewed article listed as “other than published.” Of these applicants 20.8% (n = 117) and 41.6% (n = 234) had at least 1 article designated as accepted/in-press or submitted/under review, respectively. Of the 709 articles being listed as “other than published,” 208 were listed as accepted/in-press and 501 were listed as submitted/under review. Of the accepted/ in-press articles, 90.7% (n = 189) were able to be confirmed as successfully published papers, while only 63.4% (n = 318) of articles designated as submitted/ under review went on to successful publication (p < 0.001) (Fig. 2). For 27.7% (n = 75) of applicants, none of the articles designated as “other than published” progressed to a confirmed publication (Fig. 3). The mean time to publication from the time of ERAS submission was significantly less for articles listed as accepted compared to those designated as being submitted (5.6 months vs 13.7 months, p < 0.01). Of the listed articles, those related to orthopedics accounted for 78% (n = 552) compared to other specialties at 22%
FIGURE 2. Overview of articles designated as “Other than Published” on the ERAS applications of prospective residents during the 2014 to 2015 application cycle.
Journal of Surgical Education Volume 00 /Number 00 Month 2019
3
ARTICLE IN PRESS
FIGURE 3. Proportion of applicants’ articles designated as submitted/under-review which advanced to publication.
TABLE 2. Characteristics of Articles Listed as “Other Than Published” and Factors Associated With Conversion to Publication Publications Total Published Articles-other than published Accepted/In-press Submitted/underreview Total: Orthopedicrelated Not-related
1248 208 501
% Converted to publication 90.7% 63.4%
709 552 157
Factors Associated With Conversion to Publication
Odds Ratio (95% CI)
p Value
AOA status USNWR top 25 Doximity top 25 home program Gender USMLE Step 1 score USMLE Step 2 score Orthopedic topic #Published at submission
0.79 (0.48-1.31) 3.26 (1.85-5.74) 1.04 (0.62-1.74)
0.36 <0.001* 0.89
1.29 (0.70-2.41) 0.98 (0.95-0.99)
0.42 0.01*
1.01 (0.99-1.03)
0.58
1.63 (1.08-2.50) 1.02 (1.00-1.05)
0.02* 0.11
*p < 0.05 considered to be statistically significant.
4
(n = 157). About 75.5% of orthopedic-related articles were more likely to progress to official publication, compared to those not related to orthopedics (65.4%). Regression analysis demonstrated applicants from top 25 research medical schools to be more likely to advance their articles designated as “other than published” to official publication (Odds ratio (OR) 3.26, 95% confidence interval (CI) 1.85-5.74, p < 0.001), while applicants with higher step 1 scores were less likely (OR 0.98, 95% CI 0.95-0.99, p = 0.009). Furthermore, articles related to orthopedics were more likely to progress to official publication, compared to those not related to orthopedics (OR 1.63, 95% CI 1.08-2.50, p = 0.02). (Table 2).
DISCUSSION Upon review of factors that program directors consider important when ranking applicants, multiple studies cite research activity as being one of the most important.1,19,9 Consequently, the mean number of publications cited on applicants’ curriculum vitaes at the time of applying continues to increase.7 Recently, Ramkumar et al. found that 40% of orthopedic residency applicants take dedicated research years in hopes to strengthen their application. 10 However, not all cited publications should be viewed equally as not all of them may be published by the residency application deadline. In the present study, 48% of applicants were found have at least one unpublished research article on their
Journal of Surgical Education Volume 00 /Number 00 Month 2019
ARTICLE IN PRESS residency application. While 90.7% of articles listed as being accepted/ in press were eventually published, only 63.4% of articles designated as being in submission/ under-review progressed to official publication. The factors predictive of articles which went on to official publication were having the designation of accepted/in-press at the time of ERAS submission, articles examining orthopedic topics, a lower USMLE step 1 score, and applicants from a top 25 ranked research medical school. Historically, there has been a concerning amount of misrepresented publications on the ERAS applications of orthopedic applicants with 18.0% and 20.6% of articles being misrepresented in 1999 and 2007, respectively.3,4 Interestingly, since 2014 when ERAS began to require the PubMed identifier for all articles designated as published, misrepresentation of research decreased drastically to 1.2%.4 It may be speculated that prior to this requirement, given the competitiveness of obtaining a residency spot, applicants may have purposely exaggerated their scholarly activity in order to strengthen their application. It is possible that applicants are now exaggerating or misrepresenting their research by falsely labeling something as either being accepted/in-press or in submission/under-review. In this study, 27.7% of applicants had articles listed as other than published, in which no listed article went on to be confirmed as an official publication. Understandably, authors in the aforementioned study have encouraged program directors to interpret listed articles that are other than published with caution and recommend the requirement that all in-press articles be accompanied by a verification letter from the journal. Yet, this investigation revealed that a large proportion of the articles in the present study (90.7%) listed as accepted/in-press were able to be confirmed at a mean of 5.6 months after ERAS submission. In comparison, only 63.4% of articles designated as submitted/underreview could be confirmed, suggesting that such articles should be interpreted with caution, or not be allowed to be listed on the application at all. In a 2010 meta-analysis of 13 studies that investigated research misrepresentations among residency and fellowship applicants to 7 different specialties, the average misrepresentation rate was found to be 14.5%, with orthopedic applicants having one of the highest rates of misrepresentation.11 Applicant misrepresentation of research can come in many different forms with the most common form of research misrepresentations being; listing nonexistent articles, followed by errors in authorship order, nonauthorship, and reporting abstracts as articles.11 Interestingly none of these studies examined articles designated as in-submission and only 8 of the 13 studies included articles listed as accepted/inpress.11 In the present study 48% of applicants had at
Journal of Surgical Education Volume 00 /Number 00 Month 2019
least one article designated as either in submission or inpress, with not a single article progressing to publication in 27.7% of such applicants. This indicates that there is a large form of misrepresentation that has gone unnoticed and suggested that most studies to date have underestimated the true rate of research misrepresentation in residency applicants. The only variables found to be predictive of whether an article would progress to official publication were having the designation of accepted/in-press at the time of ERAS submission, articles on orthopedic topics, a lower USMLE step 1 score and being an applicant from a top 25 ranked medical school for research. This suggests that applicants may be more likely to follow through with a project if it is related to orthopedics as it is beneficial to their future career as an orthopedic surgeon, while applicants may perceive their work which is unrelated to orthopedics less important once they obtain a residency position. Furthermore, applicants with lower Step 1 scores frequently try to bolster their application in other ways such as research to make up for their subpar test scores. Since many applicants do not receive their test scores until the end of their second or third year of medical school, they may not increase their research productivity until this time resulting in many articles not being accepted and published until after their applications are submitted. This project has several potential limitations. First, the study is retrospective in nature and only applicants during the 2014 to 2015 application pool were included in the study and therefore we are unable to comment on any time-related trends. Additionally, this data is based on applications to a single academic institution resulting in possible selection bias, however our sample includes more than 50% (563 applicants out of 810 total US allopathic applicants) of the applicants during the respective application cycle, and is likely representative of the applicant pool.12 Finally, despite a thorough search method, it is possible some articles were missed, especially those published in “gray literature” or in nonEnglish journals. In conclusion, this investigation revealed that nearly one-half of all orthopedic residency applicants report unpublished research articles on their ERAS application with less than two-thirds of articles designated as being in submission/under-review progressing to official publication. In addition, 27.7% of applicants with articles designated as in submission /under review did not have a single article which progressed to formal publication. Such data suggests that this may be the most common form of misrepresentation in orthopedic residency applicants and residency selection committees should interpret unpublished citations with caution.
5
ARTICLE IN PRESS REFERENCES 1. Turner NS, Shaughnessy WJ, Berg EJ, Larson DR,
Hanssen AD. A quantitative composite scoring tool for orthopaedic residency screening and selection. In: Clinical Orthopaedics and Related Research; 2006. p. 50–55. https://doi.org/ 10.1097/01.blo.0000224042.84839.44. 2. National Resident Matching Program. Results of the
2016 NRMP Program Director Survey. 2016;79-80. http://www.nrmp.org/wp-content/uploads/2016/ 09/NRMP-2016-Program-Director-Survey.pdf. 3. Crosby LA. Misrepresentation of research criteria by
orthopaedic residency applicants. J Bone Jt Sur Am. 2016;82:1512–1513. https://doi.org/10.2106/ 00004623-200010000-00028. 4. Meeks BD, Kiskaddon EM, Burton MG Jr, Froehle
AW, Crosby LA, Laughlin RT. Update on misrepresentation of research publications among orthopaedic surgery residency applicants. J Bone Joint Surg. 2018;121:1–6. 5. US News & World Report. Best Medical Schools:
Research. https://www.usnews.com/best-graduateschools/top-medical-schools. Accessed January 3, 2019. 6. Doximity. Residency navigator: orthopaedic surgery. 7. DePasse JM, Palumbo MA, Eberson CP, Daniels AH.
Academic characteristics of orthopaedic surgery
6
residency applicants from 2007 to 2014. J Bone Joint Surg. 2016;98:788–795. 8. Egol KA, Collins J, Zuckerman JD. Success in ortho-
paedic training: resident selection and predictors of quality performance. J Am Acad Orthop Surg. 2011. https://doi.org/10.5435/00124635-20110200000002. 9. Schenker ML, Baldwin KD, Israelite CL, Levin LS,
Mehta S, Ahn J. Selecting the best and brightest: a structured approach to orthopedic resident selection. J Surg Educ. 2016. https://doi.org/10.1016/j. jsurg.2016.04.004. 10. Ramkumar PN, Navarro SM, Chughtai M, Haeberle HS,
Taylor SA, Mont MA. The orthopaedic surgery residency application process: an analysis of the applicant experience. J Am Acad Orthop Surg. 2018;26:537– 544. https://doi.org/10.5435/JAAOS-D-16-00835. 11. Wiggins MN. A meta-analysis of studies of publication
misrepresentation by applicants to residency and fellowship programs. Acad Med. 2010;85:1470–1474. https://doi.org/10.1097/ACM.0b013e3181e2cf2b. 12. NRMP. National Resident Matching Program; Associ-
ation of American Medical Colleges. Charting Outcomes in the Match—characteristics of applicants who matched to their preferred specialty in the 2014 main residency match. 2014. http://www. nrmp.org/wp-content/uploads/2014/09/ChartingOutcomes-2014-Final.pdf.
Journal of Surgical Education Volume 00 /Number 00 Month 2019
Characterization and Fate of Unpublished Research Articles Reported by Orthopedic Surgery Residency Applicants Nicholas J. Lemme, MD, Neill Y. Li, MD, John Twomey-Kozak, BA, Steven F. DeFroda, MD, Zachary Silber, BS, Alan H. Daniels, MD, and Craig P. Eberson, MD Department of Orthopaedic Surgery, Warren Alpert Medical School of Brown University, Providence, Rhode Island OBJECTIVE: As the competitiveness of matching to an
orthopedic residency continues to increase, applicants attempt to bolster their application by participating in research activities. However, due to the brief duration of medical school, applicants’ articles may not be published at the time of applying. The purpose of this study was to identify projects that were listed under “publications-other than published” within Electronic Residency Application System (ERAS) applications of prospective orthopedic surgery residents to determine the rate and time of these projects to future publication in a peer-reviewed journal. Program directors can use this information to help interpret the importance of such articles on the applications of future residency candidates. DESIGN: Retrospective study of prospective residents’
applications to a single orthopedic residency program during the 2014 to 2015 application cycle were reviewed to identify articles designated as “other than published." Articles which advanced to official publication were confirmed using the Embase, PubMed, and Google Scholar databases. Applicant and article characteristics were recorded to identify variables associated with an increased proportion of articles that were able to be confirmed. PARTICIPANTS: Prospective residents to a single ortho-
pedic residency program during the 2014 to 2015 application cycle. Investigation performed at: Department of Orthopedics, Alpert Medical School of Brown University, Rhode Island Hospital. 593 Eddy St, Providence, RI 02903. Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Correspondence: Inquiries to Nicholas Lemme, MD, Department of Orthopaedic Surgery, Warren Alpert Medical School of Brown University, 593 Eddy St, Coop 1st Floor, Providence, RI 02903; e-mail: [email protected]
RESULTS: A total of 1957 article titles were listed amongst 563 applicants, with 48% of applicants (n = 271) having at least one peer-reviewed article listed as “other than published.” Overall, 34.2% (709) of the articles were designated as being unpublished including 208 listed as accepted/in-press and 501 listed as submitted/under review. Of the accepted/in-press articles, 90.7% (n = 189) were able to be confirmed as successfully published papers, compared to 63.4% (n = 318) of articles designated as submitted/under review (p < 0.001). Factors predictive of articles which advanced to official publication were being accepted/in-press at the time of applying, a lower United States Medical Licensing Exam (USMLE) Step 1 score, and articles on orthopedic topics. CONCLUSIONS: Nearly one-half of orthopedic resi-
dency applicants report unpublished research articles on their ERAS application. While 90.7% of the articles listed as being accepted/ in press were eventually published, less than two-thirds of the articles designated as being in submission/under-review progressed to official publication. ( J Surg Ed 000:1 6. Ó 2019 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: ERAS, residency, orthopedics, application,
research, publication COMPETENCIES: Interpersonal and Communication
Skills, Practice-Based Learning and Improvement, Systems-Based Practice
INTRODUCTION Matching into an orthopedic surgery residency program has become increasingly competitive. As the number
Journal of Surgical Education © 2019 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jsurg.2019.11.005
1
ARTICLE IN PRESS and quality of applicants continue to rise, programs are in search of objective methods to successfully select applicants that will excel as residents.1 During application review, medical school grades, AOA status, USMLE scores, and research publications drive interview consideration at many programs.2 In a 2016 survey of orthopedic surgery residency program directors, 65% cited research involvement as being highly important when determining which applicants would receive an invitation to interview.2 Research can be a time-consuming process and many projects may not be formally published in time for residency application deadlines. To combat this, applicants are allowed to list submitted works as “accepted,” “in-press,” or “in submission/under review” in the designated area on the ERAS application under “Peer-Reviewed Articles-Other Than Published.” To date, no previous study has characterized these publications nor determined what proportion advance to be successfully published. This leaves those who review applications of prospective residents skeptical when interpreting and assigning credibility to projects listed as “in submission.”4 The purpose of this study was to identify projects that were listed under “publicationsother than published” within ERAS applications in order to determine the rate and time of these projects to future publication in a peer-reviewed journal. Program directors can use this information to help interpret the importance of such articles on the applications of future residency candidates.
METHODS Data Collection Prospective applicants to a single orthopedic residency program during the 2014 to 2015 application cycle were reviewed following IRB approval. This application year was selected to allow for a minimum of 5 years to capture publications that subsequently progressed to an official publication. The following characteristics were extracted from each application; applicant gender, USMLE Step 1 score, USMLE Step 2 Score, Alpha Omega Alpha Honor Society Status, whether the applicant was from a Top 25 Ranked US News & World Report Medical School (Research) and/or a medical school associated with a Top 25 Ranked Doximity Orthopedic program (website, reputation).16,6 Research numbers included the number of peerreviewed articles published and the number listed in the other than published category. The title of each listed article identified as being “other than published” was recorded. For each article, 2 authors determined if the manuscript topic was related to orthopedics using the
2
title, abstract and full text, if necessary. Two authors (JTK and ZS) identified all of the articles listed as “other than published” that went on to successful publication using Embase, PubMed, and Google Scholar databases. Briefly, the article title was copied and pasted from the respective application into all 3 search engines. If the article was not able to be identified, keywords from the manuscript title and any combination of the authors’ last names were entered into both search engines. The articles that still could not be identified underwent the above protocol by a separate third author (NJL blinded.) to confirm lack of publication. Articles were listed as a confirmed publication under the following circumstances (1) the article title was identical to the one listed on the application (2) if the title was not identical, the abstract and/or full text was read to determine if the research focus was the same as the one listed on the application (3) the applicant was listed as an author. For each article that progressed to successful publication the time to publication was determined from the time of application submission (September 2014) until the time of online publication (in months). Statistical Analysis Shapiro-Wilk testing was performed to determine normality of data. Statistical difference amongst groups was determined using either independent t test or KruskalWallis test for parametric and nonparametric data, respectively. Binary logistic regression was used to which determine applicant and article variables were independently predictive of progression of an article designated as “other than published” to official publication. Statistical analyses were performed using SPSS V.24 (SPSS Inc., Chicago, IL). A p value < 0.05 was considered statistically significant.
RESULTS Applicant Overview A total of 563 ERAS applications were reviewed, applicants were predominantly male population representing 83% of all applicants. The average Step 1 and 2 scores were 243 and 249, respectively. One a third of eligible applicants were inducted to AOA. (Table 1). A total of 1957 article titles were listed amongst all applicants. Of these articles, 63.8% (n = 1248) were published prior to the ERAS submission deadline and 34.2% (709) were listed in other than published (Fig. 1). Published Articles Applicants had an average of 3.5 listed peer-reviewed articles published at the time of ERAS submission. The
Journal of Surgical Education Volume 00 /Number 00 Month 2019
ARTICLE IN PRESS
TABLE 1. Characteristics of Orthopedic Surgery Candidates at a Single Institution During the 2014 to 2015 Application Cycle 2014-2015 Applicant Characteristics Gender Male Female AOA
Yes No Not applicable
142 286 135 Mean Step 1 Mean Step 2
n = 468 n = 95 US News Top 25 Medical School
Doximity Top 25 Home Program
142 421 0
133 430 0 243 249
FIGURE 1. Articles listed as published or “other than published” on the ERAS applications of prospective residents during the 2014 to 2015 application cycle.
mean number of publications was found to be higher for male applicants (5.9 vs 3.0 publications, p = 0.04) and applicants from a medical school ranked in the top 25 by USNWR (research) (5.0 vs 3.0, p < 0.01). The ranking of an applicant’s home orthopedic program, USMLE Step 1 score, USMLE Step 2 score, or AOA status had no significant effect on an applicant’s overall number of publications listed. “Other Than Published” Articles Of the 563 applicants, 48% (n = 271) were found to have at least one peer-reviewed article listed as “other than published.” Of these applicants 20.8% (n = 117) and 41.6% (n = 234) had at least 1 article designated as accepted/in-press or submitted/under review, respectively. Of the 709 articles being listed as “other than published,” 208 were listed as accepted/in-press and 501 were listed as submitted/under review. Of the accepted/ in-press articles, 90.7% (n = 189) were able to be confirmed as successfully published papers, while only 63.4% (n = 318) of articles designated as submitted/ under review went on to successful publication (p < 0.001) (Fig. 2). For 27.7% (n = 75) of applicants, none of the articles designated as “other than published” progressed to a confirmed publication (Fig. 3). The mean time to publication from the time of ERAS submission was significantly less for articles listed as accepted compared to those designated as being submitted (5.6 months vs 13.7 months, p < 0.01). Of the listed articles, those related to orthopedics accounted for 78% (n = 552) compared to other specialties at 22%
FIGURE 2. Overview of articles designated as “Other than Published” on the ERAS applications of prospective residents during the 2014 to 2015 application cycle.
Journal of Surgical Education Volume 00 /Number 00 Month 2019
3
ARTICLE IN PRESS
FIGURE 3. Proportion of applicants’ articles designated as submitted/under-review which advanced to publication.
TABLE 2. Characteristics of Articles Listed as “Other Than Published” and Factors Associated With Conversion to Publication Publications Total Published Articles-other than published Accepted/In-press Submitted/underreview Total: Orthopedicrelated Not-related
1248 208 501
% Converted to publication 90.7% 63.4%
709 552 157
Factors Associated With Conversion to Publication
Odds Ratio (95% CI)
p Value
AOA status USNWR top 25 Doximity top 25 home program Gender USMLE Step 1 score USMLE Step 2 score Orthopedic topic #Published at submission
0.79 (0.48-1.31) 3.26 (1.85-5.74) 1.04 (0.62-1.74)
0.36 <0.001* 0.89
1.29 (0.70-2.41) 0.98 (0.95-0.99)
0.42 0.01*
1.01 (0.99-1.03)
0.58
1.63 (1.08-2.50) 1.02 (1.00-1.05)
0.02* 0.11
*p < 0.05 considered to be statistically significant.
4
(n = 157). About 75.5% of orthopedic-related articles were more likely to progress to official publication, compared to those not related to orthopedics (65.4%). Regression analysis demonstrated applicants from top 25 research medical schools to be more likely to advance their articles designated as “other than published” to official publication (Odds ratio (OR) 3.26, 95% confidence interval (CI) 1.85-5.74, p < 0.001), while applicants with higher step 1 scores were less likely (OR 0.98, 95% CI 0.95-0.99, p = 0.009). Furthermore, articles related to orthopedics were more likely to progress to official publication, compared to those not related to orthopedics (OR 1.63, 95% CI 1.08-2.50, p = 0.02). (Table 2).
DISCUSSION Upon review of factors that program directors consider important when ranking applicants, multiple studies cite research activity as being one of the most important.1,19,9 Consequently, the mean number of publications cited on applicants’ curriculum vitaes at the time of applying continues to increase.7 Recently, Ramkumar et al. found that 40% of orthopedic residency applicants take dedicated research years in hopes to strengthen their application. 10 However, not all cited publications should be viewed equally as not all of them may be published by the residency application deadline. In the present study, 48% of applicants were found have at least one unpublished research article on their
Journal of Surgical Education Volume 00 /Number 00 Month 2019
ARTICLE IN PRESS residency application. While 90.7% of articles listed as being accepted/ in press were eventually published, only 63.4% of articles designated as being in submission/ under-review progressed to official publication. The factors predictive of articles which went on to official publication were having the designation of accepted/in-press at the time of ERAS submission, articles examining orthopedic topics, a lower USMLE step 1 score, and applicants from a top 25 ranked research medical school. Historically, there has been a concerning amount of misrepresented publications on the ERAS applications of orthopedic applicants with 18.0% and 20.6% of articles being misrepresented in 1999 and 2007, respectively.3,4 Interestingly, since 2014 when ERAS began to require the PubMed identifier for all articles designated as published, misrepresentation of research decreased drastically to 1.2%.4 It may be speculated that prior to this requirement, given the competitiveness of obtaining a residency spot, applicants may have purposely exaggerated their scholarly activity in order to strengthen their application. It is possible that applicants are now exaggerating or misrepresenting their research by falsely labeling something as either being accepted/in-press or in submission/under-review. In this study, 27.7% of applicants had articles listed as other than published, in which no listed article went on to be confirmed as an official publication. Understandably, authors in the aforementioned study have encouraged program directors to interpret listed articles that are other than published with caution and recommend the requirement that all in-press articles be accompanied by a verification letter from the journal. Yet, this investigation revealed that a large proportion of the articles in the present study (90.7%) listed as accepted/in-press were able to be confirmed at a mean of 5.6 months after ERAS submission. In comparison, only 63.4% of articles designated as submitted/underreview could be confirmed, suggesting that such articles should be interpreted with caution, or not be allowed to be listed on the application at all. In a 2010 meta-analysis of 13 studies that investigated research misrepresentations among residency and fellowship applicants to 7 different specialties, the average misrepresentation rate was found to be 14.5%, with orthopedic applicants having one of the highest rates of misrepresentation.11 Applicant misrepresentation of research can come in many different forms with the most common form of research misrepresentations being; listing nonexistent articles, followed by errors in authorship order, nonauthorship, and reporting abstracts as articles.11 Interestingly none of these studies examined articles designated as in-submission and only 8 of the 13 studies included articles listed as accepted/inpress.11 In the present study 48% of applicants had at
Journal of Surgical Education Volume 00 /Number 00 Month 2019
least one article designated as either in submission or inpress, with not a single article progressing to publication in 27.7% of such applicants. This indicates that there is a large form of misrepresentation that has gone unnoticed and suggested that most studies to date have underestimated the true rate of research misrepresentation in residency applicants. The only variables found to be predictive of whether an article would progress to official publication were having the designation of accepted/in-press at the time of ERAS submission, articles on orthopedic topics, a lower USMLE step 1 score and being an applicant from a top 25 ranked medical school for research. This suggests that applicants may be more likely to follow through with a project if it is related to orthopedics as it is beneficial to their future career as an orthopedic surgeon, while applicants may perceive their work which is unrelated to orthopedics less important once they obtain a residency position. Furthermore, applicants with lower Step 1 scores frequently try to bolster their application in other ways such as research to make up for their subpar test scores. Since many applicants do not receive their test scores until the end of their second or third year of medical school, they may not increase their research productivity until this time resulting in many articles not being accepted and published until after their applications are submitted. This project has several potential limitations. First, the study is retrospective in nature and only applicants during the 2014 to 2015 application pool were included in the study and therefore we are unable to comment on any time-related trends. Additionally, this data is based on applications to a single academic institution resulting in possible selection bias, however our sample includes more than 50% (563 applicants out of 810 total US allopathic applicants) of the applicants during the respective application cycle, and is likely representative of the applicant pool.12 Finally, despite a thorough search method, it is possible some articles were missed, especially those published in “gray literature” or in nonEnglish journals. In conclusion, this investigation revealed that nearly one-half of all orthopedic residency applicants report unpublished research articles on their ERAS application with less than two-thirds of articles designated as being in submission/under-review progressing to official publication. In addition, 27.7% of applicants with articles designated as in submission /under review did not have a single article which progressed to formal publication. Such data suggests that this may be the most common form of misrepresentation in orthopedic residency applicants and residency selection committees should interpret unpublished citations with caution.
5
ARTICLE IN PRESS REFERENCES 1. Turner NS, Shaughnessy WJ, Berg EJ, Larson DR,
Hanssen AD. A quantitative composite scoring tool for orthopaedic residency screening and selection. In: Clinical Orthopaedics and Related Research; 2006. p. 50–55. https://doi.org/ 10.1097/01.blo.0000224042.84839.44. 2. National Resident Matching Program. Results of the
2016 NRMP Program Director Survey. 2016;79-80. http://www.nrmp.org/wp-content/uploads/2016/ 09/NRMP-2016-Program-Director-Survey.pdf. 3. Crosby LA. Misrepresentation of research criteria by
orthopaedic residency applicants. J Bone Jt Sur Am. 2016;82:1512–1513. https://doi.org/10.2106/ 00004623-200010000-00028. 4. Meeks BD, Kiskaddon EM, Burton MG Jr, Froehle
AW, Crosby LA, Laughlin RT. Update on misrepresentation of research publications among orthopaedic surgery residency applicants. J Bone Joint Surg. 2018;121:1–6. 5. US News & World Report. Best Medical Schools:
Research. https://www.usnews.com/best-graduateschools/top-medical-schools. Accessed January 3, 2019. 6. Doximity. Residency navigator: orthopaedic surgery. 7. DePasse JM, Palumbo MA, Eberson CP, Daniels AH.
Academic characteristics of orthopaedic surgery
6
residency applicants from 2007 to 2014. J Bone Joint Surg. 2016;98:788–795. 8. Egol KA, Collins J, Zuckerman JD. Success in ortho-
paedic training: resident selection and predictors of quality performance. J Am Acad Orthop Surg. 2011. https://doi.org/10.5435/00124635-20110200000002. 9. Schenker ML, Baldwin KD, Israelite CL, Levin LS,
Mehta S, Ahn J. Selecting the best and brightest: a structured approach to orthopedic resident selection. J Surg Educ. 2016. https://doi.org/10.1016/j. jsurg.2016.04.004. 10. Ramkumar PN, Navarro SM, Chughtai M, Haeberle HS,
Taylor SA, Mont MA. The orthopaedic surgery residency application process: an analysis of the applicant experience. J Am Acad Orthop Surg. 2018;26:537– 544. https://doi.org/10.5435/JAAOS-D-16-00835. 11. Wiggins MN. A meta-analysis of studies of publication
misrepresentation by applicants to residency and fellowship programs. Acad Med. 2010;85:1470–1474. https://doi.org/10.1097/ACM.0b013e3181e2cf2b. 12. NRMP. National Resident Matching Program; Associ-
ation of American Medical Colleges. Charting Outcomes in the Match—characteristics of applicants who matched to their preferred specialty in the 2014 main residency match. 2014. http://www. nrmp.org/wp-content/uploads/2014/09/ChartingOutcomes-2014-Final.pdf.
Journal of Surgical Education Volume 00 /Number 00 Month 2019