- Email: [email protected]
Challenges Facing Program Directors in the Urology Match
urologypracticejournal.com
Challenges Facing Program Directors in the Urology Match Steven J. Weissbart, Jeffrey A. Stock and Alan J. Wein* From the Department of Urology, Stony Brook University (SJW), Stony Brook and Icahn School of Medicine at Mount Sinai (JAS), New York, New York, and Division of Urology, Department of Surgery, Hospital of University of Pennsylvania (AJW), Philadelphia, Pennsylvania
Abstract
Abbreviations and Acronyms
Introduction: We investigate urology residency program directors’ perspectives on the current residency matching system. Methods: A survey was emailed to Society of Urology Chairpersons and Program Directors members. The survey queried respondents’ perspectives on the current residency matching system, with special attention to the recent surge in application volume and the usefulness of the Medical Student Performance Evaluation. Participants were also asked about their perspective on a possible application limit for students applying to urology residencies.
MSPE = Medical Student Performance Evaluation USMLE = United States Medical Licensing Examination
Results: A total of 70 members of the Society of Urology Chairpersons and Program Directors responded to the survey. The majority of respondents received more than 200 applications for their program’s residency positions (77.1%) and used a Step 1 cutoff score to screen applications (81.4%). Approximately half of the respondents (51.4%) were in favor of imposing a limit to the number of applications that applicants are permitted to submit. The Medical Student Performance Evaluation was considered important or very important by 20% and 94.3% favored including an applicant’s class rank in the evaluation. An applicant’s projected likelihood of attending a respondent’s program was considered by 76%, and 60% had previously not offered superior candidates interviews because they estimated the applicants were not truly interested in the program. Conclusions: Urology program directors exclude a large number of applications based on board scores and applicants’ perceived levels of interest in the programs. A significant number of program directors favored an application limit as well as including class rank in the Medical Student Performance Evaluation. Key Words: internship and residency; urology; education, medical, graduate
Submitted for publication August 24, 2015. No direct or indirect commercial incentive associated with publishing this article. The corresponding author certifies that, when applicable, a statement(s) has been included in the manuscript documenting institutional review board, ethics committee or ethical review board study approval; principles of Helsinki Declaration were followed in lieu of formal ethics committee approval; 2352-0779/16/36-486/0 UROLOGY PRACTICE Ó 2016 by AMERICAN UROLOGICAL ASSOCIATION EDUCATION
AND
institutional animal care and use committee approval; all human subjects provided written informed consent with guarantees of confidentiality; IRB approved protocol number; animal approved project number. * Correspondence: Department of Urology, Penn Medicine, Perelman Center for Advanced Medicine, West Pavilion, 3rd Floor, 3400 Civic Center Blvd., Philadelphia, Pennsylvania 19104.
RESEARCH, INC.
http://dx.doi.org/10.1016/j.urpr.2015.10.008 Vol. 3, 486-492, November 2016 Published by Elsevier
Challenges for Program Directors in Urology Match
The number of applications that medical students are submitting for urology residency positions has dramatically increased in recent decades.1,2 While in 1995 applicants applied to an average of 28.4 programs, for the 2014-2015 match cycle applicants submitted a mean of 63 applications.1,2 This equated to an applicant, on average, applying to more than half the number of urology programs available. Aside from the financial cost of these applications for medical students,3 and the time commitment required by program directors to review these applications, these additional applications may most importantly preclude program directors from thoroughly reviewing each component of the application and, thereby, alter the paradigm by which applicants are selected for urology residency. In addition to the increase in application volume, urology residency program directors may also find letters of recommendation and the MSPE (Dean’s letter) to be challenging to interpret. The MSPE summarizes a student’s medical school record and is an important component of the residency application. While it is asked that the MSPE include a comparative analysis of a medical student’s performance in relation to his/her classmates,4 recent data have suggested that the majority of MSPEs do not include such information.5 Furthermore, medical schools use different language in their descriptions of applicants (eg superb, excellent, distinguished), thereby making it difficult for program directors to compare applicants according to the MSPE. With the high application volume, the need for MSPEs that succinctly express medical student performance has become even more pronounced. To ensure that applicants and programs are matched most appropriately and that the best medical students are selected for the field of urology, understanding program directors’ methods of applicant selection and their perspective on the current system is important. We chose to investigate urology program directors’ perspectives on a possible application limit for medical students applying to urology residencies as well as their attitudes toward the MSPE. We hypothesized that program directors would be in favor of an application limit and that the MSPE has become less important to program directors in the current application process.
Materials and Methods
A newly developed survey was emailed to 179 urology residency program directors and/or chairpersons through the Society of Urology Chairpersons and Program Directors listserv in April 2015. The survey included 2 lines of questions regarding 1) the imposition of an application limit to the current application system, and 2) the usefulness of
487
the current MSPEs (see Appendix). SurveyMonkeyÒ was used to generate and administer the survey. Descriptive statistics were used to characterize respondents’ survey answers. Subsequently, respondents were then divided into the groups of 1) those in favor of an application limit and 2) those not in favor of an application limit. Self-reported program competitiveness, received application volume and use of a USMLE cutoff score to screen applications were then compared between the 2 groups using chi-squared tests. StataÒ 13.1 was used for statistical analysis.
Results
The study included 70 (39%) urology chairpersons and/or program directors who responded to the survey. The survey respondents were from urology residency programs across the country (Northeastern 4%, New England 9%, New York 14%, Mid Atlantic 9%, Southeastern 19%, South Central 11%, North Central 21%, Western 13%) and most respondents considered their program to be of average (42.9%) or above average (54.3%) competitiveness for gaining acceptance. Of the survey respondents 54 (77.1%) received more than 200 applications during the 2014-2015 match cycle and 57 (81.4%) used a USMLE Step 1 cutoff score to determine which applications to review in further detail. Respondents who received more than 200 applications were not more likely to use a USMLE cutoff score than those who received 200 applications or less (p¼0.48). Of the respondents using a USMLE cutoff score 14 (24.6%) used an approximate score of 220, 24 (42.1%) used an approximate score of 230, 17 (29.8%) used an approximate score of 240 and 2 (3.5%) used an approximate score of 250. Figure 1 shows the percentages of applications that respondents eliminated by a USMLE Step 1 score or another objective parameter (eg Alpha Omega Alpha status). More than half of the respondents eliminated approximately 40% or more of applications before reviewing them in further detail. Except for applications that were initially excluded, 21 (30%) respondents spent more than 10 minutes reviewing each application, 34 (48.6%) spent 6 to 10 minutes reviewing each application and 14 (20%) spent 1 to 5 minutes reviewing an application. Most institutions had multiple individuals review applications for interviewing selection (fig. 1). The idea of imposing a limit on the number of applications that applicants are permitted to submit for the urology match was favored by 36 (51.4%) of the respondents. Of those respondents in favor of an application limit slightly
488
Challenges for Program Directors in Urology Match
Figure 1. Response to survey question “What percentage of applications did you eliminate by a USMLE cutoff score or another objective parameter before you began reviewing applications in more detail?” (A) and number of individuals reviewing applications at survey respondents’ institutions (B).
more than half (51.3%) believed that an application limit of approximately 20 would be reasonable (fig. 2). There were no differences in self-reported program competitiveness, volume of applications received or use of a USMLE cutoff score between respondents who favored the imposition of an application limit and those who did not, although a relationship between application volume and favorability of an application limit approached significance (p¼0.08, ie respondents who received more applications were more likely to be in favor of an application limit) (see table). The MSPE was considered important or very important for applicant assessment by 14 (20%), moderately important by 19 (27.1%) and of little importance or unimportant by 37 (52.9%) respondents. Figure 3 illustrates how often respondents found meaningful information in the MSPE, with 13 (18.6%) respondents citing that the MSPE very frequently or frequently contained meaningful information. The inclusion of an applicant’s class rank in the MSPE was favored by 66 (94.3%) respondents. Most respondents (58, 82.9%) thought the MSPEs would be most useful if received by September (fig. 3).
An applicant’s projected likelihood of attending a respondent’s program (if offered a position) was considered by 53 (76%) respondents, and 44 (63%) respondents considered an applicant’s geographic history (ie medical school location, hometown, location of away rotations) in their selection of candidates to interview (fig. 4). More than half of the respondents (42, 60%) had previously not offered superior candidates interviews because they estimated the applicants were not truly interested in the program (fig. 4). The relationship between respondents’ perceived level of strength of an applicant and their impression of the same individual as a resident was thought to weakly, moderately and strongly correlate by 7 (10%), 48 (68.6%) and 15 (21.4%) respondents, respectively.
Discussion
In this study of program directors’ methods of selecting applicants to interview and their sentiments toward the MSPE, we found that urology residency program directors exclude a significant percentage of applications based on USMLE Step 1 score, do not overwhelmingly consider
Table. Characteristics of respondents according to favorability of an application limit No. Favor Application Limit (%)
Figure 2. Reasonable application limit number among respondents in favor of application limit.
Received more than 300 applications Used USMLE cutoff score Top third of program competitiveness
No. Do Not Favor Application Limit (%)
p Value
13 (36.1)
6 (17.6)
0.08
28 (77.8) 19 (52.7)
29 (85.3) 19 (55.9)
0.42 0.79
Challenges for Program Directors in Urology Match
489
Figure 3. Response to survey questions “How often did the Dean’s letter provide meaningful information about the applicant?” (A) and “When, to be most useful, should the Dean’s letter be received?” (B).
the MSPEs to have meaningful information and commonly take into account an applicant’s perceived level of interest in their program during their decision of whom to interview. In addition, the idea of limiting the number of applications that applicants are allowed to submit was favorably considered by more than half of those surveyed. The urology matching system was designed to facilitate the efficient pairings of applicants and programs, and a cardinal feature of the system has been its ability to produce pairings that optimize applicant and program preferences.6 Assuming program directors are rational, they should prefer to interview the top candidates. However, with the high number of applications sent to program directors, they may have to consider factors other than scholastic achievement in their selection of candidates to interview.
In this study 76% of respondents considered an applicant’s perceived level of interest in their program during their deliberations on whom to interview and 60% had previously not interviewed superior candidates based on an applicant’s perceived lack of interest in the program. The large volume of applications appears to have congested the urology residency matching market, whereby superior applicants do not obtain interviews from programs of high interest and program directors lose the opportunity to recruit these top applicants to their programs. Limiting the number of applications that applicants may submit for urology residency positions might clarify their true levels of interest in programs, provide applicants with cost savings and would not appear to decrease the
Figure 4. Response to survey questions “To what extent did you consider a candidate’s likeliness to want to come to your program to determine who would receive an interview offer?” (A) and “Have you ever not offered a superior candidate an interview because you estimated that he or she was not really interested in your program?” (B).
490
Challenges for Program Directors in Urology Match
percentage of applicants who match.7 As demonstrated by this study, program directors are currently using factors such as applicants’ geographic histories in their selection of candidates to interview, but is it not possible that a superior applicant from the Northwest would be thrilled to attend an elite program on the East Coast? Interestingly, in a pilot study investigating the factors that applicants considered when formulating their rank lists, family obligations ranked as one of the least important factors (16 out of 18).8 Therefore, a residency near home might not be essential to most applicants. With the imposition of a reasonably set application limit, applying to a program alone would signal a genuine interest in the program and could, perhaps, take away the guesswork of determining applicants’ interest levels. This could, in effect, help applicants attain interview offers from programs in which they are sincerely interested. Not surprisingly, more than half of the respondents in this study were in favor of imposing an application limit to the urology match. Applicants in the field of urology may also be disadvantaged by vague letters of recommendation and MPSEs that do not comport with the Association of American Medical Colleges (AAMC) MSPE preparation guide. In 2002 the AAMC issued a guide for preparing the MSPE to ensure that the MSPE would be meaningful to members of the graduate medical education community (among other reasons).4 According to the guide, the summary paragraph of the MSPE should include a student’s comparative medical school performance in relation to his/her classmates, and MSPE appendices should include graphic comparisons of a student’s performance in relation to his or her classmates in several categories (basic science coursework, third year clerkships etc). Unfortunately, a recent study showed that only 9% of medicals schools included this complete comparative information in their MSPEs, and there was significant heterogeneity in the descriptive words used in MSPEs (eg outstanding, excellent and distinguished).5 Needless to say, including class rank in the MSPE could provide program directors with an efficient means of distinguishing superior applicants and, thus, could explain why survey respondents were in near unanimous support of including class rank in the MSPE. Ultimately, at the end of the application and interview season, applicants and program directors have to synthesize all available information to form their rank lists. Urology applicants can attend away rotations, speak with graduates and perform second looks in order to gather the information they need to optimally judge programs. The top 3 factors that applicants considered when formulating their final rank list (resident satisfaction, “abstract” feeling and surgical
autonomy for the residents) could effectively be gleaned only during an away rotation or second look.8 Program directors, on the other hand, do not always have the opportunity to spend clinical time with an applicant, nor is there enough time to interview all applicants. Therefore, MSPEs that contain valuable information and applications from only those who are sincerely interested in the program may facilitate program directors’ selection of the best applicants for their programs and the field of urology. There is room for improvement in the selection process as only 21% of respondents thought their impression of an applicant strongly correlated with their impression of the same individual as a resident. Limitations of this study include the lack of a validated questionnaire on the topic. Additionally, the questionnaire could have been expanded to investigate the topic in greater specificity. However, it was designed so that it would be easy for respondents to complete. Additional questions and further study are needed to discern why program directors favor or oppose the idea of an application limit. An econometric analysis that studies the usefulness of an application limit for all urology match participants should be conducted. Conclusions
Urology residency program directors receive a large number of applications, and exclude a significant number of applications based on USMLE Step 1 scores and applicants’ perceived levels of interest in their programs. Including class rank in the MSPE and imposing a reasonably set application limit might allow program directors to rely less on these factors, and focus on applicants’ more meaningful clinical and academic achievements. Acknowledgements
Heather Swanson provided assistance in gathering the data. Appendix. Urology Residency Matching Program Survey 1. How many applications did you receive this year (for the 2014e2015 match) for your urology residency program? B Less than 100 B 100e200 B 201e300 B More than 300 2. Did you use a USMLE Step 1 cutoff score to determine which applications to review in greater detail? If yes, what was your cutoff score? B No, I did not use a USMLE Step 1 cutoff score B Yes, around 220 B Yes, around 230 B Yes, around 240 B Yes, around 250
Challenges for Program Directors in Urology Match
3. What percentage of applications did you eliminate by a USMLE cutoff score or another objective parameter before you began reviewing applications in more detail? B None, I did not use a cutoff score and I reviewed all applications in detail B Around 25% B Around 40% B Around 60% B Around 70% or more 4. Excluding applications that you previously screened out by USMLE score or another objective parameter, how many minutes did you spend on average reviewing each application to determine who would receive an interview offer? B Less than 1 minute B 1e5 minutes B 6e10 minutes B More than 10 minutes 5. At your institution, how many individuals reviewed applications in order to select applicants to invite for an interview? B One B Two B Three B Four B Five B More than five 6. How important was the Dean’s letter in your assessment of the applicant? B Very important B Important B Moderately important B Of little importance B Unimportant 7. How often did the Dean’s letter provide meaningful information about the applicant? B Very frequently B Frequently B Occasionally B Rarely B Very rarely 8. Would you favor the inclusion of the applicant’s class rank in the Dean’s letter? B Yes B No 9. When, to be most useful, should the Dean’s letter be received? B August B September B October B November B December 10. To what extent did you consider a candidate’s likeliness to want to come to your program to determine who would receive an interview offer? B Did not consider B Slightly considered B Moderately considered B Strongly considered 11. Did you consider an applicant’s geographic history (i.e. medical school location, hometown, location of away rotation) in your decision to grant him or her an interview? B Yes B No 12. Have you ever not offered a superior candidate an interview because you estimated that he or she was not really interested in your program? B No, never B Yes, rarely B Yes, occasionally B Yes, frequently
491
13. Do you think that there should be a limit to the number of applications that applicants are allowed to submit? B Yes B No 14. What would be a reasonable application limit? B Approximately 10 applications B Approximately 20 applications B Approximately 30 applications B Approximately 40 applications B I don’t think there should be a limit 15. How would you rate the competitiveness of your residency program? B Bottom third of program competitiveness B Middle third of program competitiveness B Top third of program competitiveness 16. In the past, how often has your perceived strength of an applicant correlated with your impression of the same individual as a resident? B Not correlated B Weakly correlated B Moderately correlated B Strongly correlated 17. Which section of the American Urological Association does your institution belong to? B Northeastern B New England B New York B Mid Atlantic B Southeastern B South Central B North Central B Western
References 1. American Urological Association: Urology and Specialty Matches. Available at https://www.auanet.org/education/urologyand-specialty-matches.cfm. Accessed August 3, 2015. 2. American Urological Association: Urology Match Directory Information for Resident Candidates and Program Directors 1999. 3. Nikonow TN, Lyon TD, Jackman SV et al: Survey of applicant experience and cost in the urology match: opportunities for reform. J Urol 2015; 194: 1063. 4. Association of American Medical Colleges: A Guide to the Preparation of the Medical Student Performance Evaluation. Report of the Dean’s Letter Advisory Committee. Available at https://www. aamc.org/download/64496/data/mspeguide.pdf. Accessed August 3, 2015. 5. Naidich JB, Grimaldi GM, Lombardi P et al: A program director’s guide to the Medical Student Performance Evaluation (former dean’s letter) with a database. J Am Coll Radiol 2014; 11: 611. 6. Weissbart SJ and Stock JA: The history and rationale of the American Urological Association residency matching program. Urol Pract 2014; 1: 205. 7. Weissbart SJ, Hall SJ, Fultz BR et al: The urology match as a prisoner’s dilemma: a game theory perspective. Urology 2013; 82: 791. 8. Pagano MJ, Cooper KL and Badalato GM: Urology residency match: student perceptions and priorities. AUANews 2015; 20: 26, May.
492
Challenges for Program Directors in Urology Match
Editorial Commentary
This article focuses on the program director’s problem of the overwhelming number of applications to review during the time-honored and often frustrating recruitment season. In 2014 the AAMC reported that there were 425 applicants to urology, with the average number of applications being 59.6 per applicant.1 ERASÒ (Electronic Residency Application Service) conveniently allows program directors to filter out medical student files based on any number of parameters, such as minimal USMLE scores, clerkship grades and citizenship. Applicants who do not meet these parameters are automatically excluded for review. Understandably, filters optimize a streamlined process but they do not, in my opinion, allow for the best review process. Often the best candidates are not those who achieve the highest scores on a standardized, multiple choice test. They might simply not be able to do well on standardized tests. Students who may make exceptional doctors, but do not meet the program criteria, may be indiscriminately filtered out. These same candidates may have some of the more elusive, nonfilterable qualities such as integrity, grit and emotional quotient, characteristics we all admire in our residents e and in ourselves. Folklore portrays urologists as smart, fun-loving surgeons who enjoy their craft. This is what attracts most
medical students to our field. As a group, we are proud of who we are and what we represent. This is why we need to make the application process perhaps a little less mechanized and more humanized by truly seeking to determine those candidates who make humane contributions, have diverse qualities and possess collegial attributes. These qualities are in the students’ files, of course, and are elusive and perhaps, nonfilterable, but if we are to make the application process fair and unbiased, we should consider that the best predictors of success in residency may be in that applicant we filtered out for the sake of convenience. Byron Joyner Vice Dean for GME & DIO (Designated Institutional Official) University of Washington School of Medicine Seattle, Washington
Reference 1. Association of American Medical Colleges: Electronic Residency Application Service (ERASÒ) Data. Applicants, Matriculants, Enrollment, Graduates, MD/PhD, and Residency Applicants Data. Table 41. Available at https://www.aamc.org/data/facts/erasmdphd/. Accessed November 1, 2015.
Reply by Authors
Selecting the best applicants to interview for urology residency is no easy task. Program directors receive approximately 275 applications and can typically only interview 30 to 40 students (reference 1 in article). Applicants are now applying to half of the residency programs and deciphering their true levels of interest in programs can be difficult. In the current urology matching system the only practical method of selecting applicants to interview includes the consideration of board scores, grades (especially in surgery) and class rank or designation in the MSPE. Although admittedly not always correct, it is a reasonable assumption that these components of the application will be predictive of resident performance. Similar criteria are used for admission to other professional schools and there seems to be a good reason for that historically. Superior performance
in college has always been regarded as a predictor of superior performance and achievement at the next level of education. The challenges facing program directors in the urology match may not be completely solvable but there is room for improvement. Implementing a reasonably set application limit would clarify students’ interests in programs and may help them receive interview offers from programs of high interest. Imposing an application limit would allow for closer scrutiny of each application and afford program directors additional time for interviewing. Including class rank in the MSPE would highlight top students and provide program directors with another metric to consider. Asking letter of recommendation writers to complete a confidential form ranking the applicant in several categories would make letters more meaningful.
Challenges Facing Program Directors in the Urology Match Steven J. Weissbart, Jeffrey A. Stock and Alan J. Wein* From the Department of Urology, Stony Brook University (SJW), Stony Brook and Icahn School of Medicine at Mount Sinai (JAS), New York, New York, and Division of Urology, Department of Surgery, Hospital of University of Pennsylvania (AJW), Philadelphia, Pennsylvania
Abstract
Abbreviations and Acronyms
Introduction: We investigate urology residency program directors’ perspectives on the current residency matching system. Methods: A survey was emailed to Society of Urology Chairpersons and Program Directors members. The survey queried respondents’ perspectives on the current residency matching system, with special attention to the recent surge in application volume and the usefulness of the Medical Student Performance Evaluation. Participants were also asked about their perspective on a possible application limit for students applying to urology residencies.
MSPE = Medical Student Performance Evaluation USMLE = United States Medical Licensing Examination
Results: A total of 70 members of the Society of Urology Chairpersons and Program Directors responded to the survey. The majority of respondents received more than 200 applications for their program’s residency positions (77.1%) and used a Step 1 cutoff score to screen applications (81.4%). Approximately half of the respondents (51.4%) were in favor of imposing a limit to the number of applications that applicants are permitted to submit. The Medical Student Performance Evaluation was considered important or very important by 20% and 94.3% favored including an applicant’s class rank in the evaluation. An applicant’s projected likelihood of attending a respondent’s program was considered by 76%, and 60% had previously not offered superior candidates interviews because they estimated the applicants were not truly interested in the program. Conclusions: Urology program directors exclude a large number of applications based on board scores and applicants’ perceived levels of interest in the programs. A significant number of program directors favored an application limit as well as including class rank in the Medical Student Performance Evaluation. Key Words: internship and residency; urology; education, medical, graduate
Submitted for publication August 24, 2015. No direct or indirect commercial incentive associated with publishing this article. The corresponding author certifies that, when applicable, a statement(s) has been included in the manuscript documenting institutional review board, ethics committee or ethical review board study approval; principles of Helsinki Declaration were followed in lieu of formal ethics committee approval; 2352-0779/16/36-486/0 UROLOGY PRACTICE Ó 2016 by AMERICAN UROLOGICAL ASSOCIATION EDUCATION
AND
institutional animal care and use committee approval; all human subjects provided written informed consent with guarantees of confidentiality; IRB approved protocol number; animal approved project number. * Correspondence: Department of Urology, Penn Medicine, Perelman Center for Advanced Medicine, West Pavilion, 3rd Floor, 3400 Civic Center Blvd., Philadelphia, Pennsylvania 19104.
RESEARCH, INC.
http://dx.doi.org/10.1016/j.urpr.2015.10.008 Vol. 3, 486-492, November 2016 Published by Elsevier
Challenges for Program Directors in Urology Match
The number of applications that medical students are submitting for urology residency positions has dramatically increased in recent decades.1,2 While in 1995 applicants applied to an average of 28.4 programs, for the 2014-2015 match cycle applicants submitted a mean of 63 applications.1,2 This equated to an applicant, on average, applying to more than half the number of urology programs available. Aside from the financial cost of these applications for medical students,3 and the time commitment required by program directors to review these applications, these additional applications may most importantly preclude program directors from thoroughly reviewing each component of the application and, thereby, alter the paradigm by which applicants are selected for urology residency. In addition to the increase in application volume, urology residency program directors may also find letters of recommendation and the MSPE (Dean’s letter) to be challenging to interpret. The MSPE summarizes a student’s medical school record and is an important component of the residency application. While it is asked that the MSPE include a comparative analysis of a medical student’s performance in relation to his/her classmates,4 recent data have suggested that the majority of MSPEs do not include such information.5 Furthermore, medical schools use different language in their descriptions of applicants (eg superb, excellent, distinguished), thereby making it difficult for program directors to compare applicants according to the MSPE. With the high application volume, the need for MSPEs that succinctly express medical student performance has become even more pronounced. To ensure that applicants and programs are matched most appropriately and that the best medical students are selected for the field of urology, understanding program directors’ methods of applicant selection and their perspective on the current system is important. We chose to investigate urology program directors’ perspectives on a possible application limit for medical students applying to urology residencies as well as their attitudes toward the MSPE. We hypothesized that program directors would be in favor of an application limit and that the MSPE has become less important to program directors in the current application process.
Materials and Methods
A newly developed survey was emailed to 179 urology residency program directors and/or chairpersons through the Society of Urology Chairpersons and Program Directors listserv in April 2015. The survey included 2 lines of questions regarding 1) the imposition of an application limit to the current application system, and 2) the usefulness of
487
the current MSPEs (see Appendix). SurveyMonkeyÒ was used to generate and administer the survey. Descriptive statistics were used to characterize respondents’ survey answers. Subsequently, respondents were then divided into the groups of 1) those in favor of an application limit and 2) those not in favor of an application limit. Self-reported program competitiveness, received application volume and use of a USMLE cutoff score to screen applications were then compared between the 2 groups using chi-squared tests. StataÒ 13.1 was used for statistical analysis.
Results
The study included 70 (39%) urology chairpersons and/or program directors who responded to the survey. The survey respondents were from urology residency programs across the country (Northeastern 4%, New England 9%, New York 14%, Mid Atlantic 9%, Southeastern 19%, South Central 11%, North Central 21%, Western 13%) and most respondents considered their program to be of average (42.9%) or above average (54.3%) competitiveness for gaining acceptance. Of the survey respondents 54 (77.1%) received more than 200 applications during the 2014-2015 match cycle and 57 (81.4%) used a USMLE Step 1 cutoff score to determine which applications to review in further detail. Respondents who received more than 200 applications were not more likely to use a USMLE cutoff score than those who received 200 applications or less (p¼0.48). Of the respondents using a USMLE cutoff score 14 (24.6%) used an approximate score of 220, 24 (42.1%) used an approximate score of 230, 17 (29.8%) used an approximate score of 240 and 2 (3.5%) used an approximate score of 250. Figure 1 shows the percentages of applications that respondents eliminated by a USMLE Step 1 score or another objective parameter (eg Alpha Omega Alpha status). More than half of the respondents eliminated approximately 40% or more of applications before reviewing them in further detail. Except for applications that were initially excluded, 21 (30%) respondents spent more than 10 minutes reviewing each application, 34 (48.6%) spent 6 to 10 minutes reviewing each application and 14 (20%) spent 1 to 5 minutes reviewing an application. Most institutions had multiple individuals review applications for interviewing selection (fig. 1). The idea of imposing a limit on the number of applications that applicants are permitted to submit for the urology match was favored by 36 (51.4%) of the respondents. Of those respondents in favor of an application limit slightly
488
Challenges for Program Directors in Urology Match
Figure 1. Response to survey question “What percentage of applications did you eliminate by a USMLE cutoff score or another objective parameter before you began reviewing applications in more detail?” (A) and number of individuals reviewing applications at survey respondents’ institutions (B).
more than half (51.3%) believed that an application limit of approximately 20 would be reasonable (fig. 2). There were no differences in self-reported program competitiveness, volume of applications received or use of a USMLE cutoff score between respondents who favored the imposition of an application limit and those who did not, although a relationship between application volume and favorability of an application limit approached significance (p¼0.08, ie respondents who received more applications were more likely to be in favor of an application limit) (see table). The MSPE was considered important or very important for applicant assessment by 14 (20%), moderately important by 19 (27.1%) and of little importance or unimportant by 37 (52.9%) respondents. Figure 3 illustrates how often respondents found meaningful information in the MSPE, with 13 (18.6%) respondents citing that the MSPE very frequently or frequently contained meaningful information. The inclusion of an applicant’s class rank in the MSPE was favored by 66 (94.3%) respondents. Most respondents (58, 82.9%) thought the MSPEs would be most useful if received by September (fig. 3).
An applicant’s projected likelihood of attending a respondent’s program (if offered a position) was considered by 53 (76%) respondents, and 44 (63%) respondents considered an applicant’s geographic history (ie medical school location, hometown, location of away rotations) in their selection of candidates to interview (fig. 4). More than half of the respondents (42, 60%) had previously not offered superior candidates interviews because they estimated the applicants were not truly interested in the program (fig. 4). The relationship between respondents’ perceived level of strength of an applicant and their impression of the same individual as a resident was thought to weakly, moderately and strongly correlate by 7 (10%), 48 (68.6%) and 15 (21.4%) respondents, respectively.
Discussion
In this study of program directors’ methods of selecting applicants to interview and their sentiments toward the MSPE, we found that urology residency program directors exclude a significant percentage of applications based on USMLE Step 1 score, do not overwhelmingly consider
Table. Characteristics of respondents according to favorability of an application limit No. Favor Application Limit (%)
Figure 2. Reasonable application limit number among respondents in favor of application limit.
Received more than 300 applications Used USMLE cutoff score Top third of program competitiveness
No. Do Not Favor Application Limit (%)
p Value
13 (36.1)
6 (17.6)
0.08
28 (77.8) 19 (52.7)
29 (85.3) 19 (55.9)
0.42 0.79
Challenges for Program Directors in Urology Match
489
Figure 3. Response to survey questions “How often did the Dean’s letter provide meaningful information about the applicant?” (A) and “When, to be most useful, should the Dean’s letter be received?” (B).
the MSPEs to have meaningful information and commonly take into account an applicant’s perceived level of interest in their program during their decision of whom to interview. In addition, the idea of limiting the number of applications that applicants are allowed to submit was favorably considered by more than half of those surveyed. The urology matching system was designed to facilitate the efficient pairings of applicants and programs, and a cardinal feature of the system has been its ability to produce pairings that optimize applicant and program preferences.6 Assuming program directors are rational, they should prefer to interview the top candidates. However, with the high number of applications sent to program directors, they may have to consider factors other than scholastic achievement in their selection of candidates to interview.
In this study 76% of respondents considered an applicant’s perceived level of interest in their program during their deliberations on whom to interview and 60% had previously not interviewed superior candidates based on an applicant’s perceived lack of interest in the program. The large volume of applications appears to have congested the urology residency matching market, whereby superior applicants do not obtain interviews from programs of high interest and program directors lose the opportunity to recruit these top applicants to their programs. Limiting the number of applications that applicants may submit for urology residency positions might clarify their true levels of interest in programs, provide applicants with cost savings and would not appear to decrease the
Figure 4. Response to survey questions “To what extent did you consider a candidate’s likeliness to want to come to your program to determine who would receive an interview offer?” (A) and “Have you ever not offered a superior candidate an interview because you estimated that he or she was not really interested in your program?” (B).
490
Challenges for Program Directors in Urology Match
percentage of applicants who match.7 As demonstrated by this study, program directors are currently using factors such as applicants’ geographic histories in their selection of candidates to interview, but is it not possible that a superior applicant from the Northwest would be thrilled to attend an elite program on the East Coast? Interestingly, in a pilot study investigating the factors that applicants considered when formulating their rank lists, family obligations ranked as one of the least important factors (16 out of 18).8 Therefore, a residency near home might not be essential to most applicants. With the imposition of a reasonably set application limit, applying to a program alone would signal a genuine interest in the program and could, perhaps, take away the guesswork of determining applicants’ interest levels. This could, in effect, help applicants attain interview offers from programs in which they are sincerely interested. Not surprisingly, more than half of the respondents in this study were in favor of imposing an application limit to the urology match. Applicants in the field of urology may also be disadvantaged by vague letters of recommendation and MPSEs that do not comport with the Association of American Medical Colleges (AAMC) MSPE preparation guide. In 2002 the AAMC issued a guide for preparing the MSPE to ensure that the MSPE would be meaningful to members of the graduate medical education community (among other reasons).4 According to the guide, the summary paragraph of the MSPE should include a student’s comparative medical school performance in relation to his/her classmates, and MSPE appendices should include graphic comparisons of a student’s performance in relation to his or her classmates in several categories (basic science coursework, third year clerkships etc). Unfortunately, a recent study showed that only 9% of medicals schools included this complete comparative information in their MSPEs, and there was significant heterogeneity in the descriptive words used in MSPEs (eg outstanding, excellent and distinguished).5 Needless to say, including class rank in the MSPE could provide program directors with an efficient means of distinguishing superior applicants and, thus, could explain why survey respondents were in near unanimous support of including class rank in the MSPE. Ultimately, at the end of the application and interview season, applicants and program directors have to synthesize all available information to form their rank lists. Urology applicants can attend away rotations, speak with graduates and perform second looks in order to gather the information they need to optimally judge programs. The top 3 factors that applicants considered when formulating their final rank list (resident satisfaction, “abstract” feeling and surgical
autonomy for the residents) could effectively be gleaned only during an away rotation or second look.8 Program directors, on the other hand, do not always have the opportunity to spend clinical time with an applicant, nor is there enough time to interview all applicants. Therefore, MSPEs that contain valuable information and applications from only those who are sincerely interested in the program may facilitate program directors’ selection of the best applicants for their programs and the field of urology. There is room for improvement in the selection process as only 21% of respondents thought their impression of an applicant strongly correlated with their impression of the same individual as a resident. Limitations of this study include the lack of a validated questionnaire on the topic. Additionally, the questionnaire could have been expanded to investigate the topic in greater specificity. However, it was designed so that it would be easy for respondents to complete. Additional questions and further study are needed to discern why program directors favor or oppose the idea of an application limit. An econometric analysis that studies the usefulness of an application limit for all urology match participants should be conducted. Conclusions
Urology residency program directors receive a large number of applications, and exclude a significant number of applications based on USMLE Step 1 scores and applicants’ perceived levels of interest in their programs. Including class rank in the MSPE and imposing a reasonably set application limit might allow program directors to rely less on these factors, and focus on applicants’ more meaningful clinical and academic achievements. Acknowledgements
Heather Swanson provided assistance in gathering the data. Appendix. Urology Residency Matching Program Survey 1. How many applications did you receive this year (for the 2014e2015 match) for your urology residency program? B Less than 100 B 100e200 B 201e300 B More than 300 2. Did you use a USMLE Step 1 cutoff score to determine which applications to review in greater detail? If yes, what was your cutoff score? B No, I did not use a USMLE Step 1 cutoff score B Yes, around 220 B Yes, around 230 B Yes, around 240 B Yes, around 250
Challenges for Program Directors in Urology Match
3. What percentage of applications did you eliminate by a USMLE cutoff score or another objective parameter before you began reviewing applications in more detail? B None, I did not use a cutoff score and I reviewed all applications in detail B Around 25% B Around 40% B Around 60% B Around 70% or more 4. Excluding applications that you previously screened out by USMLE score or another objective parameter, how many minutes did you spend on average reviewing each application to determine who would receive an interview offer? B Less than 1 minute B 1e5 minutes B 6e10 minutes B More than 10 minutes 5. At your institution, how many individuals reviewed applications in order to select applicants to invite for an interview? B One B Two B Three B Four B Five B More than five 6. How important was the Dean’s letter in your assessment of the applicant? B Very important B Important B Moderately important B Of little importance B Unimportant 7. How often did the Dean’s letter provide meaningful information about the applicant? B Very frequently B Frequently B Occasionally B Rarely B Very rarely 8. Would you favor the inclusion of the applicant’s class rank in the Dean’s letter? B Yes B No 9. When, to be most useful, should the Dean’s letter be received? B August B September B October B November B December 10. To what extent did you consider a candidate’s likeliness to want to come to your program to determine who would receive an interview offer? B Did not consider B Slightly considered B Moderately considered B Strongly considered 11. Did you consider an applicant’s geographic history (i.e. medical school location, hometown, location of away rotation) in your decision to grant him or her an interview? B Yes B No 12. Have you ever not offered a superior candidate an interview because you estimated that he or she was not really interested in your program? B No, never B Yes, rarely B Yes, occasionally B Yes, frequently
491
13. Do you think that there should be a limit to the number of applications that applicants are allowed to submit? B Yes B No 14. What would be a reasonable application limit? B Approximately 10 applications B Approximately 20 applications B Approximately 30 applications B Approximately 40 applications B I don’t think there should be a limit 15. How would you rate the competitiveness of your residency program? B Bottom third of program competitiveness B Middle third of program competitiveness B Top third of program competitiveness 16. In the past, how often has your perceived strength of an applicant correlated with your impression of the same individual as a resident? B Not correlated B Weakly correlated B Moderately correlated B Strongly correlated 17. Which section of the American Urological Association does your institution belong to? B Northeastern B New England B New York B Mid Atlantic B Southeastern B South Central B North Central B Western
References 1. American Urological Association: Urology and Specialty Matches. Available at https://www.auanet.org/education/urologyand-specialty-matches.cfm. Accessed August 3, 2015. 2. American Urological Association: Urology Match Directory Information for Resident Candidates and Program Directors 1999. 3. Nikonow TN, Lyon TD, Jackman SV et al: Survey of applicant experience and cost in the urology match: opportunities for reform. J Urol 2015; 194: 1063. 4. Association of American Medical Colleges: A Guide to the Preparation of the Medical Student Performance Evaluation. Report of the Dean’s Letter Advisory Committee. Available at https://www. aamc.org/download/64496/data/mspeguide.pdf. Accessed August 3, 2015. 5. Naidich JB, Grimaldi GM, Lombardi P et al: A program director’s guide to the Medical Student Performance Evaluation (former dean’s letter) with a database. J Am Coll Radiol 2014; 11: 611. 6. Weissbart SJ and Stock JA: The history and rationale of the American Urological Association residency matching program. Urol Pract 2014; 1: 205. 7. Weissbart SJ, Hall SJ, Fultz BR et al: The urology match as a prisoner’s dilemma: a game theory perspective. Urology 2013; 82: 791. 8. Pagano MJ, Cooper KL and Badalato GM: Urology residency match: student perceptions and priorities. AUANews 2015; 20: 26, May.
492
Challenges for Program Directors in Urology Match
Editorial Commentary
This article focuses on the program director’s problem of the overwhelming number of applications to review during the time-honored and often frustrating recruitment season. In 2014 the AAMC reported that there were 425 applicants to urology, with the average number of applications being 59.6 per applicant.1 ERASÒ (Electronic Residency Application Service) conveniently allows program directors to filter out medical student files based on any number of parameters, such as minimal USMLE scores, clerkship grades and citizenship. Applicants who do not meet these parameters are automatically excluded for review. Understandably, filters optimize a streamlined process but they do not, in my opinion, allow for the best review process. Often the best candidates are not those who achieve the highest scores on a standardized, multiple choice test. They might simply not be able to do well on standardized tests. Students who may make exceptional doctors, but do not meet the program criteria, may be indiscriminately filtered out. These same candidates may have some of the more elusive, nonfilterable qualities such as integrity, grit and emotional quotient, characteristics we all admire in our residents e and in ourselves. Folklore portrays urologists as smart, fun-loving surgeons who enjoy their craft. This is what attracts most
medical students to our field. As a group, we are proud of who we are and what we represent. This is why we need to make the application process perhaps a little less mechanized and more humanized by truly seeking to determine those candidates who make humane contributions, have diverse qualities and possess collegial attributes. These qualities are in the students’ files, of course, and are elusive and perhaps, nonfilterable, but if we are to make the application process fair and unbiased, we should consider that the best predictors of success in residency may be in that applicant we filtered out for the sake of convenience. Byron Joyner Vice Dean for GME & DIO (Designated Institutional Official) University of Washington School of Medicine Seattle, Washington
Reference 1. Association of American Medical Colleges: Electronic Residency Application Service (ERASÒ) Data. Applicants, Matriculants, Enrollment, Graduates, MD/PhD, and Residency Applicants Data. Table 41. Available at https://www.aamc.org/data/facts/erasmdphd/. Accessed November 1, 2015.
Reply by Authors
Selecting the best applicants to interview for urology residency is no easy task. Program directors receive approximately 275 applications and can typically only interview 30 to 40 students (reference 1 in article). Applicants are now applying to half of the residency programs and deciphering their true levels of interest in programs can be difficult. In the current urology matching system the only practical method of selecting applicants to interview includes the consideration of board scores, grades (especially in surgery) and class rank or designation in the MSPE. Although admittedly not always correct, it is a reasonable assumption that these components of the application will be predictive of resident performance. Similar criteria are used for admission to other professional schools and there seems to be a good reason for that historically. Superior performance
in college has always been regarded as a predictor of superior performance and achievement at the next level of education. The challenges facing program directors in the urology match may not be completely solvable but there is room for improvement. Implementing a reasonably set application limit would clarify students’ interests in programs and may help them receive interview offers from programs of high interest. Imposing an application limit would allow for closer scrutiny of each application and afford program directors additional time for interviewing. Including class rank in the MSPE would highlight top students and provide program directors with another metric to consider. Asking letter of recommendation writers to complete a confidential form ranking the applicant in several categories would make letters more meaningful.