- Email: [email protected]
Fellowship Candidate Factors Considered by Program Directors
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
ORIGINAL ARTICLE
Fellowship Candidate Factors Considered by Program Directors Q13
Elham Beheshtian, MD, Rozita Jalilianhasanpour, MD, Sadaf Sahraian, MD, David M. Yousem, MD, MBA Abstract Objective: The neuroradiology fellowship match has been in existence for about 20 years. However, the elements by which neuroradiology fellowship program directors evaluate candidates have not been clearly elucidated. We sought to identify the factors that program directors use to rank neuroradiology fellowship applicants. Methods: An anonymous Qualtrics online five-question survey about educational credentials, personal traits, extracurricular activities, and demographic characteristics was sent to 72 neuroradiology program directors in April 2019. Each question required ranking of 10 factors based on different characteristics of fellowship candidates. Items included gender, nationality, US Medical Licensing Examination scores, internal applicants, work and research experience, recommendation letters, residency program, medical school attended, and visa status. Program directors had the ability to list any new characteristics that were not included in the survey. Results: In all, 68 of 72 (94.4%) neuroradiology programs responded to the survey. The most important criteria by which candidates were assessed were (1) residency program attended for educational credential, (2) personality as assessed by faculty at interviews for candidate personal traits, (3) research performed for candidate’s extracurricular activities, and (4) likelihood of coming to or previous experience in the fellowship geographic area. Neuroradiology program directors independently stressed residency program attended, personality assessed during the interview by faculty, internal candidate status, letters of recommendation, and research activities as their top five criteria in ranking the candidates. Conclusion: Multiple factors are weighed by neuroradiology fellowship program directors in selecting fellows, but recent experiences in residency, research, and faculty interactions are prioritized. Internal candidates have an advantage for remaining as fellows within their residency programs. Key Words: Fellowship, fellowship match, neuroradiology, program director J Am Coll Radiol 2019;-:---. Copyright ª 2019 American College of Radiology
INTRODUCTION The first neuroradiology fellowship positions in the United States were offered in the 1950s, and in 2001 the first neuroradiology fellowship selection match was initiated. According to the National Resident Matching Program (NRMP), in 2019, there were 201 applicants for 240 neuroradiology fellowship positions offered in 72 programs [1-3]. The selection process for fellows in different medical fields has been studied for multiple subspecialties [4,5].
Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institution, Baltimore, Maryland. Corresponding author and reprints: Elham Beheshtian, MD, 600 N. Wolfe Street, Phipps B112D, Baltimore, MD 21287; e-mail: [email protected]. Dr Yousem reports royalties from Elsevier, personal fees from Medicolegal consultant, speaker fees from ACR, other from Analytical Informatics, outside the submitted work. The other authors state that they have no Q 2 conflict of interest related to the material discussed in this article.
In 2011, radiology program directors (PDs) were asked about objective and subjective factors in the process of selecting radiology fellows, and the results showed that radiology residency performance and the reputation of the residency program were the most important objective factors. The respondents of this survey included 44 musculoskeletal fellowship PDs, 13 pediatric fellowship PDs, 13 vascular-interventional radiology PDs, but only 11 neuroradiology PDs [6]. We thought it would be useful for neuroradiology fellowship candidates, and of interest to neuroradiology PDs, to elucidate the factors that are weighed most heavily in the selection of neuroradiology fellows. We believed that by publicizing these factors, residents could improve their application materials and focus on the most important variables that would enhance their chances to match in the candidates’ most desirable programs. Moreover, this knowledge could be used by radiology residency PDs and neuroradiology fellowship PDs
Copyright ª 2019 American College of Radiology 1546-1440/19/$36.00 n https://doi.org/10.1016/j.jacr.2019.10.013
FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
1
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106
107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158
to implement quality improvement policies that expand their fellowship program selection successes. We hypothesized that PDs would emphasize the most recent academic activities in a candidate’s profile in ranking the applicants.
METHODS
Q3
Q4
In April 2019, a cover letter explaining the importance of clarifying neuroradiology fellowship ranking criteria for candidates was attached to a questionnaire named “Neuroradiology Program Directors Match Survey” and sent through the Internet using the Qualtrics platform (Seattle, Washington) to 72 neuroradiology programs’ current or past fellowship directors. Three reminders were sent to them within 1 month of sending the survey and then again at 3 months. The questionnaire sought responses to five questions concerning various aspects of the fellowship candidate’s educational credentials, personal traits, extracurricular activities, and demographic characteristics. Each question included 10 items, and the PDs were asked to put these 10 factors in rank order from 1 (most important) to 10 (least important) based on factors they consider during candidates’ evaluation. These 10 factors were based in part on a prior NRMP survey that was sent to Radiology Residency PDs in 2018 [7] and a neuroradiology PD survey from 2016, which only produced 17 of 75 responses and was therefore of limited value [8]. The “educational credentials” question included undergraduate school attended, medical school attended, residency training program attended, US Medical Licensing Examination (USMLE) step 1 and 2 results, dean’s letters, medical school grades, failure of any USMLE steps, Alpha Omega Alpha membership or medical school honors, in-service scores, and any other criteria added by PD. The “personal traits” survey question included items such as personal statement, letters of recommendation (LOR); personality assessed by faculty, residents, and fellows at interview; demonstrated teamwork; election as chief resident; assessment of intelligence; personal knowledge of applicant from visiting rotation or previous collaborations; home institution internal candidates, and other factors added by PD. The candidate’s “extracurricular activities” that were surveyed included research and community service performed; activity in radiology politics, hobbies, ethics, or leadership; teaching skill demonstrated; English language fluency; funded grant activity; additional degrees; and others. We also asked about “demographic characteristics” of gender, underrepresented minority, American or foreign international medical graduate, years from graduating medical school, gap in career chronology, likelihood of 2
coming to or previous experience in your geographic area, visa or citizenship status, interested in 2 years of fellowship, prior subspecialty training, and any other item recorded by PDs. In the fifth question, the PDs were asked to sort the top 12 items they value for assessing a candidate. They could pick from the offerings provided in questions 1 to 4 or list any new characteristics that were not included. The study was considered closed as of July 9, 2019, and all data were tabulated and analyzed using Excel 2018 (Microsoft, Redmond, Washington).
RESULTS Of the 72 programs surveyed, 68 PDs responded to the survey, representing a 94.4% response rate by July 9, 2019. The single most important criterion for each of the four questions asked (described in the Methods section) were (mean ranking from 1 to 10 with one being the most important and 10 least important): 1. Educational credentials: residency program attended (mean 1.88 1.44) (Table 1) 2. Personal traits: personality as assessed by faculty at interviews (mean 2.77 1.44) (Table 2)
Table 1. Educational credential rankings (score of 1 is most important, 10 least important) Mean Score
SD
Residency training program attended
1.88
1.44
Failure of any USMLE step
3.99
2.61
USMLE step 1 scores
4.07
1.70
Medical school attended
4.19
1.85
USMLE step 2 (CS and CK) scores
4.75
1.64
Alpha Omega Alpha membership or medical school honors
6.04
2.35
Dean’s letter or medical school grades
6.31
1.63
Undergraduate school attended
7.10
2.00
In-service examination scores
7.40
2.55
Other
9.25
2.24
Field
CK ¼ ---; CS ¼ Examination.
---;
USMLE ¼ US Medical Licensing
Journal of the American College of Radiology Volume - n Number - n Month 2019 FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
Q11
159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210
211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262
Table 2. Personal trait rankings (score of on1 is most important, 10 least important)
Table 4. Demographic characteristic rankings (score of 1 is most important, 10 least important)
Mean Score
SD
Mean Score
SD
Personality as assessed by faculty at interviews
2.77
1.47
Likelihood of coming to or previous experience in your geographic area
3.39
2.29
Internal (home institution) candidate
3.25
2.62
American or foreign international medical graduate
3.62
2.19
Letters of recommendation
4.17
2.38
Visa or citizenship status
4.71
2.29
Personal knowledge of applicant from visiting rotation or previous collaborations (not internal candidate)
4.42
2.54
Years from graduating medical school
4.88
2.12
Underrepresented minority
4.95
2.33
Gap in career chronology
5.50
2.39
Personality as assessed by residents or fellows at interviews
4.88
Interested in 2 years of fellowship
5.55
2.79
Prior subspecialty training
6.18
2.71
Demonstrated team work
5.97
1.96
Gender
6.24
2.48
Election as chief resident
6.17
2.15
Other
9.97
0.17
Personal statement
6.43
2.35
Assessments of intelligence
6.97
2.07
Other
9.98
0.12
Field
1.86
Table 3. Extracurricular activity rankings (score of 1 is most important, 10 least important) Field
Q1
Field
3. Extracurricular activities: research performed (mean 2.27 1.4) (Table 3) 4. Demographics: likelihood of coming to or previous experience in the fellowship’s geographic area (mean 3.39 2.29) (Table 4) The least important factors in granting an interview for the four questions were (1) in-service examination scores (mean 7.4 2.55); (2) assessments of intelligence (mean 6.97 2.07); (3) hobbies (mean 7.52 1.78); and (4) gender (mean 6.24 2.48), respectively. Other factors mentioned by PDs included MD or DO education, ability to handle a large volume of work, interest in considering an academic career, quality or authenticity of the candidate’s prior experience, and upholding patient care as a primary guiding principle. The top 12 items PDs listed for assessing a candidate in response to question 5 are reported in Figure 1. Of the answers to question 5, 25%, 22.5%, and 10% of PDs considered status as an internal candidate, residency program attended, and LORs, respectively, as the most significant factors in fellows’ selection. Five percent of PDs considered personality assessed by faculties during the interview as their most important fellow selection factor.
Mean Score
SD
Research performed
2.27
1.40
Ethics or leadership demonstrated
3.45
1.89
Teaching skills demonstrated
4.47
1.89
English language fluency
4.48
2.95
Funded grant activity
5.12
2.51
Activity in radiology politics or issues
5.29
1.95
Community service performed
5.86
1.85
Additional degrees completed
6.64
2.37
DISCUSSION
Hobbies
7.52
1.78
Other
9.89
0.86
Few studies have focused on the factors considered most important in radiology subspecialty fellowship selection. Based on a 2016 NRMP PD survey, different factors were
Journal of the American College of Radiology Beheshtian et al n Fellowship Candidate Factors FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
3
263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314
315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366
Fig 1. Independently ranked top 12 factors cited by program directors for best candidates (numbers are mean). CK ¼ ---; CS ¼ ---; USMLE ¼ US Medical Licensing Examination.
identified for offering an applicant an interview and ranking the candidates, but only 17 neuroradiology PDs answered the survey. The top five priorities found in that limited survey were (1) LOR, (2) letter by the residency PD, (3) reputation of the residency program, (4) interest in research, and (5) perceived commitment to neuroradiology. The strength of our study is a 94.4% response rate by 68 of 72 neuroradiology fellowship programs from the most recent 2018 to 2019 match. Our results differ by identifying within categories (1) residency program attended, (2) personality as assessed by faculty at interviews, (3) research performed, and (4) likelihood of coming to or previous experience in fellowship’s geographic area as top qualities. When the neuroradiology PDs offered their own top factors, they listed residency program attended, personality assessment by faculty at interviews, internal candidacy, LORs, and research performed, in that order. The factor listed as number 1 by the most PDs was being an internal candidate. The advantages of selecting internally are (1) being a known entity; (2) familiarity with the host institution’s electronic medical record, PACS, and workflow; (3) feelings that one’s own trainees are superior to outside candidates; (4) elimination of the upheaval of having someone move for a 1-year fellowship; and (5) support of the residency program’s own recruitment by guaranteeing fellowship slots. Research involving other subspecialties have published field-specific criteria used to offer interviews and to rank fellowship applicants. In pediatric surgery fellowships, candidates were chosen by various criteria, including research activity, exposure to well-known pediatric surgeons, and 4
Q10
quality of LOR [9]. Likewise, in ophthalmology subspecialties, the most cited criteria were the LORs and performance at the interview [10,11]. For female pelvic medicine and reconstructive surgery fellowships, residency education, followed by clinical research experience, was the most important selection factor. Being a chief resident had little importance; thus, the authors suggested that applicants should forgo being a chief resident and instead supplement their clinical research experience to improve their fellowship chances. However, based on our results, being elected as the chief resident was in the middle of the ranking list (ranked as the 14th factor, mean score 8.97) and still could have influence in selection criteria [12]. At the same time, the demonstration of leadership skills was cited as 9th of the top 12 criteria used by our PDs (Fig. 1). The neuroradiology PDs in this survey emphasized familiarity with the candidate (either as an internal candidate or through other previous experiences) more so than the other fellowship programs cited previously. There may be different criteria used for granting an interview versus ranking the candidates for the match. Niec Q 5 et al showed that with respect to offering an interview for a hand surgery fellowship, academic educational credentials were highly important, and LOR ranked as the most important item. The institution of MD (not DO) degree, training in orthopedic surgery, and residency program reputation were the factors most highly valued by hand surgery PDs. However, personality traits, applicant integrity, commitment to hard work, ability to work well with others, and quality of interview were all among the top-rated items Journal of the American College of Radiology Volume - n Number - n Month 2019
FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418
419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470
in the final ranking of applicants to hand fellowship programs. They also reported that in-service training score was relatively unimportant in choosing fellows by PDs [4]. This finding is compatible with our results that put in-service examinations as one of the least important items (ranked as 23rd factor, mean score: 9.7) ranked by PDs. By contrast, the scores on the yearly in-training examination were cited as important for candidates applying to general surgery fellowships [13]. Based on the educational question, attendance in a residency training program was the most important factor in ranking applicants. This supports our hypothesis that the most recent achievements would be prioritized for fellowship selection. On the other hand, interestingly, USMLE failure and step 1 scores from medical school, dating to years before even entering radiology residency (let alone the fellowship), were given greater emphasis by neuroradiology PDs than residency in-service scores. Why is that? This may be because of USMLE scores’ (1) widespread use in residency selection, (2) standardized nature, (3) familiarity and universality, and (4) ease of use for screening applicants by setting a threshold. The step 1 score also is employed to assess knowledge in international medical graduates in cases in which the quality of the medical school and residency training may be unknown. We recognize that this survey is an imprecise method for assessing the factors that are used in selecting neuroradiology fellows. Our overall response rate was excellent, but some PDs did not answer all five questions in the survey despite numerous reminders. We also recognize that we are making assumptions in this article that (1) the neuroradiology PDs were being honest in their answers and (2) the PDs are actually able to systematically and explicitly analyze what factors they use to make their decisions in judging candidates. Furthermore, some factors that are considered in the selection process may not have been offered among the options provided and thereby be “devalued.”
CONCLUSION The results of this study indicate the most important factors that neuroradiology fellowship applicants should emphasize are the quality of his or her residency program, the interview, internal candidacy, becoming known to the fellowship program through potential away visitation, research, and LORs. Internal fellow candidates also have a striking advantage for staying. The implications of our study are that trainees would benefit by (1) considering observerships at programs they are interested in; (2) conducting mock
interviews to improve their interviewing skills; (3) ensuring their letters of reference are unique and laudatory; (4) putting more emphasis on the USMLE tests before residency, engaging in research during residency; (5) strongly weighing internal programs for fellowship; and (6) accepting leadership and teaching opportunities to optimize their fellowship candidacy. Q6
TAKE-HOME POINTS -
The most important criteria by which candidates were assessed by PDs are (1) residency program attended, (2) personality as assessed by faculty at interviews, (3) research performed, and (4) likelihood of coming to or previous experience in fellowship geographic area.
-
Additional important factors include internal candidate status and LOR.
Q 12
REFERENCES
1. Taveras JM. Development of the first fellowship training program in neuroradiology in North America. AJNR Am J Neuroradiol 1991;12: 587-90. 2. Yousem DM, Hammoud DA. Assessment of the neuroradiology fellowship match: year 3. AJNR Am J Neuroradiol 2004;25:897-900. 3. Results and Data Specialties Matching Service 2019 Appointment Q7 Year. Available at: www.nrmp.org. Accessed July 16, 2019. 4. Nies MS, Bollinger AJ, Cassidy C, Jebson PJL. Factors used by program directors to select hand surgery fellows. J Hand Surg Am 2014;39:2285-88.e5. 5. Benzon HA, De Oliveira GS, Jagannathan N, Suresh S. Selection of subspecialty fellows in anesthesia for pediatric anesthesia: a national survey of program directors in the United States. Paediatr Anaesth 2015;25:487-91. 6. Mulcahy H, Chew FS, Mulcahy MJ. The radiology fellowship application and selection process in the United States: experiences and perceptions from both sides. Radiol Res Pract 2012;2012:875083. 7. The Match. National Resident Matching Program. Results of the 2018 NRMP Program Director Survey Available at: https://mk0nrmp3 oyqui6wqfm.kinstacdn.com/wp-content/uploads/2018/07/NRMP-2 Q8 018-Program-Director-Survey-for-WWW.pdf. 8. Results of the 2016 NRMP. Available at: www.nrmp.org. Accessed Q9 July 16, 2019. 9. Hirthler MA, Glick PL, Hassett JM, et al. Comparative analysis of successful and unsuccessful candidates for the pediatric surgical matching program. J Pediatr Surg 1992;27:142-5; discussion 145-8. 10. Kempton JE, Shields MB, Afshari NA, Dou W, Adelman RA. Fellow selection criteria. Ophthalmology 2009;116:1020-1020.e4. 11. Meyer DR, Dewan MA. Fellowship selection criteria in ophthalmic plastic and reconstructive surgery. Ophthal Plast Reconstr Surg 2010;26:357-9. 12. Muffly TM, Penick ER, Tang F, et al. Factors used by female pelvic medicine and reconstructive surgery fellowship directors to select their fellows. Int Urogynecol J 2010;21:349-52. 13. Miller AT, Swain GW, Widmar M, Divino CM. How important are American Board of Surgery In-Training Examination scores when applying for fellowships? J Surg Educ 2010;67:149-51.
Journal of the American College of Radiology Beheshtian et al n Fellowship Candidate Factors FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
5
471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522
ORIGINAL ARTICLE
Fellowship Candidate Factors Considered by Program Directors Q13
Elham Beheshtian, MD, Rozita Jalilianhasanpour, MD, Sadaf Sahraian, MD, David M. Yousem, MD, MBA Abstract Objective: The neuroradiology fellowship match has been in existence for about 20 years. However, the elements by which neuroradiology fellowship program directors evaluate candidates have not been clearly elucidated. We sought to identify the factors that program directors use to rank neuroradiology fellowship applicants. Methods: An anonymous Qualtrics online five-question survey about educational credentials, personal traits, extracurricular activities, and demographic characteristics was sent to 72 neuroradiology program directors in April 2019. Each question required ranking of 10 factors based on different characteristics of fellowship candidates. Items included gender, nationality, US Medical Licensing Examination scores, internal applicants, work and research experience, recommendation letters, residency program, medical school attended, and visa status. Program directors had the ability to list any new characteristics that were not included in the survey. Results: In all, 68 of 72 (94.4%) neuroradiology programs responded to the survey. The most important criteria by which candidates were assessed were (1) residency program attended for educational credential, (2) personality as assessed by faculty at interviews for candidate personal traits, (3) research performed for candidate’s extracurricular activities, and (4) likelihood of coming to or previous experience in the fellowship geographic area. Neuroradiology program directors independently stressed residency program attended, personality assessed during the interview by faculty, internal candidate status, letters of recommendation, and research activities as their top five criteria in ranking the candidates. Conclusion: Multiple factors are weighed by neuroradiology fellowship program directors in selecting fellows, but recent experiences in residency, research, and faculty interactions are prioritized. Internal candidates have an advantage for remaining as fellows within their residency programs. Key Words: Fellowship, fellowship match, neuroradiology, program director J Am Coll Radiol 2019;-:---. Copyright ª 2019 American College of Radiology
INTRODUCTION The first neuroradiology fellowship positions in the United States were offered in the 1950s, and in 2001 the first neuroradiology fellowship selection match was initiated. According to the National Resident Matching Program (NRMP), in 2019, there were 201 applicants for 240 neuroradiology fellowship positions offered in 72 programs [1-3]. The selection process for fellows in different medical fields has been studied for multiple subspecialties [4,5].
Russell H. Morgan Department of Radiology and Radiological Science, Johns Hopkins Medical Institution, Baltimore, Maryland. Corresponding author and reprints: Elham Beheshtian, MD, 600 N. Wolfe Street, Phipps B112D, Baltimore, MD 21287; e-mail: [email protected]. Dr Yousem reports royalties from Elsevier, personal fees from Medicolegal consultant, speaker fees from ACR, other from Analytical Informatics, outside the submitted work. The other authors state that they have no Q 2 conflict of interest related to the material discussed in this article.
In 2011, radiology program directors (PDs) were asked about objective and subjective factors in the process of selecting radiology fellows, and the results showed that radiology residency performance and the reputation of the residency program were the most important objective factors. The respondents of this survey included 44 musculoskeletal fellowship PDs, 13 pediatric fellowship PDs, 13 vascular-interventional radiology PDs, but only 11 neuroradiology PDs [6]. We thought it would be useful for neuroradiology fellowship candidates, and of interest to neuroradiology PDs, to elucidate the factors that are weighed most heavily in the selection of neuroradiology fellows. We believed that by publicizing these factors, residents could improve their application materials and focus on the most important variables that would enhance their chances to match in the candidates’ most desirable programs. Moreover, this knowledge could be used by radiology residency PDs and neuroradiology fellowship PDs
Copyright ª 2019 American College of Radiology 1546-1440/19/$36.00 n https://doi.org/10.1016/j.jacr.2019.10.013
FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
1
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106
107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158
to implement quality improvement policies that expand their fellowship program selection successes. We hypothesized that PDs would emphasize the most recent academic activities in a candidate’s profile in ranking the applicants.
METHODS
Q3
Q4
In April 2019, a cover letter explaining the importance of clarifying neuroradiology fellowship ranking criteria for candidates was attached to a questionnaire named “Neuroradiology Program Directors Match Survey” and sent through the Internet using the Qualtrics platform (Seattle, Washington) to 72 neuroradiology programs’ current or past fellowship directors. Three reminders were sent to them within 1 month of sending the survey and then again at 3 months. The questionnaire sought responses to five questions concerning various aspects of the fellowship candidate’s educational credentials, personal traits, extracurricular activities, and demographic characteristics. Each question included 10 items, and the PDs were asked to put these 10 factors in rank order from 1 (most important) to 10 (least important) based on factors they consider during candidates’ evaluation. These 10 factors were based in part on a prior NRMP survey that was sent to Radiology Residency PDs in 2018 [7] and a neuroradiology PD survey from 2016, which only produced 17 of 75 responses and was therefore of limited value [8]. The “educational credentials” question included undergraduate school attended, medical school attended, residency training program attended, US Medical Licensing Examination (USMLE) step 1 and 2 results, dean’s letters, medical school grades, failure of any USMLE steps, Alpha Omega Alpha membership or medical school honors, in-service scores, and any other criteria added by PD. The “personal traits” survey question included items such as personal statement, letters of recommendation (LOR); personality assessed by faculty, residents, and fellows at interview; demonstrated teamwork; election as chief resident; assessment of intelligence; personal knowledge of applicant from visiting rotation or previous collaborations; home institution internal candidates, and other factors added by PD. The candidate’s “extracurricular activities” that were surveyed included research and community service performed; activity in radiology politics, hobbies, ethics, or leadership; teaching skill demonstrated; English language fluency; funded grant activity; additional degrees; and others. We also asked about “demographic characteristics” of gender, underrepresented minority, American or foreign international medical graduate, years from graduating medical school, gap in career chronology, likelihood of 2
coming to or previous experience in your geographic area, visa or citizenship status, interested in 2 years of fellowship, prior subspecialty training, and any other item recorded by PDs. In the fifth question, the PDs were asked to sort the top 12 items they value for assessing a candidate. They could pick from the offerings provided in questions 1 to 4 or list any new characteristics that were not included. The study was considered closed as of July 9, 2019, and all data were tabulated and analyzed using Excel 2018 (Microsoft, Redmond, Washington).
RESULTS Of the 72 programs surveyed, 68 PDs responded to the survey, representing a 94.4% response rate by July 9, 2019. The single most important criterion for each of the four questions asked (described in the Methods section) were (mean ranking from 1 to 10 with one being the most important and 10 least important): 1. Educational credentials: residency program attended (mean 1.88 1.44) (Table 1) 2. Personal traits: personality as assessed by faculty at interviews (mean 2.77 1.44) (Table 2)
Table 1. Educational credential rankings (score of 1 is most important, 10 least important) Mean Score
SD
Residency training program attended
1.88
1.44
Failure of any USMLE step
3.99
2.61
USMLE step 1 scores
4.07
1.70
Medical school attended
4.19
1.85
USMLE step 2 (CS and CK) scores
4.75
1.64
Alpha Omega Alpha membership or medical school honors
6.04
2.35
Dean’s letter or medical school grades
6.31
1.63
Undergraduate school attended
7.10
2.00
In-service examination scores
7.40
2.55
Other
9.25
2.24
Field
CK ¼ ---; CS ¼ Examination.
---;
USMLE ¼ US Medical Licensing
Journal of the American College of Radiology Volume - n Number - n Month 2019 FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
Q11
159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210
211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262
Table 2. Personal trait rankings (score of on1 is most important, 10 least important)
Table 4. Demographic characteristic rankings (score of 1 is most important, 10 least important)
Mean Score
SD
Mean Score
SD
Personality as assessed by faculty at interviews
2.77
1.47
Likelihood of coming to or previous experience in your geographic area
3.39
2.29
Internal (home institution) candidate
3.25
2.62
American or foreign international medical graduate
3.62
2.19
Letters of recommendation
4.17
2.38
Visa or citizenship status
4.71
2.29
Personal knowledge of applicant from visiting rotation or previous collaborations (not internal candidate)
4.42
2.54
Years from graduating medical school
4.88
2.12
Underrepresented minority
4.95
2.33
Gap in career chronology
5.50
2.39
Personality as assessed by residents or fellows at interviews
4.88
Interested in 2 years of fellowship
5.55
2.79
Prior subspecialty training
6.18
2.71
Demonstrated team work
5.97
1.96
Gender
6.24
2.48
Election as chief resident
6.17
2.15
Other
9.97
0.17
Personal statement
6.43
2.35
Assessments of intelligence
6.97
2.07
Other
9.98
0.12
Field
1.86
Table 3. Extracurricular activity rankings (score of 1 is most important, 10 least important) Field
Q1
Field
3. Extracurricular activities: research performed (mean 2.27 1.4) (Table 3) 4. Demographics: likelihood of coming to or previous experience in the fellowship’s geographic area (mean 3.39 2.29) (Table 4) The least important factors in granting an interview for the four questions were (1) in-service examination scores (mean 7.4 2.55); (2) assessments of intelligence (mean 6.97 2.07); (3) hobbies (mean 7.52 1.78); and (4) gender (mean 6.24 2.48), respectively. Other factors mentioned by PDs included MD or DO education, ability to handle a large volume of work, interest in considering an academic career, quality or authenticity of the candidate’s prior experience, and upholding patient care as a primary guiding principle. The top 12 items PDs listed for assessing a candidate in response to question 5 are reported in Figure 1. Of the answers to question 5, 25%, 22.5%, and 10% of PDs considered status as an internal candidate, residency program attended, and LORs, respectively, as the most significant factors in fellows’ selection. Five percent of PDs considered personality assessed by faculties during the interview as their most important fellow selection factor.
Mean Score
SD
Research performed
2.27
1.40
Ethics or leadership demonstrated
3.45
1.89
Teaching skills demonstrated
4.47
1.89
English language fluency
4.48
2.95
Funded grant activity
5.12
2.51
Activity in radiology politics or issues
5.29
1.95
Community service performed
5.86
1.85
Additional degrees completed
6.64
2.37
DISCUSSION
Hobbies
7.52
1.78
Other
9.89
0.86
Few studies have focused on the factors considered most important in radiology subspecialty fellowship selection. Based on a 2016 NRMP PD survey, different factors were
Journal of the American College of Radiology Beheshtian et al n Fellowship Candidate Factors FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
3
263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314
315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366
Fig 1. Independently ranked top 12 factors cited by program directors for best candidates (numbers are mean). CK ¼ ---; CS ¼ ---; USMLE ¼ US Medical Licensing Examination.
identified for offering an applicant an interview and ranking the candidates, but only 17 neuroradiology PDs answered the survey. The top five priorities found in that limited survey were (1) LOR, (2) letter by the residency PD, (3) reputation of the residency program, (4) interest in research, and (5) perceived commitment to neuroradiology. The strength of our study is a 94.4% response rate by 68 of 72 neuroradiology fellowship programs from the most recent 2018 to 2019 match. Our results differ by identifying within categories (1) residency program attended, (2) personality as assessed by faculty at interviews, (3) research performed, and (4) likelihood of coming to or previous experience in fellowship’s geographic area as top qualities. When the neuroradiology PDs offered their own top factors, they listed residency program attended, personality assessment by faculty at interviews, internal candidacy, LORs, and research performed, in that order. The factor listed as number 1 by the most PDs was being an internal candidate. The advantages of selecting internally are (1) being a known entity; (2) familiarity with the host institution’s electronic medical record, PACS, and workflow; (3) feelings that one’s own trainees are superior to outside candidates; (4) elimination of the upheaval of having someone move for a 1-year fellowship; and (5) support of the residency program’s own recruitment by guaranteeing fellowship slots. Research involving other subspecialties have published field-specific criteria used to offer interviews and to rank fellowship applicants. In pediatric surgery fellowships, candidates were chosen by various criteria, including research activity, exposure to well-known pediatric surgeons, and 4
Q10
quality of LOR [9]. Likewise, in ophthalmology subspecialties, the most cited criteria were the LORs and performance at the interview [10,11]. For female pelvic medicine and reconstructive surgery fellowships, residency education, followed by clinical research experience, was the most important selection factor. Being a chief resident had little importance; thus, the authors suggested that applicants should forgo being a chief resident and instead supplement their clinical research experience to improve their fellowship chances. However, based on our results, being elected as the chief resident was in the middle of the ranking list (ranked as the 14th factor, mean score 8.97) and still could have influence in selection criteria [12]. At the same time, the demonstration of leadership skills was cited as 9th of the top 12 criteria used by our PDs (Fig. 1). The neuroradiology PDs in this survey emphasized familiarity with the candidate (either as an internal candidate or through other previous experiences) more so than the other fellowship programs cited previously. There may be different criteria used for granting an interview versus ranking the candidates for the match. Niec Q 5 et al showed that with respect to offering an interview for a hand surgery fellowship, academic educational credentials were highly important, and LOR ranked as the most important item. The institution of MD (not DO) degree, training in orthopedic surgery, and residency program reputation were the factors most highly valued by hand surgery PDs. However, personality traits, applicant integrity, commitment to hard work, ability to work well with others, and quality of interview were all among the top-rated items Journal of the American College of Radiology Volume - n Number - n Month 2019
FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418
419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470
in the final ranking of applicants to hand fellowship programs. They also reported that in-service training score was relatively unimportant in choosing fellows by PDs [4]. This finding is compatible with our results that put in-service examinations as one of the least important items (ranked as 23rd factor, mean score: 9.7) ranked by PDs. By contrast, the scores on the yearly in-training examination were cited as important for candidates applying to general surgery fellowships [13]. Based on the educational question, attendance in a residency training program was the most important factor in ranking applicants. This supports our hypothesis that the most recent achievements would be prioritized for fellowship selection. On the other hand, interestingly, USMLE failure and step 1 scores from medical school, dating to years before even entering radiology residency (let alone the fellowship), were given greater emphasis by neuroradiology PDs than residency in-service scores. Why is that? This may be because of USMLE scores’ (1) widespread use in residency selection, (2) standardized nature, (3) familiarity and universality, and (4) ease of use for screening applicants by setting a threshold. The step 1 score also is employed to assess knowledge in international medical graduates in cases in which the quality of the medical school and residency training may be unknown. We recognize that this survey is an imprecise method for assessing the factors that are used in selecting neuroradiology fellows. Our overall response rate was excellent, but some PDs did not answer all five questions in the survey despite numerous reminders. We also recognize that we are making assumptions in this article that (1) the neuroradiology PDs were being honest in their answers and (2) the PDs are actually able to systematically and explicitly analyze what factors they use to make their decisions in judging candidates. Furthermore, some factors that are considered in the selection process may not have been offered among the options provided and thereby be “devalued.”
CONCLUSION The results of this study indicate the most important factors that neuroradiology fellowship applicants should emphasize are the quality of his or her residency program, the interview, internal candidacy, becoming known to the fellowship program through potential away visitation, research, and LORs. Internal fellow candidates also have a striking advantage for staying. The implications of our study are that trainees would benefit by (1) considering observerships at programs they are interested in; (2) conducting mock
interviews to improve their interviewing skills; (3) ensuring their letters of reference are unique and laudatory; (4) putting more emphasis on the USMLE tests before residency, engaging in research during residency; (5) strongly weighing internal programs for fellowship; and (6) accepting leadership and teaching opportunities to optimize their fellowship candidacy. Q6
TAKE-HOME POINTS -
The most important criteria by which candidates were assessed by PDs are (1) residency program attended, (2) personality as assessed by faculty at interviews, (3) research performed, and (4) likelihood of coming to or previous experience in fellowship geographic area.
-
Additional important factors include internal candidate status and LOR.
Q 12
REFERENCES
1. Taveras JM. Development of the first fellowship training program in neuroradiology in North America. AJNR Am J Neuroradiol 1991;12: 587-90. 2. Yousem DM, Hammoud DA. Assessment of the neuroradiology fellowship match: year 3. AJNR Am J Neuroradiol 2004;25:897-900. 3. Results and Data Specialties Matching Service 2019 Appointment Q7 Year. Available at: www.nrmp.org. Accessed July 16, 2019. 4. Nies MS, Bollinger AJ, Cassidy C, Jebson PJL. Factors used by program directors to select hand surgery fellows. J Hand Surg Am 2014;39:2285-88.e5. 5. Benzon HA, De Oliveira GS, Jagannathan N, Suresh S. Selection of subspecialty fellows in anesthesia for pediatric anesthesia: a national survey of program directors in the United States. Paediatr Anaesth 2015;25:487-91. 6. Mulcahy H, Chew FS, Mulcahy MJ. The radiology fellowship application and selection process in the United States: experiences and perceptions from both sides. Radiol Res Pract 2012;2012:875083. 7. The Match. National Resident Matching Program. Results of the 2018 NRMP Program Director Survey Available at: https://mk0nrmp3 oyqui6wqfm.kinstacdn.com/wp-content/uploads/2018/07/NRMP-2 Q8 018-Program-Director-Survey-for-WWW.pdf. 8. Results of the 2016 NRMP. Available at: www.nrmp.org. Accessed Q9 July 16, 2019. 9. Hirthler MA, Glick PL, Hassett JM, et al. Comparative analysis of successful and unsuccessful candidates for the pediatric surgical matching program. J Pediatr Surg 1992;27:142-5; discussion 145-8. 10. Kempton JE, Shields MB, Afshari NA, Dou W, Adelman RA. Fellow selection criteria. Ophthalmology 2009;116:1020-1020.e4. 11. Meyer DR, Dewan MA. Fellowship selection criteria in ophthalmic plastic and reconstructive surgery. Ophthal Plast Reconstr Surg 2010;26:357-9. 12. Muffly TM, Penick ER, Tang F, et al. Factors used by female pelvic medicine and reconstructive surgery fellowship directors to select their fellows. Int Urogynecol J 2010;21:349-52. 13. Miller AT, Swain GW, Widmar M, Divino CM. How important are American Board of Surgery In-Training Examination scores when applying for fellowships? J Surg Educ 2010;67:149-51.
Journal of the American College of Radiology Beheshtian et al n Fellowship Candidate Factors FLA 5.6.0 DTD JACR4998_proof 28 November 2019 6:03 pm ce
5
471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522