- Email: [email protected]
Evaluating applicants to emergency medicine residency programs
The Journal of Emergency Medicine, Vol 17, No 1, pp 131–134, 1999 Copyright © 1999 Elsevier Science Inc. Printed in the USA. All rights reserved 0736-4679/99 $–see front matter
PII S0736-4679(98)00138-3
Education
EVALUATING APPLICANTS TO EMERGENCY MEDICINE RESIDENCY PROGRAMS Jerry Balentine,
DO, FAAEM,*
Theodore Gaeta,
DO, FACEP,†
and Theodore Spevack,
DO, FACOEP‡
*St. Barnabas Hospital, Bronx, New York, USA, †Methodist Hospital, Brooklyn, New York, USA, ‡NYCOM/St. Barnabas Hospital, Bronx, New York Reprint Address: Jerry Balentine, DO, Department of Emergency Medicine, St. Barnabas Hospital, 4422 Third Avenue, Bronx, NY 10457
e Abstract—This article reviews the pertinent literature related to the selection process of medical students to emergency medicine residency programs. The impact that academic performance in medical school, the interview, letters of recommendation, and other achievements have on the performance of the future resident are reviewed. All articles identified by an English language MEDLINE search were reviewed by the authors as to significance to the subject. Review of relevant literature indicates that no precise correlation can be made between performance in medical school and achievements during the residency, although there seems to be a correlation between academic performance in medical school and similar performance on board certification examinations. © 1999 Elsevier Science Inc.
dency candidate. In 1991, EM resident applicants spent an average of $1725 and 18 days away from their medical school rotations due to interviews (1). The applicants’ perception of the program will then further determine how high the applicant ranks the program (2). Some residency programs have developed databases in order to evaluate and track information regarding their applicants while others rely on a more traditional method (3). The purpose of this article is to review the literature relating to the selection process, and then evaluate the various criteria used to predict which candidates will make successful residents. It should be noted, however, that the majority of literature has been published by specialties other than emergency medicine and it is unclear how the results of this research relate to our specialty. It is also worth noting that in the majority of cases, whether or not a resident is deemed successful depends upon performance evaluations from program directors and supervising physicians. Such evaluations harbor potential reliability problems due to possible bias, personal preferences, and a lack of standardization. The reliability of these performance assessments have not been clearly established, though reports of positive correlation do exist when two independent supervisors evaluate the same resident (4).
e Keywords—residency; selection; evaluating applicants; interview
INTRODUCTION The selection of candidates to an Emergency Medicine (EM) residency program, an annual event that will shape an individual residency program for several years, requires expenditure of considerable financial as well as temporal resources by both the institution and the resi-
Education is coordinated by Stephen R. Hayden, Diego, California
RECEIVED: 31 October 1997; FINAL ACCEPTED: 20 February 1998
SUBMISSION RECEIVED:
MD,
of The University of California San Diego Medical Center, San
6 February 1998; 131
132
J. Balentine, T. Gaeta, and T. Spevack
Table 1. Measures Used to Evaluate Applicants Interview Medical school grades and class rank Results of board examinations Letters of recommendation Clerkship Prior Experiences (research; other residency training . . . )
MEASURES USED TO EVALUATE APPLICANTS The measures available to most programs in the selection process are listed in Table 1. INTERVIEW Emergency medicine residencies require an interview to be considered for a position in their programs. When surveying residency directors on the importance of different factors in the selection process, the interview was frequently cited to be the most important tool (5). Wagoner in two separate surveys of program directors in internal medicine, family practice, surgery and pediatrics found interpersonal skills as demonstrated on the interview as the most important selection criteria (6,7). The same result was obtained in a survey of physical medicine and rehabilitation directors and interviews with 16 psychiatry residency program directors (8,9). A survey of residency directors in radiology programs showed significant differences between programs with regard to the emphasis placed on class rank and medical school grades (10). Such importance given to the interview by program directors has never been substantiated by data showing that the results of the interview correlate with future performance. In fact, in a study of PGY I residents in an internal medicine residency, no correlation between interview scores and scores at the end of their PGY I year (as determined by faculty evaluation) were apparent (11). Could an improvement in our interview process help with these results? Altmaier et al., in a study of medical students applying to radiology residencies, compared the traditional interview process with behavior based interviewing (an interview technique focusing on experiences, behaviors, knowledge and skills that are job related). They confirm the low predictive value of traditional interviews but find that the results of the behavior based interviewing add considerable predictive value (12). ACADEMIC MEASURES Medical school performance as measured by grades, class rank, or performance on a national test (NBME or
USMLE) offers the easiest way to measure performance as a student and compare to it with performance as a resident. In several studies, medical school administrators followed their graduates longitudinally. They had program directors fill out questionnaires pertaining to the performance of the residents in their program. These results were then correlated to variables from the student’s academic record. The findings of these studies are conflicting. National Board of medical examiners (NBME) Part I and Part II scores did not correlate well with performance as residents (13,14). Alternatively, two studies revealed a correlation for students who were in the top quarter and bottom quarter, each remaining in the same group when evaluations were based on their clinical performance (15,16). Similar results should be expected from the United States Medical Licensing Examination (USMLE), although not enough data are available (17). Performance on the NBME does correlate with success in passing the board certification examinations in orthopedic surgery, dermatology, preventive medicine, (18) and internal medicine (19) while faculty evaluations of residents do not correlate with success on the board examination (20). If this is our most important outcome performance measure, then the NBME results should weigh heavily. Similarly, class rank and medical school grades show only modest correlation with residency performance (21,22). There seems to be a trend towards a correlation at the extremes (top of the class and bottom of the class).
Letters of Recommendation The letter of recommendation is often expected to be a source of reliable information supplied by a person who knows and has worked with the candidate (23). Unfortunately, when letters of recommendation are reviewed, important information such as clinical judgment and interpersonal communication skills are often lacking (24). When evaluating letters of recommendation, a group of academic surgeons stated that the origin of the letter (academic rank and school of origin) were important factors in ranking the letter of recommendation, but in the same study no difference was found between the ranking of a letter whether it was provided to the evaluator on original letterhead and with signature or as a “text only” blank letter (25). In the same study, the overall “theme” of the letter often outweighed specific statements, for example, the phrase “If I can provide any additional information please call . . .” was almost uniformly identified as a negative comment, but it was also found in a letter rated as one of the highest recommendations. Emergency Medicine is currently trying to circumvent this problem by adopting a standardized letter
Evaluating Applicants
of recommendation. The results of the use of these new letters will have to be reviewed in the future.
Clerkship Evaluations The clinical clerkship provides the student with the opportunity to learn about the field of emergency medicine and acquire the clinical knowledge necessary for future training. It also gives the faculty an opportunity to observe a student. Unfortunately, the information gained about the student is frequently not interpreted and recorded appropriately (26). In addition, with different clerkships having different objectives and evaluation tools, even a standardized student evaluation form has limited predictive value when used for surgical clerkships (27). Standardized patient-based postclerkship examinations (28) and performance based assessment scores after clerkships (29) have shown a stronger predictive value than traditional evaluations, but are not universally available. Therefore, unless one is familiar with a given clerkship or its evaluation methods, limited information is gained from a clerkship evaluation from an outside institution. Direct observation of candidates during a clerkship may be the only valid model for candidates applying to a program. Unfortunately, many programs have limited numbers of student clerkships available and most candidates have not performed clerkships at their hospital.
PRIOR TRAINING AND EXPERIENCE With newer reimbursement issues, prior training will probably become a disadvantage for applicants who have previous residency training. In the radiology literature, no advantage was found in the performance of residents with prior training compared to residents with no prior training (30). This does not necessarily apply to the practice of emergency medicine. The results of other prior experiences [(prehospital training, research experience)] on future resident performance in emergency medicine has not been studied and therefore cannot be evaluated.
OTHER METHODS In one study, psychological testing was used to evaluate desirable qualities in residents (31). Once qualities thought to be advantageous to emergency medicine residents are identified, psychological tests may provide input into the application process. In fact, some institutions already use an elaborate evaluation method criteria obtained by assigning relative weights to different traits
133
that would be expected in an ideal house officer and use the results of this system to obtain a ranking list (32). These criteria will be different from specialty to specialty and location to location and are therefore not universally applicable.
LIMITATIONS Most of the literature reviewed is not specific to emergency medicine residencies. Although the basic qualities needed during residency may be similar in the different areas of medicine, certain skills needed in Emergency Medicine could be different from a radiology or surgical residency. Extrapolation of results of some of these studies may be beyond the scope of the original objective. In most studies the outcome criterion was an evaluation by faculty members or the program director, who might have had limited exposure to the resident or involved their own biases of positive and negative qualities in evaluating a resident.
DISCUSSION To date, the literature fails to provide guidance in predicting students’ performance during future residency. There is a trend towards the “top” students (academically) staying at the top as residents and the bottom staying at the bottom, but even this was not a universal truth. The lack of correlation between undergraduate and graduate performance might be inherent in the system. The expectations for medical students during the first two years of training are different from those of residents. Measures such as letters of recommendations, clerkships, and prior experiences show poor correlation with future residency success. We suspect the best correlation can be achieved if a combination of factors is considered and different weight assigned to measures depending on the importance of that quality to the program. Each residency will have a different perception of the “ideal” resident. This must be determined by the faculty and program director, and this should influence the ranking of applicants. Residencies might want to project a certain image (research oriented; academic program) or prefer to balance their program with applicants with different backgrounds and strengths and weaknesses. External forces must be considered. Issues such as less reimbursement for residents with prior training and limitations on residency positions will play into the selection process. We should expect that with mandatory decreases in residency training positions, a program might try to attract residents who will excel at measures used to evaluate residency success in their respective
134
J. Balentine, T. Gaeta, and T. Spevack
institutions. If passing the ABEM certification examination is the institution’s measure of success, then the
program might gear the curriculum and ranking of candidates towards a more academic background.
REFERENCES 1. Aghababian R, Tandberg D, Iserson K, Martin M, Sklar D. Selection of emergency medicine residents. Ann Emerg Med 1993;22: 1753– 61. 2. Koscove EM. An applicant’s evaluation of an emergency medicine internship and residency. Ann Emerg Med 1990;19:1774 – 80. 3. Ledwith M, Myers A. A database management system for residency applicants files. Computers and Biomed Res 1988;21:186 – 91. 4. Davis JK, Inamdar S. The reliability of performance assessment during residency. Acad Med 1990;65:716. 5. Galazka SS, Kikano GE, Zyzanski S. Methods of recruiting and selecting residents for US family practice residencies. Acad Med 1994;69:304 – 6. 6. Wagoner NE, Gray GT. Report of a survey of program directors regarding selection factors in graduate medical education. J Med Edu 1979;54:445–52. 7. Wagoner NE, Suriano JR, Stoner JA. Factors used by program directors to select residents. J Med Educ 1986;61:10 –21. 8. DeLisa JA, Jain SS, Campagnolo DI. Factors used by physical medicine and rehabilitation residency training directors to select residents. Am J Phys Med Rehab 1994;73:152– 6. 9. Ross CA, Leichner P. Criteria for selection residents: A reassessment. Can J Psychiatry 1984;29:681– 6. 10. Grantham JR. Radiology resident selection: Results of a survey. Residency Directors Newsl 1993;28:99 –101. 11. Komives E, Weiss ST, Rosa R. The applicant interview as a predictor of resident performance. J Med Educ 1984;59:425– 6. 12. Altmaier EM, Smith WL, O’Halloran CM, Franken EA, Jr. The predictive utility of behavior based interviewing compared with traditional interviewing in the selection of radiology residents. Invest Radiol 1992;27:385–9. 13. Smith SR. Correlations between graduates’ performance as firstyear residents and their performances as medical students. Acad Med 1993;68:633–5. 14. Wood PS, Smith WL, Altmaier EM, Tarico VS, Franken EA. A prospective study of cognitive and noncognitive selection criteria as predictors of resident performance. Investig Rad 1990; 25: 855–9. 15. Markert RJ. The relationship of academic measures in medical school to performance after graduation. Acad Med 1993;68:S31–5. 16. Yindra KJ, Rosenfeld PS, Donnelly MB. Medical school achievements as predictors of residency performance. J Med Educ 1988; 63:357– 63. 17. Berner ES, Brooks CM, Erdman JB. Use of the USMLE to select residents. Acad Med 1993;68:753–9. 18. Case SM, Swanson DB. Validity of NBME Part I and Part II
19.
20.
21. 22.
23.
24.
25.
26.
27. 28.
29.
30.
31.
32.
Scores for selection of residents in orthopaedic surgery, dermatology and preventive medicine. Acad Med 1993;68:S51–S6. Sosenko J, Stekel KW, Soto R, Gelbard M. NBME Examination part I as a predictor of clinical and ABIM certifying examination performance. J Gen Intern Med 1993;8:86 – 8. Wade TP, Andrus CH, Kaminski DL. Evaluations of surgery resident performance correlate with success in board examinations. Surgery 1993;113:644 – 8. Arnold LA, Willoughby TL. The empirical association between student and resident performances. Acad Med 1993;68:S35–S40. Loftus LS, Arnold L, Willoughby TL, Connolly A. First year residents performances compared with their medical school class ranks determined by three ranking systems. Acad Med 1992;67: 319 –23. Baker JD, Bailey MK, Brahen NH, Conroy JM, Dorman BH, Haynes GR. Selection of anesthesiology residents. Acad Med 1993;68:161–3. O’Halloran CM, Altmaier EM, Smith W, Frabken EA. Evaluation of resident applicants by letters of recommendation: A comparison of traditional and behavior-based formats. Invest Radiol 1993;3: 274 –7. Greenburg AG, Doyle J, McClure DK. Letters of recommendation for surgical residents: What they say and what they mean. J Surg Res 1994;56:192– 8. Teneyck RP, Maclean TA. Improving the quality of emergency medicine rotation/clerkship evaluations. Am J Emer Med 1984;12: 113–7. DeRosa DA, Folse R. Evaluation of a system designed to enhance the resident selection process. Surgery 1991;109:715–21. Vu NV, Distlehorst LH, Verhulst SJ, Colliver JA. Clinical performance based test sensitivity and specifivity in predicting first-year residency performance. Acad Med 1993;68:S41–5. Distelhorst LH, Verhulst SJ, Vu NV, Barrows HS. The use of performance based assessment scores and traditional measures in predicting first year residency performance. Proc Annu Conf Res Med Educ 1988;27:176 – 81. Gay SB, Hillman BJ, McNulty BC, et al. The effect of preradiology clinical training on the performance of radiology residents. Invest Radiol 1993;28:1090 – 4. McDonald JS, Lingam RP, Jacoby J, et al. Psychological testing as an aid to selection of residents in anesthesiology. Anesth Analg 1994;78:542–7. Friedman CP, Helm KP, Trier WC, et al: Predictive validity of a house office selection process at one medical school. Acad Med 1991;66:471–1.
PII S0736-4679(98)00138-3
Education
EVALUATING APPLICANTS TO EMERGENCY MEDICINE RESIDENCY PROGRAMS Jerry Balentine,
DO, FAAEM,*
Theodore Gaeta,
DO, FACEP,†
and Theodore Spevack,
DO, FACOEP‡
*St. Barnabas Hospital, Bronx, New York, USA, †Methodist Hospital, Brooklyn, New York, USA, ‡NYCOM/St. Barnabas Hospital, Bronx, New York Reprint Address: Jerry Balentine, DO, Department of Emergency Medicine, St. Barnabas Hospital, 4422 Third Avenue, Bronx, NY 10457
e Abstract—This article reviews the pertinent literature related to the selection process of medical students to emergency medicine residency programs. The impact that academic performance in medical school, the interview, letters of recommendation, and other achievements have on the performance of the future resident are reviewed. All articles identified by an English language MEDLINE search were reviewed by the authors as to significance to the subject. Review of relevant literature indicates that no precise correlation can be made between performance in medical school and achievements during the residency, although there seems to be a correlation between academic performance in medical school and similar performance on board certification examinations. © 1999 Elsevier Science Inc.
dency candidate. In 1991, EM resident applicants spent an average of $1725 and 18 days away from their medical school rotations due to interviews (1). The applicants’ perception of the program will then further determine how high the applicant ranks the program (2). Some residency programs have developed databases in order to evaluate and track information regarding their applicants while others rely on a more traditional method (3). The purpose of this article is to review the literature relating to the selection process, and then evaluate the various criteria used to predict which candidates will make successful residents. It should be noted, however, that the majority of literature has been published by specialties other than emergency medicine and it is unclear how the results of this research relate to our specialty. It is also worth noting that in the majority of cases, whether or not a resident is deemed successful depends upon performance evaluations from program directors and supervising physicians. Such evaluations harbor potential reliability problems due to possible bias, personal preferences, and a lack of standardization. The reliability of these performance assessments have not been clearly established, though reports of positive correlation do exist when two independent supervisors evaluate the same resident (4).
e Keywords—residency; selection; evaluating applicants; interview
INTRODUCTION The selection of candidates to an Emergency Medicine (EM) residency program, an annual event that will shape an individual residency program for several years, requires expenditure of considerable financial as well as temporal resources by both the institution and the resi-
Education is coordinated by Stephen R. Hayden, Diego, California
RECEIVED: 31 October 1997; FINAL ACCEPTED: 20 February 1998
SUBMISSION RECEIVED:
MD,
of The University of California San Diego Medical Center, San
6 February 1998; 131
132
J. Balentine, T. Gaeta, and T. Spevack
Table 1. Measures Used to Evaluate Applicants Interview Medical school grades and class rank Results of board examinations Letters of recommendation Clerkship Prior Experiences (research; other residency training . . . )
MEASURES USED TO EVALUATE APPLICANTS The measures available to most programs in the selection process are listed in Table 1. INTERVIEW Emergency medicine residencies require an interview to be considered for a position in their programs. When surveying residency directors on the importance of different factors in the selection process, the interview was frequently cited to be the most important tool (5). Wagoner in two separate surveys of program directors in internal medicine, family practice, surgery and pediatrics found interpersonal skills as demonstrated on the interview as the most important selection criteria (6,7). The same result was obtained in a survey of physical medicine and rehabilitation directors and interviews with 16 psychiatry residency program directors (8,9). A survey of residency directors in radiology programs showed significant differences between programs with regard to the emphasis placed on class rank and medical school grades (10). Such importance given to the interview by program directors has never been substantiated by data showing that the results of the interview correlate with future performance. In fact, in a study of PGY I residents in an internal medicine residency, no correlation between interview scores and scores at the end of their PGY I year (as determined by faculty evaluation) were apparent (11). Could an improvement in our interview process help with these results? Altmaier et al., in a study of medical students applying to radiology residencies, compared the traditional interview process with behavior based interviewing (an interview technique focusing on experiences, behaviors, knowledge and skills that are job related). They confirm the low predictive value of traditional interviews but find that the results of the behavior based interviewing add considerable predictive value (12). ACADEMIC MEASURES Medical school performance as measured by grades, class rank, or performance on a national test (NBME or
USMLE) offers the easiest way to measure performance as a student and compare to it with performance as a resident. In several studies, medical school administrators followed their graduates longitudinally. They had program directors fill out questionnaires pertaining to the performance of the residents in their program. These results were then correlated to variables from the student’s academic record. The findings of these studies are conflicting. National Board of medical examiners (NBME) Part I and Part II scores did not correlate well with performance as residents (13,14). Alternatively, two studies revealed a correlation for students who were in the top quarter and bottom quarter, each remaining in the same group when evaluations were based on their clinical performance (15,16). Similar results should be expected from the United States Medical Licensing Examination (USMLE), although not enough data are available (17). Performance on the NBME does correlate with success in passing the board certification examinations in orthopedic surgery, dermatology, preventive medicine, (18) and internal medicine (19) while faculty evaluations of residents do not correlate with success on the board examination (20). If this is our most important outcome performance measure, then the NBME results should weigh heavily. Similarly, class rank and medical school grades show only modest correlation with residency performance (21,22). There seems to be a trend towards a correlation at the extremes (top of the class and bottom of the class).
Letters of Recommendation The letter of recommendation is often expected to be a source of reliable information supplied by a person who knows and has worked with the candidate (23). Unfortunately, when letters of recommendation are reviewed, important information such as clinical judgment and interpersonal communication skills are often lacking (24). When evaluating letters of recommendation, a group of academic surgeons stated that the origin of the letter (academic rank and school of origin) were important factors in ranking the letter of recommendation, but in the same study no difference was found between the ranking of a letter whether it was provided to the evaluator on original letterhead and with signature or as a “text only” blank letter (25). In the same study, the overall “theme” of the letter often outweighed specific statements, for example, the phrase “If I can provide any additional information please call . . .” was almost uniformly identified as a negative comment, but it was also found in a letter rated as one of the highest recommendations. Emergency Medicine is currently trying to circumvent this problem by adopting a standardized letter
Evaluating Applicants
of recommendation. The results of the use of these new letters will have to be reviewed in the future.
Clerkship Evaluations The clinical clerkship provides the student with the opportunity to learn about the field of emergency medicine and acquire the clinical knowledge necessary for future training. It also gives the faculty an opportunity to observe a student. Unfortunately, the information gained about the student is frequently not interpreted and recorded appropriately (26). In addition, with different clerkships having different objectives and evaluation tools, even a standardized student evaluation form has limited predictive value when used for surgical clerkships (27). Standardized patient-based postclerkship examinations (28) and performance based assessment scores after clerkships (29) have shown a stronger predictive value than traditional evaluations, but are not universally available. Therefore, unless one is familiar with a given clerkship or its evaluation methods, limited information is gained from a clerkship evaluation from an outside institution. Direct observation of candidates during a clerkship may be the only valid model for candidates applying to a program. Unfortunately, many programs have limited numbers of student clerkships available and most candidates have not performed clerkships at their hospital.
PRIOR TRAINING AND EXPERIENCE With newer reimbursement issues, prior training will probably become a disadvantage for applicants who have previous residency training. In the radiology literature, no advantage was found in the performance of residents with prior training compared to residents with no prior training (30). This does not necessarily apply to the practice of emergency medicine. The results of other prior experiences [(prehospital training, research experience)] on future resident performance in emergency medicine has not been studied and therefore cannot be evaluated.
OTHER METHODS In one study, psychological testing was used to evaluate desirable qualities in residents (31). Once qualities thought to be advantageous to emergency medicine residents are identified, psychological tests may provide input into the application process. In fact, some institutions already use an elaborate evaluation method criteria obtained by assigning relative weights to different traits
133
that would be expected in an ideal house officer and use the results of this system to obtain a ranking list (32). These criteria will be different from specialty to specialty and location to location and are therefore not universally applicable.
LIMITATIONS Most of the literature reviewed is not specific to emergency medicine residencies. Although the basic qualities needed during residency may be similar in the different areas of medicine, certain skills needed in Emergency Medicine could be different from a radiology or surgical residency. Extrapolation of results of some of these studies may be beyond the scope of the original objective. In most studies the outcome criterion was an evaluation by faculty members or the program director, who might have had limited exposure to the resident or involved their own biases of positive and negative qualities in evaluating a resident.
DISCUSSION To date, the literature fails to provide guidance in predicting students’ performance during future residency. There is a trend towards the “top” students (academically) staying at the top as residents and the bottom staying at the bottom, but even this was not a universal truth. The lack of correlation between undergraduate and graduate performance might be inherent in the system. The expectations for medical students during the first two years of training are different from those of residents. Measures such as letters of recommendations, clerkships, and prior experiences show poor correlation with future residency success. We suspect the best correlation can be achieved if a combination of factors is considered and different weight assigned to measures depending on the importance of that quality to the program. Each residency will have a different perception of the “ideal” resident. This must be determined by the faculty and program director, and this should influence the ranking of applicants. Residencies might want to project a certain image (research oriented; academic program) or prefer to balance their program with applicants with different backgrounds and strengths and weaknesses. External forces must be considered. Issues such as less reimbursement for residents with prior training and limitations on residency positions will play into the selection process. We should expect that with mandatory decreases in residency training positions, a program might try to attract residents who will excel at measures used to evaluate residency success in their respective
134
J. Balentine, T. Gaeta, and T. Spevack
institutions. If passing the ABEM certification examination is the institution’s measure of success, then the
program might gear the curriculum and ranking of candidates towards a more academic background.
REFERENCES 1. Aghababian R, Tandberg D, Iserson K, Martin M, Sklar D. Selection of emergency medicine residents. Ann Emerg Med 1993;22: 1753– 61. 2. Koscove EM. An applicant’s evaluation of an emergency medicine internship and residency. Ann Emerg Med 1990;19:1774 – 80. 3. Ledwith M, Myers A. A database management system for residency applicants files. Computers and Biomed Res 1988;21:186 – 91. 4. Davis JK, Inamdar S. The reliability of performance assessment during residency. Acad Med 1990;65:716. 5. Galazka SS, Kikano GE, Zyzanski S. Methods of recruiting and selecting residents for US family practice residencies. Acad Med 1994;69:304 – 6. 6. Wagoner NE, Gray GT. Report of a survey of program directors regarding selection factors in graduate medical education. J Med Edu 1979;54:445–52. 7. Wagoner NE, Suriano JR, Stoner JA. Factors used by program directors to select residents. J Med Educ 1986;61:10 –21. 8. DeLisa JA, Jain SS, Campagnolo DI. Factors used by physical medicine and rehabilitation residency training directors to select residents. Am J Phys Med Rehab 1994;73:152– 6. 9. Ross CA, Leichner P. Criteria for selection residents: A reassessment. Can J Psychiatry 1984;29:681– 6. 10. Grantham JR. Radiology resident selection: Results of a survey. Residency Directors Newsl 1993;28:99 –101. 11. Komives E, Weiss ST, Rosa R. The applicant interview as a predictor of resident performance. J Med Educ 1984;59:425– 6. 12. Altmaier EM, Smith WL, O’Halloran CM, Franken EA, Jr. The predictive utility of behavior based interviewing compared with traditional interviewing in the selection of radiology residents. Invest Radiol 1992;27:385–9. 13. Smith SR. Correlations between graduates’ performance as firstyear residents and their performances as medical students. Acad Med 1993;68:633–5. 14. Wood PS, Smith WL, Altmaier EM, Tarico VS, Franken EA. A prospective study of cognitive and noncognitive selection criteria as predictors of resident performance. Investig Rad 1990; 25: 855–9. 15. Markert RJ. The relationship of academic measures in medical school to performance after graduation. Acad Med 1993;68:S31–5. 16. Yindra KJ, Rosenfeld PS, Donnelly MB. Medical school achievements as predictors of residency performance. J Med Educ 1988; 63:357– 63. 17. Berner ES, Brooks CM, Erdman JB. Use of the USMLE to select residents. Acad Med 1993;68:753–9. 18. Case SM, Swanson DB. Validity of NBME Part I and Part II
19.
20.
21. 22.
23.
24.
25.
26.
27. 28.
29.
30.
31.
32.
Scores for selection of residents in orthopaedic surgery, dermatology and preventive medicine. Acad Med 1993;68:S51–S6. Sosenko J, Stekel KW, Soto R, Gelbard M. NBME Examination part I as a predictor of clinical and ABIM certifying examination performance. J Gen Intern Med 1993;8:86 – 8. Wade TP, Andrus CH, Kaminski DL. Evaluations of surgery resident performance correlate with success in board examinations. Surgery 1993;113:644 – 8. Arnold LA, Willoughby TL. The empirical association between student and resident performances. Acad Med 1993;68:S35–S40. Loftus LS, Arnold L, Willoughby TL, Connolly A. First year residents performances compared with their medical school class ranks determined by three ranking systems. Acad Med 1992;67: 319 –23. Baker JD, Bailey MK, Brahen NH, Conroy JM, Dorman BH, Haynes GR. Selection of anesthesiology residents. Acad Med 1993;68:161–3. O’Halloran CM, Altmaier EM, Smith W, Frabken EA. Evaluation of resident applicants by letters of recommendation: A comparison of traditional and behavior-based formats. Invest Radiol 1993;3: 274 –7. Greenburg AG, Doyle J, McClure DK. Letters of recommendation for surgical residents: What they say and what they mean. J Surg Res 1994;56:192– 8. Teneyck RP, Maclean TA. Improving the quality of emergency medicine rotation/clerkship evaluations. Am J Emer Med 1984;12: 113–7. DeRosa DA, Folse R. Evaluation of a system designed to enhance the resident selection process. Surgery 1991;109:715–21. Vu NV, Distlehorst LH, Verhulst SJ, Colliver JA. Clinical performance based test sensitivity and specifivity in predicting first-year residency performance. Acad Med 1993;68:S41–5. Distelhorst LH, Verhulst SJ, Vu NV, Barrows HS. The use of performance based assessment scores and traditional measures in predicting first year residency performance. Proc Annu Conf Res Med Educ 1988;27:176 – 81. Gay SB, Hillman BJ, McNulty BC, et al. The effect of preradiology clinical training on the performance of radiology residents. Invest Radiol 1993;28:1090 – 4. McDonald JS, Lingam RP, Jacoby J, et al. Psychological testing as an aid to selection of residents in anesthesiology. Anesth Analg 1994;78:542–7. Friedman CP, Helm KP, Trier WC, et al: Predictive validity of a house office selection process at one medical school. Acad Med 1991;66:471–1.