- Email: [email protected]
Can a structured, behavior-based interview predict future resident success?
Research
www. AJOG.org
EDUCATION
Can a structured, behavior-based interview predict future resident success? Eric A. Strand, MD; Elizabeth Moore, PhD; Douglas W. Laube, MD, MEd OBJECTIVE: To determine whether a structured, behavior-based appli-
cant interview predicts future success in an obstetrics and gynecology residency program. STUDY DESIGN: Using a modified pre-post study design, we compared behavior-based interview scores of our residency applicants to a postmatch evaluation completed by the applicant’s current residency program director. Applicants were evaluated on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program. RESULTS: Information was obtained for 45 (63%) applicants. The over-
all interview score did not correlate with overall resident performance.
Applicant leadership subscore was predictive of leadership performance as a resident (P ⫽ .042). Academic record was associated with patient care performance as a resident (P ⫽ .014), but only for graduates of US medical schools. Five residents changed programs; these residents had significantly lower scores for trainability/suitability for the specialty (P ⫽ .020). CONCLUSION: Behavioral interviewing can provide predictive informa-
tion regarding success in an obstetrics and gynecology training program. Key words: internship/residency, personnel selection, professional competence
Cite this article as: Strand EA, Moore E, Laube DW. Can a structured, behavior-based interview predict future resident success? Am J Obstet Gynecol 2011;204:446.e1-13.
E
ach academic year, residency programs and applicants expend significant time, energy, and financial resources interviewing applicants for future residency positions. In the 2009 National Residency Matching Program (NRMP) main From the Department of Obstetrics and Gynecology (Drs Strand and Moore), St. Vincent Hospital, Indianapolis, IN, and the Department of Obstetrics and Gynecology (Dr Laube), University of Wisconsin, Madison, WI. Presented at the annual joint meeting of the Association of Professors of Gynecology and Obstetrics and the Council on Resident Education in Obstetrics and Gynecology, Orlando, FL, March 3-6, 2010. Received Aug. 26, 2010; revised Nov. 8, 2010; accepted Feb. 4, 2011. Reprints: Eric A. Strand, MD, Department of Obstetrics and Gynecology, St. Vincent Hospital, 8111 Township Line Rd, Indianapolis, IN 46260. [email protected]. 0002-9378/free © 2011 Mosby, Inc. All rights reserved. doi: 10.1016/j.ajog.2011.02.019
For Editors’ Commentary, see Table of Contents See related editorial, page 369
446.e1
residency match, a total of 51,882 applicants pursued the 25,185 training positions available in the United States. This included 1796 applicants for the 1185 residency positions in the specialty of obstetrics and gynecology.1 Despite these investments, little is known about the use of the residency interview in predicting an applicant’s future performance as a resident. For instance, Metro et al2 reviewed interview scores for their applicants to determine whether the scores correlated with any measures used to evaluate the residents during their training. Interview scores did not correlate with any of the measures, including knowledge, judgment, motor skills, or intrapersonal attitudes.2 Performance on the United States Medical Licensing Examination (USMLE) has been shown to be positively correlated with intraining service examinations, but not with faculty evaluation of resident performance.3,4 Other studies have shown interview scores did not predict physicians at risk of later impairment.5 Furthermore, resident attrition is a significant problem for many programs. After a cohort of 1055 residents started their obstetrics and gynecology residency training in 2001, McAlis-
American Journal of Obstetrics & Gynecology MAY 2011
ter et al6 found that 21.6% of residents either: switched programs, changed specialties, completed training on an atypical academic cycle, or left graduate medical education all together. In 2006, the St. Vincent Hospital Department of Obstetrics/Gynecoloy embarked on a program of behavior-based interviewing for all applicants to its residency program, with the hope of identifying applicants who would be successful both in the chosen specialty and in the St. Vincent program. This study reports the results of this interview process to predict future resident performance.
M ATERIALS AND M ETHODS The St. Vincent Hospital Obstetrics/Gynecology residency is a communitybased program consisting of 4 residents per academic year (approved for expansion to 5 positions in June 2007). The hospital is 1 of 4 main teaching sites for the required third year obstetrics/gynecology clerkship for medical students from the Indiana University School of Medicine. Approximately 200-250 students apply for the available residency positions, with the majority of the US graduates matriculating from programs in the Midwest region.
Education
www.AJOG.org
FIGURE 1
Overall interview score vs overall performance as assessed by program director
P ⫽ .784. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
Beginning in November 2006, the St. Vincent Hospital Department of Obstetrics/Gynecology embarked on a structured behavior-based interview for applicants to its residency program. Applicants were scored on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program. Academic record was scored by the program director through a review of the information available in the written application. The other 4 aspects were measured through a series of behavior-based questions at individual interview stations. Interviews for professionalism, leadership, and trainability/suitability were conducted simultaneously by 2 faculty members. When scheduling prevented the presence of both members, 1 faculty member conducted the interview. Fit for the program was assessed by a panel of 3 residents. Through literature review and discussions with subject matter experts, behavior-based questions pertinent to each subject area were developed, as were scoring sheets with specific examples of scores for particular responses
(measured aspects of each subject area, sample scoring sheets, and sample questions, Supplementary Figures 1-4). Faculty and residents were educated regarding the new interview protocol during a series of formal and informal meetings. During the interviews, faculty and residents could ask any of the potential questions available for their session, allowing some flexibility from interview to interview. At the end of each interview encounter, every interviewing faculty or resident completed their score sheet separately and independently from their partner(s). Scores for all 5 categories were determined by the average of the interviewer’s scores. With each category having a potential value of 36, the maximum interview score for each applicant was 180. These scores provided the initial ranking list of applicants on which all future discussions were based. A database was maintained with all interview data, as well as medical school location (USbased program or international-based program). For all applicants from November 2006 through January 2008, eventual
Research
match sites were identified through a combination of (1) the NRMP Match Results by Matched Applicant provided to the St. Vincent program, (2) the Council on Resident Education in Obstetrics and Gynecology (CREOG) resident database, and (3) individual program web site reviews. The program director for each program was identified through the listing of accredited programs at the Accreditation Council for Graduate Medical Education website (www.acgme.org). An electronic survey was developed and sent to each director of a program into which a St. Vincent applicant had matched. Program director e-mails were identified through the AGCME database. E-mails with links to the electronic survey were sent to the appropriate program directors in September 2009, at which time the former applicants would have completed either 1 or 2 years of residency training. The survey was designed to measure the following resident attributes: ● Patient care ● Medical knowledge ● Surgical skills ● Communication ● Professionalism ● Clinical documentation ● Leadership ● Teamwork ● Overall impression In addition, questions regarding awards, disciplinary actions for academic or professionalism-related issues, and the resident’s continued status in the program were included. If the resident was no longer with the training program, program directors were asked to explain the change in training status. Information on type of program (community or university) was also collected. For residents matching into the St. Vincent training program, it was thought the program director’s completion of the survey would lead to potential bias, as he is the principal investigator of this study (E.S.). Therefore, the same survey for St. Vincent residents was completed by the consensus opinion of a core group of 3 faculty members with extensive clinical experience with the residents.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e2
Research
Education
To maintain applicant confidentiality, once data were collected from the electronic survey and matched to the original interview score, all data were deidentified. When means between groups were compared, Student t tests were performed to determine statistical significance. Linear regression was used to determine relationships between pretest (interview) and posttest (electronic survey) scores. A P value of ⬍ .05 was considered significant. The project was submitted to the St. Vincent Hospital Institutional Review Board (IRB) and was granted exempt status as an educational project.
www.AJOG.org
FIGURE 2
Interview score of academic record vs patient care performance as a resident, US medical school graduates only
R ESULTS From November 2006 through January 2008, a total of 80 applicants interviewed for residency positions in obstetrics/ gynecology at St. Vincent Hospital. Of these applicants, 65 (81%) were women, 15 (19%) were men, and 74 (92.5%) were from US allopathic training programs. Of these, 72 matched into obstetrics/gynecology training programs, and 8 either did not match or matched into other specialty programs (4 family medicine, 1 pediatrics, 1 general surgery, 1 anesthesia, and 1 unmatched). The 72 applicants matched into a total of 42 obstetrics/gynecology residency training programs. Of the 42 electronic surveys sent to these program directors, a total of 24 surveys (57%) were completed, providing follow-up data for 36 of the applicants. Including the 9 residents matching into the St. Vincent program, this provided follow-up data for 45 (63%) of the applicants matching into obstetrics/gynecology training programs. In evaluating the assessment of overall resident performance by their program director, there was not a significant relationship between the overall interview score and the overall performance score (P ⫽ .784, Figure 1). This remained the case when separating results based by type of training program (university based, n ⫽ 25, P ⫽ .337; community based, n ⫽ 20, P ⫽ .952). Scores for the applicant’s academic record did not predict resident scores for patient care (P ⫽ .147) or medical knowledge 446.e3
P ⫽ .014. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
(P ⫽ .125). However, when analyzing results for only graduates of US medical schools (43/45 applicants with available data), a significant relationship did exist between academic record and patient care (P ⫽ .014, Figure 2). Scores for medical knowledge, though, remained nonsignificant (P ⫽ .075). Applicant scores for professionalism did not predict resident scores for professionalism (P ⫽ .685, Figure 3). However, a relationship did exist between applicant scores in leadership and leadership scores as a resident (P ⫽ .042, Figure 4). Of the 45 applicants with available data, 5 (11%) changed programs during the time of the study (3 changed to different obstetrics/gynecology programs, 1 to anesthesia, and 1 to emergency medicine). Based on univariate logistic regression of interview scores to program status (still in program or changed programs), only trainability/ suitability was a significant predictor of remaining in the training program (P ⫽ .020, Table 1). Fit for the program
American Journal of Obstetrics & Gynecology MAY 2011
was not predictive of resident attrition (P ⫽ .898, Table 1).
C OMMENT Developed by industrial psychologists, behavioral interviewing focuses on the premise that past performance is the best indicator of future performance. Instead of using vague questions such as “What are your strengths and weaknesses?” behavioral interviews involve questions regarding specific behaviors that often represent decision-making skills, critical thinking, and interpersonal communication styles. In one of the first studies to evaluate the use of behavioral interviews, Janz7 compared behavior-based interviews to traditional, unstructured interviews of prospective teaching assistants. Using student ratings of the teaching assistants as the outcome measure, behavioral interviews were significantly more predictive of performance than the traditional interview.7 Given the potential benefit, many businesses have changed to behavior-based interviewing, includ-
Education
www.AJOG.org
Research
FIGURE 3
TABLE
Interview professionalism score vs professionalism rating as a resident
Univariate logistic regression of interview scores vs continued status in the program
Interview item
Prediction of remaining in program (P value)
Academic record
.930
Professionalism
.916
Leadership
.430
Suitability/trainability
.020
Fit for program
.898
Overall score
.575
........................................................................................................... ........................................................................................................... ........................................................................................................... ........................................................................................................... ........................................................................................................... ...........................................................................................................
Strand. Predicting resident success. Am J Obstet Gynecol 2011.
P ⫽ .685. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
FIGURE 4
Interview leadership score vs leadership rating as a resident
P ⫽ .042. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
ing 65% of Fortune 500 companies.8 Because cognitive factors, such as grades/ transcripts, USMLE scores, and class rank have not correlated with clinical success as a resident,3-4,9,10 programs are turning to behavioral interviewing for candidate assessment.11 In this study of a standardized behavior-based interview process, important significant associations were found. Specifically, applicants with higher scores for leadership tended to be rated higher for this characteristic as a resident. In addition, applicants who changed training programs had lower interview scores in trainability/suitability. This may illustrate an important risk factor to be considered in selecting an applicant, given the significant time, expense, and disruption that occurs when a resident leaves a program and a replacement is sought. Interestingly, one measure of our applicant assessment, academic record, was not behavior-based and yet did significantly correlate with patient care performance as a resident. This stands in contrast to the cognitive factors assessed in the previous studies. Our finding was significant only when comparing data for graduates of US medical schools. The association lost significance when data for applicants from international medical schools was included. This may indicate that cognitive factors (USMLE MAY 2011 American Journal of Obstetrics & Gynecology
446.e4
Research
Education
scores, transcripts) for international graduates may be particularly poor predictors of future performance, and should be considered in the applicant’s overall assessment. Our global interview score, though, did not predict future performance as a resident. This stands in contrast to a study by Olawaiye et al,12 where an applicant’s position on the program’s NRMP rank list did predict future first year clinical performance. There may be several reasons for the different findings. In the study by Olawaiye et al,12 similar but different categories were measured during the interview— communication skills, insight into specialty, motivation, compassion, and “fit” into the program. The nature of the interview questions (behavior-based or otherwise) was not described. Finally, resident performance was not compared with the overall interview score (or subscore), but rather to “rank percentile,” which represented the applicant’s position on the final ranking list. Our study is the first attempt to use behavioral interviewing to predict future resident performance. In addition, ours is the first to go beyond global scores and investigate subscores such as leadership, professionalism, and trainability/suitability as possible predictors of future success. These findings point to areas of potential future study, as programs look for predictive characteristics of resident success or failure. One strength of the study is its assessment of residents in a number of different programs. Program directors completing the survey were unaware of the original applicant’s evaluation, which allowed for an unbiased assessment of the applicant’s performance as a resident. Removing the St. Vincent program director from the evaluation process limited this bias as well, although a potential for bias remained if the St. Vincent
446.e5
www.AJOG.org faculty involved remembered the applicant’s interview results. Interviews had occurred roughly 2 to 3 years previously, which makes significant bias unlikely. An additional strength of the study was the preparation of residents and faculty for the new interview technique, and the creation of standardized grading templates with specific examples of answers corresponding to specific marks. This should have limited variation in evaluations submitted on different interview dates. There are several weaknesses to the study. Although the survey provided data for 63% of our applicants, it is unknown if the remaining 37% were more or less successful as residents. Program directors with residents demonstrating poor performance may have been less likely to complete a survey regarding their performance, despite reassurances regarding confidentiality of the subjects. Program expectations of a quality resident may also be substantially different among programs. Although our data did not show significant differences when analyzed comparing results from community-based or university-based programs, variations in programs may create different definitions of “success.” Although we attempted to standardize our survey for the program directors, we certainly must allow that expectations of performance vary from program to program. Overall, our study points to the potential usefulness of behavioral interviewing to identify either predictors of success or risk factors for failure during residency. Trainability/suitability, in particular, may be a subject of further interest in identifying applicants at risk for changing programs. Future studies should investigate additional noncognitive traits of our applicants to help better predict future residency success. f
American Journal of Obstetrics & Gynecology MAY 2011
REFERENCES 1. National Residency Matching Program results and data: 2009 main residency match. National Residency Matching Program, 2009 (vol 2009). 2. Metro DG, Talarico JF, Patel RM, Wetmore AL.The resident application process and its correlation to future performance as a resident. Anesth Analg 2005;100:502-5. 3. Bell JG, Kanellitsas I, Shaffer L. Selection of obstetrics and gynecology residents on the basis of medical school performance. Am J Obstet Gynecol 2002;186:1091-4. 4. Brothers TE, Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ 2007;64:378-85. 5. Dubovsky SL, Gendel M, Dubovsky AN, Rosse J, Levin R, House R. Do data obtained from admissions interviews and resident evaluations predict later personal and practice problems? Acad Psychiatry 2005;29: 443-7. 6. McAlister RP, Andriole DA, Brotherton SE, Jeffe DB. Attrition in residents entering US obstetrics and gynecology residencies: analysis of national GME census data. Am J Obstet Gynecol 2008;199:574.e1-6. 7. Janz T. Initial comparisons of patterned behavior-based interviews versus unstructured interviews. J Appl Psychol 1982;67:577-80. 8. Byers M. Interview Rx: a powerful guide for making your next interview a success, 3rd ed. Conyers, GA: Nearline Publishers; 2007. 9. Erlandson EE, Calhoun JG, Barrack FM, et al. Resident selection: applicant selection criteria compared with performance. Surgery 1982;92: 270-5. 10. Hojat M, Gonnella JS, Veloski JJ, Erdmann JB. Is the glass half full or half empty? A reexamination of the associations between assessment measures during medical school and clinical competence after graduation. Acad Med 1993;68:S69-76. 11. Lyon D, Wiper D. Finding our colleagues, finding ourselves: behavioral interviewing and a critical assessment of how we identify and interview residency candidates. APGO/CREOG Annual Meeting. San Diego, CA, 2009. 12. Olawaiye A, Yeh J, Withiam-Leitch M. Resident selection process and prediction of clinical performance in an obstetrics and gynecology program. Teach Learn Med 2006; 18:310-5.
www.AJOG.org
Education
Research
SUPPLEMENTARY FIGURE 1
Factors measured in St. Vincent’s behavioral interviews
Strand. Predicting resident success. Am J Obstet Gynecol 2011.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e6
Research
Education
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets
Continued on next page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
446.e7
American Journal of Obstetrics & Gynecology MAY 2011
www.AJOG.org
Education
www.AJOG.org
Research
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets Professionalism 1. Interview Day/Appearance
1
Unkempt; rude; late
3
Well-groomed; slow or difficult to engage
5
7
9
Appropriately dressed, timely, polite; interacts well but must be led by interviewers
Pleasant, courteous, wellgroomed; enough selfconfidence to have some poise in interaction
Pleasant, courteous, wellgroomed; selfassured, wellspoken; completely comfortable with the interview
2. Ethics
1
3
5
7
9
Cannot identify ethical challenge or situation
Can identify ethical situation or challenge, but has no insight into response
Shows basic understanding of complexities within ethical dilemmas
Demonstrates understanding of ethical principles (informed consent, beneficence, etc.)
Provides engaging discussion of difficult ethical situations
3. Stress
1
3
5
7
9
Denies stressful situations or being effected by stress
Aware of stressful situations but not of personal effects
Describes stressful situation and some evidence of personal approach
Has clearly defined ways of personally handling stress
Describes stressful situation, well-adjusted response, and individual growth from experience
1
3
5
7
9
Evidence of initiative leading to personal pride
Describes several situations requiring initiative as well as how this contributed to personal growth
4. Initiative
Offers no evidence of personal initiative
Limited evidence of initiative
Repeated evidence of initiative in professional or personal life
Total Score:________________
Continued from the previous page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e8
Research
Education
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets
Continued from the previous page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
446.e9
American Journal of Obstetrics & Gynecology MAY 2011
www.AJOG.org
Education
www.AJOG.org
Research
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets
Continued from the previous page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e10
Research
Education
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets
Continued from the previous page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
446.e11
American Journal of Obstetrics & Gynecology MAY 2011
www.AJOG.org
Education
www.AJOG.org
Research
SUPPLEMENTARY FIGURE 3
Sample interview questions
Strand. Predicting resident success. Am J Obstet Gynecol 2011.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e12
Research
www. AJOG.org
EDUCATION
Can a structured, behavior-based interview predict future resident success? Eric A. Strand, MD; Elizabeth Moore, PhD; Douglas W. Laube, MD, MEd OBJECTIVE: To determine whether a structured, behavior-based appli-
cant interview predicts future success in an obstetrics and gynecology residency program. STUDY DESIGN: Using a modified pre-post study design, we compared behavior-based interview scores of our residency applicants to a postmatch evaluation completed by the applicant’s current residency program director. Applicants were evaluated on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program. RESULTS: Information was obtained for 45 (63%) applicants. The over-
all interview score did not correlate with overall resident performance.
Applicant leadership subscore was predictive of leadership performance as a resident (P ⫽ .042). Academic record was associated with patient care performance as a resident (P ⫽ .014), but only for graduates of US medical schools. Five residents changed programs; these residents had significantly lower scores for trainability/suitability for the specialty (P ⫽ .020). CONCLUSION: Behavioral interviewing can provide predictive informa-
tion regarding success in an obstetrics and gynecology training program. Key words: internship/residency, personnel selection, professional competence
Cite this article as: Strand EA, Moore E, Laube DW. Can a structured, behavior-based interview predict future resident success? Am J Obstet Gynecol 2011;204:446.e1-13.
E
ach academic year, residency programs and applicants expend significant time, energy, and financial resources interviewing applicants for future residency positions. In the 2009 National Residency Matching Program (NRMP) main From the Department of Obstetrics and Gynecology (Drs Strand and Moore), St. Vincent Hospital, Indianapolis, IN, and the Department of Obstetrics and Gynecology (Dr Laube), University of Wisconsin, Madison, WI. Presented at the annual joint meeting of the Association of Professors of Gynecology and Obstetrics and the Council on Resident Education in Obstetrics and Gynecology, Orlando, FL, March 3-6, 2010. Received Aug. 26, 2010; revised Nov. 8, 2010; accepted Feb. 4, 2011. Reprints: Eric A. Strand, MD, Department of Obstetrics and Gynecology, St. Vincent Hospital, 8111 Township Line Rd, Indianapolis, IN 46260. [email protected]. 0002-9378/free © 2011 Mosby, Inc. All rights reserved. doi: 10.1016/j.ajog.2011.02.019
For Editors’ Commentary, see Table of Contents See related editorial, page 369
446.e1
residency match, a total of 51,882 applicants pursued the 25,185 training positions available in the United States. This included 1796 applicants for the 1185 residency positions in the specialty of obstetrics and gynecology.1 Despite these investments, little is known about the use of the residency interview in predicting an applicant’s future performance as a resident. For instance, Metro et al2 reviewed interview scores for their applicants to determine whether the scores correlated with any measures used to evaluate the residents during their training. Interview scores did not correlate with any of the measures, including knowledge, judgment, motor skills, or intrapersonal attitudes.2 Performance on the United States Medical Licensing Examination (USMLE) has been shown to be positively correlated with intraining service examinations, but not with faculty evaluation of resident performance.3,4 Other studies have shown interview scores did not predict physicians at risk of later impairment.5 Furthermore, resident attrition is a significant problem for many programs. After a cohort of 1055 residents started their obstetrics and gynecology residency training in 2001, McAlis-
American Journal of Obstetrics & Gynecology MAY 2011
ter et al6 found that 21.6% of residents either: switched programs, changed specialties, completed training on an atypical academic cycle, or left graduate medical education all together. In 2006, the St. Vincent Hospital Department of Obstetrics/Gynecoloy embarked on a program of behavior-based interviewing for all applicants to its residency program, with the hope of identifying applicants who would be successful both in the chosen specialty and in the St. Vincent program. This study reports the results of this interview process to predict future resident performance.
M ATERIALS AND M ETHODS The St. Vincent Hospital Obstetrics/Gynecology residency is a communitybased program consisting of 4 residents per academic year (approved for expansion to 5 positions in June 2007). The hospital is 1 of 4 main teaching sites for the required third year obstetrics/gynecology clerkship for medical students from the Indiana University School of Medicine. Approximately 200-250 students apply for the available residency positions, with the majority of the US graduates matriculating from programs in the Midwest region.
www. AJOG.org
EDUCATION
Can a structured, behavior-based interview predict future resident success? Eric A. Strand, MD; Elizabeth Moore, PhD; Douglas W. Laube, MD, MEd OBJECTIVE: To determine whether a structured, behavior-based appli-
cant interview predicts future success in an obstetrics and gynecology residency program. STUDY DESIGN: Using a modified pre-post study design, we compared behavior-based interview scores of our residency applicants to a postmatch evaluation completed by the applicant’s current residency program director. Applicants were evaluated on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program. RESULTS: Information was obtained for 45 (63%) applicants. The over-
all interview score did not correlate with overall resident performance.
Applicant leadership subscore was predictive of leadership performance as a resident (P ⫽ .042). Academic record was associated with patient care performance as a resident (P ⫽ .014), but only for graduates of US medical schools. Five residents changed programs; these residents had significantly lower scores for trainability/suitability for the specialty (P ⫽ .020). CONCLUSION: Behavioral interviewing can provide predictive informa-
tion regarding success in an obstetrics and gynecology training program. Key words: internship/residency, personnel selection, professional competence
Cite this article as: Strand EA, Moore E, Laube DW. Can a structured, behavior-based interview predict future resident success? Am J Obstet Gynecol 2011;204:446.e1-13.
E
ach academic year, residency programs and applicants expend significant time, energy, and financial resources interviewing applicants for future residency positions. In the 2009 National Residency Matching Program (NRMP) main From the Department of Obstetrics and Gynecology (Drs Strand and Moore), St. Vincent Hospital, Indianapolis, IN, and the Department of Obstetrics and Gynecology (Dr Laube), University of Wisconsin, Madison, WI. Presented at the annual joint meeting of the Association of Professors of Gynecology and Obstetrics and the Council on Resident Education in Obstetrics and Gynecology, Orlando, FL, March 3-6, 2010. Received Aug. 26, 2010; revised Nov. 8, 2010; accepted Feb. 4, 2011. Reprints: Eric A. Strand, MD, Department of Obstetrics and Gynecology, St. Vincent Hospital, 8111 Township Line Rd, Indianapolis, IN 46260. [email protected]. 0002-9378/free © 2011 Mosby, Inc. All rights reserved. doi: 10.1016/j.ajog.2011.02.019
For Editors’ Commentary, see Table of Contents See related editorial, page 369
446.e1
residency match, a total of 51,882 applicants pursued the 25,185 training positions available in the United States. This included 1796 applicants for the 1185 residency positions in the specialty of obstetrics and gynecology.1 Despite these investments, little is known about the use of the residency interview in predicting an applicant’s future performance as a resident. For instance, Metro et al2 reviewed interview scores for their applicants to determine whether the scores correlated with any measures used to evaluate the residents during their training. Interview scores did not correlate with any of the measures, including knowledge, judgment, motor skills, or intrapersonal attitudes.2 Performance on the United States Medical Licensing Examination (USMLE) has been shown to be positively correlated with intraining service examinations, but not with faculty evaluation of resident performance.3,4 Other studies have shown interview scores did not predict physicians at risk of later impairment.5 Furthermore, resident attrition is a significant problem for many programs. After a cohort of 1055 residents started their obstetrics and gynecology residency training in 2001, McAlis-
American Journal of Obstetrics & Gynecology MAY 2011
ter et al6 found that 21.6% of residents either: switched programs, changed specialties, completed training on an atypical academic cycle, or left graduate medical education all together. In 2006, the St. Vincent Hospital Department of Obstetrics/Gynecoloy embarked on a program of behavior-based interviewing for all applicants to its residency program, with the hope of identifying applicants who would be successful both in the chosen specialty and in the St. Vincent program. This study reports the results of this interview process to predict future resident performance.
M ATERIALS AND M ETHODS The St. Vincent Hospital Obstetrics/Gynecology residency is a communitybased program consisting of 4 residents per academic year (approved for expansion to 5 positions in June 2007). The hospital is 1 of 4 main teaching sites for the required third year obstetrics/gynecology clerkship for medical students from the Indiana University School of Medicine. Approximately 200-250 students apply for the available residency positions, with the majority of the US graduates matriculating from programs in the Midwest region.
Education
www.AJOG.org
FIGURE 1
Overall interview score vs overall performance as assessed by program director
P ⫽ .784. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
Beginning in November 2006, the St. Vincent Hospital Department of Obstetrics/Gynecology embarked on a structured behavior-based interview for applicants to its residency program. Applicants were scored on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program. Academic record was scored by the program director through a review of the information available in the written application. The other 4 aspects were measured through a series of behavior-based questions at individual interview stations. Interviews for professionalism, leadership, and trainability/suitability were conducted simultaneously by 2 faculty members. When scheduling prevented the presence of both members, 1 faculty member conducted the interview. Fit for the program was assessed by a panel of 3 residents. Through literature review and discussions with subject matter experts, behavior-based questions pertinent to each subject area were developed, as were scoring sheets with specific examples of scores for particular responses
(measured aspects of each subject area, sample scoring sheets, and sample questions, Supplementary Figures 1-4). Faculty and residents were educated regarding the new interview protocol during a series of formal and informal meetings. During the interviews, faculty and residents could ask any of the potential questions available for their session, allowing some flexibility from interview to interview. At the end of each interview encounter, every interviewing faculty or resident completed their score sheet separately and independently from their partner(s). Scores for all 5 categories were determined by the average of the interviewer’s scores. With each category having a potential value of 36, the maximum interview score for each applicant was 180. These scores provided the initial ranking list of applicants on which all future discussions were based. A database was maintained with all interview data, as well as medical school location (USbased program or international-based program). For all applicants from November 2006 through January 2008, eventual
Research
match sites were identified through a combination of (1) the NRMP Match Results by Matched Applicant provided to the St. Vincent program, (2) the Council on Resident Education in Obstetrics and Gynecology (CREOG) resident database, and (3) individual program web site reviews. The program director for each program was identified through the listing of accredited programs at the Accreditation Council for Graduate Medical Education website (www.acgme.org). An electronic survey was developed and sent to each director of a program into which a St. Vincent applicant had matched. Program director e-mails were identified through the AGCME database. E-mails with links to the electronic survey were sent to the appropriate program directors in September 2009, at which time the former applicants would have completed either 1 or 2 years of residency training. The survey was designed to measure the following resident attributes: ● Patient care ● Medical knowledge ● Surgical skills ● Communication ● Professionalism ● Clinical documentation ● Leadership ● Teamwork ● Overall impression In addition, questions regarding awards, disciplinary actions for academic or professionalism-related issues, and the resident’s continued status in the program were included. If the resident was no longer with the training program, program directors were asked to explain the change in training status. Information on type of program (community or university) was also collected. For residents matching into the St. Vincent training program, it was thought the program director’s completion of the survey would lead to potential bias, as he is the principal investigator of this study (E.S.). Therefore, the same survey for St. Vincent residents was completed by the consensus opinion of a core group of 3 faculty members with extensive clinical experience with the residents.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e2
Research
Education
To maintain applicant confidentiality, once data were collected from the electronic survey and matched to the original interview score, all data were deidentified. When means between groups were compared, Student t tests were performed to determine statistical significance. Linear regression was used to determine relationships between pretest (interview) and posttest (electronic survey) scores. A P value of ⬍ .05 was considered significant. The project was submitted to the St. Vincent Hospital Institutional Review Board (IRB) and was granted exempt status as an educational project.
www.AJOG.org
FIGURE 2
Interview score of academic record vs patient care performance as a resident, US medical school graduates only
R ESULTS From November 2006 through January 2008, a total of 80 applicants interviewed for residency positions in obstetrics/ gynecology at St. Vincent Hospital. Of these applicants, 65 (81%) were women, 15 (19%) were men, and 74 (92.5%) were from US allopathic training programs. Of these, 72 matched into obstetrics/gynecology training programs, and 8 either did not match or matched into other specialty programs (4 family medicine, 1 pediatrics, 1 general surgery, 1 anesthesia, and 1 unmatched). The 72 applicants matched into a total of 42 obstetrics/gynecology residency training programs. Of the 42 electronic surveys sent to these program directors, a total of 24 surveys (57%) were completed, providing follow-up data for 36 of the applicants. Including the 9 residents matching into the St. Vincent program, this provided follow-up data for 45 (63%) of the applicants matching into obstetrics/gynecology training programs. In evaluating the assessment of overall resident performance by their program director, there was not a significant relationship between the overall interview score and the overall performance score (P ⫽ .784, Figure 1). This remained the case when separating results based by type of training program (university based, n ⫽ 25, P ⫽ .337; community based, n ⫽ 20, P ⫽ .952). Scores for the applicant’s academic record did not predict resident scores for patient care (P ⫽ .147) or medical knowledge 446.e3
P ⫽ .014. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
(P ⫽ .125). However, when analyzing results for only graduates of US medical schools (43/45 applicants with available data), a significant relationship did exist between academic record and patient care (P ⫽ .014, Figure 2). Scores for medical knowledge, though, remained nonsignificant (P ⫽ .075). Applicant scores for professionalism did not predict resident scores for professionalism (P ⫽ .685, Figure 3). However, a relationship did exist between applicant scores in leadership and leadership scores as a resident (P ⫽ .042, Figure 4). Of the 45 applicants with available data, 5 (11%) changed programs during the time of the study (3 changed to different obstetrics/gynecology programs, 1 to anesthesia, and 1 to emergency medicine). Based on univariate logistic regression of interview scores to program status (still in program or changed programs), only trainability/ suitability was a significant predictor of remaining in the training program (P ⫽ .020, Table 1). Fit for the program
American Journal of Obstetrics & Gynecology MAY 2011
was not predictive of resident attrition (P ⫽ .898, Table 1).
C OMMENT Developed by industrial psychologists, behavioral interviewing focuses on the premise that past performance is the best indicator of future performance. Instead of using vague questions such as “What are your strengths and weaknesses?” behavioral interviews involve questions regarding specific behaviors that often represent decision-making skills, critical thinking, and interpersonal communication styles. In one of the first studies to evaluate the use of behavioral interviews, Janz7 compared behavior-based interviews to traditional, unstructured interviews of prospective teaching assistants. Using student ratings of the teaching assistants as the outcome measure, behavioral interviews were significantly more predictive of performance than the traditional interview.7 Given the potential benefit, many businesses have changed to behavior-based interviewing, includ-
Education
www.AJOG.org
Research
FIGURE 3
TABLE
Interview professionalism score vs professionalism rating as a resident
Univariate logistic regression of interview scores vs continued status in the program
Interview item
Prediction of remaining in program (P value)
Academic record
.930
Professionalism
.916
Leadership
.430
Suitability/trainability
.020
Fit for program
.898
Overall score
.575
........................................................................................................... ........................................................................................................... ........................................................................................................... ........................................................................................................... ........................................................................................................... ...........................................................................................................
Strand. Predicting resident success. Am J Obstet Gynecol 2011.
P ⫽ .685. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
FIGURE 4
Interview leadership score vs leadership rating as a resident
P ⫽ .042. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
ing 65% of Fortune 500 companies.8 Because cognitive factors, such as grades/ transcripts, USMLE scores, and class rank have not correlated with clinical success as a resident,3-4,9,10 programs are turning to behavioral interviewing for candidate assessment.11 In this study of a standardized behavior-based interview process, important significant associations were found. Specifically, applicants with higher scores for leadership tended to be rated higher for this characteristic as a resident. In addition, applicants who changed training programs had lower interview scores in trainability/suitability. This may illustrate an important risk factor to be considered in selecting an applicant, given the significant time, expense, and disruption that occurs when a resident leaves a program and a replacement is sought. Interestingly, one measure of our applicant assessment, academic record, was not behavior-based and yet did significantly correlate with patient care performance as a resident. This stands in contrast to the cognitive factors assessed in the previous studies. Our finding was significant only when comparing data for graduates of US medical schools. The association lost significance when data for applicants from international medical schools was included. This may indicate that cognitive factors (USMLE MAY 2011 American Journal of Obstetrics & Gynecology
446.e4
Research
Education
scores, transcripts) for international graduates may be particularly poor predictors of future performance, and should be considered in the applicant’s overall assessment. Our global interview score, though, did not predict future performance as a resident. This stands in contrast to a study by Olawaiye et al,12 where an applicant’s position on the program’s NRMP rank list did predict future first year clinical performance. There may be several reasons for the different findings. In the study by Olawaiye et al,12 similar but different categories were measured during the interview— communication skills, insight into specialty, motivation, compassion, and “fit” into the program. The nature of the interview questions (behavior-based or otherwise) was not described. Finally, resident performance was not compared with the overall interview score (or subscore), but rather to “rank percentile,” which represented the applicant’s position on the final ranking list. Our study is the first attempt to use behavioral interviewing to predict future resident performance. In addition, ours is the first to go beyond global scores and investigate subscores such as leadership, professionalism, and trainability/suitability as possible predictors of future success. These findings point to areas of potential future study, as programs look for predictive characteristics of resident success or failure. One strength of the study is its assessment of residents in a number of different programs. Program directors completing the survey were unaware of the original applicant’s evaluation, which allowed for an unbiased assessment of the applicant’s performance as a resident. Removing the St. Vincent program director from the evaluation process limited this bias as well, although a potential for bias remained if the St. Vincent
446.e5
www.AJOG.org faculty involved remembered the applicant’s interview results. Interviews had occurred roughly 2 to 3 years previously, which makes significant bias unlikely. An additional strength of the study was the preparation of residents and faculty for the new interview technique, and the creation of standardized grading templates with specific examples of answers corresponding to specific marks. This should have limited variation in evaluations submitted on different interview dates. There are several weaknesses to the study. Although the survey provided data for 63% of our applicants, it is unknown if the remaining 37% were more or less successful as residents. Program directors with residents demonstrating poor performance may have been less likely to complete a survey regarding their performance, despite reassurances regarding confidentiality of the subjects. Program expectations of a quality resident may also be substantially different among programs. Although our data did not show significant differences when analyzed comparing results from community-based or university-based programs, variations in programs may create different definitions of “success.” Although we attempted to standardize our survey for the program directors, we certainly must allow that expectations of performance vary from program to program. Overall, our study points to the potential usefulness of behavioral interviewing to identify either predictors of success or risk factors for failure during residency. Trainability/suitability, in particular, may be a subject of further interest in identifying applicants at risk for changing programs. Future studies should investigate additional noncognitive traits of our applicants to help better predict future residency success. f
American Journal of Obstetrics & Gynecology MAY 2011
REFERENCES 1. National Residency Matching Program results and data: 2009 main residency match. National Residency Matching Program, 2009 (vol 2009). 2. Metro DG, Talarico JF, Patel RM, Wetmore AL.The resident application process and its correlation to future performance as a resident. Anesth Analg 2005;100:502-5. 3. Bell JG, Kanellitsas I, Shaffer L. Selection of obstetrics and gynecology residents on the basis of medical school performance. Am J Obstet Gynecol 2002;186:1091-4. 4. Brothers TE, Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ 2007;64:378-85. 5. Dubovsky SL, Gendel M, Dubovsky AN, Rosse J, Levin R, House R. Do data obtained from admissions interviews and resident evaluations predict later personal and practice problems? Acad Psychiatry 2005;29: 443-7. 6. McAlister RP, Andriole DA, Brotherton SE, Jeffe DB. Attrition in residents entering US obstetrics and gynecology residencies: analysis of national GME census data. Am J Obstet Gynecol 2008;199:574.e1-6. 7. Janz T. Initial comparisons of patterned behavior-based interviews versus unstructured interviews. J Appl Psychol 1982;67:577-80. 8. Byers M. Interview Rx: a powerful guide for making your next interview a success, 3rd ed. Conyers, GA: Nearline Publishers; 2007. 9. Erlandson EE, Calhoun JG, Barrack FM, et al. Resident selection: applicant selection criteria compared with performance. Surgery 1982;92: 270-5. 10. Hojat M, Gonnella JS, Veloski JJ, Erdmann JB. Is the glass half full or half empty? A reexamination of the associations between assessment measures during medical school and clinical competence after graduation. Acad Med 1993;68:S69-76. 11. Lyon D, Wiper D. Finding our colleagues, finding ourselves: behavioral interviewing and a critical assessment of how we identify and interview residency candidates. APGO/CREOG Annual Meeting. San Diego, CA, 2009. 12. Olawaiye A, Yeh J, Withiam-Leitch M. Resident selection process and prediction of clinical performance in an obstetrics and gynecology program. Teach Learn Med 2006; 18:310-5.
www.AJOG.org
Education
Research
SUPPLEMENTARY FIGURE 1
Factors measured in St. Vincent’s behavioral interviews
Strand. Predicting resident success. Am J Obstet Gynecol 2011.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e6
Research
Education
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets
Continued on next page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
446.e7
American Journal of Obstetrics & Gynecology MAY 2011
www.AJOG.org
Education
www.AJOG.org
Research
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets Professionalism 1. Interview Day/Appearance
1
Unkempt; rude; late
3
Well-groomed; slow or difficult to engage
5
7
9
Appropriately dressed, timely, polite; interacts well but must be led by interviewers
Pleasant, courteous, wellgroomed; enough selfconfidence to have some poise in interaction
Pleasant, courteous, wellgroomed; selfassured, wellspoken; completely comfortable with the interview
2. Ethics
1
3
5
7
9
Cannot identify ethical challenge or situation
Can identify ethical situation or challenge, but has no insight into response
Shows basic understanding of complexities within ethical dilemmas
Demonstrates understanding of ethical principles (informed consent, beneficence, etc.)
Provides engaging discussion of difficult ethical situations
3. Stress
1
3
5
7
9
Denies stressful situations or being effected by stress
Aware of stressful situations but not of personal effects
Describes stressful situation and some evidence of personal approach
Has clearly defined ways of personally handling stress
Describes stressful situation, well-adjusted response, and individual growth from experience
1
3
5
7
9
Evidence of initiative leading to personal pride
Describes several situations requiring initiative as well as how this contributed to personal growth
4. Initiative
Offers no evidence of personal initiative
Limited evidence of initiative
Repeated evidence of initiative in professional or personal life
Total Score:________________
Continued from the previous page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e8
Research
Education
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets
Continued from the previous page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
446.e9
American Journal of Obstetrics & Gynecology MAY 2011
www.AJOG.org
Education
www.AJOG.org
Research
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets
Continued from the previous page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e10
Research
Education
SUPPLEMENTARY FIGURE 2
Evaluation scoring sheets
Continued from the previous page. Strand. Predicting resident success. Am J Obstet Gynecol 2011.
446.e11
American Journal of Obstetrics & Gynecology MAY 2011
www.AJOG.org
Education
www.AJOG.org
Research
SUPPLEMENTARY FIGURE 3
Sample interview questions
Strand. Predicting resident success. Am J Obstet Gynecol 2011.
MAY 2011 American Journal of Obstetrics & Gynecology
446.e12
Research
www. AJOG.org
EDUCATION
Can a structured, behavior-based interview predict future resident success? Eric A. Strand, MD; Elizabeth Moore, PhD; Douglas W. Laube, MD, MEd OBJECTIVE: To determine whether a structured, behavior-based appli-
cant interview predicts future success in an obstetrics and gynecology residency program. STUDY DESIGN: Using a modified pre-post study design, we compared behavior-based interview scores of our residency applicants to a postmatch evaluation completed by the applicant’s current residency program director. Applicants were evaluated on the following areas: academic record, professionalism, leadership, trainability/suitability for the specialty, and fit for the program. RESULTS: Information was obtained for 45 (63%) applicants. The over-
all interview score did not correlate with overall resident performance.
Applicant leadership subscore was predictive of leadership performance as a resident (P ⫽ .042). Academic record was associated with patient care performance as a resident (P ⫽ .014), but only for graduates of US medical schools. Five residents changed programs; these residents had significantly lower scores for trainability/suitability for the specialty (P ⫽ .020). CONCLUSION: Behavioral interviewing can provide predictive informa-
tion regarding success in an obstetrics and gynecology training program. Key words: internship/residency, personnel selection, professional competence
Cite this article as: Strand EA, Moore E, Laube DW. Can a structured, behavior-based interview predict future resident success? Am J Obstet Gynecol 2011;204:446.e1-13.
E
ach academic year, residency programs and applicants expend significant time, energy, and financial resources interviewing applicants for future residency positions. In the 2009 National Residency Matching Program (NRMP) main From the Department of Obstetrics and Gynecology (Drs Strand and Moore), St. Vincent Hospital, Indianapolis, IN, and the Department of Obstetrics and Gynecology (Dr Laube), University of Wisconsin, Madison, WI. Presented at the annual joint meeting of the Association of Professors of Gynecology and Obstetrics and the Council on Resident Education in Obstetrics and Gynecology, Orlando, FL, March 3-6, 2010. Received Aug. 26, 2010; revised Nov. 8, 2010; accepted Feb. 4, 2011. Reprints: Eric A. Strand, MD, Department of Obstetrics and Gynecology, St. Vincent Hospital, 8111 Township Line Rd, Indianapolis, IN 46260. [email protected]. 0002-9378/free © 2011 Mosby, Inc. All rights reserved. doi: 10.1016/j.ajog.2011.02.019
For Editors’ Commentary, see Table of Contents See related editorial, page 369
446.e1
residency match, a total of 51,882 applicants pursued the 25,185 training positions available in the United States. This included 1796 applicants for the 1185 residency positions in the specialty of obstetrics and gynecology.1 Despite these investments, little is known about the use of the residency interview in predicting an applicant’s future performance as a resident. For instance, Metro et al2 reviewed interview scores for their applicants to determine whether the scores correlated with any measures used to evaluate the residents during their training. Interview scores did not correlate with any of the measures, including knowledge, judgment, motor skills, or intrapersonal attitudes.2 Performance on the United States Medical Licensing Examination (USMLE) has been shown to be positively correlated with intraining service examinations, but not with faculty evaluation of resident performance.3,4 Other studies have shown interview scores did not predict physicians at risk of later impairment.5 Furthermore, resident attrition is a significant problem for many programs. After a cohort of 1055 residents started their obstetrics and gynecology residency training in 2001, McAlis-
American Journal of Obstetrics & Gynecology MAY 2011
ter et al6 found that 21.6% of residents either: switched programs, changed specialties, completed training on an atypical academic cycle, or left graduate medical education all together. In 2006, the St. Vincent Hospital Department of Obstetrics/Gynecoloy embarked on a program of behavior-based interviewing for all applicants to its residency program, with the hope of identifying applicants who would be successful both in the chosen specialty and in the St. Vincent program. This study reports the results of this interview process to predict future resident performance.
M ATERIALS AND M ETHODS The St. Vincent Hospital Obstetrics/Gynecology residency is a communitybased program consisting of 4 residents per academic year (approved for expansion to 5 positions in June 2007). The hospital is 1 of 4 main teaching sites for the required third year obstetrics/gynecology clerkship for medical students from the Indiana University School of Medicine. Approximately 200-250 students apply for the available residency positions, with the majority of the US graduates matriculating from programs in the Midwest region.