Evaluation of the pediatric surgical matching program by the directors of pediatric surgical training programs

Evaluation of the Pediatric Surgical Matching by the Directors of Pediatric Surgical Training

Program Programs

By Maureen A. Hirthler, Philip L. Glick, James M. Hassett, Jr, and Donald R. Cooney Buffalo,

New

York and Columbus, Ohio

0 In a previous survey of successful candidates (SC) and unsuccessful candidates (USC) to the pediatric surgery matching program (PSMP), significant discriminating factors associated with SC included previous participation in basic science research, number of publications, number of national presentations, and contact with and letters of recommendation from well-known pediatric surgeons. The purpose of this study was to determine the selection criteria used by directors of the pediatric surgery training programs (PSTP) in choosing residents for the PSMP. A two-part survey of the PSMP was completed by directors of the PSTP (n = 24) to identify the factors (n = 31) that most influence their selection process when choosing a resident, and to help clarify questions (n = 11) posed by the candidates during the first survey. In part I, each factor was scored on a modified Likert Scale, from most (I) to least (5) important in evaluating candidates for training positions, and were ranked according to the mean score. Based on our previous survey, but blinded to the PSTP director respondents, these same factors were divided into groups that, in our first survey, did and did not discriminate between SC and USC for the PSMP, and will be referred to as “discriminating factors” (DF, n = 7) and “nondiscriminating factors” (NDF, n = 8). respectively. Also, a series of questions dealt with subjective factors of the candidates, not previously analyzed in our original survey, and will be referred to as “program directors’ factors” (PDF, n = 18). The three groups (DF. NDF, PDF) were analyzed by the Kruskal-Wallis test and analysis of variance (ANOVA). The five most important factors to the PSTP directors were (1) works well with people (1.24). (2) personal interview (1.33), (3) telephone conversations with other pediatric surgeons (1.4), (4) interviews with other faculty (1.48). and (5) personality (1.5). PDF (mean score, 2.13) were considered most important, followed by DF (2.88) and NDF (3.18). These groups are significantly different by ANOVA (F = 4.18,P = .02) and the Kruskal-Wallis test (H = 6.516, P = 84). Most directors supported mandatory participation in the matching process and were opposed to centralized interviews. Almost all believed that the current system provides accurate infor-

From the Departments of Surgery and Pediattics, Section of Pediatric Surgery, The School of Medicine and Biomedical Sciences, The University at Buffalo, State Universiiy of New York, The Children’s Hospital of Buffalo, Buffalo, NY, and the Departments of Pediatric Surgery and General Surgery, The Ohio State University and The Children S Hospital of Columbus, Columbus, OH. Date accepted: August 9, 1993. Presented at the 13th Annual Meeting of The Association for Surgical Education, Indianapolis, Indiana, April 14-I 7, 1993. Address reprint requests to Philip L. Glick, MD, Depattment of Pediatric Surgery, The Children’s Hospital of Buffalo, 219 Bryant St, Buffalo, NY 14222. Copyright o 1994 by W:B. Saunders Company 0022-3468194/2910-0019$03.00/0

1370

mation about the candidates, but that qualified candidates often went unmatched. Sixty percent favored an increase in the number of training programs. The PSMP now has been analyzed from both the candidates’ and PSTP directors’ points of view. The results of this survey of the PSMP by the PSTP directors confirm our previous survey’s results with respect to objective factors that distinguish between SC and USC in the PSMP. However, subjective PDF were considered the most important factors in evaluating a candidate; this emphasizes the personal intangible characteristics of the candidates and assumes stellar academic and technical abilities. Copyright o 1994 by W.B. Saunders Company INDEX WORDS: Pediatric surgery matching program; pediatric surgery training program.

T

HE PEDIATRIC surgery matching program (PSMP), from which residents are selected for pediatric surgery training positions, has been the subject of much debate. To objectively evaluate factors contributing to a positive outcome in the matching process, we previously completed and analyzed a survey of all successful and unsuccessful candidates for the PSMP throughout a 7-year period.’ This survey showed that a general surgical training program that provided a candidate with certain requisite opportunities (ie, to participate in basic science research, to write peer-reviewed publications, to make national presentations, and to have contact with and letters of recommendation from well-known pediatric surgeons) were the key factors in a successful result in the PSMP. However, there is no information available concerning the selection process of this highly competitive pool of applicants from the perspective of pediatric surgery training program (PSTP) directors. Because of continuing interest in and discussion about this topic, we submitted a second survey directly to the PSTP directors. The results are presented herein. MATERIALS AND METHODS A two-part survey was designed for rapid anonymous completion by the PSTP directors who attended the 22nd Annual APSA Meeting (n = 24). Part I requested that each attribute (n = 31) of a potential candidate be scored on a modified Likert Scale, from most (1) to least (5) important in choosing a resident for their particular program. Based on our previous survey, but blinded to the program director respondents, these attributes were divided into groups, that did and did not discriminate between successful and unsuccessful candidates for the PSMP, and will be referred to

Journal

of PediatricSurgery, Vol 29,No

10 (October),

1994: pp 1370-I374

1371

PEDIATRIC SURGERY MATCHING PROGRAM SURVEY

“discriminating factors” (n = 7) and “nondiscriminating factors” (n = 8), respectively. Also, a series of questions dealt with subjective attributes of the candidates, not previously analyzed in our original survey, and will be referred to as “program director factors” (n = 16). The attributes were ranked according to the mean score. The three groups were analyzed by the Kruskal-Wallis test and analysis of variance (ANOVA); a P value of less than .05 was considered significant (Table 1). The second part of the survey consisted of yes/no questions (n = 12) about the matching process itself (Table 2). This part was prompted by comments and questions from the candidates who participated in our initial survey.1~2 as

Table 1. Training Program Directors’ Survey, Part

I

InevaluatingBcandidatefor trainingin PediatricSurgery,how importantdo you find the followingitems? ModifiedLikertScale 1 2 3 4 5

1.

Quality of

Important

pate in the match every year? (Y/N) 33. Have you ever favored an “inside” candidate? (Y/N) 34. Do you offer an interview to all candidates? (Y/N) 35. Do you believe that the current system allows qualified candidates to go unmatched? (Y/N) 36. Do you communicete directly with the candidates prior to submission of your ranking list? (Y/N) 37. Do you provide candidates with realistic information about their chance of matching at your program? (Y/N) 38. Do you believe a visit to your hospital is essential for each can39. Would you support a centralized interviewing process? (Y/N)

Twenty-one of 24 surveys were completed and returned. The ranking (by mean score) of all data from part I of the survey is presented in Fig 1. The five factors considered most important by the PSTP directors were (1) perceived ability of the candidate to work well with staff (mean rank, 1.24), (2) personal

VW Important

32. Do you believe that all programs should be required to partici-

didate? (Y/N)

RESULTS

Most Important

Table 2. Training Program Directors’ Survey, Part II

Relatively Unimportant

Least Important

undergraduate education

2. Quality of medical school 3. Duality of surgical residency 4. Membership in honor societies, eg, AOA 5. Inservice examination scores 6. Additional clinical training during/after residency (ie, fellowship) 7. Quantity of previous pediatric surgical experience 8. Quantity of previous pediatric surgical experience 9. Performance of basic science research 10. Performance of clinical research 11. Quality of research 12. Number of publications 13. Quality of publications 14. Presentation of research at APSA 15. Presentation of research at the Surgical Section of the AAP 16. Presentation of research at the ACS Surgical Forum 17. Letters of recommendation from general surgical program

40. Do you favor an increase in the number of training positions/ programs? (Y/N) 41. How many weeks does the average candidate need to interview for the match?

(< 4,

> 4)

42. How many thousands of dollars does the average candidate spend to interview for the match?

(< $4K,

> S4K)

interview with the candidate (1.33) (3) telephone conversations with well-known pediatric surgeons (1.4), (4) evaluation by other faculty members (1.48), and (5) personality (1.5). The factors considered least important included occupation of spouse (4.6) marital status (4.3) and quantity of previous pediatric surgical experience (3.57). In part I, the “program directors’ factors” (mean score, 2.13) were considered most important, followed by “discriminating factors” (2.66) and “nondiscriminating factors” (3.16). These groups are significantly different by ANOVA (F = 4.16, P = 0.02) and the Kruskal-Wallis test (H = 6.516, P = .04) (Fig 2). The results from part II are noted in Table 3. PSTP directors agreed unanimously that a visit to their institution was essential. Twenty felt that some qualified applicants went unmatched, and 19 believed that the current process provided accurate information about candidates. The directors were divided on all other issues, with the majority opinion representing approximately two thirds of the respondents.

director 18. Letters of recommendation from other pediatric surgeons 19. Telephone conversations with pediatric surgeons well known to You 20. Your personal interview with the candidate 21. Evaluation of candidate by other faculty members 22. Evaluation of candidate by chief resident/fellow 23. Technical surgical ability of the candidate 24. Perceived ability of candidate to handle clinical work load 25. Perceived ability of candidate to work well with staff 26. Overall personality of candidate 27. Possibility of candidate becoming a full-time faculty member 28. Marital status of candidate 29. Occupation of candidate’s spouse 30. Candidate’s interest in academic surgery 31. Perceived leadership abilities of candidate

DISCUSSION

A comparison of the results of both surveys showed interesting similarities and differences. Discriminating factors were more important to the PSTP directors than were nondiscriminating factors, which validates the results of our previous survey.’ However, the program directors’ factors were considered most important, which emphasizes the subjective and intangible characteristics of the candidate and assumes stellar requisite academic and technical abilities. As Dr Jay L. Grosfeld recently expressed, “Choosing people for a pediatric surgical position is a lot different from choosing students completing medical

1372

HIRTHLER ET AL

1 Fig 1.

2

3

4

5

Pediatric surgical matching program.

school and interviewing for a [surgical] internship. In the latter instance, individuals rotate on a service for 2 to 3 months at the most; however, the pediatric. surgical resident basically lives with the faculty for 2 consecutive years (almost like a wedding contract). While basic science research, publications, and the number of national presentations are impressive on a bibliography, they may not actually indicate a person will be a good pediatric surgery resident. There are obviously a number of other factors that are more important.*” The candidates’ evaluations of programs were based primarily on their interview with the director,l and training program directors also placed notable emphasis on their personal meeting with the candidate. However, the training program directors also believed that input from other faculty involved in the training program and their current pediatric surgery residents was very important. The applicants believed that the directors placed too much emphasis on the amount of research and publications, at the expense of more subjective qualifications.3 The results of the PSMP survey do not support this belief. Only moderate significance was placed on the quantity of research and publications, as long as the quality was high. Other academic achievements were also considered only moderately important. Rather, the personality of the applicant and the perception of his/her ability to work well with

the staff were of primary importance to the director. To make that assessment, directors indicated they rely heavily on telephone conversations with pediatric surgeons with whom they were well acquainted. Highly influential in this regard was the reputation of the candidate’s general surgical residency and the letter of recommendation from the residency director. Training program directors were satisfied with this system for obtaining information about each candidate. However, they recognized that some qualified applicants went unmatched, and 13 would support an increase in the number of positions or programs, a view strongly favored by the candidates.3 Since this survey was taken, four new training positions (Buffalo Xl, Chicago Xl, and Denver X2) have been certified by the Accreditation Council for Graduate Medical Education and the Residency Review Committee, which will begin to rectify this problem. Although it appears that the directors attempted to evaluate participants objectively, there is no doubt that candidates applying from a well-established surgical residency and supported by a prominent pediatric surgeon are in a position likely to result in success. It is much more difficult for other candidates, despite their academic successes, to attain a residency position, primarily because of the training directors’ difficulty in adequately evaluating the subjective factors that they consider most important.

PEDIATRIC

SURGERY

MATCHING

PROGRAM

1373

SURVEY

PROGRAM DIRECTORS’ FACTORS

OlSCRlHlNATlNG I FACTORS

I

1.24

Works well with others

I

Surgical Residency

1.33

r

1.66

I

I I

Interview

NON-DISCRIMINATING I FACTORS

I

2.51

Basic Science Research

I

I

Program Director’s Recommendation

214

2.85

Telephone Conversrllon with Pedlalrlc Surgeons

1.40

Clinical Research

260

Additional

lmpresslonsof Faculty

1.48

Presentation a1 Surgical Forum

3.05

Honors

3.20

Personality

1.50

Number of Pubtlcatlons

3.14

Quality of Pedialric Surglcel Experience

3.20

Clinical Ability

1.62

Presentation al APSA

39g

Quantity of Pedlatrlc Surgical Experience

3.57

Leadershlp

1.76

Presentation at AAP

3.33

Meriial Status

4.30

Spouse’s

460

Chief Resident’s Technical

Impression

1.86 2.14

Abililv

Tralnlng

Occupation

f

t

Interest in AcedemicSurgery

2.80

ANTE score

2.85

Faculty appointment

3.20

Undergraduate Education

3.20

_ Fig 2. Rankings for program directors’ factors, discriminating factors, and nondiscriminating factors.

1MEAN

2.09*

*(ANOVA, F=5.83, P=O.OOS)

Candidates also questioned the “fairness” of the process.1” Seven directors admitted to having favored an “inside” candidate, and six would like the ability to opt out of the match in a given year. Only seven provided realistic information to the candidate regard-

Table 3. Results of Survey, Part II NO.

Item

32.

Mandatory

33.

“Inside”

Yes (%I

participation

15 (71)

candidate

No (%)

6 (2%

7 (33)

14 (67)

7 (33) 20 (95)

14 (67)

34.

Interview

all applicants

35.

Qualified

candidates

36.

Direct communication

37.

Realistic

information

38. 39.

Hospital

visit essential

Centralized

40.

Increase

41.

Greater

than 4 wk to interview

14 (67)

6 (29)

42.

Greater

than $4,000

15 (75)

5 (25)

going

unmatched

interview programs/positions

1 (5) 7 (33) 21 (100) 6 (29) 13 (62)

1 (5) 20 (95) 14 (67) D (0) 15 (71) 6 (36)

I

2.80 *

I

*(KRUSKALWALLIS,H=8.234,P=O.O18)

I

3.14 ’

ing his/her chance of matching at their program. Twenty directors claimed no direct communication with applicants before submitting their ranking list. In an attempt to limit unnecessary expense, 14 programs prescreen applicants for interviews. It appears that the directors try to make the matching process as fair as possible, but it also appears that occasionally this is not the case. In sharp contrast to the opinions of candidates,‘s program directors were strongly opposed to a centralized interviewing process, with six directors thinking that fewer than 4 weeks was needed to complete the interview process, and five believing that candidates spent less than $4,000 on travel and other expenses. All believed that a visit to their hospital was essential. It can be concluded that the current system works well for the program directors, and that one third are unaware of the excessive time and cost to the applicants.

1374

HIRTHLER ET AL

The results of these two surveys should stimulate discussion among PSTP directors and members of the pediatric surgery and general surgery communities to improve the PSMP. Reevaluation of the PSMP will continue to ensure that the quality of our applicants

remains high and the future of pediatric surgery is optimistic. In addition, other surgical specialties’ resident selection processes (ie, vascular surgery, cardiac surgery, plastic surgery) may also benefit from the results and discussions of these surveys.

REFERENCES 1. Hirthler MA, Glick PL, Hassett Jr JM, et al: Comparative analysis of successful and unsuccessful candidates for the pediatric surgical matching program. J Pediatr Surg 27:142-148,1992 2. Grosfeld JL: Personal communication to Philip L. Glick, January 25,1993

3. Hirthler MA, Glick PL, Allen JE, et al: Candidates’ comments on the pediatric surgical matching program. J Pediatr Surg 27:413,1992 (letter)