Residents' evaluation of a problem-based learning curriculum in a general surgery residency program

Residents’ Evaluation Learning Curriculum Residency Richard W. Schwartz,

MD, Michael 6. Donnelly, PhD, David A. Sloan, MD, William E. Strodel, MD, Lexington,

METHODS: To determine residents’ satisfaction with problem-based learning and its tutors, to determine how residents prepare for sessions, and to identify the characteristics of both effective sessions and tutors, we analyzed 132 evaluations from 24 residents who completed 1 or more of the 9 cases presented during the first 9 months of our program. The 38-item evaluation questionnaire asked residents to rate tutor characteristics, various aspects of the sessions, and methods used to prepare for the sessions. RESULTS: Residents were well satisfied with the problem-based learning sessions and with the logistics of our program; they found the overall quality of the tutor more important than that of the case; they valued an active, thought-provoking tutor more than a traditional facilitator; and they most often used standard textbooks to prepare for the sessions. CONCLUSIONS: Problem-based learning is a practical, enjoyable graduate curricular vehicle when implemented with well-written cases and active tutors. Am J Surg. 1997;173:338-341. 0 1997 by Excerpta Medica, Inc.

roblem-based learning has gained increasing popularity in undergraduate medical education.‘,’ However, P to the authors’ knowledge, it has not heretofore been used systematically in graduate medical education. After almost 5 years of experience with problem-based learning as the curricular format for a third-year surgery clerkship,’ the Division of General Surgery at the University of Kentucky Chandler Medical Center has initiated a systematic, problem-based learning curriculum for residency education. Because the clinical knowledge of the resident is greater than that of the medical student, this curricular model, in theory, should work better for graduate education than for undergraduate. Its advantages include the systematic coverage of both basic science and clinical medicine with associated ethical, legal, and socioeconomic issues; constant From the Department of Surgery, University of Kentucky College of Medicine, Lexington, Kentucky. Requests for reprints should be addressed to Richard W. Schwartz, MD, Division of Surgery Education, Department of Surgery, University of Kentucky Chandler Medical Center, 800 Rose Street, Lexington, Kentucky 40536-0084. Manuscript submitted August 13, 1996 and accepted in revised form September 26, 1996.

338

0 1997 by Excerpta All rights reserved.

Medica,

of a Problem*based in a General Surgery Program

Inc.

Kentucky

curriculum updates through ongoing case revision; consistent small-group interaction between faculty and residents; the encouragement of ongoing perusal of the pertinent medical literature and associated informatics; greater faculty enjoyment; and, most important, an emphasis on self-directed, life-long learning principles for graduate surgery education. In this initial study of problem-based learning as a systematic curricular format for general surgery residents, we found that both faculty and residents enjoyed problem-based learning and that residents preferred it to several other methods of instruction. Furthermore, our belief that group leaders should function as true tutors4 (that is, maintain an active, constant role of challenging the group) and not as traditional problem-based learning facilitators’ was strongly reinforced. The purposes of this study were to determine residents’ satisfaction with both problem-based learning and its faculty tutors, to determine residents’ satisfaction with specific aspects of the problem-based learning sessions, to determine how residents prepared for problem-based learning sessions, and to identify the characteristics of effective problem-based learning sessions and of effective problem-based learning tutors.

METHODS All categorical general surgery residents (5 at each level) and all general surgery faculty members participated in the problem-based learning curriculum. The curriculum was designed around problems in four domains based on the sections of the Division of General Surgery (Gastrointestinal/ Endocrine; Oncology; Trauma/Critical Care; Vascular). The problems were written by appropriate members of the faculty and were chosen to constitute a formal curriculum of both the basic science and clinical objectives necessary for the broad education of a general surgeon. In addition, pertinent ethical, socioeconotnic, legal, and management issues were integrated into the cases. Four study groups were formed, consisting of postgraduate year-l and -2 residents, postgraduate year-3 and research residents, senior residents, and chief residents. The curriculum was designed so that each case takes a month to complete and each domain consists of roughly 10 cases. Each group will cover approximately 3 cases from each domain per year; therefore, the curriculum takes 3 years to complete. Theoretically, each resident will then have had the opportunity to study each case both as a junior (first 3 years) and as a senior (latter 3 years) resident, thus addressinn increasingly more sophisticated case issues. 0002-961 O/97/$1 PII SOOO2-9610(96)00390-X

7.00

1 PROBLEM-BASED

During the study period (9 months), the groups met simultaneously each Saturday morning for 75- to 90.minute sessions; each problem-based learning case was covered in four Saturday sessions. Other demands on residents’ time, such as off-site rotations, clinical duties, and vacations, caused attendance at each session to vary, but 4 or 5 members of each group were usually present. Faculty members tutored the cases germane to their area of expertise; because each section of General Surgery has three or four faculty members, each tutored 3 or 4 months per year. Faculty members were expected to guide the residents through the objectives of each case and to demand preparation for each session; residents were expected to identify learning issues appropriate to their level of training and to prepare for each session by consulting textbooks, journal articles, or other expert faculty members. The more senior the resident, the greater the expectation both for reference to pertinent literature and for its proper interpretation. Each group covered 9 problem-based learning cases. Fourteen faculty members served as expert tutors for 1 or more cases. At the end of each problem-based learning case, residents were asked to evaluate the faculty tutor and the problembased learning sessions by filling out a 38-item questionnaire. Twenty-four of these items evaluated specific characteristics (such as “asked thought-provoking questions,” “kept the group on track,” “ asked open-ended questions”) that were considered important for effective tutoring. Eight items (such as “time allotted to case,” “time to prepare adequately for sessions,” “ frequency of sessions”) assessed the residents’ evaluation of the various aspects of the problembased learning sessions. Six items determined how residents prepared for the sessions, ie, whether they consulted computerized literature searches, journals, standard surgery texts, specialty texts, faculty members, or other residents. Twenty-four of the 25 residents evaluated one or more of the sessions, completing 132 evaluations of the problembased learning sessions and tutors. Descriptive statistics were used to summarize residents’ judgments about the problem-based learning sessions and tutors. A one-way, repeated-measures analysis of variance (ANOVA) was used to determine the most frequent methods residents used to prepare for problem-based learning sessions. Pearson correlations between the overall evaluation of problem-based learning sessions and tutors and the evaluation of specific characteristics were used to determine the distinctive features of effective problem-based learning sessions and tutors.

RESULTS The residents evaluated both the problem-based learning tutors and the problem-based learning sessions very positively. More than 80% of the evaluations indicated that the residents judged the tutors to be above average or outstanding in their skills in helping the residents think critically about clinical problems. Similarly, nearly 80% of the residents believed that the problem-based learning sessions were above average or outstanding in comparison to standard resident teaching methods such as conferences and rounds. A paired t-test was used to determine whether the residents found the problem-based learning tutors and prohlem-based learning sessions to be of equal overall quality. The t-test indicated that the problem-based learning tutors THE

AMERICAN

LEARNING

CURRICULUM/SCHWARTZ

ET AL 1

were viewed as being of greater educatIona value than were the problem-based learning sessions themselves (mean tutor rating = 4.11; mean problem-based learning rating = 3.94; t = 2.56; df = 121; P = 0.012). A summary of the residents’ evaluation of the time allocated to the problem-based learning cases indicated that more than 80% found the time allocation “just right.” High service demands make time for study an important issue in resident education. More than 60% of the residents thought that there was adequate time to prepare for the problembased learning sessions; however, ;I substantial minority (-30%) thought that the time necesxary to prepare for the sessions was somewhat inadequate. More than 40% of the residents thought that they usually had enough time to use appropriate resources; slightly mare than 45% thought that they sometimes had sufficient time for such use. Almost all the residents thought that the frequency of the sessions was “just right.” More than 70% of the residents thought that the problem-based learning cases were somewhat challenging; another 25% thought that the cases were very challenging. The residents were asked to indicate the methods and resources they used to prepare for the problem-based learning sessions on a i-point scale from “never” (scaled as a 1) to “sometimes” (scaled as a 2) to “usually” (xaled as a 3). A repeated-measures analysis of variance indicated significant differences in the frequency with which various methods and resource5 were used to prepare for the problembased learning sessions (F = 48.31; df = 5,585; P < 0.0001). Standard surgery texts were used mobt frequently. Specialty texts, MEDLINE searches, and journal articles were used somewhat less frequently. Other residents and faculty members were the most infrequently used resources. Table I presents the correlations between the overall evaluation of the tutor and specific tutor skills. Operationally, these correlations were divided into four different levels of importance in discriminating the effective tutors from the less effective ones: 1) high correlations (0.70 to 0.79); 2) moderately high correlations (0.60 to 0.69); 3) moderate correlations (0.50 to 0.59; and 4) 1ow correlations (<0.50). The most discriminating items were concerned with general teaching skills (comfort and enthusiasm a5 a tutor) and the active nature of the effective tutor (for example, encouraged the development of learning issues, asked thought-provoking questions, probed depth of understanding). The moderately high correlations tended to be concerned with the tutor as an effective director of the problem-based learning session (for example, kept the group on track, asked open-ended questions, encouraged use of learning resources). The moderate correlations were almost exclusively concerned with communication between resident and tutor (for example, tutor asked for feedback or gave feedback to residents). Eight items were correlated with the overall resident evaluation of the problem-based learning sessions. As can be seen from Table II, the two most important factors were the overall quality of both the tutor and the problem-based learning case; the other important factor appeared to be how well the group worked together. The correlations shown in Table II, although rather low, are considered quite good in this context. The low correlations, however, have little practical importance because of the amount of variance they explain. JOURNAL

OF SURGERYR

VOLUME

173

APRIL

1997

339

EROBLEM-BASED

LEARNING

CURRICULUM/SCHWARTZ

TABLE I Correlation

Between

ET AL 1

Overall and

Tutor

Evaluation Specific

of the Problem-based Tutor Skills

Skill

Learning

Correlation

How would you evaluate your tutor’s participation in your group? How would you evaluate the case used in these sessions? Asked for resident input, i.e., “How are we doing?” How can we do better?” The tutor asked for feedback from the resident. The tutor encouraged residents to give feedback to each other. Brought all residents into the discussion. Gave feedback to residents. Challenged residents on their statements (right or wrong). Debriefed at point of resolution of problem. Encouraged use of learning resources. Encouraged “give and take”: interchanges among residents. Kept discussion focused on the patient. Introduced a new “wrinkle” to stimulate learning. Asked open-ended questions. Gave residents time to think before answering questions. Kept the group on track. Appeared comfortable with role of facilitator. Asked thought-provoking questions pertinent to the case. Integrated learning issues into discussion as opposed to resident mini-lectures. Probed depth of understanding. Promoted clarification of concepts and terms. Was an enthusiastic participant in the problem-based learning sessions. Encouraged development of learning issues as case progressed.

TABLE II Correlations Problem-Based

Between Overall Evaluation of Problem-based Spxific Problem-based Learning Characteristics Learning

Characteristic

COMMENTS Several years ago the authors proposed a method of learning that would systematically address a defined set of clinical problems in a manner that would reinforce the fact that residents are adult learners.6 Although several authors have either advocated such a change7 or used problem-based learning in short duration for graduate medical education, this report is unique in its study of problem-based learning as a systematic, long-term curricular modality. We believe that the numerous advantages of this educational format, which have been well documented elsewhere,s may even be more appropriate for graduate than for undergraduate medical education. Several findings from this study are important. More than 80% of the residents thought that the faculty members had above-average or outstanding skills in helping the residents THE

AMERICAN

JOURNAL

OF SURGERY@

VOLUME

0.05 0.44’ 0.52* 0.52* 0.54* 0.55* 0.56* 0.57* 0.60* 0.62* 0.63* 0.63* 0.64’ 0.65’ 0.66’ 0.68’ 0.71’ 0.71’ 0.71* 0.71* 0.72* 0.72* 0.78*

Learning

and

Correlation

Sufficient scheduled time to cover problem-based learning Time to prepare adequately for sessions. Sufficient time to use resources. Time allotted to case. Case challenging. Did your groups work well together? How would you evaluate the case used in these sessions? Overall evaluation of this tutor in helping you think critically about clinical problems.

340

Tutor

173

case.

0.04 0.06 0.08 0.27’ 0.42* 0.50 0.53* 0.54*

think critically about clinical problems. The residents found the tutors of greater educational value than the problembased learning sessions themselves. More than 80% of the residents thought that the time allocation and frequency of the sessions were “just right.” Almost all residents rated the problem-based learning cases as somewhat or very challenging. Sixty percent of the residents thought there was adequate time for study (that is, session preparation). The most frequently used resource for preparation was standard surgery textbooks; the least frequently used resource was faculty members. General teaching skills, such as enthusiasm and an active tutor role (constantly probing and querying) were correlated more highly with the residents’ overall rating of the tutors than were skills in facilitating the problem-based learning session (keeping the group on track). The two most important factors relating to the resAPRIL

1997

PROBLEM-BASED

idents’ overall evaluation of the problem-based learning sessions were the quality of the tutor and that of the problembased learning case, and the next most important factor was how well the group worked together. Clearly, our residents believed that the overall quality of the tutor was the most important feature of their learning. Furthermore, as opposed to traditional dogma regarding problem-based learning,5 the most highly rated quality was clearly a constantly active tutor who consistently asked thought-provoking questions, probed the depth of residents’ understanding, and served to stimulate understanding of the important case issues. This study further reinforces our belief that traditional problem-based learning tutoring, which is more concerned with facilitation of group processes, is often passive and nonfocused (that is, results in group “Brownian motion”) and is regarded by many adult learners as unproductive. In concert with poor tutoring, poorly written cases will negate both enthusiasm for and the educational benefits of the program. We strongly advocate that clinical problem-based learning, at both the undergraduate and graduate medical levels, should he based on cases clearly integrated into a curriculum with definite objectives and with tutors who are consistently active thought-provokers who guide the students and residents through such objectives. The logistics of our program (four sessions per case, one session per week, 75 to 90 minutes per session, 4 to 6 residents per group, and Saturday morning meetings) were all well accepted by the residents. Roughly two-thirds of the residents thought that they had adequate time to prepare for the sessions; these logistics appear to be practical given the heavy clinical demands of most surgical residency programs. It was of interest to the authors that standard textbooks were used most frequently as references, and faculty members, least. We encourage the use of current literature (using computer informatics) and expert faculty members as learning resources. Faculty expertise in areas germane to each case should, theoretically, be of great benefit to the

THE

AMERICAN

LEARNING

CURRICULUM/SCHWARTZ

ET AL )

residents. We have hypothesized Art resrdents, especially at higher levels of training, may he unwilling to reveal ignorance about case ISSUESto faculty members. The results of this study have stimulated the authors to address the following issues: What percentage of residents are adequately prepared for each session? What behaviors characterize the prepared resident! What stimulates residents to prepare for sessions? Do ledming resources (such as textbooks, journal articles and tacrrlty expertise) become more sophisticated as residents advance through the program? These issues are currently being studied. In summary, problem-based learning is a practical, enjoyable curricular vehicle for graduate medical education when implemented with well-written cases and active, probing tutors.

REFERENCES 1. Ncxman GR. Schmidt HG. The osvchf txoblembased learning: a reww of the r\-id&r. Acid Med. 1992&;67:557565. 2. Romzick TM, Smith RW. Applying pn+lem-hased learning theory to the clinical clerkship. Acad Med. 1990;65:662. 3. Schwartz RW, Donnelly MB, Young B, vt al. Undergraduate surgical education for rhe twenty-first century. Ann Sur,o. 1992;216: 639-647. 4. Mayo WP, Donnelly MB, Schwartz RX’. Characteristics of the ideal problem-ha& learning tutor in clinic,\1 medicine. Eval He&h Prof. 1995;18:124-136. learning: an +pmach 5. Barrows HS, Tamhlyn RM. P r<)hl em-h.& to medical educati<>n. New York: Springer Puhlishmg Company; 1980. (Jones S, 4. Springer Series on Medical Education, Volume 1). 6. Schwartz RW. L)onnelly MB, Maw WI’. StrodeI WE. Problembased learning: a f~>rmal curriculum ‘&r postgraduate surgical education. Curr Surly. 1993;50:285-290. 7. Tweed WA, Dunen N. The experlmtlal curriculum: an alternate model for anaesthesia educatlan. Can .I Anuesth. 1994;41:12271233. 8. Albanese MA, Mitchell S. Problem-hased learning: a review of literature on its Llutcomes and implrmentatwn issues. Acad Med. 1993;68:52-81.

JOURNAL

OF

SURGERYa

VOLUME

173

APRIL

1997

341