- Email: [email protected]
A structured teaching curriculum for medical students improves their performance on the National Board of Medical Examiners shelf examination in surgery
The American Journal of Surgery (2015) -, -–-
A structured teaching curriculum for medical students improves their performance on the National Board of Medical Examiners shelf examination in surgery Keith Wirth, B.S.a,*, Bethany Malone, B.S.a, Christopher Turner, M.D.b, Robert Schulze, M.D., F.A.C.S.c, Warren Widmann, M.D., F.A.C.S.b, Aliu Sanni, M.D.d a
Department of Surgery, SUNY Downstate College of Medicine, 450 Clarkson Avenue, Brooklyn, NY 11203, USA; bDepartment of Surgery, SUNY Downstate Medical Center, 450 Clarkson Avenue, Brooklyn, NY, USA; cDepartment of Surgery, Boston University, Boston, MA, USA; dDepartment of Surgery, Emory University, Atlanta, GA, USA
KEYWORDS: Medical student; NBME shelf; National Board of Medical Examiners; Curriculum
Abstract BACKGROUND: The aim of this study was to evaluate the effect of a resident-driven, student taught educational curriculum on the medical students’ performance on the National Board of Medical Examiners surgery subject examination (NBME). METHODS: On daily morning rounds, medical students or the chief resident delivered preassigned brief presentations on 1 or 2 of the 30 common surgical topics selected for the curriculum. An initial assessment of student knowledge and an end-rotation in-house examination (multiple choice question examination) were conducted. The mean scores on the NBME examination were compared between students in teams using this teaching curriculum and those without it. RESULTS: A total of 57 third-year medical students participated in the study. The mean score on the in-house postclerkship multiple choice question examination was increased by 23.5% (P , .05). The mean NBME scores were significantly higher in the students who underwent the teaching curriculum when compared with their peers who were not exposed to the teaching curriculum (78 vs 72, P , .05). CONCLUSION: The implementation of a resident-driven structured teaching curriculum improved performance of medical students on the NBME examination. Ó 2015 Elsevier Inc. All rights reserved.
There were no relevant financial relationships or any sources of support in the form of grants, equipment, or drugs. The authors declare no conflicts of interest. Oral presentation by Ms Bethany Malone at the 2013 ACS Clinical Congress * Corresponding author. Tel.: 11-718-270-1421; fax: 11-718-270-2826. E-mail address: [email protected] Manuscript received May 27, 2014; revised manuscript August 27, 2014 0002-9610/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.amjsurg.2014.09.036
2
The American Journal of Surgery, Vol -, No -, - 2015
The settings and goals of the clinical clerkship differ significantly from those in the preclinical basic science curriculum. In addition to the factual knowledge that students are accustomed to learning in the preclinical years, they must now master procedural skills, sharpen their patient interactions, and simply learn how to think as a physician. Appropriately, students spend the majority of their time on the wards, in the clinic, and in the operating room to reach these goals. However, this also means that the time for didactic lectures and conferences is significantly reduced, often to only weekly or biweekly sessions, while the amount of clinical knowledge students must learn remains largely at random by case exposure and individual unsupervised reading. Clerkship directors have taken several approaches to the basic knowledge component of their clinical curricula. Some have adopted strategies from the basic science curricula such as team-based learning, case-based learning (CBL), and problem-based learning (PBL).1–3 Others have added unique features using online podcasts and virtual patient software.4,5 Of note, all these interventions are used outside the clinical arena. Traditionally, faculty teaching rounds served as a significant venue for clinical teaching. This aspect of medical education is becoming less frequent with attendings citing economic factors and time restrictions (work-hours rules) as significant limiting factors for the conduct of the former lengthy teaching rounds.6 In the recent past (1999), students rated independent study and tutorials as the first and second most important learning modalities. Interestingly, students rated resident teaching as third, with ward rounds and attending physician teaching falling at fifth and ninth, respectively.7 Pelletier proposed that residents act ‘‘as an important link between the theory and practice of surgery.’’ Residents spend up to 25% of their clinical time teaching students,8 are perceived as teachers even more so than attendings,9 and greatly influence medical students’ choice of specialty.10 This has been recognized among a number of medical schools and residency programs, and curricula termed ‘‘Residents as Teachers’’ programs have become popular. These programs only provide residents with tools to better their teaching skills and have resulted in improved feedback from students.11 However, the relation between perceived resident teaching quality and NBME scores accounted for only 14 of score variability.12 Pelletier and Belliveau7 pointed out that students rated the importance of residents more highly for instruction in technical and practical aspects of surgery, as opposed to basic teaching in topics such as burn management and nutrition. Residents are certainly teaching, and are being taught how to teach. However, what the residents teach and the effectiveness of this teaching remain highly variable. The goal of our study was to investigate a novel approach to teaching on the wards, recognizing the above concerns and challenges regarding the education of thirdyear medical students on their surgery clerkship. With
decreasing attending involvement in formal teaching rounds and reduced didactic time during clerkships, we sought to provide a link between our program’s didactic lectures, small group scheduled conference, and the day-today clinical exposure of the medical students on morning rounds. We studied whether we could cover the basic curriculum during morning rounds, as an adjunct to the existing didactic sessions. By having a senior/chief resident act as facilitator, we provided mentor interaction with a structure in which both residents and students teach. By having the presentations on morning rounds, group learning was facilitated in a social learning environment with student-generated content.
Patients and Methods Study design This study was conducted during a 12-month period overlapping the 2011 to 2012 and 2012 to 2013 academic years at 3 of the clinical sites of the State University of New York Downstate Medical Center. The study was approved by the Institutional Review Board at SUNY Downstate Medical Center. Each study period was 1 month, corresponding to the medical student rotation schedule. One senior/chief resident (A.S.) piloted the program. The students were assigned to this team randomly, as the scheduling at our institution is done by a lottery system; they will be referred to as the participants. Because of varied caseloads at the sites, the groups ranged in size between 3 and 9 students. Students could be assigned to only one team. A total of 57 students were assigned to the participant group.
Teaching method Thirty core surgical topics were selected by surveying surgical faculty and then supplemented by a review of commonly tested topics on the NBME examination (Table 1). The participants were e-mailed a schedule of topic assignments before the start of their rotation. Twenty of the topics were distributed among the participants, and 10 to the senior/chief resident. The topic distribution was randomized for each rotation for both the participants and the senior/chief resident. The participants were instructed to prepare a 5-minute oral presentation without the use of slides. They were required to e-mail their outline to the senior/chief resident the night before their presentation and were allowed to refer to their outline during the oral presentation. One or 2 presentations were delivered each morning during morning rounds. All the residents and medical students in the team were required to be present. The senior/chief resident would select the timing of these presentations depending on the schedule for that day and the clinical cases on the ward. If a topic was related to specific patient on the ward,
K. Wirth et al. Table 1
Student curriculum improves shelf scores
3
Thirty core surgical topics
Wound healing Perioperative management Surgical infections Fluid and electrolytes Appendicitis Small bowel obstruction Diverticular disease Gallstone disease Pancreatitis Hernias Bleeding/Coagulation cascade Surgical nutrition Breast cancer Peripheral vascular disease Colorectal polyps and cancer
Anal cancer Perianal infections Hemorrhoids Inflammatory bowel disease Colonic obstruction and volvulus Aneurysms Carotid disease Septic shock Ventilator management, acid/base Thyroid carcinoma Pheochromocytoma Portal hypertension Chronic liver disease scores Multiple endocrine neoplasia Disorders of splenic function
Topics presented during each study month.
the presentation would be given before bedside rounding on that patient. If no pertinent clinical cases related to a specific topic were available, the presentations were delivered at the end of rounds. A stopwatch was used during the presentation and students were alerted at 4 and 5 minutes. The senior/chief resident would comment only if needed for clarification or correction. Of note, throughout this intervention, students in all teams attended the traditional departmental didactic lectures.
Assessment tools A preintervention, topic-specific, multiple choice question examination (MCQE) was created using questions from Lawrence, Essentials of General Surgery, Fourth Edition.13 The MCQE consisted of 30 questions, one question on each of the topics to be presented. This MCQE was given on the first day of the rotation. After the 1-month rotation, the same MCQE was repeated. The NBME was given at the end of each surgery ‘‘block’’. Each ‘‘block’’ was 3 months in length. Therefore, the NBME examination and MCQE were not necessarily taken during the same month. As the participants would be receiving specific lectures on each of the 30 topics in the curriculum, nonparticipants from a different team would be at a significant disadvantage on the MCQE. Rather than comparing participants with nonparticipants, we thought comparing the effectiveness of the medical student presentations with those given by the senior/chief resident would be more informative. This would also control for differences in past examination performance and student motivation, as the same participants were both control and intervention subjects. The NBME scores were used to compare participants with nonparticipants. The data were obtained from the Department of Surgery and deidentified. Rotation location and time of year were retained with the data. Matched controls were selected using students who rotated the same month, on a
different team at a different rotation site. If the number of controls available was larger than the participants, controls were selected in order of appearance in the control list.
Questionnaires An optional postintervention survey was distributed along with the final MCQE. The first section of the questionnaire asked students to respond to a set of 6 questions using a Likert scale describing their agreeability with the statements. The second section asked students to rank the usefulness of the learning modalities listed in the results section with number one being the most useful.
Statistical analysis Pre- and postintervention MCQE was scored using 2 separate methods: overall percentage correct, as well as by ‘‘teacher status.’’ Teacher status refers to those topics taught by the medical students versus those topics taught by the senior/chief resident. The mean and standard deviation were calculated. A Student t test for paired data was used to evaluate for statistical significance. The NBME data were analyzed using the same method. Questionnaire data were aggregated and the percentage of each modality ranked as number one in usefulness was calculated.
Results A total of 57 students were participants in the team that engaged in the adjunctive curriculum, all of whom sat for the MCQE pre and post examinations, as well as the NBME examination. Fifty-two of the participants elected to fill out the survey. Fifty-nine percent of the participants had completed 2 or more rotations before the month of this curriculum. Fifty-seven percent of participants had
The American Journal of Surgery, Vol -, No -, - 2015
4 Table 2
Mean NBME scores 6 SD
Participants Controls
Table 4
Raw
Percentile
n
77.5 (68.7)* 69.5 (66.8)*
58.4 (626.4) 33.2 (626.0)
57 57
NBME 5 National Board of Medical Examiners; SD 5 standard deviation. *P , .05.
completed a surgical subspecialty rotation, and 65% had completed an internal medicine clerkship. The participants scored significantly higher in the NBME examination (Table 2). The results of the in-house pre- and postintervention MCQE are summarized in Table 3, with an expected significant increase in scores at the end of the rotation. Table 4 dealing with the in-house MCQE score differential between pre and post rotation shows no statistical difference between the scores on topics presented by the participants versus those presented by the senior/chief resident. The questions on the students’ perceptions of the curriculum are listed in Table 5 and show great enthusiasm for the structured curriculum. Table 6 shows that students rated preparation of presentations highly as a learning experience.
Comments The NBME Surgery Subject Examination is the most often failed clerkship examination.14 The third-year surgery clerkship is frequently medical students’ first exposure to basic surgical topics. Didactic time is significantly reduced during the third year, while teaching rounds have become a less frequent occurrence.6 In assessing the needs for an adjunct to the traditional didactic curriculum, we implemented a unique combination of a more structured environment, which used resident leadership and medical student participation in teaching. The first novel aspect of our study was the structured approach. Presentations were delivered each day during morning rounds. Participants were provided with a schedule of topic presentations and were informed that they would be responsible for understanding all these topics by the end of the rotation. The addition of this curriculum to traditional didactics resulted in significantly improved Table 3
Mean percentage of questions correct on MCQ 6 SD
Teacher status
Pretest
Post-test
P value
Overall Peer taught Resident taught
40.4% (613.3) 41.2% (614.7) 36.0% (620.2)
61.6% (614.4) 64.5% (615.8) 53.8% (620.9)
,.05 ,.05 ,.05
Pre- and post-test percentage correct for topic-specific MCQ examination. MCQ 5 multiple choice question; SD 5 standard deviation.
Mean score differential on MCQE (6SD) Student taught Resident taught P value
Score differential 23.3 (619.3)
17.9 (620.6)
.13
Difference between pre- and post-test performance on topicspecific MCQ examination. MCQE 5 multiple choice question Examination; SD 5 standard deviation.
performance in the NBME examination when compared with nonparticipant controls. The surgery clerkship can present unique challenges to medical students. In comparison with rotations such as internal medicine, the surgery rotation presents medical students with topics not even broached during the preclinical years. Although pathologic conditions are encountered and discussed in the preclinical years, it is mostly on the surgery rotation that students learn the indications for and the basics of operative procedures. By having a structured curriculum, we believe that we assure better exposure to the variety of conditions than might be obtained by the varieties of cases available on morning rounds. We believe that presentations on morning rounds bring a sense of urgency and practical importance as students recognize that precious time is taken even if a specific case is not on the ward at that moment. The second novel aspect of our study was the use of the resident as a teaching leader with the medical students acting as teachers as well. As mentioned in the Introduction, Residents as Teacher programs are becoming more popular. These programs focus on the development of teaching skills, not on specific curricula. The effectiveness of these programs remains unclear, however.15 Our curriculum provides residents with a blueprint for teaching while not requiring an excessive amount of time investment. This was accomplished by having the residents co-ordinate and facilitate with the students delivering most of the presentations. To evaluate if this would be effective as compared with resident teaching alone, we developed the MCQE
Table 5 Perceptions of curriculum effectivenessdpercent Likert score greater than 3 (agree, strongly agree) Statement
Percentage
I would like to see the structure of this teaching on other clerkships Preparation for presentations was a useful exercise Topics on the quiz correlated with topics discussed on rounds The residents took time to teach in the clinical setting My residents were capable teachers I tended to focus my studying based on topics that came up on the pretest
96.2%
Postintervention feedback results.
96.2% 96.2% 98.1% 98.1% 38.0%
K. Wirth et al.
Student curriculum improves shelf scores
Table 6 Rating of learning stylesdpercentage rated #1 (scale of 1–4) Learning style
Percentage
Preparing a topic presentation Practice questions Hearing my peers’ topic presentation Residents asking me a question on the spot
41.2% 32.4% 29.4% 23.5%
Postintervention feedback results.
and as noted above, there was no significant difference in the improvement of scores between those topics taught by the resident versus those taught by the students.
5
Applicability This curriculum was designed with the goal of providing a reproducible teaching framework that could easily be adopted by residents and faculty alike. This study lays out a blueprint for the scheduling and topics of the curriculum, with the only demand on the resident or faculty using this curriculum being 5 to 10 minutes of time before or after rounding. By having such a clear plan and with data demonstrating its efficacy, we hoped this curriculum would appeal to both dedicated teachers and those seeking a simple way to fit teaching into their day.
Limitations and strengths of the study Comparison with previous studies In the clerkship education literature, a number of studies have investigated the effect of various peer-led instructional formats on Objective Structured Clinical Examination and procedural technique learning. Fewer have investigated the effects on NBME scores, however. In the surgical education literature, curricula changes, which replaced traditional didactic lectures with PBL or CBL, reported significant improvements in NBME scores as well as positive feedback from the students.1,16 Similar results have been reported in the pediatric, internal medicine, and psychiatry literature.17,18 Studies on obstetrics and gynecology and psychiatry reported no significant changes in NBME performance, but improved student satisfaction.19,20 Although our curriculum is certainly not structured in the same manner as a PBL or CBL, it shares the common elements of required independent preparation and a social learning environment. Also reflecting on the PBL/CBL data, we hypothesized that the peer-led presentations would lead to improved performance when compared with senior/chief-led presentations. Previous studies have theorized that peer-led tutorials provide improved teaching because of a ‘‘cognitive congruence,’’ a theory also supported in our earlier investigation of resident education.21,22 This is the concept that students closer in teaching distance will have a better appreciation for what knowledge the students have already mastered.21 The results of this study, however, suggested that students and the senior/chief resident were equally effective teachers. We believe that the short timing of the presentations encouraged these 2 separate teacher groups to present effectively. With only 5 minutes of time, the presentations must be succinct and cover the salient features of each topic. Finally, the feedback data from our study correspond with that of previous studies. All the PBL/CBL research above, as well as all the peer-assisted learning studies in a recent review article have reported positive feedback from students.23 Interestingly, although students rated resident presentations higher than those given by peers, this was not reflected in higher scores on topics taught by the senior/chief resident on the in-house MCQE.
The limitations of this study include the small sample size and lack of control for previous examination performance (United States Medical Licensing Examination [USMLE], previous NBME examinations.) Also, only one senior/chief resident participated in this pilot study. A strength was our randomization process. By using the institution’s scheduling system, a lottery, we were able to control for selection bias. Evaluating performance with 2 different examination outcome measurements, one of which was a standardized national assessment, is also a strength.
Conclusions This study provides evidence that providing more structure to the third-year surgery clerkship through the implementation of a standardized adjunctive curriculum covering core surgical topics taught by students and residents can significantly improve medical student examination scores. Using residents as facilitators and both the resident and medical students as teachers is both feasible and effective and should be encouraged in the general surgery clerkship.
References 1. Cendan JC, Silver M, Ben-David K. Changing the student clerkship from traditional lectures to small group case-based sessions benefits the student and the faculty. J Surg Educ 2011;68:117–20. 2. Adamas-Rappaport WJ, Waer AL, Teeple MK, et al. A comparison of unguided vs guided case-based instruction on the surgery clerkship. J Surg Educ 2013;70:821–5. 3. McLean SF, Horn K, Tyroch AH. Case based review questions, review sessions, and call schedule type enhance knowledge gains in a surgical clerkship. J Surg Educ 2013;70:68–75. 4. White JS, Sharma N, Boora P. Surgery 101: evaluating the use of podcasting in a general surgery clerkship. Med Teach 2011;33:941–3. 5. Gormley GJ, McGlade K, Thomson C, et al. A virtual surgery in general practice: evaluation of a novel undergraduate virtual patient learning package. Med Teach 2011;33:e522–7. 6. Gonzalo JD, Heist BS, Duffy BL, et al. Identifying and overcoming the barriers to bedside rounds: a multicenter qualitative study. Acad Med 2014;89:326–34.
6
The American Journal of Surgery, Vol -, No -, - 2015
7. Pelletier M, Belliveau P. Role of surgical residents in undergraduate surgical education. Can J Surg 1999;42:451–6. 8. Sheets KJ, Hankin FM, Schwenk TL. Preparing surgery house officers for their teaching role. Am J Surg 1991;161:443–9. 9. Whittaker Jr LD, Estes NC, Ash J, et al. The value of resident teaching to improve student perceptions of surgery clerkships and surgical career choices. Am J Surg 2006;191:320–4. 10. Musunuru S, Lewis B, Rikkers LF, et al. Effective surgical residents strongly influence medical students to pursue surgical careers. J Am Coll Surg 2007;204:164–7. 11. Ostapchuk M, Patel PD, Miller KH, et al. Improving residents’ teaching skills: a program evaluation of residents as teachers course. Med Teach 2010;32:e49–56. 12. Langenfeld SJ, Helmer SD, Cusick TE, et al. Do strong resident teachers help medical students on objective examinations of knowledge? J Surg Educ 2011;68:350–4. 13. Lawrence PF, Bell RM, Dayton MT. Essentials of General Surgery. Philadelphia, PA: Lippincott Williams & Wilkins; 2006. 14. Lind DS, Deladisma AM, Cue JI, et al. Survey of student education in surgery. J Am Coll Surg 2007;204:969–74; discussion, 975–6. 15. Hill AG, Yu TC, Barrow M, et al. A systematic review of resident-asteacher programmes. Med Educ 2009;43:1129–40.
16. Schwartz RW, Donnelly MB, Nash PP, et al. Problem-based learning: an effective educational method for a surgery clerkship. J Surg Res 1992;53:326–30. 17. Curtis JA, Indyk D, Taylor B. Successful use of problem-based learning in a third-year pediatric clerkship. Ambul Pediatr 2001;1:132–5. 18. Levine RE, O’Boyle M, Haidet P, et al. Transforming a clinical clerkship with team learning. Teach Learn Med 2004;16:270–5. 19. Nalesnik SW, Heaton JO, Olsen CH, et al. Incorporating problembased learning into an obstetrics/gynecology clerkship: impact on student satisfaction and grades. Am J Obstet Gynecol 2004;190: 1375–81. 20. Morreale M, Arfken C, Bridge P, et al. Incorporating active learning into a psychiatry clerkship: does it make a difference? Acad Psychiatry 2012;36:223–5. 21. Wirth K, Malone B, Barrera K, et al. Is there a place for medical students as teachers in the education of junior residents? Am J Surg 2014; 207:271–4. 22. Ten Cate O, Durning S. Dimensions and psychology of peer teaching in medical education. Med Teach 2007;29:546–52. 23. Yu TC, Wilson NC, Singh PP, et al. Medical students-as-teachers: a systematic review of peer-assisted teaching during medical school. Adv Med Educ Pract 2011;2:157–72.
A structured teaching curriculum for medical students improves their performance on the National Board of Medical Examiners shelf examination in surgery Keith Wirth, B.S.a,*, Bethany Malone, B.S.a, Christopher Turner, M.D.b, Robert Schulze, M.D., F.A.C.S.c, Warren Widmann, M.D., F.A.C.S.b, Aliu Sanni, M.D.d a
Department of Surgery, SUNY Downstate College of Medicine, 450 Clarkson Avenue, Brooklyn, NY 11203, USA; bDepartment of Surgery, SUNY Downstate Medical Center, 450 Clarkson Avenue, Brooklyn, NY, USA; cDepartment of Surgery, Boston University, Boston, MA, USA; dDepartment of Surgery, Emory University, Atlanta, GA, USA
KEYWORDS: Medical student; NBME shelf; National Board of Medical Examiners; Curriculum
Abstract BACKGROUND: The aim of this study was to evaluate the effect of a resident-driven, student taught educational curriculum on the medical students’ performance on the National Board of Medical Examiners surgery subject examination (NBME). METHODS: On daily morning rounds, medical students or the chief resident delivered preassigned brief presentations on 1 or 2 of the 30 common surgical topics selected for the curriculum. An initial assessment of student knowledge and an end-rotation in-house examination (multiple choice question examination) were conducted. The mean scores on the NBME examination were compared between students in teams using this teaching curriculum and those without it. RESULTS: A total of 57 third-year medical students participated in the study. The mean score on the in-house postclerkship multiple choice question examination was increased by 23.5% (P , .05). The mean NBME scores were significantly higher in the students who underwent the teaching curriculum when compared with their peers who were not exposed to the teaching curriculum (78 vs 72, P , .05). CONCLUSION: The implementation of a resident-driven structured teaching curriculum improved performance of medical students on the NBME examination. Ó 2015 Elsevier Inc. All rights reserved.
There were no relevant financial relationships or any sources of support in the form of grants, equipment, or drugs. The authors declare no conflicts of interest. Oral presentation by Ms Bethany Malone at the 2013 ACS Clinical Congress * Corresponding author. Tel.: 11-718-270-1421; fax: 11-718-270-2826. E-mail address: [email protected] Manuscript received May 27, 2014; revised manuscript August 27, 2014 0002-9610/$ - see front matter Ó 2015 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.amjsurg.2014.09.036
2
The American Journal of Surgery, Vol -, No -, - 2015
The settings and goals of the clinical clerkship differ significantly from those in the preclinical basic science curriculum. In addition to the factual knowledge that students are accustomed to learning in the preclinical years, they must now master procedural skills, sharpen their patient interactions, and simply learn how to think as a physician. Appropriately, students spend the majority of their time on the wards, in the clinic, and in the operating room to reach these goals. However, this also means that the time for didactic lectures and conferences is significantly reduced, often to only weekly or biweekly sessions, while the amount of clinical knowledge students must learn remains largely at random by case exposure and individual unsupervised reading. Clerkship directors have taken several approaches to the basic knowledge component of their clinical curricula. Some have adopted strategies from the basic science curricula such as team-based learning, case-based learning (CBL), and problem-based learning (PBL).1–3 Others have added unique features using online podcasts and virtual patient software.4,5 Of note, all these interventions are used outside the clinical arena. Traditionally, faculty teaching rounds served as a significant venue for clinical teaching. This aspect of medical education is becoming less frequent with attendings citing economic factors and time restrictions (work-hours rules) as significant limiting factors for the conduct of the former lengthy teaching rounds.6 In the recent past (1999), students rated independent study and tutorials as the first and second most important learning modalities. Interestingly, students rated resident teaching as third, with ward rounds and attending physician teaching falling at fifth and ninth, respectively.7 Pelletier proposed that residents act ‘‘as an important link between the theory and practice of surgery.’’ Residents spend up to 25% of their clinical time teaching students,8 are perceived as teachers even more so than attendings,9 and greatly influence medical students’ choice of specialty.10 This has been recognized among a number of medical schools and residency programs, and curricula termed ‘‘Residents as Teachers’’ programs have become popular. These programs only provide residents with tools to better their teaching skills and have resulted in improved feedback from students.11 However, the relation between perceived resident teaching quality and NBME scores accounted for only 14 of score variability.12 Pelletier and Belliveau7 pointed out that students rated the importance of residents more highly for instruction in technical and practical aspects of surgery, as opposed to basic teaching in topics such as burn management and nutrition. Residents are certainly teaching, and are being taught how to teach. However, what the residents teach and the effectiveness of this teaching remain highly variable. The goal of our study was to investigate a novel approach to teaching on the wards, recognizing the above concerns and challenges regarding the education of thirdyear medical students on their surgery clerkship. With
decreasing attending involvement in formal teaching rounds and reduced didactic time during clerkships, we sought to provide a link between our program’s didactic lectures, small group scheduled conference, and the day-today clinical exposure of the medical students on morning rounds. We studied whether we could cover the basic curriculum during morning rounds, as an adjunct to the existing didactic sessions. By having a senior/chief resident act as facilitator, we provided mentor interaction with a structure in which both residents and students teach. By having the presentations on morning rounds, group learning was facilitated in a social learning environment with student-generated content.
Patients and Methods Study design This study was conducted during a 12-month period overlapping the 2011 to 2012 and 2012 to 2013 academic years at 3 of the clinical sites of the State University of New York Downstate Medical Center. The study was approved by the Institutional Review Board at SUNY Downstate Medical Center. Each study period was 1 month, corresponding to the medical student rotation schedule. One senior/chief resident (A.S.) piloted the program. The students were assigned to this team randomly, as the scheduling at our institution is done by a lottery system; they will be referred to as the participants. Because of varied caseloads at the sites, the groups ranged in size between 3 and 9 students. Students could be assigned to only one team. A total of 57 students were assigned to the participant group.
Teaching method Thirty core surgical topics were selected by surveying surgical faculty and then supplemented by a review of commonly tested topics on the NBME examination (Table 1). The participants were e-mailed a schedule of topic assignments before the start of their rotation. Twenty of the topics were distributed among the participants, and 10 to the senior/chief resident. The topic distribution was randomized for each rotation for both the participants and the senior/chief resident. The participants were instructed to prepare a 5-minute oral presentation without the use of slides. They were required to e-mail their outline to the senior/chief resident the night before their presentation and were allowed to refer to their outline during the oral presentation. One or 2 presentations were delivered each morning during morning rounds. All the residents and medical students in the team were required to be present. The senior/chief resident would select the timing of these presentations depending on the schedule for that day and the clinical cases on the ward. If a topic was related to specific patient on the ward,
K. Wirth et al. Table 1
Student curriculum improves shelf scores
3
Thirty core surgical topics
Wound healing Perioperative management Surgical infections Fluid and electrolytes Appendicitis Small bowel obstruction Diverticular disease Gallstone disease Pancreatitis Hernias Bleeding/Coagulation cascade Surgical nutrition Breast cancer Peripheral vascular disease Colorectal polyps and cancer
Anal cancer Perianal infections Hemorrhoids Inflammatory bowel disease Colonic obstruction and volvulus Aneurysms Carotid disease Septic shock Ventilator management, acid/base Thyroid carcinoma Pheochromocytoma Portal hypertension Chronic liver disease scores Multiple endocrine neoplasia Disorders of splenic function
Topics presented during each study month.
the presentation would be given before bedside rounding on that patient. If no pertinent clinical cases related to a specific topic were available, the presentations were delivered at the end of rounds. A stopwatch was used during the presentation and students were alerted at 4 and 5 minutes. The senior/chief resident would comment only if needed for clarification or correction. Of note, throughout this intervention, students in all teams attended the traditional departmental didactic lectures.
Assessment tools A preintervention, topic-specific, multiple choice question examination (MCQE) was created using questions from Lawrence, Essentials of General Surgery, Fourth Edition.13 The MCQE consisted of 30 questions, one question on each of the topics to be presented. This MCQE was given on the first day of the rotation. After the 1-month rotation, the same MCQE was repeated. The NBME was given at the end of each surgery ‘‘block’’. Each ‘‘block’’ was 3 months in length. Therefore, the NBME examination and MCQE were not necessarily taken during the same month. As the participants would be receiving specific lectures on each of the 30 topics in the curriculum, nonparticipants from a different team would be at a significant disadvantage on the MCQE. Rather than comparing participants with nonparticipants, we thought comparing the effectiveness of the medical student presentations with those given by the senior/chief resident would be more informative. This would also control for differences in past examination performance and student motivation, as the same participants were both control and intervention subjects. The NBME scores were used to compare participants with nonparticipants. The data were obtained from the Department of Surgery and deidentified. Rotation location and time of year were retained with the data. Matched controls were selected using students who rotated the same month, on a
different team at a different rotation site. If the number of controls available was larger than the participants, controls were selected in order of appearance in the control list.
Questionnaires An optional postintervention survey was distributed along with the final MCQE. The first section of the questionnaire asked students to respond to a set of 6 questions using a Likert scale describing their agreeability with the statements. The second section asked students to rank the usefulness of the learning modalities listed in the results section with number one being the most useful.
Statistical analysis Pre- and postintervention MCQE was scored using 2 separate methods: overall percentage correct, as well as by ‘‘teacher status.’’ Teacher status refers to those topics taught by the medical students versus those topics taught by the senior/chief resident. The mean and standard deviation were calculated. A Student t test for paired data was used to evaluate for statistical significance. The NBME data were analyzed using the same method. Questionnaire data were aggregated and the percentage of each modality ranked as number one in usefulness was calculated.
Results A total of 57 students were participants in the team that engaged in the adjunctive curriculum, all of whom sat for the MCQE pre and post examinations, as well as the NBME examination. Fifty-two of the participants elected to fill out the survey. Fifty-nine percent of the participants had completed 2 or more rotations before the month of this curriculum. Fifty-seven percent of participants had
The American Journal of Surgery, Vol -, No -, - 2015
4 Table 2
Mean NBME scores 6 SD
Participants Controls
Table 4
Raw
Percentile
n
77.5 (68.7)* 69.5 (66.8)*
58.4 (626.4) 33.2 (626.0)
57 57
NBME 5 National Board of Medical Examiners; SD 5 standard deviation. *P , .05.
completed a surgical subspecialty rotation, and 65% had completed an internal medicine clerkship. The participants scored significantly higher in the NBME examination (Table 2). The results of the in-house pre- and postintervention MCQE are summarized in Table 3, with an expected significant increase in scores at the end of the rotation. Table 4 dealing with the in-house MCQE score differential between pre and post rotation shows no statistical difference between the scores on topics presented by the participants versus those presented by the senior/chief resident. The questions on the students’ perceptions of the curriculum are listed in Table 5 and show great enthusiasm for the structured curriculum. Table 6 shows that students rated preparation of presentations highly as a learning experience.
Comments The NBME Surgery Subject Examination is the most often failed clerkship examination.14 The third-year surgery clerkship is frequently medical students’ first exposure to basic surgical topics. Didactic time is significantly reduced during the third year, while teaching rounds have become a less frequent occurrence.6 In assessing the needs for an adjunct to the traditional didactic curriculum, we implemented a unique combination of a more structured environment, which used resident leadership and medical student participation in teaching. The first novel aspect of our study was the structured approach. Presentations were delivered each day during morning rounds. Participants were provided with a schedule of topic presentations and were informed that they would be responsible for understanding all these topics by the end of the rotation. The addition of this curriculum to traditional didactics resulted in significantly improved Table 3
Mean percentage of questions correct on MCQ 6 SD
Teacher status
Pretest
Post-test
P value
Overall Peer taught Resident taught
40.4% (613.3) 41.2% (614.7) 36.0% (620.2)
61.6% (614.4) 64.5% (615.8) 53.8% (620.9)
,.05 ,.05 ,.05
Pre- and post-test percentage correct for topic-specific MCQ examination. MCQ 5 multiple choice question; SD 5 standard deviation.
Mean score differential on MCQE (6SD) Student taught Resident taught P value
Score differential 23.3 (619.3)
17.9 (620.6)
.13
Difference between pre- and post-test performance on topicspecific MCQ examination. MCQE 5 multiple choice question Examination; SD 5 standard deviation.
performance in the NBME examination when compared with nonparticipant controls. The surgery clerkship can present unique challenges to medical students. In comparison with rotations such as internal medicine, the surgery rotation presents medical students with topics not even broached during the preclinical years. Although pathologic conditions are encountered and discussed in the preclinical years, it is mostly on the surgery rotation that students learn the indications for and the basics of operative procedures. By having a structured curriculum, we believe that we assure better exposure to the variety of conditions than might be obtained by the varieties of cases available on morning rounds. We believe that presentations on morning rounds bring a sense of urgency and practical importance as students recognize that precious time is taken even if a specific case is not on the ward at that moment. The second novel aspect of our study was the use of the resident as a teaching leader with the medical students acting as teachers as well. As mentioned in the Introduction, Residents as Teacher programs are becoming more popular. These programs focus on the development of teaching skills, not on specific curricula. The effectiveness of these programs remains unclear, however.15 Our curriculum provides residents with a blueprint for teaching while not requiring an excessive amount of time investment. This was accomplished by having the residents co-ordinate and facilitate with the students delivering most of the presentations. To evaluate if this would be effective as compared with resident teaching alone, we developed the MCQE
Table 5 Perceptions of curriculum effectivenessdpercent Likert score greater than 3 (agree, strongly agree) Statement
Percentage
I would like to see the structure of this teaching on other clerkships Preparation for presentations was a useful exercise Topics on the quiz correlated with topics discussed on rounds The residents took time to teach in the clinical setting My residents were capable teachers I tended to focus my studying based on topics that came up on the pretest
96.2%
Postintervention feedback results.
96.2% 96.2% 98.1% 98.1% 38.0%
K. Wirth et al.
Student curriculum improves shelf scores
Table 6 Rating of learning stylesdpercentage rated #1 (scale of 1–4) Learning style
Percentage
Preparing a topic presentation Practice questions Hearing my peers’ topic presentation Residents asking me a question on the spot
41.2% 32.4% 29.4% 23.5%
Postintervention feedback results.
and as noted above, there was no significant difference in the improvement of scores between those topics taught by the resident versus those taught by the students.
5
Applicability This curriculum was designed with the goal of providing a reproducible teaching framework that could easily be adopted by residents and faculty alike. This study lays out a blueprint for the scheduling and topics of the curriculum, with the only demand on the resident or faculty using this curriculum being 5 to 10 minutes of time before or after rounding. By having such a clear plan and with data demonstrating its efficacy, we hoped this curriculum would appeal to both dedicated teachers and those seeking a simple way to fit teaching into their day.
Limitations and strengths of the study Comparison with previous studies In the clerkship education literature, a number of studies have investigated the effect of various peer-led instructional formats on Objective Structured Clinical Examination and procedural technique learning. Fewer have investigated the effects on NBME scores, however. In the surgical education literature, curricula changes, which replaced traditional didactic lectures with PBL or CBL, reported significant improvements in NBME scores as well as positive feedback from the students.1,16 Similar results have been reported in the pediatric, internal medicine, and psychiatry literature.17,18 Studies on obstetrics and gynecology and psychiatry reported no significant changes in NBME performance, but improved student satisfaction.19,20 Although our curriculum is certainly not structured in the same manner as a PBL or CBL, it shares the common elements of required independent preparation and a social learning environment. Also reflecting on the PBL/CBL data, we hypothesized that the peer-led presentations would lead to improved performance when compared with senior/chief-led presentations. Previous studies have theorized that peer-led tutorials provide improved teaching because of a ‘‘cognitive congruence,’’ a theory also supported in our earlier investigation of resident education.21,22 This is the concept that students closer in teaching distance will have a better appreciation for what knowledge the students have already mastered.21 The results of this study, however, suggested that students and the senior/chief resident were equally effective teachers. We believe that the short timing of the presentations encouraged these 2 separate teacher groups to present effectively. With only 5 minutes of time, the presentations must be succinct and cover the salient features of each topic. Finally, the feedback data from our study correspond with that of previous studies. All the PBL/CBL research above, as well as all the peer-assisted learning studies in a recent review article have reported positive feedback from students.23 Interestingly, although students rated resident presentations higher than those given by peers, this was not reflected in higher scores on topics taught by the senior/chief resident on the in-house MCQE.
The limitations of this study include the small sample size and lack of control for previous examination performance (United States Medical Licensing Examination [USMLE], previous NBME examinations.) Also, only one senior/chief resident participated in this pilot study. A strength was our randomization process. By using the institution’s scheduling system, a lottery, we were able to control for selection bias. Evaluating performance with 2 different examination outcome measurements, one of which was a standardized national assessment, is also a strength.
Conclusions This study provides evidence that providing more structure to the third-year surgery clerkship through the implementation of a standardized adjunctive curriculum covering core surgical topics taught by students and residents can significantly improve medical student examination scores. Using residents as facilitators and both the resident and medical students as teachers is both feasible and effective and should be encouraged in the general surgery clerkship.
References 1. Cendan JC, Silver M, Ben-David K. Changing the student clerkship from traditional lectures to small group case-based sessions benefits the student and the faculty. J Surg Educ 2011;68:117–20. 2. Adamas-Rappaport WJ, Waer AL, Teeple MK, et al. A comparison of unguided vs guided case-based instruction on the surgery clerkship. J Surg Educ 2013;70:821–5. 3. McLean SF, Horn K, Tyroch AH. Case based review questions, review sessions, and call schedule type enhance knowledge gains in a surgical clerkship. J Surg Educ 2013;70:68–75. 4. White JS, Sharma N, Boora P. Surgery 101: evaluating the use of podcasting in a general surgery clerkship. Med Teach 2011;33:941–3. 5. Gormley GJ, McGlade K, Thomson C, et al. A virtual surgery in general practice: evaluation of a novel undergraduate virtual patient learning package. Med Teach 2011;33:e522–7. 6. Gonzalo JD, Heist BS, Duffy BL, et al. Identifying and overcoming the barriers to bedside rounds: a multicenter qualitative study. Acad Med 2014;89:326–34.
6
The American Journal of Surgery, Vol -, No -, - 2015
7. Pelletier M, Belliveau P. Role of surgical residents in undergraduate surgical education. Can J Surg 1999;42:451–6. 8. Sheets KJ, Hankin FM, Schwenk TL. Preparing surgery house officers for their teaching role. Am J Surg 1991;161:443–9. 9. Whittaker Jr LD, Estes NC, Ash J, et al. The value of resident teaching to improve student perceptions of surgery clerkships and surgical career choices. Am J Surg 2006;191:320–4. 10. Musunuru S, Lewis B, Rikkers LF, et al. Effective surgical residents strongly influence medical students to pursue surgical careers. J Am Coll Surg 2007;204:164–7. 11. Ostapchuk M, Patel PD, Miller KH, et al. Improving residents’ teaching skills: a program evaluation of residents as teachers course. Med Teach 2010;32:e49–56. 12. Langenfeld SJ, Helmer SD, Cusick TE, et al. Do strong resident teachers help medical students on objective examinations of knowledge? J Surg Educ 2011;68:350–4. 13. Lawrence PF, Bell RM, Dayton MT. Essentials of General Surgery. Philadelphia, PA: Lippincott Williams & Wilkins; 2006. 14. Lind DS, Deladisma AM, Cue JI, et al. Survey of student education in surgery. J Am Coll Surg 2007;204:969–74; discussion, 975–6. 15. Hill AG, Yu TC, Barrow M, et al. A systematic review of resident-asteacher programmes. Med Educ 2009;43:1129–40.
16. Schwartz RW, Donnelly MB, Nash PP, et al. Problem-based learning: an effective educational method for a surgery clerkship. J Surg Res 1992;53:326–30. 17. Curtis JA, Indyk D, Taylor B. Successful use of problem-based learning in a third-year pediatric clerkship. Ambul Pediatr 2001;1:132–5. 18. Levine RE, O’Boyle M, Haidet P, et al. Transforming a clinical clerkship with team learning. Teach Learn Med 2004;16:270–5. 19. Nalesnik SW, Heaton JO, Olsen CH, et al. Incorporating problembased learning into an obstetrics/gynecology clerkship: impact on student satisfaction and grades. Am J Obstet Gynecol 2004;190: 1375–81. 20. Morreale M, Arfken C, Bridge P, et al. Incorporating active learning into a psychiatry clerkship: does it make a difference? Acad Psychiatry 2012;36:223–5. 21. Wirth K, Malone B, Barrera K, et al. Is there a place for medical students as teachers in the education of junior residents? Am J Surg 2014; 207:271–4. 22. Ten Cate O, Durning S. Dimensions and psychology of peer teaching in medical education. Med Teach 2007;29:546–52. 23. Yu TC, Wilson NC, Singh PP, et al. Medical students-as-teachers: a systematic review of peer-assisted teaching during medical school. Adv Med Educ Pract 2011;2:157–72.