- Email: [email protected]
Student perceptions of a simulation-based flipped classroom for the surgery clerkship: A mixed-methods study
ARTICLE IN PRESS
Student perceptions of a simulation-based flipped classroom for the surgery clerkship: A mixed-methods study Cara A. Liebert, MD, Laura Mazer, MD, MS, Sylvia Bereknyei Merrell, DrPH, MS, Dana T. Lin, MD, and James N. Lau, MD Palo Alto, CA
Background. The flipped classroom, a blended learning paradigm that uses pre-session online videos reinforced with interactive sessions, has been proposed as an alternative to traditional lectures. This article investigates medical students’ perceptions of a simulation-based, flipped classroom for the surgery clerkship and suggests best practices for implementation in this setting. Methods. A prospective cohort of students (n = 89), who were enrolled in the surgery clerkship during a 1-year period, was taught via a simulation-based, flipped classroom approach. Students completed an anonymous, end-of-clerkship survey regarding their perceptions of the curriculum. Quantitative analysis of Likert responses and qualitative analysis of narrative responses were performed. Results. Students’ perceptions of the curriculum were positive, with 90% rating it excellent or outstanding. The majority reported the curriculum should be continued (95%) and applied to other clerkships (84%). The component received most favorably by the students was the simulation-based skill sessions. Students rated the effectiveness of the Khan Academy-style videos the highest compared with other video formats (P < .001). Qualitative analysis identified 21 subthemes in 4 domains: general positive feedback, educational content, learning environment, and specific benefits to medical students. The students reported that the learning environment fostered accountability and self-directed learning. Specific perceived benefits included preparation for the clinical rotation and the National Board of Medical Examiners shelf exam, decreased class time, socialization with peers, and faculty interaction. Conclusion. Medical students’ perceptions of a simulation-based, flipped classroom in the surgery clerkship were overwhelmingly positive. The flipped classroom approach can be applied successfully in a surgery clerkship setting and may offer additional benefits compared with traditional lecture-based curricula. (Surgery 2016;j:j-j.) From the Goodman Surgical Education Center, Division of General Surgery, Department of Surgery, Stanford University School of Medicine, Palo Alto, CA
TRADITIONAL MODELS OF UNDERGRADUATE MEDICAL EDUCATION use didactic lectures as the primary mode of classroom instruction. Even when clerkship Supported by an Innovation in Teaching and Researching Online and Blended Courses Grant from the Office of the Vice Provost for Online Learning at Stanford University and the Stanford University School of Medicine. Presented as an oral paper at the 8th Annual Meeting of the Consortium of American College of Surgeons-accredited Education Institutes on March 13–14, 2015, Chicago, IL. Accepted for publication March 25, 2016. Reprint requests: Cara A. Liebert, MD, Division of General Surgery, Department of Surgery, Stanford University Medical Center, 300 Pasteur Drive, H3591, Stanford, CA 94305-5641. E-mail: [email protected]. 0039-6060/$ - see front matter Ó 2016 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.surg.2016.03.034
students rotate on clinical ward teams, the majority of formal learning experiences on required core clerkships remain traditional didactic lectures. Alternative models of education include the “flipped classroom,” which is a blended learning paradigm that uses short online videos prior to class and interactive small group sessions that reinforce the pre-session material.1 Published studies have suggested that learning gains in flipped classroom curricula may be the result of increased active learning.2 This model of education has been applied to preclinical medical education curricula in various fields, ranging from undergraduate nursing courses3-5 to pharmacy student sessions.6 This modality also has been piloted with limited scope in graduate medical education, including emergency medicine and internal medicine residency programs.7,8 Limited reports exist regarding SURGERY 1
ARTICLE IN PRESS 2 Liebert et al
application of the flipped classroom to the curriculum of a medical student clerkship. Belfi et al9 applied the flipped classroom model to an introductory clerkship in radiology with positive student perceptions. In Morgan et al,10 a flipped classroom was implemented as part of a 4-week elective in obstetrics and gynecology for graduating fourth-year medical students. To date, there have been no reports of implementing a flipped classroom in the surgery core clerkship setting. Potential benefits of applying this model of education to a core clerkship include the ability for busy students to watch lecture materials asynchronously without adding to time away from their clinical teams and the potential to increase the time devoted to active learning modalities, such as simulation- and problem-based learning during class sessions. The purpose of this mixedmethods study was to examine perceptions of third-year medical students regarding a novel, simulation-based, flipped classroom for the surgery clerkship. The students’ evaluations were intended to provide surgical educators insight into the value and benefits of applying this curricular model to the core clerkship setting. METHODS Study design and setting. A mixed-methods study design of qualitative and quantitative methods was used to explore medical students’ perceptions of a simulation-based, flipped classroom curriculum for the surgery core clerkship. The study protocol submitted to the Institutional Review Board at Stanford University was determined exempt from additional Institutional Review Board consideration. The study was conducted in an American College of Surgeonsaccredited Education Institute in the Department of Surgery at the Stanford University School of Medicine. Description of the simulation-based, flipped classroom curriculum. The flipped classroom curricular model for the surgery core clerkship consisted of an 8-week curriculum divided into 1-week modules implemented during a 1-year period from March 2014 to February 2015. The curriculum contained an orientation module, 6 clinical general surgery content modules, and a debriefing module. The orientation module was held during the first week of the clerkship and consisted of introduction to policies and procedures, discussion of the surgical learning environment, a skills session on suturing and knot tying, and a simulation-based session on procedural skills (chest tube placement, central line placement, and
Surgery j 2016
laparoscopic camera setup and navigation). The 6 general surgery content modules were: 1) breast, 2) foregut, bariatric, small intestine, and hernias, 3) hepatobiliary and pancreatic, 4) colorectal, 5) trauma, and 6) perioperative care. Each clinical content module was comprised of a pretest, presession online videos, review of pretest questions, a case-based clinical reasoning session, a simulationbased skills session, and a post-test (Fig 1). The online videos were hosted on the Open edX platform (code.edx.org), with students required to view the videos at home before attending the in-class session. Videos were produced in 1 of 3 formats: Khan Academy-style11 with digital inking, traditional PowerPoint slides, or a hybrid of PowerPoint slides with digital inking. Khan Academy-style videos were produced either with live digital inking by the faculty member or in conjunction with an artist and did not contain still photos or images. Videos produced with PowerPoint slides often included still images of anatomy, clinical findings, radiology, or pathology. Videos ranged 6–18 minutes in duration and used a black background, white text, audio voiceover, and colored digital inking (when applicable). The in-class portion of the curriculum was held 1 afternoon per week for a 4-hour block. Pretest and post-test questions were completed online during class each week. Simulation modalities incorporated in the curriculum included: knot tying (knot-tying boards), suturing (synthetic skin pads), ultrasound-guided central line placement (whole chicken tissue model),12 chest tube placement (TraumaMan, SimuLab Corporation, Seattle, WA), ultrasound-guided breast biopsy (chicken breast model), stapled bowel anastomosis (porcine intestine model), Fundamentals of Laparoscopic Surgery (FLS) tasks, upper endoscopy virtual reality (EndoscopyVR, CAE Healthcare, Sarasota, FL), mannequin-based trauma team simulations (SimMan, Laerdal, Stavanger, Norway), a low-cost, laparoscopic cholecystectomy model (Fig 2), and the Surgical Improvement of Clinical Knowledge Ops (SICKO) Online Decision-Making Video Game (https://med.stanford.edu/cme/courses/ online/sicko.html).13 SICKO Online Decisionmaking Video Game was included in 6 of the 12 months of the curriculum; the remaining 6 months were substituted with a required School of Medicine Reflection Rounds for all core clerkship students. The low-cost, laparoscopic cholecystectomy model was created by the authors for this curriculum using cardboard to represent the liver, a balloon filled with Play-Doh (Hasbro, Pawtucket, RI) to represent the gallbladder, Penrose drains of
ARTICLE IN PRESS Surgery Volume j, Number j
Liebert et al 3
Fig 1. A diagram of a simulation-based, flipped classroom curriculum model for the Surgery Core Clerkship.
Fig 2. A low-cost laparoscopic cholecystectomy model.
varying sizes to represent the cystic and common bile duct, red twine to represent the cystic artery, and polyester fiber to represent connective tissue. The model is compatible with a tower in the setup for the FLS and allows students to perform a mock cholecystectomy with laparoscopic dissectors, clips, and shears. A faculty-led, surgical education fellow(s) and simulation center staff were present at all sessions. Participants. A prospective cohort of third-year medical students who were enrolled in the surgery core clerkship (n = 89) during the first year of implementation of the flipped classroom approach (March 2014 to February 2015) were invited to complete an end-of-clerkship online curriculum evaluation survey; 77 students completed the online evaluation (87% response rate). Data collection. The end-of-clerkship, online curriculum evaluation included both open- and closed-ended questions regarding students’
perceptions of the simulation-based, flipped classroom curricular model for the surgery clerkship. Students rated the curriculum overall, the general components of the curriculum, and the specific components of each module on a 5-point Likert scale (poor, fair, average, excellent, or outstanding). Students rated their level of agreement with a series of statements on a 5-point Likert scale ranging from strongly disagree to strongly agree. Data were collected regarding user feedback on ease of use of the online video and pre- and post-test platforms, and participants were asked to rank the effectiveness of each of the 3 video formats used (1 = most effective and 3 = least effective). Qualitative data in this study consisted of student responses to 2 open-ended questions: “Describe your overall impressions of the Surgery Core Clerkship Flipped Classroom curriculum” and “please list any topics you think should be added to the Surgery Core Clerkship Flipped Classroom curriculum.” Qualitative responses were imported and coded in Dedoose qualitative analysis software (version 5.2.1., 2015, SocioCultural Research Consultants, Los Angeles, CA) for ease of data management. Data analysis. For the quantitative survey data, the number of respondents and percentages are reported for each Likert scale response. Means and standard deviations were calculated for effectiveness rankings of video format and compared statistically by one-way analysis of variance (ANOVA) with P values < .05 considered statistically significant. Open-ended responses were analyzed qualitatively using the constant comparative method in a team-based, analytic approach.14 Two study investigators identified major themes and subthemes in an iterative fashion, and a
ARTICLE IN PRESS 4 Liebert et al
Surgery j 2016
Table I. Student ratings of surgery clerkship flipped classroom curricular components Component
n
Poor % (no.)
Fair % (no.)
Average % (no.)
Excellent % (no.)
Outstanding % (no.)
Overall General component Online videos In-class clinical reasoning cases In-class simulation-based skill sessions Online pre-session practice questions Online post-session practice questions Review of pre-session questions in class Specific simulation sessions Suturing and knot tying Chest tube and central line placement US-guided breast biopsy and cyst aspiration FLS and endoscopy VR Mannequin-based trauma team simulations Laparoscopic cholecystectomy model SICKO surgical decision-making game* Stapled bowel anastomosis
76
0 (0)
0 (0)
10.5 (8)
51 (39)
38 (29)
77 77 77 76 77 77
0 1.3 0 1.3 1.3 0
(0) (1) (0) (1) (1) (0)
2.6 2.6 0 0 0 0
(2) (2) (0) (0) (0) (0)
10.4 13.0 2.6 25.0 23.4 10.4
(8) (10) (2) (19) (18) (8)
60 53 49 47 48 43
(46) (41) (38) (36) (37) (33)
27 30 48 26 27 47
(21) (23) (37) (20) (21) (36)
77 77 77 77 76 77 37 76
0 0 1.3 0 0 0 2.7 0
(0) (0) (1) (0) (0) (0) (1) (0)
0 0 2.6 0 0 0 2.7 0
(0) (0) (2) (0) (0) (0) (1) (0)
6.5 7.8 20.8 5.2 1.3 3.9 13.5 7.9
(5) (6) (16) (4) (1) (3) (5) (6)
48 47 39 39 34 39 46 43
(37) (36) (30) (30) (26) (30) (17) (33)
46 46 36 56 65 57 35 49
(35) (35) (28) (43) (49) (44) (13) (37)
*This simulation modality was included in 6 of the 12 months of the flipped classroom curriculum, representing n = 37 students. Due to rounding, percentages may not add up to 100%. US, Ultrasound; FLS, Fundamentals of Laparoscopic Surgery; VR, virtual reality; SICKO, Surgical Improvement of Clinical Knowledge Ops.
consensus-coding framework was established. The units of analysis were sentences. Both manifest and latent coding were used in an inductive approach.15 Final coding consensus was reached by discussion between the investigators (C.L. and L.M.). The data were then coded independently by the 2 investigators (C.L. and L.M.) using this consensus framework. Inter-rater reliability was calculated by Cohen’s kappa for all subthemes using Dedoose. Both reviewers were education fellows enrolled in a master’s level program in health professions education at the time of the study with prior training in qualitative methods and qualitative data analysis. A faculty investigator with experience in medical education research and curriculum design (D.L.) performed a peer audit of the narrative responses and coding framework. Representative example quotes of each subtheme were identified from the data. The number and percentage of student responses that exhibited each subtheme were calculated. RESULTS Quantitative results. A total of 77 students completed the online end-of-clerkship curriculum evaluation survey (87% response rate). Results of student ratings of the overall curricular model, general curricular components, and specific simulation-based sessions are shown in Table I. Overall, 90% of students rated the overall
curriculum excellent or outstanding, with none rating the curriculum poor or fair. Similar findings were noted for the curricular components, with the majority rating the online videos (87%), clinical reasoning cases (83%), in-class simulation-based skill sessions (98%), pre-session practice questions (74%), postsession practice questions (75%), and review of pre-session questions (90%) as excellent or outstanding. The component that was received most favorably by the students was the simulation-based skill sessions. Of these, the highest rated simulations were the FLS/EndoscopyVR, mannequin-based trauma team simulations, and the low-cost laparoscopic cholecystectomy model. Most students (77%) found the online pre- and postsession testing platform (Qualtrics software, Qualtrics, Provo, Utah) very easy to use, with the remaining students reporting it somewhat easy to use (22%) and 1 student (1%) reporting it somewhat difficult to use. Similar user feedback was given regarding the online Open edX video platform, with 79% reporting it very easy to use, 17% somewhat easy to use, and 4% somewhat difficult to use. Students rated the effectiveness of the Khan Academy-style video format the greatest compared with the hybrid of PowerPoint slides with digital inking format or traditional PowerPoint slides format (1.6 ± 0.8 vs 2.7 ± 0.5; 2.7; P < .001).
ARTICLE IN PRESS Liebert et al 5
Surgery Volume j, Number j
Table II. Student perceptions of overall surgery clerkship flipped classroom curricular model Strongly disagree % (no.)
Disagree % (no.)
Neither agree or disagree % (no.)
Agree % (no.)
Strongly agree % (no.)
5 (4)
26 (20)
69 (53)
Statement
n
The flipped classroom model for the surgery clerkship should be continued. The flipped classroom model should be applied to other core clerkships. The flipped classroom model allows students to spend more time with their clinical team. The flipped classroom model helped me feel prepared for the NBME shelf exam. The flipped classroom model provided adequate faculty exposure and interaction.
77
0 (0)
0 (0)
77
0 (0)
2.6 (2)
13 (10)
29 (22)
56 (43)
77
0 (0)
3.9 (3)
20 (15)
36 (28)
40 (31)
77
1.3 (1)
6.5 (5)
16 (12)
42 (32)
35 (27)
77
0 (0)
2.6 (2)
10 (8)
48 (37)
39 (30)
Due to rounding, percentages may not add up to 100%. NBME, National Board of Medical Examiners.
Students’ perceptions of the overall flipped classroom curricular model are shown in Table II. The vast majority of students reported that the flipped classroom model for the surgery core clerkship should be continued (95%), with the remaining students (5%) neutral in opinion. Students also reported that the flipped classroom model allowed them to spend more time with their clinical team (77%), provided adequate faculty exposure and interaction (87%), and should be applied to other core clerkships (84%). Qualitative results. Of the 77 students who completed the online, end-of-clerkship curriculum evaluation, 91% (n = 70) submitted narrative responses regarding their overall impressions of the flipped classroom curricular model. The majority of the responses were positive (96%); the remaining students (n = 3) expressed neutral impressions. No negative narrative responses were received. Qualitative analysis identified 4 major themes with 21 subthemes (Table III). The major themes identified were general positive feedback, educational content, learning environment, and specific benefits to students. Inter-rater reliability was measured with Cohen’s kappa, with a pooled kappa of 0.76 for all subthemes. Eight subthemes had an inter-rater reliability of 1.00. Other surgical topics suggested by students to be added to the flipped classroom curriculum for the surgery core clerkship included orthopedics (n = 17), vascular and pediatric (n = 7 each), cardiothoracic (n = 6), urology (n = 5), and otolaryngology, endocrine, and plastics (n # 4). Each major theme is explored in further detail below.
Representative narrative responses by the medical students are shown in the Addendum (online only version). General positive feedback. Students expressed strong positive feedback for the surgery core clerkship flipped classroom approach overall and for its specific components, including online videos, practice questions, clinical reasoning cases, simulation-based skills sessions, and the teaching faculty/staff. Multiple students also commented on the potential for this flipped classroom curricular model to be applied to other core clerkships. Educational content. Students valued several specific characteristics of the educational content, including succinct videos, high-yield material, practical and relevant topics, and the use of multiple modalities reinforcing knowledge. Learning environment. Students commented on the unique learning environment of the flipped classroom model, specifically its ability to create an interactive, engaging, and innovative environment that provided protected educational time and fostered self-directed learning and accountability in prework. Specific benefits to students. Various perceived benefits to students were identified based on the student responses. Reported benefits included preparation for the clinical rotation, preparation for the surgery shelf exam of the National Board of Medical Examiners (NBME), socialization with peers, and interaction with teaching faculty. Students additionally reported that the innovative, flipped classroom curricular model allowed for decreased class time compared with the traditional
ARTICLE IN PRESS 6 Liebert et al
Surgery j 2016
Table III. Qualitative analysis of students’ overall impressions of surgery clerkship flipped classroom Themes and subthemes General positive feedback Overall curriculum
Theme (# of responses) 30
Simulation-based skills sessions Clinical reasoning cases
8
Teaching faculty/staff
7
Online videos Practice questions Educational content Succinct videos High-yield material Multiple modalities
Practical and relevant topics Learning environment Interactive Engaging
9
6 2 3 4 3
1 5 5
Accountability
3
Self-directed learning
3
Protected educational time
4
Innovative Specific benefits to students Preparation for clinical rotation
2 5
Preparation for NBME shelf Reduced class time
3 3
Socialization with peers Interaction with faculty
1 1
Example quote “Absolutely out-of-the-box thinking with regards to clerkship education, wish other clerkships would follow example.” “The surgical skills sessions were [the] most valuable part of the training.” “I also love the clinical reasoning model that we used—case-based learning, and appreciated the chance to brainstorm differentials like you would for a real ED consult or clinic patient.” “Fantastic experience overall with great attendings and residents who care about teaching.” “Love the videos—especially the Khan Academy-style ones.” “. learned a lot going over questions .” “. videos were all really succinct, effective, and engaging.” “Reduced total class time, but the learning was much more high yield.” “Getting the information presented to me in multiple different ways (videos, questions, lectures, cases) was helpful for reinforcing what I was learning.” “.relevant to what we needed to know for the wards and the shelf.” “Really enjoyed how interactive the sessions were.” “I thought it was fun, engaging, and I learned a lot compared with other didactic sessions in other clerkships.” “I liked watching the videos and doing pre-questions because this held me accountable to the information and made me feel prepared and ready to learn when I came into class.” “It allowed us to learn the material at home at our own pace, which greatly helped me ‘get’ the learning points since I had time to look things up while watching the videos.” “. beneficial because we had scheduled time to be in didactics that were predictable for both us as students and our team.” “Innovative .” “.getting to use the EGD simulator, doing the choles, bowel anastomosis!!, running through a primary/secondary for a trauma, doing ultrasound guided activities - made me feel more confident when I was doing these activities with real patients or observing my team carry out these procedures.” “Great NBME prep.” “I appreciated that my time wasn’t wasted with boring lectures during the day that I was missing cases to attend.” “You get to see classmates.” “I also really appreciated meeting the faculty/attendings via the videos and then in person.”
n = 70; 91% of students who completed the survey submitted narrative responses. ED, Emergency Department; EGD, esophagogastroduodenoscopy; NBME, National Board of Medical Examiners.
lecture format, which translated to more time in the operating room with their clinical team. DISCUSSION This mixed-methods study explored medical students’ perceptions of a simulation-based, flipped classroom curricular approach for the surgery
core clerkship, with complementary quantitative and qualitative results. The findings suggest that the flipped classroom curricular model in the surgery core clerkship is well received by students, with the majority rating the overall curriculum and each of its subcomponents as excellent or outstanding. Qualitative analysis of narrative
ARTICLE IN PRESS Surgery Volume j, Number j
responses suggested that the vast majority of students expressed positive overall impressions, and no students expressed negative perceptions. Although all components of the curricular structure, including videos, pre- and post-test questions, clinical reasoning cases, and simulation-based skills sessions, were valued, the simulation-based skill sessions were the best-received subcomponents. An overwhelming majority (95%) reported the flipped classroom curricular model should be continued in the surgery core clerkship and applied to core clerkships in other disciplines. The qualitative thematic analysis of narrative responses revealed 4 major themes, including general positive feedback, educational content, learning environment, and specific benefits to students. The students noted that the combination of multiple complimentary teaching modalities reinforcing knowledge was a particular strength. Valued aspects of the educational content included the succinct videos, high-yield material, multiple modalities, and practical and relevant topics. Students particularly favored videos designed with a Khan Academy11 style of production <10 minutes in duration. Students reported that the flipped classroom fostered an engaging, interactive, and innovative learning environment that encouraged self-directed learning and accountability. Notably, no students complained of the required prework; in contrast, the opportunity for self-paced, independent learning at home was valued and suggests that students may desire this mode of content delivery when reinforced by active learning during class sessions. Students also identified specific benefits of the curriculum, including preparation for clinical rotations, preparation for the NBME shelf exam, decreased class time, socialization with peers, and interaction with faculty. Although one potential criticism of this flipped classroom curricular model is that students interact with fewer faculty members compared with the traditional didactic curriculum, the results of this study indicate that the majority of students perceived adequate faculty exposure and interaction. Although a smaller number of faculty members taught during the in-class sessions compared with a traditional didactic curriculum, it is possible that the level of student interaction and engagement with faculty was more intensive and meaningful in the flipped classroom. This flipped classroom curricular model could be adapted easily for surgery clerkships at other institutions or for core clerkships in other specialties. Based on our experiences and study
Liebert et al 7
Table IV. Recommended 10 best practices Consistent protected educational time Standardized overall format Topic-based clinical modules Multiple instructional methods Succinct, high-quality video prework Accountability in completion of prework Use of practice questions Case-based learning Simulation-based skills training Committed staff and faculty leads
findings, we propose best practices for implementing a flipped classroom for the core clerkship environment (Table IV). We suggest organization of the curriculum into separate topic-based modules with a repeating schedule and format, because students appreciated consistent, protected educational time each week using a standard format. We recommend inclusion of multiple instructional methods, such as practice questions, case-based clinical reasoning, and simulation-based skills sessions, to allow for knowledge application and consolidation of learning. Video prework was well received by students and should use succinct videos, such as those produced using a Khan Academy11 format, when possible. Efforts should be taken to maintain an interactive and engaging learning environment where students are held accountable for completing prework prior to the session. Multiple-choice NBME-style practice questions should be incorporated into the curriculum in the form of pre- and post-tests or practice questions, because students saw preparation for the NBME shelf exam as a specific benefit of a simulation-based curriculum. Based on student feedback, we plan to expand the content to include surgical subspecialty topics such as orthopedics, vascular surgery, cardiothoracic surgery, pediatric surgery, and urology, among others. Additional planned changes in the curriculum include replacement of traditional PowerPoint video content with Khan Academy-style videos and periodic revision of practice questions and clinical reasoning cases. It is important to highlight that the strength of a flipped classroom curriculum depends on the combination of high-quality succinct videos and relevant curricular content with effective instructional methods to foster active learning. Our study has several limitations, including that it was conducted at a single academic institution, which may limit its generalizability. The
ARTICLE IN PRESS 8 Liebert et al
curriculum created and implemented by dedicated staff at the Goodman Surgical Education Center was time-intensive, which may limit its generalizability to institutions with a small teaching staff or limited access to a simulation center. A lead faculty and surgical education fellow were present at each session to provide format consistency, continuity, and quality assurance, which may not be feasible for all institutions. Students completed the online survey 3 days prior to the NBME shelf exam, which limited their ability to determine whether the curriculum prepared them for the exam. This study demonstrated positive student ratings and perceptions of a flipped classroom curriculum for the surgery clerkship. Still, additional studies are needed to compare the effectiveness of a flipped classroom curriculum with more traditional curricula, with an eye toward knowledge acquisition and application and measurable objective outcomes. SUPPLEMENTARY DATA Supplementary data related to this article can be found online at http://dx.doi.org/10.1016/j.surg.2016.03.034. REFERENCES 1. Prober CG, Khan S. Medical education reimagined: A call to action. Acad Med 2013;88:1407-10. 2. Jensen JL, Kummer TA, Godoy PD. Improvements from a flipped classroom may simply be the fruits of active learning. CBE Life Sci Educ 2015;14:ar5. 3. Simpson V, Richards E. Flipping the classroom to teach population health: Increasing the relevance. Nurse Educ Pract 2015;15:162-7.
Surgery j 2016
4. Ratta CB. Flipping the classroom with team-based learning in undergraduate nursing education. Nurse Educ 2015;40: 71-4. 5. Schlairet MC, Green R, Benton MJ. The flipped classroom: Strategies for an undergraduate nursing course. Nurse Educ 2014;39:321-5. 6. Ferreri SP, O’Connor SK. Redesign of a large lecture course into a small-group learning course. Am J Pharm Educ 2013; 77:13. 7. Young TP, Bailey CJ, Guptill M, Thorp AW, Thomas TL. The flipped classroom: A modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med 2014;15:938-44. 8. Vincent DS. Out of the wilderness: Flipping the classroom to advance scholarship in an internal medicine residency program. Hawaii J Med Public Health 2014;73:2-3. 9. Belfi LM, Bartolotta RJ, Giambrone AE, Davi C, Min RJ. “Flipping” the introductory clerkship: Impact on medical student performance and perceptions. Acad Radiol 2015; 22:794-801. 10. Morgan H, Marzano D, Lanham M, Stein T, Curran D, Hammoud M. Preparing medical students for obstetrics and gynecology milestone level one: A description of a pilot curriculum. Med Educ Online 2014;19:25746. 11. Khan Academy: Health and Medicine. Available from: https://www.khanacademy.org/science/health-and-medicine. 12. Ault MJ, Rosen BT, Ault B. The use of tissue models for vascular access training: Phase I of the Procedural Patient Safety Initiative. J Gen Intern Med 2006;21(5):514-7. 13. Lin DT, Park J, Liebert CA, Lau JN. Validity evidence for Surgical Improvement of Clinical Knowledge Ops: A novel gaming platform to assess surgical decision making. Am J Surg 2015;209:79-85. 14. Guest G, MacQueen KM. Handbook for team-based qualitative research. Washington, D.C.: Rowman & Littlefield Pub Incorporated; 2008. 15. Miles MB, Michael Huberman A, Saldana J. Qualitative data analysis: A methods sourcebook. 3rd ed. Thousand Oaks, California: SAGE Publications, Inc; 2014.
Student perceptions of a simulation-based flipped classroom for the surgery clerkship: A mixed-methods study Cara A. Liebert, MD, Laura Mazer, MD, MS, Sylvia Bereknyei Merrell, DrPH, MS, Dana T. Lin, MD, and James N. Lau, MD Palo Alto, CA
Background. The flipped classroom, a blended learning paradigm that uses pre-session online videos reinforced with interactive sessions, has been proposed as an alternative to traditional lectures. This article investigates medical students’ perceptions of a simulation-based, flipped classroom for the surgery clerkship and suggests best practices for implementation in this setting. Methods. A prospective cohort of students (n = 89), who were enrolled in the surgery clerkship during a 1-year period, was taught via a simulation-based, flipped classroom approach. Students completed an anonymous, end-of-clerkship survey regarding their perceptions of the curriculum. Quantitative analysis of Likert responses and qualitative analysis of narrative responses were performed. Results. Students’ perceptions of the curriculum were positive, with 90% rating it excellent or outstanding. The majority reported the curriculum should be continued (95%) and applied to other clerkships (84%). The component received most favorably by the students was the simulation-based skill sessions. Students rated the effectiveness of the Khan Academy-style videos the highest compared with other video formats (P < .001). Qualitative analysis identified 21 subthemes in 4 domains: general positive feedback, educational content, learning environment, and specific benefits to medical students. The students reported that the learning environment fostered accountability and self-directed learning. Specific perceived benefits included preparation for the clinical rotation and the National Board of Medical Examiners shelf exam, decreased class time, socialization with peers, and faculty interaction. Conclusion. Medical students’ perceptions of a simulation-based, flipped classroom in the surgery clerkship were overwhelmingly positive. The flipped classroom approach can be applied successfully in a surgery clerkship setting and may offer additional benefits compared with traditional lecture-based curricula. (Surgery 2016;j:j-j.) From the Goodman Surgical Education Center, Division of General Surgery, Department of Surgery, Stanford University School of Medicine, Palo Alto, CA
TRADITIONAL MODELS OF UNDERGRADUATE MEDICAL EDUCATION use didactic lectures as the primary mode of classroom instruction. Even when clerkship Supported by an Innovation in Teaching and Researching Online and Blended Courses Grant from the Office of the Vice Provost for Online Learning at Stanford University and the Stanford University School of Medicine. Presented as an oral paper at the 8th Annual Meeting of the Consortium of American College of Surgeons-accredited Education Institutes on March 13–14, 2015, Chicago, IL. Accepted for publication March 25, 2016. Reprint requests: Cara A. Liebert, MD, Division of General Surgery, Department of Surgery, Stanford University Medical Center, 300 Pasteur Drive, H3591, Stanford, CA 94305-5641. E-mail: [email protected]. 0039-6060/$ - see front matter Ó 2016 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.surg.2016.03.034
students rotate on clinical ward teams, the majority of formal learning experiences on required core clerkships remain traditional didactic lectures. Alternative models of education include the “flipped classroom,” which is a blended learning paradigm that uses short online videos prior to class and interactive small group sessions that reinforce the pre-session material.1 Published studies have suggested that learning gains in flipped classroom curricula may be the result of increased active learning.2 This model of education has been applied to preclinical medical education curricula in various fields, ranging from undergraduate nursing courses3-5 to pharmacy student sessions.6 This modality also has been piloted with limited scope in graduate medical education, including emergency medicine and internal medicine residency programs.7,8 Limited reports exist regarding SURGERY 1
ARTICLE IN PRESS 2 Liebert et al
application of the flipped classroom to the curriculum of a medical student clerkship. Belfi et al9 applied the flipped classroom model to an introductory clerkship in radiology with positive student perceptions. In Morgan et al,10 a flipped classroom was implemented as part of a 4-week elective in obstetrics and gynecology for graduating fourth-year medical students. To date, there have been no reports of implementing a flipped classroom in the surgery core clerkship setting. Potential benefits of applying this model of education to a core clerkship include the ability for busy students to watch lecture materials asynchronously without adding to time away from their clinical teams and the potential to increase the time devoted to active learning modalities, such as simulation- and problem-based learning during class sessions. The purpose of this mixedmethods study was to examine perceptions of third-year medical students regarding a novel, simulation-based, flipped classroom for the surgery clerkship. The students’ evaluations were intended to provide surgical educators insight into the value and benefits of applying this curricular model to the core clerkship setting. METHODS Study design and setting. A mixed-methods study design of qualitative and quantitative methods was used to explore medical students’ perceptions of a simulation-based, flipped classroom curriculum for the surgery core clerkship. The study protocol submitted to the Institutional Review Board at Stanford University was determined exempt from additional Institutional Review Board consideration. The study was conducted in an American College of Surgeonsaccredited Education Institute in the Department of Surgery at the Stanford University School of Medicine. Description of the simulation-based, flipped classroom curriculum. The flipped classroom curricular model for the surgery core clerkship consisted of an 8-week curriculum divided into 1-week modules implemented during a 1-year period from March 2014 to February 2015. The curriculum contained an orientation module, 6 clinical general surgery content modules, and a debriefing module. The orientation module was held during the first week of the clerkship and consisted of introduction to policies and procedures, discussion of the surgical learning environment, a skills session on suturing and knot tying, and a simulation-based session on procedural skills (chest tube placement, central line placement, and
Surgery j 2016
laparoscopic camera setup and navigation). The 6 general surgery content modules were: 1) breast, 2) foregut, bariatric, small intestine, and hernias, 3) hepatobiliary and pancreatic, 4) colorectal, 5) trauma, and 6) perioperative care. Each clinical content module was comprised of a pretest, presession online videos, review of pretest questions, a case-based clinical reasoning session, a simulationbased skills session, and a post-test (Fig 1). The online videos were hosted on the Open edX platform (code.edx.org), with students required to view the videos at home before attending the in-class session. Videos were produced in 1 of 3 formats: Khan Academy-style11 with digital inking, traditional PowerPoint slides, or a hybrid of PowerPoint slides with digital inking. Khan Academy-style videos were produced either with live digital inking by the faculty member or in conjunction with an artist and did not contain still photos or images. Videos produced with PowerPoint slides often included still images of anatomy, clinical findings, radiology, or pathology. Videos ranged 6–18 minutes in duration and used a black background, white text, audio voiceover, and colored digital inking (when applicable). The in-class portion of the curriculum was held 1 afternoon per week for a 4-hour block. Pretest and post-test questions were completed online during class each week. Simulation modalities incorporated in the curriculum included: knot tying (knot-tying boards), suturing (synthetic skin pads), ultrasound-guided central line placement (whole chicken tissue model),12 chest tube placement (TraumaMan, SimuLab Corporation, Seattle, WA), ultrasound-guided breast biopsy (chicken breast model), stapled bowel anastomosis (porcine intestine model), Fundamentals of Laparoscopic Surgery (FLS) tasks, upper endoscopy virtual reality (EndoscopyVR, CAE Healthcare, Sarasota, FL), mannequin-based trauma team simulations (SimMan, Laerdal, Stavanger, Norway), a low-cost, laparoscopic cholecystectomy model (Fig 2), and the Surgical Improvement of Clinical Knowledge Ops (SICKO) Online Decision-Making Video Game (https://med.stanford.edu/cme/courses/ online/sicko.html).13 SICKO Online Decisionmaking Video Game was included in 6 of the 12 months of the curriculum; the remaining 6 months were substituted with a required School of Medicine Reflection Rounds for all core clerkship students. The low-cost, laparoscopic cholecystectomy model was created by the authors for this curriculum using cardboard to represent the liver, a balloon filled with Play-Doh (Hasbro, Pawtucket, RI) to represent the gallbladder, Penrose drains of
ARTICLE IN PRESS Surgery Volume j, Number j
Liebert et al 3
Fig 1. A diagram of a simulation-based, flipped classroom curriculum model for the Surgery Core Clerkship.
Fig 2. A low-cost laparoscopic cholecystectomy model.
varying sizes to represent the cystic and common bile duct, red twine to represent the cystic artery, and polyester fiber to represent connective tissue. The model is compatible with a tower in the setup for the FLS and allows students to perform a mock cholecystectomy with laparoscopic dissectors, clips, and shears. A faculty-led, surgical education fellow(s) and simulation center staff were present at all sessions. Participants. A prospective cohort of third-year medical students who were enrolled in the surgery core clerkship (n = 89) during the first year of implementation of the flipped classroom approach (March 2014 to February 2015) were invited to complete an end-of-clerkship online curriculum evaluation survey; 77 students completed the online evaluation (87% response rate). Data collection. The end-of-clerkship, online curriculum evaluation included both open- and closed-ended questions regarding students’
perceptions of the simulation-based, flipped classroom curricular model for the surgery clerkship. Students rated the curriculum overall, the general components of the curriculum, and the specific components of each module on a 5-point Likert scale (poor, fair, average, excellent, or outstanding). Students rated their level of agreement with a series of statements on a 5-point Likert scale ranging from strongly disagree to strongly agree. Data were collected regarding user feedback on ease of use of the online video and pre- and post-test platforms, and participants were asked to rank the effectiveness of each of the 3 video formats used (1 = most effective and 3 = least effective). Qualitative data in this study consisted of student responses to 2 open-ended questions: “Describe your overall impressions of the Surgery Core Clerkship Flipped Classroom curriculum” and “please list any topics you think should be added to the Surgery Core Clerkship Flipped Classroom curriculum.” Qualitative responses were imported and coded in Dedoose qualitative analysis software (version 5.2.1., 2015, SocioCultural Research Consultants, Los Angeles, CA) for ease of data management. Data analysis. For the quantitative survey data, the number of respondents and percentages are reported for each Likert scale response. Means and standard deviations were calculated for effectiveness rankings of video format and compared statistically by one-way analysis of variance (ANOVA) with P values < .05 considered statistically significant. Open-ended responses were analyzed qualitatively using the constant comparative method in a team-based, analytic approach.14 Two study investigators identified major themes and subthemes in an iterative fashion, and a
ARTICLE IN PRESS 4 Liebert et al
Surgery j 2016
Table I. Student ratings of surgery clerkship flipped classroom curricular components Component
n
Poor % (no.)
Fair % (no.)
Average % (no.)
Excellent % (no.)
Outstanding % (no.)
Overall General component Online videos In-class clinical reasoning cases In-class simulation-based skill sessions Online pre-session practice questions Online post-session practice questions Review of pre-session questions in class Specific simulation sessions Suturing and knot tying Chest tube and central line placement US-guided breast biopsy and cyst aspiration FLS and endoscopy VR Mannequin-based trauma team simulations Laparoscopic cholecystectomy model SICKO surgical decision-making game* Stapled bowel anastomosis
76
0 (0)
0 (0)
10.5 (8)
51 (39)
38 (29)
77 77 77 76 77 77
0 1.3 0 1.3 1.3 0
(0) (1) (0) (1) (1) (0)
2.6 2.6 0 0 0 0
(2) (2) (0) (0) (0) (0)
10.4 13.0 2.6 25.0 23.4 10.4
(8) (10) (2) (19) (18) (8)
60 53 49 47 48 43
(46) (41) (38) (36) (37) (33)
27 30 48 26 27 47
(21) (23) (37) (20) (21) (36)
77 77 77 77 76 77 37 76
0 0 1.3 0 0 0 2.7 0
(0) (0) (1) (0) (0) (0) (1) (0)
0 0 2.6 0 0 0 2.7 0
(0) (0) (2) (0) (0) (0) (1) (0)
6.5 7.8 20.8 5.2 1.3 3.9 13.5 7.9
(5) (6) (16) (4) (1) (3) (5) (6)
48 47 39 39 34 39 46 43
(37) (36) (30) (30) (26) (30) (17) (33)
46 46 36 56 65 57 35 49
(35) (35) (28) (43) (49) (44) (13) (37)
*This simulation modality was included in 6 of the 12 months of the flipped classroom curriculum, representing n = 37 students. Due to rounding, percentages may not add up to 100%. US, Ultrasound; FLS, Fundamentals of Laparoscopic Surgery; VR, virtual reality; SICKO, Surgical Improvement of Clinical Knowledge Ops.
consensus-coding framework was established. The units of analysis were sentences. Both manifest and latent coding were used in an inductive approach.15 Final coding consensus was reached by discussion between the investigators (C.L. and L.M.). The data were then coded independently by the 2 investigators (C.L. and L.M.) using this consensus framework. Inter-rater reliability was calculated by Cohen’s kappa for all subthemes using Dedoose. Both reviewers were education fellows enrolled in a master’s level program in health professions education at the time of the study with prior training in qualitative methods and qualitative data analysis. A faculty investigator with experience in medical education research and curriculum design (D.L.) performed a peer audit of the narrative responses and coding framework. Representative example quotes of each subtheme were identified from the data. The number and percentage of student responses that exhibited each subtheme were calculated. RESULTS Quantitative results. A total of 77 students completed the online end-of-clerkship curriculum evaluation survey (87% response rate). Results of student ratings of the overall curricular model, general curricular components, and specific simulation-based sessions are shown in Table I. Overall, 90% of students rated the overall
curriculum excellent or outstanding, with none rating the curriculum poor or fair. Similar findings were noted for the curricular components, with the majority rating the online videos (87%), clinical reasoning cases (83%), in-class simulation-based skill sessions (98%), pre-session practice questions (74%), postsession practice questions (75%), and review of pre-session questions (90%) as excellent or outstanding. The component that was received most favorably by the students was the simulation-based skill sessions. Of these, the highest rated simulations were the FLS/EndoscopyVR, mannequin-based trauma team simulations, and the low-cost laparoscopic cholecystectomy model. Most students (77%) found the online pre- and postsession testing platform (Qualtrics software, Qualtrics, Provo, Utah) very easy to use, with the remaining students reporting it somewhat easy to use (22%) and 1 student (1%) reporting it somewhat difficult to use. Similar user feedback was given regarding the online Open edX video platform, with 79% reporting it very easy to use, 17% somewhat easy to use, and 4% somewhat difficult to use. Students rated the effectiveness of the Khan Academy-style video format the greatest compared with the hybrid of PowerPoint slides with digital inking format or traditional PowerPoint slides format (1.6 ± 0.8 vs 2.7 ± 0.5; 2.7; P < .001).
ARTICLE IN PRESS Liebert et al 5
Surgery Volume j, Number j
Table II. Student perceptions of overall surgery clerkship flipped classroom curricular model Strongly disagree % (no.)
Disagree % (no.)
Neither agree or disagree % (no.)
Agree % (no.)
Strongly agree % (no.)
5 (4)
26 (20)
69 (53)
Statement
n
The flipped classroom model for the surgery clerkship should be continued. The flipped classroom model should be applied to other core clerkships. The flipped classroom model allows students to spend more time with their clinical team. The flipped classroom model helped me feel prepared for the NBME shelf exam. The flipped classroom model provided adequate faculty exposure and interaction.
77
0 (0)
0 (0)
77
0 (0)
2.6 (2)
13 (10)
29 (22)
56 (43)
77
0 (0)
3.9 (3)
20 (15)
36 (28)
40 (31)
77
1.3 (1)
6.5 (5)
16 (12)
42 (32)
35 (27)
77
0 (0)
2.6 (2)
10 (8)
48 (37)
39 (30)
Due to rounding, percentages may not add up to 100%. NBME, National Board of Medical Examiners.
Students’ perceptions of the overall flipped classroom curricular model are shown in Table II. The vast majority of students reported that the flipped classroom model for the surgery core clerkship should be continued (95%), with the remaining students (5%) neutral in opinion. Students also reported that the flipped classroom model allowed them to spend more time with their clinical team (77%), provided adequate faculty exposure and interaction (87%), and should be applied to other core clerkships (84%). Qualitative results. Of the 77 students who completed the online, end-of-clerkship curriculum evaluation, 91% (n = 70) submitted narrative responses regarding their overall impressions of the flipped classroom curricular model. The majority of the responses were positive (96%); the remaining students (n = 3) expressed neutral impressions. No negative narrative responses were received. Qualitative analysis identified 4 major themes with 21 subthemes (Table III). The major themes identified were general positive feedback, educational content, learning environment, and specific benefits to students. Inter-rater reliability was measured with Cohen’s kappa, with a pooled kappa of 0.76 for all subthemes. Eight subthemes had an inter-rater reliability of 1.00. Other surgical topics suggested by students to be added to the flipped classroom curriculum for the surgery core clerkship included orthopedics (n = 17), vascular and pediatric (n = 7 each), cardiothoracic (n = 6), urology (n = 5), and otolaryngology, endocrine, and plastics (n # 4). Each major theme is explored in further detail below.
Representative narrative responses by the medical students are shown in the Addendum (online only version). General positive feedback. Students expressed strong positive feedback for the surgery core clerkship flipped classroom approach overall and for its specific components, including online videos, practice questions, clinical reasoning cases, simulation-based skills sessions, and the teaching faculty/staff. Multiple students also commented on the potential for this flipped classroom curricular model to be applied to other core clerkships. Educational content. Students valued several specific characteristics of the educational content, including succinct videos, high-yield material, practical and relevant topics, and the use of multiple modalities reinforcing knowledge. Learning environment. Students commented on the unique learning environment of the flipped classroom model, specifically its ability to create an interactive, engaging, and innovative environment that provided protected educational time and fostered self-directed learning and accountability in prework. Specific benefits to students. Various perceived benefits to students were identified based on the student responses. Reported benefits included preparation for the clinical rotation, preparation for the surgery shelf exam of the National Board of Medical Examiners (NBME), socialization with peers, and interaction with teaching faculty. Students additionally reported that the innovative, flipped classroom curricular model allowed for decreased class time compared with the traditional
ARTICLE IN PRESS 6 Liebert et al
Surgery j 2016
Table III. Qualitative analysis of students’ overall impressions of surgery clerkship flipped classroom Themes and subthemes General positive feedback Overall curriculum
Theme (# of responses) 30
Simulation-based skills sessions Clinical reasoning cases
8
Teaching faculty/staff
7
Online videos Practice questions Educational content Succinct videos High-yield material Multiple modalities
Practical and relevant topics Learning environment Interactive Engaging
9
6 2 3 4 3
1 5 5
Accountability
3
Self-directed learning
3
Protected educational time
4
Innovative Specific benefits to students Preparation for clinical rotation
2 5
Preparation for NBME shelf Reduced class time
3 3
Socialization with peers Interaction with faculty
1 1
Example quote “Absolutely out-of-the-box thinking with regards to clerkship education, wish other clerkships would follow example.” “The surgical skills sessions were [the] most valuable part of the training.” “I also love the clinical reasoning model that we used—case-based learning, and appreciated the chance to brainstorm differentials like you would for a real ED consult or clinic patient.” “Fantastic experience overall with great attendings and residents who care about teaching.” “Love the videos—especially the Khan Academy-style ones.” “. learned a lot going over questions .” “. videos were all really succinct, effective, and engaging.” “Reduced total class time, but the learning was much more high yield.” “Getting the information presented to me in multiple different ways (videos, questions, lectures, cases) was helpful for reinforcing what I was learning.” “.relevant to what we needed to know for the wards and the shelf.” “Really enjoyed how interactive the sessions were.” “I thought it was fun, engaging, and I learned a lot compared with other didactic sessions in other clerkships.” “I liked watching the videos and doing pre-questions because this held me accountable to the information and made me feel prepared and ready to learn when I came into class.” “It allowed us to learn the material at home at our own pace, which greatly helped me ‘get’ the learning points since I had time to look things up while watching the videos.” “. beneficial because we had scheduled time to be in didactics that were predictable for both us as students and our team.” “Innovative .” “.getting to use the EGD simulator, doing the choles, bowel anastomosis!!, running through a primary/secondary for a trauma, doing ultrasound guided activities - made me feel more confident when I was doing these activities with real patients or observing my team carry out these procedures.” “Great NBME prep.” “I appreciated that my time wasn’t wasted with boring lectures during the day that I was missing cases to attend.” “You get to see classmates.” “I also really appreciated meeting the faculty/attendings via the videos and then in person.”
n = 70; 91% of students who completed the survey submitted narrative responses. ED, Emergency Department; EGD, esophagogastroduodenoscopy; NBME, National Board of Medical Examiners.
lecture format, which translated to more time in the operating room with their clinical team. DISCUSSION This mixed-methods study explored medical students’ perceptions of a simulation-based, flipped classroom curricular approach for the surgery
core clerkship, with complementary quantitative and qualitative results. The findings suggest that the flipped classroom curricular model in the surgery core clerkship is well received by students, with the majority rating the overall curriculum and each of its subcomponents as excellent or outstanding. Qualitative analysis of narrative
ARTICLE IN PRESS Surgery Volume j, Number j
responses suggested that the vast majority of students expressed positive overall impressions, and no students expressed negative perceptions. Although all components of the curricular structure, including videos, pre- and post-test questions, clinical reasoning cases, and simulation-based skills sessions, were valued, the simulation-based skill sessions were the best-received subcomponents. An overwhelming majority (95%) reported the flipped classroom curricular model should be continued in the surgery core clerkship and applied to core clerkships in other disciplines. The qualitative thematic analysis of narrative responses revealed 4 major themes, including general positive feedback, educational content, learning environment, and specific benefits to students. The students noted that the combination of multiple complimentary teaching modalities reinforcing knowledge was a particular strength. Valued aspects of the educational content included the succinct videos, high-yield material, multiple modalities, and practical and relevant topics. Students particularly favored videos designed with a Khan Academy11 style of production <10 minutes in duration. Students reported that the flipped classroom fostered an engaging, interactive, and innovative learning environment that encouraged self-directed learning and accountability. Notably, no students complained of the required prework; in contrast, the opportunity for self-paced, independent learning at home was valued and suggests that students may desire this mode of content delivery when reinforced by active learning during class sessions. Students also identified specific benefits of the curriculum, including preparation for clinical rotations, preparation for the NBME shelf exam, decreased class time, socialization with peers, and interaction with faculty. Although one potential criticism of this flipped classroom curricular model is that students interact with fewer faculty members compared with the traditional didactic curriculum, the results of this study indicate that the majority of students perceived adequate faculty exposure and interaction. Although a smaller number of faculty members taught during the in-class sessions compared with a traditional didactic curriculum, it is possible that the level of student interaction and engagement with faculty was more intensive and meaningful in the flipped classroom. This flipped classroom curricular model could be adapted easily for surgery clerkships at other institutions or for core clerkships in other specialties. Based on our experiences and study
Liebert et al 7
Table IV. Recommended 10 best practices Consistent protected educational time Standardized overall format Topic-based clinical modules Multiple instructional methods Succinct, high-quality video prework Accountability in completion of prework Use of practice questions Case-based learning Simulation-based skills training Committed staff and faculty leads
findings, we propose best practices for implementing a flipped classroom for the core clerkship environment (Table IV). We suggest organization of the curriculum into separate topic-based modules with a repeating schedule and format, because students appreciated consistent, protected educational time each week using a standard format. We recommend inclusion of multiple instructional methods, such as practice questions, case-based clinical reasoning, and simulation-based skills sessions, to allow for knowledge application and consolidation of learning. Video prework was well received by students and should use succinct videos, such as those produced using a Khan Academy11 format, when possible. Efforts should be taken to maintain an interactive and engaging learning environment where students are held accountable for completing prework prior to the session. Multiple-choice NBME-style practice questions should be incorporated into the curriculum in the form of pre- and post-tests or practice questions, because students saw preparation for the NBME shelf exam as a specific benefit of a simulation-based curriculum. Based on student feedback, we plan to expand the content to include surgical subspecialty topics such as orthopedics, vascular surgery, cardiothoracic surgery, pediatric surgery, and urology, among others. Additional planned changes in the curriculum include replacement of traditional PowerPoint video content with Khan Academy-style videos and periodic revision of practice questions and clinical reasoning cases. It is important to highlight that the strength of a flipped classroom curriculum depends on the combination of high-quality succinct videos and relevant curricular content with effective instructional methods to foster active learning. Our study has several limitations, including that it was conducted at a single academic institution, which may limit its generalizability. The
ARTICLE IN PRESS 8 Liebert et al
curriculum created and implemented by dedicated staff at the Goodman Surgical Education Center was time-intensive, which may limit its generalizability to institutions with a small teaching staff or limited access to a simulation center. A lead faculty and surgical education fellow were present at each session to provide format consistency, continuity, and quality assurance, which may not be feasible for all institutions. Students completed the online survey 3 days prior to the NBME shelf exam, which limited their ability to determine whether the curriculum prepared them for the exam. This study demonstrated positive student ratings and perceptions of a flipped classroom curriculum for the surgery clerkship. Still, additional studies are needed to compare the effectiveness of a flipped classroom curriculum with more traditional curricula, with an eye toward knowledge acquisition and application and measurable objective outcomes. SUPPLEMENTARY DATA Supplementary data related to this article can be found online at http://dx.doi.org/10.1016/j.surg.2016.03.034. REFERENCES 1. Prober CG, Khan S. Medical education reimagined: A call to action. Acad Med 2013;88:1407-10. 2. Jensen JL, Kummer TA, Godoy PD. Improvements from a flipped classroom may simply be the fruits of active learning. CBE Life Sci Educ 2015;14:ar5. 3. Simpson V, Richards E. Flipping the classroom to teach population health: Increasing the relevance. Nurse Educ Pract 2015;15:162-7.
Surgery j 2016
4. Ratta CB. Flipping the classroom with team-based learning in undergraduate nursing education. Nurse Educ 2015;40: 71-4. 5. Schlairet MC, Green R, Benton MJ. The flipped classroom: Strategies for an undergraduate nursing course. Nurse Educ 2014;39:321-5. 6. Ferreri SP, O’Connor SK. Redesign of a large lecture course into a small-group learning course. Am J Pharm Educ 2013; 77:13. 7. Young TP, Bailey CJ, Guptill M, Thorp AW, Thomas TL. The flipped classroom: A modality for mixed asynchronous and synchronous learning in a residency program. West J Emerg Med 2014;15:938-44. 8. Vincent DS. Out of the wilderness: Flipping the classroom to advance scholarship in an internal medicine residency program. Hawaii J Med Public Health 2014;73:2-3. 9. Belfi LM, Bartolotta RJ, Giambrone AE, Davi C, Min RJ. “Flipping” the introductory clerkship: Impact on medical student performance and perceptions. Acad Radiol 2015; 22:794-801. 10. Morgan H, Marzano D, Lanham M, Stein T, Curran D, Hammoud M. Preparing medical students for obstetrics and gynecology milestone level one: A description of a pilot curriculum. Med Educ Online 2014;19:25746. 11. Khan Academy: Health and Medicine. Available from: https://www.khanacademy.org/science/health-and-medicine. 12. Ault MJ, Rosen BT, Ault B. The use of tissue models for vascular access training: Phase I of the Procedural Patient Safety Initiative. J Gen Intern Med 2006;21(5):514-7. 13. Lin DT, Park J, Liebert CA, Lau JN. Validity evidence for Surgical Improvement of Clinical Knowledge Ops: A novel gaming platform to assess surgical decision making. Am J Surg 2015;209:79-85. 14. Guest G, MacQueen KM. Handbook for team-based qualitative research. Washington, D.C.: Rowman & Littlefield Pub Incorporated; 2008. 15. Miles MB, Michael Huberman A, Saldana J. Qualitative data analysis: A methods sourcebook. 3rd ed. Thousand Oaks, California: SAGE Publications, Inc; 2014.