- Email: [email protected]
The Flipped Classroom in Emergency Medicine Using Online Videos with Interpolated Questions
The Journal of Emergency Medicine, Vol. -, No. -, pp. 1–9, 2016 Ó 2016 Elsevier Inc. All rights reserved. 0736-4679/$ - see front matter
http://dx.doi.org/10.1016/j.jemermed.2016.05.033
Education THE FLIPPED CLASSROOM IN EMERGENCY MEDICINE USING ONLINE VIDEOS WITH INTERPOLATED QUESTIONS Emily Rose, MD, FAAP, FAAEM, FACEP,* Ilene Claudius, MD,* Ramin Tabatabai, MD,* Liza Kearl, MD,* Solomon Behar, MD,* and Paul Jhun, MD† *Department of Emergency Medicine, Keck School of Medicine of the University of Southern California, Los Angeles County + USC Medical Center, Los Angeles, California and †Department of Emergency Medicine, University of California San Francisco, San Francisco General Hospital, San Francisco, California Reprint Address: Emily Rose, MD, FAAP, FAAEM, FACEP, Department of Emergency Medicine, Keck School of Medicine of the University of Southern California, Los Angeles County + USC Medical Center, 1200 North State Street, Los Angeles, CA 90033
, Abstract—Background: Utilizing the flipped classroom is an opportunity for a more engaged classroom session. This educational approach is theorized to improve learner engagement and retention and allows for more complex learning during class. No studies to date have been conducted in the postgraduate medical education setting investigating the effects of interactive, interpolated questions in preclassroom online video material. Objectives: We created a flipped classroom for core pediatric emergency medicine (PEM) topics using recorded online video lectures for preclassroom material and interactive simulations for the in-classroom session. Methods: Lectures were filmed and edited to include integrated questions on an online platform called Zaption. One-half of the residents viewed the lectures uninterrupted (Group A) and the remainder (Group B) viewed with integrated questions (2–6 per 5–15-min segment). Residents were expected to view the lectures prior to in-class time (total viewing time of approximately 2½ h). The 2½-h in-class session included four simulation and three procedure stations, with six PEM faculty available for higher-level management discussion throughout the stations. Total educational time of home preparation and in-class time was approximately 5 h. Results: Residents performed better on the posttest as compared to the pretest, and their satisfaction was high with this educational innovation. In 2014, performance on the posttest between the two groups was similar. However, in 2015, the group with integrated questions performed better on the posttest. Conclusion: An
online format combined with face-to-face interaction is an effective educational model for teaching core PEM topics. Ó 2016 Elsevier Inc. All rights reserved. , Keywords—flipped classroom; online videos; interpolated questions; pediatric emergency medicine education; engaged educational sessions; curriculum innovation
INTRODUCTION The concept of the flipped classroom is increasing in popularity, as evidence exists that it improves learner engagement and learning retention, affords more efficient face time, and allows more complex learning during class (1–7). Coined by two high school chemistry teachers in 2007, the term ‘‘flipped classroom’’ refers to an educational approach that reverses the traditional lecture and homework elements of a course (8). Students are given material in advance (such as text, online lecture, or podcast), allowing class time to be used for active learning exercises. Background The pediatric emergency medicine (PEM) faculty teach a modified 1-day advanced pediatric life support (APLS) course to the postgraduate year (PGY)-3 Emergency
RECEIVED: 30 April 2016; ACCEPTED: 10 May 2016 1
2
E. Rose et al.
medicine residents at Los Angeles County + University of Southern California Medical Center annually as part of the educational curriculum. This course is an overview of basic PEM care and is given to the PGY-3 class immediately prior to the start of their PGY-4 year. Historically, this course has consisted of 6 h of live, prewritten lectures on various topics, followed by a 2-h procedure laboratory. Anecdotally, residents and faculty reported this day to be long and somewhat arduous, with a time commitment disproportionate to potential learning benefit. The PEM faculty members were inspired to create a more engaging session that maximized educational impact in a more efficient manner. Educational Objectives 1. Condense core advanced pediatric life support (APLS)/PEM curriculum to brief online lectures for online access. 2. Utilize class time for efficient, interactive sessions. MATERIALS AND METHODS Curricular Design Four PEM physicians recorded multiple PEM topics for online video viewing. One physician lectured while two
physicians served as commentators during the lectures. These sessions were filmed and produced professionally by the Hippo Education studio. After extensive editing, 13 approximately 10-min online lectures highlighting 12 topics from the APLS course were uploaded to an online service called Zaption (www. zaption.com). Zaption allows forced interpolated questions integrated into the online lectures to reinforce concepts, and it also provides instant analytics including: number of views, average viewing time percent of video watched, and answers to interpolated questions (Table 1). Explanations to incorrect questions responses were given in real time. One half of the residents were randomized to view online lectures uninterrupted (Group A), whereas the remainder (Group B) viewed with integrated questions (2–6 per 5–15-min lecture, see Table 1). All residents were expected to view the lectures prior to an in-class session (total viewing time of approximately 2½ h). The in-class session was <2½ h in length and included four simulation stations and three procedure stations led by six PEM faculty for higher-level management discussion throughout the stations (Table 2 for class schedule). The University of Southern California’s institutional review board approved this study as exempt. Seventeen PGY-3 Emergency Medicine residents participated in a
Table 1. Zaption Analytics of Pediatric Emergency Medicine Lectures
Lecture (Group)
Video Length (min:sec)
Average Viewing Time (min:sec)
Integrated Questions (#)
Percent Correct (%)
Average Skips Backward
Average Skips Forward
Airway (A) Airway (B) Bronchiolitis* (A) Bronchiolitis (B) Croup* (A) Croup (B) Diabetic ketoacidosis (A) Diabetic ketoacidosis (B) Fever I* (A) Fever I (B) Fever II* (A) Fever II (B) Head trauma (A) Head trauma (B) Intubation (A) Intubation (B) Neonatal resuscitation (A) Neonatal resuscitation (B) NPPV (A) NPPV (B) PALS update (A) PALS update (B) Sepsis (A) Sepsis (B) Status epilepticus (A) Status epilepticus (B)
7:54 7:54 8:26 8:26 10:34 10:34 12:10 12:10 13:40 13:40 11:55 11:55 9:37 9:37 14:54 14:54 5:25 5:25 13:56 13:56 13:19 13:19 13:09 13:09 14:44 14:44
6:35 5:37 7:11 8:21 9:57 10:26 11:10 11:00 11:45 12:42 10:38 10:30 8:26 9:11 12:49 13:41 5:03 5:03 12:27 12:38 11:55 12:43 12:12 12:37 13:51 13:04
0 3 0 4 0 4 0 4 0 2 0 3 0 3 0 4 0 3 0 3 0 4 0 6 0 5
NA 60 NA 72 NA 72 NA 79 NA 94 NA 88 NA 96 NA 84 NA 82 NA 82 NA 88 NA 96 NA 79
0.8 0 1.6 0 1 0 0.7 0 1.6 0 1.4 0 0.4 0 1.6 0 0.6 0 1 0 2.8 0 1.4 0 1.1 0
0.6 0.5 1.9 1.3 0.4 0.1 1 0.9 0.7 1.8 0.3 0.1 1.6 0.2 2.7 1.8 0.6 0.2 3.9 0.5 1.4 0.7 1.2 0 1.6 1.7
NPPV = noninvasive positive pressure ventilation; PALS = pediatric advanced life support. Group A viewed the lectures uninterrupted and Group B viewed the lectures with integrated questions. * These groups have medical student viewing data combined with the resident views for a separate project.
The Flipped Classroom in Emergency Medicine
3
Table 2. Class Session Schedule Time 9:00 a.m. 9:15 a.m. 9:30 a.m. 9:45 a.m. 10:00 a.m. 10:15 a.m.
Group 1
Group 2
Simulation—Status Simulation—Drowning Epilepticus Resuscitation Simulation—Aspirated Simulation—Neonatal Foreign Body Shock Simulation—Drowning Simulation—Status Resuscitation Epilepticus Simulation—Neonatal Simulation—Aspirated Shock Foreign Body Advanced Airway Techniques Needle Cricothyrotomy Umbilical Lines
10:30 a.m. 10:45 a.m. 11:00–11:30 a.m.
Group 3
Group 4
Advanced Airway Techniques Needle Cricothyrotomy Umbilical Lines
Simulation—Status Epilepticus Simulation—Aspirated Foreign Body Simulation—Drowning Resuscitation Simulation—Neonatal Shock
Simulation—Drowning Resuscitation Simulation—Neonatal Shock Simulation—Status Epilepticus Simulation—Aspirated Foreign Body
Debriefing, Posttest and Survey
Two faculty members ran two simultaneous simulation stations. One faculty member headed each procedure station. Residents were given freedom to spend as much time as needed at each procedure station.
PEM course in June 2014, and another 17 in June 2015. The residents were randomly assigned to Group A (uninterrupted online videos) or Group B (videos with integrated questions). Lecture links to 13 approximately 10-min lectures were sent to residents via e-mail with instructions to view 1 week prior to the in-class session. Lecture topics included: pediatric advanced life support update, pediatric sepsis, pediatric fever (I & II), croup, bronchiolitis, pediatric diabetic ketoacidosis, pediatric airway anatomy, pediatric intubation, neonatal resuscita-
tion, status epilepticus, head trauma, and noninvasive positive pressure ventilation (Table 1). Viewers were able to play the lectures repeatedly, pause, and scroll forward or backward. In the group with integrated questions, the lecture would pause at predesigned intervals, with a multiple-choice question requiring the user to answer prior to restarting the lecture (Figures 1 and 2). The correct answer was immediately given after one attempt with an explanation. The lecture would continue once the answer was acknowledged.
Figure 1. Screen shot of the video on a computer screen with interpolated questions.
4
E. Rose et al.
Figure 2. Screen shot of view of one lecturer, two commentators, and slides.
Zaption analytics were collected (Figure 3), allowing educators to know which lectures were viewed, the time individuals spent viewing, the number of skips forward or backward, and the score for the integrated questions (see Table 1 for Zaption analytic results). Internet access was required, but the videos could be viewed on home computers, laptops, tablets, or smart phones. Two distinct 20-question multiple-choice tests (Tests A and B) were created by three board-certified PEM faculty. Questions were peer reviewed among the involved six PEM faculty. Test A served as a pretest, completed by the learner prior to participation in online video viewing and in-classroom activities. Sixteen residents completed Test A in 2014, and 17 completed Test A in 2015. The pretest was sent to the residents electronically. Upon test completion, online lectures were provided. Residents were instructed to view 100% of the lectures prior to the in-class session. The face-to-face class time totaled approximately 2½ h. Residents were randomly placed into four equal groups. Residents rotated through stations and participated in four simulation cases (status epilepticus, aspirated foreign body, drowning resuscitation, and neonatal shock) and three skills stations (umbilical lines, needle cricothyrotomy, and airway) (see Table 2 for inclass schedule). Immediately upon completion of the in-classroom activities, residents completed Test B and a survey (both on paper) at the end of the session. Residents were given a
postevent survey evaluating their experience with the educational event. They were asked their preference of educational interaction, as well as asked to check a box next to the lecture topic if they viewed the lecture. See Appendices 1 and 2 for sample questions and survey. The pre- and posttests were evaluated and results were analyzed with a Mann-Whitney U test using median test score values. RESULTS In both years, all residents demonstrated improved performance on the posttest (Table 3). In 2014, Test A averaged 55% correct (42–63%). Test B averaged 88% correct (58–100%). In 2015, Test A averaged 71% correct (54–88%). Test B averaged 90% correct (58–96%). In 2014, there was no statistical difference between posttest scores in those that viewed lectures with integrated questions and those without questions (p value 0.961). However, in 2015, there was a difference between the groups that viewed the lectures with questions (87% correct [79-96%]) vs. those that viewed plain lectures without questions (75% correct [58–83%], p value 0.010). The residents were asked to respond on a 1–7 Likert scale to the statement, ‘‘I prefer online lectures and using protected education time for interactive educational activities rather than live lectures.’’ In 2014 the response averaged 6.2, and in 2015 the response averaged 5.9
The Flipped Classroom in Emergency Medicine
5
Figure 3. Screen shot of the Administrator’s view of Zaption analytics.
(Table 4). All residents polled preferred online lectures to live lectures, and the majority (82% in 2014 and 70% in 2015) preferred interactive questions within the online lectures to increase learning retention and interaction.
In 2014, 11 of 17 residents who responded reported viewing all 13 online lectures in an anonymous postevent survey (see Table 5). There was a technical problem rendering us unable to document actual views recorded for the 2014 group.
Table 3. Test Score Change, Pre and Post Intervention by Year (N = 34) Group A Pre Year 2014 Median Range Year 2015 Median Range
55 (55–55) 71 (71–71)
Post (n = 8) 88 (79–100) (n = 9) 75 (58–92)
Group B Change
Pre
33 (24–45)
55 (55–55)
4 ( 13 to 21)
71 (71–71)
* Mann-Whitney U test assessed median change-score between groups by year.
Post (n = 9) 92 (58–96) (n = 8) 85.5 (79–96)
Change
p-Value 0.961*
37 (3–41) 0.010* 14.5 (8–25)
6
E. Rose et al.
Table 4. Postevent Survey Responses Based on a Likert Scale of 1–7 Prefer Online Lectures (1–7)
# Responses
Satisfied with Question Prompts (1–7)
# Responses
0 0 1 1 4 3 7
1 2 3 4 5 6 7
0 0 0 0 1 3 5
0 0 1 1 4 3 7
1 2 3 4 5 6 7
0 0 0 0 2 6 3
2014 1 2 3 4 5 6 7 2015 1 2 3 4 5 6 7
In our 2015 anonymous postevent survey, 14 residents stated that they viewed all of the lectures. However, upon review of the Zaption database, only one resident viewed all 13 lecture links. Three residents viewed 12/13, three viewed 11/13, three viewed 10/13, three viewed 9/13, and three viewed 8/13. Zaption recorded if the lectures were viewed, length of viewing, the grade of the integrated questions for group B, and the number of skips forward or backward during the views. See Table 4 for detailed results. Table 5. PEM Lecture Online Video Completion Rates
Lecture Airway Bronchiolitis Croup Diabetic ketoacidosis Fever I Fever II Head trauma Intubation Neonatal resuscitation Noninvasive positive pressure ventilation PALS update Sepsis Status epilepticus
2014 Reported Views (17 Surveys Received)
2015 Reported Views (Only 16 Surveys Received)
2015 Actual Views (Per Zaption Viewing Log)
13 14 17 15
16 16 16 16
16 13 15 16
15 15 15 15 15
15 15 16 16 16
15 14 17 13 15
14
14
13
14 15 15
16 15 14
16 14 14
PEM = pediatric emergency medicine; PALS = pediatric advanced life support.
DISCUSSION The concept of the flipped classroom is gaining traction in medical education (9–13). Its success is likely multifactorial, and the method may be particularly appealing to adult learners. Malcolm Knowles’ adult educational theory emphasizes that adult learners need independence, freedom, and flexibility of education to meet their needs. The effectiveness of the flipped classroom has been demonstrated. The benefit of this method is that it frees up class time for more engaging, higher-level discussions. In medical education, the benefit is clear. Clinically relevant simulation and case discussions where interest and curiosity are piqued become better learning opportunities and the information is better retained (or, as Prober and Khan say, the material becomes more ‘‘sticky’’) (10). However, certain didactic material must be covered in residency curriculum. Online, prelecture didactic videos can cover core material and appeal to adult learners in its flexibility and pace of viewing. Online lectures have been shown to be as effective, if not more effective, than in-person lectures (14). The Department of Education published a meta-analysis that concluded, ‘‘on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction, with larger effects if the online learning was combined with face-to-face instruction’’ (15). The effectiveness of the flipped classroom and online didactic videos were presumed in this study and not directly evaluated. Two methods of online lectures (uninterrupted and with integrated questions) were evaluated, and viewing data were gathered from Zaption analytics. There was a control group for the interposed questions, but no control group for the flipped classroom concept overall, due to the clear ethical concerns of offering half of a residency class a potentially more impactful and efficient intervention. Therefore, learning retention as a direct result of the flipped classroom concept could not be adequately assessed. Overall, residents liked this intervention, and postevent scores improved in both groups. This may be partially due to the fact that they simply appreciated the time and investment of faculty to make their educational experience more efficient and of higher impact. They preferred the format of integrated questions within the lectures, and many residents who viewed lectures without questions stated that they believed they would prefer to have integrated questions when viewing lectures. Integrating questions within lectures increases the learner’s engagement with the material and highlights points of emphasis. Platforms such as Zaption
The Flipped Classroom in Emergency Medicine
are an effective resource to add integrated questions to online video lectures. Additionally, instructors are able to monitor compliance and evaluate how much time learners spent viewing the lectures, as well as assess the pace of the material for the learners. The ability to interpolate questions allows the instructor to pause and emphasize important take-home points while correcting learner error by immediately providing the correct answer and explanation to the question. In Zaption, the integrated question pops up in the viewer screen and an answer must be clicked to continue the video, but the correct answer is immediately given regardless of a(n) correct/incorrect response. This format is informative but not too cumbersome for viewers. It remains to be seen if integrated questions improve learning retention, as we have conflicting test results between the 2 years. Residency is challenging and there are competing time interests for trainees. It is often difficult for residents to complete assigned tasks ahead of time. Emergency Medicine programs should innovate educational programs to meet the needs of their trainees but may need a venue in which compliance may be monitored. Online platforms such as Zaption are tools that can be used to enhance the efficacy of flipped classroom preclassroom educational material. Limitations There are several limitations to this small descriptive educational innovation study. The small sample sizes and conflicting results between the years in this pilot trial make it difficult to reach conclusions on the effectiveness of integrated questions in online lectures. The groups were randomly selected and presumed to have similar pretest scores. Perhaps a difference in group characteristics contributed more to the difference between the years rather than the format of plain lectures vs. lectures with interpolated questions. The residents took the pretest on the computer and the posttest on paper. This difference in methods could account for a difference in scores. Additionally, three tests were created for this course. The same pretest was given both years, but the posttest was different in 2014 and 2015, which could also account for differences in scores.
7
CONCLUSION An online format combined with face-to-face interaction is an effective educational model for teaching core PEM topics in the emergency department. Interpolated questions within online videos were preferred and may increase the learner’s engagement with the material. Test scores improved and resident satisfaction was high with this educational innovation. However, completion rates may need to be monitored for successful implementation. REFERENCES 1. McDonald K, Smith CM. The flipped classroom for professional development: part I. Benefits and strategies. J Contin Educ Nurs 2013;44:437–8. 2. Critz CM, Knight D. Using the flipped classroom in graduate nursing education. Nurse Educ 2013;38:210–3. 3. McLaughlin JE, Griffin LM, Esserman DA, et al. Pharmacy student engagement, performance, and perception in a flipped satellite classroom. Am J Pharm Educ 2013;77:196. 4. Gilboy MB, Heinerichs S, Pazzaglia G. Enhancing student engagement using the flipped classroom. J Nutr Educ Behav 2015;47:109–14. 5. Missildine K, Fountain R, Summers L, Gosselin K. Flipping the classroom to improve student performance and satisfaction. J Nurs Educ 2013;52:597–9. 6. Moffett J. Twelve tips for ‘‘flipping’’ the classroom. Med Teach 2015;37:331–6. 7. Smith CM, McDonald K. The flipped classroom for professional development: part II. Making podcasts and videos. J Contin Educ Nurs 2013;44:486–7. 8. Bergmann J, Sams A. Flip your classroom. Reach every student in every class every day. Arlington, VA: International Society for Technology in Education; 2012. 9. Prober CG, Heath C. Lecture halls without lectures—a proposal for medical education. N Engl J Med 2012;366:1657–9. 10. Prober CG, Khan S. Medical education reimagined: a call to action. Acad Med 2013;88:1407–10. 11. Mehta NB, Hull AL, Young JB, Stoller JK. Just imagine: new paradigms for medical education. Acad Med 2013;88:1418–23. 12. Galway LP, Corbett KK, Takaro TK, Tairyan K, Frank E. A novel integration of online and flipped classroom instructional models in public health higher education. BMC Med Educ 2014;14:181. 13. McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Acad Med 2014;89:236–43. 14. Platz E, Liteplo A, Hurwitz S, Hwang J. Are live instructors replaceable? Computer vs. classroom lectures for EFAST training. J Emerg Med 2011;40:534–8. 15. Evaluation of evidence-based practices in online learning: a metaanalysis and review of online learning studies. Washington, DC: Department of Education, Office of Planning, Evaluation, and Policy Development; 2010. Available at: http://www2.ed.gov/ rschstat/eval/tech/evidence-based-practices/finalreport.pdf. Accessed October 20, 2015.
8
E. Rose et al.
ARTICLE SUMMARY 1. Why is this topic important? Emergency medicine residents must learn a tremendous amount of material in a condensed time period while maintaining a demanding clinical schedule. Education must be efficiently delivered in a high-yield manner, and increased engagement results in learning retention. 2. What does this study attempt to show? Online video lectures are an effective method of utilizing the flipped classroom. Interpolated questions within the videos are preferred and may increase learner engagement in the material. 3. What are the key findings? Online lectures provide flexibility to view at a convenient time and at a learner’s own pace. Traditional lectures are less flexible, efficient, and interactive. Novel educational methods such as online video lectures with/without interpolated questions may be used to increase knowledge retention, satisfaction, and engagement with the material. Residents may over-report the rate at which lectures were viewed, and it is important to monitor compliance. 4. How is patient care impacted? When education is delivered in a more efficient, interactive, and enjoyable method, emergency medicine residents are more engaged. A resident that actively participates in education results is a more informed clinician. A provider with a broad knowledge base is a more astute and effective clinician.
The Flipped Classroom in Emergency Medicine
Appendix A. Sample Test Questions.
8.e1
Appendix B. Postevent Anonymous Survey.
http://dx.doi.org/10.1016/j.jemermed.2016.05.033
Education THE FLIPPED CLASSROOM IN EMERGENCY MEDICINE USING ONLINE VIDEOS WITH INTERPOLATED QUESTIONS Emily Rose, MD, FAAP, FAAEM, FACEP,* Ilene Claudius, MD,* Ramin Tabatabai, MD,* Liza Kearl, MD,* Solomon Behar, MD,* and Paul Jhun, MD† *Department of Emergency Medicine, Keck School of Medicine of the University of Southern California, Los Angeles County + USC Medical Center, Los Angeles, California and †Department of Emergency Medicine, University of California San Francisco, San Francisco General Hospital, San Francisco, California Reprint Address: Emily Rose, MD, FAAP, FAAEM, FACEP, Department of Emergency Medicine, Keck School of Medicine of the University of Southern California, Los Angeles County + USC Medical Center, 1200 North State Street, Los Angeles, CA 90033
, Abstract—Background: Utilizing the flipped classroom is an opportunity for a more engaged classroom session. This educational approach is theorized to improve learner engagement and retention and allows for more complex learning during class. No studies to date have been conducted in the postgraduate medical education setting investigating the effects of interactive, interpolated questions in preclassroom online video material. Objectives: We created a flipped classroom for core pediatric emergency medicine (PEM) topics using recorded online video lectures for preclassroom material and interactive simulations for the in-classroom session. Methods: Lectures were filmed and edited to include integrated questions on an online platform called Zaption. One-half of the residents viewed the lectures uninterrupted (Group A) and the remainder (Group B) viewed with integrated questions (2–6 per 5–15-min segment). Residents were expected to view the lectures prior to in-class time (total viewing time of approximately 2½ h). The 2½-h in-class session included four simulation and three procedure stations, with six PEM faculty available for higher-level management discussion throughout the stations. Total educational time of home preparation and in-class time was approximately 5 h. Results: Residents performed better on the posttest as compared to the pretest, and their satisfaction was high with this educational innovation. In 2014, performance on the posttest between the two groups was similar. However, in 2015, the group with integrated questions performed better on the posttest. Conclusion: An
online format combined with face-to-face interaction is an effective educational model for teaching core PEM topics. Ó 2016 Elsevier Inc. All rights reserved. , Keywords—flipped classroom; online videos; interpolated questions; pediatric emergency medicine education; engaged educational sessions; curriculum innovation
INTRODUCTION The concept of the flipped classroom is increasing in popularity, as evidence exists that it improves learner engagement and learning retention, affords more efficient face time, and allows more complex learning during class (1–7). Coined by two high school chemistry teachers in 2007, the term ‘‘flipped classroom’’ refers to an educational approach that reverses the traditional lecture and homework elements of a course (8). Students are given material in advance (such as text, online lecture, or podcast), allowing class time to be used for active learning exercises. Background The pediatric emergency medicine (PEM) faculty teach a modified 1-day advanced pediatric life support (APLS) course to the postgraduate year (PGY)-3 Emergency
RECEIVED: 30 April 2016; ACCEPTED: 10 May 2016 1
2
E. Rose et al.
medicine residents at Los Angeles County + University of Southern California Medical Center annually as part of the educational curriculum. This course is an overview of basic PEM care and is given to the PGY-3 class immediately prior to the start of their PGY-4 year. Historically, this course has consisted of 6 h of live, prewritten lectures on various topics, followed by a 2-h procedure laboratory. Anecdotally, residents and faculty reported this day to be long and somewhat arduous, with a time commitment disproportionate to potential learning benefit. The PEM faculty members were inspired to create a more engaging session that maximized educational impact in a more efficient manner. Educational Objectives 1. Condense core advanced pediatric life support (APLS)/PEM curriculum to brief online lectures for online access. 2. Utilize class time for efficient, interactive sessions. MATERIALS AND METHODS Curricular Design Four PEM physicians recorded multiple PEM topics for online video viewing. One physician lectured while two
physicians served as commentators during the lectures. These sessions were filmed and produced professionally by the Hippo Education studio. After extensive editing, 13 approximately 10-min online lectures highlighting 12 topics from the APLS course were uploaded to an online service called Zaption (www. zaption.com). Zaption allows forced interpolated questions integrated into the online lectures to reinforce concepts, and it also provides instant analytics including: number of views, average viewing time percent of video watched, and answers to interpolated questions (Table 1). Explanations to incorrect questions responses were given in real time. One half of the residents were randomized to view online lectures uninterrupted (Group A), whereas the remainder (Group B) viewed with integrated questions (2–6 per 5–15-min lecture, see Table 1). All residents were expected to view the lectures prior to an in-class session (total viewing time of approximately 2½ h). The in-class session was <2½ h in length and included four simulation stations and three procedure stations led by six PEM faculty for higher-level management discussion throughout the stations (Table 2 for class schedule). The University of Southern California’s institutional review board approved this study as exempt. Seventeen PGY-3 Emergency Medicine residents participated in a
Table 1. Zaption Analytics of Pediatric Emergency Medicine Lectures
Lecture (Group)
Video Length (min:sec)
Average Viewing Time (min:sec)
Integrated Questions (#)
Percent Correct (%)
Average Skips Backward
Average Skips Forward
Airway (A) Airway (B) Bronchiolitis* (A) Bronchiolitis (B) Croup* (A) Croup (B) Diabetic ketoacidosis (A) Diabetic ketoacidosis (B) Fever I* (A) Fever I (B) Fever II* (A) Fever II (B) Head trauma (A) Head trauma (B) Intubation (A) Intubation (B) Neonatal resuscitation (A) Neonatal resuscitation (B) NPPV (A) NPPV (B) PALS update (A) PALS update (B) Sepsis (A) Sepsis (B) Status epilepticus (A) Status epilepticus (B)
7:54 7:54 8:26 8:26 10:34 10:34 12:10 12:10 13:40 13:40 11:55 11:55 9:37 9:37 14:54 14:54 5:25 5:25 13:56 13:56 13:19 13:19 13:09 13:09 14:44 14:44
6:35 5:37 7:11 8:21 9:57 10:26 11:10 11:00 11:45 12:42 10:38 10:30 8:26 9:11 12:49 13:41 5:03 5:03 12:27 12:38 11:55 12:43 12:12 12:37 13:51 13:04
0 3 0 4 0 4 0 4 0 2 0 3 0 3 0 4 0 3 0 3 0 4 0 6 0 5
NA 60 NA 72 NA 72 NA 79 NA 94 NA 88 NA 96 NA 84 NA 82 NA 82 NA 88 NA 96 NA 79
0.8 0 1.6 0 1 0 0.7 0 1.6 0 1.4 0 0.4 0 1.6 0 0.6 0 1 0 2.8 0 1.4 0 1.1 0
0.6 0.5 1.9 1.3 0.4 0.1 1 0.9 0.7 1.8 0.3 0.1 1.6 0.2 2.7 1.8 0.6 0.2 3.9 0.5 1.4 0.7 1.2 0 1.6 1.7
NPPV = noninvasive positive pressure ventilation; PALS = pediatric advanced life support. Group A viewed the lectures uninterrupted and Group B viewed the lectures with integrated questions. * These groups have medical student viewing data combined with the resident views for a separate project.
The Flipped Classroom in Emergency Medicine
3
Table 2. Class Session Schedule Time 9:00 a.m. 9:15 a.m. 9:30 a.m. 9:45 a.m. 10:00 a.m. 10:15 a.m.
Group 1
Group 2
Simulation—Status Simulation—Drowning Epilepticus Resuscitation Simulation—Aspirated Simulation—Neonatal Foreign Body Shock Simulation—Drowning Simulation—Status Resuscitation Epilepticus Simulation—Neonatal Simulation—Aspirated Shock Foreign Body Advanced Airway Techniques Needle Cricothyrotomy Umbilical Lines
10:30 a.m. 10:45 a.m. 11:00–11:30 a.m.
Group 3
Group 4
Advanced Airway Techniques Needle Cricothyrotomy Umbilical Lines
Simulation—Status Epilepticus Simulation—Aspirated Foreign Body Simulation—Drowning Resuscitation Simulation—Neonatal Shock
Simulation—Drowning Resuscitation Simulation—Neonatal Shock Simulation—Status Epilepticus Simulation—Aspirated Foreign Body
Debriefing, Posttest and Survey
Two faculty members ran two simultaneous simulation stations. One faculty member headed each procedure station. Residents were given freedom to spend as much time as needed at each procedure station.
PEM course in June 2014, and another 17 in June 2015. The residents were randomly assigned to Group A (uninterrupted online videos) or Group B (videos with integrated questions). Lecture links to 13 approximately 10-min lectures were sent to residents via e-mail with instructions to view 1 week prior to the in-class session. Lecture topics included: pediatric advanced life support update, pediatric sepsis, pediatric fever (I & II), croup, bronchiolitis, pediatric diabetic ketoacidosis, pediatric airway anatomy, pediatric intubation, neonatal resuscita-
tion, status epilepticus, head trauma, and noninvasive positive pressure ventilation (Table 1). Viewers were able to play the lectures repeatedly, pause, and scroll forward or backward. In the group with integrated questions, the lecture would pause at predesigned intervals, with a multiple-choice question requiring the user to answer prior to restarting the lecture (Figures 1 and 2). The correct answer was immediately given after one attempt with an explanation. The lecture would continue once the answer was acknowledged.
Figure 1. Screen shot of the video on a computer screen with interpolated questions.
4
E. Rose et al.
Figure 2. Screen shot of view of one lecturer, two commentators, and slides.
Zaption analytics were collected (Figure 3), allowing educators to know which lectures were viewed, the time individuals spent viewing, the number of skips forward or backward, and the score for the integrated questions (see Table 1 for Zaption analytic results). Internet access was required, but the videos could be viewed on home computers, laptops, tablets, or smart phones. Two distinct 20-question multiple-choice tests (Tests A and B) were created by three board-certified PEM faculty. Questions were peer reviewed among the involved six PEM faculty. Test A served as a pretest, completed by the learner prior to participation in online video viewing and in-classroom activities. Sixteen residents completed Test A in 2014, and 17 completed Test A in 2015. The pretest was sent to the residents electronically. Upon test completion, online lectures were provided. Residents were instructed to view 100% of the lectures prior to the in-class session. The face-to-face class time totaled approximately 2½ h. Residents were randomly placed into four equal groups. Residents rotated through stations and participated in four simulation cases (status epilepticus, aspirated foreign body, drowning resuscitation, and neonatal shock) and three skills stations (umbilical lines, needle cricothyrotomy, and airway) (see Table 2 for inclass schedule). Immediately upon completion of the in-classroom activities, residents completed Test B and a survey (both on paper) at the end of the session. Residents were given a
postevent survey evaluating their experience with the educational event. They were asked their preference of educational interaction, as well as asked to check a box next to the lecture topic if they viewed the lecture. See Appendices 1 and 2 for sample questions and survey. The pre- and posttests were evaluated and results were analyzed with a Mann-Whitney U test using median test score values. RESULTS In both years, all residents demonstrated improved performance on the posttest (Table 3). In 2014, Test A averaged 55% correct (42–63%). Test B averaged 88% correct (58–100%). In 2015, Test A averaged 71% correct (54–88%). Test B averaged 90% correct (58–96%). In 2014, there was no statistical difference between posttest scores in those that viewed lectures with integrated questions and those without questions (p value 0.961). However, in 2015, there was a difference between the groups that viewed the lectures with questions (87% correct [79-96%]) vs. those that viewed plain lectures without questions (75% correct [58–83%], p value 0.010). The residents were asked to respond on a 1–7 Likert scale to the statement, ‘‘I prefer online lectures and using protected education time for interactive educational activities rather than live lectures.’’ In 2014 the response averaged 6.2, and in 2015 the response averaged 5.9
The Flipped Classroom in Emergency Medicine
5
Figure 3. Screen shot of the Administrator’s view of Zaption analytics.
(Table 4). All residents polled preferred online lectures to live lectures, and the majority (82% in 2014 and 70% in 2015) preferred interactive questions within the online lectures to increase learning retention and interaction.
In 2014, 11 of 17 residents who responded reported viewing all 13 online lectures in an anonymous postevent survey (see Table 5). There was a technical problem rendering us unable to document actual views recorded for the 2014 group.
Table 3. Test Score Change, Pre and Post Intervention by Year (N = 34) Group A Pre Year 2014 Median Range Year 2015 Median Range
55 (55–55) 71 (71–71)
Post (n = 8) 88 (79–100) (n = 9) 75 (58–92)
Group B Change
Pre
33 (24–45)
55 (55–55)
4 ( 13 to 21)
71 (71–71)
* Mann-Whitney U test assessed median change-score between groups by year.
Post (n = 9) 92 (58–96) (n = 8) 85.5 (79–96)
Change
p-Value 0.961*
37 (3–41) 0.010* 14.5 (8–25)
6
E. Rose et al.
Table 4. Postevent Survey Responses Based on a Likert Scale of 1–7 Prefer Online Lectures (1–7)
# Responses
Satisfied with Question Prompts (1–7)
# Responses
0 0 1 1 4 3 7
1 2 3 4 5 6 7
0 0 0 0 1 3 5
0 0 1 1 4 3 7
1 2 3 4 5 6 7
0 0 0 0 2 6 3
2014 1 2 3 4 5 6 7 2015 1 2 3 4 5 6 7
In our 2015 anonymous postevent survey, 14 residents stated that they viewed all of the lectures. However, upon review of the Zaption database, only one resident viewed all 13 lecture links. Three residents viewed 12/13, three viewed 11/13, three viewed 10/13, three viewed 9/13, and three viewed 8/13. Zaption recorded if the lectures were viewed, length of viewing, the grade of the integrated questions for group B, and the number of skips forward or backward during the views. See Table 4 for detailed results. Table 5. PEM Lecture Online Video Completion Rates
Lecture Airway Bronchiolitis Croup Diabetic ketoacidosis Fever I Fever II Head trauma Intubation Neonatal resuscitation Noninvasive positive pressure ventilation PALS update Sepsis Status epilepticus
2014 Reported Views (17 Surveys Received)
2015 Reported Views (Only 16 Surveys Received)
2015 Actual Views (Per Zaption Viewing Log)
13 14 17 15
16 16 16 16
16 13 15 16
15 15 15 15 15
15 15 16 16 16
15 14 17 13 15
14
14
13
14 15 15
16 15 14
16 14 14
PEM = pediatric emergency medicine; PALS = pediatric advanced life support.
DISCUSSION The concept of the flipped classroom is gaining traction in medical education (9–13). Its success is likely multifactorial, and the method may be particularly appealing to adult learners. Malcolm Knowles’ adult educational theory emphasizes that adult learners need independence, freedom, and flexibility of education to meet their needs. The effectiveness of the flipped classroom has been demonstrated. The benefit of this method is that it frees up class time for more engaging, higher-level discussions. In medical education, the benefit is clear. Clinically relevant simulation and case discussions where interest and curiosity are piqued become better learning opportunities and the information is better retained (or, as Prober and Khan say, the material becomes more ‘‘sticky’’) (10). However, certain didactic material must be covered in residency curriculum. Online, prelecture didactic videos can cover core material and appeal to adult learners in its flexibility and pace of viewing. Online lectures have been shown to be as effective, if not more effective, than in-person lectures (14). The Department of Education published a meta-analysis that concluded, ‘‘on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction, with larger effects if the online learning was combined with face-to-face instruction’’ (15). The effectiveness of the flipped classroom and online didactic videos were presumed in this study and not directly evaluated. Two methods of online lectures (uninterrupted and with integrated questions) were evaluated, and viewing data were gathered from Zaption analytics. There was a control group for the interposed questions, but no control group for the flipped classroom concept overall, due to the clear ethical concerns of offering half of a residency class a potentially more impactful and efficient intervention. Therefore, learning retention as a direct result of the flipped classroom concept could not be adequately assessed. Overall, residents liked this intervention, and postevent scores improved in both groups. This may be partially due to the fact that they simply appreciated the time and investment of faculty to make their educational experience more efficient and of higher impact. They preferred the format of integrated questions within the lectures, and many residents who viewed lectures without questions stated that they believed they would prefer to have integrated questions when viewing lectures. Integrating questions within lectures increases the learner’s engagement with the material and highlights points of emphasis. Platforms such as Zaption
The Flipped Classroom in Emergency Medicine
are an effective resource to add integrated questions to online video lectures. Additionally, instructors are able to monitor compliance and evaluate how much time learners spent viewing the lectures, as well as assess the pace of the material for the learners. The ability to interpolate questions allows the instructor to pause and emphasize important take-home points while correcting learner error by immediately providing the correct answer and explanation to the question. In Zaption, the integrated question pops up in the viewer screen and an answer must be clicked to continue the video, but the correct answer is immediately given regardless of a(n) correct/incorrect response. This format is informative but not too cumbersome for viewers. It remains to be seen if integrated questions improve learning retention, as we have conflicting test results between the 2 years. Residency is challenging and there are competing time interests for trainees. It is often difficult for residents to complete assigned tasks ahead of time. Emergency Medicine programs should innovate educational programs to meet the needs of their trainees but may need a venue in which compliance may be monitored. Online platforms such as Zaption are tools that can be used to enhance the efficacy of flipped classroom preclassroom educational material. Limitations There are several limitations to this small descriptive educational innovation study. The small sample sizes and conflicting results between the years in this pilot trial make it difficult to reach conclusions on the effectiveness of integrated questions in online lectures. The groups were randomly selected and presumed to have similar pretest scores. Perhaps a difference in group characteristics contributed more to the difference between the years rather than the format of plain lectures vs. lectures with interpolated questions. The residents took the pretest on the computer and the posttest on paper. This difference in methods could account for a difference in scores. Additionally, three tests were created for this course. The same pretest was given both years, but the posttest was different in 2014 and 2015, which could also account for differences in scores.
7
CONCLUSION An online format combined with face-to-face interaction is an effective educational model for teaching core PEM topics in the emergency department. Interpolated questions within online videos were preferred and may increase the learner’s engagement with the material. Test scores improved and resident satisfaction was high with this educational innovation. However, completion rates may need to be monitored for successful implementation. REFERENCES 1. McDonald K, Smith CM. The flipped classroom for professional development: part I. Benefits and strategies. J Contin Educ Nurs 2013;44:437–8. 2. Critz CM, Knight D. Using the flipped classroom in graduate nursing education. Nurse Educ 2013;38:210–3. 3. McLaughlin JE, Griffin LM, Esserman DA, et al. Pharmacy student engagement, performance, and perception in a flipped satellite classroom. Am J Pharm Educ 2013;77:196. 4. Gilboy MB, Heinerichs S, Pazzaglia G. Enhancing student engagement using the flipped classroom. J Nutr Educ Behav 2015;47:109–14. 5. Missildine K, Fountain R, Summers L, Gosselin K. Flipping the classroom to improve student performance and satisfaction. J Nurs Educ 2013;52:597–9. 6. Moffett J. Twelve tips for ‘‘flipping’’ the classroom. Med Teach 2015;37:331–6. 7. Smith CM, McDonald K. The flipped classroom for professional development: part II. Making podcasts and videos. J Contin Educ Nurs 2013;44:486–7. 8. Bergmann J, Sams A. Flip your classroom. Reach every student in every class every day. Arlington, VA: International Society for Technology in Education; 2012. 9. Prober CG, Heath C. Lecture halls without lectures—a proposal for medical education. N Engl J Med 2012;366:1657–9. 10. Prober CG, Khan S. Medical education reimagined: a call to action. Acad Med 2013;88:1407–10. 11. Mehta NB, Hull AL, Young JB, Stoller JK. Just imagine: new paradigms for medical education. Acad Med 2013;88:1418–23. 12. Galway LP, Corbett KK, Takaro TK, Tairyan K, Frank E. A novel integration of online and flipped classroom instructional models in public health higher education. BMC Med Educ 2014;14:181. 13. McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Acad Med 2014;89:236–43. 14. Platz E, Liteplo A, Hurwitz S, Hwang J. Are live instructors replaceable? Computer vs. classroom lectures for EFAST training. J Emerg Med 2011;40:534–8. 15. Evaluation of evidence-based practices in online learning: a metaanalysis and review of online learning studies. Washington, DC: Department of Education, Office of Planning, Evaluation, and Policy Development; 2010. Available at: http://www2.ed.gov/ rschstat/eval/tech/evidence-based-practices/finalreport.pdf. Accessed October 20, 2015.
8
E. Rose et al.
ARTICLE SUMMARY 1. Why is this topic important? Emergency medicine residents must learn a tremendous amount of material in a condensed time period while maintaining a demanding clinical schedule. Education must be efficiently delivered in a high-yield manner, and increased engagement results in learning retention. 2. What does this study attempt to show? Online video lectures are an effective method of utilizing the flipped classroom. Interpolated questions within the videos are preferred and may increase learner engagement in the material. 3. What are the key findings? Online lectures provide flexibility to view at a convenient time and at a learner’s own pace. Traditional lectures are less flexible, efficient, and interactive. Novel educational methods such as online video lectures with/without interpolated questions may be used to increase knowledge retention, satisfaction, and engagement with the material. Residents may over-report the rate at which lectures were viewed, and it is important to monitor compliance. 4. How is patient care impacted? When education is delivered in a more efficient, interactive, and enjoyable method, emergency medicine residents are more engaged. A resident that actively participates in education results is a more informed clinician. A provider with a broad knowledge base is a more astute and effective clinician.
The Flipped Classroom in Emergency Medicine
Appendix A. Sample Test Questions.
8.e1
Appendix B. Postevent Anonymous Survey.