- Email: [email protected]
Children With Medical Complexity: A Web-Based Multimedia Curriculum Assessing Pediatric Residents Across North America
Children With Medical Complexity: A Web-Based Multimedia Curriculum Assessing Pediatric Residents Across North America Neha H. Shah, MD, MPH; Priti Bhansali, MD, MEd; Aisha Barber, MD; Keri Toner, MD; Michael Kahn, MAT; Meaghan MacLean, MD; Micah Kadden, MD; Jeffrey Sestokas, MEd; Dewesh Agrawal, MD From the Division of Hospitalist Medicine (Drs Shah, Bhansali, and Barber), Pediatric Residency Program (Drs Toner, Kadden, and Agrawal), Office of Graduate Medical Education (Mr Sestokas), Children’s National Medical Center, Washington, DC; School of Medicine and Health Sciences, The George Washington University (Mr Kahn), Washington, DC; and Pediatric Residency Program, British Columbia Children’s Hospital, University of British Columbia (Dr MacLean), Vancouver, British Columbia, Canada The authors have no conflicts of interest to disclose. Address correspondence to Neha H. Shah, MD, MPH, Division of Hospitalist Medicine, Children’s National Medical Center, 111 Michigan Ave NW, Suite M4800, Washington, DC 20010 (e-mail: [email protected]). Received for publication January 19, 2017; accepted August 15, 2017.
ABSTRACT OBJECTIVE: No standardized curricula exist for training residents in the special needs of children with medical complexity. We assessed resident satisfaction, knowledge, and behavior after implementing a novel online curriculum composed of multimedia modules on care of children with medical complexity utilizing virtual simulation. METHODS: We conducted a randomized controlled trial of residents across North America. A Web-based curriculum of 6 self-paced, interactive, multimedia modules was developed. Readings for each topic served as the control curriculum. Residents were randomized to 1 of 2 groups, each completing 3 modules and 3 sets of readings that were mutually exclusive. Outcomes included resident scores on satisfaction, knowledge-based assessments, and virtual simulation activities. RESULTS: Four hundred forty-two residents from 56 training programs enrolled in the curriculum, 229 of whom completed it and were included in the analysis. Subjects were more likely to report comfort with all topics if they reviewed modules compared to readings (P # .01 for all 6 topics). Posttest knowledge scores were significantly higher than pretest scores overall
(mean increase in score 17.7%; 95% confidence interval 16.0, 19.4), and the mean pre–post score increase for modules was significantly higher than readings (20.9% vs 15.4%, P < .001). Mean scores on the verbal handoff virtual simulation increased by 1.1 points (95% confidence interval 0.2, 2.0, P ¼ .02). There were no significant differences found in pre– post performance for the device-related emergency virtual simulation. CONCLUSIONS: There was high satisfaction, significant knowledge acquisition, and specific behavior change after participating in this innovative online curriculum. This is the first multisite, randomized trial assessing satisfaction, knowledge impact, and behavior change in a virtually simulated environment with pediatric trainees.
KEYWORDS: children with medical complexity; e-learning modules; online curricula; pediatric residents; virtual simulation ACADEMIC PEDIATRICS 2017;-:1–7
WHAT’S NEW
errors and longer hospital stays.4–12 CMC have higher rates of placement of medical devices such as gastrostomy tubes, tracheostomy tubes, and ventriculoperitoneal (VP) shunts than children who do not have any complex chronic conditions.2 Often, adverse medical device events result in or occur during hospitalization for CMC.13 Given these factors, CMC who are technology dependent compose a growing vulnerable population on pediatric wards who require specialized care for optimal outcomes. Providing quality care for CMC requires that pediatric trainees receive relevant training in practical aspects of their care, including common diagnoses and devices.14,15 However, there are few evidence-based clinical care guidelines, educational resources, and training requirements specific to inpatient
This multisite, randomized controlled trial of pediatric trainees utilized a Web-based curriculum on care of children with medical complexity and virtual simulation. Results demonstrated positive impacts on satisfaction, knowledge, and behavior.
RECENT LITERATURE DEMONSTRATES increased prevalence of children with medical complexity (CMC) in US hospitals.1–3 Health care resource utilization by CMC is also increasing, especially in the inpatient setting, where such children are particularly at risk for medical
ACADEMIC PEDIATRICS Copyright ª 2017 by Academic Pediatric Association
1
Volume -, Number -–- 2017
2
SHAH ET AL
ACADEMIC PEDIATRICS
care of CMC.16 Moreover, resident work-hour restrictions, geography, and institutional variability in exposure to CMC have been cited as significant barriers to appropriately train pediatric residents on the unique needs of CMC.17–20 From the trainee perspective, residents themselves have been found to doubt their own skills at caring for CMC, highlighting the need to develop a better training model that can overcome barriers to learning in the busy clinical environment.21 One such solution may be asynchronous computerassisted learning (CAL). The current generation of trainees not only prefers being taught via CAL, but various modalities, including e-learning/online video curricula, highfidelity simulation, and virtual patient simulators, have repeatedly been proven to be more effective teaching tools than reading or traditional lectures alone.22,23 To date, there are few studies that specifically discuss use of CAL in pediatric residency programs or the effectiveness of CAL to affect pediatric resident behavior.24,25 The objective of this study was to assess the effect of a Web-based multimedia curriculum focused on common diagnoses and devices in CMC on pediatric trainee satisfaction, knowledge, and behavior, incorporating virtual simulation assessment tools. We hypothesized that residents would favor, learn more, and demonstrate improved virtual environment patient care utilizing our Web-based multimedia modules compared to print media.
METHODS STUDY PARTICIPANTS This was a prospective randomized controlled trial of pediatrics and internal medicine–pediatrics residents at all years of training at accredited training programs across North America. STUDY DESIGN Residency program directors who were members of the Association of Pediatric Program Directors (APPD) were contacted by e-mail in September 2014 with study information. At the time of the study, APPD membership represented all 204 pediatric residency programs that participated in the match.26 Interested program directors were asked to identify a site liaison who was responsible for disseminating information about the study to residents as well as supplying our study coordinator with the names of interested participants. Our study coordinator created private accounts for each participant in the curriculum’s learning management system (LMS). After obtaining online informed consent, participants were asked to complete all precurriculum assessments, review curriculum content, and complete all postcurriculum assessments within the 3-month study period, January to April 2015. Each participant was randomized to one of two study groups, group A or group B, each receiving 3 modules and 3 sets of readings that were mutually exclusive. Group A participants received modules on VP shunts, dysautonomia, and spasticity, and readings on tracheostomy tubes, enteric feeding
tubes, and pressure ulcers. Group B participants received modules on tracheostomy tubes, enteric feeding tubes, and pressure ulcers, and readings on VP shunts, dysautonomia, and spasticity. The precurriculum assessments were composed of the following: a brief demographic survey; a multiple-choice knowledge test; a verbal handoff activity that required the participant to record their handoff of a fictional CMC after reviewing the virtual electronic medical record; and a virtual simulation involving a devicerelated emergency in a fictional CMC. Postcurriculum assessments were similar, but instead of a demographic survey, participants were asked to complete a final evaluation of the curriculum. A second round of recruitment was initiated in August 2015 through similar methodology, excluding residents who had previously participated in the curriculum (whether they had completed it or not). The second 3month study period extended from September to December 2015. This study was approved by the Children’s National Medical Center institutional review board. CURRICULUM CONTENT The curriculum was housed on a customized Modular Object-Oriented Dynamic Learning Environment (Moodle) installation. Moodle is a free open-source software package widely used by corporate and higher education institutions to disseminate online courseware in a constructivist fashion. The main content of the curriculum was delivered through 6 Web-based multimedia modules. Pediatric hospitalists and subspecialists originally created these modules with Microsoft PowerPoint 2010 (Microsoft, Redmond, Wash) for the purpose of faculty development in the Division of Hospitalist Medicine at Children’s National Medical Center, Washington, DC.27 Four of the modules have been published.28–31 PowerPoint presentations were converted into a Sharable Content Object Reference Model (SCORM) packages using Articulate Storyline 2 (Articulate, New York, NY) software to enable inclusion of audio voice-over narration, animations, and a custom navigation bar. Each 30- to 45minute module was based on topic-specific learning objectives developed before module development. The control curriculum was composed of 6 sets of readings (journal articles, textbook chapters, and Web resources), also housed on the LMS. For each topic, 3 to 5 readings (45 to 60 minutes of reading per topic) were identified by the original module authors that covered the same learning objectives as the Web-based modules. All of the readings used in the control curriculum were original references for the modules in order to ensure content similarity. PRE- AND POSTCURRICULUM ASSESSMENTS Each section of the curriculum and associated assessments was masked until the previous activity was completed to ensure residents completed all preassessments before viewing curricular content. All postassessments were administered automatically upon completion
ACADEMIC PEDIATRICS
of the curriculum review, ranging from 1 day to 3 months after initiation depending on the individual participant’s pace. SATISFACTION, ATTITUDES, AND PERCEIVED LEVEL OF COMFORT To assess the curriculum’s impact on learner attitudes, an online survey was administered as part of the final evaluation. This survey included 4-point Likert scale (strongly agree to strongly disagree), categorical, and shortanswer/free-text questions related to satisfaction, perceived clinical comfort, and perceived teaching comfort. These data were analyzed by frequency distributions and chi-square analysis. MEDICAL KNOWLEDGE ASSESSMENT A pretest assessed medical knowledge related to the 6 topics covered in the curriculum. This test was composed of 33 multiple-choice, case-based questions (4 to 6 questions per topic). These questions were written by the original module authors and were directly related to the main learning objectives upon which they chose their references and built the modules. Before initiation of the study, the test was reviewed by 3 independent experts in care of CMC to ensure content validity through an iterative process until consensus was reached. An identical multiplechoice test was administered after completion of all components of the curriculum. Pre- and posttest total and topic-specific raw scores were compared by matched t test for statistical significance. BEHAVIOR Subjects were prompted to complete 2 virtual activities that simulated common activities during residents’ inpatient rotations before and after review of the assigned modules and readings. The first virtual simulation activity involved subjects recording a verbal handoff of a fictional CMC after review of a multimedia electronic medical record (print media and audio narration from custom animated nurse and patient mother avatars). Fictional CMC background information and medical records were developed by a subset of the study authors, which specifically incorporated diagnoses and devices covered in the curriculum. After review of the patient’s medical record, participants were given 15 minutes to record their handoff using their preferred format. Voice recordings were captured by the LMS and securely saved on an internal database. An electronic checklist was used to grade the content of each verbal handoff. The checklist consisted of multiple dichotomous statements (yes/no) assessing for inclusion of diagnosis/device-related information thought to be relevant to patient care. Three internal and 3 external experts in hospital-based care of CMC reached consensus on the composition of the checklist through an iterative process, resulting in an instrument composed of 29 items for which content validity was established. A matched t test was used to assess pre–post handoff score differences.
CHILDREN WITH MEDICAL COMPLEXITY
3
Three research assistants were trained to score subjects’ handoffs using the checklist. Interrater reliability was established using 8 handoff recordings. After review of the initial 4 records, the primary investigator and research assistants met to reconcile discrepancies. An additional 4 records were then reviewed by each research assistant, and high interrater reliability was achieved, with k ¼ 0.89. For the second behavioral assessment, participants completed avatar-based virtual scenarios simulating emergencies related to the medical devices covered in this curriculum: tracheostomy occlusion, VP shunt malfunction, and dislodgement of a freshly placed gastrostomy tube. Each subject completed 2 different scenarios, one before and the other after the curriculum. These scenarios were based on a set of live, low-fidelity simulation scenarios that were previously created and published by a subset of the study authors.32 In these virtual simulation cases, subjects received information through a variety of audio and visual media in the form of custom animation, audio clips, videos, and interactive, user-driven data gathering. In the second segment of the emergency simulation, participants were offered a menu of possible actions to be taken and were asked to conduct a series of actions, with immediate feedback detailing the result of the action taken. The terminal action, which ends the case, was chosen to focus on diagnosis of device malfunction as the cause of the patient’s acute distress. If the terminal action is not selected within the initial 2 minutes, the activity advances to further decompensation of the patient. Upon selecting the terminal action, they are taken to the debriefing portion of the virtual simulation, which is a custom-animated and narrated explanation of the case resolution, including a description of the device malfunction along with review of cause and management of each device-specific malfunction. Data analysis for the device-related emergency virtual simulations included comparison of the proportion of participants who reached the desired terminal action (changing the tracheostomy tube, performing head computed tomographic scan to identify VP shunt malfunction, and obtaining a fluoroscopic dye study of the gastrostomy tube to verify dislodgement) and number of actions taken before terminal action, and was performed by chi-square analysis. All statistical analyses were conducted by IBM SPSS Statistics software (IBM SPSS, Chicago, Ill).
RESULTS DEMOGRAPHICS In total, 442 participants across 56 individual residency programs enrolled in the study over the 2 study periods. One hundred eighty did not complete the curriculum and 33 opted out of including their data in our analyses, leaving 229 (52%) who completed the curriculum and pre–post knowledge tests (Fig. 1). There were no significant demographic differences between those who did and did not complete the study. Overall, of those completing the curriculum, the majority of participants were women and categorical pediatric
4
SHAH ET AL
ACADEMIC PEDIATRICS
Enrolled n=442
Subjects randomized to Group A n=220
Subjects randomized to Group B n=222
Incomplete n=91
Incomplete n=89
Opt out n=17
Opt out n=16
Group A Final n=112 Modules: VP, DY, SP Readings: FT, TR, PU
Group B Final n=117 Modules: FT, TR, PU Readings: VP, DY, SP
KEY: VP=ventriculoperitoneal shunts, DY=dysautonomia, SP=spascity, FT=enteric feeding tubes, TR= tracheostomy tubes, PU= pressure ulcers
Figure 1. Flow diagram indicating subject randomization.
residents (Table 1). When analyzed by study group, there were no significant demographic differences between groups A and B. SATISFACTION, ATTITUDES, AND PERCEIVED COMFORT When presented with the statement, “Overall, I was satisfied with the content of this curriculum,” 97% either agreed or strongly agreed. Ninety-three percent agreed or strongly agreed that they would like to access modules like those in the curriculum in the future. Subjects were significantly more satisfied with modules than readings (P < .01). When presented with the statement “I provide better care to my patients as a result of participating in this curriculum,” 94% agreed or strongly agreed. For each of these questions, there were no statistically significant differences in response between group A and group B participants. When asked to provide a response to the statement “This curriculum improved my comfort in caring for patients Table 1. Comparison of Demographic Information for Group A Participants Versus Group B Participants Characteristic
Overall (n ¼ 229)
Group A (n ¼ 112)
Group B (n ¼ 117)
P
Level of training .71 PGY1 86 (37.5%) 39 (34.8%) 47 (40%) PGY2 70 (30.5%) 34 (30.4%) 36 (30.7%) PGY3 64 (28%) 35 (31.2%) 29 (24.8%) PGY4* 9 (4%) 4 (3.6%) 5 (4.5%) Type of residency .73 Pediatrics 212 (92.6%) 103 (91.9%) 109 (93.2%) Internal medicine– 17 (7.4%) 9 (8.1%) 8 (6.8%) pediatrics Gender .75 Male 59 (25.7%) 26 (23.2%) 33 (28.2%) Female 170 (74.3%) 86 (76.8%) 84 (71.8%) PGY indicates postgraduate year. *Residents in Canadian pediatric residency programs and internal medicine–pediatrics residents in US programs.
with [topic]” for each of the 6 topics covered in the curriculum, those who received modules on a topic were significantly more likely to agree or strongly agree with the statement compared to those who received readings. This finding held true for all 6 topics (Fig. 2). CHANGE IN KNOWLEDGE There was a statistically significant change in posttest score with a mean increase in score of 17.7% (95% confidence interval [CI] 16.0, 19.4, P < .001). There was no significant difference in pretest scores between group A and B participants (95% CI 0.43, 0.25, P ¼ .6). When analyzed by year of training, there was a significant difference in the mean change in pre–post scores for all levels of training. Intergroup comparison revealed a statistically significant difference between postgraduate year 1 change in score and postgraduate year 3/4 change in score (P ¼ .026) (Table 2). With respect to change in knowledge test score when comparing modules to readings, overall the mean pre– post score increase for modules was 20.9% and 15.5% for readings (P < .001). There was a significantly larger mean increase in score for 2 of the 6 topics for participants randomized to receive modules instead of readings: dysautonomia and enteric feeding tubes (P ¼ .03 and P < .001, respectively) (Table 3). CHANGE IN BEHAVIOR The mean pretest handoff score was 11.0, and the mean posttest handoff score was 12.1 (out of a possible 29 points). The mean improvement in score after the curriculum was 1.1 (95% CI 0.2, 2.0, P ¼ .02). MEDICAL DEVICE EMERGENCY VIRTUAL SIMULATION Performance on the virtual emergency simulation cases was compared before and after the curriculum, specifically looking at achievement of terminal action and number of
ACADEMIC PEDIATRICS
CHILDREN WITH MEDICAL COMPLEXITY
5
Figure 2. Reported comfort with clinical care by topic for modules versus readings.
actions taken before achievement of terminal action. Performance did not differ overall before and after the curriculum. Subgroup analysis by topic (VP shunts, tracheostomy tubes, and gastrostomy tubes) comparing participants who received modules versus those receiving readings also did not reveal any significant differences.
DISCUSSION To our knowledge, this is the first large-scale, randomized controlled study of a Web-based curriculum on the care of CMC assessing learner satisfaction, knowledge, and behavior change in a virtual environment. Participants in this study had higher satisfaction, increased knowledge, and improved scores on metrics assessing behavior change related to composition of verbal handoffs with multimedia modules compared to the more traditional format of reading, suggesting that interactive modules are not only a preferred method of content delivery but also more likely to affect resident performance. Furthermore, this is one of the first studies to examine the effect of the use of virtual simulation to assess behavior change. Overall satisfaction with the curriculum was high among all participants. Of particular interest, those who received modules had higher perceived comfort with clinical care after participating in this curriculum than those who received readings. This finding was statistically significant Table 2. Mean Change in Knowledge Test Scores by Postgraduate Year of Training Year of Training PGY1 PGY2 PGY3/4*
Mean Change in Score
P
20.2% 17.9% 14.6%
<.001 <.001 <.001
PGY indicates postgraduate year. *PGY3 and PGY4 were combined because of relatively small number of PGY4 participants.
for all 6 topics covered in the curriculum. Perhaps this was due to the inherently more engaging format of the multimedia modules. Unlike traditional print media, interactive modules can incorporate alternate modalities like audio narration, custom animation, and graphic design to allow for a broader reach to residents who may have varied learning styles.33 Similar to prior studies, when comparing modules to readings, there was a larger change in pre- to posttest scores for those receiving modules, again suggesting that this method of content delivery was better received by the trainees in this study.19,20,22–24 When analyzed by training year, all levels had a significant increase in their postcurriculum score, with the largest change noted for interns. This is not surprising and may indicate that this education is best targeted toward pediatric interns, who are often the first-line providers for hospitalized CMC. The novel aspect of this curriculum was the utilization of virtual simulation activities to assess behavior change before and after completing the curriculum. Verbal handoff of patients and assessment of urgent or emergent conditions in inpatients comprise important, high-stakes resident activities that carry with them the potential to compromise safety and quality of care if skills are underdeveloped. In the verbal handoff simulation, participants had a statistically significant increase in the amount of relevant patient Table 3. Improvements in Pre–Post Curriculum Knowledge Test Mean Scores for Modules Versus Readings Topic
Module
Reading
P
Spasticity Dysautonomia Ventriculoperitoneal shunt Tracheostomy tube Enteric feeding tube Pressure ulcer Overall mean
12.1% 23.1% 19.1% 17.7% 32.0% 21.4% 20.9%
7.4% 17.1% 17.1% 17.2% 14.6% 18.8% 15.5%
.10 .03 .77 .88 <.001 .33 <.001
Bold values indicate statistical significance.
6
SHAH ET AL
information included in their handoff after reviewing the content of the curriculum. Though not a primary aim of this study, this finding suggests that participants were translating knowledge gained through the curriculum to include additional information about the fictional patient in their handoffs. In our opinion, any improvement in verbal handoff has the potential to have clinical significance, given the growing body of evidence around the impact of verbal handoff on pediatric patient outcomes.34 In addition, the overall low scores on the verbal handoff exercise were striking, suggesting that there is still much to do in this arena. With respect to performance on the medical device emergency virtual simulation scenarios, we did not find a statistically significant change in performance pre–post curriculum. This may be related to the outcomes measured and the design of the cases. Identification of the device as the cause of the acute change in patient status was simultaneous with acute management of the patient. Although in reality one would be able to take several actions simultaneously, in these cases, the participant had to take one action at a time. So although they may have identified the problem, they may have prioritized patient stabilization before further investigation. Future iterations of these virtual scenarios should include multilevel action options. Additionally, although we specifically measured time to terminal action, or identification of the problem, this was not stated outright at the beginning of the scenario. Using CAL to augment learning and skills mastery can address previously mentioned barriers to educating residents during their time-limited and busy clinical training.33 All components of this curriculum can be used asynchronously regardless of location, level of training, or time of day. Training of this type may be especially useful for pediatric residents as their programs struggle to find time for competing educational experiences that will ensure a standard, basic level of competency for all trainees. Although training requirements do not yet specify the type of educational competency that is required of all pediatric trainees with respect to CMC, the growing presence of technologydependent, neurologically impaired CMC in inpatient settings1–3 serves as an impetus for development of standardized educational resources that have proven efficacy in improving knowledge and changing behavior. Furthermore, although use of virtual simulation for training purposes is not a new phenomenon in some industries such as aviation, it is a new modality for training and assessment in the health care profession, and it is just starting to be explored in the pediatric clinical care setting.25 Moving beyond the traditional forms of learner assessment, the use of dynamic and interactive Web technology to enhance and assess the learner experience was particularly well received by residents in this study. There were several limitations to this study. First, although hundreds of residents enrolled in the study, only half completed the curriculum and all assessments. Although trainees have shown preference to CAL compared to readings or lectures, actual participation has
ACADEMIC PEDIATRICS
been called into question, and a curriculum can only be effective when used.19,20,24,35 Additionally, as much of the literature shows efficacy of CAL in the classroom, few studies have shown that CAL can improve clinical skills and patient outcomes. Second, for those who did complete the study, it is possible that participants failed to complete the readings in their entirety despite stating completion on the LMS, which would bias the results in favor of the modules. Abiding by the intention-to-treat principle of randomized controlled trials, all subjects who stated they completed the curriculum were included in our analysis. Third, there may have been selection bias in the method by which program directors chose to disseminate information about the study to residents in their programs. We left to the discretion of the program director how to communicate information and whether they deemed the activity as mandatory for all or elective for those interested. To our knowledge, the majority of programs allowed residents to elect to participate, so it is possible that those residents more interested in this type of training chose to participate. However, we think that our large sampling of residents from a variety of training programs, as well as the demographic similarity between the study and control groups, helped mitigate any selection bias that may have affected our study. Fourth, though this study showed positive knowledge impact in the short term, no long-term assessment of knowledge retention was conducted for this cohort of participants. Finally, there are well-described limitations to the capabilities of virtual simulation technology in teaching key knowledge, skills, and attitudes to medical trainees.36,37 This may explain why we were not able to demonstrate as strong a treatment effect from the modules in terms of behavioral performance.
CONCLUSIONS We demonstrated that computer-based learning modules were not only preferred to reading materials but were also associated with higher degrees of satisfaction, positive knowledge impact, and improved verbal handoff in a virtual environment. Future directions include exploring the utility of CAL in clinical training outside of the study environment, an area where further research would be helpful in determining the return on investment of developing multimedia modules and virtual simulation activities. ACKNOWLEDGMENTS Supported in part by the Association of Pediatric Program Directors 2014 Special Project Grant; Elda Arce Teaching Scholar Award, Children’s National Medical Center; and the Children’s Academy of Pediatric Educators, Children’s National Medical Center. None of the funding sponsors had any role in study design, data collection, analysis, or interpretation of data, or decision to submit the article for publication.
REFERENCES 1. Burns KH, Casey PH, Lyle RE, et al. Increasing prevalence of medically complex children in US hospitals. Pediatrics. 2010;126: 638–646.
ACADEMIC PEDIATRICS 2. Simon TD, Berry J, Feudtner C, et al. Children with complex chronic conditions in inpatient hospital settings in the United States. Pediatrics. 2010;126:647–655. 3. Feudtner C, Levin JE, Srivastava R, et al. How well can hospital readmission be predicted in a cohort of hospitalized children? A retrospective, multicenter study. Pediatrics. 2009;123:286–293. 4. Cohen E, Berry JG, Camacho X, et al. Patterns and costs of health care use of children with medical complexity. Pediatrics. 2012;130: e1463–1470. 5. Berry JG, Agrawal R, Kuo DZ, et al. Characteristics of hospitalizations for patients who use a structured clinical care program for children with medical complexity. J Pediatr. 2011;159:284–290. 6. Berry JG, Poduri A, Bonkowsky JL, et al. Trends in resource utilization by children with neurological impairment in the United States inpatient health care system: a repeat cross-sectional study. PLoS Med. 2012;9:e1001158. 7. Srivastava R, Stone BL, Murphy NA. Hospitalist care of the medically complex child. Pediatr Clin North Am. 2005;52:1165–1187. 8. Bellet PS, Whitaker RC. Evaluation of a pediatric hospitalist service: impact on length of stay and hospital charges. Pediatrics. 2000;105(3 pt 1):478–484. 9. Gordon JB, Colby HH, Bartelt T, et al. A tertiary care–primary care partnership model for medically complex and fragile children and youth with special health care needs. Arch Pediatr Adolesc Med. 2007;161:937–944. 10. Bekmezian A, Chung PJ, Yazdani S. Staff-only pediatric hospitalist care of patients with medically complex subspecialty conditions in a major teaching hospital. Arch Pediatr Adolesc Med. 2008;162: 975–980. 11. Russell CJ, Simon TD. Care of children with medical complexity in the hospital setting. Pediatr Ann. 2014;43:e157–e162. 12. Berry JG, Hall M, Cohen E, et al. Ways to identify children with medical complexity and the importance of why. J Pediatr. 2015;167: 229–237. 13. Brady PW, Varadarajan K, Peterson LE, et al. Prevalence and nature of adverse medical device events in hospitalized children. J Hosp Med. 2013;8:390–393. 14. Burke RT, Alverson B. Impact of children with medically complex conditions. Pediatrics. 2010;126:789–790. 15. Adame N, Rocha M, Louden C, et al. Pediatric hospitalists’ perspectives on the care of children with medical complexity. Hosp Pediatr. 2011;1:30–37. 16. Accreditation Council of Graduate Med Educ. ACGME Program Requirements for Graduate Medical Education in Pediatrics; 2016. Available at: http://www.acgme.org/Portals/0/PFAssets/ProgramRe quirements/320_pediatrics_2016.pdf. Accessed April 8, 2017. 17. Bogetz JF, Bogetz AL, Rassbach CE, et al. Caring for children with medical complexity: challenges and educational opportunities identified by pediatric residents. Acad Pediatr. 2015;15:621–625. 18. Lau KH. Computer-based teaching module design:principles designed from learning theory. Med Educ. 2014;48:247–254. 19. D’Alessandro DM, Lewis TE, D’Alessandro MP. A pediatric digital storytelling system for third year medical students: the virtual pediatric patients. BMC Med Educ. 2004;4:1–6.
CHILDREN WITH MEDICAL COMPLEXITY
7
20. Wolbrink TA, Burns JP. Internet-based learning and applications for critical care medicine. J Intensive Care Med. 2012;27:322–332. 21. Bogetz JF, Bogetz AL, Bergman D, et al. Challenges and potential solutions to educating learners about pediatric complex care. Acad Pediatr. 2014;14:603–609. 22. Cook DA, Triola MM. Virtual patients: a critical literature review and proposed next steps. Med Educ. 2009;43:303–311. 23. Williams C, Aubin S, Harkin P, et al. A randomized, controlled, single-blind trial of teaching provided by a computer-based multimedia package versus lecture. Med Educ. 2001;35:847–854. 24. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and metaanalysis. JAMA. 2011;306:978–988. 25. Real FJ, DeBlasio D, Beck AF, et al. A virtual reality curriculum for pediatric residents decreases rates of influenza vaccine refusal. Acad Pediatr. 2017;17:431–435. 26. National Residency Match Program. Results and Data: 2017 Main Residency Match Report. Available at: http://www.nrmp.org/wpcontent/uploads/2017/06/Main-Match-Results-and-Data-2017.pdf. Accessed August 3, 2017. 27. Shah N, Davis A, Anspacher M, et al. Filling a gap: development of a Web-based multimedia curriculum on care of the child with medical complexity for pediatric practitioners. J Contin Educ Health Prof. 2015;35:278–283. 28. McClymont S, Evans S, Anspacher M, et al. Dysautonomia; 2012. Available at: http://www.mededportal.org/publication/9302. Accessed April 10, 2017. 29. Parikh K, Morozova O, Shah N. Pediatric Spasticity; 2013. Available at: http://www.mededportal.org/publication/9282. Accessed April 10, 2017. 30. Conroy R, Bhansali P, Anspacher M, et al. Enteric feeding tubes; 2012. Available at: http://www.mededportal.org/publication/9274. Accessed April 10, 2017. 31. Librizzi J, Jurgens V, Anspacher M. Complex Care Curriculum: Ventriculoperitoneal Shunts; 2014. Available at: https://www. mededportal.org/publication/9944. Accessed August 29, 2017. 32. Shah N, Bhansali P, McGarry M, et al. Simulation Series: Emergencies in Technology-dependent Children; 2013. Available at: http:// www.mededportal.org/publication/9310. Accessed April 10, 2017. 33. Berman NB, Fall LH, Maloney CG, et al. Computer-assisted instruction in clinical education: a roadmap to increasing CAI implementation. Adv Health Sci Educ. 2008;13:373–383. 34. Starmer AJ, Sectish TC, Simon DW, et al. Rates of medical errors and preventable adverse events among hospitalized children following implementation of a resident handoff bundle. JAMA. 2013;310: 2262–2270. 35. Lew EK, Nordquist EK. Asynchronous learning: student utilization out of sync with their preference. Med Educ Online. 2016;21:30587. 36. Gould DA. Training on simulators: limitations and relevance. Eur J Vasc Endovasc Surg. 2007;33:533–535. 37. Dillon GF, Boulet JR, Hawkins RE, et al. Simulations in the United States medical licensing examination (USMLE). Qual Saf Health Care. 2004;13(supp 1):i41–i45.
ABSTRACT OBJECTIVE: No standardized curricula exist for training residents in the special needs of children with medical complexity. We assessed resident satisfaction, knowledge, and behavior after implementing a novel online curriculum composed of multimedia modules on care of children with medical complexity utilizing virtual simulation. METHODS: We conducted a randomized controlled trial of residents across North America. A Web-based curriculum of 6 self-paced, interactive, multimedia modules was developed. Readings for each topic served as the control curriculum. Residents were randomized to 1 of 2 groups, each completing 3 modules and 3 sets of readings that were mutually exclusive. Outcomes included resident scores on satisfaction, knowledge-based assessments, and virtual simulation activities. RESULTS: Four hundred forty-two residents from 56 training programs enrolled in the curriculum, 229 of whom completed it and were included in the analysis. Subjects were more likely to report comfort with all topics if they reviewed modules compared to readings (P # .01 for all 6 topics). Posttest knowledge scores were significantly higher than pretest scores overall
(mean increase in score 17.7%; 95% confidence interval 16.0, 19.4), and the mean pre–post score increase for modules was significantly higher than readings (20.9% vs 15.4%, P < .001). Mean scores on the verbal handoff virtual simulation increased by 1.1 points (95% confidence interval 0.2, 2.0, P ¼ .02). There were no significant differences found in pre– post performance for the device-related emergency virtual simulation. CONCLUSIONS: There was high satisfaction, significant knowledge acquisition, and specific behavior change after participating in this innovative online curriculum. This is the first multisite, randomized trial assessing satisfaction, knowledge impact, and behavior change in a virtually simulated environment with pediatric trainees.
KEYWORDS: children with medical complexity; e-learning modules; online curricula; pediatric residents; virtual simulation ACADEMIC PEDIATRICS 2017;-:1–7
WHAT’S NEW
errors and longer hospital stays.4–12 CMC have higher rates of placement of medical devices such as gastrostomy tubes, tracheostomy tubes, and ventriculoperitoneal (VP) shunts than children who do not have any complex chronic conditions.2 Often, adverse medical device events result in or occur during hospitalization for CMC.13 Given these factors, CMC who are technology dependent compose a growing vulnerable population on pediatric wards who require specialized care for optimal outcomes. Providing quality care for CMC requires that pediatric trainees receive relevant training in practical aspects of their care, including common diagnoses and devices.14,15 However, there are few evidence-based clinical care guidelines, educational resources, and training requirements specific to inpatient
This multisite, randomized controlled trial of pediatric trainees utilized a Web-based curriculum on care of children with medical complexity and virtual simulation. Results demonstrated positive impacts on satisfaction, knowledge, and behavior.
RECENT LITERATURE DEMONSTRATES increased prevalence of children with medical complexity (CMC) in US hospitals.1–3 Health care resource utilization by CMC is also increasing, especially in the inpatient setting, where such children are particularly at risk for medical
ACADEMIC PEDIATRICS Copyright ª 2017 by Academic Pediatric Association
1
Volume -, Number -–- 2017
2
SHAH ET AL
ACADEMIC PEDIATRICS
care of CMC.16 Moreover, resident work-hour restrictions, geography, and institutional variability in exposure to CMC have been cited as significant barriers to appropriately train pediatric residents on the unique needs of CMC.17–20 From the trainee perspective, residents themselves have been found to doubt their own skills at caring for CMC, highlighting the need to develop a better training model that can overcome barriers to learning in the busy clinical environment.21 One such solution may be asynchronous computerassisted learning (CAL). The current generation of trainees not only prefers being taught via CAL, but various modalities, including e-learning/online video curricula, highfidelity simulation, and virtual patient simulators, have repeatedly been proven to be more effective teaching tools than reading or traditional lectures alone.22,23 To date, there are few studies that specifically discuss use of CAL in pediatric residency programs or the effectiveness of CAL to affect pediatric resident behavior.24,25 The objective of this study was to assess the effect of a Web-based multimedia curriculum focused on common diagnoses and devices in CMC on pediatric trainee satisfaction, knowledge, and behavior, incorporating virtual simulation assessment tools. We hypothesized that residents would favor, learn more, and demonstrate improved virtual environment patient care utilizing our Web-based multimedia modules compared to print media.
METHODS STUDY PARTICIPANTS This was a prospective randomized controlled trial of pediatrics and internal medicine–pediatrics residents at all years of training at accredited training programs across North America. STUDY DESIGN Residency program directors who were members of the Association of Pediatric Program Directors (APPD) were contacted by e-mail in September 2014 with study information. At the time of the study, APPD membership represented all 204 pediatric residency programs that participated in the match.26 Interested program directors were asked to identify a site liaison who was responsible for disseminating information about the study to residents as well as supplying our study coordinator with the names of interested participants. Our study coordinator created private accounts for each participant in the curriculum’s learning management system (LMS). After obtaining online informed consent, participants were asked to complete all precurriculum assessments, review curriculum content, and complete all postcurriculum assessments within the 3-month study period, January to April 2015. Each participant was randomized to one of two study groups, group A or group B, each receiving 3 modules and 3 sets of readings that were mutually exclusive. Group A participants received modules on VP shunts, dysautonomia, and spasticity, and readings on tracheostomy tubes, enteric feeding
tubes, and pressure ulcers. Group B participants received modules on tracheostomy tubes, enteric feeding tubes, and pressure ulcers, and readings on VP shunts, dysautonomia, and spasticity. The precurriculum assessments were composed of the following: a brief demographic survey; a multiple-choice knowledge test; a verbal handoff activity that required the participant to record their handoff of a fictional CMC after reviewing the virtual electronic medical record; and a virtual simulation involving a devicerelated emergency in a fictional CMC. Postcurriculum assessments were similar, but instead of a demographic survey, participants were asked to complete a final evaluation of the curriculum. A second round of recruitment was initiated in August 2015 through similar methodology, excluding residents who had previously participated in the curriculum (whether they had completed it or not). The second 3month study period extended from September to December 2015. This study was approved by the Children’s National Medical Center institutional review board. CURRICULUM CONTENT The curriculum was housed on a customized Modular Object-Oriented Dynamic Learning Environment (Moodle) installation. Moodle is a free open-source software package widely used by corporate and higher education institutions to disseminate online courseware in a constructivist fashion. The main content of the curriculum was delivered through 6 Web-based multimedia modules. Pediatric hospitalists and subspecialists originally created these modules with Microsoft PowerPoint 2010 (Microsoft, Redmond, Wash) for the purpose of faculty development in the Division of Hospitalist Medicine at Children’s National Medical Center, Washington, DC.27 Four of the modules have been published.28–31 PowerPoint presentations were converted into a Sharable Content Object Reference Model (SCORM) packages using Articulate Storyline 2 (Articulate, New York, NY) software to enable inclusion of audio voice-over narration, animations, and a custom navigation bar. Each 30- to 45minute module was based on topic-specific learning objectives developed before module development. The control curriculum was composed of 6 sets of readings (journal articles, textbook chapters, and Web resources), also housed on the LMS. For each topic, 3 to 5 readings (45 to 60 minutes of reading per topic) were identified by the original module authors that covered the same learning objectives as the Web-based modules. All of the readings used in the control curriculum were original references for the modules in order to ensure content similarity. PRE- AND POSTCURRICULUM ASSESSMENTS Each section of the curriculum and associated assessments was masked until the previous activity was completed to ensure residents completed all preassessments before viewing curricular content. All postassessments were administered automatically upon completion
ACADEMIC PEDIATRICS
of the curriculum review, ranging from 1 day to 3 months after initiation depending on the individual participant’s pace. SATISFACTION, ATTITUDES, AND PERCEIVED LEVEL OF COMFORT To assess the curriculum’s impact on learner attitudes, an online survey was administered as part of the final evaluation. This survey included 4-point Likert scale (strongly agree to strongly disagree), categorical, and shortanswer/free-text questions related to satisfaction, perceived clinical comfort, and perceived teaching comfort. These data were analyzed by frequency distributions and chi-square analysis. MEDICAL KNOWLEDGE ASSESSMENT A pretest assessed medical knowledge related to the 6 topics covered in the curriculum. This test was composed of 33 multiple-choice, case-based questions (4 to 6 questions per topic). These questions were written by the original module authors and were directly related to the main learning objectives upon which they chose their references and built the modules. Before initiation of the study, the test was reviewed by 3 independent experts in care of CMC to ensure content validity through an iterative process until consensus was reached. An identical multiplechoice test was administered after completion of all components of the curriculum. Pre- and posttest total and topic-specific raw scores were compared by matched t test for statistical significance. BEHAVIOR Subjects were prompted to complete 2 virtual activities that simulated common activities during residents’ inpatient rotations before and after review of the assigned modules and readings. The first virtual simulation activity involved subjects recording a verbal handoff of a fictional CMC after review of a multimedia electronic medical record (print media and audio narration from custom animated nurse and patient mother avatars). Fictional CMC background information and medical records were developed by a subset of the study authors, which specifically incorporated diagnoses and devices covered in the curriculum. After review of the patient’s medical record, participants were given 15 minutes to record their handoff using their preferred format. Voice recordings were captured by the LMS and securely saved on an internal database. An electronic checklist was used to grade the content of each verbal handoff. The checklist consisted of multiple dichotomous statements (yes/no) assessing for inclusion of diagnosis/device-related information thought to be relevant to patient care. Three internal and 3 external experts in hospital-based care of CMC reached consensus on the composition of the checklist through an iterative process, resulting in an instrument composed of 29 items for which content validity was established. A matched t test was used to assess pre–post handoff score differences.
CHILDREN WITH MEDICAL COMPLEXITY
3
Three research assistants were trained to score subjects’ handoffs using the checklist. Interrater reliability was established using 8 handoff recordings. After review of the initial 4 records, the primary investigator and research assistants met to reconcile discrepancies. An additional 4 records were then reviewed by each research assistant, and high interrater reliability was achieved, with k ¼ 0.89. For the second behavioral assessment, participants completed avatar-based virtual scenarios simulating emergencies related to the medical devices covered in this curriculum: tracheostomy occlusion, VP shunt malfunction, and dislodgement of a freshly placed gastrostomy tube. Each subject completed 2 different scenarios, one before and the other after the curriculum. These scenarios were based on a set of live, low-fidelity simulation scenarios that were previously created and published by a subset of the study authors.32 In these virtual simulation cases, subjects received information through a variety of audio and visual media in the form of custom animation, audio clips, videos, and interactive, user-driven data gathering. In the second segment of the emergency simulation, participants were offered a menu of possible actions to be taken and were asked to conduct a series of actions, with immediate feedback detailing the result of the action taken. The terminal action, which ends the case, was chosen to focus on diagnosis of device malfunction as the cause of the patient’s acute distress. If the terminal action is not selected within the initial 2 minutes, the activity advances to further decompensation of the patient. Upon selecting the terminal action, they are taken to the debriefing portion of the virtual simulation, which is a custom-animated and narrated explanation of the case resolution, including a description of the device malfunction along with review of cause and management of each device-specific malfunction. Data analysis for the device-related emergency virtual simulations included comparison of the proportion of participants who reached the desired terminal action (changing the tracheostomy tube, performing head computed tomographic scan to identify VP shunt malfunction, and obtaining a fluoroscopic dye study of the gastrostomy tube to verify dislodgement) and number of actions taken before terminal action, and was performed by chi-square analysis. All statistical analyses were conducted by IBM SPSS Statistics software (IBM SPSS, Chicago, Ill).
RESULTS DEMOGRAPHICS In total, 442 participants across 56 individual residency programs enrolled in the study over the 2 study periods. One hundred eighty did not complete the curriculum and 33 opted out of including their data in our analyses, leaving 229 (52%) who completed the curriculum and pre–post knowledge tests (Fig. 1). There were no significant demographic differences between those who did and did not complete the study. Overall, of those completing the curriculum, the majority of participants were women and categorical pediatric
4
SHAH ET AL
ACADEMIC PEDIATRICS
Enrolled n=442
Subjects randomized to Group A n=220
Subjects randomized to Group B n=222
Incomplete n=91
Incomplete n=89
Opt out n=17
Opt out n=16
Group A Final n=112 Modules: VP, DY, SP Readings: FT, TR, PU
Group B Final n=117 Modules: FT, TR, PU Readings: VP, DY, SP
KEY: VP=ventriculoperitoneal shunts, DY=dysautonomia, SP=spascity, FT=enteric feeding tubes, TR= tracheostomy tubes, PU= pressure ulcers
Figure 1. Flow diagram indicating subject randomization.
residents (Table 1). When analyzed by study group, there were no significant demographic differences between groups A and B. SATISFACTION, ATTITUDES, AND PERCEIVED COMFORT When presented with the statement, “Overall, I was satisfied with the content of this curriculum,” 97% either agreed or strongly agreed. Ninety-three percent agreed or strongly agreed that they would like to access modules like those in the curriculum in the future. Subjects were significantly more satisfied with modules than readings (P < .01). When presented with the statement “I provide better care to my patients as a result of participating in this curriculum,” 94% agreed or strongly agreed. For each of these questions, there were no statistically significant differences in response between group A and group B participants. When asked to provide a response to the statement “This curriculum improved my comfort in caring for patients Table 1. Comparison of Demographic Information for Group A Participants Versus Group B Participants Characteristic
Overall (n ¼ 229)
Group A (n ¼ 112)
Group B (n ¼ 117)
P
Level of training .71 PGY1 86 (37.5%) 39 (34.8%) 47 (40%) PGY2 70 (30.5%) 34 (30.4%) 36 (30.7%) PGY3 64 (28%) 35 (31.2%) 29 (24.8%) PGY4* 9 (4%) 4 (3.6%) 5 (4.5%) Type of residency .73 Pediatrics 212 (92.6%) 103 (91.9%) 109 (93.2%) Internal medicine– 17 (7.4%) 9 (8.1%) 8 (6.8%) pediatrics Gender .75 Male 59 (25.7%) 26 (23.2%) 33 (28.2%) Female 170 (74.3%) 86 (76.8%) 84 (71.8%) PGY indicates postgraduate year. *Residents in Canadian pediatric residency programs and internal medicine–pediatrics residents in US programs.
with [topic]” for each of the 6 topics covered in the curriculum, those who received modules on a topic were significantly more likely to agree or strongly agree with the statement compared to those who received readings. This finding held true for all 6 topics (Fig. 2). CHANGE IN KNOWLEDGE There was a statistically significant change in posttest score with a mean increase in score of 17.7% (95% confidence interval [CI] 16.0, 19.4, P < .001). There was no significant difference in pretest scores between group A and B participants (95% CI 0.43, 0.25, P ¼ .6). When analyzed by year of training, there was a significant difference in the mean change in pre–post scores for all levels of training. Intergroup comparison revealed a statistically significant difference between postgraduate year 1 change in score and postgraduate year 3/4 change in score (P ¼ .026) (Table 2). With respect to change in knowledge test score when comparing modules to readings, overall the mean pre– post score increase for modules was 20.9% and 15.5% for readings (P < .001). There was a significantly larger mean increase in score for 2 of the 6 topics for participants randomized to receive modules instead of readings: dysautonomia and enteric feeding tubes (P ¼ .03 and P < .001, respectively) (Table 3). CHANGE IN BEHAVIOR The mean pretest handoff score was 11.0, and the mean posttest handoff score was 12.1 (out of a possible 29 points). The mean improvement in score after the curriculum was 1.1 (95% CI 0.2, 2.0, P ¼ .02). MEDICAL DEVICE EMERGENCY VIRTUAL SIMULATION Performance on the virtual emergency simulation cases was compared before and after the curriculum, specifically looking at achievement of terminal action and number of
ACADEMIC PEDIATRICS
CHILDREN WITH MEDICAL COMPLEXITY
5
Figure 2. Reported comfort with clinical care by topic for modules versus readings.
actions taken before achievement of terminal action. Performance did not differ overall before and after the curriculum. Subgroup analysis by topic (VP shunts, tracheostomy tubes, and gastrostomy tubes) comparing participants who received modules versus those receiving readings also did not reveal any significant differences.
DISCUSSION To our knowledge, this is the first large-scale, randomized controlled study of a Web-based curriculum on the care of CMC assessing learner satisfaction, knowledge, and behavior change in a virtual environment. Participants in this study had higher satisfaction, increased knowledge, and improved scores on metrics assessing behavior change related to composition of verbal handoffs with multimedia modules compared to the more traditional format of reading, suggesting that interactive modules are not only a preferred method of content delivery but also more likely to affect resident performance. Furthermore, this is one of the first studies to examine the effect of the use of virtual simulation to assess behavior change. Overall satisfaction with the curriculum was high among all participants. Of particular interest, those who received modules had higher perceived comfort with clinical care after participating in this curriculum than those who received readings. This finding was statistically significant Table 2. Mean Change in Knowledge Test Scores by Postgraduate Year of Training Year of Training PGY1 PGY2 PGY3/4*
Mean Change in Score
P
20.2% 17.9% 14.6%
<.001 <.001 <.001
PGY indicates postgraduate year. *PGY3 and PGY4 were combined because of relatively small number of PGY4 participants.
for all 6 topics covered in the curriculum. Perhaps this was due to the inherently more engaging format of the multimedia modules. Unlike traditional print media, interactive modules can incorporate alternate modalities like audio narration, custom animation, and graphic design to allow for a broader reach to residents who may have varied learning styles.33 Similar to prior studies, when comparing modules to readings, there was a larger change in pre- to posttest scores for those receiving modules, again suggesting that this method of content delivery was better received by the trainees in this study.19,20,22–24 When analyzed by training year, all levels had a significant increase in their postcurriculum score, with the largest change noted for interns. This is not surprising and may indicate that this education is best targeted toward pediatric interns, who are often the first-line providers for hospitalized CMC. The novel aspect of this curriculum was the utilization of virtual simulation activities to assess behavior change before and after completing the curriculum. Verbal handoff of patients and assessment of urgent or emergent conditions in inpatients comprise important, high-stakes resident activities that carry with them the potential to compromise safety and quality of care if skills are underdeveloped. In the verbal handoff simulation, participants had a statistically significant increase in the amount of relevant patient Table 3. Improvements in Pre–Post Curriculum Knowledge Test Mean Scores for Modules Versus Readings Topic
Module
Reading
P
Spasticity Dysautonomia Ventriculoperitoneal shunt Tracheostomy tube Enteric feeding tube Pressure ulcer Overall mean
12.1% 23.1% 19.1% 17.7% 32.0% 21.4% 20.9%
7.4% 17.1% 17.1% 17.2% 14.6% 18.8% 15.5%
.10 .03 .77 .88 <.001 .33 <.001
Bold values indicate statistical significance.
6
SHAH ET AL
information included in their handoff after reviewing the content of the curriculum. Though not a primary aim of this study, this finding suggests that participants were translating knowledge gained through the curriculum to include additional information about the fictional patient in their handoffs. In our opinion, any improvement in verbal handoff has the potential to have clinical significance, given the growing body of evidence around the impact of verbal handoff on pediatric patient outcomes.34 In addition, the overall low scores on the verbal handoff exercise were striking, suggesting that there is still much to do in this arena. With respect to performance on the medical device emergency virtual simulation scenarios, we did not find a statistically significant change in performance pre–post curriculum. This may be related to the outcomes measured and the design of the cases. Identification of the device as the cause of the acute change in patient status was simultaneous with acute management of the patient. Although in reality one would be able to take several actions simultaneously, in these cases, the participant had to take one action at a time. So although they may have identified the problem, they may have prioritized patient stabilization before further investigation. Future iterations of these virtual scenarios should include multilevel action options. Additionally, although we specifically measured time to terminal action, or identification of the problem, this was not stated outright at the beginning of the scenario. Using CAL to augment learning and skills mastery can address previously mentioned barriers to educating residents during their time-limited and busy clinical training.33 All components of this curriculum can be used asynchronously regardless of location, level of training, or time of day. Training of this type may be especially useful for pediatric residents as their programs struggle to find time for competing educational experiences that will ensure a standard, basic level of competency for all trainees. Although training requirements do not yet specify the type of educational competency that is required of all pediatric trainees with respect to CMC, the growing presence of technologydependent, neurologically impaired CMC in inpatient settings1–3 serves as an impetus for development of standardized educational resources that have proven efficacy in improving knowledge and changing behavior. Furthermore, although use of virtual simulation for training purposes is not a new phenomenon in some industries such as aviation, it is a new modality for training and assessment in the health care profession, and it is just starting to be explored in the pediatric clinical care setting.25 Moving beyond the traditional forms of learner assessment, the use of dynamic and interactive Web technology to enhance and assess the learner experience was particularly well received by residents in this study. There were several limitations to this study. First, although hundreds of residents enrolled in the study, only half completed the curriculum and all assessments. Although trainees have shown preference to CAL compared to readings or lectures, actual participation has
ACADEMIC PEDIATRICS
been called into question, and a curriculum can only be effective when used.19,20,24,35 Additionally, as much of the literature shows efficacy of CAL in the classroom, few studies have shown that CAL can improve clinical skills and patient outcomes. Second, for those who did complete the study, it is possible that participants failed to complete the readings in their entirety despite stating completion on the LMS, which would bias the results in favor of the modules. Abiding by the intention-to-treat principle of randomized controlled trials, all subjects who stated they completed the curriculum were included in our analysis. Third, there may have been selection bias in the method by which program directors chose to disseminate information about the study to residents in their programs. We left to the discretion of the program director how to communicate information and whether they deemed the activity as mandatory for all or elective for those interested. To our knowledge, the majority of programs allowed residents to elect to participate, so it is possible that those residents more interested in this type of training chose to participate. However, we think that our large sampling of residents from a variety of training programs, as well as the demographic similarity between the study and control groups, helped mitigate any selection bias that may have affected our study. Fourth, though this study showed positive knowledge impact in the short term, no long-term assessment of knowledge retention was conducted for this cohort of participants. Finally, there are well-described limitations to the capabilities of virtual simulation technology in teaching key knowledge, skills, and attitudes to medical trainees.36,37 This may explain why we were not able to demonstrate as strong a treatment effect from the modules in terms of behavioral performance.
CONCLUSIONS We demonstrated that computer-based learning modules were not only preferred to reading materials but were also associated with higher degrees of satisfaction, positive knowledge impact, and improved verbal handoff in a virtual environment. Future directions include exploring the utility of CAL in clinical training outside of the study environment, an area where further research would be helpful in determining the return on investment of developing multimedia modules and virtual simulation activities. ACKNOWLEDGMENTS Supported in part by the Association of Pediatric Program Directors 2014 Special Project Grant; Elda Arce Teaching Scholar Award, Children’s National Medical Center; and the Children’s Academy of Pediatric Educators, Children’s National Medical Center. None of the funding sponsors had any role in study design, data collection, analysis, or interpretation of data, or decision to submit the article for publication.
REFERENCES 1. Burns KH, Casey PH, Lyle RE, et al. Increasing prevalence of medically complex children in US hospitals. Pediatrics. 2010;126: 638–646.
ACADEMIC PEDIATRICS 2. Simon TD, Berry J, Feudtner C, et al. Children with complex chronic conditions in inpatient hospital settings in the United States. Pediatrics. 2010;126:647–655. 3. Feudtner C, Levin JE, Srivastava R, et al. How well can hospital readmission be predicted in a cohort of hospitalized children? A retrospective, multicenter study. Pediatrics. 2009;123:286–293. 4. Cohen E, Berry JG, Camacho X, et al. Patterns and costs of health care use of children with medical complexity. Pediatrics. 2012;130: e1463–1470. 5. Berry JG, Agrawal R, Kuo DZ, et al. Characteristics of hospitalizations for patients who use a structured clinical care program for children with medical complexity. J Pediatr. 2011;159:284–290. 6. Berry JG, Poduri A, Bonkowsky JL, et al. Trends in resource utilization by children with neurological impairment in the United States inpatient health care system: a repeat cross-sectional study. PLoS Med. 2012;9:e1001158. 7. Srivastava R, Stone BL, Murphy NA. Hospitalist care of the medically complex child. Pediatr Clin North Am. 2005;52:1165–1187. 8. Bellet PS, Whitaker RC. Evaluation of a pediatric hospitalist service: impact on length of stay and hospital charges. Pediatrics. 2000;105(3 pt 1):478–484. 9. Gordon JB, Colby HH, Bartelt T, et al. A tertiary care–primary care partnership model for medically complex and fragile children and youth with special health care needs. Arch Pediatr Adolesc Med. 2007;161:937–944. 10. Bekmezian A, Chung PJ, Yazdani S. Staff-only pediatric hospitalist care of patients with medically complex subspecialty conditions in a major teaching hospital. Arch Pediatr Adolesc Med. 2008;162: 975–980. 11. Russell CJ, Simon TD. Care of children with medical complexity in the hospital setting. Pediatr Ann. 2014;43:e157–e162. 12. Berry JG, Hall M, Cohen E, et al. Ways to identify children with medical complexity and the importance of why. J Pediatr. 2015;167: 229–237. 13. Brady PW, Varadarajan K, Peterson LE, et al. Prevalence and nature of adverse medical device events in hospitalized children. J Hosp Med. 2013;8:390–393. 14. Burke RT, Alverson B. Impact of children with medically complex conditions. Pediatrics. 2010;126:789–790. 15. Adame N, Rocha M, Louden C, et al. Pediatric hospitalists’ perspectives on the care of children with medical complexity. Hosp Pediatr. 2011;1:30–37. 16. Accreditation Council of Graduate Med Educ. ACGME Program Requirements for Graduate Medical Education in Pediatrics; 2016. Available at: http://www.acgme.org/Portals/0/PFAssets/ProgramRe quirements/320_pediatrics_2016.pdf. Accessed April 8, 2017. 17. Bogetz JF, Bogetz AL, Rassbach CE, et al. Caring for children with medical complexity: challenges and educational opportunities identified by pediatric residents. Acad Pediatr. 2015;15:621–625. 18. Lau KH. Computer-based teaching module design:principles designed from learning theory. Med Educ. 2014;48:247–254. 19. D’Alessandro DM, Lewis TE, D’Alessandro MP. A pediatric digital storytelling system for third year medical students: the virtual pediatric patients. BMC Med Educ. 2004;4:1–6.
CHILDREN WITH MEDICAL COMPLEXITY
7
20. Wolbrink TA, Burns JP. Internet-based learning and applications for critical care medicine. J Intensive Care Med. 2012;27:322–332. 21. Bogetz JF, Bogetz AL, Bergman D, et al. Challenges and potential solutions to educating learners about pediatric complex care. Acad Pediatr. 2014;14:603–609. 22. Cook DA, Triola MM. Virtual patients: a critical literature review and proposed next steps. Med Educ. 2009;43:303–311. 23. Williams C, Aubin S, Harkin P, et al. A randomized, controlled, single-blind trial of teaching provided by a computer-based multimedia package versus lecture. Med Educ. 2001;35:847–854. 24. Cook DA, Hatala R, Brydges R, et al. Technology-enhanced simulation for health professions education: a systematic review and metaanalysis. JAMA. 2011;306:978–988. 25. Real FJ, DeBlasio D, Beck AF, et al. A virtual reality curriculum for pediatric residents decreases rates of influenza vaccine refusal. Acad Pediatr. 2017;17:431–435. 26. National Residency Match Program. Results and Data: 2017 Main Residency Match Report. Available at: http://www.nrmp.org/wpcontent/uploads/2017/06/Main-Match-Results-and-Data-2017.pdf. Accessed August 3, 2017. 27. Shah N, Davis A, Anspacher M, et al. Filling a gap: development of a Web-based multimedia curriculum on care of the child with medical complexity for pediatric practitioners. J Contin Educ Health Prof. 2015;35:278–283. 28. McClymont S, Evans S, Anspacher M, et al. Dysautonomia; 2012. Available at: http://www.mededportal.org/publication/9302. Accessed April 10, 2017. 29. Parikh K, Morozova O, Shah N. Pediatric Spasticity; 2013. Available at: http://www.mededportal.org/publication/9282. Accessed April 10, 2017. 30. Conroy R, Bhansali P, Anspacher M, et al. Enteric feeding tubes; 2012. Available at: http://www.mededportal.org/publication/9274. Accessed April 10, 2017. 31. Librizzi J, Jurgens V, Anspacher M. Complex Care Curriculum: Ventriculoperitoneal Shunts; 2014. Available at: https://www. mededportal.org/publication/9944. Accessed August 29, 2017. 32. Shah N, Bhansali P, McGarry M, et al. Simulation Series: Emergencies in Technology-dependent Children; 2013. Available at: http:// www.mededportal.org/publication/9310. Accessed April 10, 2017. 33. Berman NB, Fall LH, Maloney CG, et al. Computer-assisted instruction in clinical education: a roadmap to increasing CAI implementation. Adv Health Sci Educ. 2008;13:373–383. 34. Starmer AJ, Sectish TC, Simon DW, et al. Rates of medical errors and preventable adverse events among hospitalized children following implementation of a resident handoff bundle. JAMA. 2013;310: 2262–2270. 35. Lew EK, Nordquist EK. Asynchronous learning: student utilization out of sync with their preference. Med Educ Online. 2016;21:30587. 36. Gould DA. Training on simulators: limitations and relevance. Eur J Vasc Endovasc Surg. 2007;33:533–535. 37. Dillon GF, Boulet JR, Hawkins RE, et al. Simulations in the United States medical licensing examination (USMLE). Qual Saf Health Care. 2004;13(supp 1):i41–i45.