- Email: [email protected]
VR Simulation Leads to Enhanced Procedural Confidence for Surgical Trainees
ARTICLE IN PRESS ORIGINAL REPORTS
VR Simulation Leads to Enhanced Procedural Confidence for Surgical Trainees € rg Peters, PhD,† and Juan C. Cenda n, MD*, Heather Lesch, BS,* Evan Johnson BS,* Jo University of Central Florida College of Medicine, Orlando, Florida; and †University of Florida College of Engineering, Gainesville, Florida
*
OBJECTIVE: Active learning techniques result in greater knowledge acquisition compared to passive methods. For medical students with limited hands-on operative experiences, virtual reality platforms represent active learning and may enhance procedural training. We hypothesize that virtual reality simulators like Toolkit for Illustration of Procedures in Surgery (TIPS) are a more effective modality in teaching laparoscopic surgical techniques to medical students when compared to passive learning tools like videos.
CONCLUSIONS: Video and TIPS both enhanced different
DESIGN: In this crossover study, participants were ran-
education, laparoscopic training, surgical simulation
domly assigned to perform either a TIPS laparoscopic appendectomy followed by video of a laparoscopic cholecystectomy, or video of a laparoscopic appendectomy followed by TIPS laparoscopic cholecystectomy. A knowledge assessment followed each intervention. A postsurvey was used to gather feedback and subjective impressions of the learning experience.
aspects of student learning; however, the active TIPS platform produced greater confidence in the ability to reproduce the steps of the procedure and had greater utility as a learning strategy. Videos are simple to use and can serve a complementary role in curriculum design. ( J Surg Ed 000:1 6. Published by Elsevier Inc. on behalf of Association of Program Directors in Surgery.) KEY WORDS: surgical education, virtual reality-based COMPETENCIES: Medical Knowledge, Practice-Based
Learning and Improvement, Patient Care, Systems-Based Practice
INTRODUCTION
SETTING: University of Central Florida College of Medicine. PARTICIPANTS: Second, third, and fourth-year medical
students (n = 37). RESULTS: Validation of the content assessments revealed
strong internal consistency (Cronbach’s a = 0.73). A 2-tailed Fisher’s exact test revealed that the video had greater ease of use (p = 0.032), but TIPS had greater utility as a learning tool (p < 0.001) and instilled greater confidence in the ability to reproduce procedural steps (p < 0.001). A 2-tailed t test of the average content quiz scores revealed no significant difference in percentage correct between groups on the laparoscopic appendectomy quiz (p = 0.772), but a difference favoring video learning on the laparoscopic cholecystectomy quiz (p = 0.042) Funding Sources: This work was supported by the National Institutes of Bioengineering: Safety instruction and competency assessment for laparoscopy; NIHNIBIB RO1: EB018625. The authors have no financial interest in either TIPS or Wise-MD. Correspondence: Inquiries to Juan Cendan, MD, University of Central Florida College of Medicine, 6850 Lake Nona Blvd, Orlando, FL 32827; e-mail: [email protected]
As minimally invasive surgical techniques become more popular across all surgical specialties, there arises a concordant need to appropriately train residents and medical students to master these skills. Studies of visuospatial and psychomotor learning curves reveal distinct levels of learner aptitude for these techniques.1 Trainees, regardless of aptitude, could benefit from deliberate practice models to help hone these skills.2 Simulation is appealing as it reduces reliance on patients, cadaveric, and animal models for skills practice and acquisition. Historically, these skills have been refined under the supervision of a surgical mentor,3 but as mentorship opportunities have decreased, a larger emphasis has been placed on the use of alternative teaching modalities including textbooks, anatomical atlases, and training videos4; while valuable, these resources are limited in their ability to demonstrate nuanced information such as the texture of different types of tissue and necessary force to be exerted during dissection.5 Additionally, it has been repeatedly shown that active learning techniques result in greater knowledge acquisition and retention than
Journal of Surgical Education Published by Elsevier Inc. on behalf of Association of Program Directors in 1931-7204/$30.00 Surgery. https://doi.org/10.1016/j.jsurg.2019.08.008
1
ARTICLE IN PRESS passive methods.6,7 Confidence in one’s abilities alone has proven to enhance the learning process8,9 and can be improved by using simulations prior to the transfer of skills to actual patients. Virtual reality (VR) training can be a valuable tool in the development of technical skills while minimizing risk to patients, particularly for rare conditions that the learner might not encounter during their training.10 Given that laparoscopic surgery has a longer learning curve than open surgery, it would be beneficial to give trainees interested in procedural specialties the opportunity to develop their skills.4,11-19 As graduating residents consistently report not only decreased time in the operating room but also feeling underprepared for independent practice, it is important to provide modes of supplemental practice.16-20 We hypothesized that a VR technology-based procedural experience would complement commonly available educational videos leading to increased confidence and knowledge. Understanding the role of each of these technologies could inform curricular development efforts.
MATERIALS AND METHODS We aimed to test 2 different learning modalities handson active learning vs passive video learning. We wanted to reduce the chance that students struggled with the procedure due to inadequate knowledge of the anatomy, so participation was limited to only second to fourth year medical students at the University of Central Florida (UCF) College of Medicine who had successfully completed the first year anatomy course. Power analysis was based on an assumption of tolerance for type I a error of 0.05 and a type II error, or b power, of 80. The effect size was determined after developing the assessment. The research team concluded that a 5% difference in the assessment would yield the desired minimum required effect given the anticipated size of study cohorts. These calculations necessitated a sample size of at least 36 subjects. The Institutional Review Board for UCF gave permission (UCF IRB # SBE-18-14544). Enrollment was made by inviting all second to fourth year medical students to participate via email and posting to the class-specific Facebook page. Thirty-seven students consented to participate. Students were told the experiment would take no longer than 1 hour to complete and would be compensated for their time with a $20 giftcard. This experiment employed the use of the Toolkit for Illustration of Procedures in Surgery (TIPS) VR software. TIPS has been developed as part of a longstanding collaboration between authors (JC and JB); it is an interactive surgical illustration platform which allows the user to
2
FIGURE 1. A student participant utilizing the TIPS simulation
engage with anatomic renderings using what appears to be laparoscopic instruments.21 The user engages the environment using 2 three-dimensional (3D) haptic drawing pens, which act as facsimiles for laparoscopic instruments (see Fig. 1). Participants can choose from several common laparoscopic instrument tips (scissors, Maryland dissector, etc.) on either hand by toggling a single button. Haptic feedback is generated when the participant engages with an organ or tissue. TIPS does not provide step-wise instructions for the user; directions for procedural completion were printed showing the surgical sequence and matching screen shots taken from the TIPS program. The other learning modality utilized in this experiment is the web initiative for surgical education (WISE-MD)22 videos. These videos are used as online learning tools currently at the UCF College of Medicine, as well as other learning institutions, and display step-by-step directions on how surgical procedures are completed with real-life video paired with auditory voice-over and some animations. The laparoscopic appendectomy WISE-MD video was 1 minute 11 seconds in run-time, and the laparoscopic cholecystectomy video was 2 minutes and 30 seconds long. Students were blocked for hour-long sessions for the TIPS interaction; exact time-on-task was not measured though most sessions lasted about half hour. In this experiment, students were assigned to a group in an alternating fashion based on their selection of a timeslot. Participants completed the first intervention, post-quiz, second intervention, postquiz, and then a survey (see Fig. 2). The 2 postquizzes were developed by the authors after review of the content covered in both the TIPS simulation and the WISE-MD videos to ensure equal opportunity for correct answers across the groups. As
Journal of Surgical Education Volume 00 /Number 00 Month 2019
ARTICLE IN PRESS
FIGURE 2. Experiment flow
a form of psychometric assessment validation Cronbach’s alpha test can be viewed as the expected correlation of 2 tests that measure the same construct. It is a useful analysis of the internal reliability of an assessment; in this case the Cronbach’s alpha test demonstrated good internal reliability for the posttests (a = 0.73).23 In the survey portion of this experiment, students were asked for their feedback based on how much they liked using each learning modality, the ease of use of each, and other subjective questions based on a 5-point Likert scale. There was also a section where participants could provide any comments or suggestions. Statistical analysis included descriptive analysis of demographics, as well as a 2-tailed Fisher’s test for evaluation of 5-point Likert scale data.24 A 2-tailed t test was utilized to analyze the postquiz data.25
RESULTS For a total of 37 students, there were 19 in Group A and 18 in Group B. 67.6% of participants were male, and there were 15 second year students (40.5%), 8 third year students (21.6%), and 14 fourth year students (37.8%). Roughly a third of the students (13 students, 35.1%) reported they intend to pursue a surgical career. 62.1% of participants had already completed a surgical rotation during their third year and 67.6% had seen a laparoscopic appendectomy or laparoscopic cholecystectomy in person. Participants performed overall well on the postappendectomy quiz with no significant difference (81.6% for TIPS vs 83.3% for WISE-MD, p = 0.772). However, the postcholecystectomy quiz did demonstrate a difference in favor of the WISE-MD videos (84.2% vs 76.6%, p = 0.042).
Journal of Surgical Education Volume 00 /Number 00 Month 2019
Overall, students favored the utility of TIPS as a learning tool compared to the videos (p = 0.009; Fig. 3). This effect was particularly pronounced in the case of the laparoscopic cholecystectomy (p = 0.001). However, participants had no significant preference between the 2 modalities for the laparoscopic appendectomy (p = 0.536). Students’ confidence in reproducing the procedure was assessed (Fig. 3). Overall students reported higher confidence after completing the TIPS simulation (p = 0.001). The overall effect was, again, largely driven by the large impact shown by TIPS in laparoscopic cholecystectomy training (p = 0.002), with less effect noted in the laparoscopic appendectomy scenario (p = 0.146). As a summary question, participants were asked to report which modality they would prefer learning surgical techniques from and 83.8% (31 of 37) reported they would prefer using the TIPS simulation. Assessment of ease of use of the technology revealed a preference towards WISE-MD (p = 0.032; Fig. 3). Specifically, participants reported that the WISE-MD video was easier to use than the laparoscopic appendectomy TIPS simulation (p = 0.004); there was no significant difference in the reported ease of use for the laparoscopic cholecystectomy TIPS simulation (p = 0.507). It is worth noting that 43.2% of participants reported having technical difficulties while using the TIPS simulation. Research staff was available at all times during the investigation to intervene in typically minor technical issues.
CONCLUSIONS In this study, we aimed to examine the TIPS simulation software and evaluate its potential place in the learning environment for surgical trainees. When asked whether participants thought the TIPS simulation had good utility
3
ARTICLE IN PRESS
Ulity
Confidence
Ease of Use
25 20 15
10 5 0
TIPS
WISEMD
FIGURE 3. Survey questions
as a learning tool and whether they could confidently verbally reproduce the procedural steps after using the simulation, collectively the response was in favor of the TIPS simulation. Not only did the TIPS simulation appear to be a useful adjunct to the learning environment, but it improved student confidence. Confidence, itself, is a vital component to the learning process.8,9 Educators commonly seek ways not only to improve testable knowledge, but improve learners’ confidence in their skills. Participants felt more confident in their ability to verbally reproduce procedural steps after using the TIPS simulation, but this did not translate directly into improved postquiz scores. In fact, there was no difference for the laparoscopic appendectomy postquiz, while a meaningful difference was noted in the laparoscopic cholecystectomy postquiz in favor of the WISEMD videos. The videos were roughly 1.5 to 2.5 minutes long and paired not only reallife visuals with animations, but had a matching voice-over, pairing auditory and visual learning. The TIPS simulation paired only visual and kinesthetic learning, where participants followed step-by-step written directions with screen shots from the simulation as guide posts. Interestingly, the shorter video (laparoscopic appendectomy) did not have significant differences in post-quiz scores, while the longer video (laparoscopic cholecystectomy) did, which could demonstrate that the longer the auditory and visual reinforcement, the more likely you were to perform better on knowledge-based questions. During medical school, students’ education is nearly entirely auditory and visual for the first 2 years of the curriculum, so students are far more used to learning from these methods when compared to the kinesthetic learning they begin experiencing as they progress through their third year. The TIPS
4
simulation software could be improved by integrating auditory learning with the other 2 modalities. This could be accomplished by adding spoken instructions that triggered at the completion of the previous procedural step. With nearly half of the participants experiencing some form of technical difficulty while using the TIPS simulation, it comes as no surprise that participants reported the WISEMD videos as easier to use. Again, though, when broken down by individual procedure, there was not a significant difference when comparing ease-of-use between WISEMD videos and the TIPS simulation for the laparoscopic cholecystectomy procedure. We experienced some difficulties with platform stability of the new software, as well as instrument calibration and orientation, which likely reflects these results. The laparoscopic appendectomy TIPS procedure, especially, proved more technically challenging to participants. There are improvements that need to be made to the TIPS simulation software to improve usability and decrease technical difficulties, despite these participants still preferred learning via the TIPS simulation. Also collected was free text written feedback from participants. Common themes derived from these comments include technical difficulties, the learning curve associated with the TIPS simulation, enjoyment with using the TIPS simulation, and preference for combination of the WISEMD videos with the simulation. One participant reported that the video moved too quickly through the steps for adequate knowledge retention, so they relied on previous knowledge of the procedure to answer the postquiz questions. This brings up an interesting potential confounding factor for the results of the post-quizzes but does not change that participants generally preferred utilizing the TIPS simulation as a learning tool. Many reported that the video was very concise, but
Journal of Surgical Education Volume 00 /Number 00 Month 2019
ARTICLE IN PRESS they would have liked to have been able to repeat it for better knowledge solidification. Overall, students enjoyed using the TIPS simulation once they overcame the learning curve. A vital point echoed by many participants was the suggestion to integrate both modalities: “maybe getting to read through the steps, then watching a video, and then using the simulator would have been most effective.” Through this experiment we evaluated a new learning tool for hands-on teaching of laparoscopic techniques when compared to traditional passive learning from a video. There are clearly merits to both methods of learning and the greatest improvement on learning, confidence, and eventually performance may be the integration of the 2 instead of either individually.
ACKNOWLEDGMENTS WISE-MD, NYU School of Medicine.
through the transition to adulthood. Int J Disabil Dev Educ. 2015;62:137–139. https://doi.org/ 10.1080/1034912X.2014.985026. 7. Lin C, Gao J, Zheng H, et al. When to introduce
three-dimensional visualization technology into surgical residency: a randomized controlled trial. J Med Syst. 2019;43:71. 8. Peyre SE, Peyre CG, Sullivan ME, Towfigh S. A surgi-
cal skills elective can improve student confidence prior to internship. J Surg Res. 2006;133:11–15. 9. Stiggins RJ. Assessment, student confidence, and
school success. Phi Delta Kappan. 1999;81:191–198. 10. Petrosoniak A, Ryzynski A, Lebovic G, Woolfrey K.
Cricothyroidotomy In Situ Simulation Curriculum (CRIC Study): training residents for rare procedures. Simul Healthc. 2017;12:76–82. https://doi.org/ 10.1097/sih.0000000000000206. 11. Pugh CM, DaRosa DA, Glenn D, Bell RH Jr. A com-
REFERENCES 1. Prasad MS, Manivannan M, Manoharan G, Chandramo-
han SM. Objective assessment of laparoscopic force and psychomotor skills in a novel virtual reality-based haptic simulator. J Surg Educ. 2016;73:858–869. https://doi.org/10.1016/j.jsurg.2016.04.009. 2. Dempsey MS, Kauffman DF. Supporting third year med-
ical students’ skill acquisition and self-efficacy with coping models and process feedback during laparoscopic knot tying simulation. Front Psychol. 2017;8:1171. https://doi.org/10.3389/fpsyg.2017.01171. 3. Guedes HG, Ferreira ZMCC, de Sousa Leao LR, Mon-
tero EFS, Otoch JP, de Almeida Artifon EL. Virtual reality simulator versus box-trainer to teach minimally invasive procedures: a meta-analysis. Int J Surg. 2019;61:60–68. 4. Bridges M, Diamond DL. The financial impact of
teaching surgical residents in the operating room. Am J Surg. 1999;177:28–32. 5. Costa GOFD, Rocha HAL, Moura J unior LGD,
Medeiros FDC. Taxonomy of educational objectives and learning theories in the training of laparoscopic surgical techniques in a simulation environment. Revista do Col egio Brasileiro de Cirurgi~ oes. 2018;45:1–7. 6. Yamanashi J. Review of personalisation in prac-
tice: supporting young people with disabilities
Journal of Surgical Education Volume 00 /Number 00 Month 2019
parison of faculty and resident perception of resident learning needs in the operating room. J Surg Educ. 2007;64:250–255. https://doi.org/10.1016/j. jsurg.2007.07.007. 12. Piromchai P, Avery A, Laopaiboon M, Kennedy G,
O’Leary S. Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat. Cochrane Database Syst Rev. 2015: Cd010198. https://doi.org/10.1002/14651858.CD 010198.pub2. 13. Nagendran M, Gurusamy KS, Aggarwal R, Loizidou
M, Davidson BR. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. 2013:Cd006575. https://doi.org/ 10.1002/14651858.CD006575.pub3. 14. Sutherland LM, Middleton PF, Anthony A, et al. Sur-
gical simulation: a systematic review. Ann Surg. 2006;243:291–300. https://doi.org/10.1097/01.sla. 0000200839.93965.26. 15. Maertens H, Madani A, Landry T, Vermassen F, Van
Herzeele I, Aggarwal R. Systematic review of e-learning for surgical training. Br J Surg. 2016;103:1428– 1437. https://doi.org/10.1002/bjs.10236. 16. Steinemann S, Gardner A, Aulet T, Fitzgibbons S,
Campbell A, Acton R. American College of surgeons/Association for Surgical Education Medical Student Simulation-based Surgical Skills Curriculum: Alignment with Entrustable Professional Activities. Am J Surg. 2019;217:198–204.
5
ARTICLE IN PRESS 17. Heskin L, Simms C, Holland J, Traynor O, Galvin R.
A systematic review of the educational effectiveness of simulation used in open surgery. Simul Healthc. 2019;14:51–58. 18. Papanikolaou I, Haidopoulos D, Paschopoulos M, Chat-
zipapas I, Loutradis D, Vlahos N. Changing the way we train surgeons in the 21th century: a narrative comparative review focused on box trainers and virtual reality simulators. Eur J Obstet Gynecol Reprod Biol. 2019;235:13–18. 19. Bolton W, Aruparayil N, Quyn A, et al. Disseminat-
ing technology in global surgery. Br J Surg. 2019;106:e34–e43. 20. Samia H, Khan S, Lawrence J, Delaney CP. Simula-
tion and its role in training. Clin Colon Rectal
Surg. 2013;26:47–55. https://doi.org/10.1055/s0033-1333661. 21. Cendan J, Kim M, Kurenov S, Peters J. Developing a
multimedia environment for customized teaching of an adrenalectomy. Surg Endosc. 2007;21:1012–1016. 22. WISE-MD. http://wise-md.med.nyu.edu/Init.action.
Accessed April 8, 2019. 23. University of Connecticut Educational Research
Platform: UConnEdResearchCronbachAlpha. Download Accessed March 28, 2019. 24. Fisher Test. astatsa.com/fishertest. Accessed April 2,
2019. 25. T Test. https://www.socscistatistics.com/tests/stu-
dentttest/Default2.aspx. Accessed April 2, 2019.
SUPPLEMENTARY INFORMATION Supplementary material associated with this article can be found in the online version at doi:10.1016/j. jsurg.2019.08.008.
6
Journal of Surgical Education Volume 00 /Number 00 Month 2019
VR Simulation Leads to Enhanced Procedural Confidence for Surgical Trainees € rg Peters, PhD,† and Juan C. Cenda n, MD*, Heather Lesch, BS,* Evan Johnson BS,* Jo University of Central Florida College of Medicine, Orlando, Florida; and †University of Florida College of Engineering, Gainesville, Florida
*
OBJECTIVE: Active learning techniques result in greater knowledge acquisition compared to passive methods. For medical students with limited hands-on operative experiences, virtual reality platforms represent active learning and may enhance procedural training. We hypothesize that virtual reality simulators like Toolkit for Illustration of Procedures in Surgery (TIPS) are a more effective modality in teaching laparoscopic surgical techniques to medical students when compared to passive learning tools like videos.
CONCLUSIONS: Video and TIPS both enhanced different
DESIGN: In this crossover study, participants were ran-
education, laparoscopic training, surgical simulation
domly assigned to perform either a TIPS laparoscopic appendectomy followed by video of a laparoscopic cholecystectomy, or video of a laparoscopic appendectomy followed by TIPS laparoscopic cholecystectomy. A knowledge assessment followed each intervention. A postsurvey was used to gather feedback and subjective impressions of the learning experience.
aspects of student learning; however, the active TIPS platform produced greater confidence in the ability to reproduce the steps of the procedure and had greater utility as a learning strategy. Videos are simple to use and can serve a complementary role in curriculum design. ( J Surg Ed 000:1 6. Published by Elsevier Inc. on behalf of Association of Program Directors in Surgery.) KEY WORDS: surgical education, virtual reality-based COMPETENCIES: Medical Knowledge, Practice-Based
Learning and Improvement, Patient Care, Systems-Based Practice
INTRODUCTION
SETTING: University of Central Florida College of Medicine. PARTICIPANTS: Second, third, and fourth-year medical
students (n = 37). RESULTS: Validation of the content assessments revealed
strong internal consistency (Cronbach’s a = 0.73). A 2-tailed Fisher’s exact test revealed that the video had greater ease of use (p = 0.032), but TIPS had greater utility as a learning tool (p < 0.001) and instilled greater confidence in the ability to reproduce procedural steps (p < 0.001). A 2-tailed t test of the average content quiz scores revealed no significant difference in percentage correct between groups on the laparoscopic appendectomy quiz (p = 0.772), but a difference favoring video learning on the laparoscopic cholecystectomy quiz (p = 0.042) Funding Sources: This work was supported by the National Institutes of Bioengineering: Safety instruction and competency assessment for laparoscopy; NIHNIBIB RO1: EB018625. The authors have no financial interest in either TIPS or Wise-MD. Correspondence: Inquiries to Juan Cendan, MD, University of Central Florida College of Medicine, 6850 Lake Nona Blvd, Orlando, FL 32827; e-mail: [email protected]
As minimally invasive surgical techniques become more popular across all surgical specialties, there arises a concordant need to appropriately train residents and medical students to master these skills. Studies of visuospatial and psychomotor learning curves reveal distinct levels of learner aptitude for these techniques.1 Trainees, regardless of aptitude, could benefit from deliberate practice models to help hone these skills.2 Simulation is appealing as it reduces reliance on patients, cadaveric, and animal models for skills practice and acquisition. Historically, these skills have been refined under the supervision of a surgical mentor,3 but as mentorship opportunities have decreased, a larger emphasis has been placed on the use of alternative teaching modalities including textbooks, anatomical atlases, and training videos4; while valuable, these resources are limited in their ability to demonstrate nuanced information such as the texture of different types of tissue and necessary force to be exerted during dissection.5 Additionally, it has been repeatedly shown that active learning techniques result in greater knowledge acquisition and retention than
Journal of Surgical Education Published by Elsevier Inc. on behalf of Association of Program Directors in 1931-7204/$30.00 Surgery. https://doi.org/10.1016/j.jsurg.2019.08.008
1
ARTICLE IN PRESS passive methods.6,7 Confidence in one’s abilities alone has proven to enhance the learning process8,9 and can be improved by using simulations prior to the transfer of skills to actual patients. Virtual reality (VR) training can be a valuable tool in the development of technical skills while minimizing risk to patients, particularly for rare conditions that the learner might not encounter during their training.10 Given that laparoscopic surgery has a longer learning curve than open surgery, it would be beneficial to give trainees interested in procedural specialties the opportunity to develop their skills.4,11-19 As graduating residents consistently report not only decreased time in the operating room but also feeling underprepared for independent practice, it is important to provide modes of supplemental practice.16-20 We hypothesized that a VR technology-based procedural experience would complement commonly available educational videos leading to increased confidence and knowledge. Understanding the role of each of these technologies could inform curricular development efforts.
MATERIALS AND METHODS We aimed to test 2 different learning modalities handson active learning vs passive video learning. We wanted to reduce the chance that students struggled with the procedure due to inadequate knowledge of the anatomy, so participation was limited to only second to fourth year medical students at the University of Central Florida (UCF) College of Medicine who had successfully completed the first year anatomy course. Power analysis was based on an assumption of tolerance for type I a error of 0.05 and a type II error, or b power, of 80. The effect size was determined after developing the assessment. The research team concluded that a 5% difference in the assessment would yield the desired minimum required effect given the anticipated size of study cohorts. These calculations necessitated a sample size of at least 36 subjects. The Institutional Review Board for UCF gave permission (UCF IRB # SBE-18-14544). Enrollment was made by inviting all second to fourth year medical students to participate via email and posting to the class-specific Facebook page. Thirty-seven students consented to participate. Students were told the experiment would take no longer than 1 hour to complete and would be compensated for their time with a $20 giftcard. This experiment employed the use of the Toolkit for Illustration of Procedures in Surgery (TIPS) VR software. TIPS has been developed as part of a longstanding collaboration between authors (JC and JB); it is an interactive surgical illustration platform which allows the user to
2
FIGURE 1. A student participant utilizing the TIPS simulation
engage with anatomic renderings using what appears to be laparoscopic instruments.21 The user engages the environment using 2 three-dimensional (3D) haptic drawing pens, which act as facsimiles for laparoscopic instruments (see Fig. 1). Participants can choose from several common laparoscopic instrument tips (scissors, Maryland dissector, etc.) on either hand by toggling a single button. Haptic feedback is generated when the participant engages with an organ or tissue. TIPS does not provide step-wise instructions for the user; directions for procedural completion were printed showing the surgical sequence and matching screen shots taken from the TIPS program. The other learning modality utilized in this experiment is the web initiative for surgical education (WISE-MD)22 videos. These videos are used as online learning tools currently at the UCF College of Medicine, as well as other learning institutions, and display step-by-step directions on how surgical procedures are completed with real-life video paired with auditory voice-over and some animations. The laparoscopic appendectomy WISE-MD video was 1 minute 11 seconds in run-time, and the laparoscopic cholecystectomy video was 2 minutes and 30 seconds long. Students were blocked for hour-long sessions for the TIPS interaction; exact time-on-task was not measured though most sessions lasted about half hour. In this experiment, students were assigned to a group in an alternating fashion based on their selection of a timeslot. Participants completed the first intervention, post-quiz, second intervention, postquiz, and then a survey (see Fig. 2). The 2 postquizzes were developed by the authors after review of the content covered in both the TIPS simulation and the WISE-MD videos to ensure equal opportunity for correct answers across the groups. As
Journal of Surgical Education Volume 00 /Number 00 Month 2019
ARTICLE IN PRESS
FIGURE 2. Experiment flow
a form of psychometric assessment validation Cronbach’s alpha test can be viewed as the expected correlation of 2 tests that measure the same construct. It is a useful analysis of the internal reliability of an assessment; in this case the Cronbach’s alpha test demonstrated good internal reliability for the posttests (a = 0.73).23 In the survey portion of this experiment, students were asked for their feedback based on how much they liked using each learning modality, the ease of use of each, and other subjective questions based on a 5-point Likert scale. There was also a section where participants could provide any comments or suggestions. Statistical analysis included descriptive analysis of demographics, as well as a 2-tailed Fisher’s test for evaluation of 5-point Likert scale data.24 A 2-tailed t test was utilized to analyze the postquiz data.25
RESULTS For a total of 37 students, there were 19 in Group A and 18 in Group B. 67.6% of participants were male, and there were 15 second year students (40.5%), 8 third year students (21.6%), and 14 fourth year students (37.8%). Roughly a third of the students (13 students, 35.1%) reported they intend to pursue a surgical career. 62.1% of participants had already completed a surgical rotation during their third year and 67.6% had seen a laparoscopic appendectomy or laparoscopic cholecystectomy in person. Participants performed overall well on the postappendectomy quiz with no significant difference (81.6% for TIPS vs 83.3% for WISE-MD, p = 0.772). However, the postcholecystectomy quiz did demonstrate a difference in favor of the WISE-MD videos (84.2% vs 76.6%, p = 0.042).
Journal of Surgical Education Volume 00 /Number 00 Month 2019
Overall, students favored the utility of TIPS as a learning tool compared to the videos (p = 0.009; Fig. 3). This effect was particularly pronounced in the case of the laparoscopic cholecystectomy (p = 0.001). However, participants had no significant preference between the 2 modalities for the laparoscopic appendectomy (p = 0.536). Students’ confidence in reproducing the procedure was assessed (Fig. 3). Overall students reported higher confidence after completing the TIPS simulation (p = 0.001). The overall effect was, again, largely driven by the large impact shown by TIPS in laparoscopic cholecystectomy training (p = 0.002), with less effect noted in the laparoscopic appendectomy scenario (p = 0.146). As a summary question, participants were asked to report which modality they would prefer learning surgical techniques from and 83.8% (31 of 37) reported they would prefer using the TIPS simulation. Assessment of ease of use of the technology revealed a preference towards WISE-MD (p = 0.032; Fig. 3). Specifically, participants reported that the WISE-MD video was easier to use than the laparoscopic appendectomy TIPS simulation (p = 0.004); there was no significant difference in the reported ease of use for the laparoscopic cholecystectomy TIPS simulation (p = 0.507). It is worth noting that 43.2% of participants reported having technical difficulties while using the TIPS simulation. Research staff was available at all times during the investigation to intervene in typically minor technical issues.
CONCLUSIONS In this study, we aimed to examine the TIPS simulation software and evaluate its potential place in the learning environment for surgical trainees. When asked whether participants thought the TIPS simulation had good utility
3
ARTICLE IN PRESS
Ulity
Confidence
Ease of Use
25 20 15
10 5 0
TIPS
WISEMD
FIGURE 3. Survey questions
as a learning tool and whether they could confidently verbally reproduce the procedural steps after using the simulation, collectively the response was in favor of the TIPS simulation. Not only did the TIPS simulation appear to be a useful adjunct to the learning environment, but it improved student confidence. Confidence, itself, is a vital component to the learning process.8,9 Educators commonly seek ways not only to improve testable knowledge, but improve learners’ confidence in their skills. Participants felt more confident in their ability to verbally reproduce procedural steps after using the TIPS simulation, but this did not translate directly into improved postquiz scores. In fact, there was no difference for the laparoscopic appendectomy postquiz, while a meaningful difference was noted in the laparoscopic cholecystectomy postquiz in favor of the WISEMD videos. The videos were roughly 1.5 to 2.5 minutes long and paired not only reallife visuals with animations, but had a matching voice-over, pairing auditory and visual learning. The TIPS simulation paired only visual and kinesthetic learning, where participants followed step-by-step written directions with screen shots from the simulation as guide posts. Interestingly, the shorter video (laparoscopic appendectomy) did not have significant differences in post-quiz scores, while the longer video (laparoscopic cholecystectomy) did, which could demonstrate that the longer the auditory and visual reinforcement, the more likely you were to perform better on knowledge-based questions. During medical school, students’ education is nearly entirely auditory and visual for the first 2 years of the curriculum, so students are far more used to learning from these methods when compared to the kinesthetic learning they begin experiencing as they progress through their third year. The TIPS
4
simulation software could be improved by integrating auditory learning with the other 2 modalities. This could be accomplished by adding spoken instructions that triggered at the completion of the previous procedural step. With nearly half of the participants experiencing some form of technical difficulty while using the TIPS simulation, it comes as no surprise that participants reported the WISEMD videos as easier to use. Again, though, when broken down by individual procedure, there was not a significant difference when comparing ease-of-use between WISEMD videos and the TIPS simulation for the laparoscopic cholecystectomy procedure. We experienced some difficulties with platform stability of the new software, as well as instrument calibration and orientation, which likely reflects these results. The laparoscopic appendectomy TIPS procedure, especially, proved more technically challenging to participants. There are improvements that need to be made to the TIPS simulation software to improve usability and decrease technical difficulties, despite these participants still preferred learning via the TIPS simulation. Also collected was free text written feedback from participants. Common themes derived from these comments include technical difficulties, the learning curve associated with the TIPS simulation, enjoyment with using the TIPS simulation, and preference for combination of the WISEMD videos with the simulation. One participant reported that the video moved too quickly through the steps for adequate knowledge retention, so they relied on previous knowledge of the procedure to answer the postquiz questions. This brings up an interesting potential confounding factor for the results of the post-quizzes but does not change that participants generally preferred utilizing the TIPS simulation as a learning tool. Many reported that the video was very concise, but
Journal of Surgical Education Volume 00 /Number 00 Month 2019
ARTICLE IN PRESS they would have liked to have been able to repeat it for better knowledge solidification. Overall, students enjoyed using the TIPS simulation once they overcame the learning curve. A vital point echoed by many participants was the suggestion to integrate both modalities: “maybe getting to read through the steps, then watching a video, and then using the simulator would have been most effective.” Through this experiment we evaluated a new learning tool for hands-on teaching of laparoscopic techniques when compared to traditional passive learning from a video. There are clearly merits to both methods of learning and the greatest improvement on learning, confidence, and eventually performance may be the integration of the 2 instead of either individually.
ACKNOWLEDGMENTS WISE-MD, NYU School of Medicine.
through the transition to adulthood. Int J Disabil Dev Educ. 2015;62:137–139. https://doi.org/ 10.1080/1034912X.2014.985026. 7. Lin C, Gao J, Zheng H, et al. When to introduce
three-dimensional visualization technology into surgical residency: a randomized controlled trial. J Med Syst. 2019;43:71. 8. Peyre SE, Peyre CG, Sullivan ME, Towfigh S. A surgi-
cal skills elective can improve student confidence prior to internship. J Surg Res. 2006;133:11–15. 9. Stiggins RJ. Assessment, student confidence, and
school success. Phi Delta Kappan. 1999;81:191–198. 10. Petrosoniak A, Ryzynski A, Lebovic G, Woolfrey K.
Cricothyroidotomy In Situ Simulation Curriculum (CRIC Study): training residents for rare procedures. Simul Healthc. 2017;12:76–82. https://doi.org/ 10.1097/sih.0000000000000206. 11. Pugh CM, DaRosa DA, Glenn D, Bell RH Jr. A com-
REFERENCES 1. Prasad MS, Manivannan M, Manoharan G, Chandramo-
han SM. Objective assessment of laparoscopic force and psychomotor skills in a novel virtual reality-based haptic simulator. J Surg Educ. 2016;73:858–869. https://doi.org/10.1016/j.jsurg.2016.04.009. 2. Dempsey MS, Kauffman DF. Supporting third year med-
ical students’ skill acquisition and self-efficacy with coping models and process feedback during laparoscopic knot tying simulation. Front Psychol. 2017;8:1171. https://doi.org/10.3389/fpsyg.2017.01171. 3. Guedes HG, Ferreira ZMCC, de Sousa Leao LR, Mon-
tero EFS, Otoch JP, de Almeida Artifon EL. Virtual reality simulator versus box-trainer to teach minimally invasive procedures: a meta-analysis. Int J Surg. 2019;61:60–68. 4. Bridges M, Diamond DL. The financial impact of
teaching surgical residents in the operating room. Am J Surg. 1999;177:28–32. 5. Costa GOFD, Rocha HAL, Moura J unior LGD,
Medeiros FDC. Taxonomy of educational objectives and learning theories in the training of laparoscopic surgical techniques in a simulation environment. Revista do Col egio Brasileiro de Cirurgi~ oes. 2018;45:1–7. 6. Yamanashi J. Review of personalisation in prac-
tice: supporting young people with disabilities
Journal of Surgical Education Volume 00 /Number 00 Month 2019
parison of faculty and resident perception of resident learning needs in the operating room. J Surg Educ. 2007;64:250–255. https://doi.org/10.1016/j. jsurg.2007.07.007. 12. Piromchai P, Avery A, Laopaiboon M, Kennedy G,
O’Leary S. Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat. Cochrane Database Syst Rev. 2015: Cd010198. https://doi.org/10.1002/14651858.CD 010198.pub2. 13. Nagendran M, Gurusamy KS, Aggarwal R, Loizidou
M, Davidson BR. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. 2013:Cd006575. https://doi.org/ 10.1002/14651858.CD006575.pub3. 14. Sutherland LM, Middleton PF, Anthony A, et al. Sur-
gical simulation: a systematic review. Ann Surg. 2006;243:291–300. https://doi.org/10.1097/01.sla. 0000200839.93965.26. 15. Maertens H, Madani A, Landry T, Vermassen F, Van
Herzeele I, Aggarwal R. Systematic review of e-learning for surgical training. Br J Surg. 2016;103:1428– 1437. https://doi.org/10.1002/bjs.10236. 16. Steinemann S, Gardner A, Aulet T, Fitzgibbons S,
Campbell A, Acton R. American College of surgeons/Association for Surgical Education Medical Student Simulation-based Surgical Skills Curriculum: Alignment with Entrustable Professional Activities. Am J Surg. 2019;217:198–204.
5
ARTICLE IN PRESS 17. Heskin L, Simms C, Holland J, Traynor O, Galvin R.
A systematic review of the educational effectiveness of simulation used in open surgery. Simul Healthc. 2019;14:51–58. 18. Papanikolaou I, Haidopoulos D, Paschopoulos M, Chat-
zipapas I, Loutradis D, Vlahos N. Changing the way we train surgeons in the 21th century: a narrative comparative review focused on box trainers and virtual reality simulators. Eur J Obstet Gynecol Reprod Biol. 2019;235:13–18. 19. Bolton W, Aruparayil N, Quyn A, et al. Disseminat-
ing technology in global surgery. Br J Surg. 2019;106:e34–e43. 20. Samia H, Khan S, Lawrence J, Delaney CP. Simula-
tion and its role in training. Clin Colon Rectal
Surg. 2013;26:47–55. https://doi.org/10.1055/s0033-1333661. 21. Cendan J, Kim M, Kurenov S, Peters J. Developing a
multimedia environment for customized teaching of an adrenalectomy. Surg Endosc. 2007;21:1012–1016. 22. WISE-MD. http://wise-md.med.nyu.edu/Init.action.
Accessed April 8, 2019. 23. University of Connecticut Educational Research
Platform: UConnEdResearchCronbachAlpha. Download Accessed March 28, 2019. 24. Fisher Test. astatsa.com/fishertest. Accessed April 2,
2019. 25. T Test. https://www.socscistatistics.com/tests/stu-
dentttest/Default2.aspx. Accessed April 2, 2019.
SUPPLEMENTARY INFORMATION Supplementary material associated with this article can be found in the online version at doi:10.1016/j. jsurg.2019.08.008.
6
Journal of Surgical Education Volume 00 /Number 00 Month 2019