- Email: [email protected]
Curriculum Using the In-Situ Operating Room Setting
2017 APDS SPRING MEETING
Curriculum Using the In-Situ Operating Room Setting Raghavendra Rao, MD,* Robert C. Caskey, MD,* Lily Owei, BA,* Kathleen O’Connor, MD,* Elijah Riddle, MD,* Daniel T. Dempsey, MD,* Joshua Atkins, MD,† Dimitry Baranov, MD,† Gregory Motuk, RN,‡ Ari D. Brooks, MD,* Noel Williams, MD,* Jon Morris, MD,* and Kristoffel Dumon, MD* Department of Surgery, Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania; †Department of Anesthesia, Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania; and ‡Penn Clinical Simulation Center, Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania *
OBJECTIVE: The American College of Surgeons/Associa-
PARTICIPANTS: We performed a total of 8 sessions, for a
tion of Program Directors in Surgery is a comprehensive, simulation-based curriculum for General Surgery residents which exists in 3 phases. While phases 1 and 2 deal with core skills and advanced procedures respectively, phase 3 targets team-based skills. To date, the 3rd phase of this curriculum has not seen wide scale implementation. This is a pilot study to verify the feasibility of implementing the phase 3 curriculum in the in-situ setting.
total of 15 scenarios. Eight surgery residents at the postgraduate year 1 (PGY1)-PGY3 level, 16 anesthesia residents at the PGY3-PGY4 level, 16 nurses and 13 ancillary staff participated.
DESIGN: In our initial attempt to implement Phase 3 at
our institution, we chose to perform the training in an insitu setting within an operating room (OR) at our main hospital, despite our having a separate simulation center. By choosing the in-situ OR environment for this training we were able to minimize concerns regarding resident and faculty availability and able to successfully complete 8 separate sessions during the academic year. During 7 sessions, 2 separate scenarios were performed while a single scenario was performed in 1 session. This single session was excluded from analysis, leaving a total of 14 scenarios to evaluate. The unique scenarios included laparoscopic crisis, postoperative myocardial infarction, anaphylaxis, and postoperative hypotension. All sessions were audiovisually recorded. In order to evaluate the effect of the training, the videos were viewed by 3 independent reviewers and all surgery, anesthesia and nursing participants were rated using the NOTECHs II scale. Degree of inter-rater agreement was established. The difference between the first and second simulations on the same day was then assessed. In addition, participant opinions of the simulations were assessed through electronic surveys following the training. SETTING: Tertiary Care University Hospital.
RESULTS: From the first to the second scenario, the total team NOTECHs II score increased from 69.4 ± 1.4 to 77.3 ± 0.5 (p ¼ 0.007). The NOTECHs II scores for each subteam also improved, from 24.2 ± 0.6 to 26.4 ± 0.5 (p ¼ 0.007) for surgery residents, 23.7 ± 0.9 to 26.7 ± 0.4 (p ¼ 0.03) for anesthesia, and 21.6 ± 0.3 to 24.3 ± 0.5 (p ¼ 0.01) for nursing. The inter-rater reliability as measured by Kendall’s coefficient of concordance was modest for the whole team score. Most of the participant responses were either favorable or strongly favorable. CONCLUSION: The in-situ OR environment is both a
unique and effective setting to perform team-based training. Furthermore, training in the in-situ setting minimizes or removes many of the logistic issues involved in designing and implementing team-based training curricula for general surgery residency programs. However, we found that administrative and departmental (surgery, anesthesia, and nursing) “buy in” as well as protected faculty time for education were all necessary for in-situ training to be successful. NOTECHs II is an established scale for the evaluation of teams in this simulation setting and appears to be a valid tool based on the results of this study. However, further assessment of interrater reliability as well as improved training of evaluators are necessary to determine if inter-rater reliability can improve. C 2017 Published by Elsevier Inc. on ( J Surg Ed ]:]]]-]]]. J behalf of the Association of Program Directors in Surgery) KEY
WORDS: in-situ simulation, NOTECHs II, interpersonal skills, professionalism, APDS curriculum
Correspondence: Inquiries to Kristoffel Dumon, MD, FACS, Department of Surgery, 4 Silverstein, 3400 Spruce Street, Philadelphia, PA 19104; e-mail: Kristoffel. [email protected]
COMPETENCIES: Practice Based Learning and Improve-
ment, System Based Practice
Journal of Surgical Education & 2017 Published by Elsevier Inc. on behalf of the 1931-7204/$30.00 Association of Program Directors in Surgery http://dx.doi.org/10.1016/j.jsurg.2017.09.024
1
INTRODUCTION Nontechnical skills are critical to safety in the operating room (OR), and have been demonstrated to significantly impact surgical outcomes.1 Communication breakdowns have been repeatedly identified as a major cause of issues related to patient safety.2,3 In recent years, such concerns over patient safety have led to an increased emphasis on interdisciplinary and interprofessional teamwork skills during surgical training.4 During this same time period, simulation-based training has also increased in popularity as a means to provide technical and nontechnical skills training.5-7 Accordingly, the American College of Surgeons (ACS) and the Association of Program Directors in Surgery (APDS) have created a curriculum, which serves as a guideline for training surgery residents through simulation. While phases 1 and 2 deal with core skills and advanced procedures, respectively, phase 3 targets team-based skills.8 Specifically, phase 3 is a modular curriculum covering 10 scenarios in which cooperation between the team and a surgeon is necessary for a successful patient outcome. The third phase of the curriculum also includes suggestions on how to develop and implement simulated cases for resident team skills education and assessment. By providing a controlled environment, phase 3 of the ACS/APDS Surgery Residents Skills curriculum plays an important role in the objective assessment of 3 of the critical core competencies set forth by the ACGME,4 namely communication and interpersonal skills, professionalism, and system-based practice. Since the training is conducted in a controlled, safe environment, it also allows for errors to be committed while giving the opportunity for remediation outside the realm of clinical care. Despite the availability of this fully developed and standardized curriculum it has not seen widespread implementation to date. There are many potential reasons for this including a lack of awareness regarding the existence of the curriculum, the cost of implementation, the lack of trained faculty, and perhaps most importantly the lack of protected faculty and resident time.4,5,9 Specifically for phase 3, program directors have identified the challenges related to coordinating team-based training as a major obstacle to its implementation.5 However with the increasing accumulation of evidence demonstrating the usefulness of team training as well as other nontechnical skills training, the attitudes regarding such training will need to change.10 We recently designed a practical way of implementing phase 3 of the ACS/ADPS curriculum using an actual OR for team training. This in-situ environment was chosen for many reasons. Firstly, it obviates the need for a dedicated simulation center with its associated costs. Also, and more importantly, it increases resident and faculty availability because they are already on the site at the time of simulation. For these and more reasons, we believe that this training model will be useful for academic and 2
community surgery programs alike. In this paper, we present our experience to date with in-situ OR simulations and also present the resident/participant opinions on the usefulness of these simulations. As this was a pilot study of in-situ OR simulation training, we also chose to evaluate residents’ nontechnical skills to gauge the usefulness of the training. To do this we used a well-established tool for evaluating teamwork, the NOTECHS II scale, and hoped to demonstrate the validity of this instrument for use in this setting.11-14
METHODS We performed 8 sessions of in-situ simulation with 7 of the sessions involving 2 scenarios and 1 session involving only 1 scenario. This single scenario was excluded from data analysis. In order to prevent disruption to the OR schedule and minimize indirect costs, all simulations were performed “first case” in the morning in an OR that is reserved for add on cases and which was unstaffed at the time of the simulations. This was done with the permission and support of both the surgery and anesthesia departments as well as perioperative services. All participants chosen to participate in the training were already scheduled to be on site the day of the training and were informed of the training at least a month ahead of time. There were a total of 53 participants: 8 general surgery residents at the postgraduate year 1 (PGY1)-PGY3 level, 16 anesthesia residents at the PGY3PGY4 level, 16 circulating OR nurses, and 13 anesthesia technicians (Table 1). During each session, participants were put into teams consisting of a surgical resident, 2 anesthesia residents, 2 circulating nurses, and 1 or 2 anesthesia technicians. Based on their particular session, participants were exposed to 1 or 2 of 4 distinct scenarios that were chosen because we felt they would maximize the need for proper teamwork and communication: rapid exsanguination, air embolus due to laparoscopic trocar, patient losing vitals while performing prone position surgery and postoperative myocardial infarction with hypotension following laparoscopic cholecystectomy. We utilized the SimMan 4G (Laerdel, NY) for all scenarios. All other equipment were the same as those used in a standard OR. Each scenario was followed by systematic debriefing using crisis resource management (CRM) techniques.15 Debriefing was performed by surgery and anesthesia faculty and an experienced simulation educator. It TABLE 1. Participant Breakdown Category General surgery residents Anesthesia residents OR nurses Anesthesia technicians Total
Number 8 16 16 13 (not included in evaluation) 53
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
TABLE 2. Opinion Survey Questions Question
% Positive Response
This training will help to improve patient care The debriefings were useful I prefer in-situ team training to conventional simulation center-based training The training scenarios simulated my actual clinical experience This training helped to improve my leadership skills This training helped to improve my teamwork skills
consisted of 10 minutes of self-reflection by participants, followed by 10 minutes of guided discussion by the faculty and 5 minutes to summarize areas for improvement. Videos were recorded and stored on B-line (B-Line Medical, Washington, DC, USA). These were then reviewed by 3 independent raters, all of whom attended an intensive 2-day course on nontechnical skills training and received approximately 6 hours of training in evaluating nontechnical skills using the NOTECHs II scale. Each of the 3 subteams, namely anesthesia, nursing and surgery were evaluated using the standardized NOTECHs II scale. The difference in each team’s total average score for the first and second scenario was evaluated using a paired t-test. Inter-rater reliability was calculated using the intra-class correlation coefficient. All values were expressed using mean ± standard error. p o 0.05 was regarded as significant. Following the training participants’ opinions regarding the simulations were recorded using an electronic survey (Table 2). In this survey the participants were asked to respond to the summarized statements in Table 2 with one of the following options: Strongly Agree, Agree, Neutral, Disagree, or Strongly Disagree. A positive response was noted for any response that was either strongly agree or agree.
RESULTS There was significant improvement in each team’s total NOTECHs II score between the first and second scenarios (69.4 ± 1.4 and 77.3 ± 0.5, p ¼ 0.007) (Fig. 1). The surgery resident (24.2 ± 0.6 and 26.4 ± 0.5, p ¼ 0.007), anesthesia resident (23.7 ± 0.9 and 26.7 ± 0.4, p ¼ 0.03), and nursing staff (21.6 ± 0.3 and 24.3 ± 0.5, p ¼ 0.01)
FIGURE 1. Total NOTECHs score for scenarios 1 and 2.
91.7 87.5 96.4 83.3 75.0 66.7
subteams also demonstrated significant improvements in NOTECHs II score from the first to the second scenarios (Fig. 2). The Kendall’s coefficient of concordance was modest but statistically significant for the overall team score (W ¼ 0.57, p ¼ 0.05). For the subteams, it was statistically significant for only the nursing subteam (W ¼ 0.66. p ¼ 0.01), while it was strongly trending for surgery (W ¼ 0.54, p ¼ 0.07) and anesthesia (W ¼ 0.52, p ¼ 0.09) subteams. Participant survey responses (Table 2) demonstrated that 91.7% of participants felt that the training would help improve patient care, 87.5% felt that the debriefings were useful, 96.4% of the participants favored in-situ over conventional simulation center-based team training and 83.3% felt the training scenarios simulated their actual clinical experience. In all, 75% felt that the training helped improve their leadership skills and 66.7% felt that the training improved their teamwork skills (Table 2).
DISCUSSION The ACS/APDS curriculum16 is a comprehensive curriculum consisting of 3, modular phases covering basic surgical skills, advanced skills and procedures, and team-based training for mid to senior level residents. While the level of residency during which this training is supposed to occur is not specified, it is probably ideal that residents become comfortable with these skills during or before their 3rd year of training. This will allow for the training to be completed before the residents take on any significant leadership roles and let them spend the last 2 years of residency focusing on their clinical training. Overall, the implementation rates of
FIGURE 2. NOTECHs score by subteam for scenarios 1 and 2.
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
3
the ACS/APDS curriculum into residency programs has been poor despite the introduction of phase I in 2007, phase II in 2008, and phase III in 2009.5 In a 2013 survey of program directors, implementation rates were 36% for phase I, 19% for phase II, and 16% for phase III. Over 50% of respondents identified lack of faculty protected time, limited personnel, significant costs, and resident workhour restrictions as major obstacles9 to implementation. Since the penetration of the phase III curriculum has been the lowest,5 we elected to implement this training using an in-situ setting to try and overcome most if not all of the above listed obstacles. The coordination of team-based training, which requires the presence of multidisciplinary participants, has been identified as the greatest barrier to the implementation of the phase III curriculum.5 In-situ simulation has the ability to make such coordination easier because both the faculty and residents are at least already present on site. However, it must be stressed that in order for this to work there must be “buy in” on both the departmental and administrative levels. For our sessions, we were given use of the OR by perioperative services. We also had the support of the surgery and anesthesia departments. With such support we were able to choose residents and nursing staff who would be on site anyway and then inform them of the training at least a month beforehand. This allowed for clinical coverage to be arranged and in doing so minimized concerns regarding resident duty hours. Finally, the faculty who were involved in the scenarios and debriefing sessions consisted mostly, but not entirely, of those involved in the project and/or which have protected educational time. Using the in-situ environment for simulation training can also, at least theoretically, lead to cost savings especially when compared to the cost of staffing and running a fulltime simulation center. Indeed, we undertook in-situ training despite our having such a simulation center. We were able to do this partly because within our health system we have extra ORs, which are fully equipped, but not necessarily staffed. These rooms are usually used to accommodate extra volume on days with a very busy OR schedule. Again reflective of the need for departmental and administrative “buy in,” we were able to schedule our simulation sessions early in the morning on a day of the week (i.e., Wednesday) when our operative volume is traditionally lowest. As to the specifics of OR costs, there are no published formal data on true OR costs. It most certainly differs by institution, specialty, and country.17 Excluding physician costs, OR administrators sometimes use $15-$20 per OR minute to estimate cost for a basic surgical procedure, with at least half of that figure being fixed overhead costs.18 A more recent European study looking at examples of cost calculations for an OR and a post-anesthesia care unit quotes the indirect costs as being as high as 25% of total OR cost.19 The real variable costs depend in large part on how the OR staff is paid (e.g., hourly or salaried). Therefore, as the ORs we 4
used for these simulations were not staffed and since it is difficult/artificial to separate out the indirect cost of running a single OR for 45-90 minutes (1-2 scenarios) from the indirect costs for running over 50 other ORs we found the indirect costs of this training to negligible. The objective evaluation of the participants’ learning of nontechnical skills is a critical part of team-based training. Several validated rating scales have been developed to do this. Some are designed to evaluate specific groups such as NOTSS (nontechnical skills for surgeons) for surgeons,12 ANTS (anesthetists nontechnical skills) for anesthesiologists,20 and SPLINTS (Scrub Practitioners’ List of Intraoperative Non-Technical Skills ) for the scrub-nurses.21 Other scales are better utilized to evaluate teamwork. These include OTAS (Observational Teamwork Assessment for Surgery),22 the Clinical Teamwork Scale (CTS),23 the Mayo High Performance Teamwork Scale,24 and NOTECHs. Of these, only NOTECHs is a generic rating scale that can be used to assess both the team and subteams in multiple clinical environments and has a published high inter-rater reliability.11 However, as the original NOTECHs had a decreased ability to differentiate teams whose performance was not extreme, it was restructured to NOTECHs II in 2014.11 The NOTECHs II scale has improved scalability, and allows a better understanding on the impact of quality improvement interventions.13 For these reasons we chose NOTECHS II for use in our study.12-14 In-situ simulation has been extensively studied previously but not within the context of implantation of phase 3 of the ACS/APDS curriculum.1,6,7,25-28 Phitayakorn et al.26,27 performed a series of 5 simulations using surgery residents and evaluated the residents and other participants using NOTSS, ANTS, and SPLINTS. They found that technical and nontechnical skills did not correlate and suggested that both need to be trained together. They also reported poor inter-rater reliability for these scales in the absence of rater training. Interestingly, Nicksa et al. combined technical and nontechnical skills training in the OR, emergency department and intensive care unit and found that only the PGY2 (but not PGY1) residents improved when the technical and nontechnical skills were assessed at the beginning and the end of the academic year. Systematic debriefing, similar to our study was used in this study, but the effect of other factors on the participants’ nontechnical skills acquisition throughout the academic year is difficult to separate out.28 This particular study also used the NOTECHs II scale for the evaluation of nontechnical skills. Another study by Steinemann et al.29 implemented in-situ training for trauma resuscitation and found that there was a significant improvement in both nontechnical and technical skills in both the simulation and the clinical environments. An extensive review of in-situ OR simulation was performed by our group earlier this year.30 In it we concluded that in-situ simulation offers unique advantages compared to simulation
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
center-based training. This is especially true in relation to identification of system wide safety issues and threats. However, the efficacy of in-situ training has yet to be fully elucidated. In this pilot study, the nontechnical skills of each team improved following systematic debriefing. This was also true for each subteam. Overall the participants felt favorably regarding the simulations and the majority of the participants also preferred in-situ training to simulation centerbased training. We were also able to show that the nontechnical skills acquired were not scenario dependent as they showed improvement when distinct “pre” and “post” scenarios were used. The study was not without limitations however. The inter-rater reliability of the NOTECHs II scale was modestly significant for the total team score and the nursing subteam scores and was only strongly trending but not statistically significant for other subteams. This is likely reflective of the experience of our evaluators and not a problem inherent to the use of the NOTECHs II scale in this setting. However, further assessment of inter-rater reliability in this setting as well as an extensive review of our training of evaluators are necessary to determine if inter-rater reliability can actually improve.
3. Gawande AA, Zinner MJ, Studdert DM, Brennan TA.
Analysis of errors reported by surgeons at three teaching hospitals. Surgery. 2003;133(6):614-621. 4. Hull L, Arora S, Stefanidis D, Sevdalis N. Facilitating
the implementation of the American College of Surgeons/Association of Program Directors in Surgery phase III skills curriculum: training faculty in the assessment of team skills. Am J Surg. 2015;210 (5):933-941 e2. 5. Korndorffer JR, Arora S, Sevdalis N, Paige J, et al. The
American College of Surgeons/Association of Program Directors in Surgery National Skills Curriculum: adoption rate, challenges and strategies for effective implementation into surgical residency programs. Surgery. 2013;154(1):13-20. 6. Wetzel CM, Black SA, Hanna GB, et al. The effects of
stress and coping on surgical performance during simulations. Ann Surg. 2010;251(1):171-176. 7. Paige JT, Kozmenko V, Yang T, Paragi Gururaja R,
et al. High-fidelity, simulation-based, interdisciplinary operating room team training at the point of care. Surgery. 2009;145(2):138-146. 8. Scott DJ, Dunnington GL. The new ACS/APDS Skills
CONCLUSION Implementation of the ACS phase 3 simulation curriculum in the in-situ setting was favorably received by the participants and resulted in a significant improvement in nontechnical skills following systematic debriefing between scenarios. However, we found that administrative and departmental (surgery, anesthesia, and nursing) “buy in” as well as protected faculty time for education were essential to making the training successful. NOTECHs II is a wellestablished scale for the evaluation of teams in this simulation setting and appears to be a reliable and valid tool based on the results of this study. However, inter-rater reliability of the scale needs to be further evaluated in this setting.
Curriculum: moving the learning curve out of the operating room. J Gastrointest Surg. 2008;12(2):213-221. 9. Pentiak
PA, Schuch-Miller D, Streetman RT, Marik K, et al. Barriers to adoption of the surgical resident skills curriculum of the American College of Surgeons/Association of Program Directors in Surgery. Surgery. 2013;154(1):23-28.
10. Hull L, Sevdalis N. Advances in teaching and assessing
nontechnical skills. Surg Clin N Am. 2015;95(4): 869-884. 11. Sevdalis N, Davis R, Koutantji M, Undre S, et al.
Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg. 2008;196(2):184-190. 12. Wood TC, Raison N, Haldar S, Brunckhorst O, et al.
Training tools for nontechnical skills for surgeons—a systematic review. J Surg Educ. 2017;74(4):548-578. 13. Tyack PL, Calambokidis J, Friedlaender A, Goldbogen
REFERENCES 1. Arriaga AF, Gawande A, Raemer DB, et al. Pilot
testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams. Ann Surg. 2014;259(3):403-410. 2. Arriaga AF, Bader AM, Wong JM, Lipsitz SR, et al.
Simulation-based trial of surgical-crisis checklists. N Engl J Med. 2013;368(3):246-253.
J, et al. Oxford NOTECHS II: a modified theatre team non-technical skills scoring system. PLoS One. 2014;9(3):e90320. 14. Mishra A, Catchpole K, McCulloch P. The Oxford
NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care. 2009;18(2):104-108. 15. Blum RH, Raemer DB, Carroll JS, Sunder N. Crisis
resource management training for an anaesthesia
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
5
faculty: a new approach to continuing education. Med Educ. 2004;38(1):45-55. 16. American College of Surgeons. ACS/APDS Surgery
Resident Skills Curriculum. 2017. Available at: https:// www.facs.org/education/program/resident-skills. 17. Macario A. What does one minute of operating room
time cost? J Clin Anesthesia. 2010;22(4):233-236. 18. Park KW DC. Can efficient supply management in
the operating room save millions. Curr Opin Anaesthesiol. 2009;22(2):242-248. 19. Raft J, Millet F, Meistelman C. Example of cost
calculations for an operating room and a post-anaesthesia care unit. Anaesth Crit Care Pain Med. 2015;34 (4):211-215. 20. Flin R, Patey R. Non-technical skills for anaesthetists:
developing and applying ANTS. Best Pract Res Clin Anaesthesiol. 2011;25(2):215-227.
24. Malec JF TL, Dunn WF, Wiegmann DA, Arnold JJ,
Brown DA, Phatak V. The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc. 2007;2(1):4-10. 25. Powers KA1, Rehrig ST, Irias N, Albano HA, et al.
Simulated laparoscopic operating room crisis: an approach to enhance the surgical team performance. Surg Endosc. 2008;22(4):885-900. 26. Phitayakorn R MR, Pian-Smith MC, Hemingway MW,
Milosh-Zinkus T, Oriol-Morway D, Petrusa E. Practicality of intraoperative teamwork assessments. J Surg Res. 2014;190(1):22-28. 27. Phitayakorn R, Minehart RD, Hemingway MW, Pian-
Smith MC, Petrusa E. The relationship between intraoperative teamwork and management skills in patient care. Surgery. 2015;158(5):1434-1440. 28. Nicksa GA, Anderson C, Fidler R, Stewart L. INno-
of the scrub practitioner: the SPLINTS system. ORNAC J. 2014;32(3):33-38.
vative approach using interprofessional simulation to educate surgical residents in technical and nontechnical skills in high-risk clinical scenarios. JAMA Surg. 2015;150(3):201-207.
22. Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N.
29. Steinemann S, Berg B, Skinner A, DiTulio A, et al. In
21. Flin R, Mitchell L, McLeod B. Non-technical skills
Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg. 2011;212(2):234-243 e5.
situ, multidisciplinary, simulation-based teamwork training improves early trauma care. J Surg Educ. 2011;68(6):472-477.
23. Guise JM, Deering S, Kanki BG, et al. Validation of a
30. Owei L, Neylan CJ, Rao R, Caskey RC, et al. In situ
tool to measure and promote clinical teamwork. Simul Healthc. 2008;3(4):217-223.
operating room-based simulation: a review. J Surg Educ. 2017;74(4):579-588.
6
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
Curriculum Using the In-Situ Operating Room Setting Raghavendra Rao, MD,* Robert C. Caskey, MD,* Lily Owei, BA,* Kathleen O’Connor, MD,* Elijah Riddle, MD,* Daniel T. Dempsey, MD,* Joshua Atkins, MD,† Dimitry Baranov, MD,† Gregory Motuk, RN,‡ Ari D. Brooks, MD,* Noel Williams, MD,* Jon Morris, MD,* and Kristoffel Dumon, MD* Department of Surgery, Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania; †Department of Anesthesia, Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania; and ‡Penn Clinical Simulation Center, Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania *
OBJECTIVE: The American College of Surgeons/Associa-
PARTICIPANTS: We performed a total of 8 sessions, for a
tion of Program Directors in Surgery is a comprehensive, simulation-based curriculum for General Surgery residents which exists in 3 phases. While phases 1 and 2 deal with core skills and advanced procedures respectively, phase 3 targets team-based skills. To date, the 3rd phase of this curriculum has not seen wide scale implementation. This is a pilot study to verify the feasibility of implementing the phase 3 curriculum in the in-situ setting.
total of 15 scenarios. Eight surgery residents at the postgraduate year 1 (PGY1)-PGY3 level, 16 anesthesia residents at the PGY3-PGY4 level, 16 nurses and 13 ancillary staff participated.
DESIGN: In our initial attempt to implement Phase 3 at
our institution, we chose to perform the training in an insitu setting within an operating room (OR) at our main hospital, despite our having a separate simulation center. By choosing the in-situ OR environment for this training we were able to minimize concerns regarding resident and faculty availability and able to successfully complete 8 separate sessions during the academic year. During 7 sessions, 2 separate scenarios were performed while a single scenario was performed in 1 session. This single session was excluded from analysis, leaving a total of 14 scenarios to evaluate. The unique scenarios included laparoscopic crisis, postoperative myocardial infarction, anaphylaxis, and postoperative hypotension. All sessions were audiovisually recorded. In order to evaluate the effect of the training, the videos were viewed by 3 independent reviewers and all surgery, anesthesia and nursing participants were rated using the NOTECHs II scale. Degree of inter-rater agreement was established. The difference between the first and second simulations on the same day was then assessed. In addition, participant opinions of the simulations were assessed through electronic surveys following the training. SETTING: Tertiary Care University Hospital.
RESULTS: From the first to the second scenario, the total team NOTECHs II score increased from 69.4 ± 1.4 to 77.3 ± 0.5 (p ¼ 0.007). The NOTECHs II scores for each subteam also improved, from 24.2 ± 0.6 to 26.4 ± 0.5 (p ¼ 0.007) for surgery residents, 23.7 ± 0.9 to 26.7 ± 0.4 (p ¼ 0.03) for anesthesia, and 21.6 ± 0.3 to 24.3 ± 0.5 (p ¼ 0.01) for nursing. The inter-rater reliability as measured by Kendall’s coefficient of concordance was modest for the whole team score. Most of the participant responses were either favorable or strongly favorable. CONCLUSION: The in-situ OR environment is both a
unique and effective setting to perform team-based training. Furthermore, training in the in-situ setting minimizes or removes many of the logistic issues involved in designing and implementing team-based training curricula for general surgery residency programs. However, we found that administrative and departmental (surgery, anesthesia, and nursing) “buy in” as well as protected faculty time for education were all necessary for in-situ training to be successful. NOTECHs II is an established scale for the evaluation of teams in this simulation setting and appears to be a valid tool based on the results of this study. However, further assessment of interrater reliability as well as improved training of evaluators are necessary to determine if inter-rater reliability can improve. C 2017 Published by Elsevier Inc. on ( J Surg Ed ]:]]]-]]]. J behalf of the Association of Program Directors in Surgery) KEY
WORDS: in-situ simulation, NOTECHs II, interpersonal skills, professionalism, APDS curriculum
Correspondence: Inquiries to Kristoffel Dumon, MD, FACS, Department of Surgery, 4 Silverstein, 3400 Spruce Street, Philadelphia, PA 19104; e-mail: Kristoffel. [email protected]
COMPETENCIES: Practice Based Learning and Improve-
ment, System Based Practice
Journal of Surgical Education & 2017 Published by Elsevier Inc. on behalf of the 1931-7204/$30.00 Association of Program Directors in Surgery http://dx.doi.org/10.1016/j.jsurg.2017.09.024
1
INTRODUCTION Nontechnical skills are critical to safety in the operating room (OR), and have been demonstrated to significantly impact surgical outcomes.1 Communication breakdowns have been repeatedly identified as a major cause of issues related to patient safety.2,3 In recent years, such concerns over patient safety have led to an increased emphasis on interdisciplinary and interprofessional teamwork skills during surgical training.4 During this same time period, simulation-based training has also increased in popularity as a means to provide technical and nontechnical skills training.5-7 Accordingly, the American College of Surgeons (ACS) and the Association of Program Directors in Surgery (APDS) have created a curriculum, which serves as a guideline for training surgery residents through simulation. While phases 1 and 2 deal with core skills and advanced procedures, respectively, phase 3 targets team-based skills.8 Specifically, phase 3 is a modular curriculum covering 10 scenarios in which cooperation between the team and a surgeon is necessary for a successful patient outcome. The third phase of the curriculum also includes suggestions on how to develop and implement simulated cases for resident team skills education and assessment. By providing a controlled environment, phase 3 of the ACS/APDS Surgery Residents Skills curriculum plays an important role in the objective assessment of 3 of the critical core competencies set forth by the ACGME,4 namely communication and interpersonal skills, professionalism, and system-based practice. Since the training is conducted in a controlled, safe environment, it also allows for errors to be committed while giving the opportunity for remediation outside the realm of clinical care. Despite the availability of this fully developed and standardized curriculum it has not seen widespread implementation to date. There are many potential reasons for this including a lack of awareness regarding the existence of the curriculum, the cost of implementation, the lack of trained faculty, and perhaps most importantly the lack of protected faculty and resident time.4,5,9 Specifically for phase 3, program directors have identified the challenges related to coordinating team-based training as a major obstacle to its implementation.5 However with the increasing accumulation of evidence demonstrating the usefulness of team training as well as other nontechnical skills training, the attitudes regarding such training will need to change.10 We recently designed a practical way of implementing phase 3 of the ACS/ADPS curriculum using an actual OR for team training. This in-situ environment was chosen for many reasons. Firstly, it obviates the need for a dedicated simulation center with its associated costs. Also, and more importantly, it increases resident and faculty availability because they are already on the site at the time of simulation. For these and more reasons, we believe that this training model will be useful for academic and 2
community surgery programs alike. In this paper, we present our experience to date with in-situ OR simulations and also present the resident/participant opinions on the usefulness of these simulations. As this was a pilot study of in-situ OR simulation training, we also chose to evaluate residents’ nontechnical skills to gauge the usefulness of the training. To do this we used a well-established tool for evaluating teamwork, the NOTECHS II scale, and hoped to demonstrate the validity of this instrument for use in this setting.11-14
METHODS We performed 8 sessions of in-situ simulation with 7 of the sessions involving 2 scenarios and 1 session involving only 1 scenario. This single scenario was excluded from data analysis. In order to prevent disruption to the OR schedule and minimize indirect costs, all simulations were performed “first case” in the morning in an OR that is reserved for add on cases and which was unstaffed at the time of the simulations. This was done with the permission and support of both the surgery and anesthesia departments as well as perioperative services. All participants chosen to participate in the training were already scheduled to be on site the day of the training and were informed of the training at least a month ahead of time. There were a total of 53 participants: 8 general surgery residents at the postgraduate year 1 (PGY1)-PGY3 level, 16 anesthesia residents at the PGY3PGY4 level, 16 circulating OR nurses, and 13 anesthesia technicians (Table 1). During each session, participants were put into teams consisting of a surgical resident, 2 anesthesia residents, 2 circulating nurses, and 1 or 2 anesthesia technicians. Based on their particular session, participants were exposed to 1 or 2 of 4 distinct scenarios that were chosen because we felt they would maximize the need for proper teamwork and communication: rapid exsanguination, air embolus due to laparoscopic trocar, patient losing vitals while performing prone position surgery and postoperative myocardial infarction with hypotension following laparoscopic cholecystectomy. We utilized the SimMan 4G (Laerdel, NY) for all scenarios. All other equipment were the same as those used in a standard OR. Each scenario was followed by systematic debriefing using crisis resource management (CRM) techniques.15 Debriefing was performed by surgery and anesthesia faculty and an experienced simulation educator. It TABLE 1. Participant Breakdown Category General surgery residents Anesthesia residents OR nurses Anesthesia technicians Total
Number 8 16 16 13 (not included in evaluation) 53
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
TABLE 2. Opinion Survey Questions Question
% Positive Response
This training will help to improve patient care The debriefings were useful I prefer in-situ team training to conventional simulation center-based training The training scenarios simulated my actual clinical experience This training helped to improve my leadership skills This training helped to improve my teamwork skills
consisted of 10 minutes of self-reflection by participants, followed by 10 minutes of guided discussion by the faculty and 5 minutes to summarize areas for improvement. Videos were recorded and stored on B-line (B-Line Medical, Washington, DC, USA). These were then reviewed by 3 independent raters, all of whom attended an intensive 2-day course on nontechnical skills training and received approximately 6 hours of training in evaluating nontechnical skills using the NOTECHs II scale. Each of the 3 subteams, namely anesthesia, nursing and surgery were evaluated using the standardized NOTECHs II scale. The difference in each team’s total average score for the first and second scenario was evaluated using a paired t-test. Inter-rater reliability was calculated using the intra-class correlation coefficient. All values were expressed using mean ± standard error. p o 0.05 was regarded as significant. Following the training participants’ opinions regarding the simulations were recorded using an electronic survey (Table 2). In this survey the participants were asked to respond to the summarized statements in Table 2 with one of the following options: Strongly Agree, Agree, Neutral, Disagree, or Strongly Disagree. A positive response was noted for any response that was either strongly agree or agree.
RESULTS There was significant improvement in each team’s total NOTECHs II score between the first and second scenarios (69.4 ± 1.4 and 77.3 ± 0.5, p ¼ 0.007) (Fig. 1). The surgery resident (24.2 ± 0.6 and 26.4 ± 0.5, p ¼ 0.007), anesthesia resident (23.7 ± 0.9 and 26.7 ± 0.4, p ¼ 0.03), and nursing staff (21.6 ± 0.3 and 24.3 ± 0.5, p ¼ 0.01)
FIGURE 1. Total NOTECHs score for scenarios 1 and 2.
91.7 87.5 96.4 83.3 75.0 66.7
subteams also demonstrated significant improvements in NOTECHs II score from the first to the second scenarios (Fig. 2). The Kendall’s coefficient of concordance was modest but statistically significant for the overall team score (W ¼ 0.57, p ¼ 0.05). For the subteams, it was statistically significant for only the nursing subteam (W ¼ 0.66. p ¼ 0.01), while it was strongly trending for surgery (W ¼ 0.54, p ¼ 0.07) and anesthesia (W ¼ 0.52, p ¼ 0.09) subteams. Participant survey responses (Table 2) demonstrated that 91.7% of participants felt that the training would help improve patient care, 87.5% felt that the debriefings were useful, 96.4% of the participants favored in-situ over conventional simulation center-based team training and 83.3% felt the training scenarios simulated their actual clinical experience. In all, 75% felt that the training helped improve their leadership skills and 66.7% felt that the training improved their teamwork skills (Table 2).
DISCUSSION The ACS/APDS curriculum16 is a comprehensive curriculum consisting of 3, modular phases covering basic surgical skills, advanced skills and procedures, and team-based training for mid to senior level residents. While the level of residency during which this training is supposed to occur is not specified, it is probably ideal that residents become comfortable with these skills during or before their 3rd year of training. This will allow for the training to be completed before the residents take on any significant leadership roles and let them spend the last 2 years of residency focusing on their clinical training. Overall, the implementation rates of
FIGURE 2. NOTECHs score by subteam for scenarios 1 and 2.
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
3
the ACS/APDS curriculum into residency programs has been poor despite the introduction of phase I in 2007, phase II in 2008, and phase III in 2009.5 In a 2013 survey of program directors, implementation rates were 36% for phase I, 19% for phase II, and 16% for phase III. Over 50% of respondents identified lack of faculty protected time, limited personnel, significant costs, and resident workhour restrictions as major obstacles9 to implementation. Since the penetration of the phase III curriculum has been the lowest,5 we elected to implement this training using an in-situ setting to try and overcome most if not all of the above listed obstacles. The coordination of team-based training, which requires the presence of multidisciplinary participants, has been identified as the greatest barrier to the implementation of the phase III curriculum.5 In-situ simulation has the ability to make such coordination easier because both the faculty and residents are at least already present on site. However, it must be stressed that in order for this to work there must be “buy in” on both the departmental and administrative levels. For our sessions, we were given use of the OR by perioperative services. We also had the support of the surgery and anesthesia departments. With such support we were able to choose residents and nursing staff who would be on site anyway and then inform them of the training at least a month beforehand. This allowed for clinical coverage to be arranged and in doing so minimized concerns regarding resident duty hours. Finally, the faculty who were involved in the scenarios and debriefing sessions consisted mostly, but not entirely, of those involved in the project and/or which have protected educational time. Using the in-situ environment for simulation training can also, at least theoretically, lead to cost savings especially when compared to the cost of staffing and running a fulltime simulation center. Indeed, we undertook in-situ training despite our having such a simulation center. We were able to do this partly because within our health system we have extra ORs, which are fully equipped, but not necessarily staffed. These rooms are usually used to accommodate extra volume on days with a very busy OR schedule. Again reflective of the need for departmental and administrative “buy in,” we were able to schedule our simulation sessions early in the morning on a day of the week (i.e., Wednesday) when our operative volume is traditionally lowest. As to the specifics of OR costs, there are no published formal data on true OR costs. It most certainly differs by institution, specialty, and country.17 Excluding physician costs, OR administrators sometimes use $15-$20 per OR minute to estimate cost for a basic surgical procedure, with at least half of that figure being fixed overhead costs.18 A more recent European study looking at examples of cost calculations for an OR and a post-anesthesia care unit quotes the indirect costs as being as high as 25% of total OR cost.19 The real variable costs depend in large part on how the OR staff is paid (e.g., hourly or salaried). Therefore, as the ORs we 4
used for these simulations were not staffed and since it is difficult/artificial to separate out the indirect cost of running a single OR for 45-90 minutes (1-2 scenarios) from the indirect costs for running over 50 other ORs we found the indirect costs of this training to negligible. The objective evaluation of the participants’ learning of nontechnical skills is a critical part of team-based training. Several validated rating scales have been developed to do this. Some are designed to evaluate specific groups such as NOTSS (nontechnical skills for surgeons) for surgeons,12 ANTS (anesthetists nontechnical skills) for anesthesiologists,20 and SPLINTS (Scrub Practitioners’ List of Intraoperative Non-Technical Skills ) for the scrub-nurses.21 Other scales are better utilized to evaluate teamwork. These include OTAS (Observational Teamwork Assessment for Surgery),22 the Clinical Teamwork Scale (CTS),23 the Mayo High Performance Teamwork Scale,24 and NOTECHs. Of these, only NOTECHs is a generic rating scale that can be used to assess both the team and subteams in multiple clinical environments and has a published high inter-rater reliability.11 However, as the original NOTECHs had a decreased ability to differentiate teams whose performance was not extreme, it was restructured to NOTECHs II in 2014.11 The NOTECHs II scale has improved scalability, and allows a better understanding on the impact of quality improvement interventions.13 For these reasons we chose NOTECHS II for use in our study.12-14 In-situ simulation has been extensively studied previously but not within the context of implantation of phase 3 of the ACS/APDS curriculum.1,6,7,25-28 Phitayakorn et al.26,27 performed a series of 5 simulations using surgery residents and evaluated the residents and other participants using NOTSS, ANTS, and SPLINTS. They found that technical and nontechnical skills did not correlate and suggested that both need to be trained together. They also reported poor inter-rater reliability for these scales in the absence of rater training. Interestingly, Nicksa et al. combined technical and nontechnical skills training in the OR, emergency department and intensive care unit and found that only the PGY2 (but not PGY1) residents improved when the technical and nontechnical skills were assessed at the beginning and the end of the academic year. Systematic debriefing, similar to our study was used in this study, but the effect of other factors on the participants’ nontechnical skills acquisition throughout the academic year is difficult to separate out.28 This particular study also used the NOTECHs II scale for the evaluation of nontechnical skills. Another study by Steinemann et al.29 implemented in-situ training for trauma resuscitation and found that there was a significant improvement in both nontechnical and technical skills in both the simulation and the clinical environments. An extensive review of in-situ OR simulation was performed by our group earlier this year.30 In it we concluded that in-situ simulation offers unique advantages compared to simulation
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
center-based training. This is especially true in relation to identification of system wide safety issues and threats. However, the efficacy of in-situ training has yet to be fully elucidated. In this pilot study, the nontechnical skills of each team improved following systematic debriefing. This was also true for each subteam. Overall the participants felt favorably regarding the simulations and the majority of the participants also preferred in-situ training to simulation centerbased training. We were also able to show that the nontechnical skills acquired were not scenario dependent as they showed improvement when distinct “pre” and “post” scenarios were used. The study was not without limitations however. The inter-rater reliability of the NOTECHs II scale was modestly significant for the total team score and the nursing subteam scores and was only strongly trending but not statistically significant for other subteams. This is likely reflective of the experience of our evaluators and not a problem inherent to the use of the NOTECHs II scale in this setting. However, further assessment of inter-rater reliability in this setting as well as an extensive review of our training of evaluators are necessary to determine if inter-rater reliability can actually improve.
3. Gawande AA, Zinner MJ, Studdert DM, Brennan TA.
Analysis of errors reported by surgeons at three teaching hospitals. Surgery. 2003;133(6):614-621. 4. Hull L, Arora S, Stefanidis D, Sevdalis N. Facilitating
the implementation of the American College of Surgeons/Association of Program Directors in Surgery phase III skills curriculum: training faculty in the assessment of team skills. Am J Surg. 2015;210 (5):933-941 e2. 5. Korndorffer JR, Arora S, Sevdalis N, Paige J, et al. The
American College of Surgeons/Association of Program Directors in Surgery National Skills Curriculum: adoption rate, challenges and strategies for effective implementation into surgical residency programs. Surgery. 2013;154(1):13-20. 6. Wetzel CM, Black SA, Hanna GB, et al. The effects of
stress and coping on surgical performance during simulations. Ann Surg. 2010;251(1):171-176. 7. Paige JT, Kozmenko V, Yang T, Paragi Gururaja R,
et al. High-fidelity, simulation-based, interdisciplinary operating room team training at the point of care. Surgery. 2009;145(2):138-146. 8. Scott DJ, Dunnington GL. The new ACS/APDS Skills
CONCLUSION Implementation of the ACS phase 3 simulation curriculum in the in-situ setting was favorably received by the participants and resulted in a significant improvement in nontechnical skills following systematic debriefing between scenarios. However, we found that administrative and departmental (surgery, anesthesia, and nursing) “buy in” as well as protected faculty time for education were essential to making the training successful. NOTECHs II is a wellestablished scale for the evaluation of teams in this simulation setting and appears to be a reliable and valid tool based on the results of this study. However, inter-rater reliability of the scale needs to be further evaluated in this setting.
Curriculum: moving the learning curve out of the operating room. J Gastrointest Surg. 2008;12(2):213-221. 9. Pentiak
PA, Schuch-Miller D, Streetman RT, Marik K, et al. Barriers to adoption of the surgical resident skills curriculum of the American College of Surgeons/Association of Program Directors in Surgery. Surgery. 2013;154(1):23-28.
10. Hull L, Sevdalis N. Advances in teaching and assessing
nontechnical skills. Surg Clin N Am. 2015;95(4): 869-884. 11. Sevdalis N, Davis R, Koutantji M, Undre S, et al.
Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg. 2008;196(2):184-190. 12. Wood TC, Raison N, Haldar S, Brunckhorst O, et al.
Training tools for nontechnical skills for surgeons—a systematic review. J Surg Educ. 2017;74(4):548-578. 13. Tyack PL, Calambokidis J, Friedlaender A, Goldbogen
REFERENCES 1. Arriaga AF, Gawande A, Raemer DB, et al. Pilot
testing of a model for insurer-driven, large-scale multicenter simulation training for operating room teams. Ann Surg. 2014;259(3):403-410. 2. Arriaga AF, Bader AM, Wong JM, Lipsitz SR, et al.
Simulation-based trial of surgical-crisis checklists. N Engl J Med. 2013;368(3):246-253.
J, et al. Oxford NOTECHS II: a modified theatre team non-technical skills scoring system. PLoS One. 2014;9(3):e90320. 14. Mishra A, Catchpole K, McCulloch P. The Oxford
NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care. 2009;18(2):104-108. 15. Blum RH, Raemer DB, Carroll JS, Sunder N. Crisis
resource management training for an anaesthesia
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]
5
faculty: a new approach to continuing education. Med Educ. 2004;38(1):45-55. 16. American College of Surgeons. ACS/APDS Surgery
Resident Skills Curriculum. 2017. Available at: https:// www.facs.org/education/program/resident-skills. 17. Macario A. What does one minute of operating room
time cost? J Clin Anesthesia. 2010;22(4):233-236. 18. Park KW DC. Can efficient supply management in
the operating room save millions. Curr Opin Anaesthesiol. 2009;22(2):242-248. 19. Raft J, Millet F, Meistelman C. Example of cost
calculations for an operating room and a post-anaesthesia care unit. Anaesth Crit Care Pain Med. 2015;34 (4):211-215. 20. Flin R, Patey R. Non-technical skills for anaesthetists:
developing and applying ANTS. Best Pract Res Clin Anaesthesiol. 2011;25(2):215-227.
24. Malec JF TL, Dunn WF, Wiegmann DA, Arnold JJ,
Brown DA, Phatak V. The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc. 2007;2(1):4-10. 25. Powers KA1, Rehrig ST, Irias N, Albano HA, et al.
Simulated laparoscopic operating room crisis: an approach to enhance the surgical team performance. Surg Endosc. 2008;22(4):885-900. 26. Phitayakorn R MR, Pian-Smith MC, Hemingway MW,
Milosh-Zinkus T, Oriol-Morway D, Petrusa E. Practicality of intraoperative teamwork assessments. J Surg Res. 2014;190(1):22-28. 27. Phitayakorn R, Minehart RD, Hemingway MW, Pian-
Smith MC, Petrusa E. The relationship between intraoperative teamwork and management skills in patient care. Surgery. 2015;158(5):1434-1440. 28. Nicksa GA, Anderson C, Fidler R, Stewart L. INno-
of the scrub practitioner: the SPLINTS system. ORNAC J. 2014;32(3):33-38.
vative approach using interprofessional simulation to educate surgical residents in technical and nontechnical skills in high-risk clinical scenarios. JAMA Surg. 2015;150(3):201-207.
22. Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N.
29. Steinemann S, Berg B, Skinner A, DiTulio A, et al. In
21. Flin R, Mitchell L, McLeod B. Non-technical skills
Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg. 2011;212(2):234-243 e5.
situ, multidisciplinary, simulation-based teamwork training improves early trauma care. J Surg Educ. 2011;68(6):472-477.
23. Guise JM, Deering S, Kanki BG, et al. Validation of a
30. Owei L, Neylan CJ, Rao R, Caskey RC, et al. In situ
tool to measure and promote clinical teamwork. Simul Healthc. 2008;3(4):217-223.
operating room-based simulation: a review. J Surg Educ. 2017;74(4):579-588.
6
Journal of Surgical Education Volume ]/Number 6 November/December ]]]]