- Email: [email protected]
Simulation-Based Laparoscopic Surgery Crisis Resource Management Training—Predicting Technical and Nontechnical Skills
ORIGINAL REPORTS
Simulation-Based Laparoscopic Surgery Crisis Resource Management Training—Predicting Technical and Nontechnical Skills Mitchell G. Goldenberg, MBBS,* Kai H. Fok, MD,* Michael Ordon, MD, FRCSC,*,† Kenneth T. Pace, MD, FRCSC,*,† and Jason Y. Lee, MD, MHPE, FRCSC*,‡ Division of Urology, Department of Surgery, University of Toronto, Toronto, Ontario, Canada; †St. Michael’s Hospital, University of Toronto, Toronto, Ontario, Canada; and ‡University Health Network, University of Toronto, Ontario, Canada *
OBJECTIVES: To develop a unique simulation-based assess-
ment using a laparoscopic inferior vena cava (IVC) injury scenario that allows for the safe assessment of urology resident’s technical and nontechnical skills, and investigate the effect of personality traits performance in a surgical crisis. METHODS: Urology residents from our institution were recruited to participate in a simulation-based training laparoscopic nephrectomy exercise. Residents completed demographic and multidimensional personality questionnaires and were instructed to play the role of staff urologist. A vasovagal response to pneumoperitoneum and an IVC injury event were scripted into the scenario. Technical and nontechnical skills were assessed by expert laparoscopic surgeons using validated tools (task checklist, GOALS, and NOTSS). RESULTS: Ten junior and five senior urology residents
participated. Five residents were unable to complete the exercise safely. Senior residents outperformed juniors on technical (checklist score 15.1 vs 9.9, p o 0.01, GOALS score 18.0 vs 13.3, p o 0.01) and nontechnical performance (NOTSS score 13.8 vs 10.1, p ¼ 0.03). Technical performance scores correlated with NOTSS scores (p o 0.01) and pass/fail rating correlated with technical performance (p o 0.01 for both checklist and GOALS), NOTSS score (p ¼ 0.02), and blood loss (p o 0.01). Only the conscientiousness dimension of the big five inventory correlated with technical score (p ¼ 0.03) and pass/fail rating (p ¼ 0.04). CONCLUSIONS: Resident level of training and laparo-
scopic experience correlated with technical performance
Correspondence: Inquiries to Department of Surgery, University of Toronto, Toronto, Ontario, Canada; e-mail: [email protected]
during a simulation-based laparoscopic IVC injury crisis management scenario, as well as multiple domains of nontechnical performance. Personality traits of our surgical residents are similar and did not predict technical skill. C ( J Surg Ed ]:]]]-]]]. J 2017 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: laparoscopy, simulation-based assessment,
crisis management, technical skills, nontechnical skills COMPETENCIES: Patient Care, Practice-based Learning &
Improvement
INTRODUCTION A variety of factors in the modern surgical training landscape have provided an impetus to increase the role of simulation-based training (SBT).1 Pressures to increase the efficiency of health care delivery and mandates to decrease resident work hours have shifted surgical training away from an apprenticeship model toward a competency-based one.2 Increasing ethical demands in patient care and increased case complexity have driven interest in SBT to augment training programs.3 Several decades of development have made SBT an accepted training method with supporting validity evidence, and experience gained from SBT has been shown to translate into the clinical environment.4 For example, learners who trained with SBT methods demonstrated decreasing operative times, errors, intraoperative and postoperative complications during laparoscopic surgery when compared to those trained without SBT.5,6 Modern SBT is moving toward high-fidelity training scenarios that incorporate the teaching and assessment of
Journal of Surgical Education & 2017 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jsurg.2017.11.011
1
both technical and nontechnical skills. Simulation platforms had traditionally focused on the development of surgical technique, however, nontechnical skills such as communication, teamwork, and decision making have been increasingly emphasized and incorporated into simulation.7 Motivation of these efforts comes from clinical data with evidence suggesting that almost half of surgical errors involved failures in effective communication among medical personnel.8 Incorporation of these essential principles into simulation has yielded specific and targeted ways of educating physicians and trainees in how to be effective communicators.9-11 Beyond developing technical and nontechnical skills required for competency, the nature of high fidelity SBT offers several key advantages over traditional training methods. SBT provides a safe and standardized method of practice—learners can refresh or gain confidence in rare, emergency situations or risky procedures and can explore outside the “zone of clinical safety.”12 This advantage of simulation has been recognized in the aviation industry where simulation of crisis scenarios is routinely part of training.13 Despite this, crisis or intraoperative emergency simulation training have received little attention in surgery to date. This paper follows our work in 2012 describing a lowcost surgical model for use in a simulation-based team training laparoscopic crisis scenario.14,15 Building on this work, we present an observational study further validating this high-fidelity laparoscopic crisis scenario designed to better understand predictors of surgeon technical skill with emphasis on nontechnical performance.
FIGURE 1. immersive simulated operating room (OR), complete with full laparoscopic suite, surgical equipment, and an anesthesia machine.
This prospective single cohort simulation-based study was conducted at a single institution, after receiving Research and Ethics Board approval. Urology residents at the University of Toronto were recruited to participate in the study, giving written and verbal consent. Participants were blinded to the details of the simulation, only told that they were consenting to partake in a SBT laparoscopic nephrectomy exercise. Consenting residents completed a brief prestudy questionnaire that included participant demographics, previous SBT and laparoscopic nephrectomy experience, and self-assessment of laparoscopic skill. Participants additionally completed a multidimensional personality questionnaire, the big five inventory (BFI).16 This validated, Likert-scale questionnaire covers 5 types of personality trait: openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. The simulation scenario was held at our institutional simulation centre, in an immersive simulated operating room (OR), complete with full laparoscopic suite, surgical equipment, and an anesthesia machine (Fig. 1). Participants
were recorded during the scenario through combined endoscopic video and in-room audio and video capture via wall-mounted cameras. The simulated patient consisted of a laparoscopic box-trainer, fully draped and positioned on the table as a patient would be during a laparoscopic right nephrectomy. The intra-abdominal organs were simulated using part-task trainers (Fig. 2). Participants were instructed to act as a staff urologist during the simulation, and direct the surgical team accordingly. Confederates included in the simulation included a scrub nurse, surgical assistant, and an anesthesiologist. The standardized scenario was coordinated by a trained simulation technician observing the entire scenario through one-way mirrored glass (Fig. 3). The simulated scenario began with the participant entering the OR after scrubbing, the patient already prepped and draped, the surgical team waiting for the final preoperative time-out before beginning the laparoscopic right radical nephrectomy. The participants were oriented to the specific surgical case, including patient details, and were instructed that they were to act as the staff urologist. They were oriented to the rules of the simulated scenario but were not told that any adverse events were included in the scenario. The scripted scenario consisted of 2 crisis events. First, a vasovagal response occurred following the insertion of a Veress needle and the start of pneumoperitoneum. The simulation technician could cause alarms to ring at the anesthesiologist’s machine, as would occur in a real OR environment, indicating acute and severe bradycardia and hypotension. If the participant did not recognize the event as a vasovagal response, the anesthesiologist was scripted to prompt the participant of the diagnosis. Second, an inferior vena cava (IVC) injury was simulated during mobilization of the kidney using the model shown in Figure 2. This injury occurred at a standardized step of the procedure—after clipping and cutting the renal artery. The patient condition was linked to the management of the IVC injury scenario and controlled by the simulation technician. All participants could proceed to the second part of the simulation
2
Journal of Surgical Education Volume ]/Number ] ] 2017
METHODS
1-3) residents. Demographic data were subjected to a frequency-distribution analysis to understand the baseline characteristics of the cohorts. Next, participant mean scores from the 2 groups were compared on univariate analysis using chi square, Mann-Whitney U, and ANOVA tests. Pearson and Spearman correlations were used to understand the relationships between checklist, GOALS, and NOTSS scores and level of training, previous laparoscopic experience, estimated blood loss (EBL), and technical skill selfassessment scores. All statistical analyses were conducted using SPSS (IBM, NY, USA). FIGURE 2. Intra-abdominal organs were simulated using part-task trainers.
RESULTS Study Participants
regardless of their response to the vasovagal event. The scenario ended following the IVC injury event either with the successful laparoscopic repair of the injury, the decision to convert to an open repair, or with a maximum blood loss of 2.5 L. Technical skills were evaluated from laparoscopically captured video. The analysis was carried out by 2 blinded expert laparoscopic surgeon analysts using the Global Operative Assessment of Laparoscopic Skills (GOALS)17 tool, a Likert-scale based global rating scale, and a procedure-specific checklist (Appendix 1). The nontechnical skills for surgeons (NOTSS)9 was also used to score participant performance using the audiovisual capture from the in-room cameras by 2 blinded simulation experts with significant experience in using the NOTSS tool. Analysts were oriented with the assessment tools and the purpose of the scenario to ensure accurate ratings, and interrater reliability statistics were calculated to determine to reliability of the scores. Participants were asked to assess their own technical and communication skills using de-novo, Likerttype rating scale. To conduct the analysis, the participants were split based on training level into senior (PGY 4-5) and junior (PGY
The study included a total of 15 participants, 5 senior residents and 10 junior residents. All participants completed the study, but 5 residents were unable to complete the scenario safely (ie, reached maximum blood loss). The overall mean EBL was 1.98 L (SD ¼ 0.61). The junior and senior cohorts had similar baseline characteristics (Table 1), including prior SBT experience (p ¼ 0.45), communication skills rating (p ¼ 0.14), and BFI scores (p 4 0.10 for all domains), but senior residents had more laparoscopic nephrectomy experience (p o 0.01). Technical Performance When comparing mean performance between the 2 groups, the senior residents obtained higher technical skill scores on univariate analysis (Table 2) on the procedure-specific checklist (15.1 [SD ¼ 2.04] vs 9.9 [SD 3.38], p o 0.01) and GOALS (18.0 [SD ¼ 2.18] vs 13.3 [SD 2.69], p o 0.01) scores. Successful completion of the scenario (pass/fail rating) correlated with technical performance (p o 0.01 for both checklist and GOALS scores), NOTSS score (p ¼ 0.02), and EBL (p o 0.01). Checklist scores correlated strongly with GOALS score (r ¼ 0.953, p o 0.001), NOTSS score (r ¼ 0.878, p o 0.001), and level of training (r ¼ 0.711, p ¼ 0.00), and moderately correlated with previous laparoscopic nephrectomy experience (r ¼ 0.672, p ¼ 0.00), self-assessed laparoscopic skill (r ¼ 0.516, p ¼ 0.045), and EBL (r ¼ −0.645, p ¼ 0.01). GOALS scores correlated strongly with level of training (r ¼ 0.700, p ¼ 0.00), previous laparoscopic nephrectomy experience (r ¼ 0.705, p ¼ 0.00), NOTSS score (r ¼ 0.921, p o 0.001), and checklist score (r ¼ 0.953, p o 0.001), and moderately correlated with self-assessed laparoscopic skill (r ¼ 0.614, p ¼ 0.02) and EBL (r ¼ −0.569, p ¼ 0.03) (Table 3). Nontechnical Performance
FIGURE 3. Standardized scenarios were coordinated by a trained simulation technician observing the entire scenario through one-way mirrored glass.
Senior residents outperformed their junior colleagues in nontechnical performance scores (NOTSS ¼ 13.8
Journal of Surgical Education Volume ]/Number ] ] 2017
3
TABLE 1. Participant Demographics, Baseline Experience, and BFI Scores Sr. Residents (n ¼ 5)
Jr. Residents (n ¼ 10)
Mean (SD) Prior experience and self-rating Prior laparoscopic nephrectomy experience Prior SBT experience (1 ¼ yes, 2 ¼ no) Self-rating of technical skill (GOALS) Self-rating of communication skills (NOTSS) Big five inventory (BFI) scores Extraversion Agreeableness Conscientiousness Neuroticism Openness
p Value
3.3 2.0 3.0 3.4
(0.6) (0.4) (0.7) (0.7)
1.3 1.8 2.0 3.2
(0.5) (0.4) (0.8) (0.8)
o 0.01 0.32 0.04 0.66
3.4 4.3 4.4 3.0 3.6
(0.7) (0.5) (0.3) (0.5) (0.4)
3.8 4.0 4.1 2.6 3.4
(0.8) (0.6) (0.3) (0.5) (3.6)
0.37 0.42 0.11 0.17 0.41
[SD ¼ 1.95] vs 10.1 [SD ¼ 3.14], p ¼ 0.03). Nontechnical scores correlated moderately with level of training (0.521, p ¼ 0.05) and overall NOTSS score correlated strongly with both checklist (r ¼ 0.878, p o 0.001) and GOALS (r ¼ 0.921, p o 0.001) scores. Of the 4 NOTSS domains assessed, level of training correlated with situation awareness, decision-making, and leadership scores but not with communication scores (p ¼ 0.09). Communication and teamwork domains also did not correlate with self-rated communication scores (0.234, p ¼ 0.401). Personality Scores On the (BFI) multidimensional personality questionnaire, only the conscientiousness dimension correlated with technical GOALS score (p ¼ 0.03) and successful completion of the scenario (pass/fail rating) (p ¼ 0.04). Interrater Statistics Interrater reliability statistics were excellent across all 3 assessment modalities, NOTSS (ICC ¼ 0.955), GOALS (ICC ¼ 0.976), and procedure-specific checklist (ICC ¼ 0.836).
DISCUSSION A variety of factors pose significant challenges to meeting learning objectives for training programs in the North American academic health care climate. Diminishing clinical exposure for trainees, increasing complexity of surgeries, and increased emphasis on reduction of medical errors has resulted in less opportunities for trainees to receive directed learning and practice. In this landscape, SBT is increasingly being used as an adjunct to traditional training to provide deliberate practice, timely feedback, and opportunities to refine behavior in a safe environment. Our 2012 study described a low-cost model for use in a “laparoscopic nephrectomy reno-vascular injury” simulation scenario.14 We also provided pilot-data on validating the laparoscopic crisis scenario simulation.15 Our current study provides further evidence to support the validity of this high-fidelity simulation-based laparoscopic crisis scenario for the assessment of technical skills, with procedure-specific checklist and GOALS assessments correlating well with level of training. Beyond technical skills, increased attention has been given to surgical nontechnical skills and their role in reducing errors and adverse outcomes. High fidelity SBT provides learners with opportunities to assess and practice
TABLE 2. Univariate Analysis of Resident Performance Sr. Residents (n ¼ 5)
Jr. Residents (n ¼ 10)
p Value
Mean (SD) Estimated blood loss (EBL) “Passed” scenario Number completing scenario Safely Number converting to open Performance scores Procedure-specific checklist GOALS NOTSS 4
1.9 (0.8)
2.1 (0.6)
0.57
N (%) 4 (80) 0 (0) 4 (80) 15.1 (2.0) 18.0 (2.2) 13.8 (1.9)
6 (60) 2 (20) 4 (40) 9.9 (3.4) 13.3 (2.7) 10.1 (3.1)
0.48
o 0.01 o 0.01 0.03
Journal of Surgical Education Volume ]/Number ] ] 2017
TABLE 3. Correlation Matrix Previous Lap SelfEsmated Level of Checklist Nephrectomy Assessed Blood Training Score Experience Skill Loss
GOALS Score
NOTSS Score
Checklist Score
0.711 p=.003
0.672 p=.006
0.516 p=.049
- 0.645 p=.009
0.953 p<.001
0.878 p<.001
GOALS Score
0.700 p=.004
0.672 p=.003
0.614 p=.015
- 0.569 p=.027
0.953 p<.001
NOTSS Score
0.521 p=.047
0.544 p=.036
0.440 p=.101*
- 0.546 p=.035
0.878 p<.001
0.921 p<.001 0.921 p<.001
Strong Correlaon (>0.700) Moderate Correlaon (0.500-0.700) Weak Correlaon (0.300-0.500) *
Not statistically significant.
these nontechnical skills. In our study, NOTSS score was significantly higher amongst senior residents compared to junior residents, but only correlated moderately with level of training. Nontechnical skill correlated well with technical skill as the NOTSS score correlated strongly with the GOALS score, procedure-specific checklist, and successful completion of scenario. Nontechnical skill appears to improve during residency training and improvement is correlated with development of technical skill. Interestingly, only the communication domain of the NOTSS score did not correlate with level of training. The other domains of the NOTSS score (situation awareness, decision making, and leadership) have more potential to be biased by the participant’s technical performance in the scenario, confounding the correlative relationship between those domains and technical skill. Our finding that senior residents do not differ from junior residents in communication domain scores may be a reflection of our training program’s lack of formal teaching on communication skills. Failure in teamwork and communication have been correlated with increased surgical errors and flow disruptions.18 In recent literature, surgery has often been compared to aviation as they share the similar potential for high stress, time pressures, dependence on properly functioning equipment, and historically rigid hierarchy.19 Yet, in a survey taken by expert observers, team performance by surgical teams were inferior to cockpit crews. Importantly, surgeons had a disproportionately high perception of the their own level of teamwork and communication skills.20 Our data, collected over a decade later, suggest that surgical trainees still lack the ability to recognize when their communication skills are lacking despite advances in surgical education. There appears to be an ongoing need to emphasize assessment and development of communication skills in the OR. Participants demonstrated similar personality traits across all domains as measured by the BFI, which may be reflective of the
residency selection process or selection bias for those residents who elected to participate in this study. Unsurprisingly as a result, personality traits did not correlate well with technical skill. Only the conscientious dimension of the BFI correlated moderately with technical skill. Similar results have previously been reported in literature. A retrospective study was done to determine if a surgical aptitude test done at time of application to otolaryngology residency could predict overall resident performance as evaluated by faculty members.21 A subset attitudinal score on the aptitude test which rated how focused, determined, and not distracted the applicant was towards the task was highly predictive of overall resident performance in the program. There are several limitations to our observational study. Although sufficiently powered for the statistical analyses, our study only had 15 participants from a single institution and requires external validation in larger, multi-institution studies. The BFI scores among our participants were similar across all domains, indicating it may not be sensitive enough to detect differences in the homogenous population of residents in the same surgical residency program. Although our study demonstrated a correlation between technical and nontechnical skill, it was not designed to assess whether nontechnical skills could be an independent predictor of technical skill. Finally, our study relies on video-recording of the simulated scenarios and expert raters to determine technical and non-technical scores of participants. There may be scenario-independent cues given by the participants that may reveal their level of training. This perception of level of training by the expert observer may introduce biases in the evaluation of the participants. In conclusion, resident level of training and laparoscopic experience correlated with technical performance during a simulation-based laparoscopic IVC injury crisis management scenario. Level of training and laparoscopic experience also correlated with most nontechnical performance
Journal of Surgical Education Volume ]/Number ] ] 2017
5
domains, except for communication and teamwork. Trainees were better able to self-assess their own technical skills in comparison to nontechnical skills during surgery. Personality traits of our surgical residents are similar and did not predict technical skill.
APPENDIX-1. —PROCEDURE-SPECIFIC CHECKLIST
Yes No
N/A
Demonstrates understanding of principles of Veress needle access for pneumoperitoneum Confirms 2 clicks for fascial layers Aspiration test, drop test, etc Observes initial pneumoperitoneum pressures (o10) Recognizes vasovagal response and stops insufflation (± attempts desufflation) Demonstrates proper sequence of renal vessel ligation (artery before vein) Demonstrates appropriate ligation of artery (minimum 2 clips on stay side OR stapler) Demonstrates safe transection of artery (accurate, controlled cut without past-pointing OR stapler) Recognizes venous bleeding and communicates appropriately Notifies anesthesia and RN team Asks for open tray availability Asks about pRBCs and/or vascular surgery consult Demonstrates appropriate initial management of venous bleeding Applies direct pressure Increases pneumoperitoneum Places additional assistant ports Makes use of assistant to help with bleeding Demonstrates appropriate management of IVC laceration Exploration of injury site Appropriate use of lap instruments Appropriate use of sutures/clips Uses appropriate tension/pressure on IVC Obtains definitive control of IVC (stops all bleeding) OR appropriate and timely decision to convert to open (if necessary) (2 points)
REFERENCES 1. Aydin
A, Ahmed K, Shafi AMA, Khan MS, Dasgupta P. The role of simulation in urological training—a quantitative study of practice and opinions. Surgeon. 2015. http://dx.doi.org/10.1016/j.surge. 2015.06.003.
2. Szasz P, Louridas M, Harris KA, Aggarwal R, Grantcharov
TP. Assessing technical competence in surgical trainees: 6
3. Brydges R, Hatala R, Zendejas B, Erwin PJ, Cook DA.
Linking simulation-based educational assessments and patient-related outcomes. Acad Med. 2015;90(2): 246-256. http://dx.doi.org/10.1097/ACM.000000000 0000549. 4. Cook DA, Zendejas B, Hamstra SJ, Hatala R,
Technical Skills Checklist (1 point for each action): Action
a systematic review. Ann Surg. 2014;261(6):1-1055. http://dx.doi.org/10.1097/SLA.0000000000000866.
Brydges R. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment Adv Health Sci Educ Theory Pract. 2014;19(2):233-250. http://dx.doi.org/10.1007/ s10459-013-9458-4. 5. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L,
Rosenberg J, Funch Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004;91(2):146-150. http://dx.doi. org/10.1002/bjs.4407. 6. Zendejas B, Cook DA, Bingener J, et al. Simulation-
based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair. Ann Surg. 2011; 254(3):502-511. http://dx.doi.org/10.1097/SLA.0b 013e31822c6994. 7. Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D.
Development of a rating system for surgeons’ non-technical skills. Med Educ. 2006;40(11):1098-1104. http://dx.doi.org/10.1111/j.1365-2929.2006.02610.x. 8. Gawande AA, Zinner MJ, Studdert DM, Brennan TA.
Analysis of errors reported by surgeons at three teaching hospitals. Surgery. 2003;133(6):614-621. http://dx.doi. org/10.1067/msy.2003.169. 9. Yule S, Flin R, Maran N, Rowley D, Youngson G,
Paterson-Brown S. Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg. 2008;32(4):548-556. http://dx.doi.org/10.1007/s00268-007-9320-z. 10. Dedy NJ, Szasz P, Louridas M, Bonrath EM, Husslein H,
Grantcharov TP. Objective structured assessment of nontechnical skills: reliability of a global rating scale for the intraining assessment in the operating room. Surgery. 2015:1-12. http://dx.doi.org/10.1016/j.surg.2014.12.023. 11. Rashid P, Gianduzzo T. Urology technical and non‐
technical skills development: the emerging role of simulation. BJU Int. 2015. http://dx.doi.org/10.1111/ bju.13259/full. 12. Doumouras AG, Keshet I, Nathens AB, Ahmed N,
Hicks CM. A crisis of faith? A review of simulation in teaching team-based, crisis management skills to surgical trainees J Surg Educ. 2012;69(3):274-281. http://dx. doi.org/10.1016/j.jsurg.2011.11.004. Journal of Surgical Education Volume ]/Number ] ] 2017
13. Helmreich RL, Merritt AC, Wilhelm JA. The evolu-
tion of crew resource management training in commercial aviation. Int J Aviat Psychol. 1999;9(1):19-32. http://dx.doi.org/10.1207/s15327108ijap0901_2. 14. Lee JY, Mucksavage P, McDougall EM. Simulating
laparoscopic renal hilar vessel injuries: preliminary evaluation of a novel surgical training model for residents. J Endourol. 2012;26(4):393-397. http://dx. doi.org/10.1089/end.2011.0432. 15. Lee JY, Mucksavage P, Canales C, McDougall EM,
Lin S. High fidelity simulation based team training in urology: a preliminary interdisciplinary study of technical and nontechnical skills in laparoscopic complications management. JURO. 2012;187(4):1385-1391. http://dx. doi.org/10.1016/j.juro.2011.11.106.
16. “The next Big Five Inventory (BFI-2): Developing and
assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power”: correction to Soto and John (2016). J Pers Soc Psychol. 2017; 113(1):143. http://dx.doi.org/10.1037/pspp0000x155.
laparoscopic skills. Am J Surg. 2005;190(1):107-113. http://dx.doi.org/10.1016/j.amjsurg.2005.04.004. 18. Greenberg CC, Regenbogen SE, Studdert DM, et al.
Patterns of communication breakdowns resulting in injury to surgical patients. J Am Coll Surg. 2007; 204(4):533-540. http://dx.doi.org/10.1016/j.jamcollsurg.2007.01.010. 19. McCulloch P, Mishra A, Handa A, Dale T, Hirst G,
Catchpole K. The effects of aviation-style non-technical skills training on technical performance and outcome in the operating theatre. Qual Saf Health Care. 2009;18(2):109-115. http://dx.doi.org/10.1136/qshc. 2008.032045. 20. Sexton JB, Thomas EJ, Helmreich RL. Error, stress,
and teamwork in medicine and aviation: cross sectional surveys. Br Med J. 2000;320(7237):745-749. 21. Moore EJ, Price DL, Van Abel KM, Carlson ML. Still
global assessment tool for evaluation of intraoperative
under the microscope: Can a surgical aptitude test predict otolaryngology resident performance? Laryngoscope. 2014;125(2):E57-E61. http://dx.doi.org/10. 1002/lary.24791.
Journal of Surgical Education Volume ]/Number ] ] 2017
7
17. Vassiliou MC, Feldman LS, Andrew CG, et al. A
Simulation-Based Laparoscopic Surgery Crisis Resource Management Training—Predicting Technical and Nontechnical Skills Mitchell G. Goldenberg, MBBS,* Kai H. Fok, MD,* Michael Ordon, MD, FRCSC,*,† Kenneth T. Pace, MD, FRCSC,*,† and Jason Y. Lee, MD, MHPE, FRCSC*,‡ Division of Urology, Department of Surgery, University of Toronto, Toronto, Ontario, Canada; †St. Michael’s Hospital, University of Toronto, Toronto, Ontario, Canada; and ‡University Health Network, University of Toronto, Ontario, Canada *
OBJECTIVES: To develop a unique simulation-based assess-
ment using a laparoscopic inferior vena cava (IVC) injury scenario that allows for the safe assessment of urology resident’s technical and nontechnical skills, and investigate the effect of personality traits performance in a surgical crisis. METHODS: Urology residents from our institution were recruited to participate in a simulation-based training laparoscopic nephrectomy exercise. Residents completed demographic and multidimensional personality questionnaires and were instructed to play the role of staff urologist. A vasovagal response to pneumoperitoneum and an IVC injury event were scripted into the scenario. Technical and nontechnical skills were assessed by expert laparoscopic surgeons using validated tools (task checklist, GOALS, and NOTSS). RESULTS: Ten junior and five senior urology residents
participated. Five residents were unable to complete the exercise safely. Senior residents outperformed juniors on technical (checklist score 15.1 vs 9.9, p o 0.01, GOALS score 18.0 vs 13.3, p o 0.01) and nontechnical performance (NOTSS score 13.8 vs 10.1, p ¼ 0.03). Technical performance scores correlated with NOTSS scores (p o 0.01) and pass/fail rating correlated with technical performance (p o 0.01 for both checklist and GOALS), NOTSS score (p ¼ 0.02), and blood loss (p o 0.01). Only the conscientiousness dimension of the big five inventory correlated with technical score (p ¼ 0.03) and pass/fail rating (p ¼ 0.04). CONCLUSIONS: Resident level of training and laparo-
scopic experience correlated with technical performance
Correspondence: Inquiries to Department of Surgery, University of Toronto, Toronto, Ontario, Canada; e-mail: [email protected]
during a simulation-based laparoscopic IVC injury crisis management scenario, as well as multiple domains of nontechnical performance. Personality traits of our surgical residents are similar and did not predict technical skill. C ( J Surg Ed ]:]]]-]]]. J 2017 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: laparoscopy, simulation-based assessment,
crisis management, technical skills, nontechnical skills COMPETENCIES: Patient Care, Practice-based Learning &
Improvement
INTRODUCTION A variety of factors in the modern surgical training landscape have provided an impetus to increase the role of simulation-based training (SBT).1 Pressures to increase the efficiency of health care delivery and mandates to decrease resident work hours have shifted surgical training away from an apprenticeship model toward a competency-based one.2 Increasing ethical demands in patient care and increased case complexity have driven interest in SBT to augment training programs.3 Several decades of development have made SBT an accepted training method with supporting validity evidence, and experience gained from SBT has been shown to translate into the clinical environment.4 For example, learners who trained with SBT methods demonstrated decreasing operative times, errors, intraoperative and postoperative complications during laparoscopic surgery when compared to those trained without SBT.5,6 Modern SBT is moving toward high-fidelity training scenarios that incorporate the teaching and assessment of
Journal of Surgical Education & 2017 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jsurg.2017.11.011
1
both technical and nontechnical skills. Simulation platforms had traditionally focused on the development of surgical technique, however, nontechnical skills such as communication, teamwork, and decision making have been increasingly emphasized and incorporated into simulation.7 Motivation of these efforts comes from clinical data with evidence suggesting that almost half of surgical errors involved failures in effective communication among medical personnel.8 Incorporation of these essential principles into simulation has yielded specific and targeted ways of educating physicians and trainees in how to be effective communicators.9-11 Beyond developing technical and nontechnical skills required for competency, the nature of high fidelity SBT offers several key advantages over traditional training methods. SBT provides a safe and standardized method of practice—learners can refresh or gain confidence in rare, emergency situations or risky procedures and can explore outside the “zone of clinical safety.”12 This advantage of simulation has been recognized in the aviation industry where simulation of crisis scenarios is routinely part of training.13 Despite this, crisis or intraoperative emergency simulation training have received little attention in surgery to date. This paper follows our work in 2012 describing a lowcost surgical model for use in a simulation-based team training laparoscopic crisis scenario.14,15 Building on this work, we present an observational study further validating this high-fidelity laparoscopic crisis scenario designed to better understand predictors of surgeon technical skill with emphasis on nontechnical performance.
FIGURE 1. immersive simulated operating room (OR), complete with full laparoscopic suite, surgical equipment, and an anesthesia machine.
This prospective single cohort simulation-based study was conducted at a single institution, after receiving Research and Ethics Board approval. Urology residents at the University of Toronto were recruited to participate in the study, giving written and verbal consent. Participants were blinded to the details of the simulation, only told that they were consenting to partake in a SBT laparoscopic nephrectomy exercise. Consenting residents completed a brief prestudy questionnaire that included participant demographics, previous SBT and laparoscopic nephrectomy experience, and self-assessment of laparoscopic skill. Participants additionally completed a multidimensional personality questionnaire, the big five inventory (BFI).16 This validated, Likert-scale questionnaire covers 5 types of personality trait: openness to experience, conscientiousness, extraversion, agreeableness, and neuroticism. The simulation scenario was held at our institutional simulation centre, in an immersive simulated operating room (OR), complete with full laparoscopic suite, surgical equipment, and an anesthesia machine (Fig. 1). Participants
were recorded during the scenario through combined endoscopic video and in-room audio and video capture via wall-mounted cameras. The simulated patient consisted of a laparoscopic box-trainer, fully draped and positioned on the table as a patient would be during a laparoscopic right nephrectomy. The intra-abdominal organs were simulated using part-task trainers (Fig. 2). Participants were instructed to act as a staff urologist during the simulation, and direct the surgical team accordingly. Confederates included in the simulation included a scrub nurse, surgical assistant, and an anesthesiologist. The standardized scenario was coordinated by a trained simulation technician observing the entire scenario through one-way mirrored glass (Fig. 3). The simulated scenario began with the participant entering the OR after scrubbing, the patient already prepped and draped, the surgical team waiting for the final preoperative time-out before beginning the laparoscopic right radical nephrectomy. The participants were oriented to the specific surgical case, including patient details, and were instructed that they were to act as the staff urologist. They were oriented to the rules of the simulated scenario but were not told that any adverse events were included in the scenario. The scripted scenario consisted of 2 crisis events. First, a vasovagal response occurred following the insertion of a Veress needle and the start of pneumoperitoneum. The simulation technician could cause alarms to ring at the anesthesiologist’s machine, as would occur in a real OR environment, indicating acute and severe bradycardia and hypotension. If the participant did not recognize the event as a vasovagal response, the anesthesiologist was scripted to prompt the participant of the diagnosis. Second, an inferior vena cava (IVC) injury was simulated during mobilization of the kidney using the model shown in Figure 2. This injury occurred at a standardized step of the procedure—after clipping and cutting the renal artery. The patient condition was linked to the management of the IVC injury scenario and controlled by the simulation technician. All participants could proceed to the second part of the simulation
2
Journal of Surgical Education Volume ]/Number ] ] 2017
METHODS
1-3) residents. Demographic data were subjected to a frequency-distribution analysis to understand the baseline characteristics of the cohorts. Next, participant mean scores from the 2 groups were compared on univariate analysis using chi square, Mann-Whitney U, and ANOVA tests. Pearson and Spearman correlations were used to understand the relationships between checklist, GOALS, and NOTSS scores and level of training, previous laparoscopic experience, estimated blood loss (EBL), and technical skill selfassessment scores. All statistical analyses were conducted using SPSS (IBM, NY, USA). FIGURE 2. Intra-abdominal organs were simulated using part-task trainers.
RESULTS Study Participants
regardless of their response to the vasovagal event. The scenario ended following the IVC injury event either with the successful laparoscopic repair of the injury, the decision to convert to an open repair, or with a maximum blood loss of 2.5 L. Technical skills were evaluated from laparoscopically captured video. The analysis was carried out by 2 blinded expert laparoscopic surgeon analysts using the Global Operative Assessment of Laparoscopic Skills (GOALS)17 tool, a Likert-scale based global rating scale, and a procedure-specific checklist (Appendix 1). The nontechnical skills for surgeons (NOTSS)9 was also used to score participant performance using the audiovisual capture from the in-room cameras by 2 blinded simulation experts with significant experience in using the NOTSS tool. Analysts were oriented with the assessment tools and the purpose of the scenario to ensure accurate ratings, and interrater reliability statistics were calculated to determine to reliability of the scores. Participants were asked to assess their own technical and communication skills using de-novo, Likerttype rating scale. To conduct the analysis, the participants were split based on training level into senior (PGY 4-5) and junior (PGY
The study included a total of 15 participants, 5 senior residents and 10 junior residents. All participants completed the study, but 5 residents were unable to complete the scenario safely (ie, reached maximum blood loss). The overall mean EBL was 1.98 L (SD ¼ 0.61). The junior and senior cohorts had similar baseline characteristics (Table 1), including prior SBT experience (p ¼ 0.45), communication skills rating (p ¼ 0.14), and BFI scores (p 4 0.10 for all domains), but senior residents had more laparoscopic nephrectomy experience (p o 0.01). Technical Performance When comparing mean performance between the 2 groups, the senior residents obtained higher technical skill scores on univariate analysis (Table 2) on the procedure-specific checklist (15.1 [SD ¼ 2.04] vs 9.9 [SD 3.38], p o 0.01) and GOALS (18.0 [SD ¼ 2.18] vs 13.3 [SD 2.69], p o 0.01) scores. Successful completion of the scenario (pass/fail rating) correlated with technical performance (p o 0.01 for both checklist and GOALS scores), NOTSS score (p ¼ 0.02), and EBL (p o 0.01). Checklist scores correlated strongly with GOALS score (r ¼ 0.953, p o 0.001), NOTSS score (r ¼ 0.878, p o 0.001), and level of training (r ¼ 0.711, p ¼ 0.00), and moderately correlated with previous laparoscopic nephrectomy experience (r ¼ 0.672, p ¼ 0.00), self-assessed laparoscopic skill (r ¼ 0.516, p ¼ 0.045), and EBL (r ¼ −0.645, p ¼ 0.01). GOALS scores correlated strongly with level of training (r ¼ 0.700, p ¼ 0.00), previous laparoscopic nephrectomy experience (r ¼ 0.705, p ¼ 0.00), NOTSS score (r ¼ 0.921, p o 0.001), and checklist score (r ¼ 0.953, p o 0.001), and moderately correlated with self-assessed laparoscopic skill (r ¼ 0.614, p ¼ 0.02) and EBL (r ¼ −0.569, p ¼ 0.03) (Table 3). Nontechnical Performance
FIGURE 3. Standardized scenarios were coordinated by a trained simulation technician observing the entire scenario through one-way mirrored glass.
Senior residents outperformed their junior colleagues in nontechnical performance scores (NOTSS ¼ 13.8
Journal of Surgical Education Volume ]/Number ] ] 2017
3
TABLE 1. Participant Demographics, Baseline Experience, and BFI Scores Sr. Residents (n ¼ 5)
Jr. Residents (n ¼ 10)
Mean (SD) Prior experience and self-rating Prior laparoscopic nephrectomy experience Prior SBT experience (1 ¼ yes, 2 ¼ no) Self-rating of technical skill (GOALS) Self-rating of communication skills (NOTSS) Big five inventory (BFI) scores Extraversion Agreeableness Conscientiousness Neuroticism Openness
p Value
3.3 2.0 3.0 3.4
(0.6) (0.4) (0.7) (0.7)
1.3 1.8 2.0 3.2
(0.5) (0.4) (0.8) (0.8)
o 0.01 0.32 0.04 0.66
3.4 4.3 4.4 3.0 3.6
(0.7) (0.5) (0.3) (0.5) (0.4)
3.8 4.0 4.1 2.6 3.4
(0.8) (0.6) (0.3) (0.5) (3.6)
0.37 0.42 0.11 0.17 0.41
[SD ¼ 1.95] vs 10.1 [SD ¼ 3.14], p ¼ 0.03). Nontechnical scores correlated moderately with level of training (0.521, p ¼ 0.05) and overall NOTSS score correlated strongly with both checklist (r ¼ 0.878, p o 0.001) and GOALS (r ¼ 0.921, p o 0.001) scores. Of the 4 NOTSS domains assessed, level of training correlated with situation awareness, decision-making, and leadership scores but not with communication scores (p ¼ 0.09). Communication and teamwork domains also did not correlate with self-rated communication scores (0.234, p ¼ 0.401). Personality Scores On the (BFI) multidimensional personality questionnaire, only the conscientiousness dimension correlated with technical GOALS score (p ¼ 0.03) and successful completion of the scenario (pass/fail rating) (p ¼ 0.04). Interrater Statistics Interrater reliability statistics were excellent across all 3 assessment modalities, NOTSS (ICC ¼ 0.955), GOALS (ICC ¼ 0.976), and procedure-specific checklist (ICC ¼ 0.836).
DISCUSSION A variety of factors pose significant challenges to meeting learning objectives for training programs in the North American academic health care climate. Diminishing clinical exposure for trainees, increasing complexity of surgeries, and increased emphasis on reduction of medical errors has resulted in less opportunities for trainees to receive directed learning and practice. In this landscape, SBT is increasingly being used as an adjunct to traditional training to provide deliberate practice, timely feedback, and opportunities to refine behavior in a safe environment. Our 2012 study described a low-cost model for use in a “laparoscopic nephrectomy reno-vascular injury” simulation scenario.14 We also provided pilot-data on validating the laparoscopic crisis scenario simulation.15 Our current study provides further evidence to support the validity of this high-fidelity simulation-based laparoscopic crisis scenario for the assessment of technical skills, with procedure-specific checklist and GOALS assessments correlating well with level of training. Beyond technical skills, increased attention has been given to surgical nontechnical skills and their role in reducing errors and adverse outcomes. High fidelity SBT provides learners with opportunities to assess and practice
TABLE 2. Univariate Analysis of Resident Performance Sr. Residents (n ¼ 5)
Jr. Residents (n ¼ 10)
p Value
Mean (SD) Estimated blood loss (EBL) “Passed” scenario Number completing scenario Safely Number converting to open Performance scores Procedure-specific checklist GOALS NOTSS 4
1.9 (0.8)
2.1 (0.6)
0.57
N (%) 4 (80) 0 (0) 4 (80) 15.1 (2.0) 18.0 (2.2) 13.8 (1.9)
6 (60) 2 (20) 4 (40) 9.9 (3.4) 13.3 (2.7) 10.1 (3.1)
0.48
o 0.01 o 0.01 0.03
Journal of Surgical Education Volume ]/Number ] ] 2017
TABLE 3. Correlation Matrix Previous Lap SelfEsmated Level of Checklist Nephrectomy Assessed Blood Training Score Experience Skill Loss
GOALS Score
NOTSS Score
Checklist Score
0.711 p=.003
0.672 p=.006
0.516 p=.049
- 0.645 p=.009
0.953 p<.001
0.878 p<.001
GOALS Score
0.700 p=.004
0.672 p=.003
0.614 p=.015
- 0.569 p=.027
0.953 p<.001
NOTSS Score
0.521 p=.047
0.544 p=.036
0.440 p=.101*
- 0.546 p=.035
0.878 p<.001
0.921 p<.001 0.921 p<.001
Strong Correlaon (>0.700) Moderate Correlaon (0.500-0.700) Weak Correlaon (0.300-0.500) *
Not statistically significant.
these nontechnical skills. In our study, NOTSS score was significantly higher amongst senior residents compared to junior residents, but only correlated moderately with level of training. Nontechnical skill correlated well with technical skill as the NOTSS score correlated strongly with the GOALS score, procedure-specific checklist, and successful completion of scenario. Nontechnical skill appears to improve during residency training and improvement is correlated with development of technical skill. Interestingly, only the communication domain of the NOTSS score did not correlate with level of training. The other domains of the NOTSS score (situation awareness, decision making, and leadership) have more potential to be biased by the participant’s technical performance in the scenario, confounding the correlative relationship between those domains and technical skill. Our finding that senior residents do not differ from junior residents in communication domain scores may be a reflection of our training program’s lack of formal teaching on communication skills. Failure in teamwork and communication have been correlated with increased surgical errors and flow disruptions.18 In recent literature, surgery has often been compared to aviation as they share the similar potential for high stress, time pressures, dependence on properly functioning equipment, and historically rigid hierarchy.19 Yet, in a survey taken by expert observers, team performance by surgical teams were inferior to cockpit crews. Importantly, surgeons had a disproportionately high perception of the their own level of teamwork and communication skills.20 Our data, collected over a decade later, suggest that surgical trainees still lack the ability to recognize when their communication skills are lacking despite advances in surgical education. There appears to be an ongoing need to emphasize assessment and development of communication skills in the OR. Participants demonstrated similar personality traits across all domains as measured by the BFI, which may be reflective of the
residency selection process or selection bias for those residents who elected to participate in this study. Unsurprisingly as a result, personality traits did not correlate well with technical skill. Only the conscientious dimension of the BFI correlated moderately with technical skill. Similar results have previously been reported in literature. A retrospective study was done to determine if a surgical aptitude test done at time of application to otolaryngology residency could predict overall resident performance as evaluated by faculty members.21 A subset attitudinal score on the aptitude test which rated how focused, determined, and not distracted the applicant was towards the task was highly predictive of overall resident performance in the program. There are several limitations to our observational study. Although sufficiently powered for the statistical analyses, our study only had 15 participants from a single institution and requires external validation in larger, multi-institution studies. The BFI scores among our participants were similar across all domains, indicating it may not be sensitive enough to detect differences in the homogenous population of residents in the same surgical residency program. Although our study demonstrated a correlation between technical and nontechnical skill, it was not designed to assess whether nontechnical skills could be an independent predictor of technical skill. Finally, our study relies on video-recording of the simulated scenarios and expert raters to determine technical and non-technical scores of participants. There may be scenario-independent cues given by the participants that may reveal their level of training. This perception of level of training by the expert observer may introduce biases in the evaluation of the participants. In conclusion, resident level of training and laparoscopic experience correlated with technical performance during a simulation-based laparoscopic IVC injury crisis management scenario. Level of training and laparoscopic experience also correlated with most nontechnical performance
Journal of Surgical Education Volume ]/Number ] ] 2017
5
domains, except for communication and teamwork. Trainees were better able to self-assess their own technical skills in comparison to nontechnical skills during surgery. Personality traits of our surgical residents are similar and did not predict technical skill.
APPENDIX-1. —PROCEDURE-SPECIFIC CHECKLIST
Yes No
N/A
Demonstrates understanding of principles of Veress needle access for pneumoperitoneum Confirms 2 clicks for fascial layers Aspiration test, drop test, etc Observes initial pneumoperitoneum pressures (o10) Recognizes vasovagal response and stops insufflation (± attempts desufflation) Demonstrates proper sequence of renal vessel ligation (artery before vein) Demonstrates appropriate ligation of artery (minimum 2 clips on stay side OR stapler) Demonstrates safe transection of artery (accurate, controlled cut without past-pointing OR stapler) Recognizes venous bleeding and communicates appropriately Notifies anesthesia and RN team Asks for open tray availability Asks about pRBCs and/or vascular surgery consult Demonstrates appropriate initial management of venous bleeding Applies direct pressure Increases pneumoperitoneum Places additional assistant ports Makes use of assistant to help with bleeding Demonstrates appropriate management of IVC laceration Exploration of injury site Appropriate use of lap instruments Appropriate use of sutures/clips Uses appropriate tension/pressure on IVC Obtains definitive control of IVC (stops all bleeding) OR appropriate and timely decision to convert to open (if necessary) (2 points)
REFERENCES 1. Aydin
A, Ahmed K, Shafi AMA, Khan MS, Dasgupta P. The role of simulation in urological training—a quantitative study of practice and opinions. Surgeon. 2015. http://dx.doi.org/10.1016/j.surge. 2015.06.003.
2. Szasz P, Louridas M, Harris KA, Aggarwal R, Grantcharov
TP. Assessing technical competence in surgical trainees: 6
3. Brydges R, Hatala R, Zendejas B, Erwin PJ, Cook DA.
Linking simulation-based educational assessments and patient-related outcomes. Acad Med. 2015;90(2): 246-256. http://dx.doi.org/10.1097/ACM.000000000 0000549. 4. Cook DA, Zendejas B, Hamstra SJ, Hatala R,
Technical Skills Checklist (1 point for each action): Action
a systematic review. Ann Surg. 2014;261(6):1-1055. http://dx.doi.org/10.1097/SLA.0000000000000866.
Brydges R. What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment Adv Health Sci Educ Theory Pract. 2014;19(2):233-250. http://dx.doi.org/10.1007/ s10459-013-9458-4. 5. Grantcharov TP, Kristiansen VB, Bendix J, Bardram L,
Rosenberg J, Funch Jensen P. Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg. 2004;91(2):146-150. http://dx.doi. org/10.1002/bjs.4407. 6. Zendejas B, Cook DA, Bingener J, et al. Simulation-
based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair. Ann Surg. 2011; 254(3):502-511. http://dx.doi.org/10.1097/SLA.0b 013e31822c6994. 7. Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D.
Development of a rating system for surgeons’ non-technical skills. Med Educ. 2006;40(11):1098-1104. http://dx.doi.org/10.1111/j.1365-2929.2006.02610.x. 8. Gawande AA, Zinner MJ, Studdert DM, Brennan TA.
Analysis of errors reported by surgeons at three teaching hospitals. Surgery. 2003;133(6):614-621. http://dx.doi. org/10.1067/msy.2003.169. 9. Yule S, Flin R, Maran N, Rowley D, Youngson G,
Paterson-Brown S. Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg. 2008;32(4):548-556. http://dx.doi.org/10.1007/s00268-007-9320-z. 10. Dedy NJ, Szasz P, Louridas M, Bonrath EM, Husslein H,
Grantcharov TP. Objective structured assessment of nontechnical skills: reliability of a global rating scale for the intraining assessment in the operating room. Surgery. 2015:1-12. http://dx.doi.org/10.1016/j.surg.2014.12.023. 11. Rashid P, Gianduzzo T. Urology technical and non‐
technical skills development: the emerging role of simulation. BJU Int. 2015. http://dx.doi.org/10.1111/ bju.13259/full. 12. Doumouras AG, Keshet I, Nathens AB, Ahmed N,
Hicks CM. A crisis of faith? A review of simulation in teaching team-based, crisis management skills to surgical trainees J Surg Educ. 2012;69(3):274-281. http://dx. doi.org/10.1016/j.jsurg.2011.11.004. Journal of Surgical Education Volume ]/Number ] ] 2017
13. Helmreich RL, Merritt AC, Wilhelm JA. The evolu-
tion of crew resource management training in commercial aviation. Int J Aviat Psychol. 1999;9(1):19-32. http://dx.doi.org/10.1207/s15327108ijap0901_2. 14. Lee JY, Mucksavage P, McDougall EM. Simulating
laparoscopic renal hilar vessel injuries: preliminary evaluation of a novel surgical training model for residents. J Endourol. 2012;26(4):393-397. http://dx. doi.org/10.1089/end.2011.0432. 15. Lee JY, Mucksavage P, Canales C, McDougall EM,
Lin S. High fidelity simulation based team training in urology: a preliminary interdisciplinary study of technical and nontechnical skills in laparoscopic complications management. JURO. 2012;187(4):1385-1391. http://dx. doi.org/10.1016/j.juro.2011.11.106.
16. “The next Big Five Inventory (BFI-2): Developing and
assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power”: correction to Soto and John (2016). J Pers Soc Psychol. 2017; 113(1):143. http://dx.doi.org/10.1037/pspp0000x155.
laparoscopic skills. Am J Surg. 2005;190(1):107-113. http://dx.doi.org/10.1016/j.amjsurg.2005.04.004. 18. Greenberg CC, Regenbogen SE, Studdert DM, et al.
Patterns of communication breakdowns resulting in injury to surgical patients. J Am Coll Surg. 2007; 204(4):533-540. http://dx.doi.org/10.1016/j.jamcollsurg.2007.01.010. 19. McCulloch P, Mishra A, Handa A, Dale T, Hirst G,
Catchpole K. The effects of aviation-style non-technical skills training on technical performance and outcome in the operating theatre. Qual Saf Health Care. 2009;18(2):109-115. http://dx.doi.org/10.1136/qshc. 2008.032045. 20. Sexton JB, Thomas EJ, Helmreich RL. Error, stress,
and teamwork in medicine and aviation: cross sectional surveys. Br Med J. 2000;320(7237):745-749. 21. Moore EJ, Price DL, Van Abel KM, Carlson ML. Still
global assessment tool for evaluation of intraoperative
under the microscope: Can a surgical aptitude test predict otolaryngology resident performance? Laryngoscope. 2014;125(2):E57-E61. http://dx.doi.org/10. 1002/lary.24791.
Journal of Surgical Education Volume ]/Number ] ] 2017
7
17. Vassiliou MC, Feldman LS, Andrew CG, et al. A