- Email: [email protected]
Developing a coaching mechanism for practicing surgeons
ARTICLE IN PRESS
Developing a coaching mechanism for practicing surgeons Dimitrios Stefanidis, MD, PhD, FACS, FASMBS,a,b Brittany Anderson-Montoya, PhD,c Robert V. Higgins, MD,d Manuel E. Pimentel,b Patrick Rowland,c Madison O. Scarborough, MBA, MHA,c and Danelle Higgins, RN, BSN, MHA,c Charlotte, NC
Background. While performance feedback and assessment are hallmarks of surgical training, they abruptly cease after training is completed. In their absence, performance may stagnate and poor habits persist. Our aim was to develop a coaching mechanism for practicing surgeons with feedback provision based on objective performance assessment. Methods. Technical and nontechnical intraoperative video recordings from laparoscopic or robotic cholecystectomies, colectomies, and hysterectomies were assessed by a blinded surgeon and a human factors expert, respectively. Aspects of performance in need of improvement were noted, and a coaching session was developed for feedback provision to participating surgeons. This 4-hour coaching session consisted of a didactic lecture with video review and hands-on practice using procedural and mannequin-based simulation. Results. Thirty-two practicing surgeons (18 general; 14 gynecologists) from 6 different hospitals were assessed, and 9 of them participated in coaching. Technical aspects identified for performance improvement included suboptimal trocar placement, inadequate critical view achievement during laparoscopic cholecystectomies, poor visualization of the operating field, bimanual dexterity, and dissection techniques, while nontechnical aspects included inappropriate handling of distractions and interruptions, poor ergonomic positioning and situational awareness, and inadequate mitigation of delays. Most surgeons appropriately accomplished some of the objectives of the distraction scenario, but none was able to achieve expert levels on Fundamentals of Laparoscopy tasks. Participants perceived the coaching sessions as highly valuable. Conclusion. Our study identified several technical and nontechnical skill sets of practicing surgeons in need of improvement and provided support for the implementation of coaching programs for surgeons on an ongoing basis. (Surgery 2016;j:j-j.) From the Department of Surgery,a Carolinas Simulation Center,b Division of Quality,c and the Department of OB/Gynecology,d Carolinas HealthCare System, Charlotte, NC
SURGERY IS A HIGHLY TECHNICAL, high-volume, high-risk field of health care that impacts millions of patients. A systematic review indicated that the majority of Supported through a Leading Edge Advanced Practice Topics grant by the Center for Medicare and Medicaid Services contract number HHSM-500-2012-00028C/000005. Presented from the podium at the annual consortium meeting of the American College of Surgeons–Accredited Education Institutes in Chicago, IL, on March 14th, 2015. Accepted for publication March 11, 2016. Reprint requests: Dimitrios Stefanidis, MD, PhD, FACS, FASMBS, Carolinas Simulation Center, and Division of Bariatric Surgery, Department of Surgery, Carolinas HealthCare System, 1025 Morehead Medical Dr, Suite 300, Charlotte, NC 28204. E-mail: [email protected]. 0039-6060/$ - see front matter Ó 2016 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.surg.2016.03.036
adverse events in health care were related to operations and surgical staff.1 Therefore, it is imperative that an understanding of what is causing these adverse events be established to mitigate future adverse events. Indeed, the National Patient Safety Foundation’s (NPSF’s) Agenda for Research in Patient Safety states, “One of the most important approaches to patient safety research is to understand the nature of technical work.”2 Although it seems intuitive that surgeon skill directly impacts patient safety, it was not until recently that a study demonstrated that patients of bariatric surgeons whose skill was rated higher by their peers on video-based assessment had fewer postoperative complications compared with those of surgeons with lower skill levels.3 Further, the impact of poor nontechnical skills is increasingly recognized as a significant contributing factor to adverse events and needs to SURGERY 1
ARTICLE IN PRESS 2 Stefanidis et al
be assessed in relation to a physician’s technical skills.4,5 Despite the recognized importance and widespread endorsement of competency-based assessment for physicians in training, objective methods to assess practicing physician performance are rarely used today. Further, amidst growing demand for transparency of clinical outcomes, hospitals are struggling with the question of how to objectively measure physician competency and quality. Without an objective assessment, it is very difficult to accurately identify physicians in need of performance improvement and to help them remediate their deficiencies.6,7 Research has demonstrated that performance feedback can lead to improved clinical and technical skills, and its absence usually leads to an inability to achieve a higher level of performance.8,9 The importance of feedback for performance improvement and the lack thereof for practicing surgeons have led several authors to recommend that coaching programs for surgeons be developed.10,11 The value of coaching is well recognized in several disciplines, such as sports,12 and is geared toward performers at all levels, not just trainees. The first aim of this study was to identify technical and nontechnical deficiencies in practicing surgeons based on video review of their performance. The second aim, based on the identified deficiencies, was to develop and pilot a coaching mechanism for practicing surgeons with feedback provision. METHODS This project was an Institutional Review Board– approved, prospective quality improvement study with voluntary enrollment of practicing general and gynecologic surgeons. The study was conducted in 6 hospitals of Carolinas HealthCare System (CHS), a large not-for-profit health care system comprising 43 hospitals and more than 60,000 employees. The Center for Medicare and Medicaid Services Partnership for Patients funded the project as part of 5 Leading Edge Advanced Practice Topics projects. The project was implemented across our system between September 2013 and December 2014. Selection criteria. Three procedures (cholecystectomy, colectomy, and hysterectomy), performed either laparoscopically or robotically, were chosen for assessment. We focused on these particular procedures because of the high frequency with which they are performed at our institution. Further, most operative complications associated
Surgery j 2016
with these particular cases occur within 90 days, allowing for rapid tracking of patient outcomes. To be eligible to participate in the study, surgeons had to perform 1 of these procedures at least twice during a 6-month period; therefore, case volumes of all general and gynecologic surgeons in our system over the previous year (2013) were reviewed. Invitation to participate in the study was extended only to those meeting the eligibility criteria. Recruitment was performed through physician practice meetings, word of mouth, Physician Portal, e-mail, and one-on-one meetings. Videotaping process. At least 2 technical and nontechnical intraoperative videos were randomly recorded for participating surgeons during a 6month period from December 1, 2013 to May 31, 2014. For the technical recordings, operating room (OR) circulators were asked to record the feed from the laparoscopic/robotic camera. Recording started once the feed became available (during abdominal entry) and finished when the feed was turned off (after CO2 desufflation of the abdominal cavity). The recording was then burned on a DVD, which was collected by the study coordinator for later review. Nontechnical recordings were obtained by a videographer. To maintain confidentiality, recordings started only after the patient had been prepped and draped, just before the operative timeout, and finished when the surgeon exited the room. Further, the surgeon’s faces and any patient identifiers were pixelated, and the technical recordings were embedded in a corner of the nontechnical recording. Video assessment. All video reviewers were blinded to the identity of the surgeon they were reviewing and watched each video in its entirety. The technical recordings were reviewed by 2 minimally invasive surgeons with expertise in laparoscopic and robotic surgery. An expert minimally invasive general surgeon who has been in practice for >10 years and performs more than 350 operations a year (DS) reviewed the cholecystectomies and colectomies. An expert gynecologic surgeon (RVH) who has been in practice for 25 years with 150–200 operative cases a year reviewed the hysterectomies. The reviewers recorded any errors and skill deficiencies and identified which deficiencies were amenable to training and what type of training might best address them. To objectively assess the videos, the reviewers used established, previously validated assessment tools, such as the Global Objective Assessment of Laparoscopic
ARTICLE IN PRESS Surgery Volume j, Number j Skills,13 the Global Evaluative Assessment of Robotic Skills,14 and 3 Competency Assessment Tools15,16: 1 for laparoscopic/robotic cholecystectomy, 1 for laparoscopic colectomy, and 1 for laparoscopic/robotic hysterectomy. A human factors expert (BAM) with >10 years of training/experience reviewed the nontechnical recordings. As with the technical recordings, the reviewer recorded any existing deficiencies in nontechnical skills and identified potential training techniques to address them. The reviewer used a new rating tool that is currently going through the validation process, based on established, validated, nontechnical evaluation tools: the Oxford Non-Technical Skills,17 Observational Teamwork Assessment for Surgery,18 NonTechnical Skills for Surgeons (NOTSS),19,20 and NOn-TECHnical Skills.21 In addition, the new tool also addressed interruptions/distractions and workload/stress. Coaching curriculum development. Once video review had been completed, the study group met to develop a curriculum that would address the identified deficiencies and that could be offered to participating surgeons. After collecting feedback from our participants, we determined that a 4.5hour coaching session would best accomplish the desired training and accommodate our participants’ busy schedules. The curriculum developed for the coaching session is shown in Fig 1 and was delivered in our level I American College of Surgeons–Accredited Education Institute. The first hour and a half were led by one of the Minimally Invasive Surgery (MIS) experts who presented a brief overview of the study rationale and objectives, review of the importance of deliberate practice and feedback for performance improvement, and review of the stages of skill acquisition and the impact of stress and attentional capacity on performance. Short video segments (15 seconds to 2 minutes) demonstrating preventable errors or skill deficiencies acquired from the video review were shown to participating surgeons who were asked to comment on the observed performance. The human factors expert then provided a 30-minute presentation addressing nontechnical skill deficiencies and remedies for each of the following areas: communication, situation awareness/vigilance, cooperation/ team skills, leadership, interruptions/distractions, and workload/stress. The areas identified with the most opportunity for improvement are discussed in more detail in the results. During the second part of the coaching session, participants rotated every 30 minutes among 4 skill
Stefanidis et al 3
stations: 3 technical and 1 nontechnical. The technical stations focused on skills training, in particular peg transfer, intracorporeal suturing, and simulated laparoscopic cholecystectomies and hysterectomies. The investigators included these skill stations based on their relevance and the surgeons’ observed deficiencies during video review. The nontechnical station focused on interruptions and distractions, which were identified as primary areas of opportunity for physicians to improve their nontechnical skills (Table I). Participants were individually debriefed at the end of the nontechnical training session, and a final group debrief was performed upon completion of the curriculum. RESULTS As shown in Fig 2, 90 surgeons from 6 different CHS hospitals were invited to participate. Of these 90, 32 surgeons (18 general; 14 gynecologists) agreed to be videotaped and evaluated. Forty-two technical videos and 45 nontechnical videos were assessed by the surgeon experts and the human factors expert, respectively (Table II22). In a few cases, there was a technical problem capturing a recording, resulting in the discrepant number of total recordings. Seven surgeons volunteered to participate in the group coaching session. An additional 2 could not attend the group session but participated in one-on-one coaching sessions on technical skills. One surgeon participated in both the group and the individual coaching sessions. All surgeons appropriately accomplished some of the objectives of the distraction scenario, but no surgeon accomplished all objectives, which were to handle each interruption/distraction appropriately. Further, no participating surgeon was able to achieve expert levels23 on the 2 Fundamentals of Laparoscopic Surgery (FLS) tasks used. Participants perceived the coaching sessions as highly valuable. One participant stated in her evaluation of the sessions: “I wanted to thank you for the learning session yesterday. I found it very helpful. I just finished my first of 4 lap choles for today and found myself implementing several things we talked about yesterday. Look forward to continuing to work with you. Thanks for all your hard work on the effort.” The type and frequency of technical and nontechnical skill deficiencies identified for improvement are shown in Table III and described in greater detail in the following paragraphs. While very few significant complications were observed in the reviewed videos, several “near misses” and areas for improvement were noted.
ARTICLE IN PRESS 4 Stefanidis et al
Surgery j 2016
Fig 1. Group coaching session outline.
Identified preventable technical performance issues. Port placement. Suboptimal port placement was observed in several general and gynecologic procedures (in 42% of participating surgeons) and resulted in 2 main concerns. First, safety was compromised during introduction of trocars inside the abdominal cavity. In a number of videos, surgeons did not orient their camera appropriately to keep in view the tip of the trocar entering the abdominal cavity, which may on occasion lead to vascular or intestinal injury. Second, inappropriate angle of trocar insertion axis relevant to the
dissection target was observed in a number of videos. In the obese patient, who has a thick abdominal wall, use of instruments through the malpositioned port becomes more difficult and negatively impacts fine motor skill. Indeed, 1 surgeon had to replace the trocar to safely complete the procedure. Visualization of the operating field. In 35% of videos, visualization of the operating field was suboptimal, yet surgeons continued dissecting despite an unclear image (fogged, smudged lens, etc), and in 1 case, a surgeon dissected an area that
ARTICLE IN PRESS Stefanidis et al 5
Surgery Volume j, Number j
Table I. Description of training stations Station category Technical: FLS tasks Technical: simulated procedure
Technical: controlling bleed Nontechnical: mitigating interruptions and distractions
Description Participants practiced peg transfer and intracorporeal knot tying. Gynecology participants practiced performing a simulated laparoscopic hysterectomy on a Gaumard hysterectomy model, while general surgeons practiced performing a simulated laparoscopic cholecystectomy on a Simulab cholecystectomy model. Participants practiced controlling a bleed using a vena cava model developed by the University of Minnesota CREST Lab.22 Participants practiced mitigation techniques for interruptions and distractions in a simulated operating room while performing a simulated operative procedure.
FLS, Fundamentals of Laparoscopic Surgery; CREST, Center for Research in Education and Simulation Technologies.
was not directly visualized. Further, in some cases, excessive camera instability was noted to the point that it was difficult for the reviewers to watch the video. Given that laparoscopy and robotic surgery are directly dependent on accurate visualization of the operating field (surgeons are only aware of what they can see in the monitor), camera issues (uncleanliness, instability) may potentially lead to inadvertent errors. Retraction and exposure. We observed a plethora of exposure issues in 46% of participating surgeons, including inadequate retraction with the nondominant (ie, nondissecting) hand (ie, too weak pull leading to suboptimal exposure), wrong direction of retraction (ie, instead of pulling a structure in a direction that would optimize camera visualization, pulling it toward the camera and obstructing its view), or retraction of the wrong structure (ie, pulling on another structure would have significantly improved exposure). Dissection technique and safe use of energy. We identified a number of dissection techniques in need of improvement in 42% of procedures we recorded. Examples included dissecting adhesions off the gallbladder infundibulum using electrosurgery in a downward direction (risking injury to the duodenum if control of the dissecting instrument was lost; this issue did actually occur in 1 case where the duodenum was briefly touched with the hot hook cautery but without immediate consequences); using electrosurgery too close to the duodenum when separating it from the gallbladder; using electrosurgery too close to a metallic clip that can lead to energy transmission and burn injury to important structures, such as the bile duct, during laparoscopic cholecystectomy; dissecting with electrosurgery in close proximity to the common bile duct, or the division of the cervix during hysterectomy in a downward fashion risking injury to
underlying intestines, or using ultrasonic energy too close to bowel (touching it) when the instrument is still hot during colectomy. Bimanual dexterity. The most common and notable deficiencies in 58% of participants were poor bimanual coordination, inadequate fine motor control, and limited-to-absent nondominant hand use (which negatively impacts exposure and procedural safety as described above). Adherence to safe practices. In 54% of the videos we reviewed, we observed cases where safe practices, such as identification of the ureter(s) during hysterectomy or obtaining the critical view of safety during cholecystectomy, were not followed (eg, in several cases, the ureters were identified after the hysterectomy had been completed or not at all). Importantly, for laparoscopic cholecystectomies, we identified that very few surgeons obtained the Critical View of Safety, an established method that prevents misidentification of the bile ducts and helps to avoid biliary injury. Appropriate strategies for “unexpected” occurrences and intraoperative decision-making. In 35% of cases, we observed suboptimal decision making. Most notable was the management of unexpected bleeding occurring from the middle colic artery and vein during a colectomy. While the surgeon was able to control bleeding eventually, a poor strategy and likely limited ability to control excessive stress led to significant blood loss. Nontechnical review of the same case revealed an instrument missing from the room, which may have affected the surgeon’s ability to control the bleed expeditiously; this highlights the need to embrace and complete the preoperative checklist, thereby ensuring all required and potentially needed instruments are available and working. Other issues. In one of the recorded laparoscopic cholecystectomy procedures, a diaphragmatic
ARTICLE IN PRESS 6 Stefanidis et al
Surgery j 2016
Table II. Number of videos assessed by procedure type Number of videos assessed
Fig 2. Overview of surgeon recruitment.
injury occurred when an assistant lost control of the instrument holding the gallbladder. While not every such occurrence is preventable, surgeons’ appropriate situational awareness is paramount for the safe conduct of an operation. Identified preventable nontechnical performance issues. Response to interruptions and distractions. In all of the nontechnical recordings, a number of interruptions and distractions occurred that were suboptimally handled by surgeons 58% of the time. The OR door was observed opening on a plethora of occasions, phones rang and were answered, pagers beeped frequently, music was often on (sometimes at a very high volume), a lot of social talk among team members was observed, and discussion about other patients were commonly observed interruptions and distractions. Many of these could be managed more appropriately by requesting that only personnel essential to the operation enter and exit the OR, requesting that if a phone call or page is not an emergency to take a message that will be returned after the operation, and keeping music at an appropriate volume. Further, although all distractions need to be minimized as they can lead to deleterious consequences at any stage of the operation, there are certain task critical stages
Procedure
Technical
Nontechnical
Laparoscopic cholecystectomy Robotic cholecystectomy Laparoscopic colectomy Laparoscopic hysterectomy Robotic hysterectomy
16 1 4 14 7
16 1 4 15 9
during an operation, such as the timeout, counting, and intubation, that might benefit from instituting a “quiet OR,” similar to the sterile cockpit in aviation. Ergonomic positioning. In all reviewed videos, poor ergonomic positions were observed. In many cases, the monitor used by the surgeon was positioned too high, resulting in the surgeon looking up, which puts strain on the neck, shoulders, and upper extremities24; adjusting the monitor to a lower position would help mitigate this strain. Further, in a number of instances, the surgeons were observed adopting awkward positions of their hands and arms (eg, wrists bent at almost 908 angles, elbows high in the air), often for long periods of time. These positions can lead to strain and repetitive use disorders25 and also restrict freedom of movement,26 which could decrease operative precision and accuracy. Indeed, some of the surgeons who participated in the coaching session indicated they suffered from carpal tunnel syndrome and numbness, especially in their phalanges. Adjustment of the operating table, using step stools as necessary, and taking small breaks to stretch can help mitigate strain.26,27 Situational awareness and shared mental models. In 67% of videos, we observed that the operative team members would get into their own “bubbles” and refrain from communicating well with the rest of the OR team. Often the anesthesiologist and the surgeon did not communicate about the patient for the entire case, thus decreasing the overall situational awareness for the whole OR team as well as the ability to share mental models. Encouraging the OR team to communicate throughout the case helps each member to understand the patient’s status and to anticipate next steps and what might be needed to ensure patient safety. Mitigation of delays. Avoidable delays were observed in 92% of videos. The most common reason for delays was missing tools/equipment and equipment that was not working properly (eg, laparoscopic light source malfunctioning, monitor
ARTICLE IN PRESS Stefanidis et al 7
Surgery Volume j, Number j
Table III. Type and frequency of identified technical and nontechnical skill deficiencies Technical deficiency
%
Nontechnical deficiency
%
Port placement Visualization of the operating field Retraction and exposure Dissection technique and safe use of energy Bimanual dexterity Adherence to safe practices Decision-making
42 35 46 42 58 54 35
Response to interruptions and distractions Ergonomic positioning Mitigation of delays Situational awareness and shared mental models Pre- and postbrief
58 100 92 67 92
not turning on). Most of these delays could have been avoided by checking all the equipment prior to starting the operation. As noted previously, the lack of proper equipment contributed to a delay in controlling an intraoperative bleed. Insufficient pre- and postbrief. Our institution has adopted a modified version of the WHO checklist,28 which includes a preoperative timeout and a postoperative debrief. Our video review revealed that in 92% of cases, adherence to this process was inadequate. Although the preoperative timeout was observed for almost every case, it was often not performed in its entirety, or items were checked off without adequately performing the required task. An example previously mentioned was a lack of verification that all needed and anticipated equipment was available and working properly. Further, a postoperative debrief in which the surgeon participated was rarely observed. Handoffs of patients have been identified as an area that is prone to communication errors, and handoffs have been implicated in a number of patient harm events.29 Performing a surgeon-led postoperative debrief helps identify and confirm any patient concerns for recovery. It should also be noted that the cost of conducting this study was calculated at $2,640 per participating surgeon. This cost included video recording and editing charges, study coordination, coach salary, and supplies used and was dependent on the number of participating surgeons. DISCUSSION In this study, we aimed to identify technical and nontechnical skill deficiencies of practicing surgeons and to develop a coaching mechanism that would address the lack of performance feedback available to surgeons today. To accomplish this, we embarked on a video-based assessment of technical and nontechnical skill of attending surgeons and created a coaching program that was based on an interactive video review and simulation-based practice.
Our approach allowed us to demonstrate the importance of ongoing performance assessment, as several deficiencies amenable to training were identified that otherwise likely would have gone undetected. The observed skill deficiencies for a variety of surgeons fell into several common categories, which demonstrate the need for targeted curriculum development. These areas are highlighted in this paper and can be useful for educators and coaches in surgery. Importantly, several of our participating surgeons were very appreciative for being afforded this coaching opportunity, as they themselves felt this mechanism of performance feedback was missing from their practice. The importance of coaching for performance improvement and professional development is widely recognized across many disciplines, such as the sports industry,12 education,30 managerial training,31 and in health care.32 While coaching techniques have some overlap with traditional teaching methods, there are differences in the roles that a teacher and coach fulfill.33 Coaching focuses on an individual’s professional development through one-on-one reflective dialogue with their coach to monitor and evaluate progress toward specific goals and to modify action plans on the basis of feedback.34 The Kansas Coaching Project has identified factors that are often cited as critical for coaching success. Some of these include having a learning-friendly culture where the coachee is shown respect and provided encouragement; coachees voluntarily and willingly agree to participate; and an iterative approach to coaching is adopted, recognizing that continuous learning is imperative to success.34 Based on these principles, we designed a 5-step coaching model to be used in this study (Fig 3): (1) Assess skills through video. Our model starts with video-based performance assessment of technical and nontechnical skills through objective-based metrics. Expert surgeons assess their peers’ technical skills, while a human factors expert assesses
ARTICLE IN PRESS 8 Stefanidis et al
Fig 3. Carolinas coaching model. (Color version of this figure is available online.)
the surgeons’ nontechnical skills. The aspects of performance in need of improvement are defined by this expert-driven process. (2) Identify areas for growth. The second stage of the process is to synthesize all the observed data, thus identifying common group and individual performance deficiencies and guiding goal setting for coaching (ie, which aspects of performance an individual needs to improve). Consistent with the Kansas Coaching Project34 observations, we found it is imperative that the coachee exhibits the willingness/desire to improve his or her performance. (3) Group review and individual feedback. We propose that the next step in the process should be performance feedback by the coach. This step represents the heart of our coaching model, as it provides the necessary information and rationale to the coachee of what needs to be improved and how it can be accomplished.
Our experience in this study indicated that coachees benefit most from feedback provided in a group session and in one-on-one sessions. We identified several advantages of adding a blinded group review session to one-on-one coaching. Group sessions enable peer review; therefore, surgeons can watch other surgeons operate and learn from their mistakes and from their successes. Indeed, several of our participating surgeons indicated that they found value in reviewing each other’s operations, as they saw some techniques they liked and identified issues they clearly perceived as errors in others.
Surgery j 2016
Another advantage of the blinded group debrief session is that they may be less threatening to surgeons because the critiqued videos are deidentified, so participants can freely express their opinion without feeling vulnerable or defensive. Research indicates that humans are a lot better at critiquing others than at performing self-critique (which one-on-one coaching calls for).7 Consequently, group sessions prime participants for the one-on-one coaching sessions; deficiencies identified collectively during the group session cannot be refuted or overseen by coachees during individual sessions, making it more likely to leave a lasting impression on them and bring about the desired behavior change. On the other hand, interactive one-on-one sessions are necessary and invaluable, as they allow specific feedback tailored to individual needs. Together, coach and coachee can develop goals and strategies to achieve them. (4) Deliberate practice with coaching. The next step in our model consists of deliberate practice on the issues identified during the previous steps. Simulation can greatly enhance the effectiveness of this step, as it enables repetitive practice of deficient procedural skills at the surgeon’s preferred time, which otherwise may be impossible based on clinical practice alone for rarely used skills or encountered conditions. It further allows for the provision of formative and summative performance feedback that is essential for the acquisition of skills.35 Simulation also allows for nontechnical skill training by using clinical and/or team training scenarios that can effectively deliver the desired intervention to participants. The accompanying debrief is a great opportunity to highlight good and poor behaviors and to impart to participants important take-home messages. (5) Monitor patient outcomes. The final step in our model consists of patient outcome monitoring, which we are currently assessing for the first cohort of trainees. This step is crucial, as it provides a good measure of effectiveness and ensures that the coaching intervention benefits patients. The loop closes by repeat skill assessment and restarting of the described process. We strongly believe that this feedback loop needs to be ongoing as behavioral change happens over time, and limited interventions are less likely to be successful.
Recently, and after we had embarked on our coaching curriculum development, Greenberg et al9 reported their coaching approach: The Wisconsin Surgical Coaching Framework, which targets performance improvement in 3 domains (technical skill, cognitive skill, and nontechnical skill) and proposes
ARTICLE IN PRESS Stefanidis et al 9
Surgery Volume j, Number j
3 distinct but interrelated activities of coaching (setting goals, encouraging and motivating, and developing and guiding). The 2 models use different definitions of assessed skills (ie, cognitive skills as defined by Greenberg et al9 are included in our technical and nontechnical skill definitions). Further, our model uses both group and individual coaching sessions, employs simulation as a means for further practice and feedback, and has a focus on patient outcomes. On the other hand, both studies use intraoperative video recordings to assess the participating surgeon’s skill and focus on targeted feedback provision to the coachee. Another group from the United Kingdom has developed and assessed a coaching curriculum using medical students who were laparoscopic novices. Singh et al11 based their coaching curriculum on the Goal, Reality, Options, Way Forward (GROW) model that guides coaching conversations according to a 4-step process: initially the goals of coaching are set with the coachee, followed by performance review, formulation of effective solutions, and development of an action plan targeting goal achievement and taking into consideration potential anticipated obstacles. Using this coaching approach, Singh et al11 provided video-based coaching to an intervention group of novices training on laparoscopic skills and demonstrated enhanced quality of laparoscopic operative performance during both virtual reality and porcine laparoscopic cholecystectomies compared with a control group receiving the same amount of training but in the absence of coaching. The authors concluded that video-based coaching maximizes performance enhancement from every clinical exposure. A very important and relevant question is who the operative coach should be and what personal qualities he or she should possess. Greenberg et al9 suggest that excellent interpersonal skills including cooperative communication and active listening, ability to adapt coaching approaches to the individual’s needs, and reputability within their field are crucial traits of a good coach. Our experience indicates that the ideal coach is also dependent on the type of skill being addressed; for technical skills, it should be a respected peer with expertise in the domain that is being coached (ie, it will be difficult for a breast surgeon to coach a minimally invasive surgeon and vice versa). On the other hand, for nontechnical skills, the ideal coach might not be a peer surgeon but rather an individual with a human factors background. While peer surgeons might be able to provide feedback on nontechnical skills, they typically lack human factors expertise
and are likely to be less effective and less respected as coaches for these types of skills. Our study is not without limitations. Foremost, a fraction (28%) of our enrolled surgeons, possibly the most motivated, participated in any of the coaching sessions, thus limiting our ability to draw firm conclusions about the effectiveness of our coaching model. The limited participation rate reflects the difficulty in engaging busy practicing surgeons. It may also reflect hesitation to receive feedback. In addition, though we developed a robust coaching mechanism for practicing surgeons, we did not demonstrate conclusively its effectiveness; however, we have identified some positive trends. To address this issue, we are currently expanding our coaching paradigm and collecting evidence of effectiveness, including patient outcomes, and we hope to report our findings in upcoming publications. Another potential limitation for widespread implementation is that technical skills are easier to review blindly during group sessions (easy to obtain and reviewers only see the operative field and cannot identify who the surgeon is), while nontechnical skills are significantly more challenging (very time consuming and costly to blind them), limiting their feasibility. Further research may demonstrate the true benefits and best combination of our proposed dual approach to feedback provision. In conclusion, our study identified several technical and nontechnical skill sets of practicing surgeons in need of improvement and provided support for the implementation of coaching programs for surgeons on an ongoing basis. REFERENCES 1. de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA. The incidence and nature of in-hospital adverse events: A systematic review. Qual Saf Health Care 2008;17:216-23. 2. The National Patient Safety Foundation. Agenda for research and development in patient safety. [Internet]. Boston (MA): National Patient Safety Foundation; 2016. Available from:, https://c.ymcdn.com/sites/www.npsf. org/resource/collection/4B2E552F-48FA-4DCF-8BD8-574EE 15EFD99/Agenda_for_RD_in_Patient_Safety.pdf. 3. Bikrmeyer JD, Finks JF, O’Reilley A, Oerline M, Carlin AM, Nun AR, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med 2013;369:1434-42. 4. Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N. The impact of nontechnical skills on technical performance in surgery: A systematic review. J Am Coll Surg 2012;214:214-30. 5. Yule S, Patterson-Brown S. Surgeons’ non-technical skills. Surg Clin N Am 2012;92:37-50. 6. Hauer KE, Ciccone A, Henzel TR, Katsufrakis P, Miller SH, Norcross WA, et al. Remediation of the deficiencies of physicians across the continuum from medical school to practice: A thematic review of the literature. Acad Med 2009; 84:1822-32.
ARTICLE IN PRESS 10 Stefanidis et al
7. Davis DA, Mazmanian PE, Fordis M, Harrison RV, Thorpe KE, Perrier L. Accuracy of physician selfassessment compared with observed measures of competence: A systematic review. JAMA 2006;296:1094-102. 8. Wigton RS, Patil KD, Hoellerich VL. The effect of feedback in learning clinical diagnosis. J Med Educ 1986;61:816-22. 9. Stefanidis D. Optimal acquisition and assessment of proficiency on simulators in surgery. Surg Clin North Am 2010;90:475-89. 10. Greenberg CC, Ghousseini HN, Pavuluri Quamma SR, Beasley HL, Wiegmann DA. Surgical coaching for individual performance improvement. Ann Surg 2015;261:32-4. 11. Singh P, Aggarwal R, Tahir M, Pucher PH, Darzi A. A randomized controlled study to evaluate the role of video-based coaching in training laparoscopic skills. Ann Surg 2015;261:862-9. 12. Becker AJ. It’s not what they do, it’s how they do it: Athlete experiences of great coaching. Int J Sports Sci Coach 2009;4:93-119. 13. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondre K, Stanbridge D, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 2005;190:107-13. 14. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ. Global evaluative assessment of robotic skills: Validation of a clinical assessment tool to measure robotic surgical skills. J Urology 2012;187:247-52. 15. Cole SJ, Mackenzie H, Ha J, Hanna GB, Miskovic D. Randomized controlled trial on the effect of coaching in simulated laparoscopic training. Surg Endosc 2014;28:979-86. 16. Miskovic D, Ni M, Wyles SM, Kennedy RH, Francis NK, Parvaiz A, et al. Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg 2013;257:476-82. 17. Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS system: Reliability and validity of a tool for measuring teamwork behavior in the operating theatre. Qual Saf Health Care 2009;18:104-8. 18. Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N. Observational teamwork assessment for surgery: Content validation and tool refinement. J Am Coll Surg 2011;212:234-43. 19. Yule S, Rowley D, Flin R, Maran N, Youngson G, Duncan J, et al. Experience matters: Comparing novice and expert ratings of non-technical skills using the NOTSS system. ANZ J Surg 2009;79:154-60. 20. Yule S, Flin R, Maran N, Rowley D, Youngson G, PatersonBrown S. Surgeons’ non-technical skills in the operating room: Reliability testing of the NOTSS behavior rating system. World J Surg 2008;32:548-56. 21. Sevdalis N, Davis R, Koutantji M, Undre S, Darzi A, Vincent CA. Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg 2008;196:184-90.
Surgery j 2016
22. Office for Technology Commercialization. Synthetic inferior vena cava surgery training model. [Internet]. Minneapolis (MN): Regents of the University of Minnesota; 2016. Available from: http://license.umn.edu/technolo gies/20110220-4_synthetic-inferior-vena-cava-surgery-trainingmodel. 23. Ritter EM, Scott DJ. Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov 2007;14:107-12. 24. van Det MJ, Meijerink WJHJ, Hoff C, Totte ER, Pierie JPEN. Optimal ergonomics for laparoscopic surgery in minimally invasive surgery suites: A review and guidelines. Surg Endosc 2009;23:1279-85. 25. Miller K, Benden M, Pickens A, Shipp E, Zheng Q. Ergonomics principles associated with laparoscopic surgeon injury/illness. Hum Factors 2012;54:1087-92. 26. van Veelen MA, Kazemier G, Koopman J, Goossens RH, Meijer DW. Assessment of the ergonomically optimal operating surface height for laparoscopic surgery. J Laparoendosc Adv Surg Tech 2002;12:47-52. 27. Engelmann C, Schneider M, Kirschbaum C, Grote G, Dingemann J, Schoof S, et al. Effects of interoperative breaks in mental and somatic operator fatigue: A randomized clinical trial. Surg Endosc 2011;24:1245-50. 28. World Alliance for Patient Safety. WHO surgical safety checklist and implementation manual. [Internet]. Geneva (Switzerland): World Health Organization; 2016. Available from:, http://www.who.int/patientsafety/safesurgery/ss_ checklist/en/. 29. Riesenberg LA, Leitzsch J, Massucci JL, Jaeger J, Rosenfeld JC, Patow C, et al. Residents’ and attending physicians’ handoffs: A systematic review of the literature. Acad Med 2009;84:1775-87. 30. Costa AL, Garmston RJ. Cognitive coaching: A foundation for renaissance schools. Norwood (MA): Christopher-Gordon Publishers; 1994. 31. Longenecker CO, Neubert MJ. The practices of effective managerial coaches. Bus Horizons 2005;48:493-500. 32. Velasquez M, Shellenberger S, von Sternberg K. Training health-care practitioners in screening, brief intervention, and referral to treatment using standardized approaches and expert coaching. Addict Sci Clin Pract 2012;7(Suppl 1):A95. 33. Rupert T, Buschner C. Teaching and coaching: A comparison of instructional behaviors. J Teach Phys Educ 1989;9: 49-57. 34. Knight J. Coaching. J Staff Dev 2009;30:18–22. 35. Scerbo MW, Anderson BL. Medical simulation. In: Carayon P, editor. Handbook of human factors and ergonomics in health care and patient safety. Boca Raton (FL): Taylor and Francis Group; 2012.
Developing a coaching mechanism for practicing surgeons Dimitrios Stefanidis, MD, PhD, FACS, FASMBS,a,b Brittany Anderson-Montoya, PhD,c Robert V. Higgins, MD,d Manuel E. Pimentel,b Patrick Rowland,c Madison O. Scarborough, MBA, MHA,c and Danelle Higgins, RN, BSN, MHA,c Charlotte, NC
Background. While performance feedback and assessment are hallmarks of surgical training, they abruptly cease after training is completed. In their absence, performance may stagnate and poor habits persist. Our aim was to develop a coaching mechanism for practicing surgeons with feedback provision based on objective performance assessment. Methods. Technical and nontechnical intraoperative video recordings from laparoscopic or robotic cholecystectomies, colectomies, and hysterectomies were assessed by a blinded surgeon and a human factors expert, respectively. Aspects of performance in need of improvement were noted, and a coaching session was developed for feedback provision to participating surgeons. This 4-hour coaching session consisted of a didactic lecture with video review and hands-on practice using procedural and mannequin-based simulation. Results. Thirty-two practicing surgeons (18 general; 14 gynecologists) from 6 different hospitals were assessed, and 9 of them participated in coaching. Technical aspects identified for performance improvement included suboptimal trocar placement, inadequate critical view achievement during laparoscopic cholecystectomies, poor visualization of the operating field, bimanual dexterity, and dissection techniques, while nontechnical aspects included inappropriate handling of distractions and interruptions, poor ergonomic positioning and situational awareness, and inadequate mitigation of delays. Most surgeons appropriately accomplished some of the objectives of the distraction scenario, but none was able to achieve expert levels on Fundamentals of Laparoscopy tasks. Participants perceived the coaching sessions as highly valuable. Conclusion. Our study identified several technical and nontechnical skill sets of practicing surgeons in need of improvement and provided support for the implementation of coaching programs for surgeons on an ongoing basis. (Surgery 2016;j:j-j.) From the Department of Surgery,a Carolinas Simulation Center,b Division of Quality,c and the Department of OB/Gynecology,d Carolinas HealthCare System, Charlotte, NC
SURGERY IS A HIGHLY TECHNICAL, high-volume, high-risk field of health care that impacts millions of patients. A systematic review indicated that the majority of Supported through a Leading Edge Advanced Practice Topics grant by the Center for Medicare and Medicaid Services contract number HHSM-500-2012-00028C/000005. Presented from the podium at the annual consortium meeting of the American College of Surgeons–Accredited Education Institutes in Chicago, IL, on March 14th, 2015. Accepted for publication March 11, 2016. Reprint requests: Dimitrios Stefanidis, MD, PhD, FACS, FASMBS, Carolinas Simulation Center, and Division of Bariatric Surgery, Department of Surgery, Carolinas HealthCare System, 1025 Morehead Medical Dr, Suite 300, Charlotte, NC 28204. E-mail: [email protected]. 0039-6060/$ - see front matter Ó 2016 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.surg.2016.03.036
adverse events in health care were related to operations and surgical staff.1 Therefore, it is imperative that an understanding of what is causing these adverse events be established to mitigate future adverse events. Indeed, the National Patient Safety Foundation’s (NPSF’s) Agenda for Research in Patient Safety states, “One of the most important approaches to patient safety research is to understand the nature of technical work.”2 Although it seems intuitive that surgeon skill directly impacts patient safety, it was not until recently that a study demonstrated that patients of bariatric surgeons whose skill was rated higher by their peers on video-based assessment had fewer postoperative complications compared with those of surgeons with lower skill levels.3 Further, the impact of poor nontechnical skills is increasingly recognized as a significant contributing factor to adverse events and needs to SURGERY 1
ARTICLE IN PRESS 2 Stefanidis et al
be assessed in relation to a physician’s technical skills.4,5 Despite the recognized importance and widespread endorsement of competency-based assessment for physicians in training, objective methods to assess practicing physician performance are rarely used today. Further, amidst growing demand for transparency of clinical outcomes, hospitals are struggling with the question of how to objectively measure physician competency and quality. Without an objective assessment, it is very difficult to accurately identify physicians in need of performance improvement and to help them remediate their deficiencies.6,7 Research has demonstrated that performance feedback can lead to improved clinical and technical skills, and its absence usually leads to an inability to achieve a higher level of performance.8,9 The importance of feedback for performance improvement and the lack thereof for practicing surgeons have led several authors to recommend that coaching programs for surgeons be developed.10,11 The value of coaching is well recognized in several disciplines, such as sports,12 and is geared toward performers at all levels, not just trainees. The first aim of this study was to identify technical and nontechnical deficiencies in practicing surgeons based on video review of their performance. The second aim, based on the identified deficiencies, was to develop and pilot a coaching mechanism for practicing surgeons with feedback provision. METHODS This project was an Institutional Review Board– approved, prospective quality improvement study with voluntary enrollment of practicing general and gynecologic surgeons. The study was conducted in 6 hospitals of Carolinas HealthCare System (CHS), a large not-for-profit health care system comprising 43 hospitals and more than 60,000 employees. The Center for Medicare and Medicaid Services Partnership for Patients funded the project as part of 5 Leading Edge Advanced Practice Topics projects. The project was implemented across our system between September 2013 and December 2014. Selection criteria. Three procedures (cholecystectomy, colectomy, and hysterectomy), performed either laparoscopically or robotically, were chosen for assessment. We focused on these particular procedures because of the high frequency with which they are performed at our institution. Further, most operative complications associated
Surgery j 2016
with these particular cases occur within 90 days, allowing for rapid tracking of patient outcomes. To be eligible to participate in the study, surgeons had to perform 1 of these procedures at least twice during a 6-month period; therefore, case volumes of all general and gynecologic surgeons in our system over the previous year (2013) were reviewed. Invitation to participate in the study was extended only to those meeting the eligibility criteria. Recruitment was performed through physician practice meetings, word of mouth, Physician Portal, e-mail, and one-on-one meetings. Videotaping process. At least 2 technical and nontechnical intraoperative videos were randomly recorded for participating surgeons during a 6month period from December 1, 2013 to May 31, 2014. For the technical recordings, operating room (OR) circulators were asked to record the feed from the laparoscopic/robotic camera. Recording started once the feed became available (during abdominal entry) and finished when the feed was turned off (after CO2 desufflation of the abdominal cavity). The recording was then burned on a DVD, which was collected by the study coordinator for later review. Nontechnical recordings were obtained by a videographer. To maintain confidentiality, recordings started only after the patient had been prepped and draped, just before the operative timeout, and finished when the surgeon exited the room. Further, the surgeon’s faces and any patient identifiers were pixelated, and the technical recordings were embedded in a corner of the nontechnical recording. Video assessment. All video reviewers were blinded to the identity of the surgeon they were reviewing and watched each video in its entirety. The technical recordings were reviewed by 2 minimally invasive surgeons with expertise in laparoscopic and robotic surgery. An expert minimally invasive general surgeon who has been in practice for >10 years and performs more than 350 operations a year (DS) reviewed the cholecystectomies and colectomies. An expert gynecologic surgeon (RVH) who has been in practice for 25 years with 150–200 operative cases a year reviewed the hysterectomies. The reviewers recorded any errors and skill deficiencies and identified which deficiencies were amenable to training and what type of training might best address them. To objectively assess the videos, the reviewers used established, previously validated assessment tools, such as the Global Objective Assessment of Laparoscopic
ARTICLE IN PRESS Surgery Volume j, Number j Skills,13 the Global Evaluative Assessment of Robotic Skills,14 and 3 Competency Assessment Tools15,16: 1 for laparoscopic/robotic cholecystectomy, 1 for laparoscopic colectomy, and 1 for laparoscopic/robotic hysterectomy. A human factors expert (BAM) with >10 years of training/experience reviewed the nontechnical recordings. As with the technical recordings, the reviewer recorded any existing deficiencies in nontechnical skills and identified potential training techniques to address them. The reviewer used a new rating tool that is currently going through the validation process, based on established, validated, nontechnical evaluation tools: the Oxford Non-Technical Skills,17 Observational Teamwork Assessment for Surgery,18 NonTechnical Skills for Surgeons (NOTSS),19,20 and NOn-TECHnical Skills.21 In addition, the new tool also addressed interruptions/distractions and workload/stress. Coaching curriculum development. Once video review had been completed, the study group met to develop a curriculum that would address the identified deficiencies and that could be offered to participating surgeons. After collecting feedback from our participants, we determined that a 4.5hour coaching session would best accomplish the desired training and accommodate our participants’ busy schedules. The curriculum developed for the coaching session is shown in Fig 1 and was delivered in our level I American College of Surgeons–Accredited Education Institute. The first hour and a half were led by one of the Minimally Invasive Surgery (MIS) experts who presented a brief overview of the study rationale and objectives, review of the importance of deliberate practice and feedback for performance improvement, and review of the stages of skill acquisition and the impact of stress and attentional capacity on performance. Short video segments (15 seconds to 2 minutes) demonstrating preventable errors or skill deficiencies acquired from the video review were shown to participating surgeons who were asked to comment on the observed performance. The human factors expert then provided a 30-minute presentation addressing nontechnical skill deficiencies and remedies for each of the following areas: communication, situation awareness/vigilance, cooperation/ team skills, leadership, interruptions/distractions, and workload/stress. The areas identified with the most opportunity for improvement are discussed in more detail in the results. During the second part of the coaching session, participants rotated every 30 minutes among 4 skill
Stefanidis et al 3
stations: 3 technical and 1 nontechnical. The technical stations focused on skills training, in particular peg transfer, intracorporeal suturing, and simulated laparoscopic cholecystectomies and hysterectomies. The investigators included these skill stations based on their relevance and the surgeons’ observed deficiencies during video review. The nontechnical station focused on interruptions and distractions, which were identified as primary areas of opportunity for physicians to improve their nontechnical skills (Table I). Participants were individually debriefed at the end of the nontechnical training session, and a final group debrief was performed upon completion of the curriculum. RESULTS As shown in Fig 2, 90 surgeons from 6 different CHS hospitals were invited to participate. Of these 90, 32 surgeons (18 general; 14 gynecologists) agreed to be videotaped and evaluated. Forty-two technical videos and 45 nontechnical videos were assessed by the surgeon experts and the human factors expert, respectively (Table II22). In a few cases, there was a technical problem capturing a recording, resulting in the discrepant number of total recordings. Seven surgeons volunteered to participate in the group coaching session. An additional 2 could not attend the group session but participated in one-on-one coaching sessions on technical skills. One surgeon participated in both the group and the individual coaching sessions. All surgeons appropriately accomplished some of the objectives of the distraction scenario, but no surgeon accomplished all objectives, which were to handle each interruption/distraction appropriately. Further, no participating surgeon was able to achieve expert levels23 on the 2 Fundamentals of Laparoscopic Surgery (FLS) tasks used. Participants perceived the coaching sessions as highly valuable. One participant stated in her evaluation of the sessions: “I wanted to thank you for the learning session yesterday. I found it very helpful. I just finished my first of 4 lap choles for today and found myself implementing several things we talked about yesterday. Look forward to continuing to work with you. Thanks for all your hard work on the effort.” The type and frequency of technical and nontechnical skill deficiencies identified for improvement are shown in Table III and described in greater detail in the following paragraphs. While very few significant complications were observed in the reviewed videos, several “near misses” and areas for improvement were noted.
ARTICLE IN PRESS 4 Stefanidis et al
Surgery j 2016
Fig 1. Group coaching session outline.
Identified preventable technical performance issues. Port placement. Suboptimal port placement was observed in several general and gynecologic procedures (in 42% of participating surgeons) and resulted in 2 main concerns. First, safety was compromised during introduction of trocars inside the abdominal cavity. In a number of videos, surgeons did not orient their camera appropriately to keep in view the tip of the trocar entering the abdominal cavity, which may on occasion lead to vascular or intestinal injury. Second, inappropriate angle of trocar insertion axis relevant to the
dissection target was observed in a number of videos. In the obese patient, who has a thick abdominal wall, use of instruments through the malpositioned port becomes more difficult and negatively impacts fine motor skill. Indeed, 1 surgeon had to replace the trocar to safely complete the procedure. Visualization of the operating field. In 35% of videos, visualization of the operating field was suboptimal, yet surgeons continued dissecting despite an unclear image (fogged, smudged lens, etc), and in 1 case, a surgeon dissected an area that
ARTICLE IN PRESS Stefanidis et al 5
Surgery Volume j, Number j
Table I. Description of training stations Station category Technical: FLS tasks Technical: simulated procedure
Technical: controlling bleed Nontechnical: mitigating interruptions and distractions
Description Participants practiced peg transfer and intracorporeal knot tying. Gynecology participants practiced performing a simulated laparoscopic hysterectomy on a Gaumard hysterectomy model, while general surgeons practiced performing a simulated laparoscopic cholecystectomy on a Simulab cholecystectomy model. Participants practiced controlling a bleed using a vena cava model developed by the University of Minnesota CREST Lab.22 Participants practiced mitigation techniques for interruptions and distractions in a simulated operating room while performing a simulated operative procedure.
FLS, Fundamentals of Laparoscopic Surgery; CREST, Center for Research in Education and Simulation Technologies.
was not directly visualized. Further, in some cases, excessive camera instability was noted to the point that it was difficult for the reviewers to watch the video. Given that laparoscopy and robotic surgery are directly dependent on accurate visualization of the operating field (surgeons are only aware of what they can see in the monitor), camera issues (uncleanliness, instability) may potentially lead to inadvertent errors. Retraction and exposure. We observed a plethora of exposure issues in 46% of participating surgeons, including inadequate retraction with the nondominant (ie, nondissecting) hand (ie, too weak pull leading to suboptimal exposure), wrong direction of retraction (ie, instead of pulling a structure in a direction that would optimize camera visualization, pulling it toward the camera and obstructing its view), or retraction of the wrong structure (ie, pulling on another structure would have significantly improved exposure). Dissection technique and safe use of energy. We identified a number of dissection techniques in need of improvement in 42% of procedures we recorded. Examples included dissecting adhesions off the gallbladder infundibulum using electrosurgery in a downward direction (risking injury to the duodenum if control of the dissecting instrument was lost; this issue did actually occur in 1 case where the duodenum was briefly touched with the hot hook cautery but without immediate consequences); using electrosurgery too close to the duodenum when separating it from the gallbladder; using electrosurgery too close to a metallic clip that can lead to energy transmission and burn injury to important structures, such as the bile duct, during laparoscopic cholecystectomy; dissecting with electrosurgery in close proximity to the common bile duct, or the division of the cervix during hysterectomy in a downward fashion risking injury to
underlying intestines, or using ultrasonic energy too close to bowel (touching it) when the instrument is still hot during colectomy. Bimanual dexterity. The most common and notable deficiencies in 58% of participants were poor bimanual coordination, inadequate fine motor control, and limited-to-absent nondominant hand use (which negatively impacts exposure and procedural safety as described above). Adherence to safe practices. In 54% of the videos we reviewed, we observed cases where safe practices, such as identification of the ureter(s) during hysterectomy or obtaining the critical view of safety during cholecystectomy, were not followed (eg, in several cases, the ureters were identified after the hysterectomy had been completed or not at all). Importantly, for laparoscopic cholecystectomies, we identified that very few surgeons obtained the Critical View of Safety, an established method that prevents misidentification of the bile ducts and helps to avoid biliary injury. Appropriate strategies for “unexpected” occurrences and intraoperative decision-making. In 35% of cases, we observed suboptimal decision making. Most notable was the management of unexpected bleeding occurring from the middle colic artery and vein during a colectomy. While the surgeon was able to control bleeding eventually, a poor strategy and likely limited ability to control excessive stress led to significant blood loss. Nontechnical review of the same case revealed an instrument missing from the room, which may have affected the surgeon’s ability to control the bleed expeditiously; this highlights the need to embrace and complete the preoperative checklist, thereby ensuring all required and potentially needed instruments are available and working. Other issues. In one of the recorded laparoscopic cholecystectomy procedures, a diaphragmatic
ARTICLE IN PRESS 6 Stefanidis et al
Surgery j 2016
Table II. Number of videos assessed by procedure type Number of videos assessed
Fig 2. Overview of surgeon recruitment.
injury occurred when an assistant lost control of the instrument holding the gallbladder. While not every such occurrence is preventable, surgeons’ appropriate situational awareness is paramount for the safe conduct of an operation. Identified preventable nontechnical performance issues. Response to interruptions and distractions. In all of the nontechnical recordings, a number of interruptions and distractions occurred that were suboptimally handled by surgeons 58% of the time. The OR door was observed opening on a plethora of occasions, phones rang and were answered, pagers beeped frequently, music was often on (sometimes at a very high volume), a lot of social talk among team members was observed, and discussion about other patients were commonly observed interruptions and distractions. Many of these could be managed more appropriately by requesting that only personnel essential to the operation enter and exit the OR, requesting that if a phone call or page is not an emergency to take a message that will be returned after the operation, and keeping music at an appropriate volume. Further, although all distractions need to be minimized as they can lead to deleterious consequences at any stage of the operation, there are certain task critical stages
Procedure
Technical
Nontechnical
Laparoscopic cholecystectomy Robotic cholecystectomy Laparoscopic colectomy Laparoscopic hysterectomy Robotic hysterectomy
16 1 4 14 7
16 1 4 15 9
during an operation, such as the timeout, counting, and intubation, that might benefit from instituting a “quiet OR,” similar to the sterile cockpit in aviation. Ergonomic positioning. In all reviewed videos, poor ergonomic positions were observed. In many cases, the monitor used by the surgeon was positioned too high, resulting in the surgeon looking up, which puts strain on the neck, shoulders, and upper extremities24; adjusting the monitor to a lower position would help mitigate this strain. Further, in a number of instances, the surgeons were observed adopting awkward positions of their hands and arms (eg, wrists bent at almost 908 angles, elbows high in the air), often for long periods of time. These positions can lead to strain and repetitive use disorders25 and also restrict freedom of movement,26 which could decrease operative precision and accuracy. Indeed, some of the surgeons who participated in the coaching session indicated they suffered from carpal tunnel syndrome and numbness, especially in their phalanges. Adjustment of the operating table, using step stools as necessary, and taking small breaks to stretch can help mitigate strain.26,27 Situational awareness and shared mental models. In 67% of videos, we observed that the operative team members would get into their own “bubbles” and refrain from communicating well with the rest of the OR team. Often the anesthesiologist and the surgeon did not communicate about the patient for the entire case, thus decreasing the overall situational awareness for the whole OR team as well as the ability to share mental models. Encouraging the OR team to communicate throughout the case helps each member to understand the patient’s status and to anticipate next steps and what might be needed to ensure patient safety. Mitigation of delays. Avoidable delays were observed in 92% of videos. The most common reason for delays was missing tools/equipment and equipment that was not working properly (eg, laparoscopic light source malfunctioning, monitor
ARTICLE IN PRESS Stefanidis et al 7
Surgery Volume j, Number j
Table III. Type and frequency of identified technical and nontechnical skill deficiencies Technical deficiency
%
Nontechnical deficiency
%
Port placement Visualization of the operating field Retraction and exposure Dissection technique and safe use of energy Bimanual dexterity Adherence to safe practices Decision-making
42 35 46 42 58 54 35
Response to interruptions and distractions Ergonomic positioning Mitigation of delays Situational awareness and shared mental models Pre- and postbrief
58 100 92 67 92
not turning on). Most of these delays could have been avoided by checking all the equipment prior to starting the operation. As noted previously, the lack of proper equipment contributed to a delay in controlling an intraoperative bleed. Insufficient pre- and postbrief. Our institution has adopted a modified version of the WHO checklist,28 which includes a preoperative timeout and a postoperative debrief. Our video review revealed that in 92% of cases, adherence to this process was inadequate. Although the preoperative timeout was observed for almost every case, it was often not performed in its entirety, or items were checked off without adequately performing the required task. An example previously mentioned was a lack of verification that all needed and anticipated equipment was available and working properly. Further, a postoperative debrief in which the surgeon participated was rarely observed. Handoffs of patients have been identified as an area that is prone to communication errors, and handoffs have been implicated in a number of patient harm events.29 Performing a surgeon-led postoperative debrief helps identify and confirm any patient concerns for recovery. It should also be noted that the cost of conducting this study was calculated at $2,640 per participating surgeon. This cost included video recording and editing charges, study coordination, coach salary, and supplies used and was dependent on the number of participating surgeons. DISCUSSION In this study, we aimed to identify technical and nontechnical skill deficiencies of practicing surgeons and to develop a coaching mechanism that would address the lack of performance feedback available to surgeons today. To accomplish this, we embarked on a video-based assessment of technical and nontechnical skill of attending surgeons and created a coaching program that was based on an interactive video review and simulation-based practice.
Our approach allowed us to demonstrate the importance of ongoing performance assessment, as several deficiencies amenable to training were identified that otherwise likely would have gone undetected. The observed skill deficiencies for a variety of surgeons fell into several common categories, which demonstrate the need for targeted curriculum development. These areas are highlighted in this paper and can be useful for educators and coaches in surgery. Importantly, several of our participating surgeons were very appreciative for being afforded this coaching opportunity, as they themselves felt this mechanism of performance feedback was missing from their practice. The importance of coaching for performance improvement and professional development is widely recognized across many disciplines, such as the sports industry,12 education,30 managerial training,31 and in health care.32 While coaching techniques have some overlap with traditional teaching methods, there are differences in the roles that a teacher and coach fulfill.33 Coaching focuses on an individual’s professional development through one-on-one reflective dialogue with their coach to monitor and evaluate progress toward specific goals and to modify action plans on the basis of feedback.34 The Kansas Coaching Project has identified factors that are often cited as critical for coaching success. Some of these include having a learning-friendly culture where the coachee is shown respect and provided encouragement; coachees voluntarily and willingly agree to participate; and an iterative approach to coaching is adopted, recognizing that continuous learning is imperative to success.34 Based on these principles, we designed a 5-step coaching model to be used in this study (Fig 3): (1) Assess skills through video. Our model starts with video-based performance assessment of technical and nontechnical skills through objective-based metrics. Expert surgeons assess their peers’ technical skills, while a human factors expert assesses
ARTICLE IN PRESS 8 Stefanidis et al
Fig 3. Carolinas coaching model. (Color version of this figure is available online.)
the surgeons’ nontechnical skills. The aspects of performance in need of improvement are defined by this expert-driven process. (2) Identify areas for growth. The second stage of the process is to synthesize all the observed data, thus identifying common group and individual performance deficiencies and guiding goal setting for coaching (ie, which aspects of performance an individual needs to improve). Consistent with the Kansas Coaching Project34 observations, we found it is imperative that the coachee exhibits the willingness/desire to improve his or her performance. (3) Group review and individual feedback. We propose that the next step in the process should be performance feedback by the coach. This step represents the heart of our coaching model, as it provides the necessary information and rationale to the coachee of what needs to be improved and how it can be accomplished.
Our experience in this study indicated that coachees benefit most from feedback provided in a group session and in one-on-one sessions. We identified several advantages of adding a blinded group review session to one-on-one coaching. Group sessions enable peer review; therefore, surgeons can watch other surgeons operate and learn from their mistakes and from their successes. Indeed, several of our participating surgeons indicated that they found value in reviewing each other’s operations, as they saw some techniques they liked and identified issues they clearly perceived as errors in others.
Surgery j 2016
Another advantage of the blinded group debrief session is that they may be less threatening to surgeons because the critiqued videos are deidentified, so participants can freely express their opinion without feeling vulnerable or defensive. Research indicates that humans are a lot better at critiquing others than at performing self-critique (which one-on-one coaching calls for).7 Consequently, group sessions prime participants for the one-on-one coaching sessions; deficiencies identified collectively during the group session cannot be refuted or overseen by coachees during individual sessions, making it more likely to leave a lasting impression on them and bring about the desired behavior change. On the other hand, interactive one-on-one sessions are necessary and invaluable, as they allow specific feedback tailored to individual needs. Together, coach and coachee can develop goals and strategies to achieve them. (4) Deliberate practice with coaching. The next step in our model consists of deliberate practice on the issues identified during the previous steps. Simulation can greatly enhance the effectiveness of this step, as it enables repetitive practice of deficient procedural skills at the surgeon’s preferred time, which otherwise may be impossible based on clinical practice alone for rarely used skills or encountered conditions. It further allows for the provision of formative and summative performance feedback that is essential for the acquisition of skills.35 Simulation also allows for nontechnical skill training by using clinical and/or team training scenarios that can effectively deliver the desired intervention to participants. The accompanying debrief is a great opportunity to highlight good and poor behaviors and to impart to participants important take-home messages. (5) Monitor patient outcomes. The final step in our model consists of patient outcome monitoring, which we are currently assessing for the first cohort of trainees. This step is crucial, as it provides a good measure of effectiveness and ensures that the coaching intervention benefits patients. The loop closes by repeat skill assessment and restarting of the described process. We strongly believe that this feedback loop needs to be ongoing as behavioral change happens over time, and limited interventions are less likely to be successful.
Recently, and after we had embarked on our coaching curriculum development, Greenberg et al9 reported their coaching approach: The Wisconsin Surgical Coaching Framework, which targets performance improvement in 3 domains (technical skill, cognitive skill, and nontechnical skill) and proposes
ARTICLE IN PRESS Stefanidis et al 9
Surgery Volume j, Number j
3 distinct but interrelated activities of coaching (setting goals, encouraging and motivating, and developing and guiding). The 2 models use different definitions of assessed skills (ie, cognitive skills as defined by Greenberg et al9 are included in our technical and nontechnical skill definitions). Further, our model uses both group and individual coaching sessions, employs simulation as a means for further practice and feedback, and has a focus on patient outcomes. On the other hand, both studies use intraoperative video recordings to assess the participating surgeon’s skill and focus on targeted feedback provision to the coachee. Another group from the United Kingdom has developed and assessed a coaching curriculum using medical students who were laparoscopic novices. Singh et al11 based their coaching curriculum on the Goal, Reality, Options, Way Forward (GROW) model that guides coaching conversations according to a 4-step process: initially the goals of coaching are set with the coachee, followed by performance review, formulation of effective solutions, and development of an action plan targeting goal achievement and taking into consideration potential anticipated obstacles. Using this coaching approach, Singh et al11 provided video-based coaching to an intervention group of novices training on laparoscopic skills and demonstrated enhanced quality of laparoscopic operative performance during both virtual reality and porcine laparoscopic cholecystectomies compared with a control group receiving the same amount of training but in the absence of coaching. The authors concluded that video-based coaching maximizes performance enhancement from every clinical exposure. A very important and relevant question is who the operative coach should be and what personal qualities he or she should possess. Greenberg et al9 suggest that excellent interpersonal skills including cooperative communication and active listening, ability to adapt coaching approaches to the individual’s needs, and reputability within their field are crucial traits of a good coach. Our experience indicates that the ideal coach is also dependent on the type of skill being addressed; for technical skills, it should be a respected peer with expertise in the domain that is being coached (ie, it will be difficult for a breast surgeon to coach a minimally invasive surgeon and vice versa). On the other hand, for nontechnical skills, the ideal coach might not be a peer surgeon but rather an individual with a human factors background. While peer surgeons might be able to provide feedback on nontechnical skills, they typically lack human factors expertise
and are likely to be less effective and less respected as coaches for these types of skills. Our study is not without limitations. Foremost, a fraction (28%) of our enrolled surgeons, possibly the most motivated, participated in any of the coaching sessions, thus limiting our ability to draw firm conclusions about the effectiveness of our coaching model. The limited participation rate reflects the difficulty in engaging busy practicing surgeons. It may also reflect hesitation to receive feedback. In addition, though we developed a robust coaching mechanism for practicing surgeons, we did not demonstrate conclusively its effectiveness; however, we have identified some positive trends. To address this issue, we are currently expanding our coaching paradigm and collecting evidence of effectiveness, including patient outcomes, and we hope to report our findings in upcoming publications. Another potential limitation for widespread implementation is that technical skills are easier to review blindly during group sessions (easy to obtain and reviewers only see the operative field and cannot identify who the surgeon is), while nontechnical skills are significantly more challenging (very time consuming and costly to blind them), limiting their feasibility. Further research may demonstrate the true benefits and best combination of our proposed dual approach to feedback provision. In conclusion, our study identified several technical and nontechnical skill sets of practicing surgeons in need of improvement and provided support for the implementation of coaching programs for surgeons on an ongoing basis. REFERENCES 1. de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA. The incidence and nature of in-hospital adverse events: A systematic review. Qual Saf Health Care 2008;17:216-23. 2. The National Patient Safety Foundation. Agenda for research and development in patient safety. [Internet]. Boston (MA): National Patient Safety Foundation; 2016. Available from:, https://c.ymcdn.com/sites/www.npsf. org/resource/collection/4B2E552F-48FA-4DCF-8BD8-574EE 15EFD99/Agenda_for_RD_in_Patient_Safety.pdf. 3. Bikrmeyer JD, Finks JF, O’Reilley A, Oerline M, Carlin AM, Nun AR, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med 2013;369:1434-42. 4. Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N. The impact of nontechnical skills on technical performance in surgery: A systematic review. J Am Coll Surg 2012;214:214-30. 5. Yule S, Patterson-Brown S. Surgeons’ non-technical skills. Surg Clin N Am 2012;92:37-50. 6. Hauer KE, Ciccone A, Henzel TR, Katsufrakis P, Miller SH, Norcross WA, et al. Remediation of the deficiencies of physicians across the continuum from medical school to practice: A thematic review of the literature. Acad Med 2009; 84:1822-32.
ARTICLE IN PRESS 10 Stefanidis et al
7. Davis DA, Mazmanian PE, Fordis M, Harrison RV, Thorpe KE, Perrier L. Accuracy of physician selfassessment compared with observed measures of competence: A systematic review. JAMA 2006;296:1094-102. 8. Wigton RS, Patil KD, Hoellerich VL. The effect of feedback in learning clinical diagnosis. J Med Educ 1986;61:816-22. 9. Stefanidis D. Optimal acquisition and assessment of proficiency on simulators in surgery. Surg Clin North Am 2010;90:475-89. 10. Greenberg CC, Ghousseini HN, Pavuluri Quamma SR, Beasley HL, Wiegmann DA. Surgical coaching for individual performance improvement. Ann Surg 2015;261:32-4. 11. Singh P, Aggarwal R, Tahir M, Pucher PH, Darzi A. A randomized controlled study to evaluate the role of video-based coaching in training laparoscopic skills. Ann Surg 2015;261:862-9. 12. Becker AJ. It’s not what they do, it’s how they do it: Athlete experiences of great coaching. Int J Sports Sci Coach 2009;4:93-119. 13. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondre K, Stanbridge D, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 2005;190:107-13. 14. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ. Global evaluative assessment of robotic skills: Validation of a clinical assessment tool to measure robotic surgical skills. J Urology 2012;187:247-52. 15. Cole SJ, Mackenzie H, Ha J, Hanna GB, Miskovic D. Randomized controlled trial on the effect of coaching in simulated laparoscopic training. Surg Endosc 2014;28:979-86. 16. Miskovic D, Ni M, Wyles SM, Kennedy RH, Francis NK, Parvaiz A, et al. Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg 2013;257:476-82. 17. Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS system: Reliability and validity of a tool for measuring teamwork behavior in the operating theatre. Qual Saf Health Care 2009;18:104-8. 18. Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N. Observational teamwork assessment for surgery: Content validation and tool refinement. J Am Coll Surg 2011;212:234-43. 19. Yule S, Rowley D, Flin R, Maran N, Youngson G, Duncan J, et al. Experience matters: Comparing novice and expert ratings of non-technical skills using the NOTSS system. ANZ J Surg 2009;79:154-60. 20. Yule S, Flin R, Maran N, Rowley D, Youngson G, PatersonBrown S. Surgeons’ non-technical skills in the operating room: Reliability testing of the NOTSS behavior rating system. World J Surg 2008;32:548-56. 21. Sevdalis N, Davis R, Koutantji M, Undre S, Darzi A, Vincent CA. Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg 2008;196:184-90.
Surgery j 2016
22. Office for Technology Commercialization. Synthetic inferior vena cava surgery training model. [Internet]. Minneapolis (MN): Regents of the University of Minnesota; 2016. Available from: http://license.umn.edu/technolo gies/20110220-4_synthetic-inferior-vena-cava-surgery-trainingmodel. 23. Ritter EM, Scott DJ. Design of a proficiency-based skills training curriculum for the fundamentals of laparoscopic surgery. Surg Innov 2007;14:107-12. 24. van Det MJ, Meijerink WJHJ, Hoff C, Totte ER, Pierie JPEN. Optimal ergonomics for laparoscopic surgery in minimally invasive surgery suites: A review and guidelines. Surg Endosc 2009;23:1279-85. 25. Miller K, Benden M, Pickens A, Shipp E, Zheng Q. Ergonomics principles associated with laparoscopic surgeon injury/illness. Hum Factors 2012;54:1087-92. 26. van Veelen MA, Kazemier G, Koopman J, Goossens RH, Meijer DW. Assessment of the ergonomically optimal operating surface height for laparoscopic surgery. J Laparoendosc Adv Surg Tech 2002;12:47-52. 27. Engelmann C, Schneider M, Kirschbaum C, Grote G, Dingemann J, Schoof S, et al. Effects of interoperative breaks in mental and somatic operator fatigue: A randomized clinical trial. Surg Endosc 2011;24:1245-50. 28. World Alliance for Patient Safety. WHO surgical safety checklist and implementation manual. [Internet]. Geneva (Switzerland): World Health Organization; 2016. Available from:, http://www.who.int/patientsafety/safesurgery/ss_ checklist/en/. 29. Riesenberg LA, Leitzsch J, Massucci JL, Jaeger J, Rosenfeld JC, Patow C, et al. Residents’ and attending physicians’ handoffs: A systematic review of the literature. Acad Med 2009;84:1775-87. 30. Costa AL, Garmston RJ. Cognitive coaching: A foundation for renaissance schools. Norwood (MA): Christopher-Gordon Publishers; 1994. 31. Longenecker CO, Neubert MJ. The practices of effective managerial coaches. Bus Horizons 2005;48:493-500. 32. Velasquez M, Shellenberger S, von Sternberg K. Training health-care practitioners in screening, brief intervention, and referral to treatment using standardized approaches and expert coaching. Addict Sci Clin Pract 2012;7(Suppl 1):A95. 33. Rupert T, Buschner C. Teaching and coaching: A comparison of instructional behaviors. J Teach Phys Educ 1989;9: 49-57. 34. Knight J. Coaching. J Staff Dev 2009;30:18–22. 35. Scerbo MW, Anderson BL. Medical simulation. In: Carayon P, editor. Handbook of human factors and ergonomics in health care and patient safety. Boca Raton (FL): Taylor and Francis Group; 2012.