- Email: [email protected]
Assessing the Nontechnical Skills of Surgical Trainees: Views of the Theater Team
ORIGINAL REPORTS
Assessing the Nontechnical Skills of Surgical Trainees: Views of the Theater Team$ Wissam Al-Jundi,* Jonathan Wild,† Judith Ritchie,‡ Sarah Daniels,§ Eleanor Robertson,║,¶ and Jonathan Beard* Northern General Hospital, Sheffield Vascular Institute, Sheffield, United Kingdom; †Department of General Surgery, Doncaster Royal Infirmary, Doncaster, United Kingdom; ‡University of Leeds, Leeds, United Kingdom; § Surgical Oncology, Northern General Hospital, Sheffield, United Kingdom; ║Department of Plastic Surgery, Derriford Hospital, Plymouth, United Kingdom; and ¶Nuffield Department of Surgical Sciences, John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom *
OBJECTIVE: This study aims to explore the views of
CONCLUSIONS: Our survey demonstrates acceptability
members of theater teams regarding the proposed introduction of a workplace-based assessment of nontechnical skills of surgeons (NOTSS) into the Intercollegiate Surgical Curriculum Programme in the United Kingdom. In addition, the previous training and familiarity of the members of the surgical theater team with the concept and assessment of NOTSS would be evaluated.
among the theater team for the introduction of the NOTSS tool into the surgical curriculum. However, lack of familiarity highlights the importance of faculty training for assessors before such an introduction. ( J Surg Ed 73:222-229. C J 2015 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
DESIGN: A regional survey of members of theater teams
(consultant surgeons, anesthetists, scrub nurses, and trainees) was performed at 1 teaching and 2 district general hospitals in South Yorkshire.
KEY WORDS: task performance and analysis, patient safety, standard of care COMPETENCIES: Patient Care, Professionalism, Interper-
sonal and Communication Skills, Systems-Based Practice
RESULTS: There were 160 respondents corresponding to a
response rate of 81%. The majority (77%) were not aware of the NOTSS assessment tool with only 9% of respondents reporting to have previously used the NOTSS tool and just 3% having received training in NOTSS assessment. Overall, 81% stated that assessing NOTSS was as important as assessing technical skills. Trainees attributed less importance to nontechnical skills than the other groups (p r 0.016). Although opinion appears divided as to whether the presence of a consultant surgeon in theater could potentially make it difficult to assess a trainee’s leadership skills and decision-making capabilities, overall 60% agree that the routine use of NOTSS assessment would enhance safety in the operating theater and 80% agree that the NOTSS tool should be introduced to assess the nontechnical skills of trainees in theater. However, a significantly lower proportion of trainees (45%) agreed on the latter compared with the other groups (p ¼ 0.001). ☆ On behalf of the South Yorkshire Surgical Research Group (SYSuRG) Collaborative. Correspondence: Inquiries to Wissam Al-Jundi, Northern General Hospital, Sheffield Vascular Institute, Harries Road, Sheffield S5 7AU, United Kingdom; e-mail: [email protected]
222
INTRODUCTION In the UK, the last decade has seen major reform in postgraduate medical education, notably a national move toward competency-based curricula. For the surgical specialties this has led to the launch of the Intercollegiate Surgical Curriculum Programme (ISCP) (www.iscp.ac.uk), which has transformed postgraduate training. ISCP enables the surgical trainee to record evidence of their training progression through the accumulation of specialty specific workplace-based assessments (WBAs) in technical surgical skills (Direct Observation of Procedural Skills and Procedure-Based Assessment [PBA]), Clinical diagnosis and management (mini-Clinical Evaluation Exercise) and an overall assessment of behavior and attitudes by colleagues (mini-Peer Assessment Tool). Recently, an assessment tool to assess trainee’s nontechnical skills has been introduced to ISCP as a pilot. Currently, nontechnical skills are informally acquired by surgical trainees through apprenticeship and observation rather than being explicitly addressed in training.1,2 It is felt that this passive learning style needs transition to one of active knowledge acquisition with
Journal of Surgical Education & 2015 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2015.10.008
behavioral change. One of the key assessment tools of nontechnical skills available in literature is the nontechnical skills for surgeons (NOTSS), adapted from aviation Crew Resource Management.3,4 It is a rating system based on skill taxonomy, with examples of good and poor behavioral markers. Ratings and feedback are given on 4 categories of nontechnical skills; situation awareness, decision making, communication and teamwork, and leadership. There are, however, various questions that need to be addressed before the integration of NOTSS assessment in the surgical curriculum, namely, whom should take on the role of assessor (should they exclusively be surgeons or should nonsurgeons be recruited); should raters obtain formal training in NOTSS assessment and the effect of the presence of a more senior surgeon in theater, during the assessment on the rating of the trainee’s nontechnical skills. Engagement with NOTSS and how to ensure “buy-in” from members of the theater team, particularly trainees, is also an important consideration. In this study, we present a survey among the surgical theater team that explores current views regarding the need to introduce an assessment tool of nontechnical skills of surgical trainees in the theater environment. In addition, the familiarity of members of the surgical theater team with the concept of NOTSS as well as their previous training in this area is explored.
The study group redesigned the layout of the NOTSS assessment tool developed in Scotland to mirror the ISCP PBA form (Fig. 1). This was enclosed with each questionnaire and the participants were asked to examine this NOTSS PBA form before filling the survey. Participants were asked initially about their familiarity with NOTSS assessment before rating their views on a 5point Likert scale5 regarding the need to introduce a NOTSS assessment tool into surgical curriculum, the effect of the presence of a supervising consultant in theater on the assessment of NOTSS and the need to train assessors to use the proposed NOTSS tool.
Statistical Analysis The PASW statistical package (version 18, SPSS inc, Chicago, IL) was used for statistical analysis to examine for any differences in responses among the 4 professional groups. The Kruskal Wallis test of variance was used to analyze responses among the 4 groups with the MannWhitney U test used to compare responses among 2 groups where appropriate. The open-text responses were collected separately and analyzed manually.
RESULTS METHODS Data Collection This survey was conducted in South Yorkshire, UK between November 2014 and February 2015. Ethical approval was granted by University of Sheffield Medical School (No. SMBRER287). Theater teams at 1 teaching and 2 district general hospitals were invited to participate in the survey and to provide their views on NOTSS assessment as well as their experience of nontechnical skills. The participants were divided into 4 professional groups: consultant surgeons, surgical trainees, consultants anesthetists, and theater staff. Theater staff were invited to participate if they regularly scrubbed for operations and so included theater sisters, staff nurses, and operating department practitioners. The participants were contacted directly over a period of 3 working days at the beginning of theater lists as well as during coffee breaks and asked by a core surgical trainee (resident) to fill the hard copies of the questionnaires in theaters to hand them back at the end of the lists. Each participant was approached to participate only once. Responses were inputted onto an electronic database and the data extracted. Survey Questionnaire The questionnaires consisted of 17 multiple-choice questions and open-text space when clarification is necessary.
In all, 198 staff members were approached and 160 responded, corresponding to a response rate of 81% (25 consultant surgeons, 51 trainees (grades CT1-ST8), 26 anesthetists, and 58 theater staff (Fig. 2). A total of 38 staff members did not return the questionnaires (5 consultant surgeons, 12 trainees, 3 anesthetists, and 18 theater staff). Overall, 123 respondents (77%) were not aware of the NOTSS assessment tool with only 14 (9%) reporting to have previously used the NOTSS tool and just 4 (3%) having received training in NOTSS assessment (Table 1). However, the majority (81%) stated that nontechnical skills are as important as technical skills in improving the surgical outcomes, (6% thought more important and 13% thought less important) (Fig. 3). When compared with the other theater staff, however, trainees appear to hold less importance to nontechnical skills (p r 0.016). Overall, 114 respondents (71%) agreed that anesthetists should take part in a surgical trainee’s NOTSS assessment (Fig. 4) with 128 respondents (80%) also in favor of scrub nurses assessing trainees’ NOTSS (Fig. 5). The anesthetists and scrub nurses were particularly in favor of involvement of nonsurgeons in surgical trainee’s NOTSS assessments (anesthetists to assess NOTSS, mean response 4.04 [nonsurgeons] vs. 3.66 [surgeons], p ¼ 0.008; scrub nurses to assess NOTSS, 4.29 [nonsurgeons] vs. 3.68, [surgeons] p o 0.001). All the anesthetists who responded to this
Journal of Surgical Education Volume 73/Number 2 March/April 2016
223
FIGURE 1. The NOTSS assessment tool developed in Scotland redesigned to mirror the ISCP PBA form.
224
Journal of Surgical Education Volume 73/Number 2 March/April 2016
Consultant Surgeons
16%
36%
Trainees 32%
Consultants Anaesthests
16%
Scrub Nurses
Surgeons
n
Trainees
n
Scrub N
n
General
12
Vascular
11
CST
27
Sister
25
HST
24
Plasc
1
Nurse
15
Total
51
ODP
16
Urology
1
N Prac
2
Total
25
Total
57
Anaesthests
n
Total
26
FIGURE 2. Demographics of participants. CSTs are equivalent to year 1 and 2 residents whist HSTs are equivalent to more senior residents and clinical fellows. Abbreviations: CSTs, certified surgical trainees; HSTs, higher surgical trainees.
survey disagreed that the assessment of a trainee’s NOTSS should only be conducted by the supervising consultant surgeon, compared with 69% of nonanesthetists. In all, 65% of respondents agreed that NOTSS assessment provides a good tool for reflective practice. A total of 58% agreed that performing a NOTSS assessment would not interfere with the smooth running of theater lists, whereas 18% disagreed and 24% were not sure. Only 20% of trainees and surgeons stated that NOTSS should only be assessed for higher surgical trainees whereas the majority supported the utilization of the tool for both core surgical trainees (residents) and higher surgical trainees (fellows). Although opinion appears divided as to whether the presence of a consultant surgeon in theater could potentially make it difficult to assess a trainee’s leadership skills and decision-making capabilities (Fig. 6), overall 60% agree that the routine use of NOTSS assessment would enhance safety in the operating theater and 80% agree that the NOTSS tool should be introduced to assess the nontechnical skills of trainees in theater (Fig. 7). However, a significantly lower proportion of trainees (45%) agreed on the latter compared with the other groups (p ¼ 0.001).
DISCUSSION Analyses of adverse events in healthcare have revealed that there is more often failure in nontechnical aspects of performance rather than a lack of technical expertise.6,7 It has been shown that poor nontechnical skills in the operating theater result in a reduction in technical skill8 adverse events and poor clinical outcomes.9-12 Communication has been found to be a causal factor in 43% of errors made in surgery.12 One of the key outcomes for all surgical trainees is “professional skills and behavior;” however, reference to nontechnical skills specific to the theater environment is lacking. At present, ISCP recommends utilization of mini-Peer Assessment Tool and the assigned educational supervisor end of placement summative assessments to assess professional skills and behavior. These assessments are based on retrospective views of a select group of multidisciplinary staff assessing behaviors in all aspects of the surgical trainees’ clinical activity, and most importantly lack the important element of instant feedback. The NOTSS assessment tool has been designed to gather prospective multidisciplinary feedback on nontechnical
TABLE 1. Awareness, Experience, and Training of NOTSS Among Participants Surgeons
Aware of the NOTSS assessment tool Experienced in using the NOTSS assessment tool in the past Had previous training in using the NOTSS assessment tool
Consultants
Trainees
Anesthetists
Theater Nurses
Total
12 (48%) 4 (16%)
14 (28%) 5 (10%)
3 (12%) 2 (8%)
8 (14%) 3 (5%)
37 (23%) 14 (9%)
1 (4%)
3 (6%)
0 (0%)
0 (0%)
4 (2.5%)
Journal of Surgical Education Volume 73/Number 2 March/April 2016
225
FIGURE 3. Rating of the importance NOTSS among participants. Overall, 27% of trainees stated that NOTSS is less important than technical skills (p r 0.016).
skills that are fundamental to the theater work environment, specifically situational awareness, decision-making, communication, and teamwork or leadership skills. It is currently being piloted among surgical trainees through the ISCP platform. Our survey aimed to assess the perceptions of the theater team regarding assessment of nontechnical skills using the proposed multidisciplinary NOTSS assessment tool with a view to determining how feasible this would be to perform in practice. Most of the surveyed participants agreed that nontechnical skills are important and recognized the value of introducing such an assessment tool into the ISCP. Our survey demonstrated a lack of familiarity with the existing NOTSS tool among the vast majority of theater staff surveyed in this study.
The participants agreed that anesthetists and scrub nurses should be included in this assessment process. The contribution of anesthetists and scrub nurses, whose assessments have previously been shown to be equally reliable to consultant surgeons,13 makes it an ideal tool for theaterspecific Multi-Source Feedback and a valuable instrument to provide formative assessment. Staff also felt that formal assessor training would be required before introducing this assessment tool. This is important to ensure the assessors understand all aspects of the assessment process, as previous piloting has found that cognitive skills such as situational awareness and decision making are more difficult to assess than interpersonal skills, such as communication and teamwork,13 NOTSS assessor
FIGURE 4. Views of the theater team regarding the need for anesthetists to take part in assessing NOTSS. 226
Journal of Surgical Education Volume 73/Number 2 March/April 2016
FIGURE 5. Views of the theater team regarding the need for Scrub nurses to take part in assessing NOTSS.
training in this field has been also highlighted in recent guidelines developed by expert consensus.14 The survey also revealed concerns among one-fifth of the theater team that assessment could interfere with the smooth running of the theater lists. This has not been perceived to be an issue in a previous pilot study.13 Another issue that has been highlighted in this survey is the concern that the presence of a supervising consultant in theater may limit trainees’ ability to demonstrate their leadership and decision-making skills. This has effected on the assessment process in previous studies,13 although the surgical consultant would remain the ultimate team leader and decision maker in theater, trainees need to be given an opportunity
to develop and apply such skills under supervision, particularly when an assessment is planned, requiring the consultant to “take a step back.” It was also apparent in this survey that trainees were relatively less supportive of introducing NOTSS assessment into ISCP compared with the other members of the theater team. Free-text comments showed that there is a current frustration among many surgical trainees that WBAs are being misused as assessments of learning (i.e., summative) rather that assessments for learning (i.e., formative) with WBAs viewed as “tick-box exercises” that can negatively affect the training opportunities (Table 2). In addition, consultant feedback also suggests trainer
FIGURE 6. Views of the theater team regarding the effect consultant surgeons’ presence on the ability of trainees to demonstrate NOTSS. Journal of Surgical Education Volume 73/Number 2 March/April 2016
227
FIGURE 7. Views of the theater team regarding the need to introduce NOTSS tool.
dissatisfaction with the process, with some seeing it as a “tick-box exercise” also. This has previously been reported in studies looking at perceptions of work-based assessments.15 Local education initiatives, workshops, and online training modules help to promote the NOTSS ethos and the potential that NOTSS has to enhance service delivery and patient outcomes, something that would be appreciated by scrub staff as much as the medical trainers as well as patient outcomes.
This study is not without limitations. The authors work within general and vascular surgery departments, which resulted in a larger inclusion of these specialties at the expense of less representation of other surgical subspecialties within the ISCP. Although the number of participants was relatively small, the views of all the 4 professional groups within the theater team were represented. There is also a selection bias as the group of professionals who participated could be more likely to engage with the teaching and training process than those who did not participate.
TABLE 2. Free-Text Comments Professional Group Trainees
Surgeons
Anesthetists
228
Free-Text Comment “Too much paperwork. Is done anyway.” “Not required at all. This is covered in the other PBAs. We are being swamped with this.” “NOTSS is covered in other WBAs and does not need its own assessment tool.” “Should be part of each PBA, rather than introducing another tool.” “PBAs include already the assessment of leadership skills, decision making, communication, team working, and situational awareness hence a separate PBA might add further delays in theater. In my view, we do not need a separate PBA for it.” “This is a duplicate MSF.” “Great deal of subjectivity. Views likely to differ greatly from individuals in different roles, some likely to be unreliable. Significant bias including but not limited to Hawthorne effect.” “Much of these can be freely expressed in the 360 feedback—why add yet another assessment that people struggle to complete?” “I don't think that this assessment would provide any accurate or useful data.” “I think the tool is good but should be amalgamated with the ISCP assessments.” “It would be another tick-box exercise. Surgical trainees would be better spending their time in clinic or theater rather than filling in forms.” “Just another tick-box exercise.” “NOTSS improves communication and team dynamic which ultimately improves quality of care and patients' outcome” “Dependent on the individual surgeon or experience as to whether their presence would make it difficult for me to assess a trainee's nontechnical skills.” “Introducing NOTSS assessment would be better than nothing.” Journal of Surgical Education Volume 73/Number 2 March/April 2016
CONCLUSION Our survey demonstrates acceptability among the theater team for the introduction of the NOTSS tool into the surgical curriculum. Adequate provision of training on assessing NOTSS is required before an introduction and the findings of this survey can inform appraisal of the utility of NOTSS assessment tool that is being piloted via the ISCP.
ACKNOWLEDGMENTS The authors would like to thank members of South Yorkshire Surgical Research Group Collaborative who facilitated data collection in South Yorkshire, namely, Charlotte Gunner, Matthew Lee, Khalil Madbak, Mohamed El-Sharif, Emma Nofal, Vicky Proctor, and Stuart Stokes.
6. Bogner MSE. Human Error in Medicine. Hillsdale,
NJ: Lawrence Erlbaum Associates, Inc; 1994. 7. Fletcher G, McGeorge P, Flin RH, Glavin RJ, Maran
NJ. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth. 2002;88 (3):418-429. 8. McCulloch P, Mishra A, Handa A, Dale T, Hirst G,
Catchpole K. The effects of aviation-style non-technical skills training on technical performance and outcome in the operating theatre. Qual Saf Health Care. 2009;18(2):109-115. 9. Lingard L, Espin S, Whyte S, et al. Communication
failures in the operating room: an observational classification of recurrent types and effects. Qual Saf Health Care. 2004;13(5):330-334. 10. Mazzocco K, Petitti DB, Fong KT, et al. Surgical team
behaviors and patient outcomes. Am J Surg. 2009;197 (5):678-685.
REFERENCES
11. Barach P, Johnson JK, Ahmad A, et al. A prospective
technical skills for surgeons in the operating room: A review of the literature. Surgery. 2006;139(2):140-149.
observational study of human factors, adverse events, and patient outcomes in surgery for pediatric cardiac disease. J Thorac Cardiovasc Surg. 2008;136(6):1422-1428 [PubMed PMID: 19114184].
2. Aggarwal R, Undre S, Moorthy K, Vincent C, Darzi
12. Gawande AA, Zinner MJ, Studdert DM, Brennan TA.
A. The simulated operating theatre: comprehensive training for surgical teams. Qual Saf Health Care. 2004;13(suppl 1):i27-i32.
Analysis of errors reported by surgeons at three teaching hospitals. Surgery. 2003;133(6):614-621 [PubMed PMID: 12796727].
3. Yule S, Flin R, Maran N, Rowley D, Youngson G,
13. Beard JD, Marriott J, Purdie H, Crossley J. Assessing
Paterson-Brown S. Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg. 2008;32(4): 548-556.
the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology. Health Technol Assess. 2011;15(1):1-162 [i-xxi].
1. Yule S, Flin R, Paterson-Brown S, Maran N. Non-
4. Flin R, Yule S, Paterson-Brown S, Rowley D, Maran
N. The Non-Technical Skills for Surgeons (NOTSS) System Handbook v1.2. Scotland: University of Aberdeen; 2006. 5. Likert R. A technique for the measurement of atti-
tudes. Arch Psychol. 1932;22(140):1-55.
14. Hull L, Arora S, Symons NR, et al. Training faculty in
nontechnical skill assessment: national guidelines on program requirements. Ann Surg. 2013;258(2):370-375 [PubMed PMID: 23222032. Epub 2012/12/12. Eng]. 15. Ali J. Getting lost in translation? Workplace based
assessments in surgical training. Surgeon. 2013; 11(5):286-289.
Journal of Surgical Education Volume 73/Number 2 March/April 2016
229
Assessing the Nontechnical Skills of Surgical Trainees: Views of the Theater Team$ Wissam Al-Jundi,* Jonathan Wild,† Judith Ritchie,‡ Sarah Daniels,§ Eleanor Robertson,║,¶ and Jonathan Beard* Northern General Hospital, Sheffield Vascular Institute, Sheffield, United Kingdom; †Department of General Surgery, Doncaster Royal Infirmary, Doncaster, United Kingdom; ‡University of Leeds, Leeds, United Kingdom; § Surgical Oncology, Northern General Hospital, Sheffield, United Kingdom; ║Department of Plastic Surgery, Derriford Hospital, Plymouth, United Kingdom; and ¶Nuffield Department of Surgical Sciences, John Radcliffe Hospital, University of Oxford, Oxford, United Kingdom *
OBJECTIVE: This study aims to explore the views of
CONCLUSIONS: Our survey demonstrates acceptability
members of theater teams regarding the proposed introduction of a workplace-based assessment of nontechnical skills of surgeons (NOTSS) into the Intercollegiate Surgical Curriculum Programme in the United Kingdom. In addition, the previous training and familiarity of the members of the surgical theater team with the concept and assessment of NOTSS would be evaluated.
among the theater team for the introduction of the NOTSS tool into the surgical curriculum. However, lack of familiarity highlights the importance of faculty training for assessors before such an introduction. ( J Surg Ed 73:222-229. C J 2015 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)
DESIGN: A regional survey of members of theater teams
(consultant surgeons, anesthetists, scrub nurses, and trainees) was performed at 1 teaching and 2 district general hospitals in South Yorkshire.
KEY WORDS: task performance and analysis, patient safety, standard of care COMPETENCIES: Patient Care, Professionalism, Interper-
sonal and Communication Skills, Systems-Based Practice
RESULTS: There were 160 respondents corresponding to a
response rate of 81%. The majority (77%) were not aware of the NOTSS assessment tool with only 9% of respondents reporting to have previously used the NOTSS tool and just 3% having received training in NOTSS assessment. Overall, 81% stated that assessing NOTSS was as important as assessing technical skills. Trainees attributed less importance to nontechnical skills than the other groups (p r 0.016). Although opinion appears divided as to whether the presence of a consultant surgeon in theater could potentially make it difficult to assess a trainee’s leadership skills and decision-making capabilities, overall 60% agree that the routine use of NOTSS assessment would enhance safety in the operating theater and 80% agree that the NOTSS tool should be introduced to assess the nontechnical skills of trainees in theater. However, a significantly lower proportion of trainees (45%) agreed on the latter compared with the other groups (p ¼ 0.001). ☆ On behalf of the South Yorkshire Surgical Research Group (SYSuRG) Collaborative. Correspondence: Inquiries to Wissam Al-Jundi, Northern General Hospital, Sheffield Vascular Institute, Harries Road, Sheffield S5 7AU, United Kingdom; e-mail: [email protected]
222
INTRODUCTION In the UK, the last decade has seen major reform in postgraduate medical education, notably a national move toward competency-based curricula. For the surgical specialties this has led to the launch of the Intercollegiate Surgical Curriculum Programme (ISCP) (www.iscp.ac.uk), which has transformed postgraduate training. ISCP enables the surgical trainee to record evidence of their training progression through the accumulation of specialty specific workplace-based assessments (WBAs) in technical surgical skills (Direct Observation of Procedural Skills and Procedure-Based Assessment [PBA]), Clinical diagnosis and management (mini-Clinical Evaluation Exercise) and an overall assessment of behavior and attitudes by colleagues (mini-Peer Assessment Tool). Recently, an assessment tool to assess trainee’s nontechnical skills has been introduced to ISCP as a pilot. Currently, nontechnical skills are informally acquired by surgical trainees through apprenticeship and observation rather than being explicitly addressed in training.1,2 It is felt that this passive learning style needs transition to one of active knowledge acquisition with
Journal of Surgical Education & 2015 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2015.10.008
behavioral change. One of the key assessment tools of nontechnical skills available in literature is the nontechnical skills for surgeons (NOTSS), adapted from aviation Crew Resource Management.3,4 It is a rating system based on skill taxonomy, with examples of good and poor behavioral markers. Ratings and feedback are given on 4 categories of nontechnical skills; situation awareness, decision making, communication and teamwork, and leadership. There are, however, various questions that need to be addressed before the integration of NOTSS assessment in the surgical curriculum, namely, whom should take on the role of assessor (should they exclusively be surgeons or should nonsurgeons be recruited); should raters obtain formal training in NOTSS assessment and the effect of the presence of a more senior surgeon in theater, during the assessment on the rating of the trainee’s nontechnical skills. Engagement with NOTSS and how to ensure “buy-in” from members of the theater team, particularly trainees, is also an important consideration. In this study, we present a survey among the surgical theater team that explores current views regarding the need to introduce an assessment tool of nontechnical skills of surgical trainees in the theater environment. In addition, the familiarity of members of the surgical theater team with the concept of NOTSS as well as their previous training in this area is explored.
The study group redesigned the layout of the NOTSS assessment tool developed in Scotland to mirror the ISCP PBA form (Fig. 1). This was enclosed with each questionnaire and the participants were asked to examine this NOTSS PBA form before filling the survey. Participants were asked initially about their familiarity with NOTSS assessment before rating their views on a 5point Likert scale5 regarding the need to introduce a NOTSS assessment tool into surgical curriculum, the effect of the presence of a supervising consultant in theater on the assessment of NOTSS and the need to train assessors to use the proposed NOTSS tool.
Statistical Analysis The PASW statistical package (version 18, SPSS inc, Chicago, IL) was used for statistical analysis to examine for any differences in responses among the 4 professional groups. The Kruskal Wallis test of variance was used to analyze responses among the 4 groups with the MannWhitney U test used to compare responses among 2 groups where appropriate. The open-text responses were collected separately and analyzed manually.
RESULTS METHODS Data Collection This survey was conducted in South Yorkshire, UK between November 2014 and February 2015. Ethical approval was granted by University of Sheffield Medical School (No. SMBRER287). Theater teams at 1 teaching and 2 district general hospitals were invited to participate in the survey and to provide their views on NOTSS assessment as well as their experience of nontechnical skills. The participants were divided into 4 professional groups: consultant surgeons, surgical trainees, consultants anesthetists, and theater staff. Theater staff were invited to participate if they regularly scrubbed for operations and so included theater sisters, staff nurses, and operating department practitioners. The participants were contacted directly over a period of 3 working days at the beginning of theater lists as well as during coffee breaks and asked by a core surgical trainee (resident) to fill the hard copies of the questionnaires in theaters to hand them back at the end of the lists. Each participant was approached to participate only once. Responses were inputted onto an electronic database and the data extracted. Survey Questionnaire The questionnaires consisted of 17 multiple-choice questions and open-text space when clarification is necessary.
In all, 198 staff members were approached and 160 responded, corresponding to a response rate of 81% (25 consultant surgeons, 51 trainees (grades CT1-ST8), 26 anesthetists, and 58 theater staff (Fig. 2). A total of 38 staff members did not return the questionnaires (5 consultant surgeons, 12 trainees, 3 anesthetists, and 18 theater staff). Overall, 123 respondents (77%) were not aware of the NOTSS assessment tool with only 14 (9%) reporting to have previously used the NOTSS tool and just 4 (3%) having received training in NOTSS assessment (Table 1). However, the majority (81%) stated that nontechnical skills are as important as technical skills in improving the surgical outcomes, (6% thought more important and 13% thought less important) (Fig. 3). When compared with the other theater staff, however, trainees appear to hold less importance to nontechnical skills (p r 0.016). Overall, 114 respondents (71%) agreed that anesthetists should take part in a surgical trainee’s NOTSS assessment (Fig. 4) with 128 respondents (80%) also in favor of scrub nurses assessing trainees’ NOTSS (Fig. 5). The anesthetists and scrub nurses were particularly in favor of involvement of nonsurgeons in surgical trainee’s NOTSS assessments (anesthetists to assess NOTSS, mean response 4.04 [nonsurgeons] vs. 3.66 [surgeons], p ¼ 0.008; scrub nurses to assess NOTSS, 4.29 [nonsurgeons] vs. 3.68, [surgeons] p o 0.001). All the anesthetists who responded to this
Journal of Surgical Education Volume 73/Number 2 March/April 2016
223
FIGURE 1. The NOTSS assessment tool developed in Scotland redesigned to mirror the ISCP PBA form.
224
Journal of Surgical Education Volume 73/Number 2 March/April 2016
Consultant Surgeons
16%
36%
Trainees 32%
Consultants Anaesthests
16%
Scrub Nurses
Surgeons
n
Trainees
n
Scrub N
n
General
12
Vascular
11
CST
27
Sister
25
HST
24
Plasc
1
Nurse
15
Total
51
ODP
16
Urology
1
N Prac
2
Total
25
Total
57
Anaesthests
n
Total
26
FIGURE 2. Demographics of participants. CSTs are equivalent to year 1 and 2 residents whist HSTs are equivalent to more senior residents and clinical fellows. Abbreviations: CSTs, certified surgical trainees; HSTs, higher surgical trainees.
survey disagreed that the assessment of a trainee’s NOTSS should only be conducted by the supervising consultant surgeon, compared with 69% of nonanesthetists. In all, 65% of respondents agreed that NOTSS assessment provides a good tool for reflective practice. A total of 58% agreed that performing a NOTSS assessment would not interfere with the smooth running of theater lists, whereas 18% disagreed and 24% were not sure. Only 20% of trainees and surgeons stated that NOTSS should only be assessed for higher surgical trainees whereas the majority supported the utilization of the tool for both core surgical trainees (residents) and higher surgical trainees (fellows). Although opinion appears divided as to whether the presence of a consultant surgeon in theater could potentially make it difficult to assess a trainee’s leadership skills and decision-making capabilities (Fig. 6), overall 60% agree that the routine use of NOTSS assessment would enhance safety in the operating theater and 80% agree that the NOTSS tool should be introduced to assess the nontechnical skills of trainees in theater (Fig. 7). However, a significantly lower proportion of trainees (45%) agreed on the latter compared with the other groups (p ¼ 0.001).
DISCUSSION Analyses of adverse events in healthcare have revealed that there is more often failure in nontechnical aspects of performance rather than a lack of technical expertise.6,7 It has been shown that poor nontechnical skills in the operating theater result in a reduction in technical skill8 adverse events and poor clinical outcomes.9-12 Communication has been found to be a causal factor in 43% of errors made in surgery.12 One of the key outcomes for all surgical trainees is “professional skills and behavior;” however, reference to nontechnical skills specific to the theater environment is lacking. At present, ISCP recommends utilization of mini-Peer Assessment Tool and the assigned educational supervisor end of placement summative assessments to assess professional skills and behavior. These assessments are based on retrospective views of a select group of multidisciplinary staff assessing behaviors in all aspects of the surgical trainees’ clinical activity, and most importantly lack the important element of instant feedback. The NOTSS assessment tool has been designed to gather prospective multidisciplinary feedback on nontechnical
TABLE 1. Awareness, Experience, and Training of NOTSS Among Participants Surgeons
Aware of the NOTSS assessment tool Experienced in using the NOTSS assessment tool in the past Had previous training in using the NOTSS assessment tool
Consultants
Trainees
Anesthetists
Theater Nurses
Total
12 (48%) 4 (16%)
14 (28%) 5 (10%)
3 (12%) 2 (8%)
8 (14%) 3 (5%)
37 (23%) 14 (9%)
1 (4%)
3 (6%)
0 (0%)
0 (0%)
4 (2.5%)
Journal of Surgical Education Volume 73/Number 2 March/April 2016
225
FIGURE 3. Rating of the importance NOTSS among participants. Overall, 27% of trainees stated that NOTSS is less important than technical skills (p r 0.016).
skills that are fundamental to the theater work environment, specifically situational awareness, decision-making, communication, and teamwork or leadership skills. It is currently being piloted among surgical trainees through the ISCP platform. Our survey aimed to assess the perceptions of the theater team regarding assessment of nontechnical skills using the proposed multidisciplinary NOTSS assessment tool with a view to determining how feasible this would be to perform in practice. Most of the surveyed participants agreed that nontechnical skills are important and recognized the value of introducing such an assessment tool into the ISCP. Our survey demonstrated a lack of familiarity with the existing NOTSS tool among the vast majority of theater staff surveyed in this study.
The participants agreed that anesthetists and scrub nurses should be included in this assessment process. The contribution of anesthetists and scrub nurses, whose assessments have previously been shown to be equally reliable to consultant surgeons,13 makes it an ideal tool for theaterspecific Multi-Source Feedback and a valuable instrument to provide formative assessment. Staff also felt that formal assessor training would be required before introducing this assessment tool. This is important to ensure the assessors understand all aspects of the assessment process, as previous piloting has found that cognitive skills such as situational awareness and decision making are more difficult to assess than interpersonal skills, such as communication and teamwork,13 NOTSS assessor
FIGURE 4. Views of the theater team regarding the need for anesthetists to take part in assessing NOTSS. 226
Journal of Surgical Education Volume 73/Number 2 March/April 2016
FIGURE 5. Views of the theater team regarding the need for Scrub nurses to take part in assessing NOTSS.
training in this field has been also highlighted in recent guidelines developed by expert consensus.14 The survey also revealed concerns among one-fifth of the theater team that assessment could interfere with the smooth running of the theater lists. This has not been perceived to be an issue in a previous pilot study.13 Another issue that has been highlighted in this survey is the concern that the presence of a supervising consultant in theater may limit trainees’ ability to demonstrate their leadership and decision-making skills. This has effected on the assessment process in previous studies,13 although the surgical consultant would remain the ultimate team leader and decision maker in theater, trainees need to be given an opportunity
to develop and apply such skills under supervision, particularly when an assessment is planned, requiring the consultant to “take a step back.” It was also apparent in this survey that trainees were relatively less supportive of introducing NOTSS assessment into ISCP compared with the other members of the theater team. Free-text comments showed that there is a current frustration among many surgical trainees that WBAs are being misused as assessments of learning (i.e., summative) rather that assessments for learning (i.e., formative) with WBAs viewed as “tick-box exercises” that can negatively affect the training opportunities (Table 2). In addition, consultant feedback also suggests trainer
FIGURE 6. Views of the theater team regarding the effect consultant surgeons’ presence on the ability of trainees to demonstrate NOTSS. Journal of Surgical Education Volume 73/Number 2 March/April 2016
227
FIGURE 7. Views of the theater team regarding the need to introduce NOTSS tool.
dissatisfaction with the process, with some seeing it as a “tick-box exercise” also. This has previously been reported in studies looking at perceptions of work-based assessments.15 Local education initiatives, workshops, and online training modules help to promote the NOTSS ethos and the potential that NOTSS has to enhance service delivery and patient outcomes, something that would be appreciated by scrub staff as much as the medical trainers as well as patient outcomes.
This study is not without limitations. The authors work within general and vascular surgery departments, which resulted in a larger inclusion of these specialties at the expense of less representation of other surgical subspecialties within the ISCP. Although the number of participants was relatively small, the views of all the 4 professional groups within the theater team were represented. There is also a selection bias as the group of professionals who participated could be more likely to engage with the teaching and training process than those who did not participate.
TABLE 2. Free-Text Comments Professional Group Trainees
Surgeons
Anesthetists
228
Free-Text Comment “Too much paperwork. Is done anyway.” “Not required at all. This is covered in the other PBAs. We are being swamped with this.” “NOTSS is covered in other WBAs and does not need its own assessment tool.” “Should be part of each PBA, rather than introducing another tool.” “PBAs include already the assessment of leadership skills, decision making, communication, team working, and situational awareness hence a separate PBA might add further delays in theater. In my view, we do not need a separate PBA for it.” “This is a duplicate MSF.” “Great deal of subjectivity. Views likely to differ greatly from individuals in different roles, some likely to be unreliable. Significant bias including but not limited to Hawthorne effect.” “Much of these can be freely expressed in the 360 feedback—why add yet another assessment that people struggle to complete?” “I don't think that this assessment would provide any accurate or useful data.” “I think the tool is good but should be amalgamated with the ISCP assessments.” “It would be another tick-box exercise. Surgical trainees would be better spending their time in clinic or theater rather than filling in forms.” “Just another tick-box exercise.” “NOTSS improves communication and team dynamic which ultimately improves quality of care and patients' outcome” “Dependent on the individual surgeon or experience as to whether their presence would make it difficult for me to assess a trainee's nontechnical skills.” “Introducing NOTSS assessment would be better than nothing.” Journal of Surgical Education Volume 73/Number 2 March/April 2016
CONCLUSION Our survey demonstrates acceptability among the theater team for the introduction of the NOTSS tool into the surgical curriculum. Adequate provision of training on assessing NOTSS is required before an introduction and the findings of this survey can inform appraisal of the utility of NOTSS assessment tool that is being piloted via the ISCP.
ACKNOWLEDGMENTS The authors would like to thank members of South Yorkshire Surgical Research Group Collaborative who facilitated data collection in South Yorkshire, namely, Charlotte Gunner, Matthew Lee, Khalil Madbak, Mohamed El-Sharif, Emma Nofal, Vicky Proctor, and Stuart Stokes.
6. Bogner MSE. Human Error in Medicine. Hillsdale,
NJ: Lawrence Erlbaum Associates, Inc; 1994. 7. Fletcher G, McGeorge P, Flin RH, Glavin RJ, Maran
NJ. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth. 2002;88 (3):418-429. 8. McCulloch P, Mishra A, Handa A, Dale T, Hirst G,
Catchpole K. The effects of aviation-style non-technical skills training on technical performance and outcome in the operating theatre. Qual Saf Health Care. 2009;18(2):109-115. 9. Lingard L, Espin S, Whyte S, et al. Communication
failures in the operating room: an observational classification of recurrent types and effects. Qual Saf Health Care. 2004;13(5):330-334. 10. Mazzocco K, Petitti DB, Fong KT, et al. Surgical team
behaviors and patient outcomes. Am J Surg. 2009;197 (5):678-685.
REFERENCES
11. Barach P, Johnson JK, Ahmad A, et al. A prospective
technical skills for surgeons in the operating room: A review of the literature. Surgery. 2006;139(2):140-149.
observational study of human factors, adverse events, and patient outcomes in surgery for pediatric cardiac disease. J Thorac Cardiovasc Surg. 2008;136(6):1422-1428 [PubMed PMID: 19114184].
2. Aggarwal R, Undre S, Moorthy K, Vincent C, Darzi
12. Gawande AA, Zinner MJ, Studdert DM, Brennan TA.
A. The simulated operating theatre: comprehensive training for surgical teams. Qual Saf Health Care. 2004;13(suppl 1):i27-i32.
Analysis of errors reported by surgeons at three teaching hospitals. Surgery. 2003;133(6):614-621 [PubMed PMID: 12796727].
3. Yule S, Flin R, Maran N, Rowley D, Youngson G,
13. Beard JD, Marriott J, Purdie H, Crossley J. Assessing
Paterson-Brown S. Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg. 2008;32(4): 548-556.
the surgical skills of trainees in the operating theatre: a prospective observational study of the methodology. Health Technol Assess. 2011;15(1):1-162 [i-xxi].
1. Yule S, Flin R, Paterson-Brown S, Maran N. Non-
4. Flin R, Yule S, Paterson-Brown S, Rowley D, Maran
N. The Non-Technical Skills for Surgeons (NOTSS) System Handbook v1.2. Scotland: University of Aberdeen; 2006. 5. Likert R. A technique for the measurement of atti-
tudes. Arch Psychol. 1932;22(140):1-55.
14. Hull L, Arora S, Symons NR, et al. Training faculty in
nontechnical skill assessment: national guidelines on program requirements. Ann Surg. 2013;258(2):370-375 [PubMed PMID: 23222032. Epub 2012/12/12. Eng]. 15. Ali J. Getting lost in translation? Workplace based
assessments in surgical training. Surgeon. 2013; 11(5):286-289.
Journal of Surgical Education Volume 73/Number 2 March/April 2016
229