- Email: [email protected]
Video Self-Assessment of Basic Suturing and Knot Tying Skills by Novice Trainees
ORIGINAL REPORTS
Video Self-Assessment of Basic Suturing and Knot Tying Skills by Novice Trainees Yinin Hu, MD, Debbie Tiemann, RN, and L. Michael Brunt, MD Department of Surgery and Institute for Minimally Invasive Surgery, Washington University School of Medicine, St. Louis, Missouri BACKGROUND: Self-assessment is important to learning
CONCLUSIONS: Novice trainees over-estimate their basic
but few studies have utilized video self-assessment of basic surgical skills. We compared a video self-assessment of suturing and knot tying skills by novice trainees to the assessment by a senior attending surgeon.
technical skills performance compared to the assessment by a senior surgeon. Video self-assessment may be a valuable addition to a pre-residency and surgical internship preparatory curriculum in basic suturing and knot tying. ( J Surg C 2013 Association of Program Directors in 70:279-283. J Surgery. Published by Elsevier Inc. All rights reserved.)
METHODS: Sixteen senior medical students and 7 beginner
surgical interns were video-recorded while performing five suturing and knot tying tasks. All videos were analyzed using an objective structured assessment of technical skills (OSATS) metrics (1-5 scale; 1 ¼ novice, 5 ¼ expert). Video self-assessment was carried out within 4 weeks of an instructional session and subsequently by one senior surgery instructor (blinded to the individual). Both a Global score and total combined OSATS scores were analyzed. Total possible OSATS scores were: interrupted suture—30, subcuticular closure—30, one and two-handed knot tying—25 each, tying in a restricted space 20; maximum combined score—130 points). Confidence levels in performing the tasks pre-test and the value of video self-assessment were rated on a 1-5 Likert scale (1 ¼ low and 5 ¼ high). Data are mean⫾SD and statistical significance was evaluated using Friedman’s test. RESULTS: Self-assessment scoring was significantly higher
than the assessment by a senior instructor for three tasks by global score and all five tasks by combined OSATS score (self-assessment 71.8⫾16.7 vs attending assessment 56.7⫾11.0, p ¼ 0.007). Mean self-assessment Global scores ranged from 2.5 to 2.8 for all tasks performed compared to 1.8-2.3 for attending surgeon assessment (po0.05). Confidence levels demonstrated no correlation to performance speed or proficiency. The video selfassessment was rated as a highly valuable (mean 4.3⫾0.8) component to skills training.
Correspondence: Inquiries to L. Michael Brunt, MD, Department of Surgery, Washington University School of Medicine, 660 S Euclid Ave., Box 8109, St. Louis, MO 63110; fax: 314-222-6256; e-mail: [email protected] Presented as a poster at Association for Surgical Education Meeting, Boston, MA, March 22, 2011.
KEY WORDS: trainee video self-assessment COMPETENCIES: Practice-Based Learning and Improve-
ment, Medical Knowledge, Professionalism
INTRODUCTION The importance of simulation-based training in surgical education has been increasingly recognized in recent years, and skills training is now widely incorporated into surgical training programs in North America. Training away from the operating room has demonstrated transferability to operative performance in the form of improved speed, economy of movement, error rate, and independence.1 This transferability has primarily been studied in the context of virtual-reality simulation of minimally invasive procedures among surgical residents.2 More recently, many programs have begun extending technical training to senior medical students as part of courses designed to prepare students for surgical residency. These courses may vary widely in content and intensity at different institutions, but there is evidence for significant impact both on technical proficiency and trainee confidence level.3-5 Self-instructed performance is an important source of self-efficacy. Self-efficacy is the belief about one’s capacity to produce a designated level of performance, and has been linked with superior goal-setting and perseverence.6 Effective self-instruction can only take place in the setting of accurate self-evaluation. Studies in personality psychology have demonstrated that relatively incompetent individuals are unable to accurately evaluate their own ability.7 This observation is supported in medical education by differences between perceived and OSATS performance among
Journal of Surgical Education & 2013 Association of Program Directors in Surgery. 1931-7204/$30.00 279 Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2012.10.003
residents.8 However, few studies have formally analyzed the role of self-assessment in the evaluation of surgical technical skills in a simulated environment. We hypothesize that a training regimen targeting selfefficacy among surgical trainees optimally includes both self- and objective-evaluation. The purpose of this study is to utilize video self-assessment to compare perceived vs actual competency in the basic surgical techniques of suturing and knot tying.
MATERIALS AND METHODS Basic suturing and knot tying skills were taught as a part of an Accelerated Skills Preparation at a Surgical Internship Course for senior medical students with matching surgical specialty.9,10 A similar basic suturing program was also held for beginner category surgical interns. A number of assessment tools have been incorporated into these sessions, which include a survey questionnaire that participants take prior to beginning the course in which they need to rate their level of confidence in several basic surgical tasks on a Likert scale of 1-5. In addition, participants are assessed using an OSATS form for five basic suturing and tying tasks as previously described3,10: interrupted suture, subcuticular closure, two-handed knot-tie, one-handed knottie, and tying on a pass. Testing was carried out using a suture pad (Simulab Corp, Seattle, WA) and knot tying board (Ethicon, Inc, Somerville, NJ). Participants were timed and videorecorded from two angles. Videotaping was carried out in the Simulation Center of the Washington University School of Medicine using a dual camera video capture system (Simbridge, B-Line Medical, Washington, DC) and an Internet-based viewing platform. Subsequently, participants were asked to independently review their own video recordings and rate their competency based on two scoring systems. The objective structured assessment of technical skills scoring system marks performance of each task based on characteristics such as motion/accuracy, instrument handling, anchoring and finishing stitch, needle safety, tying quality, and respect for tissue.3,10 The Combined Metric Score was the sum total of each of these scores for motion/accuracy, instrument handling, etc. The Global score comprised an overall impression of the performance level for each of the tasks performed. Participants were
unaware of the scoring format at the time of the pre-course testing. Following participant self-assessments, each video recording was reviewed objectively by a senior attending general surgeon, who was blinded to the participant, and evaluated based on the same OSATS and Global scoring systems. At the end of the course, a post-course survey asked participants to rate the value of the video self-assessment process based on a Likert scale of 1-5, and to indicate which basic techniques required additional practice. Data were analyzed using the paired Friedman test when comparing self- and instructor-assessment results. Spearman test for correlation was used to demonstrate correlations between participant confidence level, speed, and competency. The analyses were accomplished using SPSS software, with p values o0.05 considered statistically significant.
RESULTS A total of 16 medical students and 7 surgical interns participated in the video self-assessment pre-test. All video recordings were evaluated and scored by both the participant and an objective senior surgical instructor. The results of the self-assessment compared to instructor rating are shown in Tables 1 and 2. Participant self-evaluations were significantly higher than the instructor ratings for every surgical task performed for the Combined Metrics OSATS scores (Table 1). A similar relationship was seen for the Global score for all tasks except for interrupted suture and subcuticular suture (Table 2). Mean self-assessment Global scores ranged from 2.5 to 2.8 for all tasks performed, compared to 1.8-2.3 for the attending surgeon/instructor assessment. Of the five tasks evaluated, one-handed knot tying and tying on a pass had the largest divergence in the Global score between self- and instructor-evaluation. Pre-course surveys indicated that confidence levels in these basic surgical tasks on average fell between ‘‘reasonably confident, some guidance needed’’ and ‘‘highly knowledgeable and confident, independent’’. Confidence levels varied substantially between participants; however, there was no demonstrable relationship between confidence and performance speed or instructor-rated competency for four of the five tasks (Fig. 1). When asked to rate the value of the self-assessment process, participants on average rated the experience to be ‘‘highly valuable’’ (4.3⫾0.8). The trainees considered
TABLE 1. Combined Metric Score Task
Self-Assessment
Instructor Rating
p Value
Interrupted suture Subcuticular One-handed tie Two-handed tie Tie on a pass Total combined score
18.5⫾9.4 15.5⫾4.1 14.1⫾4.4 13.2⫾4.1 11.6⫾3.1 71.8⫾16.7
13.4⫾2.6 13.4⫾3.4 10.6⫾2.9 10.4⫾2.5 8.4⫾2.5 56.7⫾11.0
0.003 0.039 0.016 0.05 0.011 0.007
280
Journal of Surgical Education Volume 70/Number 2 March/April 2013
TABLE 2. Global Score Task
SelfAssessment
Instructor Rating
p Value
Interrupted suture Subcuticular One-handed tie Two-handed tie Tie on a pass Total combined score
2.6⫾0.8
2.3⫾0.4
0.151
2.5⫾0.8 2.7⫾1.0 2.5⫾0.9 2.8⫾0.8 2.6⫾0.9
2.0⫾0.7 1.8⫾0.8 1.8⫾0.7 1.9⫾0.7 2.0⫾0.7
0.054 0.004 0.013 o0.001 0.001
subcuticular closure and one-handed knot tying to be the two tasks most in need of practice.
DISCUSSION With the current trend of increasing hospital costs, decreasing resident work hours, and patient safety concerns, there is a strong move toward competency-based training that may produce proficient trainees more efficiently.11 Exacerbating this problem is the insufficiency of national compensation for medical teaching and the higher cost Confidence vs Time
1200.0 Total time (4 tasks)
1000.0 800.0 600.0 400.0 200.0 0.0
12.0
14.0
15.0
16.0
17.0
18.0
19.0
20.0
19.0
20.0
Confidence (4 tasks) Confidence vs Metric Score
Metric Score Total (4 tasks)
70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
12.0
14.0
15.0
16.0
17.0
18.0
Confidence (4 tasks)
FIGURE 1. Plots of student confidence levels vs performance. Top panel—confidence vs total time required to complete four suturing and knot tying tasks (simple and subcuticular suture and one and two handed tying). Bottom panel—confidence levels vs technical proficiency score (metrics) for the four tasks. Note: Confidence levels were not assessed pre-course for the tying on a pass task.
structure of academic institutions compared to community hospitals. Studies into the shortcomings of current surgical training reveal advanced specialty operations and endoscopic procedures as areas in which residents receive insufficient exposure.12 To optimize time during training to accommodate exposure to advanced techniques, it follows to reason that acquisition of basic surgical skills should begin at the medical student level. However, procedural exposure among medical students varies drastically between institutions and even by rotations within a given institution. Furthermore, the degree to which medical students receive specific training, instruction, and feedback in basic technical skills during medical school clerkships also varies widely.13,14 To augment surgery technical training during medical school, over the last several years many institutions have adopted the framework of accelerated mini-courses or ‘‘boot camps’’ to prepare medical students for surgical internship. The structure and content of these courses are highly variable, and cover aspects ranging from residency interviewing to clinical patient management to laparoscopic skills.4,15-18 Most camps provide instruction in basic surgical skills such as knot-tying and suturing, and some utilize video-based instruction. Computer-based video instruction for surgical technique has been shown to be a valid teaching tool,19 and, at our own institution, a combination of verbal instruction and video instruction has produced a significant improvement in technique.3 Not only do these courses address inadequacies in basic technical training, several studies have shown added benefits of reduced anxiety and improved confidence in surgical trainees prior to starting residency training. This holds true even for mini-courses as short as 1 week in duration.4,5 Confidence can have both positive and negative influences on many experiences that require perseverance en route to acquiring proficiency. Originally described in 1977 by Bandura, self-efficacy describes one’s beliefs about their capabilities to produce a certain level of performance, and greatly influences goal-setting and sustained effort.6 Individuals with high self-efficacy attribute failures to skill deficiencies which are fixable rather than permanent inadequacies, and have coping capabilities that reduce stress. They are also more apt to set challenging goals and participate in difficult situations. More recently, the selfefficacy theory has been further qualified with use of manipulated goal levels to sustain motivation and performance in the face of challenging initial personal goals that might otherwise negatively influence subsequent performance.20,21 Among the major sources of efficacy is the experience of performance accomplishments, which are induced through modeling, exposure, and self-instructed performance. Corresponding tasks in surgical skills instruction could involve video-based instruction, a surgical technique pre-test, and self-assessment followed by independent practice.
Journal of Surgical Education Volume 70/Number 2 March/April 2013
281
While confidence can positively influence performance through self-efficacy, it can also have a negative impact through lack of motivation for further effort. The latter problem generally becomes apparent when a trainee has a misperception of his actual level of competency. Unfortunately, such instances are not uncommon. Kruger demonstrated that relatively unskilled individuals are particularly poor at recognizing deficiency, a pattern that holds true across a variety of competencies.7 In the medical field, competency has been shown to be intimately related to previous experiences of a task, while confidence influences willingness to undertake a task, regardless of past experiences.5 While self-evaluation has often been utilized as a modality for reflection and analysis, its accuracy as a measure of competency is frequently questioned. For example, while self-perceived competence has been shown to improve significantly during the course of medical school education in areas such as consultation and clinical skills, clinical skills assessments through OSCE exams have shown no correlation between confidence and measured competency among post-graduate year one (PGY1) trainees.8,22 This disparity holds true in surgical education, as mid-clerkship self-evaluations have demonstrated significant inaccuracy in performance estimation among both male and female medical students.23 Proficiency-based training emphasizes self-directed practice to achieve a predetermined level of proficiency. While such methods have demonstrated validity in basic surgical skills training,24,25 it stands to reason that accurate self-evaluation is a prerequisite for its implementation. It is our belief that video self-assessment can be a useful tool for basic surgical skills training. Through a combination of selfand objective-assessment, these assessments guide independent practice and induce self-efficacy without propagating a detrimental disparity between confidence and competency. The use of video assessments in surgical training has been studied among surgery residents. Driscoll et al.26 compared real-time assessment and video assessment of tissue-handling skills for surgery trainees performing open inguinal hernia repairs and found both modalities to be reliable tools for evaluation. Video assessment scores were able to distinguish trainees from experienced operators, and there was good trainer-trainee agreement in scoring. However, because the video assessments were blinded, this does not fully address the utility of a video self-assessment. Jamshidi et al.27 used resident video recordings of basic laparoscopic skills to analyze the impact of selfassessment on technical improvement. The group that was allowed to review their own video recordings demonstrated significantly greater improvement after independent practice. To our knowledge, the present study is the first to document the use of video self-assessments for basic open surgical skills among entry-level surgical trainees. Overall disparity between self- and objective-assessment allows trainees to better understand misconceptions of competency and address deficiencies on an individual basis. 282
Additionally, the greater disparity noted in more advanced techniques such as one-handed and restricted-space knot tying demonstrates shortcomings in technical exposure during medical school. In concordance with previous studies relating confidence to competency, our results demonstrate no correlation between confidence level and performance speed or proficiency.10 Most importantly, video self-assessment guided further practice, thus contributing to aforementioned sources of self-efficacy. Not surprisingly, the experience was considered a highlyvaluable component of the accelerated surgical skills mini-course. We recognize several limitations in the present study. Relatively low sample size prevented differentiation of our study subjects into a medical student group and a surgical intern group. Because PGY1 participants performed their video self-assessments early in their first year of residency, it was considered valid to incorporate them into the same group as senior medical students. However, entering PGY1’s may experience different levels of exposure to basic surgical skills that can confound pre-test performance. Future studies should accommodate larger sample numbers to provide two independent subject pools. Another limitation is the dependence upon only one blinded senior surgical instructor for objective-assessment due to time constraints. Because judgment of technical skills is operator-dependent, recruitment of additional senior instructors for video review may reduce personal biases. Accurate self-assessment is crucial both to promote selfefficacy and to prevent over confidence. Video selfassessment provides opportunities for self-reflection and guidance for proficiency-based training, especially among relatively low-competency trainees such as senior medical students and those entering surgery PGY1’s. Over the ensuing years, evolving pressures on residency programs will demand extension of basic surgical technical training into medical school curricula, and refinement of accelerated mini-courses will be a continuing challenge for surgical educators. The use of video self-assessment can be a valuable tool in this setting when used in conjunction with objective assessment by an experienced evaluator.
REFERENCES 1. Sturm LP, Windsor JA, Cosman PH, Cregan P,
Hewett PJ, Maddern GJ. A systematic review of skills transfer after surgical simulation training. Ann Surg. 2008;248:166-179. 2. Seymour NE, Gallagher AG, Roman SA, et al. Virtual
reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236(458):63, [discussion 463–464]. 3. Brunt LM, Halpin VJ, Klingensmith ME, et al.
Accelerated skills preparation and assessment for
Journal of Surgical Education Volume 70/Number 2 March/April 2013
senior medical students entering surgical internship. J Am Coll Surg. 2008;206(897):904, [discussion 904–907]. 4. Peyre SE, Peyre CG, Sullivan ME, Towfigh S. A
surgical skills elective can improve student confidence prior to internship. J Surg Res. 2006;1(133):11-15. 5. Stewart J, O’Halloran C, Barton JR, Singleton SJ,
17. Stewart RA, Hauge LS, Stewart RD, Rosen RL,
Charnot-Katsikas A, Prinz RA. Association for Surgical Education. A CRASH course in procedural skills improves medical students’ self-assessment of proficiency, confidence, and anxiety. Am J Surg. 2007; 193:771-773.
Harrigan P, Spencer J. Clarifying the concepts of confidence and competence to produce appropriate self-evaluation measurement scales. Med Educ. 2000; 34:903-909.
18. Klingensmith ME, Brunt LM. Focused surgical skills
6. Bandura A. Self-efficacy: toward a unifying theory of
Leblanc V, Dubrowski A. Teaching suturing and knot-tying skills to medical students: a randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback. Surgery. 2007;141:442-449.
behavioral change. Psychol Rev. 1977;84:191-215. 7. Kruger J, Dunning D. Unskilled and unaware of it:
how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77:1121-1134. 8. Barnsley L, Lyon PM, Ralston SJ, et al. Clinical skills
in junior medical officers: a comparison of selfreported confidence and observed competence. Med Educ. 2004;38:358-367. 9. Smith CD. Teaching surgical techniques and proce-
dures using advanced educational tools and concepts. Asian J Surg. 2005;28:159-162. 10. Zeng W, Woodhouse J, Brunt LM. Do preclinical
background and clerkship experience impact skills performance in an accelerated skills course for senior medical students? Surgery. 2010;148:768-776. 11. Schill M, Tiemann D, Klingensmith ME, Brunt LM.
Year one outcomes assessment of a masters suturing and knot-tying program for surgical interns. J Surg Educ. 2011;68:526-533. 12. Brennan MF, Debas HT. Surgical education in the
United States: portents for change. Ann Surg. 2004; 240:565-572. 13. Irani JL, Greenberg JA, Blanco MA, et al. Educational
value of the operating room experience during a core surgical clerkship. Am J Surg. 2010;200:167-172.
training for senior medical students and interns. Surg Clin North Am. 2010;90:505-518. 19. Xeroulis GJ, Park J, Moulton CA, Reznick RK,
20. Vancouver JB, Thompson CM, Williams AA. The
changing signs in the relationships among self-efficacy, personal goals, and performance. J Appl Psychol. 2001;86:605-620. 21. Powers WT. Feedback: beyond behaviorism. Science.
1973;179(71):351-356. 22. Kramer AW, Zuithoff P, Jansen JJ, Tan LH, Grol RP,
Van der Vleuten CP. Growth of self-perceived clinical competence in postgraduate training for general practice and its relation to potentially influencing factors. Adv Health Sci Educ: Theory Pract. 2007; 12:135-145. 23. Lind DS, Rekkas S, Bui V, Lam T, Beierle E,
Copeland EM 3rd. Competency-based student selfassessment on a surgery rotation. J Surg Res. 2002; 105:31-34. 24. Jowett N, LeBlanc V, Xeroulis G, MacRae H,
Dubrowski A. Surgical skill acquisition with selfdirected practice using computer-based video training. Am J Surg. 2007;193:237-242. 25. Goova MT, Hollett LA, Tesfay ST, et al. Implemen-
with surgical procedures as medical students. Am J Surg. 1990;159(341):3, [discussion 344].
tation, construct validity, and benefit of a proficiencybased knot-tying and suturing curriculum. J Surg Educ. 2008;65(4):309-315.
15. Boehler ML, Rogers DA, Schwind CJ, Fortune J,
26. Driscoll PJ, Paisley AM, Paterson-Brown S. Video
Ketchum J, Dunnington G. A senior elective designed to prepare medical students for surgical residency. Am J Surg. 2004;187:695-697.
27. Jamshidi R, LaMasters T, Eisenberg D, Duh QY,
14. Nakayama DK, Steiber A. Surgery interns’ experience
16. Naylor RA, Hollett LA, Castellvi A, Valentine RJ,
Scott DJ. Preparing medical students to enter surgery residencies. Am J Surg. 2010;199:105-109.
assessment of basic surgical trainees’ operative skills. Am J Surg. 2008;196:265-272. Curet M. Video self-assessment augments development of videoscopic suturing skill. J Am Coll Surg. 2009;209:622-625.
Journal of Surgical Education Volume 70/Number 2 March/April 2013
283
Video Self-Assessment of Basic Suturing and Knot Tying Skills by Novice Trainees Yinin Hu, MD, Debbie Tiemann, RN, and L. Michael Brunt, MD Department of Surgery and Institute for Minimally Invasive Surgery, Washington University School of Medicine, St. Louis, Missouri BACKGROUND: Self-assessment is important to learning
CONCLUSIONS: Novice trainees over-estimate their basic
but few studies have utilized video self-assessment of basic surgical skills. We compared a video self-assessment of suturing and knot tying skills by novice trainees to the assessment by a senior attending surgeon.
technical skills performance compared to the assessment by a senior surgeon. Video self-assessment may be a valuable addition to a pre-residency and surgical internship preparatory curriculum in basic suturing and knot tying. ( J Surg C 2013 Association of Program Directors in 70:279-283. J Surgery. Published by Elsevier Inc. All rights reserved.)
METHODS: Sixteen senior medical students and 7 beginner
surgical interns were video-recorded while performing five suturing and knot tying tasks. All videos were analyzed using an objective structured assessment of technical skills (OSATS) metrics (1-5 scale; 1 ¼ novice, 5 ¼ expert). Video self-assessment was carried out within 4 weeks of an instructional session and subsequently by one senior surgery instructor (blinded to the individual). Both a Global score and total combined OSATS scores were analyzed. Total possible OSATS scores were: interrupted suture—30, subcuticular closure—30, one and two-handed knot tying—25 each, tying in a restricted space 20; maximum combined score—130 points). Confidence levels in performing the tasks pre-test and the value of video self-assessment were rated on a 1-5 Likert scale (1 ¼ low and 5 ¼ high). Data are mean⫾SD and statistical significance was evaluated using Friedman’s test. RESULTS: Self-assessment scoring was significantly higher
than the assessment by a senior instructor for three tasks by global score and all five tasks by combined OSATS score (self-assessment 71.8⫾16.7 vs attending assessment 56.7⫾11.0, p ¼ 0.007). Mean self-assessment Global scores ranged from 2.5 to 2.8 for all tasks performed compared to 1.8-2.3 for attending surgeon assessment (po0.05). Confidence levels demonstrated no correlation to performance speed or proficiency. The video selfassessment was rated as a highly valuable (mean 4.3⫾0.8) component to skills training.
Correspondence: Inquiries to L. Michael Brunt, MD, Department of Surgery, Washington University School of Medicine, 660 S Euclid Ave., Box 8109, St. Louis, MO 63110; fax: 314-222-6256; e-mail: [email protected] Presented as a poster at Association for Surgical Education Meeting, Boston, MA, March 22, 2011.
KEY WORDS: trainee video self-assessment COMPETENCIES: Practice-Based Learning and Improve-
ment, Medical Knowledge, Professionalism
INTRODUCTION The importance of simulation-based training in surgical education has been increasingly recognized in recent years, and skills training is now widely incorporated into surgical training programs in North America. Training away from the operating room has demonstrated transferability to operative performance in the form of improved speed, economy of movement, error rate, and independence.1 This transferability has primarily been studied in the context of virtual-reality simulation of minimally invasive procedures among surgical residents.2 More recently, many programs have begun extending technical training to senior medical students as part of courses designed to prepare students for surgical residency. These courses may vary widely in content and intensity at different institutions, but there is evidence for significant impact both on technical proficiency and trainee confidence level.3-5 Self-instructed performance is an important source of self-efficacy. Self-efficacy is the belief about one’s capacity to produce a designated level of performance, and has been linked with superior goal-setting and perseverence.6 Effective self-instruction can only take place in the setting of accurate self-evaluation. Studies in personality psychology have demonstrated that relatively incompetent individuals are unable to accurately evaluate their own ability.7 This observation is supported in medical education by differences between perceived and OSATS performance among
Journal of Surgical Education & 2013 Association of Program Directors in Surgery. 1931-7204/$30.00 279 Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2012.10.003
residents.8 However, few studies have formally analyzed the role of self-assessment in the evaluation of surgical technical skills in a simulated environment. We hypothesize that a training regimen targeting selfefficacy among surgical trainees optimally includes both self- and objective-evaluation. The purpose of this study is to utilize video self-assessment to compare perceived vs actual competency in the basic surgical techniques of suturing and knot tying.
MATERIALS AND METHODS Basic suturing and knot tying skills were taught as a part of an Accelerated Skills Preparation at a Surgical Internship Course for senior medical students with matching surgical specialty.9,10 A similar basic suturing program was also held for beginner category surgical interns. A number of assessment tools have been incorporated into these sessions, which include a survey questionnaire that participants take prior to beginning the course in which they need to rate their level of confidence in several basic surgical tasks on a Likert scale of 1-5. In addition, participants are assessed using an OSATS form for five basic suturing and tying tasks as previously described3,10: interrupted suture, subcuticular closure, two-handed knot-tie, one-handed knottie, and tying on a pass. Testing was carried out using a suture pad (Simulab Corp, Seattle, WA) and knot tying board (Ethicon, Inc, Somerville, NJ). Participants were timed and videorecorded from two angles. Videotaping was carried out in the Simulation Center of the Washington University School of Medicine using a dual camera video capture system (Simbridge, B-Line Medical, Washington, DC) and an Internet-based viewing platform. Subsequently, participants were asked to independently review their own video recordings and rate their competency based on two scoring systems. The objective structured assessment of technical skills scoring system marks performance of each task based on characteristics such as motion/accuracy, instrument handling, anchoring and finishing stitch, needle safety, tying quality, and respect for tissue.3,10 The Combined Metric Score was the sum total of each of these scores for motion/accuracy, instrument handling, etc. The Global score comprised an overall impression of the performance level for each of the tasks performed. Participants were
unaware of the scoring format at the time of the pre-course testing. Following participant self-assessments, each video recording was reviewed objectively by a senior attending general surgeon, who was blinded to the participant, and evaluated based on the same OSATS and Global scoring systems. At the end of the course, a post-course survey asked participants to rate the value of the video self-assessment process based on a Likert scale of 1-5, and to indicate which basic techniques required additional practice. Data were analyzed using the paired Friedman test when comparing self- and instructor-assessment results. Spearman test for correlation was used to demonstrate correlations between participant confidence level, speed, and competency. The analyses were accomplished using SPSS software, with p values o0.05 considered statistically significant.
RESULTS A total of 16 medical students and 7 surgical interns participated in the video self-assessment pre-test. All video recordings were evaluated and scored by both the participant and an objective senior surgical instructor. The results of the self-assessment compared to instructor rating are shown in Tables 1 and 2. Participant self-evaluations were significantly higher than the instructor ratings for every surgical task performed for the Combined Metrics OSATS scores (Table 1). A similar relationship was seen for the Global score for all tasks except for interrupted suture and subcuticular suture (Table 2). Mean self-assessment Global scores ranged from 2.5 to 2.8 for all tasks performed, compared to 1.8-2.3 for the attending surgeon/instructor assessment. Of the five tasks evaluated, one-handed knot tying and tying on a pass had the largest divergence in the Global score between self- and instructor-evaluation. Pre-course surveys indicated that confidence levels in these basic surgical tasks on average fell between ‘‘reasonably confident, some guidance needed’’ and ‘‘highly knowledgeable and confident, independent’’. Confidence levels varied substantially between participants; however, there was no demonstrable relationship between confidence and performance speed or instructor-rated competency for four of the five tasks (Fig. 1). When asked to rate the value of the self-assessment process, participants on average rated the experience to be ‘‘highly valuable’’ (4.3⫾0.8). The trainees considered
TABLE 1. Combined Metric Score Task
Self-Assessment
Instructor Rating
p Value
Interrupted suture Subcuticular One-handed tie Two-handed tie Tie on a pass Total combined score
18.5⫾9.4 15.5⫾4.1 14.1⫾4.4 13.2⫾4.1 11.6⫾3.1 71.8⫾16.7
13.4⫾2.6 13.4⫾3.4 10.6⫾2.9 10.4⫾2.5 8.4⫾2.5 56.7⫾11.0
0.003 0.039 0.016 0.05 0.011 0.007
280
Journal of Surgical Education Volume 70/Number 2 March/April 2013
TABLE 2. Global Score Task
SelfAssessment
Instructor Rating
p Value
Interrupted suture Subcuticular One-handed tie Two-handed tie Tie on a pass Total combined score
2.6⫾0.8
2.3⫾0.4
0.151
2.5⫾0.8 2.7⫾1.0 2.5⫾0.9 2.8⫾0.8 2.6⫾0.9
2.0⫾0.7 1.8⫾0.8 1.8⫾0.7 1.9⫾0.7 2.0⫾0.7
0.054 0.004 0.013 o0.001 0.001
subcuticular closure and one-handed knot tying to be the two tasks most in need of practice.
DISCUSSION With the current trend of increasing hospital costs, decreasing resident work hours, and patient safety concerns, there is a strong move toward competency-based training that may produce proficient trainees more efficiently.11 Exacerbating this problem is the insufficiency of national compensation for medical teaching and the higher cost Confidence vs Time
1200.0 Total time (4 tasks)
1000.0 800.0 600.0 400.0 200.0 0.0
12.0
14.0
15.0
16.0
17.0
18.0
19.0
20.0
19.0
20.0
Confidence (4 tasks) Confidence vs Metric Score
Metric Score Total (4 tasks)
70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
12.0
14.0
15.0
16.0
17.0
18.0
Confidence (4 tasks)
FIGURE 1. Plots of student confidence levels vs performance. Top panel—confidence vs total time required to complete four suturing and knot tying tasks (simple and subcuticular suture and one and two handed tying). Bottom panel—confidence levels vs technical proficiency score (metrics) for the four tasks. Note: Confidence levels were not assessed pre-course for the tying on a pass task.
structure of academic institutions compared to community hospitals. Studies into the shortcomings of current surgical training reveal advanced specialty operations and endoscopic procedures as areas in which residents receive insufficient exposure.12 To optimize time during training to accommodate exposure to advanced techniques, it follows to reason that acquisition of basic surgical skills should begin at the medical student level. However, procedural exposure among medical students varies drastically between institutions and even by rotations within a given institution. Furthermore, the degree to which medical students receive specific training, instruction, and feedback in basic technical skills during medical school clerkships also varies widely.13,14 To augment surgery technical training during medical school, over the last several years many institutions have adopted the framework of accelerated mini-courses or ‘‘boot camps’’ to prepare medical students for surgical internship. The structure and content of these courses are highly variable, and cover aspects ranging from residency interviewing to clinical patient management to laparoscopic skills.4,15-18 Most camps provide instruction in basic surgical skills such as knot-tying and suturing, and some utilize video-based instruction. Computer-based video instruction for surgical technique has been shown to be a valid teaching tool,19 and, at our own institution, a combination of verbal instruction and video instruction has produced a significant improvement in technique.3 Not only do these courses address inadequacies in basic technical training, several studies have shown added benefits of reduced anxiety and improved confidence in surgical trainees prior to starting residency training. This holds true even for mini-courses as short as 1 week in duration.4,5 Confidence can have both positive and negative influences on many experiences that require perseverance en route to acquiring proficiency. Originally described in 1977 by Bandura, self-efficacy describes one’s beliefs about their capabilities to produce a certain level of performance, and greatly influences goal-setting and sustained effort.6 Individuals with high self-efficacy attribute failures to skill deficiencies which are fixable rather than permanent inadequacies, and have coping capabilities that reduce stress. They are also more apt to set challenging goals and participate in difficult situations. More recently, the selfefficacy theory has been further qualified with use of manipulated goal levels to sustain motivation and performance in the face of challenging initial personal goals that might otherwise negatively influence subsequent performance.20,21 Among the major sources of efficacy is the experience of performance accomplishments, which are induced through modeling, exposure, and self-instructed performance. Corresponding tasks in surgical skills instruction could involve video-based instruction, a surgical technique pre-test, and self-assessment followed by independent practice.
Journal of Surgical Education Volume 70/Number 2 March/April 2013
281
While confidence can positively influence performance through self-efficacy, it can also have a negative impact through lack of motivation for further effort. The latter problem generally becomes apparent when a trainee has a misperception of his actual level of competency. Unfortunately, such instances are not uncommon. Kruger demonstrated that relatively unskilled individuals are particularly poor at recognizing deficiency, a pattern that holds true across a variety of competencies.7 In the medical field, competency has been shown to be intimately related to previous experiences of a task, while confidence influences willingness to undertake a task, regardless of past experiences.5 While self-evaluation has often been utilized as a modality for reflection and analysis, its accuracy as a measure of competency is frequently questioned. For example, while self-perceived competence has been shown to improve significantly during the course of medical school education in areas such as consultation and clinical skills, clinical skills assessments through OSCE exams have shown no correlation between confidence and measured competency among post-graduate year one (PGY1) trainees.8,22 This disparity holds true in surgical education, as mid-clerkship self-evaluations have demonstrated significant inaccuracy in performance estimation among both male and female medical students.23 Proficiency-based training emphasizes self-directed practice to achieve a predetermined level of proficiency. While such methods have demonstrated validity in basic surgical skills training,24,25 it stands to reason that accurate self-evaluation is a prerequisite for its implementation. It is our belief that video self-assessment can be a useful tool for basic surgical skills training. Through a combination of selfand objective-assessment, these assessments guide independent practice and induce self-efficacy without propagating a detrimental disparity between confidence and competency. The use of video assessments in surgical training has been studied among surgery residents. Driscoll et al.26 compared real-time assessment and video assessment of tissue-handling skills for surgery trainees performing open inguinal hernia repairs and found both modalities to be reliable tools for evaluation. Video assessment scores were able to distinguish trainees from experienced operators, and there was good trainer-trainee agreement in scoring. However, because the video assessments were blinded, this does not fully address the utility of a video self-assessment. Jamshidi et al.27 used resident video recordings of basic laparoscopic skills to analyze the impact of selfassessment on technical improvement. The group that was allowed to review their own video recordings demonstrated significantly greater improvement after independent practice. To our knowledge, the present study is the first to document the use of video self-assessments for basic open surgical skills among entry-level surgical trainees. Overall disparity between self- and objective-assessment allows trainees to better understand misconceptions of competency and address deficiencies on an individual basis. 282
Additionally, the greater disparity noted in more advanced techniques such as one-handed and restricted-space knot tying demonstrates shortcomings in technical exposure during medical school. In concordance with previous studies relating confidence to competency, our results demonstrate no correlation between confidence level and performance speed or proficiency.10 Most importantly, video self-assessment guided further practice, thus contributing to aforementioned sources of self-efficacy. Not surprisingly, the experience was considered a highlyvaluable component of the accelerated surgical skills mini-course. We recognize several limitations in the present study. Relatively low sample size prevented differentiation of our study subjects into a medical student group and a surgical intern group. Because PGY1 participants performed their video self-assessments early in their first year of residency, it was considered valid to incorporate them into the same group as senior medical students. However, entering PGY1’s may experience different levels of exposure to basic surgical skills that can confound pre-test performance. Future studies should accommodate larger sample numbers to provide two independent subject pools. Another limitation is the dependence upon only one blinded senior surgical instructor for objective-assessment due to time constraints. Because judgment of technical skills is operator-dependent, recruitment of additional senior instructors for video review may reduce personal biases. Accurate self-assessment is crucial both to promote selfefficacy and to prevent over confidence. Video selfassessment provides opportunities for self-reflection and guidance for proficiency-based training, especially among relatively low-competency trainees such as senior medical students and those entering surgery PGY1’s. Over the ensuing years, evolving pressures on residency programs will demand extension of basic surgical technical training into medical school curricula, and refinement of accelerated mini-courses will be a continuing challenge for surgical educators. The use of video self-assessment can be a valuable tool in this setting when used in conjunction with objective assessment by an experienced evaluator.
REFERENCES 1. Sturm LP, Windsor JA, Cosman PH, Cregan P,
Hewett PJ, Maddern GJ. A systematic review of skills transfer after surgical simulation training. Ann Surg. 2008;248:166-179. 2. Seymour NE, Gallagher AG, Roman SA, et al. Virtual
reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg. 2002;236(458):63, [discussion 463–464]. 3. Brunt LM, Halpin VJ, Klingensmith ME, et al.
Accelerated skills preparation and assessment for
Journal of Surgical Education Volume 70/Number 2 March/April 2013
senior medical students entering surgical internship. J Am Coll Surg. 2008;206(897):904, [discussion 904–907]. 4. Peyre SE, Peyre CG, Sullivan ME, Towfigh S. A
surgical skills elective can improve student confidence prior to internship. J Surg Res. 2006;1(133):11-15. 5. Stewart J, O’Halloran C, Barton JR, Singleton SJ,
17. Stewart RA, Hauge LS, Stewart RD, Rosen RL,
Charnot-Katsikas A, Prinz RA. Association for Surgical Education. A CRASH course in procedural skills improves medical students’ self-assessment of proficiency, confidence, and anxiety. Am J Surg. 2007; 193:771-773.
Harrigan P, Spencer J. Clarifying the concepts of confidence and competence to produce appropriate self-evaluation measurement scales. Med Educ. 2000; 34:903-909.
18. Klingensmith ME, Brunt LM. Focused surgical skills
6. Bandura A. Self-efficacy: toward a unifying theory of
Leblanc V, Dubrowski A. Teaching suturing and knot-tying skills to medical students: a randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback. Surgery. 2007;141:442-449.
behavioral change. Psychol Rev. 1977;84:191-215. 7. Kruger J, Dunning D. Unskilled and unaware of it:
how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999;77:1121-1134. 8. Barnsley L, Lyon PM, Ralston SJ, et al. Clinical skills
in junior medical officers: a comparison of selfreported confidence and observed competence. Med Educ. 2004;38:358-367. 9. Smith CD. Teaching surgical techniques and proce-
dures using advanced educational tools and concepts. Asian J Surg. 2005;28:159-162. 10. Zeng W, Woodhouse J, Brunt LM. Do preclinical
background and clerkship experience impact skills performance in an accelerated skills course for senior medical students? Surgery. 2010;148:768-776. 11. Schill M, Tiemann D, Klingensmith ME, Brunt LM.
Year one outcomes assessment of a masters suturing and knot-tying program for surgical interns. J Surg Educ. 2011;68:526-533. 12. Brennan MF, Debas HT. Surgical education in the
United States: portents for change. Ann Surg. 2004; 240:565-572. 13. Irani JL, Greenberg JA, Blanco MA, et al. Educational
value of the operating room experience during a core surgical clerkship. Am J Surg. 2010;200:167-172.
training for senior medical students and interns. Surg Clin North Am. 2010;90:505-518. 19. Xeroulis GJ, Park J, Moulton CA, Reznick RK,
20. Vancouver JB, Thompson CM, Williams AA. The
changing signs in the relationships among self-efficacy, personal goals, and performance. J Appl Psychol. 2001;86:605-620. 21. Powers WT. Feedback: beyond behaviorism. Science.
1973;179(71):351-356. 22. Kramer AW, Zuithoff P, Jansen JJ, Tan LH, Grol RP,
Van der Vleuten CP. Growth of self-perceived clinical competence in postgraduate training for general practice and its relation to potentially influencing factors. Adv Health Sci Educ: Theory Pract. 2007; 12:135-145. 23. Lind DS, Rekkas S, Bui V, Lam T, Beierle E,
Copeland EM 3rd. Competency-based student selfassessment on a surgery rotation. J Surg Res. 2002; 105:31-34. 24. Jowett N, LeBlanc V, Xeroulis G, MacRae H,
Dubrowski A. Surgical skill acquisition with selfdirected practice using computer-based video training. Am J Surg. 2007;193:237-242. 25. Goova MT, Hollett LA, Tesfay ST, et al. Implemen-
with surgical procedures as medical students. Am J Surg. 1990;159(341):3, [discussion 344].
tation, construct validity, and benefit of a proficiencybased knot-tying and suturing curriculum. J Surg Educ. 2008;65(4):309-315.
15. Boehler ML, Rogers DA, Schwind CJ, Fortune J,
26. Driscoll PJ, Paisley AM, Paterson-Brown S. Video
Ketchum J, Dunnington G. A senior elective designed to prepare medical students for surgical residency. Am J Surg. 2004;187:695-697.
27. Jamshidi R, LaMasters T, Eisenberg D, Duh QY,
14. Nakayama DK, Steiber A. Surgery interns’ experience
16. Naylor RA, Hollett LA, Castellvi A, Valentine RJ,
Scott DJ. Preparing medical students to enter surgery residencies. Am J Surg. 2010;199:105-109.
assessment of basic surgical trainees’ operative skills. Am J Surg. 2008;196:265-272. Curet M. Video self-assessment augments development of videoscopic suturing skill. J Am Coll Surg. 2009;209:622-625.
Journal of Surgical Education Volume 70/Number 2 March/April 2013
283