- Email: [email protected]
Assessing the Reliability and Validity of Technical Skill Assessment Tools and the Impact of Rater Training
S172
Scientific Forum Abstracts
J Am Coll Surg
prediction and identifying morbidity causality may be related. We intend to further development this response platform to demonstrate discrimination between experienced and inexperienced respondents based on case complexity, trend learner performance over time, and ultimately, to monitor future targeted educational interventions. Assessing the Reliability and Validity of Technical Skill Assessment Tools and the Impact of Rater Training Reagan Maniar, MD, Ashley Vergis, MD, FRCSC, FACS, Lawrence Gillman, MD, FRCSC, FACS, Jason Park, MD, FRCSC, FACS University of Manitoba, Winnipeg, MB INTRODUCTION: Competency-based education frameworks are increasingly being used in surgical training. Such frameworks require robust assessments of technical skill, yet the optimal use of assessment tools has not been well defined. We assessed the effect of a rater training (RT) program on the reliability and validity of 4 technical assessment tools. METHODS: Surgeons were randomized to RT and no-training groups. A pass-fail system, a visual analog scale (VAS), a task-specific checklist, and an OSATS global rating scale (GRS) were used to assess trainees performing a suturing and knot-tying task. Trainee performances were assessed again 2 weeks later. Internal consistency, intra-rater and inter-rater reliability, and construct validity were measured. RESULTS: Forty-seven surgeons were randomized to RT (n ¼ 24) and nontraining (n ¼ 23). The OSATS GRS had superior reliability and validity compared to the other assessment tools regardless of training. Although there were no significant differences in reliability between training groups, the RT group trended to improved reliability for all assessment forms at both assessments (Table). Table. Assessment form Pass-fail RT No training VAS RT No training Checklist RT No training GRS RT No training
Inter-rater reliability Initial
Delayed
Intra-rater reliability
0.22 (0.18e0.26) 0.20 (0.17e0.25)
0.16 (0.13e0.20) 0.11 (0.07e0.15)
0.45 (0.32e0.59) 0.41 (0.26e0.57)
0.71 (0.50e0.91) 0.46 (0.27e0.75)
0.54 (0.34e0.80) 0.43 (0.24e0.73)
0.71 (0.64e0.77) 0.62 (0.54e0.70)
0.46 (0.27e0.75) 0.33 (0.17e0.64)
0.42 (0.23e0.71) 0.27 (0.13e0.57)
0.53 (0.43e0.61) 0.46 (0.35e0.56)
0.71 (0.52e0.89) 0.61 (0.41e0.85)
0.64 (0.45e0.86) 0.52 (0.32e0.79)
0.73 (0.67e0.79) 0.66 (0.58e0.73)
CONCLUSIONS: RT did not lead to significant or sustained improvements in technical skill assessments. Trainee evaluations should be performed with standardized assessment tools, such as the OSATS GRS, due to their superior reliability and validity over less standardized methods. Despite our findings, training
remains an important tool to ensure the most effective technical skill evaluations. More effective training methods or repeated sessions may be required for sustained or significant effects. Can Operative Trauma Simulation Eliminate Sex Differences in Self-Confidence? Audrey C Pendleton, MD, Afshin Parsikia, MD, MPH, Sheldon H Teperman, MD, FACS, Melvin E Stone, MD, FACS, Srinivas H Reddy, MD, Edward Chao, MBBS, FACS, Jody M Kaban, MD, FACS Jacobi Medical Center, Bronx, NY INTRODUCTION: Simulation is playing an increasingly significant role in training surgical residents as paradigm shifts in residency have evolved and work hour restrictions have limited exposure to certain clinical areas of general surgery. Simulation can improve technical skill as well enhance self-confidence, the latter being especially important in high acuity situations such as trauma and emergency surgery. Previously published studies have documented sex differences in confidence among surgical residents, with females expressing less confidence in their skills. The goal of our study was to compare confidence levels of senior surgical residents participating in an operative simulation course by sex. METHODS: This was a retrospective review of senior residents’ participation in the Advanced Operative Trauma Management (ATOM) course at our institution from 2007 to 2016. Standard components of the course include pre and post self-confidence questionnaires and knowledge tests, in addition to evaluation of operative skill. Course components were compared by sex using a paired Student’s t-test. RESULTS: There were 247 residents who took the simulation course; 60.7% were male. Female residents scored significantly lower on pre-course self-evaluation (p < 0.001) compared to the men. This difference was eliminated in post self-confidence assessments (p ¼ 0.099). There was no difference between the sexes in knowledge or operative performance, as judged by instructors. Table. Comparison of Participants by Sex Male Female residents, residents, n (%) n (%) Variable (SD) (n ¼ 150) (n ¼ 97) Pre self-evaluation 63.61 (14.85) 56.60 (15.59) Post self-evaluation 99.91 (14.97) 96.87 (12.68) Self-confidence improvement 36.29 (16.68) 40.27 (16.43) Course evaluation 37.13 (6.28) 37.98 (3.29) Instructor post-evaluation 38.23 (6.01) 38.70 (4.67) Pre exam 63.04 (15.22) 60.54 (15.05) Post exam 76.64 (11.49) 75.30 (11.36) Instructor post-evaluation 78.53 (7.19) 76.41 (14.24)
Total, n (%) (n ¼ 247) p Value 60.86 (15.50) <0.001 98.71 (14.16) 0.099 37.85 (16.66) 37.46 (5.32)
0.067 0.219
38.42 (5.52) 62.06 (15.17) 76.11 (11.44)
0.517 0.206 0.369
77.70 (10.56)
0.176
CONCLUSIONS: Female residents had equivalent knowledge and skill when compared to male residents, but expressed less confidence. After simulation, the female residents’ confidence equaled
Scientific Forum Abstracts
J Am Coll Surg
prediction and identifying morbidity causality may be related. We intend to further development this response platform to demonstrate discrimination between experienced and inexperienced respondents based on case complexity, trend learner performance over time, and ultimately, to monitor future targeted educational interventions. Assessing the Reliability and Validity of Technical Skill Assessment Tools and the Impact of Rater Training Reagan Maniar, MD, Ashley Vergis, MD, FRCSC, FACS, Lawrence Gillman, MD, FRCSC, FACS, Jason Park, MD, FRCSC, FACS University of Manitoba, Winnipeg, MB INTRODUCTION: Competency-based education frameworks are increasingly being used in surgical training. Such frameworks require robust assessments of technical skill, yet the optimal use of assessment tools has not been well defined. We assessed the effect of a rater training (RT) program on the reliability and validity of 4 technical assessment tools. METHODS: Surgeons were randomized to RT and no-training groups. A pass-fail system, a visual analog scale (VAS), a task-specific checklist, and an OSATS global rating scale (GRS) were used to assess trainees performing a suturing and knot-tying task. Trainee performances were assessed again 2 weeks later. Internal consistency, intra-rater and inter-rater reliability, and construct validity were measured. RESULTS: Forty-seven surgeons were randomized to RT (n ¼ 24) and nontraining (n ¼ 23). The OSATS GRS had superior reliability and validity compared to the other assessment tools regardless of training. Although there were no significant differences in reliability between training groups, the RT group trended to improved reliability for all assessment forms at both assessments (Table). Table. Assessment form Pass-fail RT No training VAS RT No training Checklist RT No training GRS RT No training
Inter-rater reliability Initial
Delayed
Intra-rater reliability
0.22 (0.18e0.26) 0.20 (0.17e0.25)
0.16 (0.13e0.20) 0.11 (0.07e0.15)
0.45 (0.32e0.59) 0.41 (0.26e0.57)
0.71 (0.50e0.91) 0.46 (0.27e0.75)
0.54 (0.34e0.80) 0.43 (0.24e0.73)
0.71 (0.64e0.77) 0.62 (0.54e0.70)
0.46 (0.27e0.75) 0.33 (0.17e0.64)
0.42 (0.23e0.71) 0.27 (0.13e0.57)
0.53 (0.43e0.61) 0.46 (0.35e0.56)
0.71 (0.52e0.89) 0.61 (0.41e0.85)
0.64 (0.45e0.86) 0.52 (0.32e0.79)
0.73 (0.67e0.79) 0.66 (0.58e0.73)
CONCLUSIONS: RT did not lead to significant or sustained improvements in technical skill assessments. Trainee evaluations should be performed with standardized assessment tools, such as the OSATS GRS, due to their superior reliability and validity over less standardized methods. Despite our findings, training
remains an important tool to ensure the most effective technical skill evaluations. More effective training methods or repeated sessions may be required for sustained or significant effects. Can Operative Trauma Simulation Eliminate Sex Differences in Self-Confidence? Audrey C Pendleton, MD, Afshin Parsikia, MD, MPH, Sheldon H Teperman, MD, FACS, Melvin E Stone, MD, FACS, Srinivas H Reddy, MD, Edward Chao, MBBS, FACS, Jody M Kaban, MD, FACS Jacobi Medical Center, Bronx, NY INTRODUCTION: Simulation is playing an increasingly significant role in training surgical residents as paradigm shifts in residency have evolved and work hour restrictions have limited exposure to certain clinical areas of general surgery. Simulation can improve technical skill as well enhance self-confidence, the latter being especially important in high acuity situations such as trauma and emergency surgery. Previously published studies have documented sex differences in confidence among surgical residents, with females expressing less confidence in their skills. The goal of our study was to compare confidence levels of senior surgical residents participating in an operative simulation course by sex. METHODS: This was a retrospective review of senior residents’ participation in the Advanced Operative Trauma Management (ATOM) course at our institution from 2007 to 2016. Standard components of the course include pre and post self-confidence questionnaires and knowledge tests, in addition to evaluation of operative skill. Course components were compared by sex using a paired Student’s t-test. RESULTS: There were 247 residents who took the simulation course; 60.7% were male. Female residents scored significantly lower on pre-course self-evaluation (p < 0.001) compared to the men. This difference was eliminated in post self-confidence assessments (p ¼ 0.099). There was no difference between the sexes in knowledge or operative performance, as judged by instructors. Table. Comparison of Participants by Sex Male Female residents, residents, n (%) n (%) Variable (SD) (n ¼ 150) (n ¼ 97) Pre self-evaluation 63.61 (14.85) 56.60 (15.59) Post self-evaluation 99.91 (14.97) 96.87 (12.68) Self-confidence improvement 36.29 (16.68) 40.27 (16.43) Course evaluation 37.13 (6.28) 37.98 (3.29) Instructor post-evaluation 38.23 (6.01) 38.70 (4.67) Pre exam 63.04 (15.22) 60.54 (15.05) Post exam 76.64 (11.49) 75.30 (11.36) Instructor post-evaluation 78.53 (7.19) 76.41 (14.24)
Total, n (%) (n ¼ 247) p Value 60.86 (15.50) <0.001 98.71 (14.16) 0.099 37.85 (16.66) 37.46 (5.32)
0.067 0.219
38.42 (5.52) 62.06 (15.17) 76.11 (11.44)
0.517 0.206 0.369
77.70 (10.56)
0.176
CONCLUSIONS: Female residents had equivalent knowledge and skill when compared to male residents, but expressed less confidence. After simulation, the female residents’ confidence equaled