- Email: [email protected]
Establishing Passing Scores for Technical Performance in Surgery: Lessons Learned From a 10-Year Experience of Resident Assessment
S122
Surgical Forum Abstracts
Safety domains
OR Personnel, n¼63 Surgical residents, n¼48 Junior residents, n¼26 Senior residents, n¼22
J Am Coll Surg
Safety culture Teamwork Speaking up
68% 46% 42% 50%
78% 67% 62% 73%
71% 60% 62% 59%
residents into junior (PGY 1-2) and senior (PGY 3-5) levels, all three domains remained lower. Senior and junior residents’ perceptions of safety culture were not significantly different. CONCLUSIONS: Surgical residents’ perceptions of perioperative safety remain suboptimal despite significant emphasis on a systems-based safety approach in residency training. Optimal curricula for surgical residents are lacking; trials demonstrating improvements in safety culture are needed. Establishing Passing Scores for Technical Performance in Surgery: Lessons Learned From a 10-Year Experience of Resident Assessment Sandra de Montbrun, MD, Med, FRCSC, Lisa M Satterthwaite, RPN ORT, Shunne Leung, BSc, Teodor P Grantcharov, MD, PhD, FACS University of Toronto, Toronto, Ontario INTRODUCTION: Competency based assessment is a paradigm shift in surgical training. One of the major challenges is defining a passing score for competent technical performance. The objective of this study was to set standards for the Objective Structured Assessment of Technical Skill (OSATS) exam using the world’s largest technical skills performance database. METHODS: A retrospective analysis of prospectively collected OSATS performance data from 432 PGY 1 surgical residents from 2002-2012 was used for this study. Residents completed a multistation technical skills exam at the end of their PGY 1 year. Three standard setting methodologies were applied to the dataset to set passing scores 1) contrasting groups (CG); 2) borderline group (BG); 3) borderline regression (BR).
Trainee Participation in Pancreaticoduodenectomy is Associated with Reduced Mortality and Equivalent Morbidity Patrick Varley, John R Klune, MD, David A Geller, MD, FACS, Allan Tsung, MD, FACS University of Pittsburgh, Pittsburgh, PA INTRODUCTION: As one of the most complex surgeries performed by general surgeons, pancreaticoduodenectomy (PD) remains an important target for surgical quality improvement. Numerous studies have evaluated the importance of institutional and surgeon-specific factors, but the impact of surgical trainee participation is less clear. METHODS: Patients undergoing PD as the primary procedure for both benign and malignant disease were identified from the 20052012 NSQIP Participant Data Use (PUF) files. Multivariate logistic regression was used to adjust for patient-level risk factors before the impact of trainee participation and trainee level on 30-day morbidity and mortality were assessed. RESULTS: Trainee participation was associated with significantly less mortality, but equivalent overall morbidity. Cases performed by an attending alone were also associated with increased failure to save. Though overall complication rate was equivalent, traineeassisted surgery was associated with increased risk for urinary tract infection and superficial surgical site infection. On subgroup analysis, trainee level (intern, junion, chief, fellow) was not associated with a difference in morbidity or mortality (p ¼ 0.096 and 0.131, respectively). Trainee-assisted surgery was also associated with longer operative time (350 vs 379 minutes, p<0.05) but shorter length of stay (14.4 vs 13.4 days, p < 0.05) (Table).
Outcomes
30-day mortality Any complication Failure to rescue
RESULTS: The three standard setting methodologies produced stable passing scores and passing rates for all stations. A conjunctive scoring method was used; a ‘pass’ was defined as achieving both the overall passing score (the mean passing scores for the combined stations) and passing a minimum of 50% of the stations. General surgery residents were further analyzed to establish overall passing rates for the exam. The three methodologies produced similar overall passing rates (CG, 82%, BG, 87.2%, BR, 82.7%), with the three methodologies being 93% consistent in candidate pass/ fail status. CONCLUSIONS: This is the first large scale study to apply standard setting methodologies to a validated technical skills assessment. Further analysis is looking at the predictive validity of the passing scores with future technical skill performance.
Attending alone (%), n¼1149
Residentassisted (%), n¼9208
4.6
2.6
42
42.6
p Value
Adjusted OR
95% CI
0
0.59
0.44 - 0.81
0.41
1.05
0.93 - 1.2
0
0.56
0.41 - 0.79
9.9
5.6
Any wound complication
19.8
22.8
0.01
1.21
1.04 - 1.41
Superficial SSI
7.1
10.5
0
1.54
1.22 - 1.96
11
0.39
1.09
0.9 - 1.34
Organ Space SSI
10.4
Renal failure (HD)
2.1
1.1
0.02
0.58
0.38 - 0.93
UTI
3.8
5.7
0
1.57
1.16 - 2.17
Cardiac arrest
2.3
1.3
0.01
0.59
0.39 - 0.91
CONCLUSIONS: While PD continues to be an area of quality improvement, efforts to reduce postoperative morbidity and mortality should not include limiting resident participation in these
Surgical Forum Abstracts
Safety domains
OR Personnel, n¼63 Surgical residents, n¼48 Junior residents, n¼26 Senior residents, n¼22
J Am Coll Surg
Safety culture Teamwork Speaking up
68% 46% 42% 50%
78% 67% 62% 73%
71% 60% 62% 59%
residents into junior (PGY 1-2) and senior (PGY 3-5) levels, all three domains remained lower. Senior and junior residents’ perceptions of safety culture were not significantly different. CONCLUSIONS: Surgical residents’ perceptions of perioperative safety remain suboptimal despite significant emphasis on a systems-based safety approach in residency training. Optimal curricula for surgical residents are lacking; trials demonstrating improvements in safety culture are needed. Establishing Passing Scores for Technical Performance in Surgery: Lessons Learned From a 10-Year Experience of Resident Assessment Sandra de Montbrun, MD, Med, FRCSC, Lisa M Satterthwaite, RPN ORT, Shunne Leung, BSc, Teodor P Grantcharov, MD, PhD, FACS University of Toronto, Toronto, Ontario INTRODUCTION: Competency based assessment is a paradigm shift in surgical training. One of the major challenges is defining a passing score for competent technical performance. The objective of this study was to set standards for the Objective Structured Assessment of Technical Skill (OSATS) exam using the world’s largest technical skills performance database. METHODS: A retrospective analysis of prospectively collected OSATS performance data from 432 PGY 1 surgical residents from 2002-2012 was used for this study. Residents completed a multistation technical skills exam at the end of their PGY 1 year. Three standard setting methodologies were applied to the dataset to set passing scores 1) contrasting groups (CG); 2) borderline group (BG); 3) borderline regression (BR).
Trainee Participation in Pancreaticoduodenectomy is Associated with Reduced Mortality and Equivalent Morbidity Patrick Varley, John R Klune, MD, David A Geller, MD, FACS, Allan Tsung, MD, FACS University of Pittsburgh, Pittsburgh, PA INTRODUCTION: As one of the most complex surgeries performed by general surgeons, pancreaticoduodenectomy (PD) remains an important target for surgical quality improvement. Numerous studies have evaluated the importance of institutional and surgeon-specific factors, but the impact of surgical trainee participation is less clear. METHODS: Patients undergoing PD as the primary procedure for both benign and malignant disease were identified from the 20052012 NSQIP Participant Data Use (PUF) files. Multivariate logistic regression was used to adjust for patient-level risk factors before the impact of trainee participation and trainee level on 30-day morbidity and mortality were assessed. RESULTS: Trainee participation was associated with significantly less mortality, but equivalent overall morbidity. Cases performed by an attending alone were also associated with increased failure to save. Though overall complication rate was equivalent, traineeassisted surgery was associated with increased risk for urinary tract infection and superficial surgical site infection. On subgroup analysis, trainee level (intern, junion, chief, fellow) was not associated with a difference in morbidity or mortality (p ¼ 0.096 and 0.131, respectively). Trainee-assisted surgery was also associated with longer operative time (350 vs 379 minutes, p<0.05) but shorter length of stay (14.4 vs 13.4 days, p < 0.05) (Table).
Outcomes
30-day mortality Any complication Failure to rescue
RESULTS: The three standard setting methodologies produced stable passing scores and passing rates for all stations. A conjunctive scoring method was used; a ‘pass’ was defined as achieving both the overall passing score (the mean passing scores for the combined stations) and passing a minimum of 50% of the stations. General surgery residents were further analyzed to establish overall passing rates for the exam. The three methodologies produced similar overall passing rates (CG, 82%, BG, 87.2%, BR, 82.7%), with the three methodologies being 93% consistent in candidate pass/ fail status. CONCLUSIONS: This is the first large scale study to apply standard setting methodologies to a validated technical skills assessment. Further analysis is looking at the predictive validity of the passing scores with future technical skill performance.
Attending alone (%), n¼1149
Residentassisted (%), n¼9208
4.6
2.6
42
42.6
p Value
Adjusted OR
95% CI
0
0.59
0.44 - 0.81
0.41
1.05
0.93 - 1.2
0
0.56
0.41 - 0.79
9.9
5.6
Any wound complication
19.8
22.8
0.01
1.21
1.04 - 1.41
Superficial SSI
7.1
10.5
0
1.54
1.22 - 1.96
11
0.39
1.09
0.9 - 1.34
Organ Space SSI
10.4
Renal failure (HD)
2.1
1.1
0.02
0.58
0.38 - 0.93
UTI
3.8
5.7
0
1.57
1.16 - 2.17
Cardiac arrest
2.3
1.3
0.01
0.59
0.39 - 0.91
CONCLUSIONS: While PD continues to be an area of quality improvement, efforts to reduce postoperative morbidity and mortality should not include limiting resident participation in these