- Email: [email protected]
A novel rating system effectively differentiates resident surgical performance on an open colectomy animal model
Vol. 207, No. 3S, September 2008
Surgical Forum Abstracts
time (r ⫽ ⫺0.83). Clinicians who successfully managed the airway had a total response time of ⬍ 80 seconds. Time (sec)
AAM Score
Event 1 (dyspnea)
Successful management
Survival
r ⫽⫺0.66
Yes
45.2 ⫾ 17.8 sec
48.4 ⫾ 25.2 sec
No
87.2 ⫾ 31.9 sec
87.1 ⫾ 31.6 sec
p ⬍ 0.05
p ⬍ 0.05
Event 2 (emesis)
r ⫽⫺0.47
Event 3 (hypoxia)
r ⫽⫺0.06
Total response
r ⫽⫺0.83
CONCLUSIONS: Time required to intervene during critical events in the management of an acute airway scenario in an HPS demonstrated a negative association with management scores and survival.
Parkinsonian movement metrics measured with the DaVinci surgical robot correlate with improving surgical coordination John N Afthinos MD, Avinash Burra MS, Marc Attiyeh BS, M Jawad Latif MD, Faiz Y Bhora MD, James J McGinty MD, FACS, Julio A Teixeira MD, FACS, George J Todd MD, FACS, Scott J Belsley MD St Luke’s-Roosevelt Hospital, New York, NY INTRODUCTION: The need to objectively assess progress in surgical training is of increasing importance. We have previously reported a reproducible measure of surgical coordination by applying Parkinsonian movement metrics measured through the DaVinci surgical robot advanced programming interface (API). The aim of this study is to validate the use of these metrics as a measure of improving surgical coordination with a task employing the novel psychomotor challenge of time delay and then to correlate these with visual grading. METHODS: The DaVinci robot API provided motion data during a bead transfer task performed with a 1 second time delay (Ovation Systems, UK). Subjects (n ⫽ 11) of various surgical skill levels ranging from student to attending performed the task and were then randomly assigned to practice on either the robot or a laparoscopic surgery trainer (FLS). Time to task completion, path length, jerk magnitude, jerk metric, jerk cost (JC), jerk cost per unit time and normalized jerk score (NJS) were calculated using MATLAB (MathWorks). The metrics were compared to the scores from a blinded subjective grading session. RESULTS: (See Table) Time (s) All
Pre-training Avg SEM Post-training Avg SEM
Novices Pre-training Avg SEM Post-training Avg SEM Experts Pre-training Avg SEM Post-training Avg SEM
255.8 19.8
p-value 0.0003
JC (m2/s5) 300.58 41.83
p-value 0.009
NJS 7238857 1698051
166.6
180.19
2365467
9.1
14.25
454859
279.9
311.81
8855794
24.1
0.001
55.92
0.031
2271020
173.4
179.87
2713419
11.4
18.87
612015
191.6 16.1
0.056
270.65 38.31
0.047
2927025 494772
148.3
181.03
1437593
11.5
15.36
207397
p-value 0.008
0.013
0.014
S89
CONCLUSIONS: Movement metrics are easily calculated from the DaVinci API. Time to task completion, JC and NJS were all statistically different between virtually all pre- and post-training groups. Additionally, scores correlated with traditional indicators of surgical skill including experience and subjective video grading. Improvement was similar whether subjects practiced with the robot or the FLS. Our experiment suggests a novel application of an established grading system as a quantitative evaluation of progress in surgical training.
A novel rating system effectively differentiates resident surgical performance on an open colectomy animal model Jeremy Lipman MD, Eric Marderstein MD, MPH, Farhad Zeinali MD, Roy Phitayakorn MD, Steve Schomisch BS, Michael Rosen MD, FACS, Jeffrey Marks MD, FACS, Jeffrey L Ponsky MD, FACS, Bradley Champagne MD, Conor Delaney MD, PhD, FACS University Hospitals Case Medical Center, Cleveland, OH INTRODUCTION: Evaluation of trainee operative performance is based on overall impressions and rarely involves objective rating systems. We hypothesized that a new rating system would objectively differentiate between residents of variable surgical skill performing standardized operative steps. METHODS: General surgery residents were instructed using an open porcine colectomy model. The procedure was deconstructed into a series of discrete steps and a standardized reporting form was developed for performance evaluation of each step by a trained observer using Likert scales for different components of the procedure. RESULTS: Resident task performance was divided into thirds for analysis. The table shows that the difference in scores between the Top, Middle and Bottom performing residents was highly significant (p⬍0.001). There was a correlation between time to completion and performance (63 min for Bottom and 50 min for Top), but this trend was not signficant (p⫽0.093). Postgraduate training year 2 residents had lower scores than 4th year residents (p⫽0.007) but there were no differences when compared with 3rd year residents or when comparing 3rd and 4th year residents to each other. Staff physicians blinded to these experimental results rated resident operative performance on the basis of previous clinical evaluations and correctly predicted resident group assignments with 80% accuracy. Group Name
Mean Score
p Value
Bottom third
8.3 ⫾ 1.5
Middle third Top third
14.6 ⫾ 2.1 21.5 ⫾ 1.6
⬍0.001 v. Top 0.0015 v. Middle 0.0002 v. Top
CONCLUSIONS: Using an open colectomy operative training model, this novel operative-step based rating system effectively differentiated resident operative performance into Top, Middle and Bottom groups. This rating system enhances the assessment of resident operative skills, and is an ideal instrument to evaluate the effectiveness of future operative skills-training curricula.
Surgical Forum Abstracts
time (r ⫽ ⫺0.83). Clinicians who successfully managed the airway had a total response time of ⬍ 80 seconds. Time (sec)
AAM Score
Event 1 (dyspnea)
Successful management
Survival
r ⫽⫺0.66
Yes
45.2 ⫾ 17.8 sec
48.4 ⫾ 25.2 sec
No
87.2 ⫾ 31.9 sec
87.1 ⫾ 31.6 sec
p ⬍ 0.05
p ⬍ 0.05
Event 2 (emesis)
r ⫽⫺0.47
Event 3 (hypoxia)
r ⫽⫺0.06
Total response
r ⫽⫺0.83
CONCLUSIONS: Time required to intervene during critical events in the management of an acute airway scenario in an HPS demonstrated a negative association with management scores and survival.
Parkinsonian movement metrics measured with the DaVinci surgical robot correlate with improving surgical coordination John N Afthinos MD, Avinash Burra MS, Marc Attiyeh BS, M Jawad Latif MD, Faiz Y Bhora MD, James J McGinty MD, FACS, Julio A Teixeira MD, FACS, George J Todd MD, FACS, Scott J Belsley MD St Luke’s-Roosevelt Hospital, New York, NY INTRODUCTION: The need to objectively assess progress in surgical training is of increasing importance. We have previously reported a reproducible measure of surgical coordination by applying Parkinsonian movement metrics measured through the DaVinci surgical robot advanced programming interface (API). The aim of this study is to validate the use of these metrics as a measure of improving surgical coordination with a task employing the novel psychomotor challenge of time delay and then to correlate these with visual grading. METHODS: The DaVinci robot API provided motion data during a bead transfer task performed with a 1 second time delay (Ovation Systems, UK). Subjects (n ⫽ 11) of various surgical skill levels ranging from student to attending performed the task and were then randomly assigned to practice on either the robot or a laparoscopic surgery trainer (FLS). Time to task completion, path length, jerk magnitude, jerk metric, jerk cost (JC), jerk cost per unit time and normalized jerk score (NJS) were calculated using MATLAB (MathWorks). The metrics were compared to the scores from a blinded subjective grading session. RESULTS: (See Table) Time (s) All
Pre-training Avg SEM Post-training Avg SEM
Novices Pre-training Avg SEM Post-training Avg SEM Experts Pre-training Avg SEM Post-training Avg SEM
255.8 19.8
p-value 0.0003
JC (m2/s5) 300.58 41.83
p-value 0.009
NJS 7238857 1698051
166.6
180.19
2365467
9.1
14.25
454859
279.9
311.81
8855794
24.1
0.001
55.92
0.031
2271020
173.4
179.87
2713419
11.4
18.87
612015
191.6 16.1
0.056
270.65 38.31
0.047
2927025 494772
148.3
181.03
1437593
11.5
15.36
207397
p-value 0.008
0.013
0.014
S89
CONCLUSIONS: Movement metrics are easily calculated from the DaVinci API. Time to task completion, JC and NJS were all statistically different between virtually all pre- and post-training groups. Additionally, scores correlated with traditional indicators of surgical skill including experience and subjective video grading. Improvement was similar whether subjects practiced with the robot or the FLS. Our experiment suggests a novel application of an established grading system as a quantitative evaluation of progress in surgical training.
A novel rating system effectively differentiates resident surgical performance on an open colectomy animal model Jeremy Lipman MD, Eric Marderstein MD, MPH, Farhad Zeinali MD, Roy Phitayakorn MD, Steve Schomisch BS, Michael Rosen MD, FACS, Jeffrey Marks MD, FACS, Jeffrey L Ponsky MD, FACS, Bradley Champagne MD, Conor Delaney MD, PhD, FACS University Hospitals Case Medical Center, Cleveland, OH INTRODUCTION: Evaluation of trainee operative performance is based on overall impressions and rarely involves objective rating systems. We hypothesized that a new rating system would objectively differentiate between residents of variable surgical skill performing standardized operative steps. METHODS: General surgery residents were instructed using an open porcine colectomy model. The procedure was deconstructed into a series of discrete steps and a standardized reporting form was developed for performance evaluation of each step by a trained observer using Likert scales for different components of the procedure. RESULTS: Resident task performance was divided into thirds for analysis. The table shows that the difference in scores between the Top, Middle and Bottom performing residents was highly significant (p⬍0.001). There was a correlation between time to completion and performance (63 min for Bottom and 50 min for Top), but this trend was not signficant (p⫽0.093). Postgraduate training year 2 residents had lower scores than 4th year residents (p⫽0.007) but there were no differences when compared with 3rd year residents or when comparing 3rd and 4th year residents to each other. Staff physicians blinded to these experimental results rated resident operative performance on the basis of previous clinical evaluations and correctly predicted resident group assignments with 80% accuracy. Group Name
Mean Score
p Value
Bottom third
8.3 ⫾ 1.5
Middle third Top third
14.6 ⫾ 2.1 21.5 ⫾ 1.6
⬍0.001 v. Top 0.0015 v. Middle 0.0002 v. Top
CONCLUSIONS: Using an open colectomy operative training model, this novel operative-step based rating system effectively differentiated resident operative performance into Top, Middle and Bottom groups. This rating system enhances the assessment of resident operative skills, and is an ideal instrument to evaluate the effectiveness of future operative skills-training curricula.