- Email: [email protected]
Moving skills training closer to application: In-rotation skills curriculum is feasible and effective
Accepted Manuscript Moving skills training closer to application: In-rotation skills curriculum is feasible and effective Stephen Haggerty, Tomokazu Kishiki, Michael Ujiki, Chi Wang, Nancy Schindler PII:
S0002-9610(17)30713-4
DOI:
10.1016/j.amjsurg.2017.10.043
Reference:
AJS 12614
To appear in:
The American Journal of Surgery
Received Date: 11 May 2017 Revised Date:
16 September 2017
Accepted Date: 19 October 2017
Please cite this article as: Haggerty S, Kishiki T, Ujiki M, Wang C, Schindler N, Moving skills training closer to application: In-rotation skills curriculum is feasible and effective, The American Journal of Surgery (2017), doi: 10.1016/j.amjsurg.2017.10.043. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Stephen Haggerty, MD1 Tomokazu Kishiki, MD1 Michael Ujiki, MD1 Chi Wang1 Nancy Schindler, MD1 of Surgery NorthShore University HealthSystem, 2650 Ridge Ave, Evanston, IL 60201, USA
SC
1Department
AC C
EP
TE D
M AN U
Corresponding Author: Stephen Haggerty, MD 777 Park Avenue West, Ste 2640 Highland Park, IL 60045 847-570-1700 [email protected]
RI PT
Moving skills training closer to application: In-rotation skills curriculum is feasible and effective
ACCEPTED MANUSCRIPT
Abstract: Purpose: Using simulation can help surgical trainees acquire surgical skills but at the expense of clinical learning time. We postulate an in-rotation skills curriculum is feasible and minimizes time away from clinical experiences.
RI PT
Methods: Surgical residents (PGY2-5) were allotted two hours of weekly protected time for rotation specific simulation modules that included assessment, mentoring, and practice. Between September 2015 and February 2016 performance data was collected and participants were surveyed.
SC
Results: Completion rates of 87-100% were achieved and post-test scores improved significantly, indicating improved performance. The survey (29/30 RR) revealed that 81.5% felt 2 hours a week was ‘just right’ and 79.3% agreed or strongly agreed the in-rotation aspect was a benefit. Improved confidence in the OR was reported by 86.2% of residents Intra-operative skill was self-assessed as improved in 79.3%.
AC C
EP
TE D
M AN U
Conclusion: In-rotation skills curriculum with high completion rates is feasible and allows training in close proximity to clinical application. Performance in the simulated environment significantly improved with corresponding improvements in confidence and self-assessed skill in the operating room.
ACCEPTED MANUSCRIPT
M AN U
SC
RI PT
Introduction: Surgical trainees must acquire complex technical skills so that they may operate independently upon graduation. During the five-year general surgery residency, trainees are expected to reach a certain level of competency for many operations. To do this it may take up to 10,000 hours of deliberate practice to develop expert skill [1, 2]. However, residents appear to be given less autonomy to learn during surgery than in the past and thus limited opportunity to “practice” [1, 3, 4]. There are several reasons for this including: An increased focus on patient outcomes, a desire to increase efficiency and finish operations expeditiously, and expectations of attending surgeon involvement by the hospital and patients [4]. To offset these difficulties in educating surgical residents, simulation training began early in this millennium and by 2008 the Residency Review Committee in surgery mandated that all programs have a simulation curriculum. Driven by demand and improved technology, virtual reality simulators have become readily available with modules for basic skills and specific procedures. In addition, many versions of video (box) trainers, and inanimate and animate bench models have been developed and provide an opportunity to learn and practice surgical skills. Abundant evidence now exists that all these mechanisms allow residents to learn and practice away from live patients, and can improve skill performance, increase confidence in performing the procedure and even lead to improved performance in the operating room [5-11].
EP
TE D
It is clear that resident simulation training is here to stay. However, it is not clear how much time should be allotted and when the training should take place. In the past residents have utilized off- hours time, part or whole day pull out sessions, and open labs available for the residents to use. Unfortunately these can be disruptive to clinical activities and may not offer reliable skills training. At our institution, pulling residents out of rotations for skills labs was often met with complaints from faculty and residents alike that they were missing out on important clinical activities or cases. In addition, depending on individual schedules, a specific skills lab may have been temporally separated from actual clinical experience by six months or more. With this in mind, we developed an in-rotation skills curriculum using rotation-relevant procedure based modules and postulated that this would be feasible and would maximize learning while minimizing time away from valuable clinical experiences.
AC C
Methods: Surgical residents (PGY2-5) assigned to four week clinical rotations at NorthShore University HealthSystem (NSUHS) were included in the curriculum and thus in the study (Table 1.). For each four week rotation, two hours of weekly protected time was allotted at a time when valued clinical experiences were least likely to occur. The Team Education Coordinator of each rotation chose the weekly time slot. Residents were assigned procedure based modules simulating the cases they would see on their rotations and thus corresponding to their level of training (Table 1). The curriculum for each module included a pre-test in week one, instruction and mentoring by an attending in week two, and practice time in weeks three and four, concluding with a post-test. The pre and post-tests are multi-step assessment tools that were either designed and validated by our group or have been published in the literature. For example, the laparoscopic ventral hernia module uses Global Operative Assessment of
ACCEPTED MANUSCRIPT
SC
RI PT
Laparoscopic Skills-Incisional Hernia assessment tool [12]. The mentoring session included oneon-one instruction by an attending surgeon for about one hour. The residents performed multiple practice sessions on the modules but the exact number was not recorded. Between September 2015 through March 2017 performance data was collected in the form of pre and post-tests as scored by an independent trained observer and also self assessment scores completed by the resident. The same rater performed the assessments throughout the study period. In addition, after completion of their first module at our institution, the participants were surveyed within six weeks after completion of the curriculum (Figure 1). The nonvalidated surveys were designed by the authors and not specific to the modules. Therefore, residents were only surveyed once about the in-rotation curriculum even if they were multiple different rotations and did more than one module at our institution during the study period.
M AN U
Statistical analysis: Data were summarized using descriptive statistics (mean and standard deviation for continuous variables; frequency and percentage for categorical variables). Pre- and post-test scores were compared using paired t-test. For each module, a passing score was pre-established to determine the passing rate of the residents. Pre- and post-test passing rates were then compared using McNemar paired chi-square test. All data were analyzed using SAS 9.4 (SAS Inc., Cary, NC) and a p value <0.05 was considered statistically significant.
AC C
EP
TE D
Results: Over 18 months 60 residents were eligible for the curriculum on five rotations at NSUHS. It should be noted there were not 60 different residents since some residents rotated on two or three rotations at our institution over the study period. Eight in-rotation skills modules made up the entire curriculum with four being offered on the Minimally Invasive Surgery (MIS) service and one on each of the others (Table 1). A 100% completion rate was achieved on 6/8 modules, with 92% on the other two (table 2). All modules used procedure specific skills assessment tools with unique passing scores that were set at 80% of average expert scores. Rater observed and self assessed Pre-test scores were compared to post-test scores. The rater observed post-test scores were significantly higher on all modules, indicating improved performance (Table 2). The self assessments showed similar significant improvements on all modules except advanced endoscopy. All but one module achieved a 100% passing rate (exception was Gastric Bypass, 83%) at post-test as compared to the significantly lower passing rates observed at pre-test (Table 2). The survey (29/31 responded) revealed that 81.5% felt 2 hours a week was ‘just right’, while 11% though it was too much and 7.5% thought it was not enough. 86.2% agreed or strongly agreed that an in-rotation skills curriculum was a valuable addition to their educational experience. When asked if in-rotation skills experiences provide benefits that are unique when compared to out-of-rotation skills labs or pure skills rotations, 79.3% agreed or strongly agreed. Improved confidence in the OR was reported by 86.2% of residents (moderate in 51.7%, significant in 27.6% and very significant in 6.9%). Intra-operative skill was self-assessed as
ACCEPTED MANUSCRIPT
improved in 79.3% (moderate in 44.8%, significant in 27.6% and very significant in 6.9%) of the total residents.
SC
RI PT
Discussion: Surgical simulation training is an important part of general surgery residency. It emerged early in the millennium as a way to acquire and practice laparoscopic skills and has subsequently become a necessary adjunct to the traditional model of learning on live patients. With workhour restrictions, slightly lower case volume [13] and societal pressure for attendings to “over supervise” in the operating room [1, 4], there is some evidence that residents are less prepared to go into independent practice [1, 14]. To counter this trend, simulation training facilitates deliberate practice and many repetitions, which can shorten the learning curve to obtain proficiency in performing a case [15].
EP
TE D
M AN U
Both video box trainers and virtual reality simulators (VRS) have been extensively studied over the last several years. Two randomized controlled trials have compared a curriculum using VRS for skills training versus conventional surgical training and they both showed that performance was better after the VRS curriculum [6, 7]. Furthermore, a systematic review of skills transfer after simulation training concluded that skills acquired by simulation-based training seem to be transferable to the operative setting. But the studies included in the review were of variable quality and did not use comparable simulation-based training methodologies, which limited the strength of the conclusions. They believe that more studies are required to strengthen the evidence base regarding which simulation should become a part of surgical training programs [16]. In addition to laparoscopic trainers, simulators of open operations are prevalent and take the form of live animals, cadavers and bench models. Multiple studies have confirmed the benefit of open simulator training for surgical residents [17]. All residency programs are supposed to offer access simulation training. Unfortunately several barriers exist including staff, cost and resident time, which is limited to 80 total clinical hours per week including simulation training [18].
AC C
There is no consensus on the best way to incorporate simulation into a surgical training program. Options include occasional pull out sessions of either all day or half day skills training, open labs to allow residents to practice on their own time, simulation rotations or dedicated simulation training time on short intervals during clinical rotations. It has been proposed that multiple short practice sessions (interval practice) result in better skill acquisition than one long session (massed practice) [19]. In addition, half or all day pull out sessions can interrupt valuable clinical experiences. Finally, Gostlow has shown that providing unrestricted access to simulator equipment and allowing voluntary participation in laparoscopic skills training is not effective [20] and a survey by Shetty found that the most important factor influencing resident participation in a skills curriculum was mandatory protected time in a simulation environment [21]. We propose that the ideal resident skills curriculum allows for learning, mentoring and practice of specific procedures in close proximity to operating on live patients. Therefore, we designed
ACCEPTED MANUSCRIPT
M AN U
SC
RI PT
our skills curriculum using procedure specific modules and allotted two hours of protected time each week for residents on every clinical service. The curriculum included a pre-test administered by a trained, experienced rater, a one hour mentoring session by an attending surgeon, dedicated practice time and then a post-test. As you can see in table one, the simulated procedure was appropriate for the PGY level and coordinated with cases they would be performing on that specific rotation. Most rotations involved one training module, however, the MIS rotation included four modules since some of these were in place prior to expanding our simulation curriculum to all rotations. The resident schedule at our institution is variable, with some residents rotating for only two weeks. Since they could not complete the four week curriculum with adequate mentoring and practice time, they were not offered the simulation curriculum and therefore not included in our data. In addition, we did not include fellows in our skills curriculum but do have plans to include them in mentoring. This explains why the various rotations had differing numbers of participants. We scheduled mentoring sessions with attending surgeons in advance and they were about one hour of one-on-one time. These sessions were considered a highlight of the curriculum for the trainee as the attending helped break down the steps of the operation and give tips, tricks and feedback about the performance of the simulated case.
EP
TE D
We have shown that this structured skills curriculum is feasible, obtaining 100% completion rates on most modules. Not surprisingly resident’s performance on the skills assessment significantly improved from pre to post test on all modules. Our pre-test pass rates were relatively low, but this is not alarming since the passing scores were set at 80% of expert scores using assessment tools that had been previously validated in the literature or validated by our group using experienced attending surgeons as experts. In addition, the pretest is taken by the examinee with some warm up but no other instruction on the simulator or the operation. This may contribute to the low pre test scores. More importantly these assessments break down the steps of the operations and the low passing scores on the pre-tests may be a reflection that residents are not routinely taught the major steps in surgical operations and these steps may not be covered in their current academic setting.
AC C
Those who completed the curriculum were surveyed after completing their first exposure to our in-rotation curriculum. Several residents rotated on two or even three services during the 18 months but were not surveyed more than once. Their self-assessed skill improved on all but one module, while 86% of the trainees reported improved confidence in the operating room and almost 80% believe their skill improved in the operating room. This indicates our curriculum enhances the resident’s educational experience and likely improves their surgical skill. The ultimate goal of any simulation program is to improve resident’s skill in the operating room. However, this project did not include operative assessments of live surgery. To assess the success of our curriculum, several options exist. Random operating room assessments of resident’s skill when performing the cases we are simulating may give some information about skill acquisition. A more significant conclusion could be made if we obtained pre-curriculum and post-curriculum assessments of resident’s operative performance from attending surgeons.
ACCEPTED MANUSCRIPT
SC
RI PT
However, the ideal study would involve a randomized trial comparing a group of residents who received a simulation curriculum to one that did not. Unfortunately this would deprive one group of valuable simulation education. We plan to continue to evaluate the effectiveness of this curriculum with intra-operative assessment of these procedural skills to determine if the curriculum results in comparable performance in the operating room and higher degrees of resident autonomy than is seen without an in-rotation skills program. This study is also limited by a lack of comparison to prior methods of pulling residents out of rotations for non-rotation specific labs. Our anecdotal experience along with our resident survey suggests that there are significant advantages to the current approach. We strongly believe this approach improves learning, enhances transferability due to the temporal relationship to application in the operating room, and minimizes disruption of clinically important experiences. We therefore do not intend to perform a comparison with our prior approach.
M AN U
Conclusion: An In-rotation skills curriculum with high completion rates is feasible and allows interval training in close proximity to application in live surgery. Performance in the simulated environment significantly improved with corresponding improvements in confidence and selfassessed skill in the operating room. Further research is needed to objectively confirm transferability of skills from lab to operating room.
References:
4. 5. 6. 7. 8. 9.
10. 11.
TE D
3.
EP
2.
Soper, N.J. and D.A. DaRosa, Presidential address: Engendering operative autonomy in surgical training. Surgery, 2014. 156(4): p. 745-51. Ericsson KA, K.R., Tesch-Romer C. , The role of deliberate practice in the acquisition of expert performance. Psychol Rev, 1993. 100: p. 393-4. Kantor, O., et al., Implementing a resident acute care surgery service: Improving resident education and patient care. Surgery, 2017. 161(3): p. 876-883. Teman, N.R., et al., Entrustment of general surgery residents in the operating room: factors contributing to provision of resident autonomy. J Am Coll Surg, 2014. 219(4): p. 778-87. Aggarwal, R., et al., Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg, 2007. 246(5): p. 771-9. Grantcharov, T.P., et al., Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg, 2004. 91(2): p. 146-50. Seymour, N.E., et al., Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg, 2002. 236(4): p. 458-63; discussion 463-4. Wohaibi, E.M., et al., Surgical resident performance on a virtual reality simulator correlates with operating room performance. J Surg Res, 2010. 160(1): p. 67-72. Gravante, G. and D. Venditti, A systematic review on low-cost box models to achieve basic and advanced laparoscopic skills during modern surgical training. Surg Laparosc Endosc Percutan Tech, 2013. 23(2): p. 109-20. Seymour, N.E., VR to OR: a review of the evidence that virtual reality simulation improves operating room performance. World J Surg, 2008. 32(2): p. 182-8. Gurusamy, K., et al., Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. Br J Surg, 2008. 95(9): p. 1088-97.
AC C
1.
ACCEPTED MANUSCRIPT
18. 19. 20.
21.
RI PT
17.
SC
16.
M AN U
15.
TE D
14.
EP
13.
Vaillancourt, M., et al., GOALS-incisional hernia: a valid assessment of simulated laparoscopic incisional hernia repair. Surg Innov, 2011. 18(1): p. 48-54. Drake, F.T., et al., The general surgery chief resident operative experience: 23 years of national ACGME case logs. JAMA Surg, 2013. 148(9): p. 841-7. Mattar, S.G., et al., General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors. Ann Surg, 2013. 258(3): p. 440-9. De Win, G., et al., An evidence-based laparoscopic simulation curriculum shortens the clinical learning curve and reduces surgical adverse events. Adv Med Educ Pract, 2016. 7: p. 357-70. Sturm, L.P., et al., A systematic review of skills transfer after surgical simulation training. Ann Surg, 2008. 248(2): p. 166-79. Davies, J., M. Khatib, and F. Bello, Open surgical simulation--a review. J Surg Educ, 2013. 70(5): p. 618-27. Pentiak, P.A., et al., Barriers to adoption of the surgical resident skills curriculum of the American College of Surgeons/Association of Program Directors in Surgery. Surgery, 2013. 154(1): p. 23-8. Andersen, S.A., et al., Cognitive load in distributed and massed practice in virtual reality mastoidectomy simulation. Laryngoscope, 2016. 126(2): p. E74-9. Gostlow, H., et al., Systematic Review of Voluntary Participation in Simulation-Based Laparoscopic Skills Training: Motivators and Barriers for Surgical Trainee Attendance. J Surg Educ, 2017. 74(2): p. 306-318. Shetty, S., et al., Perceptions, training experiences, and preferences of surgical residents toward laparoscopic simulation training: a resident survey. J Surg Educ, 2014. 71(5): p. 727-33.
AC C
12.
ACCEPTED MANUSCRIPT
Table 1. Summary of rotation specific simulation modules
Minimally Invasive Surgery Minimally Invasive Surgery Minimally Invasive Surgery Minimally Invasive Surgery
RI PT
Description Virtual Reality Simulator
Virtual Reality Simulator
SC
Laparoscopic Sigmoidectomy
M AN U
Surgical Oncology
Laparoscopic Ventral Hernia
Virtual Reality Simulator
Thyroid Ultrasound and Biopsy
Bench Training Model
Laparoscopic Gastric Bypass
Virtual Reality and animate model
Laparoscopic Heller Myotomy
Box Trainer with animate model
TE D
Colorectal Surgery
Module Laparoscopic Cholecystectomy
Laparoscopic Nissen Fundoplication and Hiatal Hernia Repair Advanced Endoscopy
EP
Highland Park General and Minimally Invasive Surgery
Resident Level 1-PGY4 2-PGY3 9- PGY2 2-PGY5 6-PGY4 7-PGY3 7-PGY5 5-PGY4 1-PGY3 6-PGY3 2-PGY2 5-PGY5 7-PGY4 5-PGY5 7-PGY4 5-PGY5 7-PGY4 5-PGY5 7-PGY4
AC C
Rotation Glenbrook General Surgery
Box Trainer with animate model Box Trainer with animate model
ACCEPTED MANUSCRIPT
Table 2. Pre- and post-test scores and passing rates comparison by simulation modules Self-assessment by residents Pre-test Post-test P value Score Score 32.5±8.2 41.6±3.9 0.002
Passing Score 38
21.1±3.6
28.8±2.5
<.0001
25.4±3.6
30.1±2.1
25
Module Laparoscopic Cholecystectomy Laparoscopic Ventral Hernia
Completion Rate 12/12 (100%) 15/15 (100%)
Laparoscopic Sigmoidectomy
12/13 (92%)
26.0±2.6
36.3±3.3
<.0001
28.8±7.1
35.6±4.0
0.004
Thyroid Ultrasound and Biopsy Laparoscopic Gastric Bypass Laparoscopic Heller Myotomy Laparoscopic Fundoplication and Hiatal Hernia Repair Advanced Endoscopy
8/8 (100%) 12/12 (100%) 12/12 (100%) 12/12 (100%)
22.4±1.5
35.8±1.9
<.0001
22.1±2.9
35.3±1.9
45.8±5.1
55.3±6.7
0.0002
14.9±2.2
19.6±1.8
<.0001
16.8±2.5
21.4±1.2
<.0001
11/12 (92%)
16.9±2.3
20.2±1.7
0.0018
Pre-test Rate 0/12 (0%) 3/15 (20%)
Post-test Rate 12/12 (100%) 15/15 (100%)
25.6
8/13 (62%)
12/12 (100%)
0.0163
<.0001
30
0/8 (0%) 4/12 (33%) 5/12 (42%) 8/12 (67%)
8/8 (100%) 10/12 (83%) 12/12 (100%) 12/12 (100%)
0.0002
11/12 (92%)
12/12 (100%)
0.9999
M AN U
SC
<.0001
46.6±13.6
54.8±13.5
0.016
48
16.0±4.1
18.9±4.1
0.046
16
16.7±3.7
19.2±2.4
0.008
16
18.4±2.9
19.3±3.4
0.620
14.8
TE D
EP AC C
Passing rate
RI PT
Assessment by independent trained rater Pre-test Post-test P value Score Score 28.3±5.9 42.2±3.2 <.0001
P value <.0001 <.0001
0.0361 0.0046 0.0285
ACCEPTED MANUSCRIPT
Figure 1. Post rotation resident survey
Not enough time
Just the right amount of time
RI PT
1. Two hours per week were allotted for skills training on this rotation. Was this amount of time? Too much time
Strongly disagree
Disagree
Neutral
SC
2. In-rotation skills training modules were a valuable addition to my educational experience. Agree Strongly agree
Not at all
M AN U
3. In-rotation skills experiences improved my confidence in the operating room. A little
Moderately
Significantly Very significantly
4. In-rotation skills experiences improved my surgical skill in the operating room. A little
Moderately
TE D
Not at all
Significantly
Very significantly
5. In-rotation skills experiences provide benefits that are unique when compared to out-ofrotation skills labs or pure skills rotations.
EP
Disagree
AC C
Strongly disagree
Neutral
Agree
Strongly agree
ACCEPTED MANUSCRIPT
RI PT
We designed a resident skills curriculum that allows for learning, mentoring and practice of specific procedures in close proximity to operating on live patients. The curriculum based on case specific simulation modules included a pre-test administered by a trained, experienced rater, a one hour mentoring session by an attending surgeon, dedicated practice time and then a post-test. We have shown that this structured skills curriculum is feasible, obtaining 100% completion rates on most modules. It is also effective as resident’s performance on the skills assessment significantly improved from pre to post test on all modules.
SC
Resident self-assessed skill improved on all but one module, while 86% of the trainees reported improved confidence in the operating room and almost 80% believe their skill improved in the operating room. This indicates our curriculum enhances the resident’s educational experience and likely improves their surgical skill.
AC C
EP
TE D
M AN U
An In-rotation skills curriculum with high completion rates is feasible and allows interval training in close proximity to application in live surgery. Performance in the simulated environment significantly improved with corresponding improvements in confidence and selfassessed skill in the operating room. Further research is needed to objectively confirm transferability of skills from lab to operating room.
S0002-9610(17)30713-4
DOI:
10.1016/j.amjsurg.2017.10.043
Reference:
AJS 12614
To appear in:
The American Journal of Surgery
Received Date: 11 May 2017 Revised Date:
16 September 2017
Accepted Date: 19 October 2017
Please cite this article as: Haggerty S, Kishiki T, Ujiki M, Wang C, Schindler N, Moving skills training closer to application: In-rotation skills curriculum is feasible and effective, The American Journal of Surgery (2017), doi: 10.1016/j.amjsurg.2017.10.043. This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Stephen Haggerty, MD1 Tomokazu Kishiki, MD1 Michael Ujiki, MD1 Chi Wang1 Nancy Schindler, MD1 of Surgery NorthShore University HealthSystem, 2650 Ridge Ave, Evanston, IL 60201, USA
SC
1Department
AC C
EP
TE D
M AN U
Corresponding Author: Stephen Haggerty, MD 777 Park Avenue West, Ste 2640 Highland Park, IL 60045 847-570-1700 [email protected]
RI PT
Moving skills training closer to application: In-rotation skills curriculum is feasible and effective
ACCEPTED MANUSCRIPT
Abstract: Purpose: Using simulation can help surgical trainees acquire surgical skills but at the expense of clinical learning time. We postulate an in-rotation skills curriculum is feasible and minimizes time away from clinical experiences.
RI PT
Methods: Surgical residents (PGY2-5) were allotted two hours of weekly protected time for rotation specific simulation modules that included assessment, mentoring, and practice. Between September 2015 and February 2016 performance data was collected and participants were surveyed.
SC
Results: Completion rates of 87-100% were achieved and post-test scores improved significantly, indicating improved performance. The survey (29/30 RR) revealed that 81.5% felt 2 hours a week was ‘just right’ and 79.3% agreed or strongly agreed the in-rotation aspect was a benefit. Improved confidence in the OR was reported by 86.2% of residents Intra-operative skill was self-assessed as improved in 79.3%.
AC C
EP
TE D
M AN U
Conclusion: In-rotation skills curriculum with high completion rates is feasible and allows training in close proximity to clinical application. Performance in the simulated environment significantly improved with corresponding improvements in confidence and self-assessed skill in the operating room.
ACCEPTED MANUSCRIPT
M AN U
SC
RI PT
Introduction: Surgical trainees must acquire complex technical skills so that they may operate independently upon graduation. During the five-year general surgery residency, trainees are expected to reach a certain level of competency for many operations. To do this it may take up to 10,000 hours of deliberate practice to develop expert skill [1, 2]. However, residents appear to be given less autonomy to learn during surgery than in the past and thus limited opportunity to “practice” [1, 3, 4]. There are several reasons for this including: An increased focus on patient outcomes, a desire to increase efficiency and finish operations expeditiously, and expectations of attending surgeon involvement by the hospital and patients [4]. To offset these difficulties in educating surgical residents, simulation training began early in this millennium and by 2008 the Residency Review Committee in surgery mandated that all programs have a simulation curriculum. Driven by demand and improved technology, virtual reality simulators have become readily available with modules for basic skills and specific procedures. In addition, many versions of video (box) trainers, and inanimate and animate bench models have been developed and provide an opportunity to learn and practice surgical skills. Abundant evidence now exists that all these mechanisms allow residents to learn and practice away from live patients, and can improve skill performance, increase confidence in performing the procedure and even lead to improved performance in the operating room [5-11].
EP
TE D
It is clear that resident simulation training is here to stay. However, it is not clear how much time should be allotted and when the training should take place. In the past residents have utilized off- hours time, part or whole day pull out sessions, and open labs available for the residents to use. Unfortunately these can be disruptive to clinical activities and may not offer reliable skills training. At our institution, pulling residents out of rotations for skills labs was often met with complaints from faculty and residents alike that they were missing out on important clinical activities or cases. In addition, depending on individual schedules, a specific skills lab may have been temporally separated from actual clinical experience by six months or more. With this in mind, we developed an in-rotation skills curriculum using rotation-relevant procedure based modules and postulated that this would be feasible and would maximize learning while minimizing time away from valuable clinical experiences.
AC C
Methods: Surgical residents (PGY2-5) assigned to four week clinical rotations at NorthShore University HealthSystem (NSUHS) were included in the curriculum and thus in the study (Table 1.). For each four week rotation, two hours of weekly protected time was allotted at a time when valued clinical experiences were least likely to occur. The Team Education Coordinator of each rotation chose the weekly time slot. Residents were assigned procedure based modules simulating the cases they would see on their rotations and thus corresponding to their level of training (Table 1). The curriculum for each module included a pre-test in week one, instruction and mentoring by an attending in week two, and practice time in weeks three and four, concluding with a post-test. The pre and post-tests are multi-step assessment tools that were either designed and validated by our group or have been published in the literature. For example, the laparoscopic ventral hernia module uses Global Operative Assessment of
ACCEPTED MANUSCRIPT
SC
RI PT
Laparoscopic Skills-Incisional Hernia assessment tool [12]. The mentoring session included oneon-one instruction by an attending surgeon for about one hour. The residents performed multiple practice sessions on the modules but the exact number was not recorded. Between September 2015 through March 2017 performance data was collected in the form of pre and post-tests as scored by an independent trained observer and also self assessment scores completed by the resident. The same rater performed the assessments throughout the study period. In addition, after completion of their first module at our institution, the participants were surveyed within six weeks after completion of the curriculum (Figure 1). The nonvalidated surveys were designed by the authors and not specific to the modules. Therefore, residents were only surveyed once about the in-rotation curriculum even if they were multiple different rotations and did more than one module at our institution during the study period.
M AN U
Statistical analysis: Data were summarized using descriptive statistics (mean and standard deviation for continuous variables; frequency and percentage for categorical variables). Pre- and post-test scores were compared using paired t-test. For each module, a passing score was pre-established to determine the passing rate of the residents. Pre- and post-test passing rates were then compared using McNemar paired chi-square test. All data were analyzed using SAS 9.4 (SAS Inc., Cary, NC) and a p value <0.05 was considered statistically significant.
AC C
EP
TE D
Results: Over 18 months 60 residents were eligible for the curriculum on five rotations at NSUHS. It should be noted there were not 60 different residents since some residents rotated on two or three rotations at our institution over the study period. Eight in-rotation skills modules made up the entire curriculum with four being offered on the Minimally Invasive Surgery (MIS) service and one on each of the others (Table 1). A 100% completion rate was achieved on 6/8 modules, with 92% on the other two (table 2). All modules used procedure specific skills assessment tools with unique passing scores that were set at 80% of average expert scores. Rater observed and self assessed Pre-test scores were compared to post-test scores. The rater observed post-test scores were significantly higher on all modules, indicating improved performance (Table 2). The self assessments showed similar significant improvements on all modules except advanced endoscopy. All but one module achieved a 100% passing rate (exception was Gastric Bypass, 83%) at post-test as compared to the significantly lower passing rates observed at pre-test (Table 2). The survey (29/31 responded) revealed that 81.5% felt 2 hours a week was ‘just right’, while 11% though it was too much and 7.5% thought it was not enough. 86.2% agreed or strongly agreed that an in-rotation skills curriculum was a valuable addition to their educational experience. When asked if in-rotation skills experiences provide benefits that are unique when compared to out-of-rotation skills labs or pure skills rotations, 79.3% agreed or strongly agreed. Improved confidence in the OR was reported by 86.2% of residents (moderate in 51.7%, significant in 27.6% and very significant in 6.9%). Intra-operative skill was self-assessed as
ACCEPTED MANUSCRIPT
improved in 79.3% (moderate in 44.8%, significant in 27.6% and very significant in 6.9%) of the total residents.
SC
RI PT
Discussion: Surgical simulation training is an important part of general surgery residency. It emerged early in the millennium as a way to acquire and practice laparoscopic skills and has subsequently become a necessary adjunct to the traditional model of learning on live patients. With workhour restrictions, slightly lower case volume [13] and societal pressure for attendings to “over supervise” in the operating room [1, 4], there is some evidence that residents are less prepared to go into independent practice [1, 14]. To counter this trend, simulation training facilitates deliberate practice and many repetitions, which can shorten the learning curve to obtain proficiency in performing a case [15].
EP
TE D
M AN U
Both video box trainers and virtual reality simulators (VRS) have been extensively studied over the last several years. Two randomized controlled trials have compared a curriculum using VRS for skills training versus conventional surgical training and they both showed that performance was better after the VRS curriculum [6, 7]. Furthermore, a systematic review of skills transfer after simulation training concluded that skills acquired by simulation-based training seem to be transferable to the operative setting. But the studies included in the review were of variable quality and did not use comparable simulation-based training methodologies, which limited the strength of the conclusions. They believe that more studies are required to strengthen the evidence base regarding which simulation should become a part of surgical training programs [16]. In addition to laparoscopic trainers, simulators of open operations are prevalent and take the form of live animals, cadavers and bench models. Multiple studies have confirmed the benefit of open simulator training for surgical residents [17]. All residency programs are supposed to offer access simulation training. Unfortunately several barriers exist including staff, cost and resident time, which is limited to 80 total clinical hours per week including simulation training [18].
AC C
There is no consensus on the best way to incorporate simulation into a surgical training program. Options include occasional pull out sessions of either all day or half day skills training, open labs to allow residents to practice on their own time, simulation rotations or dedicated simulation training time on short intervals during clinical rotations. It has been proposed that multiple short practice sessions (interval practice) result in better skill acquisition than one long session (massed practice) [19]. In addition, half or all day pull out sessions can interrupt valuable clinical experiences. Finally, Gostlow has shown that providing unrestricted access to simulator equipment and allowing voluntary participation in laparoscopic skills training is not effective [20] and a survey by Shetty found that the most important factor influencing resident participation in a skills curriculum was mandatory protected time in a simulation environment [21]. We propose that the ideal resident skills curriculum allows for learning, mentoring and practice of specific procedures in close proximity to operating on live patients. Therefore, we designed
ACCEPTED MANUSCRIPT
M AN U
SC
RI PT
our skills curriculum using procedure specific modules and allotted two hours of protected time each week for residents on every clinical service. The curriculum included a pre-test administered by a trained, experienced rater, a one hour mentoring session by an attending surgeon, dedicated practice time and then a post-test. As you can see in table one, the simulated procedure was appropriate for the PGY level and coordinated with cases they would be performing on that specific rotation. Most rotations involved one training module, however, the MIS rotation included four modules since some of these were in place prior to expanding our simulation curriculum to all rotations. The resident schedule at our institution is variable, with some residents rotating for only two weeks. Since they could not complete the four week curriculum with adequate mentoring and practice time, they were not offered the simulation curriculum and therefore not included in our data. In addition, we did not include fellows in our skills curriculum but do have plans to include them in mentoring. This explains why the various rotations had differing numbers of participants. We scheduled mentoring sessions with attending surgeons in advance and they were about one hour of one-on-one time. These sessions were considered a highlight of the curriculum for the trainee as the attending helped break down the steps of the operation and give tips, tricks and feedback about the performance of the simulated case.
EP
TE D
We have shown that this structured skills curriculum is feasible, obtaining 100% completion rates on most modules. Not surprisingly resident’s performance on the skills assessment significantly improved from pre to post test on all modules. Our pre-test pass rates were relatively low, but this is not alarming since the passing scores were set at 80% of expert scores using assessment tools that had been previously validated in the literature or validated by our group using experienced attending surgeons as experts. In addition, the pretest is taken by the examinee with some warm up but no other instruction on the simulator or the operation. This may contribute to the low pre test scores. More importantly these assessments break down the steps of the operations and the low passing scores on the pre-tests may be a reflection that residents are not routinely taught the major steps in surgical operations and these steps may not be covered in their current academic setting.
AC C
Those who completed the curriculum were surveyed after completing their first exposure to our in-rotation curriculum. Several residents rotated on two or even three services during the 18 months but were not surveyed more than once. Their self-assessed skill improved on all but one module, while 86% of the trainees reported improved confidence in the operating room and almost 80% believe their skill improved in the operating room. This indicates our curriculum enhances the resident’s educational experience and likely improves their surgical skill. The ultimate goal of any simulation program is to improve resident’s skill in the operating room. However, this project did not include operative assessments of live surgery. To assess the success of our curriculum, several options exist. Random operating room assessments of resident’s skill when performing the cases we are simulating may give some information about skill acquisition. A more significant conclusion could be made if we obtained pre-curriculum and post-curriculum assessments of resident’s operative performance from attending surgeons.
ACCEPTED MANUSCRIPT
SC
RI PT
However, the ideal study would involve a randomized trial comparing a group of residents who received a simulation curriculum to one that did not. Unfortunately this would deprive one group of valuable simulation education. We plan to continue to evaluate the effectiveness of this curriculum with intra-operative assessment of these procedural skills to determine if the curriculum results in comparable performance in the operating room and higher degrees of resident autonomy than is seen without an in-rotation skills program. This study is also limited by a lack of comparison to prior methods of pulling residents out of rotations for non-rotation specific labs. Our anecdotal experience along with our resident survey suggests that there are significant advantages to the current approach. We strongly believe this approach improves learning, enhances transferability due to the temporal relationship to application in the operating room, and minimizes disruption of clinically important experiences. We therefore do not intend to perform a comparison with our prior approach.
M AN U
Conclusion: An In-rotation skills curriculum with high completion rates is feasible and allows interval training in close proximity to application in live surgery. Performance in the simulated environment significantly improved with corresponding improvements in confidence and selfassessed skill in the operating room. Further research is needed to objectively confirm transferability of skills from lab to operating room.
References:
4. 5. 6. 7. 8. 9.
10. 11.
TE D
3.
EP
2.
Soper, N.J. and D.A. DaRosa, Presidential address: Engendering operative autonomy in surgical training. Surgery, 2014. 156(4): p. 745-51. Ericsson KA, K.R., Tesch-Romer C. , The role of deliberate practice in the acquisition of expert performance. Psychol Rev, 1993. 100: p. 393-4. Kantor, O., et al., Implementing a resident acute care surgery service: Improving resident education and patient care. Surgery, 2017. 161(3): p. 876-883. Teman, N.R., et al., Entrustment of general surgery residents in the operating room: factors contributing to provision of resident autonomy. J Am Coll Surg, 2014. 219(4): p. 778-87. Aggarwal, R., et al., Proving the effectiveness of virtual reality simulation for training in laparoscopic surgery. Ann Surg, 2007. 246(5): p. 771-9. Grantcharov, T.P., et al., Randomized clinical trial of virtual reality simulation for laparoscopic skills training. Br J Surg, 2004. 91(2): p. 146-50. Seymour, N.E., et al., Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg, 2002. 236(4): p. 458-63; discussion 463-4. Wohaibi, E.M., et al., Surgical resident performance on a virtual reality simulator correlates with operating room performance. J Surg Res, 2010. 160(1): p. 67-72. Gravante, G. and D. Venditti, A systematic review on low-cost box models to achieve basic and advanced laparoscopic skills during modern surgical training. Surg Laparosc Endosc Percutan Tech, 2013. 23(2): p. 109-20. Seymour, N.E., VR to OR: a review of the evidence that virtual reality simulation improves operating room performance. World J Surg, 2008. 32(2): p. 182-8. Gurusamy, K., et al., Systematic review of randomized controlled trials on the effectiveness of virtual reality training for laparoscopic surgery. Br J Surg, 2008. 95(9): p. 1088-97.
AC C
1.
ACCEPTED MANUSCRIPT
18. 19. 20.
21.
RI PT
17.
SC
16.
M AN U
15.
TE D
14.
EP
13.
Vaillancourt, M., et al., GOALS-incisional hernia: a valid assessment of simulated laparoscopic incisional hernia repair. Surg Innov, 2011. 18(1): p. 48-54. Drake, F.T., et al., The general surgery chief resident operative experience: 23 years of national ACGME case logs. JAMA Surg, 2013. 148(9): p. 841-7. Mattar, S.G., et al., General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors. Ann Surg, 2013. 258(3): p. 440-9. De Win, G., et al., An evidence-based laparoscopic simulation curriculum shortens the clinical learning curve and reduces surgical adverse events. Adv Med Educ Pract, 2016. 7: p. 357-70. Sturm, L.P., et al., A systematic review of skills transfer after surgical simulation training. Ann Surg, 2008. 248(2): p. 166-79. Davies, J., M. Khatib, and F. Bello, Open surgical simulation--a review. J Surg Educ, 2013. 70(5): p. 618-27. Pentiak, P.A., et al., Barriers to adoption of the surgical resident skills curriculum of the American College of Surgeons/Association of Program Directors in Surgery. Surgery, 2013. 154(1): p. 23-8. Andersen, S.A., et al., Cognitive load in distributed and massed practice in virtual reality mastoidectomy simulation. Laryngoscope, 2016. 126(2): p. E74-9. Gostlow, H., et al., Systematic Review of Voluntary Participation in Simulation-Based Laparoscopic Skills Training: Motivators and Barriers for Surgical Trainee Attendance. J Surg Educ, 2017. 74(2): p. 306-318. Shetty, S., et al., Perceptions, training experiences, and preferences of surgical residents toward laparoscopic simulation training: a resident survey. J Surg Educ, 2014. 71(5): p. 727-33.
AC C
12.
ACCEPTED MANUSCRIPT
Table 1. Summary of rotation specific simulation modules
Minimally Invasive Surgery Minimally Invasive Surgery Minimally Invasive Surgery Minimally Invasive Surgery
RI PT
Description Virtual Reality Simulator
Virtual Reality Simulator
SC
Laparoscopic Sigmoidectomy
M AN U
Surgical Oncology
Laparoscopic Ventral Hernia
Virtual Reality Simulator
Thyroid Ultrasound and Biopsy
Bench Training Model
Laparoscopic Gastric Bypass
Virtual Reality and animate model
Laparoscopic Heller Myotomy
Box Trainer with animate model
TE D
Colorectal Surgery
Module Laparoscopic Cholecystectomy
Laparoscopic Nissen Fundoplication and Hiatal Hernia Repair Advanced Endoscopy
EP
Highland Park General and Minimally Invasive Surgery
Resident Level 1-PGY4 2-PGY3 9- PGY2 2-PGY5 6-PGY4 7-PGY3 7-PGY5 5-PGY4 1-PGY3 6-PGY3 2-PGY2 5-PGY5 7-PGY4 5-PGY5 7-PGY4 5-PGY5 7-PGY4 5-PGY5 7-PGY4
AC C
Rotation Glenbrook General Surgery
Box Trainer with animate model Box Trainer with animate model
ACCEPTED MANUSCRIPT
Table 2. Pre- and post-test scores and passing rates comparison by simulation modules Self-assessment by residents Pre-test Post-test P value Score Score 32.5±8.2 41.6±3.9 0.002
Passing Score 38
21.1±3.6
28.8±2.5
<.0001
25.4±3.6
30.1±2.1
25
Module Laparoscopic Cholecystectomy Laparoscopic Ventral Hernia
Completion Rate 12/12 (100%) 15/15 (100%)
Laparoscopic Sigmoidectomy
12/13 (92%)
26.0±2.6
36.3±3.3
<.0001
28.8±7.1
35.6±4.0
0.004
Thyroid Ultrasound and Biopsy Laparoscopic Gastric Bypass Laparoscopic Heller Myotomy Laparoscopic Fundoplication and Hiatal Hernia Repair Advanced Endoscopy
8/8 (100%) 12/12 (100%) 12/12 (100%) 12/12 (100%)
22.4±1.5
35.8±1.9
<.0001
22.1±2.9
35.3±1.9
45.8±5.1
55.3±6.7
0.0002
14.9±2.2
19.6±1.8
<.0001
16.8±2.5
21.4±1.2
<.0001
11/12 (92%)
16.9±2.3
20.2±1.7
0.0018
Pre-test Rate 0/12 (0%) 3/15 (20%)
Post-test Rate 12/12 (100%) 15/15 (100%)
25.6
8/13 (62%)
12/12 (100%)
0.0163
<.0001
30
0/8 (0%) 4/12 (33%) 5/12 (42%) 8/12 (67%)
8/8 (100%) 10/12 (83%) 12/12 (100%) 12/12 (100%)
0.0002
11/12 (92%)
12/12 (100%)
0.9999
M AN U
SC
<.0001
46.6±13.6
54.8±13.5
0.016
48
16.0±4.1
18.9±4.1
0.046
16
16.7±3.7
19.2±2.4
0.008
16
18.4±2.9
19.3±3.4
0.620
14.8
TE D
EP AC C
Passing rate
RI PT
Assessment by independent trained rater Pre-test Post-test P value Score Score 28.3±5.9 42.2±3.2 <.0001
P value <.0001 <.0001
0.0361 0.0046 0.0285
ACCEPTED MANUSCRIPT
Figure 1. Post rotation resident survey
Not enough time
Just the right amount of time
RI PT
1. Two hours per week were allotted for skills training on this rotation. Was this amount of time? Too much time
Strongly disagree
Disagree
Neutral
SC
2. In-rotation skills training modules were a valuable addition to my educational experience. Agree Strongly agree
Not at all
M AN U
3. In-rotation skills experiences improved my confidence in the operating room. A little
Moderately
Significantly Very significantly
4. In-rotation skills experiences improved my surgical skill in the operating room. A little
Moderately
TE D
Not at all
Significantly
Very significantly
5. In-rotation skills experiences provide benefits that are unique when compared to out-ofrotation skills labs or pure skills rotations.
EP
Disagree
AC C
Strongly disagree
Neutral
Agree
Strongly agree
ACCEPTED MANUSCRIPT
RI PT
We designed a resident skills curriculum that allows for learning, mentoring and practice of specific procedures in close proximity to operating on live patients. The curriculum based on case specific simulation modules included a pre-test administered by a trained, experienced rater, a one hour mentoring session by an attending surgeon, dedicated practice time and then a post-test. We have shown that this structured skills curriculum is feasible, obtaining 100% completion rates on most modules. It is also effective as resident’s performance on the skills assessment significantly improved from pre to post test on all modules.
SC
Resident self-assessed skill improved on all but one module, while 86% of the trainees reported improved confidence in the operating room and almost 80% believe their skill improved in the operating room. This indicates our curriculum enhances the resident’s educational experience and likely improves their surgical skill.
AC C
EP
TE D
M AN U
An In-rotation skills curriculum with high completion rates is feasible and allows interval training in close proximity to application in live surgery. Performance in the simulated environment significantly improved with corresponding improvements in confidence and selfassessed skill in the operating room. Further research is needed to objectively confirm transferability of skills from lab to operating room.