- Email: [email protected]
The FES Test: Are We Ready?
2018 APDS SPRING MEETING
The FES Test: Are We Ready? Yazan Aljamal, MD,*,† Humza Saleem, MD,† Nicholas Prabhakar, BA,† and David R. Farley, MD FACS† *
Mayo Clinic Multidisciplinary Simulation Center, Mayo Clinic College of Medicine and Science, Rochester, Minnesota; and †Department of General Surgery, Mayo Clinic, Rochester, Minnesota
PURPOSE: The FES hands-on skills test is administered
using a $100,000 computer-based simulator. Few of our trainees have practiced on this device. Our aim was to evaluate our GS residents’ baseline endoscopic skills and eventually develop a simulation-based endoscopy curriculum and clarify performance-based assessment criteria. METHODS: General surgery residents’ colonoscopy
skills were assessed using a computer-based endoscopy simulator (CBES) during their biannual simulation-based OSCE-type assessments. Trainees were asked to reach the ileum in <5 minutes with minimal patient pain and complications. Module 1 (easy) was assigned to PGY 1-4 residents and module 5 (hard) to both PGY 4s and 5s. The colonoscope insertion length, % time with no pain, % time in “red out”, and complications were recorded. Performance grading criteria were driven by literature review and expert opinion. Residents were assessed in the fall 2017; they were then given scoring criteria, a step-by-step instruction manual, and a voluntary handson session with the CBES. Residents repeated the same assessment in the spring 2018. RESULTS: 30 PGY-1s, 12 PGY-2s, 8 PGY-3s, 9 PGY-4s and
7 PGY-5s GS residents participated in the fall colonoscopy assessment. In module 1, 66% of PGY-4s, 50% of PGY-3s, 8% of PGY-2s and 0% of the PGY-1s intubated the ileum (p<0.05). In module 5, 30% of PGY 5 and 22% of PGY 4 residents completed the task (p<0.05). 15 PGY-1s, 5 PGY-2s, 1 PGY-3, 2 PGY-4s, and 1 PGY-5 participated in the voluntary hands-on session. All residents completing the fall assessment undertook the same task
Correspondence: Inquiries to David R. Farley, MD, FACS, Department of Surgery, Mayo Clinic, 200 First Street, SW, Rochester, Minnesota 55905; fax: (507)2845196; e-mail: [email protected] Financial support: There was no external funding. Authorship: Authors YA and DRF conceived the study; YA acquired data; YA, DRF, HS, and NP planned the analysis; and YA drafted the initial manuscript. All authors were involved in interpreting data and revising the manuscript, and all approved the final manuscript. Ethical approval: This study was judged Exempt by the Mayo Institutional Review Board. All participants provided consent. Conflicts of interest: We are not aware of any conflicts of interest.
in the spring. In module 1, 89% of PGY-4s, 100% of the PGY-3s, 75% of PGY-2s and 70% of the PGY-1s completed the task. In module 5, 30% of PGY 5 and 34 % of PGY 4 residents completed the task. Residents who participated in the voluntary hands-on session (n= 24, 96% task completion) outperformed residents (n= 42, 64% task completion) that did not participate (p<0.05). CONCLUSIONS: Most of our GS residents could not ini-
tially intubate the ileum using the CBES. Prior experience with the CBES was the only factor strongly correlated with successful task completion. A voluntary hands-on teaching session allowed 96% of participating trainees to subsequently achieve CBES task completion. Developing a formal simulation-based curriculum suggests we can better prepare surgical trainees for the FES exam. ( J Surg Ed 75:e212 e217. Ó 2018 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: FES, endoscopy, colonoscopy, surgery, simulation, education COMPETENCIES: Medical Knowledge, Interpersonal
and Communication Skills, Practice-Based Learning and Improvement, Systems-Based Practice
INTRODUCTION The Fundamentals of Endoscopic Surgery (FES) is a highstakes examination of the basic knowledge and skills that are required to perform safe endoscopy. This test is administered using a Virtual Reality (VR) Computer-Based Endoscopy-Simulator (CBES). FES certification is required before sitting for The American Board of Surgery (ABS) qualifying examination and thus it is mandatory to pass the FES test in order to be board-certified in the USA.1 Early data reveals a passing rate of 68% among general surgery (GS) chief residents on the FES exam.2 It appears that performing a minimum of 106 endoscopic cases greatly enhances trainees’ ability to pass the FES test.2
e212 Journal of Surgical Education © 2018 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jsurg.2018.06.022
While one solution for improving FES pass rates might include increasing trainee experience with real endoscopic cases, such an effort teaching novice trainees might have a negative impact on patient safety and likely increases institutional costs. Offering simulation practice time, either with an endoscopic computer simulator or a lower cost simulator may prove useful to trainees, patients, and lead to higher FES exam pass rates. Since the implementation of FES is a recent addition to GS training programs, we sought to determine the baseline competency levels of our residents using a CBES, and investigate whether a simulation-based self-practice endoscopy curriculum could effectively improve trainees’ endoscopic performance on the VR CBES.
METHODS General Surgery (GS) residents at Mayo Clinic, Rochester are required to undertake objective simulation-based skills assessments known as the Surgical Olympics for postgraduate year (PGY) 1s and the Surgical X-Games for PGY2, PGY-3, PGY-4, PGY-5 GS residents. The competitions occur biannually, with identical tests held in the fall and spring at our simulation center. The assessment consists of nine stations for PGY-1s and six stations for PGY-2s, 3s, 4s, and 5s. The colonoscopy skill station was identical (time allotment, scoring rubric, and type of simulator used) for each PGY class assessment. The simulator, Acucutouch CBES, has six colonoscopy “cases” of varying complexity. Trainees were instructed to reach and subsequently intubate the ileocecal junction within 5 minutes while trying to minimize patient pain and complications. At the end of each case, multiple performance parameters were recorded: completion task (cecum intubation), total procedure time, time in red-out, percentage of mucosa visualized, percentage of time with no pain, and maximum length of scope inserted (cm). PGY 1s, 2s and 3s were assigned case 1 (easy) while PGY 5s were assigned case 5 (hard). PGY 4s completed both modules (easy & hard). In between the fall and spring assessments, residents were provided with: (1) A “learning colonoscopy” single-sheet handout with clear step-by-step instructions of how to complete the task successfully. (2) CBES performance criteria which were established previously by Sedlack et al., based on the experience of ten ‘‘expert’’ faculty colonoscopists with the Accutouch CBES.3,4 (3) Voluntary hands-on sessions with live feedback while residents practiced on the simulator. Results were analyzed using JMP 13.0.0 software. Statistical analysis was performed using the Student’s t-test with a significance level set at 0.05.
RESULTS A total of 70 clinical PGY-1 to -5 GS residents participated in our OSCE-type assessments: 69 residents completed the fall 2017 test and 66 completed the spring 2018 assessment. Sixty-five residents participated in both assessments. Fall Assessment Thirty PGY-1s, 12 PGY-2s, 8 PGY-3s, 9 PGY-4s and 10 PGY-5s GS residents completed the colonoscopy station. One PGY 4 and 3 PGY-5s did not complete either the fall or spring assessments and were excluded from the analysis. In module 1 (easy), 66% of PGY-4s, 50% of the PGY3s, 8% of PGY-2s and 0% of the PGY-1s completed the task (ileal intubation). Past experience with the simulator correlated with successful task completion (70% with past experience vs. 20% with no experience; p < 0.05). In module 5 (hard), 30% of PGY 5 and 22% of PGY 4 residents completed the task (p < 0.05). The mean maximum scope insertion length was 48, 88, 110, 89 cm for PGY 1,2,3, and 4 respectively in module 1 (p < 0.05) and 84 and 78 cm for PGY 4 and 5 respectively in module 5, p = 0.5. The overall complication rate (colon perforation) was 0.5%. Intervention Between the two assessments, 15 PGY-1 GS residents voluntarily attended a practice session lasting between 1 and 2 hours on the Accutouch CBES. Five PGY-2s, 1 PGY-3, 2 PGY-4s, and 1 PGY-5 attended an hour-long, 1on-1 practice session using the Accutouch CBES. All 24 residents receiving this voluntary training reported an average of three additional hours (range: 1-6 hours) of practice by themselves over multiple dates using different cases on the same CBES. Spring Assessment 30 PGY-1, 12 PGY-2, 8 PGY-3, 9 PGY-4, and 7 PGY-5 GS residents completed the colonoscopy station. In module 1, 89% of PGY-4s, 100% of the PGY-3s, 75% of PGY-2s and 70% of the PGY-1s completed the task. In module 5, 30% of PGY 5 and 34 % of PGY 4 residents completed the task. Total procedure time, maximum depth of scope insertion, percentage of mucosa visualized, and percentage of total time in red-out significantly improved in the post-training assessment (p < 0.05) (Table 1). Twenty junior residents (15 PGY-1s, 5 PGY-2s) attended the added teaching session; only four senior trainees (1 PGY-3, 2 PGY-4, 1 PGY-5) did so. Of the PGY 1s and 2s that practiced between tests, 100% successfully completed the task (vs 50% for those not practicing, p < 0.05%). No significant change in performance
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
e213
TABLE 1. Comparison of Multiple PGYs Performance on the Simulator
Completed Full Colonoscopy (N (%))
Total Procedure Time for Participants Who Completed Full Colonoscopy (s) (mean (SD))
Max Depth of Scope Insertion (cm) (mean (SD))
Percentage of Mucosa Visualized (%) (mean (SD)
Level of No Discomfort % of Total Procedure Time (%) (mean (SD))
Colonic Perforation
Percentage of Total Time in Red-Out (%) (mean (SD))
was found in overall among senior residents (p > 0.05). The percentage of mucosa visualized, max scope insertion lengths, and percentage of total time in red-out were better with the group that attended the practice sessions and independently practiced with simulator (Table 2).
DISCUSSION Our study revealed several important results: (1) Task completion rates using the CBES were low among all GS residents with their first attempt, (2) Clinical experience
e214
Fall
Spring
P value
PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1)
0 1 (8) 4(50) 6(66) 2(22) 3(30) NA
21 (70) 9 (75) 8(100) 8(89) 3(33) 3(30) 250 (62)
NA 0.0005 0.08 0.05 0.1 NA NA
PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5)
256(21) 210(27) 207(58) 239(27) 300(13) 52(30) 73(43) 109(6) 88(32) 67(33) 72(26) 9(7) 25(20) 22(8) 33(21) 8(5) 22(31) 92(16) 93(5) 89(2) 94(5) 91(6) 92(4) 0 1(8) 0 1(11) 2(22) 0 31(26) 38(31) 9(12) 10(11) 16(10) 8(4)
267(64) 263(23) 196(43) 253(89) 317(11) 92(28) 94(16) 104(6) 99(13) 85(17) 95(13) 78(26) 84(18) 93(2) 85(15) 82(5) 88(19) 93(5) 96(4) 97(2) 93(8) 90(4) 90(7) 0 0 0 0 1(11) 1(10) 10(8) 8(6) 11(9) 8(5) 8(3) 11(10)
0.7 0.1 0.7 0.8 0.6 <0.001 0.1 0.6 0.3 0.1 0.04 <0.001 <0.001 <0.001 <0.001 <0.001 0.0002 0.7 0.1 0.03 0.7 0.4 0.4 NA NA NA NA 0.2 NA <0.001 0.0025 0.7 0.6 0.04 0.4
correlates with task completion, (3) Providing residents with clear instruction on how to perform the task successfully and elucidating the performance criteria helped improve scores in spring, and (4) Junior residents can significantly improve their basic endoscopic skills with a 1 to 2 hours practice session. The examination results from the fall of 2017 showed that there was a low completion rate on both the “easy” and “hard” CBES tests for our GS residents. While this was disappointing, it is not entirely surprising. FES was implemented as a means to combat inexperience with endoscopy and to increase competency with endoscopic skills. Whether this ill-preparation relates to less
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
TABLE 2. Comparison of Training Versus No Training Groups Performance on the Simulator
Completed Full Colonoscopy (N (%))
Total Procedure Time for Participants Who Completed Full Colonoscopy (s) (mean (SD))
Max Depth of Scope Insertion (cm) (mean (SD))
Percentage of Mucosa Visualized (%) (mean (SD)
Level of No Discomfort % of Total Procedure Time (%) (mean (SD))
Colonic Perforation (N (%))
Percentage of Total Time in Red-out (%) (mean (SD))
PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5)
time on endoscopic rotations, weak curricula, complexity of the VR simulator, or other factors is unclear. Many would argue that education and training of GS residents in endoscopic skills have long been sub-par.2,5 Gardner et al. recently showed that current GS residents are still not adequately prepared to pass the FES exam. The failure rates found in their study indicated a lack of both cognitive knowledge and manual skills with regards to colonoscopy. Their suggestion, backed by Receiver Operating Characteristic (ROC) analysis, was to increase the ABS colonoscopy requirement from 50 to at least 63 procedures, ideally with a target number of 103 total
Spring X Games +training (No)training
P value
15(100) 5(100) 1(100) 2(100) 1(50) 1(100) 210(67) 243(60) 186 181(31) 239(19) 270 105(11) 98(5) 105 113(12) 97(1) 110 92(5) 94(0.8) 95 91(4) 94(0.9) 96 93(5) 97(2) 94 87(5) 90(8) 90 0 0 0 0 0 0 8(6) 7(2) 6 10.5(6) 7(0.7) 9
<0.001 0.04 NA 0.08 0.2 NA <0.001 0.3 0.2 0.6 0.3 0.6 0.006 0.4 NA 0.08 0.3 NA 0.0004 0.04 NA 0.5 0.4 NA 0.5 0.4 NA 0.3 0.7 NA NA NA NA NA NA NA 0.3 0.8 NA 0.5 0.08 NA
6(40) 4(57) 7(100) 6(86) 2(28) 2(33) 295(14) 284(66) 274(67) 200(48) 302(84) 325(119) 79(32) 92(20) 104(13) 95(11) 82(18) 92(11) 62(29) 76(20) 92(2) 84(16) 78(24) 86(20) 94(5) 95(4) 97(5) 94(3) 89(3) 90(8) 0 0 0 1(14) 1(14) 1(17) 12(10) 8(8) 11(10) 7(5) 12(4) 11(11)
cases in order to best prepare residents for both portions of the FES exam.2 This change may eventually prove to be effective in preparing residents for the FES examination as there has been a strong correlation with increased clinical colonoscopy performance and FES performance.6 However, numerous factors make simply increasing endoscopic experience difficult: program directors and trainees are short on rotation time for many specialties, attendings are often pressured to move endoscopic cases along, and residents and fellows often compete for endoscopy rotations and experience. Due to these constraints, we decided to approach the
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
e215
high failure rate in the fall of 2017 with a simulation education approach one we thought would be more time efficient and cost effective. The purpose of the FES hands-on test is to establish competency levels of endoscopic skills in vivo without any potential for patient harm. In order to effectively do this, the tool must differentiate between novices with little experience and professional endoscopists with ample repetitions.5 Our study shows a gradual increase of scores from PGY 1s (most with no exposure to colonoscopy) to PGY 5s, demonstrating that clinical experience is relevant, and typically correlates with task completion. This is consistent with other studies that show an increase in scores respective to clinical exposure; such studies were used for the validation of the FES as an assessment tool.7,8 Mastery Learning Theory (MLT) was first popularized in the late 1960s with the aim of educating and confirming students achieve a proficient set of skills before moving on to the next stage. If these skills are subpar then students should have additional practice until they reach ‘Mastery Level’. In MLT, the responsibility of the student’s failure lies in the lack of instruction rather than the student’s own ability to perform the task. Ritter et al.7 demonstrated that simulation-based MLT, through written instructions and simulation-based practice, improved colonoscopy skills in FES. We tried to incorporate a self-regulating MLT strategy through a multifaceted approach: stepby-step instructions, CBES validated performance criteria and simulation based practice. By giving residents access to step-by-step instructions, tips, and tricks, we hoped to provide them with the essential knowledge that they were lacking prior to conducting an endoscopic examination. Additionally, a CBES performance criterion was given to allow resident self-assessment of their level of proficiency against validated criteria, allowing them to determine when they have reached ‘Mastery level’. Through individual one-on-one simulation based training we hoped to provide residents with individualized feedback. We were encouraged to find one added teaching session positively impacted resident performance, particularly with our junior residents. PGY1s are generally inexperienced with the principles of performing a colonoscopy, and this was reflected in their fall CBES scores. However, their improvement from the fall to spring was tremendous all PGY-1 trainees who undertook the added simulation based training successfully passed the exam. During this session both MLT and competencybased learning techniques were adopted. There are certainly many limitations to this study, but objectivity in grading was not problematic as scores were generated by the simulator. The very small number
e216
of senior residents participating in the intervention session compared to those that did participate limits the power to reflect statistical differences. The cost of the CBES itself is a limiting factor that makes it difficult for most institutions and their trainees to obtain time for practice. The computer simulator is prone to mechanical malfunction with usage, which has severely limited the amount of practice time available as repair is neither quick nor cheap. Participation in added training time (either formal or individual) was completely voluntary and subjects were not randomized which may lead to a selection bias: residents attending voluntary sessions were more likely to be more motived and practice on their own time. During the period between the two assessments, residents receiving this voluntary training reported an average of three additional hours of practice by themselves over multiple dates using different cases on the same CBES; therefore changes in test scores may be based on unrecognized confounding variables. We have no data yet to show whether our efforts have affected the success rate among GS trainees with the formal FES examination; we aim to prospectively follow our trainees and national data for several years to address the effectiveness of educational efforts.
CONCLUSION While the introduction of FES has been a frustrating and expensive proposition for surgical programs and their trainees, it has simultaneously been a positive stimulus for improving clinical competency with endoscopy. Given that endoscopy remains the number 1 procedure of rural general surgeons and a common procedure for others, generating better education and offering more practice opportunities for young learners is paramount.9 Our study suggests we are nearly ready for the FES test additional time, practice, and education clearly enhance FES performance. Hopefully, such efforts similarly lead to better overall endoscopic skills and better patient outcomes.
REFERENCES 1. Inc TABoS. Flexible Endoscopy Curriculum for Gen-
eral Surgery residents, Inc TABoS 2. Gardner AK, Scott DJ, Willis RE, et al. Is current sur-
gery resident and GI fellow training adequate to pass FES? Surg Endosc. 2017;31(1):352–358. 3. Sedlack RE, Kolars JC. Colonoscopy curriculum
development and performance-based assessment
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
criteria on a computer-based endoscopy simulator. Acad Med J Assoc Am Med Coll. 2002;77(7):750–751. 4. Sedlack RE, Kolars JC. Validation of a computer-based
colonoscopy simulator. 2003;57(2):214–218.
Gastrointest
Endosc.
5. Hazey JW, Marks JM, Mellinger JD, et al. Why funda-
mentals of endoscopic surgery (FES)? Surg Endosc. 2014;28(3):701–703. 6. Mueller CL, Kaneva P, Fried GM, et al. Colonoscopy
performance correlates with scores on the FES manual skills test. Surg Endosc. 2014;28(11):3081–3085. 7. Ritter EM, Taylor ZA, Wolf KR, et al. Simulation-based
mastery learning for endoscopy using the endoscopy
training system: a strategy to improve endoscopic skills and prepare for the fundamentals of endoscopic surgery (FES) manual skills exam. Surg Endosc. 2018;32(1):413–420. 8. Vassiliou MC, Kaneva PA, Poulose BK, et al. Global
Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc. 2010;24 (8):1834–1841. 9. Ritchie WP Jr., Rhodes RS, Biester TW. Work loads
and practice patterns of general surgeons in the United States, 1995-1997: a report from the American Board of Surgery. Ann Surg. 1999;230(4):533–542. Discussion 42-3.
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
e217
The FES Test: Are We Ready? Yazan Aljamal, MD,*,† Humza Saleem, MD,† Nicholas Prabhakar, BA,† and David R. Farley, MD FACS† *
Mayo Clinic Multidisciplinary Simulation Center, Mayo Clinic College of Medicine and Science, Rochester, Minnesota; and †Department of General Surgery, Mayo Clinic, Rochester, Minnesota
PURPOSE: The FES hands-on skills test is administered
using a $100,000 computer-based simulator. Few of our trainees have practiced on this device. Our aim was to evaluate our GS residents’ baseline endoscopic skills and eventually develop a simulation-based endoscopy curriculum and clarify performance-based assessment criteria. METHODS: General surgery residents’ colonoscopy
skills were assessed using a computer-based endoscopy simulator (CBES) during their biannual simulation-based OSCE-type assessments. Trainees were asked to reach the ileum in <5 minutes with minimal patient pain and complications. Module 1 (easy) was assigned to PGY 1-4 residents and module 5 (hard) to both PGY 4s and 5s. The colonoscope insertion length, % time with no pain, % time in “red out”, and complications were recorded. Performance grading criteria were driven by literature review and expert opinion. Residents were assessed in the fall 2017; they were then given scoring criteria, a step-by-step instruction manual, and a voluntary handson session with the CBES. Residents repeated the same assessment in the spring 2018. RESULTS: 30 PGY-1s, 12 PGY-2s, 8 PGY-3s, 9 PGY-4s and
7 PGY-5s GS residents participated in the fall colonoscopy assessment. In module 1, 66% of PGY-4s, 50% of PGY-3s, 8% of PGY-2s and 0% of the PGY-1s intubated the ileum (p<0.05). In module 5, 30% of PGY 5 and 22% of PGY 4 residents completed the task (p<0.05). 15 PGY-1s, 5 PGY-2s, 1 PGY-3, 2 PGY-4s, and 1 PGY-5 participated in the voluntary hands-on session. All residents completing the fall assessment undertook the same task
Correspondence: Inquiries to David R. Farley, MD, FACS, Department of Surgery, Mayo Clinic, 200 First Street, SW, Rochester, Minnesota 55905; fax: (507)2845196; e-mail: [email protected] Financial support: There was no external funding. Authorship: Authors YA and DRF conceived the study; YA acquired data; YA, DRF, HS, and NP planned the analysis; and YA drafted the initial manuscript. All authors were involved in interpreting data and revising the manuscript, and all approved the final manuscript. Ethical approval: This study was judged Exempt by the Mayo Institutional Review Board. All participants provided consent. Conflicts of interest: We are not aware of any conflicts of interest.
in the spring. In module 1, 89% of PGY-4s, 100% of the PGY-3s, 75% of PGY-2s and 70% of the PGY-1s completed the task. In module 5, 30% of PGY 5 and 34 % of PGY 4 residents completed the task. Residents who participated in the voluntary hands-on session (n= 24, 96% task completion) outperformed residents (n= 42, 64% task completion) that did not participate (p<0.05). CONCLUSIONS: Most of our GS residents could not ini-
tially intubate the ileum using the CBES. Prior experience with the CBES was the only factor strongly correlated with successful task completion. A voluntary hands-on teaching session allowed 96% of participating trainees to subsequently achieve CBES task completion. Developing a formal simulation-based curriculum suggests we can better prepare surgical trainees for the FES exam. ( J Surg Ed 75:e212 e217. Ó 2018 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: FES, endoscopy, colonoscopy, surgery, simulation, education COMPETENCIES: Medical Knowledge, Interpersonal
and Communication Skills, Practice-Based Learning and Improvement, Systems-Based Practice
INTRODUCTION The Fundamentals of Endoscopic Surgery (FES) is a highstakes examination of the basic knowledge and skills that are required to perform safe endoscopy. This test is administered using a Virtual Reality (VR) Computer-Based Endoscopy-Simulator (CBES). FES certification is required before sitting for The American Board of Surgery (ABS) qualifying examination and thus it is mandatory to pass the FES test in order to be board-certified in the USA.1 Early data reveals a passing rate of 68% among general surgery (GS) chief residents on the FES exam.2 It appears that performing a minimum of 106 endoscopic cases greatly enhances trainees’ ability to pass the FES test.2
e212 Journal of Surgical Education © 2018 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. https://doi.org/10.1016/j.jsurg.2018.06.022
While one solution for improving FES pass rates might include increasing trainee experience with real endoscopic cases, such an effort teaching novice trainees might have a negative impact on patient safety and likely increases institutional costs. Offering simulation practice time, either with an endoscopic computer simulator or a lower cost simulator may prove useful to trainees, patients, and lead to higher FES exam pass rates. Since the implementation of FES is a recent addition to GS training programs, we sought to determine the baseline competency levels of our residents using a CBES, and investigate whether a simulation-based self-practice endoscopy curriculum could effectively improve trainees’ endoscopic performance on the VR CBES.
METHODS General Surgery (GS) residents at Mayo Clinic, Rochester are required to undertake objective simulation-based skills assessments known as the Surgical Olympics for postgraduate year (PGY) 1s and the Surgical X-Games for PGY2, PGY-3, PGY-4, PGY-5 GS residents. The competitions occur biannually, with identical tests held in the fall and spring at our simulation center. The assessment consists of nine stations for PGY-1s and six stations for PGY-2s, 3s, 4s, and 5s. The colonoscopy skill station was identical (time allotment, scoring rubric, and type of simulator used) for each PGY class assessment. The simulator, Acucutouch CBES, has six colonoscopy “cases” of varying complexity. Trainees were instructed to reach and subsequently intubate the ileocecal junction within 5 minutes while trying to minimize patient pain and complications. At the end of each case, multiple performance parameters were recorded: completion task (cecum intubation), total procedure time, time in red-out, percentage of mucosa visualized, percentage of time with no pain, and maximum length of scope inserted (cm). PGY 1s, 2s and 3s were assigned case 1 (easy) while PGY 5s were assigned case 5 (hard). PGY 4s completed both modules (easy & hard). In between the fall and spring assessments, residents were provided with: (1) A “learning colonoscopy” single-sheet handout with clear step-by-step instructions of how to complete the task successfully. (2) CBES performance criteria which were established previously by Sedlack et al., based on the experience of ten ‘‘expert’’ faculty colonoscopists with the Accutouch CBES.3,4 (3) Voluntary hands-on sessions with live feedback while residents practiced on the simulator. Results were analyzed using JMP 13.0.0 software. Statistical analysis was performed using the Student’s t-test with a significance level set at 0.05.
RESULTS A total of 70 clinical PGY-1 to -5 GS residents participated in our OSCE-type assessments: 69 residents completed the fall 2017 test and 66 completed the spring 2018 assessment. Sixty-five residents participated in both assessments. Fall Assessment Thirty PGY-1s, 12 PGY-2s, 8 PGY-3s, 9 PGY-4s and 10 PGY-5s GS residents completed the colonoscopy station. One PGY 4 and 3 PGY-5s did not complete either the fall or spring assessments and were excluded from the analysis. In module 1 (easy), 66% of PGY-4s, 50% of the PGY3s, 8% of PGY-2s and 0% of the PGY-1s completed the task (ileal intubation). Past experience with the simulator correlated with successful task completion (70% with past experience vs. 20% with no experience; p < 0.05). In module 5 (hard), 30% of PGY 5 and 22% of PGY 4 residents completed the task (p < 0.05). The mean maximum scope insertion length was 48, 88, 110, 89 cm for PGY 1,2,3, and 4 respectively in module 1 (p < 0.05) and 84 and 78 cm for PGY 4 and 5 respectively in module 5, p = 0.5. The overall complication rate (colon perforation) was 0.5%. Intervention Between the two assessments, 15 PGY-1 GS residents voluntarily attended a practice session lasting between 1 and 2 hours on the Accutouch CBES. Five PGY-2s, 1 PGY-3, 2 PGY-4s, and 1 PGY-5 attended an hour-long, 1on-1 practice session using the Accutouch CBES. All 24 residents receiving this voluntary training reported an average of three additional hours (range: 1-6 hours) of practice by themselves over multiple dates using different cases on the same CBES. Spring Assessment 30 PGY-1, 12 PGY-2, 8 PGY-3, 9 PGY-4, and 7 PGY-5 GS residents completed the colonoscopy station. In module 1, 89% of PGY-4s, 100% of the PGY-3s, 75% of PGY-2s and 70% of the PGY-1s completed the task. In module 5, 30% of PGY 5 and 34 % of PGY 4 residents completed the task. Total procedure time, maximum depth of scope insertion, percentage of mucosa visualized, and percentage of total time in red-out significantly improved in the post-training assessment (p < 0.05) (Table 1). Twenty junior residents (15 PGY-1s, 5 PGY-2s) attended the added teaching session; only four senior trainees (1 PGY-3, 2 PGY-4, 1 PGY-5) did so. Of the PGY 1s and 2s that practiced between tests, 100% successfully completed the task (vs 50% for those not practicing, p < 0.05%). No significant change in performance
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
e213
TABLE 1. Comparison of Multiple PGYs Performance on the Simulator
Completed Full Colonoscopy (N (%))
Total Procedure Time for Participants Who Completed Full Colonoscopy (s) (mean (SD))
Max Depth of Scope Insertion (cm) (mean (SD))
Percentage of Mucosa Visualized (%) (mean (SD)
Level of No Discomfort % of Total Procedure Time (%) (mean (SD))
Colonic Perforation
Percentage of Total Time in Red-Out (%) (mean (SD))
was found in overall among senior residents (p > 0.05). The percentage of mucosa visualized, max scope insertion lengths, and percentage of total time in red-out were better with the group that attended the practice sessions and independently practiced with simulator (Table 2).
DISCUSSION Our study revealed several important results: (1) Task completion rates using the CBES were low among all GS residents with their first attempt, (2) Clinical experience
e214
Fall
Spring
P value
PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1)
0 1 (8) 4(50) 6(66) 2(22) 3(30) NA
21 (70) 9 (75) 8(100) 8(89) 3(33) 3(30) 250 (62)
NA 0.0005 0.08 0.05 0.1 NA NA
PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5)
256(21) 210(27) 207(58) 239(27) 300(13) 52(30) 73(43) 109(6) 88(32) 67(33) 72(26) 9(7) 25(20) 22(8) 33(21) 8(5) 22(31) 92(16) 93(5) 89(2) 94(5) 91(6) 92(4) 0 1(8) 0 1(11) 2(22) 0 31(26) 38(31) 9(12) 10(11) 16(10) 8(4)
267(64) 263(23) 196(43) 253(89) 317(11) 92(28) 94(16) 104(6) 99(13) 85(17) 95(13) 78(26) 84(18) 93(2) 85(15) 82(5) 88(19) 93(5) 96(4) 97(2) 93(8) 90(4) 90(7) 0 0 0 0 1(11) 1(10) 10(8) 8(6) 11(9) 8(5) 8(3) 11(10)
0.7 0.1 0.7 0.8 0.6 <0.001 0.1 0.6 0.3 0.1 0.04 <0.001 <0.001 <0.001 <0.001 <0.001 0.0002 0.7 0.1 0.03 0.7 0.4 0.4 NA NA NA NA 0.2 NA <0.001 0.0025 0.7 0.6 0.04 0.4
correlates with task completion, (3) Providing residents with clear instruction on how to perform the task successfully and elucidating the performance criteria helped improve scores in spring, and (4) Junior residents can significantly improve their basic endoscopic skills with a 1 to 2 hours practice session. The examination results from the fall of 2017 showed that there was a low completion rate on both the “easy” and “hard” CBES tests for our GS residents. While this was disappointing, it is not entirely surprising. FES was implemented as a means to combat inexperience with endoscopy and to increase competency with endoscopic skills. Whether this ill-preparation relates to less
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
TABLE 2. Comparison of Training Versus No Training Groups Performance on the Simulator
Completed Full Colonoscopy (N (%))
Total Procedure Time for Participants Who Completed Full Colonoscopy (s) (mean (SD))
Max Depth of Scope Insertion (cm) (mean (SD))
Percentage of Mucosa Visualized (%) (mean (SD)
Level of No Discomfort % of Total Procedure Time (%) (mean (SD))
Colonic Perforation (N (%))
Percentage of Total Time in Red-out (%) (mean (SD))
PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5) PGY1s (Case 1) PGY2s (1) PGY3s (1) PGY4s (1) PGY4s (case 5) PGY5s (5)
time on endoscopic rotations, weak curricula, complexity of the VR simulator, or other factors is unclear. Many would argue that education and training of GS residents in endoscopic skills have long been sub-par.2,5 Gardner et al. recently showed that current GS residents are still not adequately prepared to pass the FES exam. The failure rates found in their study indicated a lack of both cognitive knowledge and manual skills with regards to colonoscopy. Their suggestion, backed by Receiver Operating Characteristic (ROC) analysis, was to increase the ABS colonoscopy requirement from 50 to at least 63 procedures, ideally with a target number of 103 total
Spring X Games +training (No)training
P value
15(100) 5(100) 1(100) 2(100) 1(50) 1(100) 210(67) 243(60) 186 181(31) 239(19) 270 105(11) 98(5) 105 113(12) 97(1) 110 92(5) 94(0.8) 95 91(4) 94(0.9) 96 93(5) 97(2) 94 87(5) 90(8) 90 0 0 0 0 0 0 8(6) 7(2) 6 10.5(6) 7(0.7) 9
<0.001 0.04 NA 0.08 0.2 NA <0.001 0.3 0.2 0.6 0.3 0.6 0.006 0.4 NA 0.08 0.3 NA 0.0004 0.04 NA 0.5 0.4 NA 0.5 0.4 NA 0.3 0.7 NA NA NA NA NA NA NA 0.3 0.8 NA 0.5 0.08 NA
6(40) 4(57) 7(100) 6(86) 2(28) 2(33) 295(14) 284(66) 274(67) 200(48) 302(84) 325(119) 79(32) 92(20) 104(13) 95(11) 82(18) 92(11) 62(29) 76(20) 92(2) 84(16) 78(24) 86(20) 94(5) 95(4) 97(5) 94(3) 89(3) 90(8) 0 0 0 1(14) 1(14) 1(17) 12(10) 8(8) 11(10) 7(5) 12(4) 11(11)
cases in order to best prepare residents for both portions of the FES exam.2 This change may eventually prove to be effective in preparing residents for the FES examination as there has been a strong correlation with increased clinical colonoscopy performance and FES performance.6 However, numerous factors make simply increasing endoscopic experience difficult: program directors and trainees are short on rotation time for many specialties, attendings are often pressured to move endoscopic cases along, and residents and fellows often compete for endoscopy rotations and experience. Due to these constraints, we decided to approach the
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
e215
high failure rate in the fall of 2017 with a simulation education approach one we thought would be more time efficient and cost effective. The purpose of the FES hands-on test is to establish competency levels of endoscopic skills in vivo without any potential for patient harm. In order to effectively do this, the tool must differentiate between novices with little experience and professional endoscopists with ample repetitions.5 Our study shows a gradual increase of scores from PGY 1s (most with no exposure to colonoscopy) to PGY 5s, demonstrating that clinical experience is relevant, and typically correlates with task completion. This is consistent with other studies that show an increase in scores respective to clinical exposure; such studies were used for the validation of the FES as an assessment tool.7,8 Mastery Learning Theory (MLT) was first popularized in the late 1960s with the aim of educating and confirming students achieve a proficient set of skills before moving on to the next stage. If these skills are subpar then students should have additional practice until they reach ‘Mastery Level’. In MLT, the responsibility of the student’s failure lies in the lack of instruction rather than the student’s own ability to perform the task. Ritter et al.7 demonstrated that simulation-based MLT, through written instructions and simulation-based practice, improved colonoscopy skills in FES. We tried to incorporate a self-regulating MLT strategy through a multifaceted approach: stepby-step instructions, CBES validated performance criteria and simulation based practice. By giving residents access to step-by-step instructions, tips, and tricks, we hoped to provide them with the essential knowledge that they were lacking prior to conducting an endoscopic examination. Additionally, a CBES performance criterion was given to allow resident self-assessment of their level of proficiency against validated criteria, allowing them to determine when they have reached ‘Mastery level’. Through individual one-on-one simulation based training we hoped to provide residents with individualized feedback. We were encouraged to find one added teaching session positively impacted resident performance, particularly with our junior residents. PGY1s are generally inexperienced with the principles of performing a colonoscopy, and this was reflected in their fall CBES scores. However, their improvement from the fall to spring was tremendous all PGY-1 trainees who undertook the added simulation based training successfully passed the exam. During this session both MLT and competencybased learning techniques were adopted. There are certainly many limitations to this study, but objectivity in grading was not problematic as scores were generated by the simulator. The very small number
e216
of senior residents participating in the intervention session compared to those that did participate limits the power to reflect statistical differences. The cost of the CBES itself is a limiting factor that makes it difficult for most institutions and their trainees to obtain time for practice. The computer simulator is prone to mechanical malfunction with usage, which has severely limited the amount of practice time available as repair is neither quick nor cheap. Participation in added training time (either formal or individual) was completely voluntary and subjects were not randomized which may lead to a selection bias: residents attending voluntary sessions were more likely to be more motived and practice on their own time. During the period between the two assessments, residents receiving this voluntary training reported an average of three additional hours of practice by themselves over multiple dates using different cases on the same CBES; therefore changes in test scores may be based on unrecognized confounding variables. We have no data yet to show whether our efforts have affected the success rate among GS trainees with the formal FES examination; we aim to prospectively follow our trainees and national data for several years to address the effectiveness of educational efforts.
CONCLUSION While the introduction of FES has been a frustrating and expensive proposition for surgical programs and their trainees, it has simultaneously been a positive stimulus for improving clinical competency with endoscopy. Given that endoscopy remains the number 1 procedure of rural general surgeons and a common procedure for others, generating better education and offering more practice opportunities for young learners is paramount.9 Our study suggests we are nearly ready for the FES test additional time, practice, and education clearly enhance FES performance. Hopefully, such efforts similarly lead to better overall endoscopic skills and better patient outcomes.
REFERENCES 1. Inc TABoS. Flexible Endoscopy Curriculum for Gen-
eral Surgery residents, Inc TABoS 2. Gardner AK, Scott DJ, Willis RE, et al. Is current sur-
gery resident and GI fellow training adequate to pass FES? Surg Endosc. 2017;31(1):352–358. 3. Sedlack RE, Kolars JC. Colonoscopy curriculum
development and performance-based assessment
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
criteria on a computer-based endoscopy simulator. Acad Med J Assoc Am Med Coll. 2002;77(7):750–751. 4. Sedlack RE, Kolars JC. Validation of a computer-based
colonoscopy simulator. 2003;57(2):214–218.
Gastrointest
Endosc.
5. Hazey JW, Marks JM, Mellinger JD, et al. Why funda-
mentals of endoscopic surgery (FES)? Surg Endosc. 2014;28(3):701–703. 6. Mueller CL, Kaneva P, Fried GM, et al. Colonoscopy
performance correlates with scores on the FES manual skills test. Surg Endosc. 2014;28(11):3081–3085. 7. Ritter EM, Taylor ZA, Wolf KR, et al. Simulation-based
mastery learning for endoscopy using the endoscopy
training system: a strategy to improve endoscopic skills and prepare for the fundamentals of endoscopic surgery (FES) manual skills exam. Surg Endosc. 2018;32(1):413–420. 8. Vassiliou MC, Kaneva PA, Poulose BK, et al. Global
Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc. 2010;24 (8):1834–1841. 9. Ritchie WP Jr., Rhodes RS, Biester TW. Work loads
and practice patterns of general surgeons in the United States, 1995-1997: a report from the American Board of Surgery. Ann Surg. 1999;230(4):533–542. Discussion 42-3.
Journal of Surgical Education Volume 75 /Number 6 November/December 2018
e217