- Email: [email protected]
Su1553 Can Novice Endoscopists Accurately Self-Assess Performance During Their Initial Clinical Colonoscopies? a Prospective, Cross-Sectional Study
Abstracts
demonstrate that progression from low- to high-fidelity simulator training results in superior skill transfer, compared to either low- or high-fidelity simulation alone. Aims: To determine whether an SBT curriculum of progressive fidelity and task complexity improves colonoscopy skill acquisition and transfer to the clinical setting, compared to a curriculum utilizing high-fidelity simulation in isolation. Methods: 37 novice endoscopists were randomized to 2 groups. The progressive group received 6 hours of simulation training, with the first hour on a bench-top simulator (lowfidelity) and 5 hours on a virtual reality (VR) simulator (high-fidelity), performing tasks of sequentially increasing complexity. The high-fidelity group received 6 hours of training on the VR simulator, with tasks arranged in random order of complexity. Both groups received feedback from an expert endoscopist, and 4 hours of lectures. The primary outcome measure was performance during the first 2 colonoscopies in the clinical setting (4-6 weeks after training), assessed by a blinded reviewer using the JAG DOPS scale, a task-specific colonoscopy assessment tool. Secondary outcome measures were differences in: (1) procedural knowledge, evaluated using a multiple-choice test; (2) performance on a VR simulator task (immediately and 4-6 weeks after training), measured by a modified JAG DOPS scale; and (3) performance during an integrated scenario (where participants perform a VR colonoscopy while interacting with a standardized patient) 4-6 weeks after training, measured by JAG DOPS, and validated communication and integrated scenario global rating scales. Results: At baseline, there were no significant differences between groups in demographics, VR performance or procedural knowledge (pO0.05). The progressive group demonstrated superior performance compared to the high-fidelity group on first (p!0.001, dZ1.39), and second (p!0.01, dZ1.18) clinical colonoscopies. The progressive group displayed superior technical skills on the VR simulator at the end of practice (p!0.05, dZ0.96), and performed significantly better during the integrated scenario in terms of communication (p!0.001, dZ0.62), global performance (p!0.001, dZ0.81), and colonoscopy-specific performance (p!0.01, dZ1.51). There was no difference in knowledge acquisition between groups (pO0.05). Conclusion: A SBT curriculum in colonoscopy incorporating progressive fidelity and increasing task complexity is associated with improved skill retention and transfer, as compared to high-fidelity training alone. This finding is commensurate with challenge point theory and suggests that SBT is likely most effective when trainees are exposed to endoscopy in a graded fashion.
Su1552 Comparison of Classroom Didactic Teaching (DT) and Computer Based Self-Learning (Sl) in Recognizing Narrow Band Imaging (NBI) Patterns for Diminutive Polyp (DP) Histology Characterization Taimur Khan*1, Birtukan B. Cinnor4, Neil Gupta2, Lindsay Hosford4, Ajay Bansal1,3, Mojtaba S. Olyaee1, Sachin Wani4, Amit Rastogi1,3 1 Gastroenterology, University of Kansas School of Medicine, Kansas City, KS; 2Gastroenterology, Loyola University Medical Center, Maywood, IL; 3 Gastroenterology, Kansas City VA Medical Center, Kansas City, MO; 4 Gastroenterology, University of Colorado, Aurora, CO Background: The “diagnose, resect and discard” strategy for DP and “do not resect” strategy for diminutive hyperplastic polyps in the recto-sigmoid have been proposed with significant cost saving estimates. While experts can accurately predict DP histology, teaching methods to train non-experts have not been compared. Aim: To compare the accuracy in DP histology characterization between participants undergoing classroom DT versus computer based SL of NBI polyp pattern recognition. Methods: Trainees, both in GI fellowship and Internal Medicine (IM) Residency programs at two institutions, without any prior experience in NBI, were invited to participate. They were randomized to classroom DT and SL group. For the classroom DT group, an experienced endoscopist at each institution reviewed a 20minute audio-visual Power Point (Microsoft Inc.) presentation. This detailed NBI patterns for adenomas and hyperplastic polyps, with several images. Then 40 short videos of DP under NBI were shown to the group with interactive discussion regarding patterns seen and actual histology. Participants in the SL group reviewed the same presentation with the 40 teaching videos and the answer key on their own without any interactive discussion. After this, all trainees reviewed a testing set of 40 videos of DP under NBI and entered their responses: pattern recognized, predicted histology, confidence in diagnosis, are you confident not to send polyp for histology, quality of the video. Predicted histology was compared to actual histology to calculate the performance characteristics. Fisher’s exact test was used and p ! 0.05 was considered as significant. Results: A total of 17 trainees (15 GI fellows, 2 residents in IM) participated: 8 (7 fellows and 1 resident) in classroom DT group and 9 (8 fellows, 1 resident) in SL group. There was no difference in the accuracy of histology characterization between the 2 groups (83.4% vs 87.2%; p Z 0.19) - Table 1. Similar results were noted comparing sensitivity and specificity. A larger proportion of polyps were diagnosed with high confidence (HC) in the classroom DT group (66.5% vs 50.8%; p!0.01). However, as shown in Table 1 the sensitivity and the accuracy of HC predictions were higher in the SL group. Performance characteristics of trainees at the two institutions were similar (Table 2) except for a higher proportion of HC predictions at Site 1. Conclusion: There was no difference in the ability to characterize DP histology between participants exposed to two different teaching methods. SL with a well-designed teaching module showed results at par or even superior (for HC predictions) to classroom DT. These results are encouraging
www.giejournal.org
as computer based SL is more amenable for widespread dissemination compared to classroom DT. This can then help in widespread clinical implementation of real-time polyp histology characterization. Table 1. Performance Characteristics for Predicting Adenomatous Histology Diagnostic Characteristics All Polyps Sensitivity (%) Specificity (%) Accuracy (%) % High Confidence High Confidence only Sensitivity (%) Specificity (%) Accuracy (%)
Classroom Didactic Teaching
SelfLearning
p Value
85.6 79.5 83.4 66.5
88.5 84.9 87.2 50.8
0.4 0.31 0.19 !0.01
86.9 83.1 85.7
95 91.9 93.9
0.03 0.18 0.01
Table 2. Performance Characteristics of Trainees at the Two
Institutions Diagnostic Characteristics All Polyps Sensitivity (%) Specificity (%) Accuracy (%) % High Confidence High Confidence only Sensitivity (%) Specificity (%) Accuracy (%)
Site 1
Site 2
p Value
88 80.4 85.3 69
86.3 84.1 85.6 48.4
0.67 0.5 O0.99 !0.01
88.5 86.1 87.7
93.1 89.1 91.8
0.29 0.79 0.24
Su1553 Can Novice Endoscopists Accurately Self-Assess Performance During Their Initial Clinical Colonoscopies? a Prospective, CrossSectional Study Jennifer M. Amadio, Catharine M. Walsh, Michael A. Scaffidi, Ankit Garg, Samir C. Grover* University of Toronto, Toronto, ON, Canada Background: Self-assessment is recognized as an important and integral component of health professions education and expertise development. With respect to endoscopy, the ability of novices to self-assess allows for self-regulated learning to take place. There is no prior literature examining the self-assessment accuracy of novice colonoscopists. Aims: To determine if novice endoscopists, who have performed less than 20 prior procedures, can accurately self-assess their ability to perform colonoscopy in the clinical setting. Methods: Twenty-two novice endoscopists (less than 20 prior procedures) were delivered an introductory endoscopy training program consisting of 4 hours of lectures and 8 hours of intensive endoscopic virtual-reality simulation-based training. Upon completion of training, participants performed two clinical colonoscopies. Video recordings of each colonoscopy procedure were assessed by two blinded expert endoscopists. Assessments utilized the Joint Advisory Group on Gastrointestinal Endoscopy’s Direct Observation of Procedure or Skills (JAG DOPS), a validated structured procedurespecific assessment tool. Immediately following each colonoscopy, participants also self-assessed performance using the JAG DOPS tool. Pearson’s coefficient was used to determine the correlation between self and assessors’ scores. Results: Mean JAG DOPS self-assessment scores were 57.3% (SDZ17.2%) and 56.5% (SDZ24.2%) for participants’ first and second colonoscopies, respectively. Mean expert assessment scores were 51.0% (SDZ21.9%) and 49.0% (SDZ25.4%) for participants’ first and second colonoscopies, respectively. There was excellent inter-rater reliability between the two expert raters: Pearson’s correlation coefficient was rZ0.80, rZ0.81, and rZ0.79, respectively, for the first colonoscopy, second colonoscopy, and both colonoscopies combined (all p!0.05). When comparing novices’ ability to evaluate their performance to expert assessments, the Pearson’s correlation coefficient was rZ0.21 (p!0.05) for participants’ first clinical colonoscopy, rZ0.43 (p!0.05) for their second clinical colonoscopy and rZ0.34 (p!0.05) overall for both clinical colonoscopies combined. Conclusions: This study shows that residents’ self-assessment accuracy in clinical colonoscopy is poor. This is comparable to prior literature indicating that there is weak correlation between expert and self-assessments during simulated colonoscopic polypectomy by novices. Further research is required to examine interventions intended to potentially improve self-assessment accuracy, such as video-based self-observation or exposure to relevant “benchmark” standards of performance.
Volume 81, No. 5S : 2015 GASTROINTESTINAL ENDOSCOPY AB325
demonstrate that progression from low- to high-fidelity simulator training results in superior skill transfer, compared to either low- or high-fidelity simulation alone. Aims: To determine whether an SBT curriculum of progressive fidelity and task complexity improves colonoscopy skill acquisition and transfer to the clinical setting, compared to a curriculum utilizing high-fidelity simulation in isolation. Methods: 37 novice endoscopists were randomized to 2 groups. The progressive group received 6 hours of simulation training, with the first hour on a bench-top simulator (lowfidelity) and 5 hours on a virtual reality (VR) simulator (high-fidelity), performing tasks of sequentially increasing complexity. The high-fidelity group received 6 hours of training on the VR simulator, with tasks arranged in random order of complexity. Both groups received feedback from an expert endoscopist, and 4 hours of lectures. The primary outcome measure was performance during the first 2 colonoscopies in the clinical setting (4-6 weeks after training), assessed by a blinded reviewer using the JAG DOPS scale, a task-specific colonoscopy assessment tool. Secondary outcome measures were differences in: (1) procedural knowledge, evaluated using a multiple-choice test; (2) performance on a VR simulator task (immediately and 4-6 weeks after training), measured by a modified JAG DOPS scale; and (3) performance during an integrated scenario (where participants perform a VR colonoscopy while interacting with a standardized patient) 4-6 weeks after training, measured by JAG DOPS, and validated communication and integrated scenario global rating scales. Results: At baseline, there were no significant differences between groups in demographics, VR performance or procedural knowledge (pO0.05). The progressive group demonstrated superior performance compared to the high-fidelity group on first (p!0.001, dZ1.39), and second (p!0.01, dZ1.18) clinical colonoscopies. The progressive group displayed superior technical skills on the VR simulator at the end of practice (p!0.05, dZ0.96), and performed significantly better during the integrated scenario in terms of communication (p!0.001, dZ0.62), global performance (p!0.001, dZ0.81), and colonoscopy-specific performance (p!0.01, dZ1.51). There was no difference in knowledge acquisition between groups (pO0.05). Conclusion: A SBT curriculum in colonoscopy incorporating progressive fidelity and increasing task complexity is associated with improved skill retention and transfer, as compared to high-fidelity training alone. This finding is commensurate with challenge point theory and suggests that SBT is likely most effective when trainees are exposed to endoscopy in a graded fashion.
Su1552 Comparison of Classroom Didactic Teaching (DT) and Computer Based Self-Learning (Sl) in Recognizing Narrow Band Imaging (NBI) Patterns for Diminutive Polyp (DP) Histology Characterization Taimur Khan*1, Birtukan B. Cinnor4, Neil Gupta2, Lindsay Hosford4, Ajay Bansal1,3, Mojtaba S. Olyaee1, Sachin Wani4, Amit Rastogi1,3 1 Gastroenterology, University of Kansas School of Medicine, Kansas City, KS; 2Gastroenterology, Loyola University Medical Center, Maywood, IL; 3 Gastroenterology, Kansas City VA Medical Center, Kansas City, MO; 4 Gastroenterology, University of Colorado, Aurora, CO Background: The “diagnose, resect and discard” strategy for DP and “do not resect” strategy for diminutive hyperplastic polyps in the recto-sigmoid have been proposed with significant cost saving estimates. While experts can accurately predict DP histology, teaching methods to train non-experts have not been compared. Aim: To compare the accuracy in DP histology characterization between participants undergoing classroom DT versus computer based SL of NBI polyp pattern recognition. Methods: Trainees, both in GI fellowship and Internal Medicine (IM) Residency programs at two institutions, without any prior experience in NBI, were invited to participate. They were randomized to classroom DT and SL group. For the classroom DT group, an experienced endoscopist at each institution reviewed a 20minute audio-visual Power Point (Microsoft Inc.) presentation. This detailed NBI patterns for adenomas and hyperplastic polyps, with several images. Then 40 short videos of DP under NBI were shown to the group with interactive discussion regarding patterns seen and actual histology. Participants in the SL group reviewed the same presentation with the 40 teaching videos and the answer key on their own without any interactive discussion. After this, all trainees reviewed a testing set of 40 videos of DP under NBI and entered their responses: pattern recognized, predicted histology, confidence in diagnosis, are you confident not to send polyp for histology, quality of the video. Predicted histology was compared to actual histology to calculate the performance characteristics. Fisher’s exact test was used and p ! 0.05 was considered as significant. Results: A total of 17 trainees (15 GI fellows, 2 residents in IM) participated: 8 (7 fellows and 1 resident) in classroom DT group and 9 (8 fellows, 1 resident) in SL group. There was no difference in the accuracy of histology characterization between the 2 groups (83.4% vs 87.2%; p Z 0.19) - Table 1. Similar results were noted comparing sensitivity and specificity. A larger proportion of polyps were diagnosed with high confidence (HC) in the classroom DT group (66.5% vs 50.8%; p!0.01). However, as shown in Table 1 the sensitivity and the accuracy of HC predictions were higher in the SL group. Performance characteristics of trainees at the two institutions were similar (Table 2) except for a higher proportion of HC predictions at Site 1. Conclusion: There was no difference in the ability to characterize DP histology between participants exposed to two different teaching methods. SL with a well-designed teaching module showed results at par or even superior (for HC predictions) to classroom DT. These results are encouraging
www.giejournal.org
as computer based SL is more amenable for widespread dissemination compared to classroom DT. This can then help in widespread clinical implementation of real-time polyp histology characterization. Table 1. Performance Characteristics for Predicting Adenomatous Histology Diagnostic Characteristics All Polyps Sensitivity (%) Specificity (%) Accuracy (%) % High Confidence High Confidence only Sensitivity (%) Specificity (%) Accuracy (%)
Classroom Didactic Teaching
SelfLearning
p Value
85.6 79.5 83.4 66.5
88.5 84.9 87.2 50.8
0.4 0.31 0.19 !0.01
86.9 83.1 85.7
95 91.9 93.9
0.03 0.18 0.01
Table 2. Performance Characteristics of Trainees at the Two
Institutions Diagnostic Characteristics All Polyps Sensitivity (%) Specificity (%) Accuracy (%) % High Confidence High Confidence only Sensitivity (%) Specificity (%) Accuracy (%)
Site 1
Site 2
p Value
88 80.4 85.3 69
86.3 84.1 85.6 48.4
0.67 0.5 O0.99 !0.01
88.5 86.1 87.7
93.1 89.1 91.8
0.29 0.79 0.24
Su1553 Can Novice Endoscopists Accurately Self-Assess Performance During Their Initial Clinical Colonoscopies? a Prospective, CrossSectional Study Jennifer M. Amadio, Catharine M. Walsh, Michael A. Scaffidi, Ankit Garg, Samir C. Grover* University of Toronto, Toronto, ON, Canada Background: Self-assessment is recognized as an important and integral component of health professions education and expertise development. With respect to endoscopy, the ability of novices to self-assess allows for self-regulated learning to take place. There is no prior literature examining the self-assessment accuracy of novice colonoscopists. Aims: To determine if novice endoscopists, who have performed less than 20 prior procedures, can accurately self-assess their ability to perform colonoscopy in the clinical setting. Methods: Twenty-two novice endoscopists (less than 20 prior procedures) were delivered an introductory endoscopy training program consisting of 4 hours of lectures and 8 hours of intensive endoscopic virtual-reality simulation-based training. Upon completion of training, participants performed two clinical colonoscopies. Video recordings of each colonoscopy procedure were assessed by two blinded expert endoscopists. Assessments utilized the Joint Advisory Group on Gastrointestinal Endoscopy’s Direct Observation of Procedure or Skills (JAG DOPS), a validated structured procedurespecific assessment tool. Immediately following each colonoscopy, participants also self-assessed performance using the JAG DOPS tool. Pearson’s coefficient was used to determine the correlation between self and assessors’ scores. Results: Mean JAG DOPS self-assessment scores were 57.3% (SDZ17.2%) and 56.5% (SDZ24.2%) for participants’ first and second colonoscopies, respectively. Mean expert assessment scores were 51.0% (SDZ21.9%) and 49.0% (SDZ25.4%) for participants’ first and second colonoscopies, respectively. There was excellent inter-rater reliability between the two expert raters: Pearson’s correlation coefficient was rZ0.80, rZ0.81, and rZ0.79, respectively, for the first colonoscopy, second colonoscopy, and both colonoscopies combined (all p!0.05). When comparing novices’ ability to evaluate their performance to expert assessments, the Pearson’s correlation coefficient was rZ0.21 (p!0.05) for participants’ first clinical colonoscopy, rZ0.43 (p!0.05) for their second clinical colonoscopy and rZ0.34 (p!0.05) overall for both clinical colonoscopies combined. Conclusions: This study shows that residents’ self-assessment accuracy in clinical colonoscopy is poor. This is comparable to prior literature indicating that there is weak correlation between expert and self-assessments during simulated colonoscopic polypectomy by novices. Further research is required to examine interventions intended to potentially improve self-assessment accuracy, such as video-based self-observation or exposure to relevant “benchmark” standards of performance.
Volume 81, No. 5S : 2015 GASTROINTESTINAL ENDOSCOPY AB325