- Email: [email protected]
Training with a computer-based simulator achieves basic manual skills required for upper endoscopy: a randomized controlled trial
Training with a computer-based simulator achieves basic manual skills required for upper endoscopy: a randomized controlled trial Emilio Di Giulio, MD, Diego Fregonese, MD, Tino Casetti, MD, Renzo Cestari, MD, Fausto Chilovi, MD, Giancarlo D’Ambra, MD, Giovanni Di Matteo, MD, Leonardo Ficano, MD, Gianfranco Delle Fave, MD Roma, Camposampiero, Ravenna, Brescia, Bolzano, Castellana Grotte, and Palermo, Italy
Background: Changes in medical practice have constrained the time available for education and the availability of patients for training. Computer-based simulators have been devised that can be used to achieve manual skills without patient contact. This study prospectively compared, in a clinical setting, the efficacy of a computer-based simulator for training in upper endoscopy. Methods: Twenty-two fellows with no experience in endoscopy were randomly assigned to two groups: one group underwent 10 hours of preclinical training with a computer-based simulator, and the other did not. Each trainee performed upper endoscopy in 19 or 20 patients. Performance parameters evaluated included the following: esophageal intubation, procedure duration and completeness, and request for assistance. The performance of the trainees also was evaluated by the endoscopy instructor. Results: A total of 420 upper endoscopies were performed; the computer pretrained group performed 212 and the non-pretrained group, 208. The pretrained group performed more complete procedures (87.8% vs. 70.0%; p < 0.0001), required less assistance (41.3% vs. 97.9%; p < 0.0001), and the instructor assessed performance as ‘‘positive’’ more often for this group (86.8% vs. 56.7%; p < 0.0001). The length of procedures was comparable for the two groups. Conclusions: The computer-based simulator is effective in providing novice trainees with the skills needed for identification of anatomical landmarks and basic endoscopic maneuvers, and in reducing the need for assistance by instructors. (Gastrointest Endosc 2004;60:196-200.)
A primary aim of training programs, as defined by international professional societies devoted to GI endoscopy, is to ensure that graduates can perform specific procedures safely, rapidly, and completely.1 Among the greatest obstacles to teaching and learning GI endoscopy is the difficulty of measuring progress, including manual skills.2,3 Initially, the Received August 7, 2003. For revision November 12, 2003. Accepted April 3, 2004. Current affiliations: Divisione di Malattie Digestive e del Fegato, Ospedale Sant ‘Andrea, Universita`’ di Roma ‘‘La Sapienza,’’ Rome, Servizio di Gastroenterologia Clinica, Ospedale Civile, Camposampiero, Servizio di Gastroenterologia e Dietetica Clinica, Ospedale Santa Maria delle Croci, AUSL, Ravenna, Cattedra di Chirurgia Generale, Centro di Endoscopia Digestiva, Universita` di Brescia, Brescia, Divisione di Gastroenterologia, Ospedale Regionale, Bolzano, Ospedale Civile, IRCCS, Castellana Grotte, Dipartimento di Discipline Chirurgiche ed Oncologiche, Universita` di Palermo, Palermo, Italy. The present study was supported by an educational grant from Bracco SpA, Milano, Italy. Presented, in part, as a poster at the annual meeting of Digestive Diseases Week, May 20-23, 2001, Atlanta, Georgia (Gastrointest Endosc 2001;53:AB81). Reprint requests: Emilio Di Giulio, MD, Via della Balduina 80, 00136 Roma, Italy. Copyright Ó 2004 by the American Society for Gastrointestinal Endoscopy 0016-5107/$30.00 PII: S0016-5107(04)01566-4 196
GASTROINTESTINAL ENDOSCOPY
approach to standardized and objective training was to define a specific minimum number of supervised procedures at which proficiency could reasonably be assessed, the so-called threshold for assessing competence.1 Because of the variable rates of learning among individuals, however, the evaluation of manual skills has to be tailored to each trainee, and, thus, standards for assessment of performance are needed that are independent of the number of procedures performed under supervision.4 Several training modalities are available. Supervised participation in the daily performance of procedures has the advantage of offering direct experience in the clinical setting but also the disadvantages of being time consuming, both for trainees and instructors, as well as being unpleasant, and occasionally unsafe, for patients.5 Demonstrations of procedures offer the trainee direct observation of the skills and the techniques of experts in complex clinical contexts, but this also reduces trainees to the role of passive observer and also may pose certain risks for patients.6 Simulators are, theoretically, ideal for training and have several advantages. They offer a direct experience for trainees in the manual manipulation of an endoscope and eliminate discomfort and risk for patients. They reduce the time VOLUME 60, NO. 2, 2004
EGD: computer-based simulator training achieves basic manual skills
committed to training for instructors, and they are not subject to the time constraints typically associated with supervised instruction. Simulators based on various technologies are now available, both for diagnostic and therapeutic GI endoscopy.7-14 Studies have evaluated different aspects of computer-based simulators. Two assessed whether the measurements of performance, on a computer-based simulator, can differentiate operators with varying levels of competence.15,16 In these studies, which used two different computer-based simulators, varying levels of competence were detected. However, neither study included an evaluation of skills presumably acquired with the simulator in a clinical setting, i.e., an assessment of the expected advantages of preclinical training with simulator in the care of real patients. One randomized controlled trial in which training with a computer-based simulator was compared with traditional bedside training in flexible sigmoidoscopy did not demonstrate any advantage for a computer-based simulator when the subjects performed actual examinations in 5 patients.17 This study evaluated the usefulness of a computerbased simulator for upper-GI endoscopy by defining certain indicators of satisfactory performance and then comparing subjects who received training with the simulator to those who did not during performance of upper endoscopy in patients. MATERIALS AND METHODS Simulator The interactive computer-based simulator (GI-Mentor; Simbionix Ltd., Lod, Israel) used in the study includes a specially designed mannequin, a computer simulation program, an endoscope and accessories specifically devised by the manufacturer, a force-feedback system, and 10 scenarios for upper endoscopy in realistic clinical situations. The case scenarios are arranged from simple to more difficult. The system records the time taken to reach the duodenum, the lesions correctly identified, maneuvers regarded as inappropriate, and excessive wall pressure or air insufflation. Study design Twenty-two trainees in the early phase of training in gastroenterology (5-year program) from 7 teaching hospitals took part in the study between March and May 2000. No trainee had prior direct experience with the performance of endoscopy. Each trainee was supervised by an expert endoscopist (instructor) with experience in endoscopic training. After a 2-hour session in which the workings of the endoscope were explained and correct methods for performance of upper endoscopy were described, trainees were randomized into two groups (SIM group and non-SIM group) by using randomization lists VOLUME 60, NO. 2, 2004
E Giulio, D Fregonese, T Casetti, et al.
created independently in each hospital. Eleven were randomized to the SIM group and were required to undergo 10 hours of preclinical training by using the simulator; the 11 trainees randomized to the non-SIM group did not receive training with the simulator. Computer-based simulator before training Trainees randomized to the SIM group were given basic directions by the instructors for use of the simulator and then completed the 10 hours preclinical training in 3 to 5 sessions without supervision. They were permitted to try each of the 10 simulated cases available on the simulator within the times and in the sequence they preferred, without any supervision by the instructor. Performance evaluation All 22 trainees were required to perform upper endoscopy in 20 consecutive patients scheduled for diagnostic endoscopy. Patient exclusion criteria were the following: age less than 18 years, pregnancy, prior digestive surgery, major risk factors for the procedure (severe respiratory failure, severe cardiac failure, patients in an intensive care unit, GI bleeding), coagulation abnormalities, and dysphagia. For the procedure, patients were premedicated with midazolam (2.5 mg intravenously) or diazepam (5 mg intravenously); topical pharyngeal anesthesia was induced by spraying lidocaine. Continuous supervision by the instructor was required throughout each procedure. At the end of each procedure, trainees were required to record the following in a logbook: the duration of the procedure (defined as the length of time the light source was switched on), the number of attempts at intubation, and, in the event of failure, the reasons for interruption of the procedure and/or the need of assistance in completing the procedure. Failed esophageal intubation was defined as 3 unsuccessful attempts. When esophageal intubation was achieved, the procedures were defined as ‘‘complete’’ only if trainees also were able to identify, within 20 minutes, all anatomical landmarks (esophagogastric mucosal junction, gastric angulus, pylorus) and to perform certain basic maneuvers (aspiration of gastric juice, pyloric intubation [no more than 3 attempts], duodenal bulb exploration, intubation of the second part of the duodenum, and retroflexion). Incomplete procedures were defined as ‘‘procedure failures.’’ When a trainee required assistance but was able to complete the procedure with only verbal direction and advice, and without manual assistance, the procedure was judged as ‘‘complete.’’ The number of times assistance was required and reasons for this were recorded. A procedure was defined as ‘‘not complete’’ when manual assistance from the instructor was required. Instructors were required to record the following: the ‘‘completeness’’ of the procedure, the number of identified or missed lesions, and overall subjective judgment of the performance, expressed as a score from 0 (bad) to 10 (good), based on the completeness of the examination, the need for assistance and the presumed difficulty of the procedure (i.e., vigorous peristalsis, pyloric spasm, bulbar deformity, GASTROINTESTINAL ENDOSCOPY
197
E Giulio, D Fregonese, T Casetti, et al.
EGD: computer-based simulator training achieves basic manual skills
Table 1. Characteristics of patients
Male gender Mean age (SD), y Indication Dyspepsia Follow-up Iron deficiency anemia Screening for varices Others Endoscopic findings Negative Esophagitis Gastritis Neoplasia Peptic ulcer Duodenal fold anomalies Barrett’s esophagus Other
SIM group (n = 204)
Non-SIM group (n = 203)
n (%)
n (%)
p Value
72 (35.3) 52.5 (16.3)
75 (36.9) 52.2 (14.7)
0.757 0.889
134 42 16 4 8
(65.7) (20.6) (7.8) (2.0) (3.9)
128 51 11 8 5
(63.0) (25.1) (5.4) (3.9) (2.5)
0.605 0.289 0.426 0.259 0.574
161 16 9 4 8 1 4 1
(78.9) (7.8) (4.4) (2.0) (3.9) (0.5) (2.0) (0.5)
157 20 6 2 9 0 3 6
(77.3) (9.8) (3.0) (1.0) (4.4)
0.720 0.490 0.600 0.685 0.810 1.000 1.000 0.067
(1.5) (3.0)
SIM, Simulator; SD, standard deviation.
patient discomfort). If the score was 5 or less, the procedure was classified, by the instructor, as ‘‘negative’’; for scores of 6 or greater, the procedure was classified as ‘‘positive.’’ Instructors also were asked to record complications. The instructors were not blinded as to whether trainees had or had not used the simulator. Informed consent was obtained from all patients. The study was approved by the institutional review board of each participating center. The study was not funded by the company that manufactures the simulator, and none of the investigators has any financial relationship with the company. Statistical analysis Data are expressed as mean (6 standard error of the mean) and are evaluated by means of appropriate statistical tests (t test or Mann-Whitney U test). Relative frequencies for subgroups (percentages) were compared by using the Fisher exact test. A p value <0.05 was considered statistically significant. It is recognized that there was multiple testing of outcome data; however, correction of p values by the method of Bonferroni would not have removed any finding of statistical significance.
RESULTS All trainees randomized to the SIM group completed the 10 hours of preclinical training, as planned. More than 200 procedures were performed in patients per each group. The trainees performed up to 20 upper-GI procedures. However, 6 trainees in the SIM group and 7 in the non-SIM group performed one or two procedures less than planned because of 198
GASTROINTESTINAL ENDOSCOPY
the temporary assignment to other clinical activities, with the result that the sequence of consecutive procedures was interrupted. No attempted procedure was excluded from statistical analysis. The characteristics of the patients assigned to the two groups of trainees were comparable (Table 1). No difference was observed between the two groups in terms of failure to effect esophageal intubation (SIM group, 8/212; non-SIM group, 5/208) or the number of attempts (SIM group, mean 1.7; non-SIM group, mean 1.8), with 96% to 98% of attempts being successful in both groups. Trainees in the SIM group were able to complete, without manual assistance, significantly more procedures than trainees in the no-SIM group; the SIM group also performed significantly more complete procedures without verbal assistance and received significantly more ‘‘positive’’ judgments from instructors (Table 2). Verbal assistance was required in both the SIM and non-SIM groups. The main reasons for requesting verbal assistance were the following: pylorus intubation (SIM group, 16%; non-SIM group, 21%), retroflexion (SIM group, 7%; non-SIM group, 15%), and intubation of the second part of the duodenum (SIM group, 3%; non-SIM group, 18%). There was no difference in the time taken to achieve a complete procedure (mean 10.5 minutes, SIM group; mean 12.4 minutes, non-SIM group). No complication occurred in either group. DISCUSSION Endoscopy simulators have several expected advantages: patients are spared prolonged, possibly incomplete and painful procedures; the number of supervised procedures required to achieve competence is reduced; the level of assistance from instructors is reduced; and the learning time is shortened. To our knowledge, there is no prospective randomized study of the efficacy of a computer-based upper-endoscopy simulator in which its potential contributions to training are evaluated in a clinical setting. Esophageal intubation is one of the more difficult maneuvers for trainees and patients. In this study, the number of successful intubations and the number of intubation attempts were similar for trainees who received preclinical training with the simulator and those who did not. Instructors who participated in the present study judged the simulated anatomy of the oral cavity and the pharynx to be not sufficiently realistic and inadequate in terms of reproduction of the mechanical resistance characteristics of this anatomic region. In effect, simulation occurs only when the endoscope is introduced about 20 cm. VOLUME 60, NO. 2, 2004
EGD: computer-based simulator training achieves basic manual skills
E Giulio, D Fregonese, T Casetti, et al.
Table 2. Results SIM group 204 Non-SIM group 203
Complete procedures Non assistance Instructor’s judgment Positive Negative
n/total (%)
n/total (%)
Fisher test
OR (95%CI)
179/204 (87.7) 105/179 (58.6)
142/203 (69.9) 3/142 (2.1)
p < 0.0001 3.08 (1.8, 5.1) p < 0.0001 65.7 (20.1, 214.4)
177/204 (86.8) 27/204 (13.2)
115/203 (56.7) 88/203 (43.3)
p < 0.0001
5.0 (3.1, 8.2)
SIM, Simulator; OR, odds ratio; CI, confidence interval.
The results of the present study confirm the generally held opinion that preclinical training with a GI endoscopy simulator may not impart the skills needed for esophageal intubation.7,9 A major but difficult goal in the evaluation of training programs is to define standardized and measurable parameters of acquired skill.2-4 The completeness of the examination, defined as the ability to identify and to explore all anatomical regions, is a main criterion. The anatomic knowledge acquired with the assistance of an instructor is another basic criterion. The major limitation in the present study is that the instructors who evaluated the trainees were not blinded as to whether they received preclinical training by using the simulator. To improve the reliability of the evaluation, two cross-evaluation methods were used: (1) the ‘‘completeness of the procedure in a given time’’ by using previously defined standards and the ‘‘need for assistance’’ as reported by trainees; and (2) an ‘‘overall judgment’’ by instructors, which included not only the completeness of the procedure and the need for assistance but also their assessment of the difficulty of each specific examination. The second method of evaluation is more subjective. The American Society for Gastrointestinal Endoscopy (ASGE) has defined objective performance criteria for upper-GI endoscopy (e.g., esophageal intubation, pyloric intubation).4 Additional evaluation criteria were added for the purposes of the present study. Moreover, the ASGE advises that one of the expected effects of progressive skill acquisition is a reduced need for supervision.4 The present study evaluated only technical skill and not clinical competence. Therefore, only the request for assistance to perform a ‘‘complete procedure’’ was evaluated. The data reveal that trainees and instructors were in agreement regarding the efficacy of simulator pretraining, the ability to perform a ‘‘complete procedure’’ and to gain confidence in the performance of upper endoscopy without instructor assistance. In the SIM group, trainees were able to VOLUME 60, NO. 2, 2004
perform a complete procedure without any assistance in almost 60% of cases vs. 2% in the non-SIM group. The request for assistance in pyloric intubation was comparable in the two groups (29 cases, SIM group; 30 cases, non-SIM group), although more assistance was required in the non-SIM group for retroflexion (12 vs. 21) and intubation of the second part of the duodenum (5 vs. 25). The simulator used in the present study does not reproduce the mechanical resistance typically encountered in vivo during esophageal or pyloric intubation. The results of the current study demonstrate that preclinical training with the simulator is ineffective when mechanical resistance plays a role; but, contrary to the opinion of others,8 the simulator does provide a good notion of orientation in space, endoscope manipulation, and progression in the performance of complex maneuvers (i.e., retroflexion, intubation of the second part of the duodenum). The 3-dimensional quality and the tactile feedback provided by the simulator during these maneuvers are sufficiently realistic to allow the trainee to improve hand-eye skills. The design of the present study arbitrarily predefined simulator training as 10 hours. Moreover, use of the simulator was not permitted during the in vivo evaluation portion of the study. However, a more effective contribution of the simulator can be expected when trainees are allowed unrestricted use of the machine before and after supervised instruction in the performance of procedures in patients is initiated. The simulator used in the current study records some performance parameters (time to reach duodenum, correct identification of lesions, maneuvers regarded as inappropriate, and excessive wall pressure or air inflation). In our opinion, regardless of the time specified for training with the simulator, for individual trainees, it is important to reach a high standard of performance with the simulator before undertakingsupervisedproceduresinpatients.Moreover, a much more vast and varied file of simulated GASTROINTESTINAL ENDOSCOPY
199
E Giulio, D Fregonese, T Casetti, et al.
EGD: computer-based simulator training achieves basic manual skills
cases of varying difficulty would be useful and would probably enhance the learning process for trainees. Teaching is time consuming for instructors, and an expected advantage of the simulator is a reduction in the need for instructor assistance. In this respect, the results of the present study were positive because instructor assistance was not permitted during preclinical training with the simulator. Nevertheless, compared with the non-SIM group, the performance of the SIM group was superior during actual endoscopic procedures in patients. In conclusion, the present study found the GIMentor simulator to be efficient and effective in teaching the basic manual skills required for upper endoscopy, with the exception of esophageal intubation. While this computer-based simulator may be considered useful for training purposes, improvements in the technology are necessary, in particular, the simulation of mechanical resistance in certain anatomical areas. REFERENCES 1. Principles of training in gastrointestinal endoscopy. Manchester (MA): American Society for Gastrointestinal Endoscopy; 1991. 2. Cass OW, Freeman ML, Peine CJ, Zera RT, Onstad GR. Objective evaluation of endoscopy skills during training. Ann Intern Med 1993;118:40-4. 3. Bond JH. Evaluation of trainee competence. Gastrointest Endosc Clin N Am 1995;5:337-46. 4. ASGE guidelines: principles of training in gastrointestinal endoscopy. Gastrointest Endosc 1999;49:845-53. 5. Bini EJ, Firoozi B, Choung RJ, Ali EM, Osman M, Weinshel EH. Systematic evaluation of complications related to
6. 7. 8.
9.
10. 11. 12.
13.
14.
15.
16.
17.
endoscopy in a training setting: a prospective 30-day outcomes study. Gastrointest Endosc 2003;57:8-16. Cotton P. Live endoscopy demonstrations are great, but . . . [editorial]. Gastrointest Endosc 2000;51:627-9. Gessner CE, Jowell PS, Baillie J. Novel methods for endoscopic training. Gastrointest Endosc Clin N Am 1995;5:323-36. Hochberger J, Maiss J, Hahn E. The use of simulators for training in GI endoscopy [editorial]. Endoscopy 2002;34: 727-9. Hochberger J, Maiss J, Magdeburg B, Cohen J, Hahn EG. Training simulators and education in gastrointestinal endoscopy: current status and perspectives in 2001 [editorial]. Endoscopy 2001;33:541-9. Adamsen S. Simulators and gastrointestinal endoscopy training [editorial]. Endoscopy 2000;32:895-7. Bar-Meir S. A new endoscopic simulator [editorial]. Endoscopy 2000;32:898-900. Williams CB, Saunders BP, Bladen JS. Development of colonoscopy teaching simulation [editorial]. Endoscopy 2000;32: 901-5. Neumann M, Mayer G, Ell C, Felzmann T, Reinbruber B, Horbach T, et al. The Erlangen Endo-Trainer: life-like simulation for diagnostic and interventional endoscopic retrograde cholangiography. Endoscopy 2000;32:906-10. Gerson LB, Van Dam J. The future of simulators in GI endoscopy: an unlikely possibility or a virtual reality? [editorial] Gastrointest Endosc 2002;55:608-11. Sedlack RE, Kolars JC. Validation of computer-based endoscopy simulators in training [abstract]. Gastrointest Endosc 2002;55:AB77. Ferlitsch A, Glauninger P, Gupper A, Schillinger M, Haefner M, Gangl A, et al. Evaluation of a virtual endoscopy simulator for training in gastrointestinal endoscopy. Endoscopy 2002;34:698-702. Gerson LB, Van Dam J. A randomized controlled trial comparing an endoscopic simulator with traditional bedside teaching for training in flexible sigmoidoscopy [abstract]. Gastrointest Endosc 2002;55:AB78.
Submission of ACCEPTED manuscript on diskette Gastrointestinal Endoscopy strongly encourages the submission of final manuscripts on disk. Although files created with WordPerfect are preferred, please send your final manuscript in any electronic format. On your disk, please indicate computer system (e.g., IBM, MacIntosh) and word processing software used (e.g., WordPerfect 6.1). Please refer to complete Instructions to Authors in the most recent January or July issue of the Journal.
200
GASTROINTESTINAL ENDOSCOPY
VOLUME 60, NO. 2, 2004
Background: Changes in medical practice have constrained the time available for education and the availability of patients for training. Computer-based simulators have been devised that can be used to achieve manual skills without patient contact. This study prospectively compared, in a clinical setting, the efficacy of a computer-based simulator for training in upper endoscopy. Methods: Twenty-two fellows with no experience in endoscopy were randomly assigned to two groups: one group underwent 10 hours of preclinical training with a computer-based simulator, and the other did not. Each trainee performed upper endoscopy in 19 or 20 patients. Performance parameters evaluated included the following: esophageal intubation, procedure duration and completeness, and request for assistance. The performance of the trainees also was evaluated by the endoscopy instructor. Results: A total of 420 upper endoscopies were performed; the computer pretrained group performed 212 and the non-pretrained group, 208. The pretrained group performed more complete procedures (87.8% vs. 70.0%; p < 0.0001), required less assistance (41.3% vs. 97.9%; p < 0.0001), and the instructor assessed performance as ‘‘positive’’ more often for this group (86.8% vs. 56.7%; p < 0.0001). The length of procedures was comparable for the two groups. Conclusions: The computer-based simulator is effective in providing novice trainees with the skills needed for identification of anatomical landmarks and basic endoscopic maneuvers, and in reducing the need for assistance by instructors. (Gastrointest Endosc 2004;60:196-200.)
A primary aim of training programs, as defined by international professional societies devoted to GI endoscopy, is to ensure that graduates can perform specific procedures safely, rapidly, and completely.1 Among the greatest obstacles to teaching and learning GI endoscopy is the difficulty of measuring progress, including manual skills.2,3 Initially, the Received August 7, 2003. For revision November 12, 2003. Accepted April 3, 2004. Current affiliations: Divisione di Malattie Digestive e del Fegato, Ospedale Sant ‘Andrea, Universita`’ di Roma ‘‘La Sapienza,’’ Rome, Servizio di Gastroenterologia Clinica, Ospedale Civile, Camposampiero, Servizio di Gastroenterologia e Dietetica Clinica, Ospedale Santa Maria delle Croci, AUSL, Ravenna, Cattedra di Chirurgia Generale, Centro di Endoscopia Digestiva, Universita` di Brescia, Brescia, Divisione di Gastroenterologia, Ospedale Regionale, Bolzano, Ospedale Civile, IRCCS, Castellana Grotte, Dipartimento di Discipline Chirurgiche ed Oncologiche, Universita` di Palermo, Palermo, Italy. The present study was supported by an educational grant from Bracco SpA, Milano, Italy. Presented, in part, as a poster at the annual meeting of Digestive Diseases Week, May 20-23, 2001, Atlanta, Georgia (Gastrointest Endosc 2001;53:AB81). Reprint requests: Emilio Di Giulio, MD, Via della Balduina 80, 00136 Roma, Italy. Copyright Ó 2004 by the American Society for Gastrointestinal Endoscopy 0016-5107/$30.00 PII: S0016-5107(04)01566-4 196
GASTROINTESTINAL ENDOSCOPY
approach to standardized and objective training was to define a specific minimum number of supervised procedures at which proficiency could reasonably be assessed, the so-called threshold for assessing competence.1 Because of the variable rates of learning among individuals, however, the evaluation of manual skills has to be tailored to each trainee, and, thus, standards for assessment of performance are needed that are independent of the number of procedures performed under supervision.4 Several training modalities are available. Supervised participation in the daily performance of procedures has the advantage of offering direct experience in the clinical setting but also the disadvantages of being time consuming, both for trainees and instructors, as well as being unpleasant, and occasionally unsafe, for patients.5 Demonstrations of procedures offer the trainee direct observation of the skills and the techniques of experts in complex clinical contexts, but this also reduces trainees to the role of passive observer and also may pose certain risks for patients.6 Simulators are, theoretically, ideal for training and have several advantages. They offer a direct experience for trainees in the manual manipulation of an endoscope and eliminate discomfort and risk for patients. They reduce the time VOLUME 60, NO. 2, 2004
EGD: computer-based simulator training achieves basic manual skills
committed to training for instructors, and they are not subject to the time constraints typically associated with supervised instruction. Simulators based on various technologies are now available, both for diagnostic and therapeutic GI endoscopy.7-14 Studies have evaluated different aspects of computer-based simulators. Two assessed whether the measurements of performance, on a computer-based simulator, can differentiate operators with varying levels of competence.15,16 In these studies, which used two different computer-based simulators, varying levels of competence were detected. However, neither study included an evaluation of skills presumably acquired with the simulator in a clinical setting, i.e., an assessment of the expected advantages of preclinical training with simulator in the care of real patients. One randomized controlled trial in which training with a computer-based simulator was compared with traditional bedside training in flexible sigmoidoscopy did not demonstrate any advantage for a computer-based simulator when the subjects performed actual examinations in 5 patients.17 This study evaluated the usefulness of a computerbased simulator for upper-GI endoscopy by defining certain indicators of satisfactory performance and then comparing subjects who received training with the simulator to those who did not during performance of upper endoscopy in patients. MATERIALS AND METHODS Simulator The interactive computer-based simulator (GI-Mentor; Simbionix Ltd., Lod, Israel) used in the study includes a specially designed mannequin, a computer simulation program, an endoscope and accessories specifically devised by the manufacturer, a force-feedback system, and 10 scenarios for upper endoscopy in realistic clinical situations. The case scenarios are arranged from simple to more difficult. The system records the time taken to reach the duodenum, the lesions correctly identified, maneuvers regarded as inappropriate, and excessive wall pressure or air insufflation. Study design Twenty-two trainees in the early phase of training in gastroenterology (5-year program) from 7 teaching hospitals took part in the study between March and May 2000. No trainee had prior direct experience with the performance of endoscopy. Each trainee was supervised by an expert endoscopist (instructor) with experience in endoscopic training. After a 2-hour session in which the workings of the endoscope were explained and correct methods for performance of upper endoscopy were described, trainees were randomized into two groups (SIM group and non-SIM group) by using randomization lists VOLUME 60, NO. 2, 2004
E Giulio, D Fregonese, T Casetti, et al.
created independently in each hospital. Eleven were randomized to the SIM group and were required to undergo 10 hours of preclinical training by using the simulator; the 11 trainees randomized to the non-SIM group did not receive training with the simulator. Computer-based simulator before training Trainees randomized to the SIM group were given basic directions by the instructors for use of the simulator and then completed the 10 hours preclinical training in 3 to 5 sessions without supervision. They were permitted to try each of the 10 simulated cases available on the simulator within the times and in the sequence they preferred, without any supervision by the instructor. Performance evaluation All 22 trainees were required to perform upper endoscopy in 20 consecutive patients scheduled for diagnostic endoscopy. Patient exclusion criteria were the following: age less than 18 years, pregnancy, prior digestive surgery, major risk factors for the procedure (severe respiratory failure, severe cardiac failure, patients in an intensive care unit, GI bleeding), coagulation abnormalities, and dysphagia. For the procedure, patients were premedicated with midazolam (2.5 mg intravenously) or diazepam (5 mg intravenously); topical pharyngeal anesthesia was induced by spraying lidocaine. Continuous supervision by the instructor was required throughout each procedure. At the end of each procedure, trainees were required to record the following in a logbook: the duration of the procedure (defined as the length of time the light source was switched on), the number of attempts at intubation, and, in the event of failure, the reasons for interruption of the procedure and/or the need of assistance in completing the procedure. Failed esophageal intubation was defined as 3 unsuccessful attempts. When esophageal intubation was achieved, the procedures were defined as ‘‘complete’’ only if trainees also were able to identify, within 20 minutes, all anatomical landmarks (esophagogastric mucosal junction, gastric angulus, pylorus) and to perform certain basic maneuvers (aspiration of gastric juice, pyloric intubation [no more than 3 attempts], duodenal bulb exploration, intubation of the second part of the duodenum, and retroflexion). Incomplete procedures were defined as ‘‘procedure failures.’’ When a trainee required assistance but was able to complete the procedure with only verbal direction and advice, and without manual assistance, the procedure was judged as ‘‘complete.’’ The number of times assistance was required and reasons for this were recorded. A procedure was defined as ‘‘not complete’’ when manual assistance from the instructor was required. Instructors were required to record the following: the ‘‘completeness’’ of the procedure, the number of identified or missed lesions, and overall subjective judgment of the performance, expressed as a score from 0 (bad) to 10 (good), based on the completeness of the examination, the need for assistance and the presumed difficulty of the procedure (i.e., vigorous peristalsis, pyloric spasm, bulbar deformity, GASTROINTESTINAL ENDOSCOPY
197
E Giulio, D Fregonese, T Casetti, et al.
EGD: computer-based simulator training achieves basic manual skills
Table 1. Characteristics of patients
Male gender Mean age (SD), y Indication Dyspepsia Follow-up Iron deficiency anemia Screening for varices Others Endoscopic findings Negative Esophagitis Gastritis Neoplasia Peptic ulcer Duodenal fold anomalies Barrett’s esophagus Other
SIM group (n = 204)
Non-SIM group (n = 203)
n (%)
n (%)
p Value
72 (35.3) 52.5 (16.3)
75 (36.9) 52.2 (14.7)
0.757 0.889
134 42 16 4 8
(65.7) (20.6) (7.8) (2.0) (3.9)
128 51 11 8 5
(63.0) (25.1) (5.4) (3.9) (2.5)
0.605 0.289 0.426 0.259 0.574
161 16 9 4 8 1 4 1
(78.9) (7.8) (4.4) (2.0) (3.9) (0.5) (2.0) (0.5)
157 20 6 2 9 0 3 6
(77.3) (9.8) (3.0) (1.0) (4.4)
0.720 0.490 0.600 0.685 0.810 1.000 1.000 0.067
(1.5) (3.0)
SIM, Simulator; SD, standard deviation.
patient discomfort). If the score was 5 or less, the procedure was classified, by the instructor, as ‘‘negative’’; for scores of 6 or greater, the procedure was classified as ‘‘positive.’’ Instructors also were asked to record complications. The instructors were not blinded as to whether trainees had or had not used the simulator. Informed consent was obtained from all patients. The study was approved by the institutional review board of each participating center. The study was not funded by the company that manufactures the simulator, and none of the investigators has any financial relationship with the company. Statistical analysis Data are expressed as mean (6 standard error of the mean) and are evaluated by means of appropriate statistical tests (t test or Mann-Whitney U test). Relative frequencies for subgroups (percentages) were compared by using the Fisher exact test. A p value <0.05 was considered statistically significant. It is recognized that there was multiple testing of outcome data; however, correction of p values by the method of Bonferroni would not have removed any finding of statistical significance.
RESULTS All trainees randomized to the SIM group completed the 10 hours of preclinical training, as planned. More than 200 procedures were performed in patients per each group. The trainees performed up to 20 upper-GI procedures. However, 6 trainees in the SIM group and 7 in the non-SIM group performed one or two procedures less than planned because of 198
GASTROINTESTINAL ENDOSCOPY
the temporary assignment to other clinical activities, with the result that the sequence of consecutive procedures was interrupted. No attempted procedure was excluded from statistical analysis. The characteristics of the patients assigned to the two groups of trainees were comparable (Table 1). No difference was observed between the two groups in terms of failure to effect esophageal intubation (SIM group, 8/212; non-SIM group, 5/208) or the number of attempts (SIM group, mean 1.7; non-SIM group, mean 1.8), with 96% to 98% of attempts being successful in both groups. Trainees in the SIM group were able to complete, without manual assistance, significantly more procedures than trainees in the no-SIM group; the SIM group also performed significantly more complete procedures without verbal assistance and received significantly more ‘‘positive’’ judgments from instructors (Table 2). Verbal assistance was required in both the SIM and non-SIM groups. The main reasons for requesting verbal assistance were the following: pylorus intubation (SIM group, 16%; non-SIM group, 21%), retroflexion (SIM group, 7%; non-SIM group, 15%), and intubation of the second part of the duodenum (SIM group, 3%; non-SIM group, 18%). There was no difference in the time taken to achieve a complete procedure (mean 10.5 minutes, SIM group; mean 12.4 minutes, non-SIM group). No complication occurred in either group. DISCUSSION Endoscopy simulators have several expected advantages: patients are spared prolonged, possibly incomplete and painful procedures; the number of supervised procedures required to achieve competence is reduced; the level of assistance from instructors is reduced; and the learning time is shortened. To our knowledge, there is no prospective randomized study of the efficacy of a computer-based upper-endoscopy simulator in which its potential contributions to training are evaluated in a clinical setting. Esophageal intubation is one of the more difficult maneuvers for trainees and patients. In this study, the number of successful intubations and the number of intubation attempts were similar for trainees who received preclinical training with the simulator and those who did not. Instructors who participated in the present study judged the simulated anatomy of the oral cavity and the pharynx to be not sufficiently realistic and inadequate in terms of reproduction of the mechanical resistance characteristics of this anatomic region. In effect, simulation occurs only when the endoscope is introduced about 20 cm. VOLUME 60, NO. 2, 2004
EGD: computer-based simulator training achieves basic manual skills
E Giulio, D Fregonese, T Casetti, et al.
Table 2. Results SIM group 204 Non-SIM group 203
Complete procedures Non assistance Instructor’s judgment Positive Negative
n/total (%)
n/total (%)
Fisher test
OR (95%CI)
179/204 (87.7) 105/179 (58.6)
142/203 (69.9) 3/142 (2.1)
p < 0.0001 3.08 (1.8, 5.1) p < 0.0001 65.7 (20.1, 214.4)
177/204 (86.8) 27/204 (13.2)
115/203 (56.7) 88/203 (43.3)
p < 0.0001
5.0 (3.1, 8.2)
SIM, Simulator; OR, odds ratio; CI, confidence interval.
The results of the present study confirm the generally held opinion that preclinical training with a GI endoscopy simulator may not impart the skills needed for esophageal intubation.7,9 A major but difficult goal in the evaluation of training programs is to define standardized and measurable parameters of acquired skill.2-4 The completeness of the examination, defined as the ability to identify and to explore all anatomical regions, is a main criterion. The anatomic knowledge acquired with the assistance of an instructor is another basic criterion. The major limitation in the present study is that the instructors who evaluated the trainees were not blinded as to whether they received preclinical training by using the simulator. To improve the reliability of the evaluation, two cross-evaluation methods were used: (1) the ‘‘completeness of the procedure in a given time’’ by using previously defined standards and the ‘‘need for assistance’’ as reported by trainees; and (2) an ‘‘overall judgment’’ by instructors, which included not only the completeness of the procedure and the need for assistance but also their assessment of the difficulty of each specific examination. The second method of evaluation is more subjective. The American Society for Gastrointestinal Endoscopy (ASGE) has defined objective performance criteria for upper-GI endoscopy (e.g., esophageal intubation, pyloric intubation).4 Additional evaluation criteria were added for the purposes of the present study. Moreover, the ASGE advises that one of the expected effects of progressive skill acquisition is a reduced need for supervision.4 The present study evaluated only technical skill and not clinical competence. Therefore, only the request for assistance to perform a ‘‘complete procedure’’ was evaluated. The data reveal that trainees and instructors were in agreement regarding the efficacy of simulator pretraining, the ability to perform a ‘‘complete procedure’’ and to gain confidence in the performance of upper endoscopy without instructor assistance. In the SIM group, trainees were able to VOLUME 60, NO. 2, 2004
perform a complete procedure without any assistance in almost 60% of cases vs. 2% in the non-SIM group. The request for assistance in pyloric intubation was comparable in the two groups (29 cases, SIM group; 30 cases, non-SIM group), although more assistance was required in the non-SIM group for retroflexion (12 vs. 21) and intubation of the second part of the duodenum (5 vs. 25). The simulator used in the present study does not reproduce the mechanical resistance typically encountered in vivo during esophageal or pyloric intubation. The results of the current study demonstrate that preclinical training with the simulator is ineffective when mechanical resistance plays a role; but, contrary to the opinion of others,8 the simulator does provide a good notion of orientation in space, endoscope manipulation, and progression in the performance of complex maneuvers (i.e., retroflexion, intubation of the second part of the duodenum). The 3-dimensional quality and the tactile feedback provided by the simulator during these maneuvers are sufficiently realistic to allow the trainee to improve hand-eye skills. The design of the present study arbitrarily predefined simulator training as 10 hours. Moreover, use of the simulator was not permitted during the in vivo evaluation portion of the study. However, a more effective contribution of the simulator can be expected when trainees are allowed unrestricted use of the machine before and after supervised instruction in the performance of procedures in patients is initiated. The simulator used in the current study records some performance parameters (time to reach duodenum, correct identification of lesions, maneuvers regarded as inappropriate, and excessive wall pressure or air inflation). In our opinion, regardless of the time specified for training with the simulator, for individual trainees, it is important to reach a high standard of performance with the simulator before undertakingsupervisedproceduresinpatients.Moreover, a much more vast and varied file of simulated GASTROINTESTINAL ENDOSCOPY
199
E Giulio, D Fregonese, T Casetti, et al.
EGD: computer-based simulator training achieves basic manual skills
cases of varying difficulty would be useful and would probably enhance the learning process for trainees. Teaching is time consuming for instructors, and an expected advantage of the simulator is a reduction in the need for instructor assistance. In this respect, the results of the present study were positive because instructor assistance was not permitted during preclinical training with the simulator. Nevertheless, compared with the non-SIM group, the performance of the SIM group was superior during actual endoscopic procedures in patients. In conclusion, the present study found the GIMentor simulator to be efficient and effective in teaching the basic manual skills required for upper endoscopy, with the exception of esophageal intubation. While this computer-based simulator may be considered useful for training purposes, improvements in the technology are necessary, in particular, the simulation of mechanical resistance in certain anatomical areas. REFERENCES 1. Principles of training in gastrointestinal endoscopy. Manchester (MA): American Society for Gastrointestinal Endoscopy; 1991. 2. Cass OW, Freeman ML, Peine CJ, Zera RT, Onstad GR. Objective evaluation of endoscopy skills during training. Ann Intern Med 1993;118:40-4. 3. Bond JH. Evaluation of trainee competence. Gastrointest Endosc Clin N Am 1995;5:337-46. 4. ASGE guidelines: principles of training in gastrointestinal endoscopy. Gastrointest Endosc 1999;49:845-53. 5. Bini EJ, Firoozi B, Choung RJ, Ali EM, Osman M, Weinshel EH. Systematic evaluation of complications related to
6. 7. 8.
9.
10. 11. 12.
13.
14.
15.
16.
17.
endoscopy in a training setting: a prospective 30-day outcomes study. Gastrointest Endosc 2003;57:8-16. Cotton P. Live endoscopy demonstrations are great, but . . . [editorial]. Gastrointest Endosc 2000;51:627-9. Gessner CE, Jowell PS, Baillie J. Novel methods for endoscopic training. Gastrointest Endosc Clin N Am 1995;5:323-36. Hochberger J, Maiss J, Hahn E. The use of simulators for training in GI endoscopy [editorial]. Endoscopy 2002;34: 727-9. Hochberger J, Maiss J, Magdeburg B, Cohen J, Hahn EG. Training simulators and education in gastrointestinal endoscopy: current status and perspectives in 2001 [editorial]. Endoscopy 2001;33:541-9. Adamsen S. Simulators and gastrointestinal endoscopy training [editorial]. Endoscopy 2000;32:895-7. Bar-Meir S. A new endoscopic simulator [editorial]. Endoscopy 2000;32:898-900. Williams CB, Saunders BP, Bladen JS. Development of colonoscopy teaching simulation [editorial]. Endoscopy 2000;32: 901-5. Neumann M, Mayer G, Ell C, Felzmann T, Reinbruber B, Horbach T, et al. The Erlangen Endo-Trainer: life-like simulation for diagnostic and interventional endoscopic retrograde cholangiography. Endoscopy 2000;32:906-10. Gerson LB, Van Dam J. The future of simulators in GI endoscopy: an unlikely possibility or a virtual reality? [editorial] Gastrointest Endosc 2002;55:608-11. Sedlack RE, Kolars JC. Validation of computer-based endoscopy simulators in training [abstract]. Gastrointest Endosc 2002;55:AB77. Ferlitsch A, Glauninger P, Gupper A, Schillinger M, Haefner M, Gangl A, et al. Evaluation of a virtual endoscopy simulator for training in gastrointestinal endoscopy. Endoscopy 2002;34:698-702. Gerson LB, Van Dam J. A randomized controlled trial comparing an endoscopic simulator with traditional bedside teaching for training in flexible sigmoidoscopy [abstract]. Gastrointest Endosc 2002;55:AB78.
Submission of ACCEPTED manuscript on diskette Gastrointestinal Endoscopy strongly encourages the submission of final manuscripts on disk. Although files created with WordPerfect are preferred, please send your final manuscript in any electronic format. On your disk, please indicate computer system (e.g., IBM, MacIntosh) and word processing software used (e.g., WordPerfect 6.1). Please refer to complete Instructions to Authors in the most recent January or July issue of the Journal.
200
GASTROINTESTINAL ENDOSCOPY
VOLUME 60, NO. 2, 2004