- Email: [email protected]
THE EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS: A RANDOMIZED CONTROLLED STUDY
0022-5347/02/1673-1243/0 THE JOURNAL OF UROLOGY® Copyright © 2002 by AMERICAN UROLOGICAL ASSOCIATION, INC.®
Vol. 167, 1243–1247, March 2002 Printed in U.S.A.
THE EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS: A RANDOMIZED CONTROLLED STUDY EDWARD D. MATSUMOTO, STANLEY J. HAMSTRA, SIDNEY B. RADOMSKI AND MICHAEL D. CUSIMANO From the Department of Surgery and Centre for Research in Education at the University Health Network, Faculty of Medicine, University of Toronto, Division of Urology, University Health Network, Division of Neurosurgery, St. Michael’s Hospital and Ontario Institute for Studies in Education-University of Toronto, Toronto, Ontario, Canada
ABSTRACT
Purpose: Complex skills, such as ureteroscopy and stone extraction, are increasingly taught to novice urology trainees using bench models in surgical skills laboratories. We determined whether hands-on training improved the performance of novices more than those taught only by a didactic session and whether there was a difference in the performance of subjects taught on a low versus a high fidelity model. Materials and Methods: We randomized 40 final year medical students to a didactic session or 1 of 2 hands-on training groups involving low or high fidelity bench model practice. Training sessions were supervised by experienced endourologists. Testing involved removal of a mid ureteral stone using a semirigid ureteroscope and a basket. Blinded examiners tested subjects before and after training. Performance was measured by a global rating scale, checklist, pass rating and time needed to complete the task. Results: There was a significant effect of hands-on training on endourological performance (p ⬍0.01). With respect to bench model fidelity the low fidelity group did significantly better than the didactic group (p ⬍0.05). However, no significant difference was found between the high and low fidelity groups (p ⬎0.05). The low fidelity model cost Canadian $20 to produce, while the high fidelity model cost Canadian $3,700 to purchase. Conclusions: Hands-on training using bench models can be successful for teaching novices complex endourological skills. A low fidelity bench model is a more cost-effective means of teaching ureteroscopic skills to novices than a high fidelity model. KEY WORDS: ureter; calculi; education, medical; clinical competence; ureteroscopy
It has been shown that cases performed by trainees require more time and cost more than those performed by staff surgeons only.1 Increasing time constraints in the operating room make the acquisition of complex skills, such as endoscopic management of stones in urology, more challenging. With these constraints there is the risk that the education of trainees in the operating room may suffer to expedite the case. In response to this situation training centers are increasingly using surgical skills laboratories to prepare trainees for the operating room experience.2– 6 Surgical skills laboratories rely on bench models, which serve as surrogates for the human body. Increasingly the focus of surgical skills training, especially for more senior trainees, has been on complex skills such as laparoscopy and endoscopy.7–9 A bench model can help to facilitate the acquisition of skills deemed essential to the procedure. Model designers have responded by creating bench models that are realistic in appearance, in other words high fidelity models. In a recent survey of residents completing a core surgical curriculum at our surgical skills center a recurring concern was the fidelity or realism of the bench models.10 Unfortunately the acquisition of high fidelity bench models is limited by the associated high cost. Furthermore, with increasing use of technology such as virtual reality there is concern that the cost of equipping a surgical skills laboratory continues to increase.11 For training purposes the key to designing bench models does not necessarily lie in the ability to reproduce anatomy
accurately. It involves identifying the constructs considered essential to the procedure and ensuring that the bench models incorporate these constructs. With this idea in mind we assessed 2 models for learning the technique of mid ureteral stone removal. One model was inexpensive and considered less realistic, while the other was more expensive but more realistic. We determined whether hands-on training using bench models improved the performance of novices more than those taught only by a didactic session and whether there was a difference in the performance of subjects taught on a low versus a high fidelity model. To date there have been no studies of the role of bench model fidelity on surgical skills training with validated objective evaluation instruments, specifically global rating scales and checklists.4, 5
MATERIALS AND METHODS
Institutional ethics approval was obtained. Medical students were recruited at the beginning of the final year surgical rotation. Informed consent was obtained. Subjects were asked to complete a questionnaire to assess previous endoscopic experience. Those who had performed any cystoscopy, ureteroscopy or stent insertion unassisted were excluded from study. All 40 subjects received and reviewed a manual detailing the management of ureteral stones before the study day. On the study day all subjects watched a 15-minute video reviewing the instruments and demonstrating removal of a mid ureteral stone on a high fidelity bench model. Subjects were randomized by drawing candidate numbers to 1 of 3 teaching arms, namely didactic teaching, hands-on training
Accepted for publication October 12, 2001. Third Prize, American Urological Association-ACMI Circon Essay Competition, 2001. 1243
1244
EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS
using a low fidelity bench model and hands-on training using a high fidelity bench model. Subjects randomized to didactic teaching received 1 hour of teaching given by an endourologist involved in the development of the manual. Subjects randomized to the low fidelity bench model group practiced stone removal on a low fidelity bench model that we designed using task analysis (fig. 1). Subjects randomized to the high fidelity bench model group practiced stone removal on a high fidelity bench model (fig. 2). Practice sessions were 1 hour in duration and supervised by experienced endourologists. Subjects were provided with feedback as they practiced on the models. All subjects were administered a pre-test to assess baseline endoscopic ability before the study and a post-test after the training intervention. Subjects were asked to remove a mid ureteral stone from the high fidelity bench model. Test stations were equipped with a 21Fr rigid cystoscope with a 30-degree lens and catheterizable bridge, a 7.5Fr semirigid ureteroscope, a 0.038-inch guide wire and a 3Fr nitinol stone extractor. A customized digital camera and video monitors were used to enable the examiners to assess endoscopic performance. The models were on benches that allowed the monitor to be placed directly in front of the subjects. Monitor position was the same for all subjects. A surgical assistant (operating nurse) was present to assist the subject during the procedure. A research assistant timed the procedure. Two staff examiners blinded to subject study group assessed subject ability to remove a mid ureteral stone. Blinding was maintained by allowing the examiners to be present only during the testing phases of the study. Assessment instruments included a checklist, global rating scale, pass rating and time needed to complete the procedure. The checklist and global rating scale were validated in a previous study.12 Subjects received verbal instruction to remove a left or right mid ureteral stone. The task was broken into 2 components, that is the cystoscopy phase with guide wire insertion and the ureteroscopy phase with stone basket extraction. The procedure was timed from cystoscope insertion into the model meatus to mid ureteral stone removal. The timer was stopped when the subject changed from cystoscope to ureteroscope and the timer was restarted when the ureteroscope entered the meatus. Statistical analysis was done using commercially available software. Analysis of variance (ANOVA) and nonparametric statistical methods were used for analyzing baseline perfor-
FIG. 2. High fidelity model (by Limbs and Things, Bristol, United Kingdom).
mance among the groups. Repeated measures ANOVA is a statistical analysis that allows simultaneous within-subject analysis, which in our series was the change from pre-test to post-test results, and between-group analysis, which in our series was the intervention group. Nonparametric statistical tests are used to analyze data that is not normally distributed, such as measures of time. The nonparametric tests that we used included the Kruskal-Wallis test for 3 or more independent groups, the chi-square for incidence and the MannWhitney U test, which is a signed rank test for comparing 2 groups. Cronbach’s ␣ was applied to assess the degree to which items on a measurement instrument correlated with the total score on that instrument. Global rating and checklist scores were analyzed using repeated measures ANOVA. Pass ratings were analyzed using the chi-square test. The difference in post-test and pretest times needed to complete task was analyzed using the Kruskal-Wallis test. The Mann-Whitney U test was done for post hoc time analysis. Cronbach’s ␣ was used to measure the internal consistency of the checklist and global rating scale. Results were considered significant at 2-tailed p ⬍0.05. Sample size was calculated a priori. An earlier pilot study showed a significant effect of hands-on training for inexperienced first year residents. In the pilot study hands-on training compared with didactic teaching led to a large effect size of 1.8 standard deviation (SD). This value was used to calculate the didactic group sample size of 7 to provide 0.8 power (2-tailed ␣ ⫽ 0.05). With respect to bench model fidelity no studies were available to guide us. In the field of psychology an effect size of 1 SD is considered a large but acceptable difference for assessing teaching intervention.13 We agreed that an effect size of 1 SD would be required to demonstrate a practically significant teaching advantage of high or low fidelity bench model training. Based on an effect size of 1 SD and a power of 0.8 we calculated a sample size of 16 for the low and high fidelity groups (2-tailed ␣ ⫽ 0.05). RESULTS
FIG. 1. Low fidelity ureteroscopy model consisted of Penrose drain, inverted cup, molded latex in portable plastic case and 2 embedded straws approximately 8 mm. in diameter as substitutes for urethra, bladder dome, bladder base and bilateral ureters, respectively. Openings were cut midway up straws to facilitate placement of mid ureteral stone.
A total of 40 final year medical students participated in the study, including 7 randomized to the didactic teaching group, 16 randomized to the low fidelity training group and 17 randomized to the high fidelity training group. There were no differences in mean age or gender distribution in the groups (see table). No subjects were excluded from study based on previous cystoscopy, ureteroscopy or stent insertion experience. There were no significant differences in pre-test global
EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS
1245
Subject demographics and pre-test performance Didactic No. subjects 7 No. men/No. women 4/3 Mean age 25.7 Mean pre-test global rating score ⫾ SD (max. 35) 19.3 ⫾ 6.5 Mean pre-test checklist scores ⫾ SD (max. 24) 18.4 ⫾ 4.7 No. pass rating during pre-test/total No. 1/7 Median pre-test secs. to complete task (interquartile 587 (429) range) No significant differences among groups. Value of any cystoscopy-ureteroscopy done independently (greater than 80% of case) was 0.
rating scores, checklist scores, pass rating or time needed in the groups (see table). Repeated measures ANOVA revealed overall group-bytime interaction, as indicated by the global rating and checklist scores (F[2,37] ⫽ 9.6, p ⬍0.001 and F[2,37] ⫽ 10.4, p ⬍0.001, respectively). There was significant group-by-time interaction on the pass rating (chi-square[2] 16.7, p ⬍0.001). There was no significant group-by-time interaction on time needed to complete task (Kruskal-Wallis test p ⫽ 0.15). With respect to bench model fidelity analysis revealed that the group that trained on the low fidelity model did significantly better than the group that received didactic teaching only, as measured by the global rating scale (F[1,21] ⫽ 7.6, p ⫽ 0.012), checklist score (F[1,21] ⫽ 9.5, p ⫽ 0.006), pass rating (chi-square[1] ⫽ 11.5, p ⫽ 0.001) and time needed to complete the task (Mann-Whitney U test p ⫽ 0.013). There were no significant differences in performance in the low and high fidelity groups with respect to the same measures (figs. 3 to 5). Cronbach’s ␣ was used to measure internal consistency for the pre-test and post-test checklist scores, which were 0.71 and 0.63, respectively. Cronbach’s ␣ for pre-test and post-test global rating scores was 0.94 and 0.91, respectively. DISCUSSION
The cornerstone of surgical skills acquisition is practice. Practice on bench models facilitates the acquisition of new skills and allows ample opportunity for supervising staff urologists to provide constructive feedback to trainees. This
Low Fidelity
High Fidelity
Overall
16 10/6 26.6 17 ⫾ 7.3 15.9 ⫾ 3.7 2/16 513 (204.3)
17 8/9 26.6 15.2 ⫾ 5.4 15.5 ⫾ 2.7 0/17 510 (140.5)
40 22/18 26.5 16.6 ⫾ 6.4 16.2 ⫾ 3.6 3/40 512.5 (188)
constructive feedback allows trainees to recognize poor technique and guides them in practicing the appropriate technique. Unlike the operating room environment, the surgical skills center allows the supervising staff surgeon to focus on the needs of the trainee instead of on the patient. In addition to feedback, practice in a video environment may also enhance endoscopic skills. It is likely that repetitive practice in a video environment enables the trainee to develop the appropriate depth perception and psychomotor sense necessary to become efficient at minimally invasive surgery. Hanna et al noted that even video monitor location has a significant influence on endoscopic performance.14 The final important point of practice is reinforcement of the steps of a procedure. Based on the work of Fitts and Posner15 Kopta described the process of acquiring a new psychomotor skill in 3 phases as the cognitive, the integrative and the autonomous phase.16 In the cognitive phase the learner approaches a task by making a mental inventory of the steps required to perform the procedure. The learner essentially learns to talk his/her way through the procedure. After the cognitive phase is established the learner begins to transform this knowledge into the appropriate motor behavior. Kopta stressed the importance of feedback during the integrative phase and believed that “there is little benefit and great potential harm for a learner to practice and not know whether or not he is performing correctly.”16 As the integrative phase progresses, performance becomes less irregular. The autonomous phase is achieved when performance reaches the point at which routine procedures no longer require cognitive input. Train-
FIG. 3. Effect of bench model fidelity on global rating and checklist scores. Bars indicate standard error
1246
EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS
FIG. 4. Effect of bench model fidelity on pass rating
FIG. 5. Effect of bench model fidelity on post hoc time. Boxes indicate interquartile range. Vertical lines indicate range. Short horizontal lines indicate median time.
ing in a surgical skills laboratory involves all 3 phases of learning. With respect to the issue of fidelity our results indicate that the low fidelity model is equally effective for teaching novices endourological skills compared with the high fidelity model. Before designing the low fidelity model, experts were consulted to help identify the key maneuvers during ureteroscopic extraction of a mid ureteral stone. After a consensus was reached on the key steps a low fidelity model was designed to incorporate the essential maneuvers required in the real procedure. For example, inserting the ureteroscope at a proper angle and atraumatically were considered key steps in the procedure. The skills necessary were used during practice on the low fidelity bench model, while visual appearance had only a superficial resemblance to what may be encountered during the real procedure. Although our low fidelity bench model was specific to an endourological proce-
dure, the methodology of designing the model may be applied to virtually any other surgical procedure, including laparoscopy. Cost is always a concern, whether establishing or maintaining a surgical skills laboratory. In fiscal year 1998 to 1999 at the Surgical Skills Center in Toronto approximately Canadian $62,000 was budgeted to acquire models, accounting for 40% of center expenses. In the current study the cost of building 1 low fidelity model was Canadian $20. The cost of acquiring a high fidelity bench model was 185-fold more or Canadian $3,700. Virtual reality models are now appearing and the cost of these models is likely to be significant. Costeffectiveness analysis has an important role in the selection of models for surgical skills training. The consensus in surgical education is that the evaluation of surgical courses should occur in an operating room since the goal of any surgical skills course is better performance in
EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS
the operating room. Although the focus of our study was the effect of bench model fidelity, we also demonstrated that hands-on practice can lead to significant improvement in ureteroscopic performance in the laboratory environment. We intend to continue this research with intraoperative assessment of performance. Several limitations must be overcome before extending this study to the operating room. We were limited by the number of patients with mid ureteral stones in a limited time frame as well as the ethical issue of allowing a medical student to perform ureteroscopy unassisted for study purposes. Other factors, such as patient anatomy, stone size and location, made intraoperative assessment difficult. The thrust of future research must be directed toward overcoming these difficulties and developing new tools of intraoperative assessment. CONCLUSIONS
Training outside an operating room environment is an innovative way of complementing surgical training. When choosing or designing bench models, attention must be given to ensure that key constructs are incorporated into the models, so that that they have value for surgical skills training. Our study shows that hands-on training is far superior to didactic teaching only and a low cost but thoughtfully designed low fidelity ureteroscopy model is as effective as a more expensive high fidelity bench model for endourological training. University of Toronto Surgical Skills Center at Mount Sinai Hospital, Division of Urology, University of Toronto, Storz Canada, Cook Urology Canada and Abbott Laboratories provided resources. Drs. Glenn Regehr and Richard Reznick provided helpful discussion and Michael Brandt assisted. REFERENCES
1. Bridges, M. and Diamond, D. L.: The financial impact of teaching surgical residents in the operating room. Am J Surg, 177: 28, 1999
1247
2. Lossing, A. G., Hatswell, E. M., Gilas, T. et al: A technical-skills course for 1st-year residents in general surgery: a descriptive study. Can J Surg, 35: 536 – 40, 1992 3. Heppell, J., Beauchamp, G., Chollet, A.: Ten-year experience with a basic technical skills and perioperative management workshop for first-year residents. Can J Surg, 38: 27–32, 1995 4. Reznick, R., Regehr, G., MacRae, H. et al: Testing technical skill via an innovative “bench station” examination. Am J Surg, 173: 226, 1997 5. Martin, J. A., Regehr, G., Reznick, R. et al: Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg, 84: 273, 1997 6. Anastakis, D. J., Regehr, G., Reznick, R. K. et al: Assessment of technical skills transfer from the bench training model to the human model. Am J Surg, 177: 167, 1999 7. Derossis, A. M., Bothwell, J., Sigman, H. H. et al: The effect of practice on performance in a laparoscopic simulator. Surg Endosc, 12: 1117, 1998 8. Rosser, J. C., Jr., Rosser, L. E. and Savalgi, R. S.: Objective evaluation of a laparoscopic surgical skill program for residents and senior surgeons. Arch Surg, 133: 657, 1998 9. Derossis, A. M., Antoniuk, M. and Fried, G. M.: Evaluation of laparoscopic skills: a 2-year follow-up during residency training. Can J Surg, 42: 293, 1999 10. Hutchison, C., Hamstra, S. J. and Leadbetter, W.: The University of Toronto Surgical Skills Centre opens. Focus Surg Educ, 16: 22, 1998 11. Haluck, R. S. and Krummel, T. M.: Computers and virtual reality for surgical education in the 21st century. Arch Surg, 135: 786, 2000 12. Matsumoto, E. D., Hamstra, S. J., Radomski, S. B. et al: A novel approach to endourological training: training at the Surgical Skills Centre. J Urol, 166: 1261, 2001 13. Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. New York: Academic Press, 1977 14. Hanna, G. B., Shimi, S. M. and Cuschieri, A.: Task performance in endoscopic surgery is influenced by location of the image display. Ann Surg, 227: 481, 1998 15. Fitts, P. M. and Posner, M. I.: Human Performance. Belmont: Brooks/Cole Publishing, 1967 16. Kopta, J. A.: The development of motor skills in orthopaedic education. Clin Orthop, 75: 80, 1971
Vol. 167, 1243–1247, March 2002 Printed in U.S.A.
THE EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS: A RANDOMIZED CONTROLLED STUDY EDWARD D. MATSUMOTO, STANLEY J. HAMSTRA, SIDNEY B. RADOMSKI AND MICHAEL D. CUSIMANO From the Department of Surgery and Centre for Research in Education at the University Health Network, Faculty of Medicine, University of Toronto, Division of Urology, University Health Network, Division of Neurosurgery, St. Michael’s Hospital and Ontario Institute for Studies in Education-University of Toronto, Toronto, Ontario, Canada
ABSTRACT
Purpose: Complex skills, such as ureteroscopy and stone extraction, are increasingly taught to novice urology trainees using bench models in surgical skills laboratories. We determined whether hands-on training improved the performance of novices more than those taught only by a didactic session and whether there was a difference in the performance of subjects taught on a low versus a high fidelity model. Materials and Methods: We randomized 40 final year medical students to a didactic session or 1 of 2 hands-on training groups involving low or high fidelity bench model practice. Training sessions were supervised by experienced endourologists. Testing involved removal of a mid ureteral stone using a semirigid ureteroscope and a basket. Blinded examiners tested subjects before and after training. Performance was measured by a global rating scale, checklist, pass rating and time needed to complete the task. Results: There was a significant effect of hands-on training on endourological performance (p ⬍0.01). With respect to bench model fidelity the low fidelity group did significantly better than the didactic group (p ⬍0.05). However, no significant difference was found between the high and low fidelity groups (p ⬎0.05). The low fidelity model cost Canadian $20 to produce, while the high fidelity model cost Canadian $3,700 to purchase. Conclusions: Hands-on training using bench models can be successful for teaching novices complex endourological skills. A low fidelity bench model is a more cost-effective means of teaching ureteroscopic skills to novices than a high fidelity model. KEY WORDS: ureter; calculi; education, medical; clinical competence; ureteroscopy
It has been shown that cases performed by trainees require more time and cost more than those performed by staff surgeons only.1 Increasing time constraints in the operating room make the acquisition of complex skills, such as endoscopic management of stones in urology, more challenging. With these constraints there is the risk that the education of trainees in the operating room may suffer to expedite the case. In response to this situation training centers are increasingly using surgical skills laboratories to prepare trainees for the operating room experience.2– 6 Surgical skills laboratories rely on bench models, which serve as surrogates for the human body. Increasingly the focus of surgical skills training, especially for more senior trainees, has been on complex skills such as laparoscopy and endoscopy.7–9 A bench model can help to facilitate the acquisition of skills deemed essential to the procedure. Model designers have responded by creating bench models that are realistic in appearance, in other words high fidelity models. In a recent survey of residents completing a core surgical curriculum at our surgical skills center a recurring concern was the fidelity or realism of the bench models.10 Unfortunately the acquisition of high fidelity bench models is limited by the associated high cost. Furthermore, with increasing use of technology such as virtual reality there is concern that the cost of equipping a surgical skills laboratory continues to increase.11 For training purposes the key to designing bench models does not necessarily lie in the ability to reproduce anatomy
accurately. It involves identifying the constructs considered essential to the procedure and ensuring that the bench models incorporate these constructs. With this idea in mind we assessed 2 models for learning the technique of mid ureteral stone removal. One model was inexpensive and considered less realistic, while the other was more expensive but more realistic. We determined whether hands-on training using bench models improved the performance of novices more than those taught only by a didactic session and whether there was a difference in the performance of subjects taught on a low versus a high fidelity model. To date there have been no studies of the role of bench model fidelity on surgical skills training with validated objective evaluation instruments, specifically global rating scales and checklists.4, 5
MATERIALS AND METHODS
Institutional ethics approval was obtained. Medical students were recruited at the beginning of the final year surgical rotation. Informed consent was obtained. Subjects were asked to complete a questionnaire to assess previous endoscopic experience. Those who had performed any cystoscopy, ureteroscopy or stent insertion unassisted were excluded from study. All 40 subjects received and reviewed a manual detailing the management of ureteral stones before the study day. On the study day all subjects watched a 15-minute video reviewing the instruments and demonstrating removal of a mid ureteral stone on a high fidelity bench model. Subjects were randomized by drawing candidate numbers to 1 of 3 teaching arms, namely didactic teaching, hands-on training
Accepted for publication October 12, 2001. Third Prize, American Urological Association-ACMI Circon Essay Competition, 2001. 1243
1244
EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS
using a low fidelity bench model and hands-on training using a high fidelity bench model. Subjects randomized to didactic teaching received 1 hour of teaching given by an endourologist involved in the development of the manual. Subjects randomized to the low fidelity bench model group practiced stone removal on a low fidelity bench model that we designed using task analysis (fig. 1). Subjects randomized to the high fidelity bench model group practiced stone removal on a high fidelity bench model (fig. 2). Practice sessions were 1 hour in duration and supervised by experienced endourologists. Subjects were provided with feedback as they practiced on the models. All subjects were administered a pre-test to assess baseline endoscopic ability before the study and a post-test after the training intervention. Subjects were asked to remove a mid ureteral stone from the high fidelity bench model. Test stations were equipped with a 21Fr rigid cystoscope with a 30-degree lens and catheterizable bridge, a 7.5Fr semirigid ureteroscope, a 0.038-inch guide wire and a 3Fr nitinol stone extractor. A customized digital camera and video monitors were used to enable the examiners to assess endoscopic performance. The models were on benches that allowed the monitor to be placed directly in front of the subjects. Monitor position was the same for all subjects. A surgical assistant (operating nurse) was present to assist the subject during the procedure. A research assistant timed the procedure. Two staff examiners blinded to subject study group assessed subject ability to remove a mid ureteral stone. Blinding was maintained by allowing the examiners to be present only during the testing phases of the study. Assessment instruments included a checklist, global rating scale, pass rating and time needed to complete the procedure. The checklist and global rating scale were validated in a previous study.12 Subjects received verbal instruction to remove a left or right mid ureteral stone. The task was broken into 2 components, that is the cystoscopy phase with guide wire insertion and the ureteroscopy phase with stone basket extraction. The procedure was timed from cystoscope insertion into the model meatus to mid ureteral stone removal. The timer was stopped when the subject changed from cystoscope to ureteroscope and the timer was restarted when the ureteroscope entered the meatus. Statistical analysis was done using commercially available software. Analysis of variance (ANOVA) and nonparametric statistical methods were used for analyzing baseline perfor-
FIG. 2. High fidelity model (by Limbs and Things, Bristol, United Kingdom).
mance among the groups. Repeated measures ANOVA is a statistical analysis that allows simultaneous within-subject analysis, which in our series was the change from pre-test to post-test results, and between-group analysis, which in our series was the intervention group. Nonparametric statistical tests are used to analyze data that is not normally distributed, such as measures of time. The nonparametric tests that we used included the Kruskal-Wallis test for 3 or more independent groups, the chi-square for incidence and the MannWhitney U test, which is a signed rank test for comparing 2 groups. Cronbach’s ␣ was applied to assess the degree to which items on a measurement instrument correlated with the total score on that instrument. Global rating and checklist scores were analyzed using repeated measures ANOVA. Pass ratings were analyzed using the chi-square test. The difference in post-test and pretest times needed to complete task was analyzed using the Kruskal-Wallis test. The Mann-Whitney U test was done for post hoc time analysis. Cronbach’s ␣ was used to measure the internal consistency of the checklist and global rating scale. Results were considered significant at 2-tailed p ⬍0.05. Sample size was calculated a priori. An earlier pilot study showed a significant effect of hands-on training for inexperienced first year residents. In the pilot study hands-on training compared with didactic teaching led to a large effect size of 1.8 standard deviation (SD). This value was used to calculate the didactic group sample size of 7 to provide 0.8 power (2-tailed ␣ ⫽ 0.05). With respect to bench model fidelity no studies were available to guide us. In the field of psychology an effect size of 1 SD is considered a large but acceptable difference for assessing teaching intervention.13 We agreed that an effect size of 1 SD would be required to demonstrate a practically significant teaching advantage of high or low fidelity bench model training. Based on an effect size of 1 SD and a power of 0.8 we calculated a sample size of 16 for the low and high fidelity groups (2-tailed ␣ ⫽ 0.05). RESULTS
FIG. 1. Low fidelity ureteroscopy model consisted of Penrose drain, inverted cup, molded latex in portable plastic case and 2 embedded straws approximately 8 mm. in diameter as substitutes for urethra, bladder dome, bladder base and bilateral ureters, respectively. Openings were cut midway up straws to facilitate placement of mid ureteral stone.
A total of 40 final year medical students participated in the study, including 7 randomized to the didactic teaching group, 16 randomized to the low fidelity training group and 17 randomized to the high fidelity training group. There were no differences in mean age or gender distribution in the groups (see table). No subjects were excluded from study based on previous cystoscopy, ureteroscopy or stent insertion experience. There were no significant differences in pre-test global
EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS
1245
Subject demographics and pre-test performance Didactic No. subjects 7 No. men/No. women 4/3 Mean age 25.7 Mean pre-test global rating score ⫾ SD (max. 35) 19.3 ⫾ 6.5 Mean pre-test checklist scores ⫾ SD (max. 24) 18.4 ⫾ 4.7 No. pass rating during pre-test/total No. 1/7 Median pre-test secs. to complete task (interquartile 587 (429) range) No significant differences among groups. Value of any cystoscopy-ureteroscopy done independently (greater than 80% of case) was 0.
rating scores, checklist scores, pass rating or time needed in the groups (see table). Repeated measures ANOVA revealed overall group-bytime interaction, as indicated by the global rating and checklist scores (F[2,37] ⫽ 9.6, p ⬍0.001 and F[2,37] ⫽ 10.4, p ⬍0.001, respectively). There was significant group-by-time interaction on the pass rating (chi-square[2] 16.7, p ⬍0.001). There was no significant group-by-time interaction on time needed to complete task (Kruskal-Wallis test p ⫽ 0.15). With respect to bench model fidelity analysis revealed that the group that trained on the low fidelity model did significantly better than the group that received didactic teaching only, as measured by the global rating scale (F[1,21] ⫽ 7.6, p ⫽ 0.012), checklist score (F[1,21] ⫽ 9.5, p ⫽ 0.006), pass rating (chi-square[1] ⫽ 11.5, p ⫽ 0.001) and time needed to complete the task (Mann-Whitney U test p ⫽ 0.013). There were no significant differences in performance in the low and high fidelity groups with respect to the same measures (figs. 3 to 5). Cronbach’s ␣ was used to measure internal consistency for the pre-test and post-test checklist scores, which were 0.71 and 0.63, respectively. Cronbach’s ␣ for pre-test and post-test global rating scores was 0.94 and 0.91, respectively. DISCUSSION
The cornerstone of surgical skills acquisition is practice. Practice on bench models facilitates the acquisition of new skills and allows ample opportunity for supervising staff urologists to provide constructive feedback to trainees. This
Low Fidelity
High Fidelity
Overall
16 10/6 26.6 17 ⫾ 7.3 15.9 ⫾ 3.7 2/16 513 (204.3)
17 8/9 26.6 15.2 ⫾ 5.4 15.5 ⫾ 2.7 0/17 510 (140.5)
40 22/18 26.5 16.6 ⫾ 6.4 16.2 ⫾ 3.6 3/40 512.5 (188)
constructive feedback allows trainees to recognize poor technique and guides them in practicing the appropriate technique. Unlike the operating room environment, the surgical skills center allows the supervising staff surgeon to focus on the needs of the trainee instead of on the patient. In addition to feedback, practice in a video environment may also enhance endoscopic skills. It is likely that repetitive practice in a video environment enables the trainee to develop the appropriate depth perception and psychomotor sense necessary to become efficient at minimally invasive surgery. Hanna et al noted that even video monitor location has a significant influence on endoscopic performance.14 The final important point of practice is reinforcement of the steps of a procedure. Based on the work of Fitts and Posner15 Kopta described the process of acquiring a new psychomotor skill in 3 phases as the cognitive, the integrative and the autonomous phase.16 In the cognitive phase the learner approaches a task by making a mental inventory of the steps required to perform the procedure. The learner essentially learns to talk his/her way through the procedure. After the cognitive phase is established the learner begins to transform this knowledge into the appropriate motor behavior. Kopta stressed the importance of feedback during the integrative phase and believed that “there is little benefit and great potential harm for a learner to practice and not know whether or not he is performing correctly.”16 As the integrative phase progresses, performance becomes less irregular. The autonomous phase is achieved when performance reaches the point at which routine procedures no longer require cognitive input. Train-
FIG. 3. Effect of bench model fidelity on global rating and checklist scores. Bars indicate standard error
1246
EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS
FIG. 4. Effect of bench model fidelity on pass rating
FIG. 5. Effect of bench model fidelity on post hoc time. Boxes indicate interquartile range. Vertical lines indicate range. Short horizontal lines indicate median time.
ing in a surgical skills laboratory involves all 3 phases of learning. With respect to the issue of fidelity our results indicate that the low fidelity model is equally effective for teaching novices endourological skills compared with the high fidelity model. Before designing the low fidelity model, experts were consulted to help identify the key maneuvers during ureteroscopic extraction of a mid ureteral stone. After a consensus was reached on the key steps a low fidelity model was designed to incorporate the essential maneuvers required in the real procedure. For example, inserting the ureteroscope at a proper angle and atraumatically were considered key steps in the procedure. The skills necessary were used during practice on the low fidelity bench model, while visual appearance had only a superficial resemblance to what may be encountered during the real procedure. Although our low fidelity bench model was specific to an endourological proce-
dure, the methodology of designing the model may be applied to virtually any other surgical procedure, including laparoscopy. Cost is always a concern, whether establishing or maintaining a surgical skills laboratory. In fiscal year 1998 to 1999 at the Surgical Skills Center in Toronto approximately Canadian $62,000 was budgeted to acquire models, accounting for 40% of center expenses. In the current study the cost of building 1 low fidelity model was Canadian $20. The cost of acquiring a high fidelity bench model was 185-fold more or Canadian $3,700. Virtual reality models are now appearing and the cost of these models is likely to be significant. Costeffectiveness analysis has an important role in the selection of models for surgical skills training. The consensus in surgical education is that the evaluation of surgical courses should occur in an operating room since the goal of any surgical skills course is better performance in
EFFECT OF BENCH MODEL FIDELITY ON ENDOUROLOGICAL SKILLS
the operating room. Although the focus of our study was the effect of bench model fidelity, we also demonstrated that hands-on practice can lead to significant improvement in ureteroscopic performance in the laboratory environment. We intend to continue this research with intraoperative assessment of performance. Several limitations must be overcome before extending this study to the operating room. We were limited by the number of patients with mid ureteral stones in a limited time frame as well as the ethical issue of allowing a medical student to perform ureteroscopy unassisted for study purposes. Other factors, such as patient anatomy, stone size and location, made intraoperative assessment difficult. The thrust of future research must be directed toward overcoming these difficulties and developing new tools of intraoperative assessment. CONCLUSIONS
Training outside an operating room environment is an innovative way of complementing surgical training. When choosing or designing bench models, attention must be given to ensure that key constructs are incorporated into the models, so that that they have value for surgical skills training. Our study shows that hands-on training is far superior to didactic teaching only and a low cost but thoughtfully designed low fidelity ureteroscopy model is as effective as a more expensive high fidelity bench model for endourological training. University of Toronto Surgical Skills Center at Mount Sinai Hospital, Division of Urology, University of Toronto, Storz Canada, Cook Urology Canada and Abbott Laboratories provided resources. Drs. Glenn Regehr and Richard Reznick provided helpful discussion and Michael Brandt assisted. REFERENCES
1. Bridges, M. and Diamond, D. L.: The financial impact of teaching surgical residents in the operating room. Am J Surg, 177: 28, 1999
1247
2. Lossing, A. G., Hatswell, E. M., Gilas, T. et al: A technical-skills course for 1st-year residents in general surgery: a descriptive study. Can J Surg, 35: 536 – 40, 1992 3. Heppell, J., Beauchamp, G., Chollet, A.: Ten-year experience with a basic technical skills and perioperative management workshop for first-year residents. Can J Surg, 38: 27–32, 1995 4. Reznick, R., Regehr, G., MacRae, H. et al: Testing technical skill via an innovative “bench station” examination. Am J Surg, 173: 226, 1997 5. Martin, J. A., Regehr, G., Reznick, R. et al: Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg, 84: 273, 1997 6. Anastakis, D. J., Regehr, G., Reznick, R. K. et al: Assessment of technical skills transfer from the bench training model to the human model. Am J Surg, 177: 167, 1999 7. Derossis, A. M., Bothwell, J., Sigman, H. H. et al: The effect of practice on performance in a laparoscopic simulator. Surg Endosc, 12: 1117, 1998 8. Rosser, J. C., Jr., Rosser, L. E. and Savalgi, R. S.: Objective evaluation of a laparoscopic surgical skill program for residents and senior surgeons. Arch Surg, 133: 657, 1998 9. Derossis, A. M., Antoniuk, M. and Fried, G. M.: Evaluation of laparoscopic skills: a 2-year follow-up during residency training. Can J Surg, 42: 293, 1999 10. Hutchison, C., Hamstra, S. J. and Leadbetter, W.: The University of Toronto Surgical Skills Centre opens. Focus Surg Educ, 16: 22, 1998 11. Haluck, R. S. and Krummel, T. M.: Computers and virtual reality for surgical education in the 21st century. Arch Surg, 135: 786, 2000 12. Matsumoto, E. D., Hamstra, S. J., Radomski, S. B. et al: A novel approach to endourological training: training at the Surgical Skills Centre. J Urol, 166: 1261, 2001 13. Cohen, J.: Statistical Power Analysis for the Behavioral Sciences. New York: Academic Press, 1977 14. Hanna, G. B., Shimi, S. M. and Cuschieri, A.: Task performance in endoscopic surgery is influenced by location of the image display. Ann Surg, 227: 481, 1998 15. Fitts, P. M. and Posner, M. I.: Human Performance. Belmont: Brooks/Cole Publishing, 1967 16. Kopta, J. A.: The development of motor skills in orthopaedic education. Clin Orthop, 75: 80, 1971