- Email: [email protected]
Endoscopic correction of vesicoureteral reflux simulator curriculum as an effective teaching tool: Pilot study
Journal of Pediatric Urology (2016) 12, 45.e1e45.e6
Endoscopic correction of vesicoureteral reflux simulator curriculum as an effective teaching tool: Pilot study
a
Department of Urology, CHOC Children’s Hospital, University of California, Irvine, CA, USA
Tandis Soltani a,1, Guy Hidas a,1, Maryellen S. Kelly a, Adam Kaplan a, Blake Selby a, John Billimek a, Elias Wehbi a, Elspeth McDougall b, Irene McAleer a, Gordon McLorie a, Antoine E. Khoury a Summary
b
Department of Urologic Sciences, University of British Columbia, Vancouver, BC, Canada Correspondence to: A.E. Khoury, Department of Urology, University of California, Irvine, Head of Pediatric Urology Children’s Hospital of Orange County, 505 S. Main Street, Suite 100, Orange, CA 92868, USA, Tel.: þ1 714 509 3914; fax: þ1 714 509 3916 [email protected] (A.E. Khoury) Keywords Dextranomer/hyaluronic acid injection; Simulator; Surgery teaching tool; Vesicoureteral reflux Received 9 December 2014 Accepted 18 June 2015 Available online 19 August 2015
Introduction It has been well recognized that simulators are effective tools to teach and evaluate technical skills in laparoscopic surgery. Endoscopic injection for the correction of vesicourteral reflux has a definite learning curve. Surgeon experience has also been demonstrated to have an important role in the outcome of the procedure. Simulated training allows for practice in a realistic setting without the inherent risk of harm to the patient. This stress free environment allows the trainee to focus on the acquisition of surgical skills without worry about surgical outcome. Objective The aim was to validate a porcine bladder simulator curriculum for training and assessment of the surgical skills for the endoscopic correction of vesicoureteral reflux. Study design We developed a porcine bladder-based dextranomer/hyaluronic acid (Dx/HA) injection simulator consisting of a dissected ex vivo porcine bladder in a polystyrene box with the distal ureters and urethra secured (Figure). We performed content validation by five experienced pediatric urologists. We then organized a simulator curriculum, which included lecture, demonstration, and a 2-h hands-on training on the simulator. Content, discriminant, and concurrent validation of the simulator curriculum were carried out using 11 urology trainees at different levels of expertise. All the trainees were evaluated for each step of the procedure of both their first and last performances on the simulator. Results Overall, the model demonstrated good content validity by all experts (mean questionnaire score 92%). The simulator curriculum demonstrated a significant improvement in the performance of the trainees
1
between their first and last evaluations (56e92%; p Z 0.008). Specific parts of the procedure that showed significant improvement (p < 0.05) were identification of the ureteral orifice, ureteral orifice hydrodistention, first and second injection, and location, size, and depth of the mound after injection. Discussion The Dx/HA endoscopic injection simulator is an effective training tool to improve the performance of the surgeon carrying out the procedure. This teaching tool may be used to help improve the performance of the surgeon carrying out the procedure. This teaching curriculum may shorten the early learning curve historically associated with the procedure and provide a greater understanding of the technical components of successful endoscopic vesicoureteral reflux correction. Additionally, the implementation of this simulator within the developed curriculum can improve the performance of training urologists in all steps of the challenging technique of Dx/HA needle injection confirming concurrent validity. The next step in evaluation of this surgical skill-training curriculum would be to determine if the improvement in skill performance observed during training translates to improved performance in the clinical realm, or predictive validity. Limitations Some small differences exist between the porcine model and human ureteral orifices. In the porcine model the ureteral orifices are located medially and distally in the bladder neck, which make injection more challenging. Participants suggested that after practicing with the simulator endoscopic injection to a human ureteral orifice would be easier. Conclusion The simulator curriculum was able to improve the performance of the surgeon carrying out the procedure during subsequent simulations.
These authors contributed equally to the study.
http://dx.doi.org/10.1016/j.jpurol.2015.06.017 1477-5131/ª 2015 Journal of Pediatric Urology Company. Published by Elsevier Ltd. All rights reserved.
45.e2
T. Soltani et al.
Figure
Introduction Published success rates of dextranomer/hyaluronic acid (Dx/HA) injection for correction of pediatric vesicoureteral reflux (VUR) vary widely; reported success rates range from 50% to 100% [1]. The probability for correction with Dx/HA injection varies, largely based on initial grade of VUR. Children with grade 1 VUR have been found to have a success rate of correction of 89% with Dx/HA injection, while those with grade 5 VUR had a lower success rate of 62% [1]. Kirsch et al. [2] and Capozza et al. [3] eluded to the fact that endoscopic injection for VUR has a definite learning curve, especially for higher VUR grades. This is supported by the study of Dave et al. [4], which compared the success rates from their most experienced surgeon with the combined results of all the other surgeons in their Toronto group (p Z 0.025). The Toronto group’s results further strengthened the evidence that surgeon experience and preoperative grade of VUR are correlated with increased successful of Dx/HA injection success [4]. Ever-growing constraints on training in surgery exist, including reduced trainee work hours, increased operating room and supply costs, as well as a focus on medical error and ethics surrounding the acquisition of surgical skills on patients [5]. In response to this, many surgical fields are utilizing training with surgical simulators outside of the operating room. For example, it is well recognized that simulators are effective teaching tools and allow for evaluation of technical skills in laparoscopic surgery [6e10]. Dx/HA needle injection is a 1e2-min, highly confined, unforgiving implantation of a discrete 1e3 ml of paste into an extremely small and delicate part of tissue with little room for technical compensation, adjustment, intraoperative correction, or revision. Nevertheless, to our knowledge, no validated simulator has been developed to provide the opportunity to learn and practice the skills that can be transferred to the operating room. The aim of this study was to develop and validate a porcine simulator model for training and assessment of Dx/ HA endoscopic injection surgical skills. We hypothesize that
this simulator will be an effective teaching tool and improve the slope of the learning curve toward earlier learning, and allow for a greater understanding of the components of successful endoscopic VUR correction. Because there is no gold standard (or criterion) to compare simulator results to, in order to validate a simulator we assessed a number of questions: (1) Does the simulator adequately simulate relevant aspects of the real procedure (content validity)? (2) Does the simulator discriminate between different levels of expertise (discriminant or construct validity)? (3) Does the simulator-training curriculum improve the performance of the surgeon carrying out the procedure (construct/convergent validity)?
Materials and methods Simulator design The endoscopic injection simulator consisted of a dissected ex vivo porcine bladder in a polystyrene box with the distal ureters and urethra secured. An opening was made in the base of the box to allow the urethra to be pulled to the outside. Since the porcine ureteral orifices are very distal on the bladder neck and covered with a mucosal fold, we opened the bladder longitudinally in the midline and unroofed the ureteral orifice’s mucosal fold. After suturing the bladder wall back together, the bladder, including the urethra and the distal ureters, was inserted into the polystyrene box. Previously used Dx/HA syringes and rigid needles were used for the injection. In order to simulate Dx/HA consistency, a water-based lubricant gel was used as the injecting compound. Methylene blue was added to facilitate visualization of the injection by dissection of the bladder after the procedure. A pediatric Wolf 9.5F (Richard Wolf Medical Instruments Corporation, Vernon Hills, IL, USA) working cystoscope was used (Fig. 1).
Content validity study Content validity addresses the issue of whether a measure adequately reflects the breadth of the concept being
Endoscopic correction of vesicoureteral reflux simulator
Figure 1
45.e3
Porcine simulator cystoscopic view of double hydrodistention implantation technique injection.
measured, in this case effective performance of endoscopic Dx/HA injection. In order to demonstrate that the simulator effectively simulates the real procedure, we asked five expert pediatric urologists to participate in the study. The five experts were from different institutions, each having over 20 years of experience in urology surgery and over 15 endoscopic injections of Dx/HA annually. We asked each pediatric urologist to perform a cystoscopy and bilateral double-hydrodistention implantation technique (HIT) Dx/HA injection into the two ureteral orifices of the simulator bladder. After the procedure, the experts completed a questionnaire (see next section) designed to evaluate the realism of each step of the injection procedure on the simulator compared to an actual procedure in the operating room.
Questionnaire design A post-injection questionnaire was designed by dividing the endoscopic Dx/HA injection procedure into five steps. The steps were cystoscopy, identification of the ureteral orifices, hydrodistension of the ureteral orifices, first injection, and second injection. For each step of the procedure, the pediatric urologist experts were asked to rate how realistically the simulator replicated the actual procedure. We also asked about the ability of the injected compound to realistically mimic the viscosity of the actual procedure, and the shape of the achieved injection mound and how the post-injection hydrodistension realistically simulated the actual procedure. A Likert rating scale ranged from 1 (poorly simulates the actual procedure) to 4 (excellent simulation of the actual procedure) was used to rate each question. In order to facilitate interpretation, results were transformed to a 0e100 scale by subtracting the mean item score from the theoretical minimum scale score dividing by the theoretical maximum minus the theoretical minimum scale scoring and multiplying the result by 100. A composite score was created to assess overall performance as an average of the eight items. Reliability of this measure was assessed using Cronbach’s alpha.
Construct/discriminative validity study In order to evaluate the ability of the simulator to differentiate between different levels of expertise we asked residents, fellows, and pediatric urologists to perform the procedure on the simulator. Eleven trainees
were assessed. Before the activity, all participants viewed a short video clip demonstrating the actual procedure being carried out on a patient and on the simulator. Both videos included narration that identified each step of the procedure. Participants were asked to perform the injection on the simulator bladder while no assistance was given. The injection performed by each participant was video taped and evaluated by a pediatric urologist (A.K.) who was blinded to participant identity, background, and previous experience.
Technical skills assessment Technical skills were assessed using a structured rating scale focusing on the subject’s ability to perform each part of the procedure: cystoscopy, ureteral orifice identification, ureteral orifice hydrodistension, endoscopic submucosal practice injection in the bladder wall, first HIT injection to the ureteral orifice, and second sub-trigonal injection. Each one of the technical steps outlined above was graded on a scale of 1 (attempt to complete a step but failure to do so correctly) to 4 (flawless performance). Examples of criteria for a fail grade (score Z 1) would be frequently stopping the procedure, clearly being unsure of the next move, awkward or inappropriate movements of the cystoscope, or needle injection in the wrong spot. Examples of flawless performance (score Z 4) would be consistent stable handling of the cystoscope and the needle, clear economy of motion and efficiency, correct needle placement and injection.
Mound appearance evaluation In order to evaluate the appearance of the mound, we generated a questionnaire consisting of five subcategories of the mound: the appearance of the ureteral orifice slit, hydrodistensibility, depth, size (submucosally, intramuscular, or outside the bladder), and location of the mound (at the ureteral orifice or migration to the sides). Each parameter was given 1 (poor), 2 (marginal), or 3 (excellent) points with a total score of 5e15. In order to facilitate interpretation, the results were transformed to a 0e100 scale using the next, we correlated the performance score with years of cystoscopy experience. Pearson’s correlation coefficient was used to measure the strength and direction of the relationship between the injection skills and operator experience.
45.e4
T. Soltani et al.
Concurrent validity study
Discriminant validity study
Intervention: simulator curriculum In order to assess whether the simulator-training curriculum was able to improve the performance of the surgeon carrying out the procedure, we conducted a hands-on simulator curriculum workshop, which included theoretical detailed explanation on the technique of the different surgical steps done by a pediatric urologist (A.K.), demonstration of the actual procedure on the simulator by a pediatric urologist (A.K.), hands-on training on the simulator using four dissected pig bladders (eight injections) with one-on-one coaching by a pediatric urologist (A.K.) and fellow (G.H.) for 2 h.
Eleven participants in different stages of training (postgraduate year 1 through to pediatric urology fellow) performed the procedure. The video of the injections performed by each of the participants was blindly graded by a pediatric urologist (A.K.) using the two questionnaires created (technical skills and mound appearance). A positive correlation was found between years of experience and the ability to perform the procedure (r Z 0.90, p Z 0.001). Interestingly, great improvement is achieved after the surgeon completed his pediatric urology rotation during the fourth year of training at this facility.
Concurrent validity study Performance evaluation Both technical skills and a mound configuration were included on an evaluation form during the second phase of the study, to measure participant performance. The first and last injection from each study participant was evaluated by a pediatric urologist (A.K.). A paired t test was used to measure the performance differences between the initial and the last surgical attempt. Values of p < 0.05 were considered significant. In order to preserve integrity of comparison between variables and allow easier interpretation, the data were transformed to a 0e100 scale.
Results Content validity study Immediately after performing the procedure on the simulator, participants were asked to complete a questionnaire evaluating the ability of the simulator to realistically simulate the actual procedure. Table 1 summarizes the results of each question.
Table 1 (1e4).
As shown in Table 2, most surgical skill parameters had improved after 2 h of hands-on simulator curriculum training with a total improvement change noted to be 38% (p Z 0.01). Table 3 demonstrates the percentage change in the scores of the mound configuration grading after training. Total improvement in mound configuration increased 52% (p Z 0.01).
Discussion The teaching and assessment of technical skills on highfidelity simulation systems has its roots in the aviation industry and has become an important consideration in surgical education [11,12]. Simulated training allows for practice in a realistic setting without the inherent risk of harm to the patient. This stress free environment allows the trainee to specifically focus on the acquisition of surgical skills without any worry about surgical outcome. Another unique feature of the simulator is that mistakes, which are a crucial part of the learning experience, are encouraged, allowing the development of methods to avoid or manage procedural complications. During the past
Questionnaire evaluating the ability of the simulator to realistically simulate each step of the actual procedure. Score
Question
Most common response score (range)
Does the cystoscopy part of the procedure realistically simulate the actual procedure? Is the identification of the ureteral orifice part of the procedure a realistic simulation of the actual procedure? Does the ureteral orifice hydrodistended part of the procedure realistically simulate the actual procedure? Does the first injection (submucosal implantation within the intramural ureter) part of the procedure realistically simulate the actual procedure Does the second injection (submucosal implantation at the ureteral orifice) part of the procedure realistically simulate the actual procedure? Does the viscosity of the simulated injection material realistically simulate the actual procedure? Does the shape of the achieved injection mound realistically simulate that seen at an actual procedure? Does the post-injection hydrodistension part of the procedure realistically simulate that seen at an actual procedure?
4 (3e4) 4 (3e4) 4 (3e4) 4 (4) 4 (3e4) 3 (2e4) 4 (4) 4 (3e4)
Endoscopic correction of vesicoureteral reflux simulator Table 2
45.e5
Percentage change from before to after 2 h of hands-on simulator curriculum training.
Steps
Percentage change of score from before to after training (%)
p
Cystoscopy initiation Identification of the ureteral orifice Hydrodistension of the ureteral orifice First Dx/HA-like material injection Second Dx/HA-like material injection Post-injection lack of hydrodistension at the ureteral orifice Total
37 33 33 46 46 33 38
0.017 0.025 0.010 0.006 0.006 0.082 0.01
Dx/HA Z dextranomer/hyaluronic acid.
decade, surgical simulators have been developed and popularized as a component of skills training for a variety of minimally invasive surgical procedures [7e11]. Multiple studies have demonstrated improved performance in medical students, residents, fellows, and attending physicians of multiple specialties after simulation-based training, with generally positive feedback from learners after focused educational sessions [6e11]. More recent applications of simulation technology have focused on skills assessment, but it is critical that simulation-based training be implemented on the secure platform of a well-developed curriculum. Dx/HA needle injection is a 1e2-min, highly confined, unforgiving implantation of a discrete 1e3 ml of paste into an extremely small and delicate part of the tissue with little room for technical compensation, adjustment, intraoperative correction, or revision. In this study, we demonstrated that a dissected porcine bladder simulator can realistically simulate the actual surgical procedure, and it may be a tool for improving surgical skills. This provides a cost effective model on which to practice this challenging surgical procedure. It is known that the appearance of the Dx/HA mound does not sufficiently correlate with procedure outcome. A suboptimal appearing mound will sometimes correct the reflux and perfectly appearing mounds sometimes fail [13]. Since there is no validated tool to predict if a mound is successful, this simulator curriculum aimed to teach and allow practice in order to be able to perform the technically best achievable mound, and evaluate the participant on the basis of mound configuration.
Table 3 Percent change in mound configuration score before and after 2 h of hands-on simulator curriculum training. Criteria
Percentage change in score p from before to after training (%)
Appearance of slit Hydrodistension Depth Size Location Total
50 42 58 50 50 52
0.076 0.141 0.013 0.004 0.041 0.01
This simple porcine bladder model simulator is able to distinguish surgeons dependent on their clinical experience with cystoscopy, thereby demonstrating construct validity. Additionally, the implementation of this simulator within the developed curriculum can improve the performance of training urologists in all steps of the challenging technique of Dx/HA needle injection confirming concurrent validity. It was noted that the trainees had great improvement in their skills after their pediatric urology rotation (year 4). This would be expected, as they have greater experience with Dx/HA injection at this time and beyond. From this information it may be best to have residents complete their simulator training prior to their pediatric urology rotation to increase their skill set before patient contact. The next step in evaluation of this surgical skill-training curriculum would be to determine if the improvement in skill performance observed during training translates to improved performance in the clinical realm, or predictive validity. Some small differences exist between the porcine model and human ureteral orifices. In the porcine model, the ureteral orifices are located medially and distally in the bladder neck, which make injection more challenging. Participants suggested that after practicing with the simulator, endoscopic injection to a human ureteral orifice (which is usually laterally displaced) would be easier. Since there is no gold standard against which to assess the performance of those performing Dx/HA injection and therefore the simulation curriculum, we are unable to calculate criterion validity; therefore, we relied on discriminant and convergent validity to evaluate the value as an effective teaching tool. A single senior pediatric urologist evaluated the performance of all operators of the simulator. Absence of a control group, lack of testeretest, and non-blinding to the study of the evaluator goals were potential sources of bias. We as yet have no data on actual performance of the procedure in patients of subjects exposed to the simulation curriculum, nor do we have health outcomes of those patients. Further research is needed to compare the effectiveness of the simulator curriculum training methods on patient outcomes.
Conclusion The Dx/HA endoscopic injection simulator realistically simulates the actual clinical procedure. The simulator was
45.e6 able to differentiate between different levels of experience (before and after pediatric urology training). The simulator curriculum was able to improve the performance of the surgeon carrying out the procedure during subsequent simulations. This simulator may be an effective teaching tool and may improve the early learning curve for the clinical performance of this technique and provide a greater understanding of the components of successful endoscopic VUR correction.
Conflict of interest None.
Funding None.
References [1] Routh JC, Inman BA, Reinberg Y. Dextranomer/hyaluronic acid for pediatric vesicoureteral reflux: systematic review. Pediatrics 2010;125:1010. [2] Kirsch AJ, Perez-Brayfield M, Scherz HC. Minimally invasive treatment of vesicoureteral reflux with endoscopic injection of dextranomer/hyaluronic acid copolymer: the Children’s Hospitals of Atlanta experience. J Urol 2003;170:211. [3] Capozza N, Lais A, Nappo S, et al. The role of endoscopic treatment of vesicoureteral reflux: a 17-year experience. J Urol 2004;172:1626.
T. Soltani et al. [4] Dave S, Lorenzo AJ, Khoury AE, et al. Learning from the learning curve: Factors associated with successful endoscopic correction of vesicoureteral reflux using dextranomer/hyaluronic acid copolymer. J Urol 2008;180: 1594. [5] Carlin AM, Gasevic E, Shepard AD. Effect of the 80-hour work week on resident operative experience in general surgery. Am J Surg 2007;193:326. [6] Fried GM, Feldman LS, Vassiliou MC, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg 2004;240:518. [7] Rebolledo BJ, Hammann-Scala J, Leali A, et al. Arthroscopy skills development with a surgical simulator: a comparative study in orthopaedic surgery residents. Am J Sports Med 2015; 43:1526e9. [8] Martin KD, Patterson D, Phisitkul P, et al. Ankle arthroscopy simulation improves basic skills, anatomic recognition, and proficiency during diagnostic examination of residents in training. Foot Ankle Int 2015;36:827e35. [9] Hu Y, Le IA, Goodrich RN, et al. Construct validation of a costeffective vessel ligation benchtop simulator. J Surg Educ 2015; 72:381e6. [10] Makiyama K, Yamanaka H, Ueno D, et al. Validation of a patient-specific simulator for laparoscopic renal surgery. Int J Urol 2015;22:572e6. [11] Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidencebased virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244:310. [12] Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002;236: 458. [13] Hidas G, Soltani T, Watts B, et al. Is the appearance of the dextranomer/hyaluronic acid mound predictive of reflux resolution? J Urol 2013;189:1882e5.
Endoscopic correction of vesicoureteral reflux simulator curriculum as an effective teaching tool: Pilot study
a
Department of Urology, CHOC Children’s Hospital, University of California, Irvine, CA, USA
Tandis Soltani a,1, Guy Hidas a,1, Maryellen S. Kelly a, Adam Kaplan a, Blake Selby a, John Billimek a, Elias Wehbi a, Elspeth McDougall b, Irene McAleer a, Gordon McLorie a, Antoine E. Khoury a Summary
b
Department of Urologic Sciences, University of British Columbia, Vancouver, BC, Canada Correspondence to: A.E. Khoury, Department of Urology, University of California, Irvine, Head of Pediatric Urology Children’s Hospital of Orange County, 505 S. Main Street, Suite 100, Orange, CA 92868, USA, Tel.: þ1 714 509 3914; fax: þ1 714 509 3916 [email protected] (A.E. Khoury) Keywords Dextranomer/hyaluronic acid injection; Simulator; Surgery teaching tool; Vesicoureteral reflux Received 9 December 2014 Accepted 18 June 2015 Available online 19 August 2015
Introduction It has been well recognized that simulators are effective tools to teach and evaluate technical skills in laparoscopic surgery. Endoscopic injection for the correction of vesicourteral reflux has a definite learning curve. Surgeon experience has also been demonstrated to have an important role in the outcome of the procedure. Simulated training allows for practice in a realistic setting without the inherent risk of harm to the patient. This stress free environment allows the trainee to focus on the acquisition of surgical skills without worry about surgical outcome. Objective The aim was to validate a porcine bladder simulator curriculum for training and assessment of the surgical skills for the endoscopic correction of vesicoureteral reflux. Study design We developed a porcine bladder-based dextranomer/hyaluronic acid (Dx/HA) injection simulator consisting of a dissected ex vivo porcine bladder in a polystyrene box with the distal ureters and urethra secured (Figure). We performed content validation by five experienced pediatric urologists. We then organized a simulator curriculum, which included lecture, demonstration, and a 2-h hands-on training on the simulator. Content, discriminant, and concurrent validation of the simulator curriculum were carried out using 11 urology trainees at different levels of expertise. All the trainees were evaluated for each step of the procedure of both their first and last performances on the simulator. Results Overall, the model demonstrated good content validity by all experts (mean questionnaire score 92%). The simulator curriculum demonstrated a significant improvement in the performance of the trainees
1
between their first and last evaluations (56e92%; p Z 0.008). Specific parts of the procedure that showed significant improvement (p < 0.05) were identification of the ureteral orifice, ureteral orifice hydrodistention, first and second injection, and location, size, and depth of the mound after injection. Discussion The Dx/HA endoscopic injection simulator is an effective training tool to improve the performance of the surgeon carrying out the procedure. This teaching tool may be used to help improve the performance of the surgeon carrying out the procedure. This teaching curriculum may shorten the early learning curve historically associated with the procedure and provide a greater understanding of the technical components of successful endoscopic vesicoureteral reflux correction. Additionally, the implementation of this simulator within the developed curriculum can improve the performance of training urologists in all steps of the challenging technique of Dx/HA needle injection confirming concurrent validity. The next step in evaluation of this surgical skill-training curriculum would be to determine if the improvement in skill performance observed during training translates to improved performance in the clinical realm, or predictive validity. Limitations Some small differences exist between the porcine model and human ureteral orifices. In the porcine model the ureteral orifices are located medially and distally in the bladder neck, which make injection more challenging. Participants suggested that after practicing with the simulator endoscopic injection to a human ureteral orifice would be easier. Conclusion The simulator curriculum was able to improve the performance of the surgeon carrying out the procedure during subsequent simulations.
These authors contributed equally to the study.
http://dx.doi.org/10.1016/j.jpurol.2015.06.017 1477-5131/ª 2015 Journal of Pediatric Urology Company. Published by Elsevier Ltd. All rights reserved.
45.e2
T. Soltani et al.
Figure
Introduction Published success rates of dextranomer/hyaluronic acid (Dx/HA) injection for correction of pediatric vesicoureteral reflux (VUR) vary widely; reported success rates range from 50% to 100% [1]. The probability for correction with Dx/HA injection varies, largely based on initial grade of VUR. Children with grade 1 VUR have been found to have a success rate of correction of 89% with Dx/HA injection, while those with grade 5 VUR had a lower success rate of 62% [1]. Kirsch et al. [2] and Capozza et al. [3] eluded to the fact that endoscopic injection for VUR has a definite learning curve, especially for higher VUR grades. This is supported by the study of Dave et al. [4], which compared the success rates from their most experienced surgeon with the combined results of all the other surgeons in their Toronto group (p Z 0.025). The Toronto group’s results further strengthened the evidence that surgeon experience and preoperative grade of VUR are correlated with increased successful of Dx/HA injection success [4]. Ever-growing constraints on training in surgery exist, including reduced trainee work hours, increased operating room and supply costs, as well as a focus on medical error and ethics surrounding the acquisition of surgical skills on patients [5]. In response to this, many surgical fields are utilizing training with surgical simulators outside of the operating room. For example, it is well recognized that simulators are effective teaching tools and allow for evaluation of technical skills in laparoscopic surgery [6e10]. Dx/HA needle injection is a 1e2-min, highly confined, unforgiving implantation of a discrete 1e3 ml of paste into an extremely small and delicate part of tissue with little room for technical compensation, adjustment, intraoperative correction, or revision. Nevertheless, to our knowledge, no validated simulator has been developed to provide the opportunity to learn and practice the skills that can be transferred to the operating room. The aim of this study was to develop and validate a porcine simulator model for training and assessment of Dx/ HA endoscopic injection surgical skills. We hypothesize that
this simulator will be an effective teaching tool and improve the slope of the learning curve toward earlier learning, and allow for a greater understanding of the components of successful endoscopic VUR correction. Because there is no gold standard (or criterion) to compare simulator results to, in order to validate a simulator we assessed a number of questions: (1) Does the simulator adequately simulate relevant aspects of the real procedure (content validity)? (2) Does the simulator discriminate between different levels of expertise (discriminant or construct validity)? (3) Does the simulator-training curriculum improve the performance of the surgeon carrying out the procedure (construct/convergent validity)?
Materials and methods Simulator design The endoscopic injection simulator consisted of a dissected ex vivo porcine bladder in a polystyrene box with the distal ureters and urethra secured. An opening was made in the base of the box to allow the urethra to be pulled to the outside. Since the porcine ureteral orifices are very distal on the bladder neck and covered with a mucosal fold, we opened the bladder longitudinally in the midline and unroofed the ureteral orifice’s mucosal fold. After suturing the bladder wall back together, the bladder, including the urethra and the distal ureters, was inserted into the polystyrene box. Previously used Dx/HA syringes and rigid needles were used for the injection. In order to simulate Dx/HA consistency, a water-based lubricant gel was used as the injecting compound. Methylene blue was added to facilitate visualization of the injection by dissection of the bladder after the procedure. A pediatric Wolf 9.5F (Richard Wolf Medical Instruments Corporation, Vernon Hills, IL, USA) working cystoscope was used (Fig. 1).
Content validity study Content validity addresses the issue of whether a measure adequately reflects the breadth of the concept being
Endoscopic correction of vesicoureteral reflux simulator
Figure 1
45.e3
Porcine simulator cystoscopic view of double hydrodistention implantation technique injection.
measured, in this case effective performance of endoscopic Dx/HA injection. In order to demonstrate that the simulator effectively simulates the real procedure, we asked five expert pediatric urologists to participate in the study. The five experts were from different institutions, each having over 20 years of experience in urology surgery and over 15 endoscopic injections of Dx/HA annually. We asked each pediatric urologist to perform a cystoscopy and bilateral double-hydrodistention implantation technique (HIT) Dx/HA injection into the two ureteral orifices of the simulator bladder. After the procedure, the experts completed a questionnaire (see next section) designed to evaluate the realism of each step of the injection procedure on the simulator compared to an actual procedure in the operating room.
Questionnaire design A post-injection questionnaire was designed by dividing the endoscopic Dx/HA injection procedure into five steps. The steps were cystoscopy, identification of the ureteral orifices, hydrodistension of the ureteral orifices, first injection, and second injection. For each step of the procedure, the pediatric urologist experts were asked to rate how realistically the simulator replicated the actual procedure. We also asked about the ability of the injected compound to realistically mimic the viscosity of the actual procedure, and the shape of the achieved injection mound and how the post-injection hydrodistension realistically simulated the actual procedure. A Likert rating scale ranged from 1 (poorly simulates the actual procedure) to 4 (excellent simulation of the actual procedure) was used to rate each question. In order to facilitate interpretation, results were transformed to a 0e100 scale by subtracting the mean item score from the theoretical minimum scale score dividing by the theoretical maximum minus the theoretical minimum scale scoring and multiplying the result by 100. A composite score was created to assess overall performance as an average of the eight items. Reliability of this measure was assessed using Cronbach’s alpha.
Construct/discriminative validity study In order to evaluate the ability of the simulator to differentiate between different levels of expertise we asked residents, fellows, and pediatric urologists to perform the procedure on the simulator. Eleven trainees
were assessed. Before the activity, all participants viewed a short video clip demonstrating the actual procedure being carried out on a patient and on the simulator. Both videos included narration that identified each step of the procedure. Participants were asked to perform the injection on the simulator bladder while no assistance was given. The injection performed by each participant was video taped and evaluated by a pediatric urologist (A.K.) who was blinded to participant identity, background, and previous experience.
Technical skills assessment Technical skills were assessed using a structured rating scale focusing on the subject’s ability to perform each part of the procedure: cystoscopy, ureteral orifice identification, ureteral orifice hydrodistension, endoscopic submucosal practice injection in the bladder wall, first HIT injection to the ureteral orifice, and second sub-trigonal injection. Each one of the technical steps outlined above was graded on a scale of 1 (attempt to complete a step but failure to do so correctly) to 4 (flawless performance). Examples of criteria for a fail grade (score Z 1) would be frequently stopping the procedure, clearly being unsure of the next move, awkward or inappropriate movements of the cystoscope, or needle injection in the wrong spot. Examples of flawless performance (score Z 4) would be consistent stable handling of the cystoscope and the needle, clear economy of motion and efficiency, correct needle placement and injection.
Mound appearance evaluation In order to evaluate the appearance of the mound, we generated a questionnaire consisting of five subcategories of the mound: the appearance of the ureteral orifice slit, hydrodistensibility, depth, size (submucosally, intramuscular, or outside the bladder), and location of the mound (at the ureteral orifice or migration to the sides). Each parameter was given 1 (poor), 2 (marginal), or 3 (excellent) points with a total score of 5e15. In order to facilitate interpretation, the results were transformed to a 0e100 scale using the next, we correlated the performance score with years of cystoscopy experience. Pearson’s correlation coefficient was used to measure the strength and direction of the relationship between the injection skills and operator experience.
45.e4
T. Soltani et al.
Concurrent validity study
Discriminant validity study
Intervention: simulator curriculum In order to assess whether the simulator-training curriculum was able to improve the performance of the surgeon carrying out the procedure, we conducted a hands-on simulator curriculum workshop, which included theoretical detailed explanation on the technique of the different surgical steps done by a pediatric urologist (A.K.), demonstration of the actual procedure on the simulator by a pediatric urologist (A.K.), hands-on training on the simulator using four dissected pig bladders (eight injections) with one-on-one coaching by a pediatric urologist (A.K.) and fellow (G.H.) for 2 h.
Eleven participants in different stages of training (postgraduate year 1 through to pediatric urology fellow) performed the procedure. The video of the injections performed by each of the participants was blindly graded by a pediatric urologist (A.K.) using the two questionnaires created (technical skills and mound appearance). A positive correlation was found between years of experience and the ability to perform the procedure (r Z 0.90, p Z 0.001). Interestingly, great improvement is achieved after the surgeon completed his pediatric urology rotation during the fourth year of training at this facility.
Concurrent validity study Performance evaluation Both technical skills and a mound configuration were included on an evaluation form during the second phase of the study, to measure participant performance. The first and last injection from each study participant was evaluated by a pediatric urologist (A.K.). A paired t test was used to measure the performance differences between the initial and the last surgical attempt. Values of p < 0.05 were considered significant. In order to preserve integrity of comparison between variables and allow easier interpretation, the data were transformed to a 0e100 scale.
Results Content validity study Immediately after performing the procedure on the simulator, participants were asked to complete a questionnaire evaluating the ability of the simulator to realistically simulate the actual procedure. Table 1 summarizes the results of each question.
Table 1 (1e4).
As shown in Table 2, most surgical skill parameters had improved after 2 h of hands-on simulator curriculum training with a total improvement change noted to be 38% (p Z 0.01). Table 3 demonstrates the percentage change in the scores of the mound configuration grading after training. Total improvement in mound configuration increased 52% (p Z 0.01).
Discussion The teaching and assessment of technical skills on highfidelity simulation systems has its roots in the aviation industry and has become an important consideration in surgical education [11,12]. Simulated training allows for practice in a realistic setting without the inherent risk of harm to the patient. This stress free environment allows the trainee to specifically focus on the acquisition of surgical skills without any worry about surgical outcome. Another unique feature of the simulator is that mistakes, which are a crucial part of the learning experience, are encouraged, allowing the development of methods to avoid or manage procedural complications. During the past
Questionnaire evaluating the ability of the simulator to realistically simulate each step of the actual procedure. Score
Question
Most common response score (range)
Does the cystoscopy part of the procedure realistically simulate the actual procedure? Is the identification of the ureteral orifice part of the procedure a realistic simulation of the actual procedure? Does the ureteral orifice hydrodistended part of the procedure realistically simulate the actual procedure? Does the first injection (submucosal implantation within the intramural ureter) part of the procedure realistically simulate the actual procedure Does the second injection (submucosal implantation at the ureteral orifice) part of the procedure realistically simulate the actual procedure? Does the viscosity of the simulated injection material realistically simulate the actual procedure? Does the shape of the achieved injection mound realistically simulate that seen at an actual procedure? Does the post-injection hydrodistension part of the procedure realistically simulate that seen at an actual procedure?
4 (3e4) 4 (3e4) 4 (3e4) 4 (4) 4 (3e4) 3 (2e4) 4 (4) 4 (3e4)
Endoscopic correction of vesicoureteral reflux simulator Table 2
45.e5
Percentage change from before to after 2 h of hands-on simulator curriculum training.
Steps
Percentage change of score from before to after training (%)
p
Cystoscopy initiation Identification of the ureteral orifice Hydrodistension of the ureteral orifice First Dx/HA-like material injection Second Dx/HA-like material injection Post-injection lack of hydrodistension at the ureteral orifice Total
37 33 33 46 46 33 38
0.017 0.025 0.010 0.006 0.006 0.082 0.01
Dx/HA Z dextranomer/hyaluronic acid.
decade, surgical simulators have been developed and popularized as a component of skills training for a variety of minimally invasive surgical procedures [7e11]. Multiple studies have demonstrated improved performance in medical students, residents, fellows, and attending physicians of multiple specialties after simulation-based training, with generally positive feedback from learners after focused educational sessions [6e11]. More recent applications of simulation technology have focused on skills assessment, but it is critical that simulation-based training be implemented on the secure platform of a well-developed curriculum. Dx/HA needle injection is a 1e2-min, highly confined, unforgiving implantation of a discrete 1e3 ml of paste into an extremely small and delicate part of the tissue with little room for technical compensation, adjustment, intraoperative correction, or revision. In this study, we demonstrated that a dissected porcine bladder simulator can realistically simulate the actual surgical procedure, and it may be a tool for improving surgical skills. This provides a cost effective model on which to practice this challenging surgical procedure. It is known that the appearance of the Dx/HA mound does not sufficiently correlate with procedure outcome. A suboptimal appearing mound will sometimes correct the reflux and perfectly appearing mounds sometimes fail [13]. Since there is no validated tool to predict if a mound is successful, this simulator curriculum aimed to teach and allow practice in order to be able to perform the technically best achievable mound, and evaluate the participant on the basis of mound configuration.
Table 3 Percent change in mound configuration score before and after 2 h of hands-on simulator curriculum training. Criteria
Percentage change in score p from before to after training (%)
Appearance of slit Hydrodistension Depth Size Location Total
50 42 58 50 50 52
0.076 0.141 0.013 0.004 0.041 0.01
This simple porcine bladder model simulator is able to distinguish surgeons dependent on their clinical experience with cystoscopy, thereby demonstrating construct validity. Additionally, the implementation of this simulator within the developed curriculum can improve the performance of training urologists in all steps of the challenging technique of Dx/HA needle injection confirming concurrent validity. It was noted that the trainees had great improvement in their skills after their pediatric urology rotation (year 4). This would be expected, as they have greater experience with Dx/HA injection at this time and beyond. From this information it may be best to have residents complete their simulator training prior to their pediatric urology rotation to increase their skill set before patient contact. The next step in evaluation of this surgical skill-training curriculum would be to determine if the improvement in skill performance observed during training translates to improved performance in the clinical realm, or predictive validity. Some small differences exist between the porcine model and human ureteral orifices. In the porcine model, the ureteral orifices are located medially and distally in the bladder neck, which make injection more challenging. Participants suggested that after practicing with the simulator, endoscopic injection to a human ureteral orifice (which is usually laterally displaced) would be easier. Since there is no gold standard against which to assess the performance of those performing Dx/HA injection and therefore the simulation curriculum, we are unable to calculate criterion validity; therefore, we relied on discriminant and convergent validity to evaluate the value as an effective teaching tool. A single senior pediatric urologist evaluated the performance of all operators of the simulator. Absence of a control group, lack of testeretest, and non-blinding to the study of the evaluator goals were potential sources of bias. We as yet have no data on actual performance of the procedure in patients of subjects exposed to the simulation curriculum, nor do we have health outcomes of those patients. Further research is needed to compare the effectiveness of the simulator curriculum training methods on patient outcomes.
Conclusion The Dx/HA endoscopic injection simulator realistically simulates the actual clinical procedure. The simulator was
45.e6 able to differentiate between different levels of experience (before and after pediatric urology training). The simulator curriculum was able to improve the performance of the surgeon carrying out the procedure during subsequent simulations. This simulator may be an effective teaching tool and may improve the early learning curve for the clinical performance of this technique and provide a greater understanding of the components of successful endoscopic VUR correction.
Conflict of interest None.
Funding None.
References [1] Routh JC, Inman BA, Reinberg Y. Dextranomer/hyaluronic acid for pediatric vesicoureteral reflux: systematic review. Pediatrics 2010;125:1010. [2] Kirsch AJ, Perez-Brayfield M, Scherz HC. Minimally invasive treatment of vesicoureteral reflux with endoscopic injection of dextranomer/hyaluronic acid copolymer: the Children’s Hospitals of Atlanta experience. J Urol 2003;170:211. [3] Capozza N, Lais A, Nappo S, et al. The role of endoscopic treatment of vesicoureteral reflux: a 17-year experience. J Urol 2004;172:1626.
T. Soltani et al. [4] Dave S, Lorenzo AJ, Khoury AE, et al. Learning from the learning curve: Factors associated with successful endoscopic correction of vesicoureteral reflux using dextranomer/hyaluronic acid copolymer. J Urol 2008;180: 1594. [5] Carlin AM, Gasevic E, Shepard AD. Effect of the 80-hour work week on resident operative experience in general surgery. Am J Surg 2007;193:326. [6] Fried GM, Feldman LS, Vassiliou MC, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg 2004;240:518. [7] Rebolledo BJ, Hammann-Scala J, Leali A, et al. Arthroscopy skills development with a surgical simulator: a comparative study in orthopaedic surgery residents. Am J Sports Med 2015; 43:1526e9. [8] Martin KD, Patterson D, Phisitkul P, et al. Ankle arthroscopy simulation improves basic skills, anatomic recognition, and proficiency during diagnostic examination of residents in training. Foot Ankle Int 2015;36:827e35. [9] Hu Y, Le IA, Goodrich RN, et al. Construct validation of a costeffective vessel ligation benchtop simulator. J Surg Educ 2015; 72:381e6. [10] Makiyama K, Yamanaka H, Ueno D, et al. Validation of a patient-specific simulator for laparoscopic renal surgery. Int J Urol 2015;22:572e6. [11] Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidencebased virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244:310. [12] Seymour NE, Gallagher AG, Roman SA, et al. Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Ann Surg 2002;236: 458. [13] Hidas G, Soltani T, Watts B, et al. Is the appearance of the dextranomer/hyaluronic acid mound predictive of reflux resolution? J Urol 2013;189:1882e5.