- Email: [email protected]
Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training
+
MODEL
Journal of Pediatric Urology (xxxx) xxx xxx
Design and validation of a low-cost, highfidelity model for robotic pyeloplasty simulation training a
Texas Tech University Health Science Center, Lubbock, TX, USA
Matthew D. Timberlake a,1,2, Alaina Garbens b,1,3, Bruce J. Schlomer b,4, Nicholas L. Kavoussi c,5, Adam J.M. Kern b,6, Craig A. Peters b,7, Jeffery C. Gahan b,*
b
University of Texas Southwestern Medical School, Dallas, TX, USA
c Vanderbilt University, Nashville, TN, USA
* Corresponding author. West Campus Building 3, 4th floor, 2001 Inwood Road, Dallas, TX, 75390, USA. Tel.: þ214 645 8765. [email protected] (J.C. Gahan) Keywords Simulation training; Robotic surgery; Pediatric urology; Reconstructive urology; Ureteropelvic junction obstruction
Abbreviations ANOVA, analysis of variance; GEARS, global evaluative assessment of robotic skills; IRB, internal review board; UPJ, ureteropelvic junction; USD, US dollars; VR, virtual reality
Summary
subjective physical evaluation of repair quality (1e10 scale), and flow rate between experts and novices.
Introduction/Background
Results
Owing to restrictions in operative experiences, urology residents can no longer solely rely on ‘hands-on’ operative time to master their surgical skills by the end of residency. Simulation training could help residents master basic surgical skills and steps of a procedure to maximize time in the operative room. However, simulators can be expensive or tedious to set up, limiting the availability to residents and training programs.
Objective The authors sought to develop and validate an inexpensive, high-fidelity training model for robotic pyeloplasty.
Study design Pyeloplasty models were created using Dragon Skinâ FX-Pro tissue-mimicking silicone cast over 3-dimensional molds. Urology faculty and trainees completed a demographic questionnaire. The participants viewed a brief instructional video and then independently performed robotic dismembered pyeloplasty on the model. Acceptability and content validity were evaluated via post-task evaluation of the model. Construct validity was evaluated by comparing procedure completion time, the Global Evaluative Assessment of Robotic Skills (GEARS) score, blinded
In total, 5 urology faculty, 6 fellows, and 14 residents participated. The median robotic console experience among faculty, fellows, and residents was 8 years (interquartile range [IQR] Z 6e11), 3.5 years (IQR Z 2e4 years), and 0 years (IQR Z 0e0.5 years), respectively. The median procedure completion time was 29 min (IQR Z 26e40 min), and the median flow rate was 1.11 mL/s (IQR Z 0e1.34 mL/s). All faculty had flow rates >1.25 mL/s and procedure times <30 min compared with 2 of 6 fellows and none of the residents (P < 0.001). All faculty, half of the fellows, and none of the residents achieved a GEARS score 20, with a median resident score of 12.5 (IQR Z 8e13) (P < 0.001). For repair quality, all faculty scored 9 (out of 10), all fellows scored 8, and the median score among residents was 6 (IQR Z 2e6) (P < 0.001). The material cost was $1.32/model, and the average production time was 0.12 person-hours/model.
Discussion and conclusion This low-cost pyeloplasty model exhibits acceptability and content validity. Construct validity is supported by significant correlation between participant expertise and simulator performance across multiple assessment domains. The model has excellent potential to be used as a training tool in urology and allows for repetitive practice of pyeloplasty skills before live cases.
Received 11 November 2019 Accepted 1 February 2020 Available online xxx
Summary figure Figure showing completed pyeloplasty model mounted on the platform (left) and on the right, a participant completing a pyeloplasty using the model. 1 Denotes equal contribution and co-authorship. 2 3601 4th St, Stop 7260, Lubbock, TX, 79430e7260. Tel.: þ806 743 1810; fax: N/A. 3 West Campus Building 3, 4th floor, 2001 Inwood Road, Dallas, TX, 75390. Tel.: þ214 645 8408; fax: N/A. 4 2350 N Stemmons Fwy, Suite F4300, Dallas, TX, 75207. Tel.: þ214 456 2444; fax: þ214 456 8803. 5 1301 Medical Center Dr Suite 3823, Nashville, TN, 37232. Tel.: þ443 465 1485. 6 600 Ridgely Ave Ste 222, Annapolis, MD, 21401. Tel.: þ410 266 8049; fax: þ410 919 1481. 7 2350 N Stemmons Fwy, Suite F4300, Dallas, TX, 75207. Tel.: þ214 456 2444; fax: þ214 456 8803.
https://doi.org/10.1016/j.jpurol.2020.02.003 1477-5131/Published by Elsevier Ltd on behalf of Journal of Pediatric Urology Company.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
1.e2
Introduction Urology trainees face the challenge of mastering advanced robotic procedures during a brief residency experience. Traditionally, residents would master their skills through observation and participation in the operative theater (apprenticeship model). However, with limits on residency training hours, time constraints, patient safety concerns, and the adoption of increasingly complex technologies (i.e., robotics), there is less time for residents to learn surgical skills during the traditional operative experience [1]. Simulation training offers the ability for surgical trainees to master basic robotic and surgical skills before participating in the operative theater and has been shown to improve basic surgical skills in trainees [2,3]. A number of virtual reality (VR) platforms and physical analogue models (animal tissue and synthetic materials) have been designed to enhance robotic surgical training in urology. Although VR holds promise, to date, it has been unable to accurately replicate complicated tasks such as suturing and tissue dissection [4]. Although ideal in replicating real-life surgery, complex simulation training using animal or cadaveric models are expensive and logistically cumbersome. Simple models using synthetic materials can offer more cost-effective means for residents to learn fundamental steps and techniques (i.e., suturing) specific to a procedure. Indeed, the model’s cost is a significant
M.D. Timberlake et al. barrier to adopting new simulation technologies, and the number of validated inexpensive physical models is limited [5]. Although an ideal simulator should achieve tissue quality realism, operative task reproduction, and skill level differentiation [6,7], it must also be usable and affordable to allow wide adoption and sustainable repetitive practice. The anastomotic step of a robotic pyeloplasty is an ideal procedure for simulation training, given its technical complexity and skill transferability. Often, this procedure is performed too infrequently, especially in pediatrics to provide trainees with sufficient repetition to fully master the procedure during residency [8]. Although previous studies have described various anastomotic models, each had significant limitations and questionable usability [8]. In this study, the authors sought to design a low-cost, easily reproducible training model for robotic pyeloplasty and to evaluate the validity of the results using the Messick’s framework [9,10].
Methods Pyeloplasty model design and creation Three-dimensional molds were created using repurposed laparoscopic insufflation needles (ureter), modeling clay (dilated renal pelvis), and glue stick cartridges (posterolateral appendage) (Fig. 1A). The molds were cast with
Fig. 1 Design and construction of the pyeloplasty model. (A) Three-dimensional molds of the ureter and renal pelvis were constructed from repurposed laparoscopic insufflation needles (ureter), modeling clay (dilated renal pelvis), and glue stick cartridges. This mold made 4 models at once. (B) Completed model of the dilated renal pelvis and ureter showing the lateral appendage (A) for anchoring in the holder and post-task infusion of fluid, and longitudinal ‘vasculature’ (B) for post-task assessment of anastomotic alignment. (C) The model is positioned within its holder for simulation. (D) Appearance of the model from the robotic console.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
Low-cost pyeloplasty model for simulation training Dragon Skinâ FX-Pro tissue-mimicking silicone (Smooth-On Inc, Easton, PA). Model dimensions were intended to reproduce the size of a non-dilated ureter and dilated renal pelvis in a pediatric patient. The dilated renal pelvis measured 25 mm in short axis and 30 mm in long axis, tapering to a ureteropelvic junction (UPJ) and ureter with an inner diameter of 4 mm and wall thickness of 1 mm. Silicone infused with red acrylic was used to create the appearance of renal pelvic and ureteral blood vessels which traversed the UPJ in a longitudinal fashion, allowing for post-task subjective assessment of anastomotic alignment (Fig. 1B). A lateral hollow appendage was designed to anchor the model and permit injection of fluid after the procedure to assess for patency, leak, and flow rate (Fig. 1B). A platform to support the model at 45 during operation was used (Fig. 1C). Further details on model and platform construction are provided in the supplemental text. Seven batches of 4 models were produced using the molds shown in Fig. 1A. Production time for each batch (mixing silicone, casting molds, harvesting completed models) was measured, and mean ‘batch’ production time was used to calculate per-model production time (28 models). Each batch required 30 min of drying time. Production time was calculated as the mean man-hours per model, excluding drying time. Material cost was calculated based on volume of silicone required for production of each model.
Evaluation of model Participant recruitment Urology faculty, fellows, and residents at the UT Southwestern Medical Center were invited to participate. Participation was voluntary, and there were no participant exclusion criteria. The participants viewed a 3-min instructional video outlining the basic steps of the procedure and then performed a dismembered pyeloplasty on the model (dismemberment, spatulation, pyeloureteral anastomosis) using the DaVinciâ robotic surgical system (Intuitive Surgical, Sunnyvale, California). Pyeloplasty anastomosis was completed in running fashion using two 50 monocryl suture on an RB-1 needle (one anterior and one posterior). All participants completed the model once (Fig. 1D). There were no time limitations. Completed models were assigned an ID number for subsequent blinded assessment. All participants completed a post-task electronic questionnaire to report participant demographics and evaluate the model. Assessing validity The authors assessed the validity of the results using Messick’s framework [9]. Content evidence was assessed using expert review and score performance on the model. Response process was assessed using a 5-point Likert-based questionnaire for the examinees with the opportunity for open-ended/free-text feedback. The authors assessed relationships with other variables and consequences based on simulator performance relative to the surgeon’s experience (i.e., comparing expert and novice surgeons). Simulator performance was assessed based on (i) operative time, (ii)
1.e3 subjective quality of repair score, (iii) flow rate, and (iv) blinded video review of task performance. One blinded reviewer (M.D.T.) assessed each of the deidentified models for quality of repair and flow rate using subjective criteria based on consensus expert opinion. Only one review per model was possible as the process involved deconstructing the completed model. Subjective quality of the repair score was calculated based on 10 binary (yes/no) criteria (see Table 3). One point was awarded for each criterion, with a maximum possible score of 10. To measure the flow rate, 10 mL water was injected over 3 s, at a flow rate of 3.3 mL/ s into each completed model, and drainage time was measured. Drainage time was calculated from the start of water being injected into the model until the flow ended at the ureteral stump. The mean of three drainage trials was used to calculate flow rate (mL/second) for each participant. Models that had a completely obstructed anastomosis were included in the analysis and given a flow rate of 0 mL/ s. Models that had an anastomotic leak were excluded from flow rate analysis. Fig. 2 shows examples of repairs as well as flow rate and patency assessments. Videos of participant performances were edited to include 1 min of dismemberment/spatulation and 4 min of anastomotic suturing to keep the assessment time within practical limits for expert reviewers. Files were randomized and uploaded to Viddlerâ, a secure online video platform (Viddler, Inc., Bethlehem, PA). Three surgeons familiar with the Global Evaluative Assessment of Robotic Skills (GEARS) rating (J.C.G., A.J.M.K., and M.D.T.) reviewed the deidentified videos and scored participant performances as per the GEARS [11]. Surgical autonomy/independent task performance was not included in the final GEARS assessment as every participant was given a video demonstrating the task at the beginning without further feedback. Therefore, a modified GEARS score with a maximum score of 25 was used. Statistical analysis The participants were grouped into faculty, fellows, and residents based on their level of training. Descriptive statistics were performed on demographic variables. Dichotomous variables were compared across the three groups using the chi-squared test or Fisher’s exact test. Ordinal and continuous variables were compared across the three groups using the Kruskal-Wallis test. Post hoc testing was performed to determine which groups were statistically different. Differences in group means were analyzed using one-way analysis of variance tests. All P-values were two sided and set to <0.05 for significance. All statistical analyses were performed using SPSS, version 25 (International Business Machines Corporation, Armonk, NY).
Results Participant demographics Twenty-five urologists participated in the study, including 5 faculty (20%), 6 fellows (24%), and 14 residents (56%). Demographic data are presented in Table 1. Four of the five (80%) faculty were pediatric urologists. Fellows included all subspecialties of urology, including pediatrics.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
1.e4
M.D. Timberlake et al.
Fig. 2 Evaluation of participants’ performance on pyeloplasty model simulation. (A) Visual comparison of completed anastomosis from experienced (a) and inexperienced (b) participants. (B) Completed model undergoing flow rate analysis. (C) Completed model undergoing evaluation for patency.
All faculty rated their robotic surgical self-confidence with a score of 5 (1e5 scale), compared with 1 fellow (17%) and none of the residents (P < 0.001). Twelve residents (86%) rated their confidence as 1. The median robotic console Table 1
experience among faculty, fellows, and residents was 8 years (interquartile range [IQR] Z 6e11), 3.5 years (IQR Z 2e4 years), and 0 years (IQR Z 0e0.5 years), respectively (P < 0.001).
Participant characteristics.
Operative experience
Faculty (n Z 5)
Fellows (n Z 6)
Residents (n Z 14)
Median console experience Median robotic surgeries, career >5 RAL pyeloplasties, past 12 months
8 years (IQR Z 6e11) 250 (IQR Z 55e500) 5/5 (100%)
3.5 years (IQR Z 2e4) 55 (IQR Z 50e85) 3/6 (50%)
0 years (IQR Z 0e0.5 years) 0 (IQR Z 0e10) 2/14 (14%)
IQR, interquartile range; RAL, robotic assisted laparoscopic.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
Low-cost pyeloplasty model for simulation training
Response process, consequences, and content evidence Response processes were calculated based on responses from all participants and were also calculated based on responses from expert surgeons, faculty, and fellows only (n Z 11). Participants were excluded if they had not performed a pyeloplasty previously. Summary data are shown in Table 2. All faculty and fellows (11/11) agreed/strongly agreed that the model accurately reproduced dismemberment/spatulation skills and suturing/tying skills of robotic pyeloplasty. All agreed/strongly agreed that the model reproduced skills necessary for real pyeloplasty and that trainees should use this model to practice before surgery. All fellows and residents (20/20) agreed or strongly agreed that they would use the model to practice if it were available. Twenty-six participants (100%) agreed or strongly agreed that the model was easy to set up and use. The participants indicated that the model did not behave exactly like human tissue (2.85 0.80), describing the ureter as stiffer than normal human tissue, making the anastomosis on the model more challenging.
Subjective repair quality Subjective repair quality data are summarized in Table 3. Of faculty and fellows, none had an anastomotic leak, whereas leak was observed for 7 of 14 residents (50%). All 7 of these had been performed by residents with <6 months of console experience. All faculty and fellows had patent repairs. Two of 14 residents (14%) had no measurable flow through the repair, and both had an anastomotic leak. For the overall subjective repair quality score, all 5 faculty
Table 2
1.e5 scored 9 or higher (1e10 scale), all 6 fellows scored 8 or higher, and the median score among residents was 5 (IQR Z 2e6) (P < 0.001).
Content evidence, internal structure, and relationships with other variables Procedure time The results of procedure time are shown in Table 3. The overall, median procedure completion time was 29 min (IQR Z 26e40 min). The median procedure time for faculty was 26 min (IQR Z 23e28 min) compared with 27 min (IQR Z 24e31 min) for fellows and 37 min for residents (IQR Z 28e58 min) (P Z 0.02). Flow rate Seven models were excluded owing to significant anastomotic leak (all 7 performed by residents). The median flow rate for faculty was 1.38 mL/s (IQR Z 1.34e1.43 mL/s), compared with 1.23 mL/s (IQR Z 1.18e1.43 mL/s) for fellows and 1.04 mL/s (IQR Z 0.86e1.17) for residents (P Z 0.03). GEARS score All faculty (5/5), 3 of 6 fellows, and none of the 14 residents achieved a modified GEARS score 20/25. The mean score for faculty was 21.2 (IQR Z 20e23), for fellows was 20.1 (IQR Z 19e23), and for residents was 11.6 (IQR Z 8e13) (P < 0.001). On post hoc testing, a significant difference was found between the faculty’s GEARS score and residents’ scores and the fellows’ scores and residents’ scores (P Z 0.001 for both, Table 3) There was no significant difference between the faculty’s and fellows’ scores (P Z 0.9).
Summary of the participants’ evaluations of the model.
Realism assessment (1e5 Likert)a
Mean, SD
Content validity (1e5 Likert)a
Mean, SD
Size, shape anatomically accurate
4.31 0.48
4.77 0.44
Appearance from console Angle/vantage point
4.23 0.60 3.85 0.80
Thickness of the pelvis
4.15 0.55
Reproduces dismemberment, spatulation skills Reproduces suturing/tying skills Trainees should use this model before a real case Skills necessary for the model are similar to skills necessary for real pyeloplasty
Thickness of the ureter Behaved like human tissue overall Tissue behavior, needle pass, pelvis
3.46 0.97 2.85 0.80 3.62 1.12
Tissue behavior with needle pass, ureter
3.23 0.93
Tissue behavior, cutting Tissue behavior, suturing Gestalt (felt like a real pyeloplasty)
3.23 1.01 3.38 0.87 4.54 0.52
Acceptability (1e5 Likert)b Improved my ability to perform pyeloplasty I feel better prepared to perform dismemberment and spatulation I feel better prepared for anastomosis Model is easy to set up and use I would use this model to practice if available
4.85 0.38 4.92 0.28 4.62 0.51
Mean, SD 4.24 0.62 4.29 0.64 4.29 0.78 4.35 0.50 4.67 0.48
SD, standard deviation. a Limited to responses from experienced participants (>50 robotic cases and >1 robotic pyeloplasty as a primary surgeon). b Limited to responses from trainees (residents and fellows).
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
1.e6 Table 3
M.D. Timberlake et al. Subjective and objective scoring of the participants’ performance using the robotic pyeloplasty model.
Surgical descriptor
Faculty (n Z 5)
Fellows (n Z 6)
Residents (n Z 14)
P-value
No false cuts/dog ears Spatulation cut lateral Placed stitches evenly Knots tight/flat Avoided visible gaps Avoided back wall Repair patent No leak Anastomosis aligned Avoided bunching/folding Mean modified GEARS score (SD)a ANOVA results Faculty vs. fellows Faculty vs. residents Fellows vs. residents Median procedure time (IQR), minimum Median flow rate (ml/sec), (IQR)
5/5 (100%) 5/5 (100%) 5/5 (100%) 3/5 (60%) 5/5 (100%) 5/5 (100%) 5/5 (100%) 5/5 (100%) 5/5 (100%) 5/5 (100%) 21.2 (4.2)
6/6 (100%) 6/6 (100%) 6/6 (100%) 4/6 (67%) 5/6 (83%) 5/6 (83%) 6/6 (100%) 6/6 (100%) 6/6 (100%) 4/6 (67%) 20.1 (4.1)
1/14 (7%) 8/14 (57%) 3/14 (21%) 4/14 (29%) 3/14 (21%) 11/14 (79%) 12/14 (86%) 7/14 (50%) 8/14 (57%) 3/14 (21%) 11.6 (3.8)
<0.0001
37 (28e58) 1.04 (0.86e1.17)
0.9 0.001 0.001 0.02 0.03
26 (23e28) 1.38 (1.34e1.43)
27 (24e31) 1.23 (1.18e1.43)
IQR, interquartile range; SD, standard deviation; ANOVA, analysis of variance. a GEARS Z Global Evaluative Assessment of Robotic Skills. The original score is out of 30; the modified score is out of 25.
Combining factors All 5 faculty had patent repairs with flow rates >1.25 mL/s and procedure times <30 min, compared with 2 of 6 fellows and none of the 14 residents (P < 0.001).
Cost of materials The material cost of the four 3D molds (used laparoscopic insufflation needles, modeling clay, glue stick cartridges) was USD $5 and that of the mounting platform was USD $3. The material cost for each individual simulation model was USD $1.32, and the average per-model production time was 0.12 person-hours/model.
Discussion The authors sought to design and validate a low-cost and realistic pyeloplasty model that would allow for distributed repetition and implemented Messick’s framework to evaluate the validity of their findings. Content evidence, internal structure, and relationships with other variables were supported by significant correlation between the participant’s experience and simulator performance across multiple domains, including procedure time, flow rate, GEARS score, and subjective quality of repair. The authors observed discrimination of faculty from residents using a procedure time <30 min and flow rate >1.25 mL/s. The authors’ model performed very well in evaluating objective outcomes (flow, time, and GEARS score) after pyeloplasty. Furthermore, the models can be produced rapidly (0.12 person-hours per model) and at very low cost ($1.32 per model). The design does not require access to advanced technologies or materials expertise and is simple enough to be widely reproduced at any institution for resident training.
Robotic surgical simulation for task training continues to be an evolving field. Soft tissueelike materials have been 3D printed with embedded electronics/sensors that may allow force sensitivity data analysis and patient-specific rehearsal to be performed [12]. In addition, VR simulators continue to improve with advances in computing power and improved software design. However, these simulators take time to manufacture and to be assembled and are expensive. Less complex solutions are currently needed to allow residents to hone procedure-specific skills through training repetition. Indeed, in a 2013 meta-analysis, Cook et al. [13] quantified the effectiveness of several instructional design features for simulation-based education. Factors demonstrating greatest effect when pooled included repetitive and temporally distributed practice. Distributed repetition has been shown to be superior to massed practice (e.g., annual robotic laboratory) with respect to laparoscopic skill acquisition and short- and long-term skill retention. The effect was particularly pronounced for difficult tasks such as intracorporeal suturing [14]. Of the simulation models described in urology, many are proprietary, not commercially available, or associated with significant cost [5,7]. Other physical analogue models for minimally invasive pyeloplasty are much more costly. For example, the BLAST pyeloplasty model costs US $55/model [15], whereas the SimuLab pediatric pyeloplasty model costs US $650 [8]. Although 3D printing is emerging as a way to develop very specific models with more realistic materials, this technology involves expensive materials, specialized expertise, and time-consuming production when performed at scale [8,16]. Currently, there exist very few low-cost pyeloplasty models. Ooi et al. [17] created a pyeloplasty model using chicken thighs and skin, and Ramachandran et al. [18] developed a model using the chicken esophagus and croup to simulate the ureter and renal pelvis, respectively. The chicken croup model was subsequently validated by Jiang
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
Low-cost pyeloplasty model for simulation training et al. [19] and found to have good construct validity. However, because these are animal-based models, significant preparation of the specimen must be performed, limiting large-scale use and limiting repetitive practice. Furthermore, the simulation area and robotic instruments must be thoroughly cleaned after each use, limiting availability especially in the operating room. Other ex vivo models using porcine and fish bladders have been proposed; however, they have similar limitations [20,21]. Rod et al. [22] have developed a low-cost balloon model for open dismembered pyeloplasty. However, assessment of the anastomosis was not performed, and many participants found that the model was not difficult enough as it did not replicate the restricted working space and anatomical position of the renal pelvis in children. There are several limitations to the study model. Most notably, the silicone used is stiffer and more resistant to tearing than human and animal tissue. Although reviewers commented on anatomical accuracy, realistic appearance, and appropriate renal pelvic thickness, they noted that tissue behavior was not the same as real tissue. For example, the participants indicated that ureteral spatulation did not produce a realistic splaying of cut edges. Four faculty participants indicated that these factors made performance of ureteral anastomosis more difficult than actual surgery. While increased technical difficulty may be acceptable in a training model, the participants also indicated thatsilicone was more difficult to tear than human tissue. This is problematic as trainees may apply improper forces when they transition to actual patients, causing tissue tearing. The authors further chose not to require review of the entire video when assigning a GEARS score. The authors instead chose to edit the videos and shorten each to 4 representative minutes in total. With a median time of nearly 30 min, it would be impractical for each of the 3 reviewers to watch each of the 26 videos in their entirety. Finally, the study model did not replicate the restricted workspace as would be seen in an actual pediatric pyeloplasty. Since this study, the authors now use a small box trainer to further limit space and range of motion. This study is also limited by a relatively small sample size (n Z 25) and a small number of faculty participants. Despite this limitation, the authors were able to demonstrate construct validity with their model. The cost calculations are limited to model material cost and do not include the considerable costs acquiring and/or maintaining the robotic training system and equipment. However, as this model does not contain any animal products, it could easily be brought into the operative theater if programs do not have a dedicated robotic system for training. Furthermore, this model could also be used for laparoscopic surgery without modification, expanding its use to programs without robotic systems.
Conclusions The authors developed a simple, low-cost pyeloplasty model that is highly acceptable to trainees and faculty and demonstrates preliminary content and construct validity. Subjective quality of repair, GEARS score, and reported
1.e7 self-confidence also were accurately discriminated using this model. Experienced robotic surgeons indicated that the model accurately reproduced pyeloplasty skills and can be used by trainees for repetitive practice. Future investigations are needed to improve tissue realism and examine the impact of repetitive simulator training on pyeloplasty performance.
Author statements Acknowledgments This study would not have been possible without the collaborative expertise of Bradley Mueller and Matthew Kozemond in the UT Southwestern surgical simulation laboratory.
Ethical approval This study did not meet the definition of human research that requires ethics review or approval at the authors’ institution, and a formal review by the internal review board was waived.
Funding The authors have no funding to disclose.
Competing interests The authors have no conflicts of interest to disclose.
References [1] Hutter MM, Kellogg KC, Ferguson CM, Abbott WM, Warshaw AL. The impact of the 80-hour resident workweek on surgical residents and attending surgeons. Ann Surg 2006;243(6): 864e71. https: //doi.org/10.1097/01.sla.0000220042.48310.66. [2] Akhtar KSN, Chen A, Standfield NJ, Gupte CM. The role of simulation in developing surgical skills. Curr Rev Musculoskel Med 2014;7(2):155e60. https://doi.org/10.1007/s12178-0149209-z. [3] Howells NR, Gill HS, Carr AJ, Price AJ, Rees JL. Transferring simulated arthroscopic skills to the operating theatre: a randomised blinded study. J Bone Joint Surg Br 2008;90(4): 494e9. https://doi.org/10.1302/0301-620X.90B4.20414. [4] Liu M, Curet M. A Review of Training Research and Virtual Reality Simulators for the da Vinci Surgical System. Teach Learn Med 2015;27(1):12e26. https://doi.org/10.1080/ 10401334.2014.979181. [5] Aydin A, Raison N, Khan MS, Dasgupta P, Ahmed K. Simulationbased training and assessment in urological surgery. Nat Rev Urol 2016;13(9):503e19. https://doi.org/10.1038/ nrurol.2016.147. [6] Wignall GR, Denstedt JD, Preminger GM, Cadeddu JA, Pearle M, Sweet R, et al. Surgical simulation: a urological perspective. J Urol 2008;179(5):1690e9. https: //doi.org/10.1016/j.juro.2008.01.014. [7] Moglia A, Ferrari V, Morelli L, Ferrari M, Mosca F, Cuschieri A. A systematic review of virtual reality simulators for robot-
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
1.e8
[8]
[9]
[10]
[11]
[12]
[13]
[14]
M.D. Timberlake et al. assisted surgery. Eur Urol 2016;69(6):1065e80. https: //doi.org/10.1016/j.eururo.2015.09.021. Cheung CL, Looi T, Lendvay TS, Drake JM, Farhat WA. Use of 3dimensional printing technology and silicone modeling in surgical simulation: development and face validation in pediatric laparoscopic pyeloplasty. J Surg Educ 2014;71(5): 762e7. https://doi.org/10.1016/j.jsurg.2014.03.001. Cook DA, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Adv Simulat 2016;1(1):31. https://doi.org/10.1186/s41077-016-0033-y. Noureldin YA, Lee JY, McDougall EM, Sweet RM. Competencybased training and simulation: making a “valid” argument. J Endourol 2018;32(2):84e93. https://doi.org/10.1089/ end.2017.0650. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ. Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 2012; 187(1):247e52. https://doi.org/10.1016/j.juro.2011.09.032. Garcia J, Yang ZL, Mongrain R, Leask RL, Lachapelle K. 3D printing materials and their use in medical education: a review of current technology and trends for the future. BMJ Simulat Technol Enhanc Learn 2018;4(1):27e40. https: //doi.org/10.1136/bmjstel-2017-000234. Cook DA, Hamstra SJ, Brydges R, Zendejas B, Szostek J, Wang A, et al. Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis. Med Teach 2013;35(1). https: //doi.org/10.3109/0142159X.2012.714886. Spruit EN, Band GPH, Hamming JF. Increasing efficiency of surgical training: effects of spacing practice on skill acquisition and retention in laparoscopy training. Surg Endosc 2015; 29(8):2235e43. https://doi.org/10.1007/s00464-014-3931-x.
[15] Poniatowski LH, Wolf JS, Nakada SY, Reihsen TE, Sainfort F, Sweet RM. Validity and acceptability of a high-fidelity physical simulation model for training of laparoscopic pyeloplasty. J Endourol 2014;28(4):393e8. https://doi.org/10.1089/end. 2013.0678. [16] Adams F, Qiu T, Mark A, Fritz B, Kramer L, Schlager D, et al. Soft 3D-printed phantom of the human kidney with collecting system. Ann Biomed Eng 2017;45(4):963e72. https: //doi.org/10.1007/s10439-016-1757-5. [17] Ooi J, Lawrentschuk N, Murphy DL. Training model for open or laparoscopic pyeloplasty, vol. 20; 2006. www.liebertpub.com. [18] Ramachandran A, Kurien A, Symons S, Ganpule A, Muthu V, Ch M, et al. A novel training model for laparoscopic pyeloplasty using chicken crop. J Endour ol 2008;22(4). https://doi.org/10.1089/end.2007.0380. [19] Jiang C, Liu M, Chen J, Wang P, Lin T, Xu K, et al. Construct validity of the chicken crop model in the simulation of laparoscopic pyeloplasty. J Endourol 2013. https: //doi.org/10.1089/end.2013.0085. [20] Yang B, Zhang Z, Xiao L, Wang L, Xu C, Sun Y. A novel training model for retroperitoneal laparoscopic dismembered pyeloplasty. J Endourol 2010;24(8):1345e9. https://doi.org/10. 1089/end.2009.0408. [21] Teber D, Guven S, Yaycioglu O, Ugurlu O, Sanli O, Gozen A, et al. Single-knot running suture anastomosis (one-knot pyeloplasty) for laparoscopic dismembered pyeloplasty: training model on a porcine bladder and clinical results. Int Urol Nephrol 2010;42(3):609e14. https://doi.org/10.1007/s11255009-9668-0. [22] Rod J, Marret JB, Kohaut J, Aigrain Y, Jais J, de Vries P, et al. Low-cost training simulator for open dismembered pyeloplasty: development and face validation. J Surg Educ 2018; 75(1):188e94. https://doi.org/10.1016/j.jsurg.2017.06.010.
Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi.org/10.1016/j.jpurol.2020.02.003.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
MODEL
Journal of Pediatric Urology (xxxx) xxx xxx
Design and validation of a low-cost, highfidelity model for robotic pyeloplasty simulation training a
Texas Tech University Health Science Center, Lubbock, TX, USA
Matthew D. Timberlake a,1,2, Alaina Garbens b,1,3, Bruce J. Schlomer b,4, Nicholas L. Kavoussi c,5, Adam J.M. Kern b,6, Craig A. Peters b,7, Jeffery C. Gahan b,*
b
University of Texas Southwestern Medical School, Dallas, TX, USA
c Vanderbilt University, Nashville, TN, USA
* Corresponding author. West Campus Building 3, 4th floor, 2001 Inwood Road, Dallas, TX, 75390, USA. Tel.: þ214 645 8765. [email protected] (J.C. Gahan) Keywords Simulation training; Robotic surgery; Pediatric urology; Reconstructive urology; Ureteropelvic junction obstruction
Abbreviations ANOVA, analysis of variance; GEARS, global evaluative assessment of robotic skills; IRB, internal review board; UPJ, ureteropelvic junction; USD, US dollars; VR, virtual reality
Summary
subjective physical evaluation of repair quality (1e10 scale), and flow rate between experts and novices.
Introduction/Background
Results
Owing to restrictions in operative experiences, urology residents can no longer solely rely on ‘hands-on’ operative time to master their surgical skills by the end of residency. Simulation training could help residents master basic surgical skills and steps of a procedure to maximize time in the operative room. However, simulators can be expensive or tedious to set up, limiting the availability to residents and training programs.
Objective The authors sought to develop and validate an inexpensive, high-fidelity training model for robotic pyeloplasty.
Study design Pyeloplasty models were created using Dragon Skinâ FX-Pro tissue-mimicking silicone cast over 3-dimensional molds. Urology faculty and trainees completed a demographic questionnaire. The participants viewed a brief instructional video and then independently performed robotic dismembered pyeloplasty on the model. Acceptability and content validity were evaluated via post-task evaluation of the model. Construct validity was evaluated by comparing procedure completion time, the Global Evaluative Assessment of Robotic Skills (GEARS) score, blinded
In total, 5 urology faculty, 6 fellows, and 14 residents participated. The median robotic console experience among faculty, fellows, and residents was 8 years (interquartile range [IQR] Z 6e11), 3.5 years (IQR Z 2e4 years), and 0 years (IQR Z 0e0.5 years), respectively. The median procedure completion time was 29 min (IQR Z 26e40 min), and the median flow rate was 1.11 mL/s (IQR Z 0e1.34 mL/s). All faculty had flow rates >1.25 mL/s and procedure times <30 min compared with 2 of 6 fellows and none of the residents (P < 0.001). All faculty, half of the fellows, and none of the residents achieved a GEARS score 20, with a median resident score of 12.5 (IQR Z 8e13) (P < 0.001). For repair quality, all faculty scored 9 (out of 10), all fellows scored 8, and the median score among residents was 6 (IQR Z 2e6) (P < 0.001). The material cost was $1.32/model, and the average production time was 0.12 person-hours/model.
Discussion and conclusion This low-cost pyeloplasty model exhibits acceptability and content validity. Construct validity is supported by significant correlation between participant expertise and simulator performance across multiple assessment domains. The model has excellent potential to be used as a training tool in urology and allows for repetitive practice of pyeloplasty skills before live cases.
Received 11 November 2019 Accepted 1 February 2020 Available online xxx
Summary figure Figure showing completed pyeloplasty model mounted on the platform (left) and on the right, a participant completing a pyeloplasty using the model. 1 Denotes equal contribution and co-authorship. 2 3601 4th St, Stop 7260, Lubbock, TX, 79430e7260. Tel.: þ806 743 1810; fax: N/A. 3 West Campus Building 3, 4th floor, 2001 Inwood Road, Dallas, TX, 75390. Tel.: þ214 645 8408; fax: N/A. 4 2350 N Stemmons Fwy, Suite F4300, Dallas, TX, 75207. Tel.: þ214 456 2444; fax: þ214 456 8803. 5 1301 Medical Center Dr Suite 3823, Nashville, TN, 37232. Tel.: þ443 465 1485. 6 600 Ridgely Ave Ste 222, Annapolis, MD, 21401. Tel.: þ410 266 8049; fax: þ410 919 1481. 7 2350 N Stemmons Fwy, Suite F4300, Dallas, TX, 75207. Tel.: þ214 456 2444; fax: þ214 456 8803.
https://doi.org/10.1016/j.jpurol.2020.02.003 1477-5131/Published by Elsevier Ltd on behalf of Journal of Pediatric Urology Company.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
1.e2
Introduction Urology trainees face the challenge of mastering advanced robotic procedures during a brief residency experience. Traditionally, residents would master their skills through observation and participation in the operative theater (apprenticeship model). However, with limits on residency training hours, time constraints, patient safety concerns, and the adoption of increasingly complex technologies (i.e., robotics), there is less time for residents to learn surgical skills during the traditional operative experience [1]. Simulation training offers the ability for surgical trainees to master basic robotic and surgical skills before participating in the operative theater and has been shown to improve basic surgical skills in trainees [2,3]. A number of virtual reality (VR) platforms and physical analogue models (animal tissue and synthetic materials) have been designed to enhance robotic surgical training in urology. Although VR holds promise, to date, it has been unable to accurately replicate complicated tasks such as suturing and tissue dissection [4]. Although ideal in replicating real-life surgery, complex simulation training using animal or cadaveric models are expensive and logistically cumbersome. Simple models using synthetic materials can offer more cost-effective means for residents to learn fundamental steps and techniques (i.e., suturing) specific to a procedure. Indeed, the model’s cost is a significant
M.D. Timberlake et al. barrier to adopting new simulation technologies, and the number of validated inexpensive physical models is limited [5]. Although an ideal simulator should achieve tissue quality realism, operative task reproduction, and skill level differentiation [6,7], it must also be usable and affordable to allow wide adoption and sustainable repetitive practice. The anastomotic step of a robotic pyeloplasty is an ideal procedure for simulation training, given its technical complexity and skill transferability. Often, this procedure is performed too infrequently, especially in pediatrics to provide trainees with sufficient repetition to fully master the procedure during residency [8]. Although previous studies have described various anastomotic models, each had significant limitations and questionable usability [8]. In this study, the authors sought to design a low-cost, easily reproducible training model for robotic pyeloplasty and to evaluate the validity of the results using the Messick’s framework [9,10].
Methods Pyeloplasty model design and creation Three-dimensional molds were created using repurposed laparoscopic insufflation needles (ureter), modeling clay (dilated renal pelvis), and glue stick cartridges (posterolateral appendage) (Fig. 1A). The molds were cast with
Fig. 1 Design and construction of the pyeloplasty model. (A) Three-dimensional molds of the ureter and renal pelvis were constructed from repurposed laparoscopic insufflation needles (ureter), modeling clay (dilated renal pelvis), and glue stick cartridges. This mold made 4 models at once. (B) Completed model of the dilated renal pelvis and ureter showing the lateral appendage (A) for anchoring in the holder and post-task infusion of fluid, and longitudinal ‘vasculature’ (B) for post-task assessment of anastomotic alignment. (C) The model is positioned within its holder for simulation. (D) Appearance of the model from the robotic console.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
Low-cost pyeloplasty model for simulation training Dragon Skinâ FX-Pro tissue-mimicking silicone (Smooth-On Inc, Easton, PA). Model dimensions were intended to reproduce the size of a non-dilated ureter and dilated renal pelvis in a pediatric patient. The dilated renal pelvis measured 25 mm in short axis and 30 mm in long axis, tapering to a ureteropelvic junction (UPJ) and ureter with an inner diameter of 4 mm and wall thickness of 1 mm. Silicone infused with red acrylic was used to create the appearance of renal pelvic and ureteral blood vessels which traversed the UPJ in a longitudinal fashion, allowing for post-task subjective assessment of anastomotic alignment (Fig. 1B). A lateral hollow appendage was designed to anchor the model and permit injection of fluid after the procedure to assess for patency, leak, and flow rate (Fig. 1B). A platform to support the model at 45 during operation was used (Fig. 1C). Further details on model and platform construction are provided in the supplemental text. Seven batches of 4 models were produced using the molds shown in Fig. 1A. Production time for each batch (mixing silicone, casting molds, harvesting completed models) was measured, and mean ‘batch’ production time was used to calculate per-model production time (28 models). Each batch required 30 min of drying time. Production time was calculated as the mean man-hours per model, excluding drying time. Material cost was calculated based on volume of silicone required for production of each model.
Evaluation of model Participant recruitment Urology faculty, fellows, and residents at the UT Southwestern Medical Center were invited to participate. Participation was voluntary, and there were no participant exclusion criteria. The participants viewed a 3-min instructional video outlining the basic steps of the procedure and then performed a dismembered pyeloplasty on the model (dismemberment, spatulation, pyeloureteral anastomosis) using the DaVinciâ robotic surgical system (Intuitive Surgical, Sunnyvale, California). Pyeloplasty anastomosis was completed in running fashion using two 50 monocryl suture on an RB-1 needle (one anterior and one posterior). All participants completed the model once (Fig. 1D). There were no time limitations. Completed models were assigned an ID number for subsequent blinded assessment. All participants completed a post-task electronic questionnaire to report participant demographics and evaluate the model. Assessing validity The authors assessed the validity of the results using Messick’s framework [9]. Content evidence was assessed using expert review and score performance on the model. Response process was assessed using a 5-point Likert-based questionnaire for the examinees with the opportunity for open-ended/free-text feedback. The authors assessed relationships with other variables and consequences based on simulator performance relative to the surgeon’s experience (i.e., comparing expert and novice surgeons). Simulator performance was assessed based on (i) operative time, (ii)
1.e3 subjective quality of repair score, (iii) flow rate, and (iv) blinded video review of task performance. One blinded reviewer (M.D.T.) assessed each of the deidentified models for quality of repair and flow rate using subjective criteria based on consensus expert opinion. Only one review per model was possible as the process involved deconstructing the completed model. Subjective quality of the repair score was calculated based on 10 binary (yes/no) criteria (see Table 3). One point was awarded for each criterion, with a maximum possible score of 10. To measure the flow rate, 10 mL water was injected over 3 s, at a flow rate of 3.3 mL/ s into each completed model, and drainage time was measured. Drainage time was calculated from the start of water being injected into the model until the flow ended at the ureteral stump. The mean of three drainage trials was used to calculate flow rate (mL/second) for each participant. Models that had a completely obstructed anastomosis were included in the analysis and given a flow rate of 0 mL/ s. Models that had an anastomotic leak were excluded from flow rate analysis. Fig. 2 shows examples of repairs as well as flow rate and patency assessments. Videos of participant performances were edited to include 1 min of dismemberment/spatulation and 4 min of anastomotic suturing to keep the assessment time within practical limits for expert reviewers. Files were randomized and uploaded to Viddlerâ, a secure online video platform (Viddler, Inc., Bethlehem, PA). Three surgeons familiar with the Global Evaluative Assessment of Robotic Skills (GEARS) rating (J.C.G., A.J.M.K., and M.D.T.) reviewed the deidentified videos and scored participant performances as per the GEARS [11]. Surgical autonomy/independent task performance was not included in the final GEARS assessment as every participant was given a video demonstrating the task at the beginning without further feedback. Therefore, a modified GEARS score with a maximum score of 25 was used. Statistical analysis The participants were grouped into faculty, fellows, and residents based on their level of training. Descriptive statistics were performed on demographic variables. Dichotomous variables were compared across the three groups using the chi-squared test or Fisher’s exact test. Ordinal and continuous variables were compared across the three groups using the Kruskal-Wallis test. Post hoc testing was performed to determine which groups were statistically different. Differences in group means were analyzed using one-way analysis of variance tests. All P-values were two sided and set to <0.05 for significance. All statistical analyses were performed using SPSS, version 25 (International Business Machines Corporation, Armonk, NY).
Results Participant demographics Twenty-five urologists participated in the study, including 5 faculty (20%), 6 fellows (24%), and 14 residents (56%). Demographic data are presented in Table 1. Four of the five (80%) faculty were pediatric urologists. Fellows included all subspecialties of urology, including pediatrics.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
1.e4
M.D. Timberlake et al.
Fig. 2 Evaluation of participants’ performance on pyeloplasty model simulation. (A) Visual comparison of completed anastomosis from experienced (a) and inexperienced (b) participants. (B) Completed model undergoing flow rate analysis. (C) Completed model undergoing evaluation for patency.
All faculty rated their robotic surgical self-confidence with a score of 5 (1e5 scale), compared with 1 fellow (17%) and none of the residents (P < 0.001). Twelve residents (86%) rated their confidence as 1. The median robotic console Table 1
experience among faculty, fellows, and residents was 8 years (interquartile range [IQR] Z 6e11), 3.5 years (IQR Z 2e4 years), and 0 years (IQR Z 0e0.5 years), respectively (P < 0.001).
Participant characteristics.
Operative experience
Faculty (n Z 5)
Fellows (n Z 6)
Residents (n Z 14)
Median console experience Median robotic surgeries, career >5 RAL pyeloplasties, past 12 months
8 years (IQR Z 6e11) 250 (IQR Z 55e500) 5/5 (100%)
3.5 years (IQR Z 2e4) 55 (IQR Z 50e85) 3/6 (50%)
0 years (IQR Z 0e0.5 years) 0 (IQR Z 0e10) 2/14 (14%)
IQR, interquartile range; RAL, robotic assisted laparoscopic.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
Low-cost pyeloplasty model for simulation training
Response process, consequences, and content evidence Response processes were calculated based on responses from all participants and were also calculated based on responses from expert surgeons, faculty, and fellows only (n Z 11). Participants were excluded if they had not performed a pyeloplasty previously. Summary data are shown in Table 2. All faculty and fellows (11/11) agreed/strongly agreed that the model accurately reproduced dismemberment/spatulation skills and suturing/tying skills of robotic pyeloplasty. All agreed/strongly agreed that the model reproduced skills necessary for real pyeloplasty and that trainees should use this model to practice before surgery. All fellows and residents (20/20) agreed or strongly agreed that they would use the model to practice if it were available. Twenty-six participants (100%) agreed or strongly agreed that the model was easy to set up and use. The participants indicated that the model did not behave exactly like human tissue (2.85 0.80), describing the ureter as stiffer than normal human tissue, making the anastomosis on the model more challenging.
Subjective repair quality Subjective repair quality data are summarized in Table 3. Of faculty and fellows, none had an anastomotic leak, whereas leak was observed for 7 of 14 residents (50%). All 7 of these had been performed by residents with <6 months of console experience. All faculty and fellows had patent repairs. Two of 14 residents (14%) had no measurable flow through the repair, and both had an anastomotic leak. For the overall subjective repair quality score, all 5 faculty
Table 2
1.e5 scored 9 or higher (1e10 scale), all 6 fellows scored 8 or higher, and the median score among residents was 5 (IQR Z 2e6) (P < 0.001).
Content evidence, internal structure, and relationships with other variables Procedure time The results of procedure time are shown in Table 3. The overall, median procedure completion time was 29 min (IQR Z 26e40 min). The median procedure time for faculty was 26 min (IQR Z 23e28 min) compared with 27 min (IQR Z 24e31 min) for fellows and 37 min for residents (IQR Z 28e58 min) (P Z 0.02). Flow rate Seven models were excluded owing to significant anastomotic leak (all 7 performed by residents). The median flow rate for faculty was 1.38 mL/s (IQR Z 1.34e1.43 mL/s), compared with 1.23 mL/s (IQR Z 1.18e1.43 mL/s) for fellows and 1.04 mL/s (IQR Z 0.86e1.17) for residents (P Z 0.03). GEARS score All faculty (5/5), 3 of 6 fellows, and none of the 14 residents achieved a modified GEARS score 20/25. The mean score for faculty was 21.2 (IQR Z 20e23), for fellows was 20.1 (IQR Z 19e23), and for residents was 11.6 (IQR Z 8e13) (P < 0.001). On post hoc testing, a significant difference was found between the faculty’s GEARS score and residents’ scores and the fellows’ scores and residents’ scores (P Z 0.001 for both, Table 3) There was no significant difference between the faculty’s and fellows’ scores (P Z 0.9).
Summary of the participants’ evaluations of the model.
Realism assessment (1e5 Likert)a
Mean, SD
Content validity (1e5 Likert)a
Mean, SD
Size, shape anatomically accurate
4.31 0.48
4.77 0.44
Appearance from console Angle/vantage point
4.23 0.60 3.85 0.80
Thickness of the pelvis
4.15 0.55
Reproduces dismemberment, spatulation skills Reproduces suturing/tying skills Trainees should use this model before a real case Skills necessary for the model are similar to skills necessary for real pyeloplasty
Thickness of the ureter Behaved like human tissue overall Tissue behavior, needle pass, pelvis
3.46 0.97 2.85 0.80 3.62 1.12
Tissue behavior with needle pass, ureter
3.23 0.93
Tissue behavior, cutting Tissue behavior, suturing Gestalt (felt like a real pyeloplasty)
3.23 1.01 3.38 0.87 4.54 0.52
Acceptability (1e5 Likert)b Improved my ability to perform pyeloplasty I feel better prepared to perform dismemberment and spatulation I feel better prepared for anastomosis Model is easy to set up and use I would use this model to practice if available
4.85 0.38 4.92 0.28 4.62 0.51
Mean, SD 4.24 0.62 4.29 0.64 4.29 0.78 4.35 0.50 4.67 0.48
SD, standard deviation. a Limited to responses from experienced participants (>50 robotic cases and >1 robotic pyeloplasty as a primary surgeon). b Limited to responses from trainees (residents and fellows).
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
1.e6 Table 3
M.D. Timberlake et al. Subjective and objective scoring of the participants’ performance using the robotic pyeloplasty model.
Surgical descriptor
Faculty (n Z 5)
Fellows (n Z 6)
Residents (n Z 14)
P-value
No false cuts/dog ears Spatulation cut lateral Placed stitches evenly Knots tight/flat Avoided visible gaps Avoided back wall Repair patent No leak Anastomosis aligned Avoided bunching/folding Mean modified GEARS score (SD)a ANOVA results Faculty vs. fellows Faculty vs. residents Fellows vs. residents Median procedure time (IQR), minimum Median flow rate (ml/sec), (IQR)
5/5 (100%) 5/5 (100%) 5/5 (100%) 3/5 (60%) 5/5 (100%) 5/5 (100%) 5/5 (100%) 5/5 (100%) 5/5 (100%) 5/5 (100%) 21.2 (4.2)
6/6 (100%) 6/6 (100%) 6/6 (100%) 4/6 (67%) 5/6 (83%) 5/6 (83%) 6/6 (100%) 6/6 (100%) 6/6 (100%) 4/6 (67%) 20.1 (4.1)
1/14 (7%) 8/14 (57%) 3/14 (21%) 4/14 (29%) 3/14 (21%) 11/14 (79%) 12/14 (86%) 7/14 (50%) 8/14 (57%) 3/14 (21%) 11.6 (3.8)
<0.0001
37 (28e58) 1.04 (0.86e1.17)
0.9 0.001 0.001 0.02 0.03
26 (23e28) 1.38 (1.34e1.43)
27 (24e31) 1.23 (1.18e1.43)
IQR, interquartile range; SD, standard deviation; ANOVA, analysis of variance. a GEARS Z Global Evaluative Assessment of Robotic Skills. The original score is out of 30; the modified score is out of 25.
Combining factors All 5 faculty had patent repairs with flow rates >1.25 mL/s and procedure times <30 min, compared with 2 of 6 fellows and none of the 14 residents (P < 0.001).
Cost of materials The material cost of the four 3D molds (used laparoscopic insufflation needles, modeling clay, glue stick cartridges) was USD $5 and that of the mounting platform was USD $3. The material cost for each individual simulation model was USD $1.32, and the average per-model production time was 0.12 person-hours/model.
Discussion The authors sought to design and validate a low-cost and realistic pyeloplasty model that would allow for distributed repetition and implemented Messick’s framework to evaluate the validity of their findings. Content evidence, internal structure, and relationships with other variables were supported by significant correlation between the participant’s experience and simulator performance across multiple domains, including procedure time, flow rate, GEARS score, and subjective quality of repair. The authors observed discrimination of faculty from residents using a procedure time <30 min and flow rate >1.25 mL/s. The authors’ model performed very well in evaluating objective outcomes (flow, time, and GEARS score) after pyeloplasty. Furthermore, the models can be produced rapidly (0.12 person-hours per model) and at very low cost ($1.32 per model). The design does not require access to advanced technologies or materials expertise and is simple enough to be widely reproduced at any institution for resident training.
Robotic surgical simulation for task training continues to be an evolving field. Soft tissueelike materials have been 3D printed with embedded electronics/sensors that may allow force sensitivity data analysis and patient-specific rehearsal to be performed [12]. In addition, VR simulators continue to improve with advances in computing power and improved software design. However, these simulators take time to manufacture and to be assembled and are expensive. Less complex solutions are currently needed to allow residents to hone procedure-specific skills through training repetition. Indeed, in a 2013 meta-analysis, Cook et al. [13] quantified the effectiveness of several instructional design features for simulation-based education. Factors demonstrating greatest effect when pooled included repetitive and temporally distributed practice. Distributed repetition has been shown to be superior to massed practice (e.g., annual robotic laboratory) with respect to laparoscopic skill acquisition and short- and long-term skill retention. The effect was particularly pronounced for difficult tasks such as intracorporeal suturing [14]. Of the simulation models described in urology, many are proprietary, not commercially available, or associated with significant cost [5,7]. Other physical analogue models for minimally invasive pyeloplasty are much more costly. For example, the BLAST pyeloplasty model costs US $55/model [15], whereas the SimuLab pediatric pyeloplasty model costs US $650 [8]. Although 3D printing is emerging as a way to develop very specific models with more realistic materials, this technology involves expensive materials, specialized expertise, and time-consuming production when performed at scale [8,16]. Currently, there exist very few low-cost pyeloplasty models. Ooi et al. [17] created a pyeloplasty model using chicken thighs and skin, and Ramachandran et al. [18] developed a model using the chicken esophagus and croup to simulate the ureter and renal pelvis, respectively. The chicken croup model was subsequently validated by Jiang
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
Low-cost pyeloplasty model for simulation training et al. [19] and found to have good construct validity. However, because these are animal-based models, significant preparation of the specimen must be performed, limiting large-scale use and limiting repetitive practice. Furthermore, the simulation area and robotic instruments must be thoroughly cleaned after each use, limiting availability especially in the operating room. Other ex vivo models using porcine and fish bladders have been proposed; however, they have similar limitations [20,21]. Rod et al. [22] have developed a low-cost balloon model for open dismembered pyeloplasty. However, assessment of the anastomosis was not performed, and many participants found that the model was not difficult enough as it did not replicate the restricted working space and anatomical position of the renal pelvis in children. There are several limitations to the study model. Most notably, the silicone used is stiffer and more resistant to tearing than human and animal tissue. Although reviewers commented on anatomical accuracy, realistic appearance, and appropriate renal pelvic thickness, they noted that tissue behavior was not the same as real tissue. For example, the participants indicated that ureteral spatulation did not produce a realistic splaying of cut edges. Four faculty participants indicated that these factors made performance of ureteral anastomosis more difficult than actual surgery. While increased technical difficulty may be acceptable in a training model, the participants also indicated thatsilicone was more difficult to tear than human tissue. This is problematic as trainees may apply improper forces when they transition to actual patients, causing tissue tearing. The authors further chose not to require review of the entire video when assigning a GEARS score. The authors instead chose to edit the videos and shorten each to 4 representative minutes in total. With a median time of nearly 30 min, it would be impractical for each of the 3 reviewers to watch each of the 26 videos in their entirety. Finally, the study model did not replicate the restricted workspace as would be seen in an actual pediatric pyeloplasty. Since this study, the authors now use a small box trainer to further limit space and range of motion. This study is also limited by a relatively small sample size (n Z 25) and a small number of faculty participants. Despite this limitation, the authors were able to demonstrate construct validity with their model. The cost calculations are limited to model material cost and do not include the considerable costs acquiring and/or maintaining the robotic training system and equipment. However, as this model does not contain any animal products, it could easily be brought into the operative theater if programs do not have a dedicated robotic system for training. Furthermore, this model could also be used for laparoscopic surgery without modification, expanding its use to programs without robotic systems.
Conclusions The authors developed a simple, low-cost pyeloplasty model that is highly acceptable to trainees and faculty and demonstrates preliminary content and construct validity. Subjective quality of repair, GEARS score, and reported
1.e7 self-confidence also were accurately discriminated using this model. Experienced robotic surgeons indicated that the model accurately reproduced pyeloplasty skills and can be used by trainees for repetitive practice. Future investigations are needed to improve tissue realism and examine the impact of repetitive simulator training on pyeloplasty performance.
Author statements Acknowledgments This study would not have been possible without the collaborative expertise of Bradley Mueller and Matthew Kozemond in the UT Southwestern surgical simulation laboratory.
Ethical approval This study did not meet the definition of human research that requires ethics review or approval at the authors’ institution, and a formal review by the internal review board was waived.
Funding The authors have no funding to disclose.
Competing interests The authors have no conflicts of interest to disclose.
References [1] Hutter MM, Kellogg KC, Ferguson CM, Abbott WM, Warshaw AL. The impact of the 80-hour resident workweek on surgical residents and attending surgeons. Ann Surg 2006;243(6): 864e71. https: //doi.org/10.1097/01.sla.0000220042.48310.66. [2] Akhtar KSN, Chen A, Standfield NJ, Gupte CM. The role of simulation in developing surgical skills. Curr Rev Musculoskel Med 2014;7(2):155e60. https://doi.org/10.1007/s12178-0149209-z. [3] Howells NR, Gill HS, Carr AJ, Price AJ, Rees JL. Transferring simulated arthroscopic skills to the operating theatre: a randomised blinded study. J Bone Joint Surg Br 2008;90(4): 494e9. https://doi.org/10.1302/0301-620X.90B4.20414. [4] Liu M, Curet M. A Review of Training Research and Virtual Reality Simulators for the da Vinci Surgical System. Teach Learn Med 2015;27(1):12e26. https://doi.org/10.1080/ 10401334.2014.979181. [5] Aydin A, Raison N, Khan MS, Dasgupta P, Ahmed K. Simulationbased training and assessment in urological surgery. Nat Rev Urol 2016;13(9):503e19. https://doi.org/10.1038/ nrurol.2016.147. [6] Wignall GR, Denstedt JD, Preminger GM, Cadeddu JA, Pearle M, Sweet R, et al. Surgical simulation: a urological perspective. J Urol 2008;179(5):1690e9. https: //doi.org/10.1016/j.juro.2008.01.014. [7] Moglia A, Ferrari V, Morelli L, Ferrari M, Mosca F, Cuschieri A. A systematic review of virtual reality simulators for robot-
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003
+
MODEL
1.e8
[8]
[9]
[10]
[11]
[12]
[13]
[14]
M.D. Timberlake et al. assisted surgery. Eur Urol 2016;69(6):1065e80. https: //doi.org/10.1016/j.eururo.2015.09.021. Cheung CL, Looi T, Lendvay TS, Drake JM, Farhat WA. Use of 3dimensional printing technology and silicone modeling in surgical simulation: development and face validation in pediatric laparoscopic pyeloplasty. J Surg Educ 2014;71(5): 762e7. https://doi.org/10.1016/j.jsurg.2014.03.001. Cook DA, Hatala R. Validation of educational assessments: a primer for simulation and beyond. Adv Simulat 2016;1(1):31. https://doi.org/10.1186/s41077-016-0033-y. Noureldin YA, Lee JY, McDougall EM, Sweet RM. Competencybased training and simulation: making a “valid” argument. J Endourol 2018;32(2):84e93. https://doi.org/10.1089/ end.2017.0650. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ. Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 2012; 187(1):247e52. https://doi.org/10.1016/j.juro.2011.09.032. Garcia J, Yang ZL, Mongrain R, Leask RL, Lachapelle K. 3D printing materials and their use in medical education: a review of current technology and trends for the future. BMJ Simulat Technol Enhanc Learn 2018;4(1):27e40. https: //doi.org/10.1136/bmjstel-2017-000234. Cook DA, Hamstra SJ, Brydges R, Zendejas B, Szostek J, Wang A, et al. Comparative effectiveness of instructional design features in simulation-based education: systematic review and meta-analysis. Med Teach 2013;35(1). https: //doi.org/10.3109/0142159X.2012.714886. Spruit EN, Band GPH, Hamming JF. Increasing efficiency of surgical training: effects of spacing practice on skill acquisition and retention in laparoscopy training. Surg Endosc 2015; 29(8):2235e43. https://doi.org/10.1007/s00464-014-3931-x.
[15] Poniatowski LH, Wolf JS, Nakada SY, Reihsen TE, Sainfort F, Sweet RM. Validity and acceptability of a high-fidelity physical simulation model for training of laparoscopic pyeloplasty. J Endourol 2014;28(4):393e8. https://doi.org/10.1089/end. 2013.0678. [16] Adams F, Qiu T, Mark A, Fritz B, Kramer L, Schlager D, et al. Soft 3D-printed phantom of the human kidney with collecting system. Ann Biomed Eng 2017;45(4):963e72. https: //doi.org/10.1007/s10439-016-1757-5. [17] Ooi J, Lawrentschuk N, Murphy DL. Training model for open or laparoscopic pyeloplasty, vol. 20; 2006. www.liebertpub.com. [18] Ramachandran A, Kurien A, Symons S, Ganpule A, Muthu V, Ch M, et al. A novel training model for laparoscopic pyeloplasty using chicken crop. J Endour ol 2008;22(4). https://doi.org/10.1089/end.2007.0380. [19] Jiang C, Liu M, Chen J, Wang P, Lin T, Xu K, et al. Construct validity of the chicken crop model in the simulation of laparoscopic pyeloplasty. J Endourol 2013. https: //doi.org/10.1089/end.2013.0085. [20] Yang B, Zhang Z, Xiao L, Wang L, Xu C, Sun Y. A novel training model for retroperitoneal laparoscopic dismembered pyeloplasty. J Endourol 2010;24(8):1345e9. https://doi.org/10. 1089/end.2009.0408. [21] Teber D, Guven S, Yaycioglu O, Ugurlu O, Sanli O, Gozen A, et al. Single-knot running suture anastomosis (one-knot pyeloplasty) for laparoscopic dismembered pyeloplasty: training model on a porcine bladder and clinical results. Int Urol Nephrol 2010;42(3):609e14. https://doi.org/10.1007/s11255009-9668-0. [22] Rod J, Marret JB, Kohaut J, Aigrain Y, Jais J, de Vries P, et al. Low-cost training simulator for open dismembered pyeloplasty: development and face validation. J Surg Educ 2018; 75(1):188e94. https://doi.org/10.1016/j.jsurg.2017.06.010.
Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi.org/10.1016/j.jpurol.2020.02.003.
Please cite this article as: Timberlake MD et al., Design and validation of a low-cost, high-fidelity model for robotic pyeloplasty simulation training, Journal of Pediatric Urology, https://doi.org/10.1016/j.jpurol.2020.02.003