- Email: [email protected]
Box- or Virtual-Reality Trainer: Which Tool Results in Better Transfer of Laparoscopic Basic Skills?—A Prospective Randomized Trial
ORIGINAL REPORTS
Box- or Virtual-Reality Trainer: Which Tool Results in Better Transfer of Laparoscopic Basic Skills?—A Prospective Randomized Trial Christian Brinkmann, MD,1 Mathias Fritz, MD,1 Ulrich Pankratius, MD, Ralf Bahde, MD, Philipp Neumann, MD, Steffen Schlueter, MD, Norbert Senninger, MD and Emile Rijcken, MD Department of General and Visceral Surgery, University Hospital Muenster, Muenster, Germany OBJECTIVE: Simulation training improves laparoscopic
performance. Laparoscopic basic skills can be learned in simulators as box- or virtual-reality (VR) trainers. However, there is no clear recommendation for either box or VR trainers as the most appropriate tool for the transfer of acquired laparoscopic basic skills into a surgical procedure. DESIGN: Both training tools were compared, using vali-
dated and well-established curricula in the acquirement of basic skills, in a prospective randomized trial in a 5-day structured laparoscopic training course. Participants completed either a box- or VR-trainer curriculum and then applied the learned skills performing an ex situ laparoscopic cholecystectomy on a pig liver. The performance was recorded on video and evaluated offline by 4 blinded observers using the Global Operative Assessment of Laparoscopic Skills (GOALS) score. Learning curves of the various exercises included in the training course were compared and the improvement in each exercise was analyzed. SETTING: Surgical Skills Lab of the Department of General and Visceral Surgery, University Hospital Muenster. PARTICIPANTS: Surgical novices without prior surgical
experience (medical students, n ¼ 36).
RESULTS: Posttraining evaluation showed significant improvement compared with baseline in both groups, indicating acquisition of laparoscopic basic skills. Learning curves showed almost the same progression with no significant differences. In simulated laparoscopic cholecystectomy, total
Correspondence: Inquiries to Emile Rijcken, MD, Department of General and Visceral Surgery, University Hospital Muenster, Albert-Schweitzer-Campus 1, W1, 48149 Muenster, Germany; fax: (251) 83-5-24-00; e-mail: [email protected], [email protected] 1
Both first authors contributed equally to this work
GOALS score was significantly higher for the box-trained group than the VR-trained group (box: 15.31 ⫾ 3.61 vs. VR: 12.92 ⫾ 3.06; p ¼ 0.039; Hedge's g* ¼ 0.699), indicating higher technical skill levels. CONCLUSIONS: Despite both systems having advantages and disadvantages, they can both be used for simulation training for laparoscopic skills. In the setting with 2 structured, validated and almost identical curricula, the box-trained group appears to be superior in the better transfer of basic skills into an experimental but structured C 2016 Association surgical procedure. ( J Surg Ed ]:]]]-]]]. J of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: education, simulation training, computer
simulation, GOALS, laparoscopy, prospective studies COMPETENCIES: Practice-based Learning and Improve-
ment, Systems-based Practice
INTRODUCTION It is well known that beginners in laparoscopic surgery can be trained by using simulators,1,2 and thereby are better prepared when doing their first laparoscopic procedures.3-5 Most complications in the operating room (OR) are caused by human factors.6 The experience of a surgeon and its complication rate are correlated negatively.7,8 Hence, laparoscopic training should be mandatory to warrant patient safety. Especially in the beginning of learning laparoscopic techniques, every surgical novice has to deal with their characteristics and difficulties like, e.g., 2-dimensional view, eye-hand coordination, fulcrum effect, reduced tactile feedback, and instrument handling.9,10 Time-consuming training process is necessary for increment of learning
Journal of Surgical Education & 2016 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2016.12.009
1
TABLE 1. Demographic Details of Study Population
Age Semester Sex (men, n (%)/women, n (%)) Dominant hand (right, n (%)/left, n (%)) Prior surgical experience (5-point Likert scale) Prior laparoscopic experience (5-point Likert scale) Prior experience with camera navigation (5-point Likert scale) Experience in video games (5-point Likert scale)
Box-Trainer Group (N ¼ 18)
VR-Trainer Group (N ¼ 18)
Median (Range) n (%) M ⫾ SD*
Median (Range) n (%) M ⫾ SD*
23 7 7 16 3.61 3.06 3.33
(21-29) (3-11) (39)/11 (61) (89)/2 (11) ⫾ 0.85 ⫾ 0.8 ⫾ 1.09
2.44 ⫾ 1.15
23.5 7 11 18 3.33 2.78 2.5
Mann-Whitney U/Chi-Square Test p Value†
(22-40) (4-9) (61)/7 (39) (100)/0 (0) ⫾ 1.14 ⫾ 1.17 ⫾ 1.65
2.72 ⫾ 1.41
0.291 0.301 0.182 0.146 0.656 0.613 0.139 0.559
*Values for age and semester shown as median with range in parentheses; sex and dominant hand shown as quantity with percentage value in parentheses; experience was obtained by self-assessment (1 ¼ no experience, 5 ¼ much experience) and is shown as average ⫾ standard deviation. † p Values: Mann-Whitney U test for age, semester, and experience; chi-square for sex and dominant hand.
curves.7,8,11-14 Ideally, a step-by-step training is performed, starting with the acquisition of basic laparoscopic skills in an abstract model, which are then transferred in a simulated surgical procedure accompanied by a comprehensive description of the procedure using printed or e-learning media, before the trainees apply their new skills in the OR for a real surgical procedure under supervision of an experienced surgeon.15,16 There are several different training methods available to learn laparoscopic basic skills. Box and VR trainers as well as training on live animals or cadavers are the most common.9,17 Advanced simulators include the possibility to learn complex surgical procedures. Basic skills include handling of laparoscopic instruments and to deal with the difficulties of laparoscopic techniques, such as grasping, cutting, clipping, or suturing. A structured curriculum separated into different training sessions2,18 seems to be superior to voluntary training in achieving optimal training effects.19 Although simulation-based laparoscopy training itself is approved as a valid training method by now2 after plenty of research has been done, there are still various questions on the details of simulation training.20 Expert panels see a need for determining which simulation methods are effective in specific skill acquisition. These specific questions can only be answered in well-designed randomized studies which in part repeat and confirm prior research work but highlight special questions which might be missed before. For example, there is still no clear recommendation for either box or VR trainers in the acquisition of basic skills. Whereas some studies assess both types of trainers as equally effective for the acquisition of laparoscopic skills,21-26 there are others favoring box2,27 or VR trainer.28-30 This inconsistency might be because of the lack of studies comparing both methods in validated and similar curricula. In some studies, focus was set on basic skills tasks
only, whereas others concentrated on the transfer of learned skills into complete surgical procedures in the OR. Moreover, acquired and successfully transferred basic skills should be assessed by using validated tools. This demand might be met best by the Global Operative Assessment of Laparoscopic Skills (GOALS) score,31 which was developed specifically for the evaluation of technical skills required in laparoscopic surgery. The aim of this prospectively randomized study was to find out which training method leads to better acquisition of basic skills as the first step of simulation training, before application of these skills in a simulated surgical procedure follows. For this purpose, 2 structured, validated and similar curricula on either box or VR trainers were compared for the transfer of laparoscopic basic skills into a simulated laparoscopic cholecystectomy using a validated assessment tool.
2
Journal of Surgical Education Volume ]/Number ] ] 2016
METHODS Participants A total of 36 medical students being surgical novices participated in this study. All students gave their informed consent for the analysis of the data obtained during the course. Every student answered a questionnaire on demographic data, prior surgical experience, and experience in video games. There were no statistically significant differences between the 2 groups in demographic factors (age, semester, sex, and dominant hand), previous clinical experience, and video gaming (Table 1). Course Design and Curriculum The participants completed a 5-day structured laparoscopy training curriculum (Fig. 1). On day 1, students were
Introduction into laparoscopy and the course curriculum, demonstration of the tasks and of the laparoscopic cholecystectomy (N = 36), randomisation into two groups
day 3
day 4
day 5
Box-Trainer-Curriculum (n=18)
day 2
2 cycles, containing 9 tasks each
Baseline evaluation (BL)
2 cycles, containing 9 tasks each
2 cycles, containing 9 tasks each
Coaching
2 cycles, containing 9 tasks each
2 cycles, containing 9 tasks each
Coaching
2 cycles, containing 9 tasks each
2 cycles, containing 9 tasks each
Post training evaluation (PT)
2 cycles, containing 9 tasks each
VR-Trainer-Curriculum (n=18)
day 1
laparoscopic cholecystectomy on the pig model (N = 36)
FIGURE 1. A 5-day structured curriculum.
introduced into laparoscopic techniques and equipment handling. Course's exercises and the different steps for performing a laparoscopic cholecystectomy were explained and demonstrated. Therefore, each single exercise of the box- and the VR-trainer curriculum was done by an experienced surgeon in front of the participants and a video of a laparoscopic cholecystectomy was shown while an experienced surgeon described and commented every step. All participants were pairwise randomized into 2 groups (group 1: box; group 2: VR). Two students completed the whole curriculum as a team, including the laparoscopic cholecystectomy. Once allocated to a session, no changes between groups and especially their training tools (box or VR) or partners were allowed. Furthermore, neither participant had the possibility to practice between their sessions. On day 2, participants performed each task twice without further instruction for baseline (BL) evaluation. At the end of that day, every participant completed a questionnaire for self-assessment.
Day 3 and 4 were training sessions with a total of 4 course cycles performed by each participant. Fully trained surgical tutors observed the students and gave intensive feedback and individual coaching. Tutors rotated to ensure equal coaching of all participants. On day 5, every participant performed another 2 cycles of the exercises without further instructions as a posttraining (PL) evaluation and completed the questionnaire for selfassessment for a second time. After that, every participant performed an ex situ laparoscopic cholecystectomy on a pig liver. Recorded videos of the procedure were evaluated offline by blinded observers using the GOALS score and some additional items.
Journal of Surgical Education Volume ]/Number ] ] 2016
3
Training Tools and Tasks Two different training methods were used, a box trainer build and developed in our department and the LAP Mentor II, constructed by 3D Systems, Simbionix
1: camera navigation
2: grasping
3: peg transfer
4: positioning/placement
5: pattern cutting
6: placement of ligating loop
7: extracorporeal knot
8: intracorporeal knot
9: clipping
FIGURE 2. Tasks of the box-trainer curriculum. The 9 tasks of the box-trainer curriculum are shown. Numeration matches the order in the curriculum. Pictures 3, 4, 6, and 9 are showing the view through the laparoscopic camera. In picture 7, the laparoscopic camera view is in the background, whereas the extracorporeal knot tying is happening in the foreground. Pictures 1, 2, 5, and 8 represent the direct view onto the training field.
Products. For each training tool, a curriculum consisting of 9 tasks was used, both previously validated. The box-trainer curriculum was oriented on the Fundamentals of Laparoscopic Surgery (FLS) program of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES).32 Minor modifications were made and the program was scientifically validated and established locally at the University Hospital Muenster by Bonrath et al.1 The 9 tasks of our box-trainer curriculum were camera navigation, grasping, peg transfer, positioning/placement, pattern cutting, placement of ligating loop, extracorporeal knot, intracorporeal knot, and clipping (Fig. 2). The VR-trainer curriculum was based on the validated Imperial College curriculum of Aggarwal et al.33 for laparoscopic cholecystectomy. For our particular purpose, we had to exclude procedural tasks for cholecystectomy. This was necessary as Okrainec et al.34 showed that this type of tasks would have led to an advantage for the VR-trained group. Furthermore, the task selection was matched to the tasks of the box trainer to reach maximum concordance and to ensure that the same basic skills were taught. The boxtrainer task extracorporeal knot tying was not available for the VR trainer. The 9 tasks of our VR-trainer curriculum 4
were camera manipulation 301, eye-hand coordination, 2handed maneuvers, peg transfer, translocation of objects, pattern cutting, placement of ligating loop, knot tying, clipping, and grasping (Fig. 3). Comparison of Learning Curves A direct comparison of the task time was not possible because of differences between both training tools. Therefore, the percentage improvement in the 4 most concordant tasks was compared (transfer, pattern cutting, placement of ligating loop, and intracorporeal knot). In addition, the percentage improvement of each task for the whole training period was analyzed to demonstrate correlation between learning progress and training method. Self-assessment Every participant completed the questionnaire concerning the self-assessment 2 times—on day 2 and day 5. The questionnaire consisted of the following 6 different items: “success of learning over the whole week,” “orientation,” “control of instruments,” “utilization of non-dominant Journal of Surgical Education Volume ]/Number ] ] 2016
1: camera manipulation 30°
2: eye-hand coordination
3: two handed maneuvers
4: peg transfer
5: translocation of objects
6: pattern cutting
7: placement of ligating loop
8: knot tying
9: clipping and grasping
FIGURE 3. Tasks of the VR-trainer curriculum. The 9 tasks of the VR-trainer curriculum are shown. Numeration matches the order in the curriculum. Pictures provided by LAP Mentor, 3D Systems, Simbionix Products.
hand,” “change of the instrument's extent of movement,” and “confidence in own skills—ability to perform a simple laparoscopic operation.”
Performance Measure and Scoring
On day 5 of our curriculum, every participant performed an ex situ laparoscopic cholecystectomy on an explanted pig liver in the box trainer. Therefore, in each of the 36 operations, one participant did the operation and the other one was the camera assistant, and, thereafter, the participants switched roles. The animal model consisted of pigs' livers with adherent gallbladders. Livers were placed into an aluminum tray in a fashion that gallbladders were laying on top and the cystic duct and cystic artery were located medially to the gallbladder. All performances were recorded using digital cameras. Every participant got an individual identifier, allowing blinded scoring and correct assignment of the recorded videos to each participant.
For the box trainer, each task was assessed quantitatively (time in seconds), with an additional error score, which was later converted into penalty seconds added to the total time of each task. Every participant was advised of the error score on the first day of the curriculum. Errors were classified as minor errors such as deviation from marked targets, penalized with 1 second per millimeter, or as more substantial errors such as loose knots, incorrect placed loop tie/clip or leakage from clipped tubing, penalized with 10 seconds. Every task was limited to 300 seconds. In the VR trainer, performance was measured by the computer system itself. The time in seconds was registered in all tasks. Each task was limited by the computer system, except for the knot tying task which was manually finished after 300 seconds or when the knot was tied and the task “translocation of objects,” which had no limit. For the evaluation of laparoscopic cholecystectomies, the GOALS score was used,31 consisting of 5 items (depth perception, bimanual dexterity, efficiency, tissue handling, and autonomy). Each item was rated on a 5-point Likert
Journal of Surgical Education Volume ]/Number ] ] 2016
5
Laparoscopic Cholecystectomy
scale with anchors at points 1, 3, and 5 with “1” being the lowest level of performance and “5” being the best performance. The total score was calculated as the sum of these 5 items. In addition, 5 further items were created: “quality of camera operation,” “team play,” “difficulty of the animal model,” “complete removal of gallbladder,” and “injury to the gallbladder.” The first 3 items had the same structure as the original GOALS score. The item “complete removal of the gallbladder” gave extra 5 points when this goal was achieved within 30 minutes as well as the participant got 5 additional points when he did no injury to the gallbladder, whereas only one minor lesion was rated with 2.5 points. In total, 36 videos were scored by 4 surgeons of the University Hospital Muenster who all had several years of experience in laparoscopic surgery. Every juror was instructed in the GOALS score and the extra items. Every observer scored 18 videos and was randomly assigned to them so that every performance was scored by 2 observers.
In addition, 4 parameters (“orientation,” “control of instruments,” “utilization of non-dominant hand,” and “change of the instrument's extent of movement”) were summed up to a score. The correlation coefficient between the score (subjective data) and the total task time on either training device in percentage (objective data) was calculated. To compare the results of the laparoscopic cholecystectomy, t-tests were used. The correlation between the “difficulty of the animal model” and the GOALS score was calculated. Hedge's g* (effect size) was used for the GOALS score and the extra items to calculate the effect size. For interpretation, the suggestions of Cohen was used (0.20.5 ¼ smaller effect; 0.5-0.8 ¼ medium effect; and 40.8 ¼ large effect).35 Cronbach's α was calculated as a measurement for the internal consistency of the GOALS score and the intraclass correlation coefficients (ICC) to determine the interrater reliability.31
Statistical Analysis
RESULTS
Demographic data were analyzed using chi-square test for categorial and Mann-Whitney U test for continuous variables. The effects of training were analyzed with the Wilcoxon test comparing BL and PT data. Mann-Whitney U test was used to compare percentaged box- and VR-trainer task times as well as a curve diagram displaying the mean time of all 9 tasks. For the evaluation of the self-assessment, the MannWhitney U test was used. Changes between BL and PT evaluation were analyzed by using Wilcoxon test for the items “success of learning over the whole week,” “orientation,” “control of instruments,” “utilization of nondominant hand,” and “change of the instrument's extent of movement.” For the last item, whether participants are feeling themselves comfortable to perform an uncomplicated laparoscopic cholecystectomy or appendectomy, we used chi-square test.
All 36 students completed the laparoscopy courses successfully. Data of all participants were available for evaluation. Box Trainer and VR Trainer: BL vs. PT Values As shown in Tables 2 and 3, participants in both groups showed improved performance every time (p r 0.001). In the box-trained group, participants improved approximately 30% to 60% in tasks 2 to 9. The smallest difference with approximately 12.5% was found in task 1 “camera navigation”. In the VR-trained group, improvement was similar with shortening of needed time in all tasks approximately 33% to 53%. Comparison of Learning Curves Comparison of the 4 most congruent tasks showed that there was a significant improvement in both groups. There
TABLE 2. Comparison of BL and PT Values for the Box-Trainer Group (n ¼ 18) BL (Day 2)
PT (Day 5)
*
Tasks
M ⫾ SD
Camera navigation (s) Grasping (s) Transfer (s) Positioning/placement (s) Pattern cutting (s) Placement of ligating loop (s) Extracorporeal knot (s) Intracorporeal knot (s) Clipping (s)
279 133 265 163 274 86 185 285 75
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
*
M ⫾ SD
53 53 48 56 51 45 53 22 37
244 54 127 80 187 46 93 179 44
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
56 20 55 32 52 13 22 42 17
Wilcoxon Test BL vs. PT p Value† 0.007 o0.001 o0.001 o0.001 o0.001 0.001 o0.001 o0.001 0.003
*Values shown as mean (M) ⫾ standard deviation (SD) (rounded integers). † p o 0.05 are in bold. 6
Journal of Surgical Education Volume ]/Number ] ] 2016
TABLE 3. Comparison of BL and PT Values for the VR-Trainer Group (n ¼ 18)
Tasks Camera manipulation 301 (s) Eye-hand coordination (s) 2-Handed maneuvers (s) Peg transfer (s) Translocation of objects (s) Pattern cutting (s) Placement of ligating loop (s) Knot tying (s) Clipping and grasping (s)
BL (Day 2)
PT (Day 5)
Wilcoxon Test BL vs. PT
M ⫾ SD*
M ⫾ SD*
p Value†
109 46 121 159 366 256 104 290 106
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
27 7 34 33 126 34 35 21 19
65 31 72 88 186 147 49 166 71
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
o0.001 o0.001 o0.001 o0.001 o0.001 o0.001 o0.001 o0.001 o0.001
12 10 12 17 61 46 17 55 14
*Values shown as mean (M) ⫾ standard deviation (SD) (rounded integers) † p o 0.05 are in bold.
was no significant difference between the box- and the VRtrainer group (Table 4). In addition, the learning curve, displaying the task time of all 9 tasks together in percentage, showed no significant difference in the curve progression (Fig. 4; all days p 4 0.05).
perform an uncomplicated laparoscopic cholecystectomy or appendectomy (BL: box/VR ¼ 0/1; PT: box/VR ¼ 5/7). Correlation coefficient between objective (task time in percentage) and subjective (self-assessment score) data was r ¼ 0.640 (p o 0.001) (Fig. 5), indicating that selfassessment can reflect objective measurement.
Self-assessment Self-assessment of “orientation,” “control of instruments,” and “utilization of non-dominant hand,” showed improvement after PT with a p o 0.05. There was no significant difference between the box- and the VR-trained group. Except in the “utilization of non-dominant hand” at BL evaluation, where the box-trained group ranked himself higher (box: 5.5 ⫾ 2.18; VR: 6.83 ⫾ 1.18) with a p ¼ 0.036. But that difference disappeared in the PT evaluation (box: 3.56 ⫾ 1.54; VR: 3.11 ⫾ 1.18) with a p ¼ 0.843. The item “success of learning over the whole week” was rated similar in both groups, as well as the “change of the instrument's extent of movement”. There was a significant difference for the movement values of BL and PT evaluation in both groups, indicating smaller movements in the PT evaluation. There was no significant difference in the question whether the participants thought to be able to
Laparoscopic Cholecystectomy In comparison between both groups, the box-trained group achieved a significantly higher total GOALS score as well as significantly higher scores in the items “depth perception” and “efficiency”. The effect size was significantly higher in the box-trained group for 4 of 5 items with Hedge's g* 4 0.5, which can be ranked as a medium effect (“bimanual dexterity,” “efficiency,” and “autonomy” and with Hedge's g* 4 0.8 as a large effect for “depth perception” (Table 5). In the additional items, no significant differences were found (Table 5). Correlation coefficient between the “difficulty of the animal model” and the total GOALS score showed no statistical relationship (r ¼ 0.067, p ¼ 0.698). Cronbach's α, as a measure for internal consistency of the GOALS score, was α ¼ 0.855, indicating good internal
TABLE 4. Comparison of the Task Time Improvement in the 4 Most Concordant Tasks in Percentage
Task on Box Task on VR Trainer Trainer Peg transfer Peg transfer Pattern cutting Pattern cutting Placement of Placement of ligating loop ligating loop Intracorporeal Knot tying knot
Box: Improvement in Task Time From BL to PT In Percentage*
VR: Improvement in Task Time From BL to PT in Percentage*
M ⫾ SD
M ⫾ SD
MannWhitney U Test p Value
52 ⫾ 21.3 31.6 ⫾ 17.3 46.3 ⫾ 54
44.4 ⫾ 14.9 42.6 ⫾ 16.9 53.1 ⫾ 32.5
0.311 0.088 0.174
37.2 ⫾ 11.9
42.6 ⫾ 16.4
0.174
*The average of each task at the BL evaluation was set as 100%. Based on that the value for PT was calculated as the percentage of the BL value. The difference between BL and PT is displayed in the diagram. Journal of Surgical Education Volume ]/Number ] ] 2016
7
100
FIGURE 5. Correlation of subjective and objective data. Objective data (mean task time in percentage, smaller values ¼ shorter task time) and subjective data (lower ¼ better) are shown for BL and PT. Selfassessment score is the sum of 4 parameters of the self-evaluation (“orientation,” “control of instruments,” “utilization of non-dominant hand,” and “change of the instrument's extent of movement”). Regression line for subjective data and objective data is shown.
laparoscopic cholecystectomy using a validated assessment tool. The results showed that first both groups improved their laparoscopic basic skills. Learning curves and comparison of the 4 most concordant tasks showed similar improvement in both groups, indicating that both curricula are able to convey laparoscopic basic skills. Second, the boxtrained group scored higher in the total GOALS score as well as in some of the items of the GOALS score, suggesting that the transfer of learned skills to the procedural model succeeded better after training on a box trainer. We, thereby, give an answer to one of the topics determined by Johnston et al.20 as future research directives for simulation issues still not being investigated sufficiently. There are several publications comparing box with VR training with a wide diversity in methodical approaches and results. Several studies approached the comparison between box and VR trainers by assessing the transfer of learned skills between both trainers (box to VR or vice versa). The results are not consistent, sometimes showing better results for the box trainer in the possibility to transfer learned skills to a VR trainer27 or in contrast showing that a box-trained group had difficulties transferring learned skills to a VR trainer.28 In conformity with our results, a meta-analysis by Zendejas et al.2 demonstrated that box trainers had moderately greater outcomes than VR trainers, although different parameters (learner satisfaction and task time) were measured. On the contrary, another meta-analysis by Nagendran et al.29 displayed an improvement in operation time and performance in the VR-trained group compared with the box-trained group. However, the amount of effect was not shown because only 2 studies were used for the meta-analysis. Furthermore, the study population in one study consisted of surgery residents38 and not medical students, being complete novices as in our study, making this not completely comparable to the present trial. Another study demonstrated that a VR-trained group performed better than a box-trained group in a live surgical task in a porcine model.30 However, the box-trainer curriculum used there consisted of only 3 tasks and, therefore, was not as comprehensive as our curriculum. Besides, a not widely spread mirror-based tower trainer was used and not a characteristic box trainer. Most review studies on the topic had not found any differences between box and VR trainers.21-23 This is in agreement with our results according to learning curves, but not when skill transfer is measured. This highlights the fact that equal outcome measures should be applied to make studies comparable. Two studies applied structured curricula and an evaluation of the learned skills in a laparoscopic cholecystectomy as in the present study once using VR and once using a pulsating organ perfusion trainer with a porcine liver.24,26 The cholecystectomy on the porcine liver was scored by blinded observers. Both studies demonstrated no significant difference in the box- and the VR-trained groups regarding their performance in the laparoscopic cholecystectomy.24,26
8
Journal of Surgical Education Volume ]/Number ] ] 2016
90 task time in percent
80 70 60 50
Box
40
VR
30 20 10 0 2 (BL)
3
5 (PT)
4 course day
FIGURE 4. Comparison of learning curves of box and VR trainer. The mean of each task at the BL (day 1 of training) evaluation was set as 100%. Based on that, the value for the other days was calculated as the percentage of the BL value. The mean of all 9 tasks in each group is displayed in the diagram with standard deviation.
consistency.36 The ICCs, as a measure for the interrater reliability, were greater than 0.75 for 5 of the 6 pairs of observers, which is rated as a good concordance of the observers.37
DISCUSSION The aim of this prospectively randomized study was to find out which training method leads to better acquisition of basic skills as the first step of simulation training, before application of these skills in a simulated surgical procedure follows. For this purpose, 2 structured, validated and similar curricula on either box or VR trainers were compared for the transfer of laparoscopic basic skills into a simulated 40 50 60
task time in percent
70 80 BL 90 PT 100 regression line r = .640
110 120 130 140 40
30
20
10
0
self-assessment score
TABLE 5. Results of the Laparoscopic Cholecystectomy
GOALS (Items 1-5 and Score)* Depth perception Bimanual dexterity Efficiency Tissue handling Autonomy GOALS score (sum of items 1-5) Quality of camera operation Team play Injury to the gallbladder Complete removal of gallbladder? Difficulty of the animal model
Group Box (n ¼ 18)
Group VR (n ¼ 18)
t-Test Box vs. VR
Effect Size Hedge's g*
M ⫾ SD†
M ⫾ SD†
p Value‡
g*‡
0.016 0.103 0.037 0.634 0.112 0.039 0.695 0.914 0.384 0.285 0.230
0.823 0.547 0.707 0.157 0.531 0.699 0.129 0.036 0.291 0.35 0.398
3.31 3.22 3.06 2.58 3.14 15.31 2.97 3.25 1.81 1.11 2.53
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
0.84 0.91 0.94 0.79 0,85 3.61 1.02 0.91 2.24 2.14 0.9
2.67 2.75 2.42 2.44 2.64 12.92 2.83 3.22 2.5 1.94 2.17
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
0.66 0.77 0.83 0.94 0.98 3.06 1.08 0.57 2.43 2.51 0.87
*GOALS, Global Operative Assessment of Laparoscopic Skills. † Values shown as mean (M) ⫾ standard deviation (SD); “1” was indicating the lowest level of performance and “5” indicating the highest level of performance. ‡ p o 0.05 and Hedge's g* 4 0.5 are in bold.
Some important differences in the study designs have to be highlighted: whereas Vitish-Sharma et al.24 had 3 hours training per group and the PT evaluation was done in the VR trainer, Nickel et al.26 surveyed the effects of e-learning too. Therefore, the box-trained group had less practical training time than the group trained on the VR trainer in their study. It has to be noted that the VR-trained group had procedural tasks in their curriculum and, therefore, was possibly favored through the task selection.34 Though the number of participants was relatively small, in our study, the box-trained group was superior to the VRtrained group in the assessment of a simulated operation evaluated by the GOALS score. Former training environment may play a role in the box-trained group. So, we have to ask whether we would have got different results doing the final assessment on the VR trainer, too. Vitish-Sharma et al.24 illustrated in their study that box- and VR-trained groups had equal results in their final assessment on the VR trainer. In our study, we wanted to show the adoption on real tissue. For ethical considerations, neither evaluation in an anesthetized live pig model nor on real patients in the OR was possible when medical students participated. Therefore, we decided on the pig liver model. Both simulators offer a different training situation. On the VR trainer, the camera is navigated by a joystick-like stick, completely different from a real laparoscopic camera as used on the box trainer and in the OR. In addition, the camera is navigated by the computer itself in 8 of 9 basic skill tasks, so that most time participants trained for themselves. VR trainers simulate haptic feedback using electric motors, which, still today, is not as accurate as real haptic feedback, whereas the box trainer enables to use the same instruments as in the OR getting real haptic feedback. It has been shown that the implementation of haptic feedback into VR trainers has positive influence on the
effectiveness of training although it is different from real haptics.39-41 Surprisingly, in our assessment, subscores for tissue handling were not statistically different between the groups. In our study, there were also no differences between the 2 simulators in the additional items concerning camera navigation and team play, which we added to the GOALS score. This was not expected as these aspects are not commonly trained in VR trainers as pointed out earlier. These results are in conformity with the current literature reporting that laparoscopic camera navigation is learnable on both types of simulators.42,43 As we already pointed out, there are several structural differences between both simulators. It is possible that the box-trained group just performed better because they did the assessment in the box trainer, too, and, therefore, had advantages. However, there is always a gap between the situation on a simulator and a simulated operation or even an actual operation. This gap is smaller when using the box trainer because it is closer to the situation in a real operation —using real instruments, getting a real haptic feedback, and seeing the tasks through a laparoscopic camera giving the same field of view as in an actual operation. In summary, it might be easier to transfer learned skills when they are acquired in a similar situation. In our study, we used only one certain type of VR trainer and one certain type of box trainer. In general, there are numerous different training models from different suppliers available and many researchers use self-customized materials. The generalization and comparison of results is difficult due to this variety. Otherwise, it has been shown before that different types of simulators correlate with each other.44 In addition, our tasks on both trainers were not exactly the same though very similar. Hence, it is possible that one group had advantages because of their training program, but both curricula were
Journal of Surgical Education Volume ]/Number ] ] 2016
9
validated and have shown that it is possible to gain laparoscopic skills.1,33 We only did PT evaluation by cholecystectomy and, therefore, the differences between both groups might possibly have been present before training. To eliminate that we had absolute novices in laparoscopic techniques as participants and we randomly divided them into 2 groups. They showed no significant differences in demographic factors, previous surgical experience, or experience in video gaming, which can have influence on laparoscopic basic skills45 and, therefore, might shorten the learning curve.46 In contrast to residents, medical students are all on the same beginner level and not biased by performing various surgical operations before testing. Residents have different amounts of clinical practice despite being in the same year of training, and they are influenced by viewing experienced surgeons in daily practice. Therefore, to judge only acquisition of basic skills, medical students might be purer than residents. We had a low ICC in 1 of 6 observer teams for the laparoscopic cholecystectomy, whereas in the other 5 there was good correlation. A possible reason for that is that the observers probably focused on different aspects of the participants' skills and weighted it differently. Because in 5 of 6 teams we had a rather high ICC, we believe that our results are reliable. The GOALS score was developed as a measurement tool for technical skills and not for evaluation of specific procedural skills used in laparoscopic cholecystectomy.31 It, therefore, differs from surgical outcome parameters such as operation time, complication rates, or hospital stay. Essentially, the GOALS score measures the items depth perception, bimanual dexterity, and tissue handling, so that a conclusion about the acquirement of basic skills as trained in our study is possible. GOALS was applied on evaluation of technical skills in a porcine gallbladder model in other experimental studies as well.47,48 Furthermore, Vassiliou et al.31 concluded that GOALS score is superior to task checklists or visual analog scales for the assessment of laparoscopic skills. When comparing both trainers, it is inevitable to distinguish some of the major differences between both simulators. One of the main advantages of the VR trainer is its intuitive and sometimes easy way of starting and doing a training session, whereas with the box trainer it takes much more preparation. Every task has to be arranged which takes some time and, too, a tutor to take care of the setup and the coaching. However, in contrast, the box trainer is a more complete training tool regarding, e.g., the instruments' handling. Despite technical innovations, there is still a lack in haptic feedback of the VR trainer: especially grasping is impaired, because of the complete absence of haptic feedback for the tip of the instrument during grasping. Moreover, VR trainer tasks sometimes just seem too easy, as indicated by general shorter times for VR exercises despite of high accordance for each task. However, though not 10
shown in this study, the strengths of VR training appears to be in the procedural training of more complex laparoscopic operations, which are not adequately depicted in dry or wet models in box-trainer environments.
CONCLUSIONS Despite both box trainers and VR trainers have advantages and disadvantages, they can both successfully be used for acquisition of basic laparoscopic skills. In our specific experimental setting with 2 structured, validated and almost identical curricula, the box-trained group appears to be superior in the transfer of basic skills into a structured but simulated surgical procedure as the next step of laparoscopic training.
DISCLOSURES Emile Rijcken works as a consultant for 3D Systems, Simbionix Products. Mathias Fritz, Christian Brinkmann, Ralf Bahde, Ulrich Pankratius, Philipp Neumann, Steffen Schlueter, and Norbert Senninger have no conflicts of interest or financial ties to disclose. This research did not receive any specific grant from funding agencies in the public, commercial, or not-forprofit sectors.
REFERENCES 1. Bonrath EM, Weber BK, Fritz M, et al. Laparoscopic
simulation training: testing for skill acquisition and retention. Surgery. 2012;152(1):12-20. 2. Zendejas B, Brydges R, Hamstra SJ, Cook DA. State
of the evidence on simulation-based training for laparoscopic surgery: a systematic review. Ann Surg. 2013;257(4):586-593. 3. Aggarwal R, Mytton OT, Derbrew M, et al. Training
and simulation for patient safety. Qual Saf Health Care. 2010;19(suppl 2):i34-i43. 4. Ahlberg G, Enochsson L, Gallagher AG, et al.
Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg. 2007;193 (6):797-804. 5. Sickle Van, Kent R, Ritter EM, Smith CD. The
pretrained novice: using simulation-based training to improve learning in the operating room. Surg Innov. 2006;13(3):198-204. 6. Fabri PJ, Zayas-Castro JL. Human error, not commu-
nication and systems, underlies surgical complications. Surgery. 2008;144(4):557-565. Journal of Surgical Education Volume ]/Number ] ] 2016
7. Barrie J, Jayne DG, Wright J, Murray CJ, Collinson
FJ, Pavitt SH. Attaining surgical competency and its implications in surgical clinical trial design: a systematic review of the learning curve in laparoscopic and robot-assisted laparoscopic colorectal cancer surgery. Ann Surg Oncol. 2014;21(3):829-840. 8. Miskovic D, Ni M, Wyles SM, Tekkis PP, Hanna GB.
Learning curve and case selection in laparoscopic colorectal surgery: systematic review and international multicenter analysis of 4852 cases. Dis Colon Rectum. 2012;55(12):1300-1310. 9. Roberts KE, Bell RL, Duffy AJ. Evolution of surgical
skills training. World J Gastroenterol. 2006;12(20): 3219-3224. 10. Tsuda S, Scott DJ, Doyle J, Jones DB. Surgical skills
training and simulation. Curr Probl Surg. 2009;46(4): 271-370. 11. Talebpour M, Alijani A, Hanna GB, Moosa Z, Tang
B, Cuschieri A. Proficiency-gain curve for an advanced laparoscopic procedure defined by observation clinical human reliability assessment (OCHRA). Surg Endosc. 2009;23(4):869-875. 12. Kluger MD, Vigano L, Barroso R, Cherqui D. The
learning curve in laparoscopic major liver resection. J Hepatobiliary Pancreat Sci. 2013;20(2):131-136. 13. Zevin B, Aggarwal R, Grantcharov TP. Simulation-
based training and learning curves in laparoscopic Roux-en-Y gastric bypass. Br J Surg. 2012;99 (7):887-895. 14. Rogers DA, Elstein AS, Bordage G. Improving con-
tinuing medical education for surgical techniques: applying the lessons learned in the first decade of minimal access surgery. Ann Surg. 2001;233 (2):159-166. 15. Palter VN, Grantcharov TP. Development and vali-
dation of a comprehensive curriculum to teach an advanced minimally invasive procedure: a randomized controlled trial. Ann Surg. 2012;256(1):25-32. 16. Palter VN, Orzech N, Reznick RK, Grantcharov TP.
Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg. 2013;257(2):224-230. 17. Bonrath EM, Buckl L, Brüwer M, Senninger N,
Rijcken E. Education in laparoscopic surgery: national survey on current strategies and relevance of simulation training. Zentralbl Chir. 2012;137(2):160-164.
a randomized controlled trial. Surg Endosc. 2002;16 (6):957-961. 19. van Dongen KW, van der Wal WA, Rinkes IHMB,
Schijven MP, Broeders IA. Virtual reality training for endoscopic surgery: voluntary or obligatory? Surg Endosc. 2008;22(3):664-667. 20. Johnston MJ, Paige JT, Aggarwal R, et al. An overview of
research priorities in surgical simulation: what the literature shows has been achieved during the 21st century and what remains. Am J Surg. 2016;211(1):214225. 21. Fagotti A, Petrillo M, Rossitto C, Scambia G. Stand-
ardized training programmes for advanced laparoscopic gynaecological surgery. Curr Opin Obstet Gynecol. 2013;25(4):327-331. 22. Ikonen TS, Antikainen T, Silvennoinen M, Isojärvi J,
Mäkinen E, Scheinin TM. Virtual reality simulator training of laparoscopic cholecystectomies – a systematic review. Scand J Surg. 2012;101(1):5-12. 23. Willaert W, van de Putte D, van Renterghem K, van
Nieuwenhove Y, Ceelen W, Pattyn P. Training models in laparoscopy: a systematic review comparing their effectiveness in learning surgical skills. Acta Chir Belg. 2013;113(2):77-95. 24. Vitish-Sharma P, Knowles J, Patel B. Acquisition of
fundamental laparoscopic skills: is a box really as good as a virtual reality trainer? Int J Surg. 2011;9 (8):659-661. 25. Kimura T, Kawabe A, Suzuki K, Wada H. Usefulness
of a virtual reality simulator or training box for endoscopic surgery training. Surg Endosc.. 2006;20 (4):656-659. 26. Nickel F, Brzoska JA, Gondan M, et al. Virtual reality
training versus blended learning of laparoscopic cholecystectomy: a randomized controlled trial with laparoscopic novices. Medicine (Baltimore). 2015;94(20): e764. 27. Mulla M, Sharma D, Moghul M, et al. Learning basic
laparoscopic skills: a randomized controlled study comparing box trainer, virtual reality simulator, and mental training. J Surg Educ. 2012;69(2):190-195. 28. Debes AJ, Aggarwal R, Balasundaram I, Jacobsen MB.
A tale of two trainers: virtual reality versus a video trainer for acquisition of basic laparoscopic skills. Am J Surg. 2010;199(6):840-845. 29. Nagendran M, Gurusamy KS, Aggarwal R, Loizidou
Practice distribution in procedural skills training:
M, Davidson BR. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. 2013;8:CD006575.
Journal of Surgical Education Volume ]/Number ] ] 2016
11
18. Mackay S, Morgan P, Datta V, Chang A, Darzi A.
30. Youngblood PL, Srivastava S, Curet M, Heinrichs
40. Strom P, Hedman L, Sarna L, Kjellin A, Wredmark T,
WL, Dev P, Wren SM. Comparison of training on two laparoscopic simulators and assessment of skills transfer to surgical performance. J Am Coll Surg. 2005;200(4):546-551.
Felländer-Tsai L. Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surg Endosc. 2006;20(9):1383-1388.
31. Vassiliou MC, Feldman LS, Andrew CG, et al. A
41. Kim HK, Rattner DW, Srinivasan MA. Virtual-reality-
global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005;190(1):107-113.
based laparoscopic surgical training: the role of simulation fidelity in haptic feedback. Comput Aided Surg. 2004;9(5):227-234.
32. Fried GM, Feldman LS, Vassiliou MC, et al. Proving
the value of simulation in laparoscopic surgery. Ann Surg. 2004;240(3):518-528. 33. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P,
Darzi A. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg. 2009;96(9):1086-1093. 34. Okrainec A, Tekian A, Aarts M-A, Grantcharov TP,
Escallon J, Reznick RK. Virtual Reality Training on Basic Laparoscopic Tasks vs. Virtual Reality Training of an Entire Surgical Procedure. A Randomized Controlled Trial Using Real World Operations as an Outcome. Salt Lake City, Utah: Annual Meeting of the Association of Surgical Educators (ASE); 2009. 35. Sedlmeier P, Renkewitz F. Forschungsmethoden und
Statistik in der Psychologie. Studium; 2008.
München: Pearson
42. Diesen DL, Erhunmwunsee L, Bennett KM, et al.
Effectiveness of laparoscopic computer simulator versus usage of box trainer for endoscopic surgery training of novices. J Surg Educ. 2011;68(4):282-289. 43. Stefanidis D, Haluck R, Pham T, et al. Construct and
face validity and task workload for laparoscopic camera navigation: virtual reality versus videotrainer systems at the SAGES Learning Center. Surg Endosc. 2007;21 (7):1158-1164. 44. Newmark J, Dandolu V, Milner R, Grewal H,
Harbison S, Hernandez E. Correlating virtual reality and box trainer tasks in the assessment of laparoscopic surgical skills. Am J Obstet Gynecol. 2007;197 (5):546.e1–e4. 45. Jalink MB, Goris J, Heineman E, Pierie J-PE, Ten
36. Bortz J, Döring N. Forschungsmethoden und Evalua-
Cate Hoedemaker HO. The effects of video games on laparoscopic simulator skills. Am J Surg. 2014;208(1): 151-156.
tion für Human- und Sozialwissenschaftler. 4th ed. Berlin, Heidelberg, New York: Springer; 2006.
46. Shane MD, Pettitt BJ, Morgenthal CB, Smith CD.
37. Wirtz M, Caspar F. Beurteilerübereinstimmung und
Beurteilerreliabilität. Methoden zur Bestimmung und Verbesserung der Zuverlässigkeit von Einschätzungen mittels Kategoriensystemen und Ratingskalen. Göttingen, Bern, Toronto, Seattle: Hogrefe; 2002. 38. Hamilton EC, Scott DJ, Fleming JB, et al. Compar-
ison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc. 2002;16(3):406-411. 39. Yiasemidou M, Glassman D, Badiani S, Vasas P, Patel
B. Faster simulated laparoscopic cholecystectomy with haptic feedback technology. OAS. 2011;2011:39-44.
12
Should surgical novices trade their retractors for joysticks? Videogame experience decreases the time needed to acquire surgical skills Surg Endosc. 2008;22 (5):1294-1297. 47. Hogle NJ, Widmann WD, Ude AO, Hardy MA,
Fowler DL. Does training novices to criteria and does rapid acquisition of skills on laparoscopic simulators have predictive validity or are we just playing video games? J Surg Educ. 2008;65(6):431-435. 48. Vassiliou MC, Feldman LS, Fraser SA, et al. Evaluat-
ing intraoperative laparoscopic skill: direct observation versus blinded videotaped performances. Surg Innov. 2007;14(3):211-216.
Journal of Surgical Education Volume ]/Number ] ] 2016
Box- or Virtual-Reality Trainer: Which Tool Results in Better Transfer of Laparoscopic Basic Skills?—A Prospective Randomized Trial Christian Brinkmann, MD,1 Mathias Fritz, MD,1 Ulrich Pankratius, MD, Ralf Bahde, MD, Philipp Neumann, MD, Steffen Schlueter, MD, Norbert Senninger, MD and Emile Rijcken, MD Department of General and Visceral Surgery, University Hospital Muenster, Muenster, Germany OBJECTIVE: Simulation training improves laparoscopic
performance. Laparoscopic basic skills can be learned in simulators as box- or virtual-reality (VR) trainers. However, there is no clear recommendation for either box or VR trainers as the most appropriate tool for the transfer of acquired laparoscopic basic skills into a surgical procedure. DESIGN: Both training tools were compared, using vali-
dated and well-established curricula in the acquirement of basic skills, in a prospective randomized trial in a 5-day structured laparoscopic training course. Participants completed either a box- or VR-trainer curriculum and then applied the learned skills performing an ex situ laparoscopic cholecystectomy on a pig liver. The performance was recorded on video and evaluated offline by 4 blinded observers using the Global Operative Assessment of Laparoscopic Skills (GOALS) score. Learning curves of the various exercises included in the training course were compared and the improvement in each exercise was analyzed. SETTING: Surgical Skills Lab of the Department of General and Visceral Surgery, University Hospital Muenster. PARTICIPANTS: Surgical novices without prior surgical
experience (medical students, n ¼ 36).
RESULTS: Posttraining evaluation showed significant improvement compared with baseline in both groups, indicating acquisition of laparoscopic basic skills. Learning curves showed almost the same progression with no significant differences. In simulated laparoscopic cholecystectomy, total
Correspondence: Inquiries to Emile Rijcken, MD, Department of General and Visceral Surgery, University Hospital Muenster, Albert-Schweitzer-Campus 1, W1, 48149 Muenster, Germany; fax: (251) 83-5-24-00; e-mail: [email protected], [email protected] 1
Both first authors contributed equally to this work
GOALS score was significantly higher for the box-trained group than the VR-trained group (box: 15.31 ⫾ 3.61 vs. VR: 12.92 ⫾ 3.06; p ¼ 0.039; Hedge's g* ¼ 0.699), indicating higher technical skill levels. CONCLUSIONS: Despite both systems having advantages and disadvantages, they can both be used for simulation training for laparoscopic skills. In the setting with 2 structured, validated and almost identical curricula, the box-trained group appears to be superior in the better transfer of basic skills into an experimental but structured C 2016 Association surgical procedure. ( J Surg Ed ]:]]]-]]]. J of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.) KEY WORDS: education, simulation training, computer
simulation, GOALS, laparoscopy, prospective studies COMPETENCIES: Practice-based Learning and Improve-
ment, Systems-based Practice
INTRODUCTION It is well known that beginners in laparoscopic surgery can be trained by using simulators,1,2 and thereby are better prepared when doing their first laparoscopic procedures.3-5 Most complications in the operating room (OR) are caused by human factors.6 The experience of a surgeon and its complication rate are correlated negatively.7,8 Hence, laparoscopic training should be mandatory to warrant patient safety. Especially in the beginning of learning laparoscopic techniques, every surgical novice has to deal with their characteristics and difficulties like, e.g., 2-dimensional view, eye-hand coordination, fulcrum effect, reduced tactile feedback, and instrument handling.9,10 Time-consuming training process is necessary for increment of learning
Journal of Surgical Education & 2016 Association of Program Directors in Surgery. Published by 1931-7204/$30.00 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jsurg.2016.12.009
1
TABLE 1. Demographic Details of Study Population
Age Semester Sex (men, n (%)/women, n (%)) Dominant hand (right, n (%)/left, n (%)) Prior surgical experience (5-point Likert scale) Prior laparoscopic experience (5-point Likert scale) Prior experience with camera navigation (5-point Likert scale) Experience in video games (5-point Likert scale)
Box-Trainer Group (N ¼ 18)
VR-Trainer Group (N ¼ 18)
Median (Range) n (%) M ⫾ SD*
Median (Range) n (%) M ⫾ SD*
23 7 7 16 3.61 3.06 3.33
(21-29) (3-11) (39)/11 (61) (89)/2 (11) ⫾ 0.85 ⫾ 0.8 ⫾ 1.09
2.44 ⫾ 1.15
23.5 7 11 18 3.33 2.78 2.5
Mann-Whitney U/Chi-Square Test p Value†
(22-40) (4-9) (61)/7 (39) (100)/0 (0) ⫾ 1.14 ⫾ 1.17 ⫾ 1.65
2.72 ⫾ 1.41
0.291 0.301 0.182 0.146 0.656 0.613 0.139 0.559
*Values for age and semester shown as median with range in parentheses; sex and dominant hand shown as quantity with percentage value in parentheses; experience was obtained by self-assessment (1 ¼ no experience, 5 ¼ much experience) and is shown as average ⫾ standard deviation. † p Values: Mann-Whitney U test for age, semester, and experience; chi-square for sex and dominant hand.
curves.7,8,11-14 Ideally, a step-by-step training is performed, starting with the acquisition of basic laparoscopic skills in an abstract model, which are then transferred in a simulated surgical procedure accompanied by a comprehensive description of the procedure using printed or e-learning media, before the trainees apply their new skills in the OR for a real surgical procedure under supervision of an experienced surgeon.15,16 There are several different training methods available to learn laparoscopic basic skills. Box and VR trainers as well as training on live animals or cadavers are the most common.9,17 Advanced simulators include the possibility to learn complex surgical procedures. Basic skills include handling of laparoscopic instruments and to deal with the difficulties of laparoscopic techniques, such as grasping, cutting, clipping, or suturing. A structured curriculum separated into different training sessions2,18 seems to be superior to voluntary training in achieving optimal training effects.19 Although simulation-based laparoscopy training itself is approved as a valid training method by now2 after plenty of research has been done, there are still various questions on the details of simulation training.20 Expert panels see a need for determining which simulation methods are effective in specific skill acquisition. These specific questions can only be answered in well-designed randomized studies which in part repeat and confirm prior research work but highlight special questions which might be missed before. For example, there is still no clear recommendation for either box or VR trainers in the acquisition of basic skills. Whereas some studies assess both types of trainers as equally effective for the acquisition of laparoscopic skills,21-26 there are others favoring box2,27 or VR trainer.28-30 This inconsistency might be because of the lack of studies comparing both methods in validated and similar curricula. In some studies, focus was set on basic skills tasks
only, whereas others concentrated on the transfer of learned skills into complete surgical procedures in the OR. Moreover, acquired and successfully transferred basic skills should be assessed by using validated tools. This demand might be met best by the Global Operative Assessment of Laparoscopic Skills (GOALS) score,31 which was developed specifically for the evaluation of technical skills required in laparoscopic surgery. The aim of this prospectively randomized study was to find out which training method leads to better acquisition of basic skills as the first step of simulation training, before application of these skills in a simulated surgical procedure follows. For this purpose, 2 structured, validated and similar curricula on either box or VR trainers were compared for the transfer of laparoscopic basic skills into a simulated laparoscopic cholecystectomy using a validated assessment tool.
2
Journal of Surgical Education Volume ]/Number ] ] 2016
METHODS Participants A total of 36 medical students being surgical novices participated in this study. All students gave their informed consent for the analysis of the data obtained during the course. Every student answered a questionnaire on demographic data, prior surgical experience, and experience in video games. There were no statistically significant differences between the 2 groups in demographic factors (age, semester, sex, and dominant hand), previous clinical experience, and video gaming (Table 1). Course Design and Curriculum The participants completed a 5-day structured laparoscopy training curriculum (Fig. 1). On day 1, students were
Introduction into laparoscopy and the course curriculum, demonstration of the tasks and of the laparoscopic cholecystectomy (N = 36), randomisation into two groups
day 3
day 4
day 5
Box-Trainer-Curriculum (n=18)
day 2
2 cycles, containing 9 tasks each
Baseline evaluation (BL)
2 cycles, containing 9 tasks each
2 cycles, containing 9 tasks each
Coaching
2 cycles, containing 9 tasks each
2 cycles, containing 9 tasks each
Coaching
2 cycles, containing 9 tasks each
2 cycles, containing 9 tasks each
Post training evaluation (PT)
2 cycles, containing 9 tasks each
VR-Trainer-Curriculum (n=18)
day 1
laparoscopic cholecystectomy on the pig model (N = 36)
FIGURE 1. A 5-day structured curriculum.
introduced into laparoscopic techniques and equipment handling. Course's exercises and the different steps for performing a laparoscopic cholecystectomy were explained and demonstrated. Therefore, each single exercise of the box- and the VR-trainer curriculum was done by an experienced surgeon in front of the participants and a video of a laparoscopic cholecystectomy was shown while an experienced surgeon described and commented every step. All participants were pairwise randomized into 2 groups (group 1: box; group 2: VR). Two students completed the whole curriculum as a team, including the laparoscopic cholecystectomy. Once allocated to a session, no changes between groups and especially their training tools (box or VR) or partners were allowed. Furthermore, neither participant had the possibility to practice between their sessions. On day 2, participants performed each task twice without further instruction for baseline (BL) evaluation. At the end of that day, every participant completed a questionnaire for self-assessment.
Day 3 and 4 were training sessions with a total of 4 course cycles performed by each participant. Fully trained surgical tutors observed the students and gave intensive feedback and individual coaching. Tutors rotated to ensure equal coaching of all participants. On day 5, every participant performed another 2 cycles of the exercises without further instructions as a posttraining (PL) evaluation and completed the questionnaire for selfassessment for a second time. After that, every participant performed an ex situ laparoscopic cholecystectomy on a pig liver. Recorded videos of the procedure were evaluated offline by blinded observers using the GOALS score and some additional items.
Journal of Surgical Education Volume ]/Number ] ] 2016
3
Training Tools and Tasks Two different training methods were used, a box trainer build and developed in our department and the LAP Mentor II, constructed by 3D Systems, Simbionix
1: camera navigation
2: grasping
3: peg transfer
4: positioning/placement
5: pattern cutting
6: placement of ligating loop
7: extracorporeal knot
8: intracorporeal knot
9: clipping
FIGURE 2. Tasks of the box-trainer curriculum. The 9 tasks of the box-trainer curriculum are shown. Numeration matches the order in the curriculum. Pictures 3, 4, 6, and 9 are showing the view through the laparoscopic camera. In picture 7, the laparoscopic camera view is in the background, whereas the extracorporeal knot tying is happening in the foreground. Pictures 1, 2, 5, and 8 represent the direct view onto the training field.
Products. For each training tool, a curriculum consisting of 9 tasks was used, both previously validated. The box-trainer curriculum was oriented on the Fundamentals of Laparoscopic Surgery (FLS) program of the Society of American Gastrointestinal and Endoscopic Surgeons (SAGES).32 Minor modifications were made and the program was scientifically validated and established locally at the University Hospital Muenster by Bonrath et al.1 The 9 tasks of our box-trainer curriculum were camera navigation, grasping, peg transfer, positioning/placement, pattern cutting, placement of ligating loop, extracorporeal knot, intracorporeal knot, and clipping (Fig. 2). The VR-trainer curriculum was based on the validated Imperial College curriculum of Aggarwal et al.33 for laparoscopic cholecystectomy. For our particular purpose, we had to exclude procedural tasks for cholecystectomy. This was necessary as Okrainec et al.34 showed that this type of tasks would have led to an advantage for the VR-trained group. Furthermore, the task selection was matched to the tasks of the box trainer to reach maximum concordance and to ensure that the same basic skills were taught. The boxtrainer task extracorporeal knot tying was not available for the VR trainer. The 9 tasks of our VR-trainer curriculum 4
were camera manipulation 301, eye-hand coordination, 2handed maneuvers, peg transfer, translocation of objects, pattern cutting, placement of ligating loop, knot tying, clipping, and grasping (Fig. 3). Comparison of Learning Curves A direct comparison of the task time was not possible because of differences between both training tools. Therefore, the percentage improvement in the 4 most concordant tasks was compared (transfer, pattern cutting, placement of ligating loop, and intracorporeal knot). In addition, the percentage improvement of each task for the whole training period was analyzed to demonstrate correlation between learning progress and training method. Self-assessment Every participant completed the questionnaire concerning the self-assessment 2 times—on day 2 and day 5. The questionnaire consisted of the following 6 different items: “success of learning over the whole week,” “orientation,” “control of instruments,” “utilization of non-dominant Journal of Surgical Education Volume ]/Number ] ] 2016
1: camera manipulation 30°
2: eye-hand coordination
3: two handed maneuvers
4: peg transfer
5: translocation of objects
6: pattern cutting
7: placement of ligating loop
8: knot tying
9: clipping and grasping
FIGURE 3. Tasks of the VR-trainer curriculum. The 9 tasks of the VR-trainer curriculum are shown. Numeration matches the order in the curriculum. Pictures provided by LAP Mentor, 3D Systems, Simbionix Products.
hand,” “change of the instrument's extent of movement,” and “confidence in own skills—ability to perform a simple laparoscopic operation.”
Performance Measure and Scoring
On day 5 of our curriculum, every participant performed an ex situ laparoscopic cholecystectomy on an explanted pig liver in the box trainer. Therefore, in each of the 36 operations, one participant did the operation and the other one was the camera assistant, and, thereafter, the participants switched roles. The animal model consisted of pigs' livers with adherent gallbladders. Livers were placed into an aluminum tray in a fashion that gallbladders were laying on top and the cystic duct and cystic artery were located medially to the gallbladder. All performances were recorded using digital cameras. Every participant got an individual identifier, allowing blinded scoring and correct assignment of the recorded videos to each participant.
For the box trainer, each task was assessed quantitatively (time in seconds), with an additional error score, which was later converted into penalty seconds added to the total time of each task. Every participant was advised of the error score on the first day of the curriculum. Errors were classified as minor errors such as deviation from marked targets, penalized with 1 second per millimeter, or as more substantial errors such as loose knots, incorrect placed loop tie/clip or leakage from clipped tubing, penalized with 10 seconds. Every task was limited to 300 seconds. In the VR trainer, performance was measured by the computer system itself. The time in seconds was registered in all tasks. Each task was limited by the computer system, except for the knot tying task which was manually finished after 300 seconds or when the knot was tied and the task “translocation of objects,” which had no limit. For the evaluation of laparoscopic cholecystectomies, the GOALS score was used,31 consisting of 5 items (depth perception, bimanual dexterity, efficiency, tissue handling, and autonomy). Each item was rated on a 5-point Likert
Journal of Surgical Education Volume ]/Number ] ] 2016
5
Laparoscopic Cholecystectomy
scale with anchors at points 1, 3, and 5 with “1” being the lowest level of performance and “5” being the best performance. The total score was calculated as the sum of these 5 items. In addition, 5 further items were created: “quality of camera operation,” “team play,” “difficulty of the animal model,” “complete removal of gallbladder,” and “injury to the gallbladder.” The first 3 items had the same structure as the original GOALS score. The item “complete removal of the gallbladder” gave extra 5 points when this goal was achieved within 30 minutes as well as the participant got 5 additional points when he did no injury to the gallbladder, whereas only one minor lesion was rated with 2.5 points. In total, 36 videos were scored by 4 surgeons of the University Hospital Muenster who all had several years of experience in laparoscopic surgery. Every juror was instructed in the GOALS score and the extra items. Every observer scored 18 videos and was randomly assigned to them so that every performance was scored by 2 observers.
In addition, 4 parameters (“orientation,” “control of instruments,” “utilization of non-dominant hand,” and “change of the instrument's extent of movement”) were summed up to a score. The correlation coefficient between the score (subjective data) and the total task time on either training device in percentage (objective data) was calculated. To compare the results of the laparoscopic cholecystectomy, t-tests were used. The correlation between the “difficulty of the animal model” and the GOALS score was calculated. Hedge's g* (effect size) was used for the GOALS score and the extra items to calculate the effect size. For interpretation, the suggestions of Cohen was used (0.20.5 ¼ smaller effect; 0.5-0.8 ¼ medium effect; and 40.8 ¼ large effect).35 Cronbach's α was calculated as a measurement for the internal consistency of the GOALS score and the intraclass correlation coefficients (ICC) to determine the interrater reliability.31
Statistical Analysis
RESULTS
Demographic data were analyzed using chi-square test for categorial and Mann-Whitney U test for continuous variables. The effects of training were analyzed with the Wilcoxon test comparing BL and PT data. Mann-Whitney U test was used to compare percentaged box- and VR-trainer task times as well as a curve diagram displaying the mean time of all 9 tasks. For the evaluation of the self-assessment, the MannWhitney U test was used. Changes between BL and PT evaluation were analyzed by using Wilcoxon test for the items “success of learning over the whole week,” “orientation,” “control of instruments,” “utilization of nondominant hand,” and “change of the instrument's extent of movement.” For the last item, whether participants are feeling themselves comfortable to perform an uncomplicated laparoscopic cholecystectomy or appendectomy, we used chi-square test.
All 36 students completed the laparoscopy courses successfully. Data of all participants were available for evaluation. Box Trainer and VR Trainer: BL vs. PT Values As shown in Tables 2 and 3, participants in both groups showed improved performance every time (p r 0.001). In the box-trained group, participants improved approximately 30% to 60% in tasks 2 to 9. The smallest difference with approximately 12.5% was found in task 1 “camera navigation”. In the VR-trained group, improvement was similar with shortening of needed time in all tasks approximately 33% to 53%. Comparison of Learning Curves Comparison of the 4 most congruent tasks showed that there was a significant improvement in both groups. There
TABLE 2. Comparison of BL and PT Values for the Box-Trainer Group (n ¼ 18) BL (Day 2)
PT (Day 5)
*
Tasks
M ⫾ SD
Camera navigation (s) Grasping (s) Transfer (s) Positioning/placement (s) Pattern cutting (s) Placement of ligating loop (s) Extracorporeal knot (s) Intracorporeal knot (s) Clipping (s)
279 133 265 163 274 86 185 285 75
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
*
M ⫾ SD
53 53 48 56 51 45 53 22 37
244 54 127 80 187 46 93 179 44
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
56 20 55 32 52 13 22 42 17
Wilcoxon Test BL vs. PT p Value† 0.007 o0.001 o0.001 o0.001 o0.001 0.001 o0.001 o0.001 0.003
*Values shown as mean (M) ⫾ standard deviation (SD) (rounded integers). † p o 0.05 are in bold. 6
Journal of Surgical Education Volume ]/Number ] ] 2016
TABLE 3. Comparison of BL and PT Values for the VR-Trainer Group (n ¼ 18)
Tasks Camera manipulation 301 (s) Eye-hand coordination (s) 2-Handed maneuvers (s) Peg transfer (s) Translocation of objects (s) Pattern cutting (s) Placement of ligating loop (s) Knot tying (s) Clipping and grasping (s)
BL (Day 2)
PT (Day 5)
Wilcoxon Test BL vs. PT
M ⫾ SD*
M ⫾ SD*
p Value†
109 46 121 159 366 256 104 290 106
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
27 7 34 33 126 34 35 21 19
65 31 72 88 186 147 49 166 71
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
o0.001 o0.001 o0.001 o0.001 o0.001 o0.001 o0.001 o0.001 o0.001
12 10 12 17 61 46 17 55 14
*Values shown as mean (M) ⫾ standard deviation (SD) (rounded integers) † p o 0.05 are in bold.
was no significant difference between the box- and the VRtrainer group (Table 4). In addition, the learning curve, displaying the task time of all 9 tasks together in percentage, showed no significant difference in the curve progression (Fig. 4; all days p 4 0.05).
perform an uncomplicated laparoscopic cholecystectomy or appendectomy (BL: box/VR ¼ 0/1; PT: box/VR ¼ 5/7). Correlation coefficient between objective (task time in percentage) and subjective (self-assessment score) data was r ¼ 0.640 (p o 0.001) (Fig. 5), indicating that selfassessment can reflect objective measurement.
Self-assessment Self-assessment of “orientation,” “control of instruments,” and “utilization of non-dominant hand,” showed improvement after PT with a p o 0.05. There was no significant difference between the box- and the VR-trained group. Except in the “utilization of non-dominant hand” at BL evaluation, where the box-trained group ranked himself higher (box: 5.5 ⫾ 2.18; VR: 6.83 ⫾ 1.18) with a p ¼ 0.036. But that difference disappeared in the PT evaluation (box: 3.56 ⫾ 1.54; VR: 3.11 ⫾ 1.18) with a p ¼ 0.843. The item “success of learning over the whole week” was rated similar in both groups, as well as the “change of the instrument's extent of movement”. There was a significant difference for the movement values of BL and PT evaluation in both groups, indicating smaller movements in the PT evaluation. There was no significant difference in the question whether the participants thought to be able to
Laparoscopic Cholecystectomy In comparison between both groups, the box-trained group achieved a significantly higher total GOALS score as well as significantly higher scores in the items “depth perception” and “efficiency”. The effect size was significantly higher in the box-trained group for 4 of 5 items with Hedge's g* 4 0.5, which can be ranked as a medium effect (“bimanual dexterity,” “efficiency,” and “autonomy” and with Hedge's g* 4 0.8 as a large effect for “depth perception” (Table 5). In the additional items, no significant differences were found (Table 5). Correlation coefficient between the “difficulty of the animal model” and the total GOALS score showed no statistical relationship (r ¼ 0.067, p ¼ 0.698). Cronbach's α, as a measure for internal consistency of the GOALS score, was α ¼ 0.855, indicating good internal
TABLE 4. Comparison of the Task Time Improvement in the 4 Most Concordant Tasks in Percentage
Task on Box Task on VR Trainer Trainer Peg transfer Peg transfer Pattern cutting Pattern cutting Placement of Placement of ligating loop ligating loop Intracorporeal Knot tying knot
Box: Improvement in Task Time From BL to PT In Percentage*
VR: Improvement in Task Time From BL to PT in Percentage*
M ⫾ SD
M ⫾ SD
MannWhitney U Test p Value
52 ⫾ 21.3 31.6 ⫾ 17.3 46.3 ⫾ 54
44.4 ⫾ 14.9 42.6 ⫾ 16.9 53.1 ⫾ 32.5
0.311 0.088 0.174
37.2 ⫾ 11.9
42.6 ⫾ 16.4
0.174
*The average of each task at the BL evaluation was set as 100%. Based on that the value for PT was calculated as the percentage of the BL value. The difference between BL and PT is displayed in the diagram. Journal of Surgical Education Volume ]/Number ] ] 2016
7
100
FIGURE 5. Correlation of subjective and objective data. Objective data (mean task time in percentage, smaller values ¼ shorter task time) and subjective data (lower ¼ better) are shown for BL and PT. Selfassessment score is the sum of 4 parameters of the self-evaluation (“orientation,” “control of instruments,” “utilization of non-dominant hand,” and “change of the instrument's extent of movement”). Regression line for subjective data and objective data is shown.
laparoscopic cholecystectomy using a validated assessment tool. The results showed that first both groups improved their laparoscopic basic skills. Learning curves and comparison of the 4 most concordant tasks showed similar improvement in both groups, indicating that both curricula are able to convey laparoscopic basic skills. Second, the boxtrained group scored higher in the total GOALS score as well as in some of the items of the GOALS score, suggesting that the transfer of learned skills to the procedural model succeeded better after training on a box trainer. We, thereby, give an answer to one of the topics determined by Johnston et al.20 as future research directives for simulation issues still not being investigated sufficiently. There are several publications comparing box with VR training with a wide diversity in methodical approaches and results. Several studies approached the comparison between box and VR trainers by assessing the transfer of learned skills between both trainers (box to VR or vice versa). The results are not consistent, sometimes showing better results for the box trainer in the possibility to transfer learned skills to a VR trainer27 or in contrast showing that a box-trained group had difficulties transferring learned skills to a VR trainer.28 In conformity with our results, a meta-analysis by Zendejas et al.2 demonstrated that box trainers had moderately greater outcomes than VR trainers, although different parameters (learner satisfaction and task time) were measured. On the contrary, another meta-analysis by Nagendran et al.29 displayed an improvement in operation time and performance in the VR-trained group compared with the box-trained group. However, the amount of effect was not shown because only 2 studies were used for the meta-analysis. Furthermore, the study population in one study consisted of surgery residents38 and not medical students, being complete novices as in our study, making this not completely comparable to the present trial. Another study demonstrated that a VR-trained group performed better than a box-trained group in a live surgical task in a porcine model.30 However, the box-trainer curriculum used there consisted of only 3 tasks and, therefore, was not as comprehensive as our curriculum. Besides, a not widely spread mirror-based tower trainer was used and not a characteristic box trainer. Most review studies on the topic had not found any differences between box and VR trainers.21-23 This is in agreement with our results according to learning curves, but not when skill transfer is measured. This highlights the fact that equal outcome measures should be applied to make studies comparable. Two studies applied structured curricula and an evaluation of the learned skills in a laparoscopic cholecystectomy as in the present study once using VR and once using a pulsating organ perfusion trainer with a porcine liver.24,26 The cholecystectomy on the porcine liver was scored by blinded observers. Both studies demonstrated no significant difference in the box- and the VR-trained groups regarding their performance in the laparoscopic cholecystectomy.24,26
8
Journal of Surgical Education Volume ]/Number ] ] 2016
90 task time in percent
80 70 60 50
Box
40
VR
30 20 10 0 2 (BL)
3
5 (PT)
4 course day
FIGURE 4. Comparison of learning curves of box and VR trainer. The mean of each task at the BL (day 1 of training) evaluation was set as 100%. Based on that, the value for the other days was calculated as the percentage of the BL value. The mean of all 9 tasks in each group is displayed in the diagram with standard deviation.
consistency.36 The ICCs, as a measure for the interrater reliability, were greater than 0.75 for 5 of the 6 pairs of observers, which is rated as a good concordance of the observers.37
DISCUSSION The aim of this prospectively randomized study was to find out which training method leads to better acquisition of basic skills as the first step of simulation training, before application of these skills in a simulated surgical procedure follows. For this purpose, 2 structured, validated and similar curricula on either box or VR trainers were compared for the transfer of laparoscopic basic skills into a simulated 40 50 60
task time in percent
70 80 BL 90 PT 100 regression line r = .640
110 120 130 140 40
30
20
10
0
self-assessment score
TABLE 5. Results of the Laparoscopic Cholecystectomy
GOALS (Items 1-5 and Score)* Depth perception Bimanual dexterity Efficiency Tissue handling Autonomy GOALS score (sum of items 1-5) Quality of camera operation Team play Injury to the gallbladder Complete removal of gallbladder? Difficulty of the animal model
Group Box (n ¼ 18)
Group VR (n ¼ 18)
t-Test Box vs. VR
Effect Size Hedge's g*
M ⫾ SD†
M ⫾ SD†
p Value‡
g*‡
0.016 0.103 0.037 0.634 0.112 0.039 0.695 0.914 0.384 0.285 0.230
0.823 0.547 0.707 0.157 0.531 0.699 0.129 0.036 0.291 0.35 0.398
3.31 3.22 3.06 2.58 3.14 15.31 2.97 3.25 1.81 1.11 2.53
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
0.84 0.91 0.94 0.79 0,85 3.61 1.02 0.91 2.24 2.14 0.9
2.67 2.75 2.42 2.44 2.64 12.92 2.83 3.22 2.5 1.94 2.17
⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾ ⫾
0.66 0.77 0.83 0.94 0.98 3.06 1.08 0.57 2.43 2.51 0.87
*GOALS, Global Operative Assessment of Laparoscopic Skills. † Values shown as mean (M) ⫾ standard deviation (SD); “1” was indicating the lowest level of performance and “5” indicating the highest level of performance. ‡ p o 0.05 and Hedge's g* 4 0.5 are in bold.
Some important differences in the study designs have to be highlighted: whereas Vitish-Sharma et al.24 had 3 hours training per group and the PT evaluation was done in the VR trainer, Nickel et al.26 surveyed the effects of e-learning too. Therefore, the box-trained group had less practical training time than the group trained on the VR trainer in their study. It has to be noted that the VR-trained group had procedural tasks in their curriculum and, therefore, was possibly favored through the task selection.34 Though the number of participants was relatively small, in our study, the box-trained group was superior to the VRtrained group in the assessment of a simulated operation evaluated by the GOALS score. Former training environment may play a role in the box-trained group. So, we have to ask whether we would have got different results doing the final assessment on the VR trainer, too. Vitish-Sharma et al.24 illustrated in their study that box- and VR-trained groups had equal results in their final assessment on the VR trainer. In our study, we wanted to show the adoption on real tissue. For ethical considerations, neither evaluation in an anesthetized live pig model nor on real patients in the OR was possible when medical students participated. Therefore, we decided on the pig liver model. Both simulators offer a different training situation. On the VR trainer, the camera is navigated by a joystick-like stick, completely different from a real laparoscopic camera as used on the box trainer and in the OR. In addition, the camera is navigated by the computer itself in 8 of 9 basic skill tasks, so that most time participants trained for themselves. VR trainers simulate haptic feedback using electric motors, which, still today, is not as accurate as real haptic feedback, whereas the box trainer enables to use the same instruments as in the OR getting real haptic feedback. It has been shown that the implementation of haptic feedback into VR trainers has positive influence on the
effectiveness of training although it is different from real haptics.39-41 Surprisingly, in our assessment, subscores for tissue handling were not statistically different between the groups. In our study, there were also no differences between the 2 simulators in the additional items concerning camera navigation and team play, which we added to the GOALS score. This was not expected as these aspects are not commonly trained in VR trainers as pointed out earlier. These results are in conformity with the current literature reporting that laparoscopic camera navigation is learnable on both types of simulators.42,43 As we already pointed out, there are several structural differences between both simulators. It is possible that the box-trained group just performed better because they did the assessment in the box trainer, too, and, therefore, had advantages. However, there is always a gap between the situation on a simulator and a simulated operation or even an actual operation. This gap is smaller when using the box trainer because it is closer to the situation in a real operation —using real instruments, getting a real haptic feedback, and seeing the tasks through a laparoscopic camera giving the same field of view as in an actual operation. In summary, it might be easier to transfer learned skills when they are acquired in a similar situation. In our study, we used only one certain type of VR trainer and one certain type of box trainer. In general, there are numerous different training models from different suppliers available and many researchers use self-customized materials. The generalization and comparison of results is difficult due to this variety. Otherwise, it has been shown before that different types of simulators correlate with each other.44 In addition, our tasks on both trainers were not exactly the same though very similar. Hence, it is possible that one group had advantages because of their training program, but both curricula were
Journal of Surgical Education Volume ]/Number ] ] 2016
9
validated and have shown that it is possible to gain laparoscopic skills.1,33 We only did PT evaluation by cholecystectomy and, therefore, the differences between both groups might possibly have been present before training. To eliminate that we had absolute novices in laparoscopic techniques as participants and we randomly divided them into 2 groups. They showed no significant differences in demographic factors, previous surgical experience, or experience in video gaming, which can have influence on laparoscopic basic skills45 and, therefore, might shorten the learning curve.46 In contrast to residents, medical students are all on the same beginner level and not biased by performing various surgical operations before testing. Residents have different amounts of clinical practice despite being in the same year of training, and they are influenced by viewing experienced surgeons in daily practice. Therefore, to judge only acquisition of basic skills, medical students might be purer than residents. We had a low ICC in 1 of 6 observer teams for the laparoscopic cholecystectomy, whereas in the other 5 there was good correlation. A possible reason for that is that the observers probably focused on different aspects of the participants' skills and weighted it differently. Because in 5 of 6 teams we had a rather high ICC, we believe that our results are reliable. The GOALS score was developed as a measurement tool for technical skills and not for evaluation of specific procedural skills used in laparoscopic cholecystectomy.31 It, therefore, differs from surgical outcome parameters such as operation time, complication rates, or hospital stay. Essentially, the GOALS score measures the items depth perception, bimanual dexterity, and tissue handling, so that a conclusion about the acquirement of basic skills as trained in our study is possible. GOALS was applied on evaluation of technical skills in a porcine gallbladder model in other experimental studies as well.47,48 Furthermore, Vassiliou et al.31 concluded that GOALS score is superior to task checklists or visual analog scales for the assessment of laparoscopic skills. When comparing both trainers, it is inevitable to distinguish some of the major differences between both simulators. One of the main advantages of the VR trainer is its intuitive and sometimes easy way of starting and doing a training session, whereas with the box trainer it takes much more preparation. Every task has to be arranged which takes some time and, too, a tutor to take care of the setup and the coaching. However, in contrast, the box trainer is a more complete training tool regarding, e.g., the instruments' handling. Despite technical innovations, there is still a lack in haptic feedback of the VR trainer: especially grasping is impaired, because of the complete absence of haptic feedback for the tip of the instrument during grasping. Moreover, VR trainer tasks sometimes just seem too easy, as indicated by general shorter times for VR exercises despite of high accordance for each task. However, though not 10
shown in this study, the strengths of VR training appears to be in the procedural training of more complex laparoscopic operations, which are not adequately depicted in dry or wet models in box-trainer environments.
CONCLUSIONS Despite both box trainers and VR trainers have advantages and disadvantages, they can both successfully be used for acquisition of basic laparoscopic skills. In our specific experimental setting with 2 structured, validated and almost identical curricula, the box-trained group appears to be superior in the transfer of basic skills into a structured but simulated surgical procedure as the next step of laparoscopic training.
DISCLOSURES Emile Rijcken works as a consultant for 3D Systems, Simbionix Products. Mathias Fritz, Christian Brinkmann, Ralf Bahde, Ulrich Pankratius, Philipp Neumann, Steffen Schlueter, and Norbert Senninger have no conflicts of interest or financial ties to disclose. This research did not receive any specific grant from funding agencies in the public, commercial, or not-forprofit sectors.
REFERENCES 1. Bonrath EM, Weber BK, Fritz M, et al. Laparoscopic
simulation training: testing for skill acquisition and retention. Surgery. 2012;152(1):12-20. 2. Zendejas B, Brydges R, Hamstra SJ, Cook DA. State
of the evidence on simulation-based training for laparoscopic surgery: a systematic review. Ann Surg. 2013;257(4):586-593. 3. Aggarwal R, Mytton OT, Derbrew M, et al. Training
and simulation for patient safety. Qual Saf Health Care. 2010;19(suppl 2):i34-i43. 4. Ahlberg G, Enochsson L, Gallagher AG, et al.
Proficiency-based virtual reality training significantly reduces the error rate for residents during their first 10 laparoscopic cholecystectomies. Am J Surg. 2007;193 (6):797-804. 5. Sickle Van, Kent R, Ritter EM, Smith CD. The
pretrained novice: using simulation-based training to improve learning in the operating room. Surg Innov. 2006;13(3):198-204. 6. Fabri PJ, Zayas-Castro JL. Human error, not commu-
nication and systems, underlies surgical complications. Surgery. 2008;144(4):557-565. Journal of Surgical Education Volume ]/Number ] ] 2016
7. Barrie J, Jayne DG, Wright J, Murray CJ, Collinson
FJ, Pavitt SH. Attaining surgical competency and its implications in surgical clinical trial design: a systematic review of the learning curve in laparoscopic and robot-assisted laparoscopic colorectal cancer surgery. Ann Surg Oncol. 2014;21(3):829-840. 8. Miskovic D, Ni M, Wyles SM, Tekkis PP, Hanna GB.
Learning curve and case selection in laparoscopic colorectal surgery: systematic review and international multicenter analysis of 4852 cases. Dis Colon Rectum. 2012;55(12):1300-1310. 9. Roberts KE, Bell RL, Duffy AJ. Evolution of surgical
skills training. World J Gastroenterol. 2006;12(20): 3219-3224. 10. Tsuda S, Scott DJ, Doyle J, Jones DB. Surgical skills
training and simulation. Curr Probl Surg. 2009;46(4): 271-370. 11. Talebpour M, Alijani A, Hanna GB, Moosa Z, Tang
B, Cuschieri A. Proficiency-gain curve for an advanced laparoscopic procedure defined by observation clinical human reliability assessment (OCHRA). Surg Endosc. 2009;23(4):869-875. 12. Kluger MD, Vigano L, Barroso R, Cherqui D. The
learning curve in laparoscopic major liver resection. J Hepatobiliary Pancreat Sci. 2013;20(2):131-136. 13. Zevin B, Aggarwal R, Grantcharov TP. Simulation-
based training and learning curves in laparoscopic Roux-en-Y gastric bypass. Br J Surg. 2012;99 (7):887-895. 14. Rogers DA, Elstein AS, Bordage G. Improving con-
tinuing medical education for surgical techniques: applying the lessons learned in the first decade of minimal access surgery. Ann Surg. 2001;233 (2):159-166. 15. Palter VN, Grantcharov TP. Development and vali-
dation of a comprehensive curriculum to teach an advanced minimally invasive procedure: a randomized controlled trial. Ann Surg. 2012;256(1):25-32. 16. Palter VN, Orzech N, Reznick RK, Grantcharov TP.
Validation of a structured training and assessment curriculum for technical skill acquisition in minimally invasive surgery: a randomized controlled trial. Ann Surg. 2013;257(2):224-230. 17. Bonrath EM, Buckl L, Brüwer M, Senninger N,
Rijcken E. Education in laparoscopic surgery: national survey on current strategies and relevance of simulation training. Zentralbl Chir. 2012;137(2):160-164.
a randomized controlled trial. Surg Endosc. 2002;16 (6):957-961. 19. van Dongen KW, van der Wal WA, Rinkes IHMB,
Schijven MP, Broeders IA. Virtual reality training for endoscopic surgery: voluntary or obligatory? Surg Endosc. 2008;22(3):664-667. 20. Johnston MJ, Paige JT, Aggarwal R, et al. An overview of
research priorities in surgical simulation: what the literature shows has been achieved during the 21st century and what remains. Am J Surg. 2016;211(1):214225. 21. Fagotti A, Petrillo M, Rossitto C, Scambia G. Stand-
ardized training programmes for advanced laparoscopic gynaecological surgery. Curr Opin Obstet Gynecol. 2013;25(4):327-331. 22. Ikonen TS, Antikainen T, Silvennoinen M, Isojärvi J,
Mäkinen E, Scheinin TM. Virtual reality simulator training of laparoscopic cholecystectomies – a systematic review. Scand J Surg. 2012;101(1):5-12. 23. Willaert W, van de Putte D, van Renterghem K, van
Nieuwenhove Y, Ceelen W, Pattyn P. Training models in laparoscopy: a systematic review comparing their effectiveness in learning surgical skills. Acta Chir Belg. 2013;113(2):77-95. 24. Vitish-Sharma P, Knowles J, Patel B. Acquisition of
fundamental laparoscopic skills: is a box really as good as a virtual reality trainer? Int J Surg. 2011;9 (8):659-661. 25. Kimura T, Kawabe A, Suzuki K, Wada H. Usefulness
of a virtual reality simulator or training box for endoscopic surgery training. Surg Endosc.. 2006;20 (4):656-659. 26. Nickel F, Brzoska JA, Gondan M, et al. Virtual reality
training versus blended learning of laparoscopic cholecystectomy: a randomized controlled trial with laparoscopic novices. Medicine (Baltimore). 2015;94(20): e764. 27. Mulla M, Sharma D, Moghul M, et al. Learning basic
laparoscopic skills: a randomized controlled study comparing box trainer, virtual reality simulator, and mental training. J Surg Educ. 2012;69(2):190-195. 28. Debes AJ, Aggarwal R, Balasundaram I, Jacobsen MB.
A tale of two trainers: virtual reality versus a video trainer for acquisition of basic laparoscopic skills. Am J Surg. 2010;199(6):840-845. 29. Nagendran M, Gurusamy KS, Aggarwal R, Loizidou
Practice distribution in procedural skills training:
M, Davidson BR. Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst Rev. 2013;8:CD006575.
Journal of Surgical Education Volume ]/Number ] ] 2016
11
18. Mackay S, Morgan P, Datta V, Chang A, Darzi A.
30. Youngblood PL, Srivastava S, Curet M, Heinrichs
40. Strom P, Hedman L, Sarna L, Kjellin A, Wredmark T,
WL, Dev P, Wren SM. Comparison of training on two laparoscopic simulators and assessment of skills transfer to surgical performance. J Am Coll Surg. 2005;200(4):546-551.
Felländer-Tsai L. Early exposure to haptic feedback enhances performance in surgical simulator training: a prospective randomized crossover study in surgical residents. Surg Endosc. 2006;20(9):1383-1388.
31. Vassiliou MC, Feldman LS, Andrew CG, et al. A
41. Kim HK, Rattner DW, Srinivasan MA. Virtual-reality-
global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005;190(1):107-113.
based laparoscopic surgical training: the role of simulation fidelity in haptic feedback. Comput Aided Surg. 2004;9(5):227-234.
32. Fried GM, Feldman LS, Vassiliou MC, et al. Proving
the value of simulation in laparoscopic surgery. Ann Surg. 2004;240(3):518-528. 33. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P,
Darzi A. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg. 2009;96(9):1086-1093. 34. Okrainec A, Tekian A, Aarts M-A, Grantcharov TP,
Escallon J, Reznick RK. Virtual Reality Training on Basic Laparoscopic Tasks vs. Virtual Reality Training of an Entire Surgical Procedure. A Randomized Controlled Trial Using Real World Operations as an Outcome. Salt Lake City, Utah: Annual Meeting of the Association of Surgical Educators (ASE); 2009. 35. Sedlmeier P, Renkewitz F. Forschungsmethoden und
Statistik in der Psychologie. Studium; 2008.
München: Pearson
42. Diesen DL, Erhunmwunsee L, Bennett KM, et al.
Effectiveness of laparoscopic computer simulator versus usage of box trainer for endoscopic surgery training of novices. J Surg Educ. 2011;68(4):282-289. 43. Stefanidis D, Haluck R, Pham T, et al. Construct and
face validity and task workload for laparoscopic camera navigation: virtual reality versus videotrainer systems at the SAGES Learning Center. Surg Endosc. 2007;21 (7):1158-1164. 44. Newmark J, Dandolu V, Milner R, Grewal H,
Harbison S, Hernandez E. Correlating virtual reality and box trainer tasks in the assessment of laparoscopic surgical skills. Am J Obstet Gynecol. 2007;197 (5):546.e1–e4. 45. Jalink MB, Goris J, Heineman E, Pierie J-PE, Ten
36. Bortz J, Döring N. Forschungsmethoden und Evalua-
Cate Hoedemaker HO. The effects of video games on laparoscopic simulator skills. Am J Surg. 2014;208(1): 151-156.
tion für Human- und Sozialwissenschaftler. 4th ed. Berlin, Heidelberg, New York: Springer; 2006.
46. Shane MD, Pettitt BJ, Morgenthal CB, Smith CD.
37. Wirtz M, Caspar F. Beurteilerübereinstimmung und
Beurteilerreliabilität. Methoden zur Bestimmung und Verbesserung der Zuverlässigkeit von Einschätzungen mittels Kategoriensystemen und Ratingskalen. Göttingen, Bern, Toronto, Seattle: Hogrefe; 2002. 38. Hamilton EC, Scott DJ, Fleming JB, et al. Compar-
ison of video trainer and virtual reality training systems on acquisition of laparoscopic skills. Surg Endosc. 2002;16(3):406-411. 39. Yiasemidou M, Glassman D, Badiani S, Vasas P, Patel
B. Faster simulated laparoscopic cholecystectomy with haptic feedback technology. OAS. 2011;2011:39-44.
12
Should surgical novices trade their retractors for joysticks? Videogame experience decreases the time needed to acquire surgical skills Surg Endosc. 2008;22 (5):1294-1297. 47. Hogle NJ, Widmann WD, Ude AO, Hardy MA,
Fowler DL. Does training novices to criteria and does rapid acquisition of skills on laparoscopic simulators have predictive validity or are we just playing video games? J Surg Educ. 2008;65(6):431-435. 48. Vassiliou MC, Feldman LS, Fraser SA, et al. Evaluat-
ing intraoperative laparoscopic skill: direct observation versus blinded videotaped performances. Surg Innov. 2007;14(3):211-216.
Journal of Surgical Education Volume ]/Number ] ] 2016