- Email: [email protected]
Fundamental Arthroscopic Skill Differentiation With Virtual Reality Simulation
Fundamental Arthroscopic Skill Differentiation With Virtual Reality Simulation Kelsey Rose, B.A., and Robert Pedowitz, M.D., Ph.D.
Purpose: The purpose of this study was to investigate the use and validity of virtual reality modules as part of the educational approach to mastering arthroscopy in a safe environment by assessing the ability to distinguish between experience levels. Additionally, the study aimed to evaluate whether experts have greater ambidexterity than do novices. Methods: Three virtual reality modules (Swemac/Augmented Reality Systems, Linkoping, Sweden) were created to test fundamental arthroscopic skills. Thirty participantsd10 experts consisting of faculty, 10 intermediate participants consisting of orthopaedic residents, and 10 novices consisting of medical studentsdperformed each exercise. Steady and Telescope was designed to train centering and image stability. Steady and Probe was designed to train basic triangulation. Track and Moving Target was designed to train coordinated motions of arthroscope and probe. Metrics reflecting speed, accuracy, and efficiency of motion were used to measure construct validity. Results: Steady and Probe and Track a Moving Target both exhibited construct validity, with better performance by experts and intermediate participants than by novices (P < .05), whereas Steady and Telescope did not show validity. There was an overall trend toward better ambidexterity as a function of greater surgical experience, with experts consistently more proficient than novices throughout all 3 modules. Conclusions: This study represents a new way to assess basic arthroscopy skills using virtual reality modules developed through task deconstruction. Participants with the most arthroscopic experience performed better and were more consistent than novices on all 3 virtual reality modules. Greater arthroscopic experience correlates with more symmetry of ambidextrous performance. However, further adjustment of the modules may better simulate fundamental arthroscopic skills and discriminate between experience levels. Clinical Relevance: Arthroscopy training is a critical element of orthopaedic surgery resident training. Developing techniques to safely and effectively train these skills is critical for patient safety and resident education.
A
n important aspect of orthopaedic residency training is teaching residents the motor skills required for arthroscopic surgery. The development of virtual reality simulators to teach the fundamental skills of arthroscopy may facilitate teaching and proficiency assessment before performance of these skills on patients.1 Learning basic tasks and then progressing in procedural complexity is the most logical way to train
From the Department of Orthopaedic Surgery, David Geffen School of Medicine at UCLA, Santa Monica, California, U.S.A. Supported by the Arthroscopy Association of North American Education and by academic enrichment funds provided by the David Geffen School of Medicine at UCLA. The authors report the following potential conflict of interest or source of funding: R.P. receives support from Stryker and DJ Orthopaedics. Received March 4, 2014; accepted August 15, 2014. Address correspondence to Kelsey Rose, B.A., David Geffen School of Medicine at UCLA, 334 20th St, Santa Monica, CA 90402, U.S.A. E-mail: [email protected] Ó 2015 by the Arthroscopy Association of North America 0749-8063/14184/$36.00 http://dx.doi.org/10.1016/j.arthro.2014.08.016
technical skills, and this systematic approach may be enhanced by virtual reality simulation.2,3 The Fundamentals of Arthroscopic Surgery Training (FAST) program is a collaborative effort of the Arthroscopy Association of North America, the American Academy of Orthopaedic Surgery, and the American Board of Orthopaedic Surgery. The basic objectives of the FAST program are to break down arthroscopy into its most fundamental and basic skills (task deconstruction), to teach these skills to residents in an organized and progressive fashion, and to have residents practice using simulation until they have reached a determined level of proficiency. Karahan et al.4 showed that basic motor skills are correlated with arthroscopic competency, which supports the goals of the FAST program. Performance on virtual reality simulators correlates with surgical experience.5 Those with the most surgical experience show the most rapid learning curves on virtual reality modules and require the fewest number of repetitions to reach proficiency.6-8 This is a good indicator that the modules require motor skills similar to those actually used in surgery.6 Appropriate modules
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol 31, No 2 (February), 2015: pp 299-305
299
300
K. ROSE AND R. PEDOWITZ
not only can differentiate expertise levels (construct validity) but also are effective at improving arthroscopic skills in inexperienced surgeons.9 Andersen et al.9 showed that those who were allowed to train on the simulator between a pre- and post-test performed better than those who did not, suggesting that repetition facilitates skill development. The purpose of this study was to evaluate the use of 3 virtual reality simulator modules that were designed to teach several basic skills of arthroscopy as educational tools for arthroscopic training by comparing their ability to distinguish between groups with different levels of arthroscopy experience. In the general sense, this is part of an iterative developmental process that should improve our educational curriculum. Additionally, this study aimed to look at ambidexterity across experience levels. Intuitively, experts who have been using both hands for precise movements repeatedly for years should show better equality of performance between hands than do novice surgeons. However, to our knowledge, this has never been studied regarding arthroscopic surgery. It is hypothesized that those with the most arthroscopic experience will perform more consistently and more efficiently and show better ambidexterity on the virtual reality modules, proving that experts are more ambidextrous and that these modules are testing fundamental arthroscopic skills.
Methods Three virtual reality modules were created for the dual-haptic ArthroVision platform (Swemac/Augmented Reality Systems, Linkoping, Sweden). These tasks were defined after thorough deconstruction of arthroscopic surgical skills into their most basic forms. Participants visualized the exercises on the screen of a laptop computer connected to both haptic devices. The ergonomics mimicked the distance between the common knee arthroscopy portals (inferolateral and inferomedial), with triangulation depth selected to reflect the typical working distance from skin portal to medial meniscus. The computer software modeled the 30 arthroscope, which is the standard device for knee arthroscopy. Participants were divided into 3 groups based on experience levels: (1) experts, defined as master faculty at an Arthroscopy Association of North America Resident’s Arthroscopy Fundamental Skills Course (n ¼ 10); (2) intermediate participants, defined as orthopaedic residents in training (n ¼ 10) who were attending the same course; and (3) novices, defined as medical students without any previous training in arthroscopy (n ¼ 10). The protocol was approved by our institutional review board and all participants provided informed consent. Participants were asked to fill out a questionnaire to determine handedness and arthroscopic experience.
Each module was performed twice, and then the same module was repeated twice using the contralateral hand. Instructions were provided using the same standardized script for all participants, and up to 2 minutes of practice was allowed for each module. Randomization was used to determine which hand each participant would use first for each module. Each virtual reality module was designed to train participants on different basic arthroscopy skills. Each module involved spherical targets and a circular outline in the center of the screen to coach image centering. The first module, Steady and Telescope, trained image centering and image stability. Participants were instructed to use the arthroscope to center a highlighted yellow target and to hold it steady for 5 seconds. A contralateral probe was not used in this module. After successfully centering the target, it would turn gray and participants would find the next highlighted yellow target. This would be repeated until all targets were completed. The second module, Steady and Probe, trained basic triangulation, with the virtual arthroscope in one hand and the virtual probe in the contralateral hand. Participants were instructed to use the arthroscope to center a highlighted yellow sphere and then use the simulated probe to push smaller blue balls on the surface of the sphere into the target (Fig 1). When all the smaller balls were pushed in completely, the sphere would turn gray and a new target would appear. The third module, Track a Moving Target, trained coordination of movement of arthroscope and probe. Participants were instructed to keep the target centered
Fig 1. Steady and Probe module is shown with probe pushing in a blue ball into sphere in ArthroVision virtual reality platform.
USE OF VIRTUAL REALITY IN ARTHROSCOPY EDUCATION
with their arthroscope while pushing the target along a winding track with a probe using the contralateral hand. The target remained yellow when properly centered and would turn red to signify a deviation from center. The target could still move if not centered properly, but deviation from the center was measured. Various metrics were recorded by the computer for each of the virtual reality modules. For Steady and Telescope, the computer recorded time of target in focus, arthroscope path length, and time to completion of the module. For Steady and Probe, metrics included probe path length, arthroscope path length, and time to module completion. For Track a Moving Target, the computer measured out-of-focus time, distance deviation, center deviation, and module completion time. Distance deviation was defined as the arthroscope distance to the target minus the ideal arthroscope distance to the target (i.e. net deviation from perfect arthroscope distance). This can also be referred to as telescoping error. Center deviation was defined as the angle between the actual camera-arthroscope center line and the optimal camera-arthroscope line (i.e. deviation from perfect centering). The average of the 2 trials was used for data analysis (for each module and for each hand). The difference between dominant hand and nondominant hand was then calculated for each metric. A Kruskal-Wallis analysis was performed with a significance level of .05 and a confidence interval of 95.0 (STATA 13.0; StataCorp LP, College Station, TX). Comparison of performance across experience levels was used to assess construct validity for each module. Data outside 3 standard deviations (SD) from the mean were considered outliers and were not included in the statistical analysis.
Results All participants (n ¼ 10 per group) completed 2 trials of each module with each hand, except one novice who completed only one trial of Steady and Telescope with the dominant hand. General characteristics of the 3 groups were similar. Most participants were men. Table 1. Participant Demographics Variable Number of participants Male sex Right hand dominance No. of arthroscopic procedures performed Average no. of yr playing video games Played handdominant sport
Novice Intermediate Expert P Value 10 10 10 NA 70% 90% 90% .68 90% 100% 100% .91 0 11.5 270 .0001 (in lifetime) (per year) 11.8
80%
11
80%
7.5
80%
.48
1.00
301
Twenty-nine participants were right handed and one novice participant was left handed. All but 2 participants in each group played competitive sports that involved the use of their hands (Table 1). Construct validity was established for the Steady and Probe module and the Track a Moving Target module, based primarily on significantly shorter time to module completion as a function of greater surgical experience. For some modules, performance difference between the dominant and nondominant hands decreased with greater arthroscopy experience (in other words, better ambidextrous performance as a function of surgical experience). Steady and Telescope Performance on Steady and Telescope (Table 2) showed no statistically significant differences in performance across the study groups. There were significant differences in time to module completion between the dominant and nondominant hands (P < .05). Residents and experts were statistically more ambidextrous than novices, with less difference in time to completion between the dominant and nondominant hands compared with novices (P < .04). Although novices were always less proficient than experts across all parameters, the findings were not statistically significant, likely because of data variability, which was particularly high in the novice group. Steady and Probe In the Steady and Probe module (Table 3), there was one outlier in the expert group whose data fell more than 3 SD from the mean. Data for that participant were not used for analysis because the participant was deemed a statistical outlier (we made an a priori decision to exclude participants more than 3 SD from the mean.) When the probe was used in the left hand, time to completion was statistically significant across groups (P ¼ .02), with experts performing faster than novices (P ¼ .004). With the probe in the right hand, time to completion bordered on statistical significance (P ¼ .05), although paired comparisons showed that experts were statistically faster than novices (P ¼ .03). There was a statistically significant effect for the difference between right and left hand probe path for experts versus novices (P ¼ .05). Expert performance was generally more consistent, which is reflected by the lower standard deviations of the expert data for most variables (Fig 2). Track a Moving Target There was another outlier in the expert group (not the same individual as in the previous module) in Track a Moving Target (Table 4), whose data fell outside 3 SD from the mean, and thus the data for that individual were not used for statistical analysis based on the same a priori criteria noted previously.
302
K. ROSE AND R. PEDOWITZ
Table 2. Performance on Steady and Telescope Variable Right arthroscope time, seconds Left arthroscope time, seconds Difference in time between hands, seconds Right arthroscope path, mm Left arthroscope path, mm Difference in arthroscope path between hands, mm
Novice 73.9 20.3 75.2 23.6 13.6 13.6
Intermediate 63.7 12.6 61.8 5.4 7.6 8.1
Expert 63.6 10.9 66.7 8.4 8.0 4.7
P Value .17 .12 .04
275.2 131.4 241.2 91.9 41.1 43.3
223.1 28.3 230.7 36.7 34.6 19.7
241.8 48.2 271.3 110.5 57.4 102.8
.53 .51 .80
NOTE. Data presented as mean standard deviation.
Time to completion showed a significant difference between groups (P ¼ .003). Both intermediate participants and experts were significantly faster than the novice group in dominant hand performance (P ¼ .007 and P ¼ .003, respectively) and in nondominant hand performance (P ¼ .004 and P ¼ .003, respectively). Statistical significance was also found between groups for distance deviation when the probe was in the nondominant hand (P ¼ .02), with intermediate participants and experts showing statistically superior proficiency compared with novices (P ¼ .01 and P ¼ .03, respectively). Deviation from center showed statistical significance between groups with a left-handed probe (P ¼ .007). Experts had significantly less deviation from center than did novices (P ¼ .003). When the probe was in the right hand (P ¼ .03), intermediate participants and experts showed less deviation from center than did novices (P ¼ .03 and P ¼ .02, respectively). Intermediate participants had significantly less variability in center deviation than did novices as a function of right versus left hand on the arthroscope (P < .05). Overall, standard deviation was narrower with increasing expertise levels, showing improved consistency for left probe times (Fig 3) and right probe times (Fig 4) for experts compared with the other groups.
Discussion Virtual Reality Curriculum The development of an arthroscopic curriculum, with simulation training at its core, has been a long time
coming. Further manipulations to these modules will be required before the evaluated modules are acceptable enough for use in resident education. Adjustment of technical difficulty is easy with virtual reality simulation; it is accomplished by manipulation of skill challenges and metric tolerances that are built into the computer software. The Steady and Probe module and the Track a Moving Target module both displayed construct validity for certain parameters. Track a Moving Target was able to appropriately differentiate expertise levels for probe time for both hands and deviation from center for both hands, whereas Steady and Probe differentiated levels for probe time for both hands. Together these 2 modules suggest that time to completion may be a metric that can be used (in some cases) as a surrogate for technical proficiency; however, this remains to be evaluated. In our opinion, surgical speed should not come at the expense of technical precision and patient safety. This is a particularly important concept during the early training period. Steady and Telescope did not show construct validity overall. This educational module was designed to teach the concepts of image centering and stability, which are fundamental skills for effective and efficient arthroscopy. It is likely that this exercise would be better able to discriminate between beginners and experts if the task was harder, e.g., by using different-sized targets or by requiring a narrower margin for centering error. These are easily programmed adjustments that can be built into the virtual reality modules.
Table 3. Performance on Steady and Probe Variable Left probe time to completion, seconds Right probe time to completion, seconds Difference in time to completion between hands, seconds Left probe path, mm Right probe path, mm Difference in probe path between hands, mm Left arthroscope path, mm Right arthroscope path, mm Difference in arthroscope path between hands, mm NOTE. Data presented as mean standard deviation.
Novice 222.3 37.5 188.4 40.0 42.0 26.1 1261.9 417.0 1108.2 382.9 314.6 259.7 248.1 74.4 286.5 116.9 82.8 82.6
Intermediate 203.1 56.4 169.9 35.0 33.2 31.5 1152.2 321.3 1102.1 191.2 158.11 91.8 264.4 60.1 262.0 98.1 54.54 42.5
Expert 164.1 32.7 152.7 44.6 22.9 20.1 976.1 289.6 1011.4 270.7 117.1 64.3 243.5 39.7 243.4 59.6 39.2 26.2
P Value .02 .05 .24 .25 .70 .11 .81 .80 .61
303
USE OF VIRTUAL REALITY IN ARTHROSCOPY EDUCATION
Fig 2. Time to completion of Steady and Probe when probe is in left hand based on experience level. Time decreases with experience level (P ¼ .02), and experts were more consistent than novices based on narrower standard deviation (SD). Orange dots represent mean time to completion; shaded gray represents one SD; dashed lines represent 2 SD; and open dots represents greater than one SD from the mean.
Throughout the study, we observed decreasing standard deviation of performance metrics as a function of increased surgical experience (i.e. greater performance consistency with greater experience). Although the data did not always show a significant difference in mean values, the standard deviation was wider for novices in most cases. This can be taken as another way to measure the effectiveness of simulation to discriminate across experience levels. A wider standard deviation shows that novices are not as consistent as experts, and consistency is an important general attribute of surgical proficiency. Also interesting to note is that the mean value for novices is greater than one SD from the mean for experts as seen in all 3 graphs presented in Figures 1-3. This also shows the ability of these modules to differentiate between expertise levels. The modules assessed in this study were created and selected to assess a few of these fundamental arthroscopy skills, including triangulation, centering,
and stabilization; tracking a moving target; and coordinated movements of arthroscope and probe. The first requirement, from an educational perspective, is that the modules clearly differentiate between skill levels (construct validity). Once that is established, performance benchmarks can be developed. Those benchmarks can be used to evaluate whether a resident has reached a proficiency level that is sufficient for progression to more advanced motor skills and for performance of surgery on patients. Additionally, these modules will allow residents to build good habits early in their careers by learning the concepts sequentially and progressively building on them. Virtual reality exercises can be altered so that they are difficult enough to discriminate experience levels while still teaching the fundamental skills. Similar concepts can be applied to simulation using relatively simple physical models. With some adjustments, the modules used here all have the potential to become part of a basic motor skills curriculum. Aggarwal et al.10 showed that learning 12 basic skills rather than just the 2 most complex skills was a more efficient way to teach residents. Although it is possible to become proficient eventually, it takes a lot more repetition if one skips straight to the most complex basic skills, rather than first mastering the fundamental elements of a complex task. Although this study looked at only 3 basic skills of arthroscopy, the general goal of the FAST program is development of a comprehensive curriculum that covers all the fundamental skills defined by thorough task deconstruction. The FAST program curriculum and supporting materials are currently open access at http://www.ABOS.org. Additional studies are required to determine what parameters should be used as the best markers of motor skills proficiency. Although this study showed that time is an easy and reliable marker, it is important to see if there are better markers to assess performance quality. For example, prevalence of instrument loss and “look downs,” as well as path length have been shown to be consistent metrics of skill level and proficiency11,12 and are possible metrics to evaluate in future studies using
Table 4. Performance on Track a Moving Target Variable Time to completion, probe in left hand, seconds Time to completion, probe in right hand, seconds Difference in time between hands, seconds Distance deviation, probe left, mm Distance deviation, probe right, mm Difference in distance deviation between hands, mm Arthroscope center deviation, probe in left hand, Arthroscope center deviation, probe in right hand, Difference in center deviation between hands, NOTE. Data presented as mean standard deviation.
Novice 61.6 15.3 65.6 21.6 14.4 13.2 199.7 64.4 237.6 141.1 37.9 124.9 176.6 31.4 204.2 117.3 68.1 81.8
Intermediate 39.2 10.7 38.5 13.0 6.4 4.7 130.7 26.8 149.6 58.2 18.9 38.5 149.8 38.3 129.3 31.8 21.5 25.8
Expert 38.4 9.3 36.7 8.7 8.3 5.0 137.9 47.3 158.9 51.9 21.0 64.4 117.8 32.7 126.2 38.7 27.2 35.6
P Value .003 .004 .31 .02 .22 .84 .007 .03 .09
304
K. ROSE AND R. PEDOWITZ
as a way to measure skill improvement over time toward eventual proficiency (i.e. convergence of performance for dominant and nondominant hands as the trainee progresses in the arthroscopy motor skills curriculum).
Fig 3. Time to completion of Track A Moving Target when probe is in left hand based on experience level. Time decreases with experience level (P ¼ .003), as does consistency. Orange dots represent mean time to completion; shaded gray represents one standard deviation (SD); dashed lines represent 2 SD; and open dots represent greater than one SD from the mean.
these modules. Performance targets are also important in the learning process, because they allow trainees to have tangible goals.13 As virtual reality modules are altered and improved, construct validity should be reconfirmed, because this is a key element for rational incorporation into a generalized residency curriculum. Ambidexterity It is expected that experts at arthroscopic surgery would be more ambidextrous at arthroscopic skills than novices, because they have practice using both hands repeatedly during these precise movements. To our knowledge, this is the first study to evaluate arthroscopic ambidexterity across expertise levels. Statistically significant differences in ambidexterity between experts and novices were observed for completion time in the Steady and Telescope module. Although some comparisons did not reach statistical significance in the other modules (probably because of an inadequate number of participants), experts were generally more consistent and performed better with their nondominant hand than did novices. As novices practice their motor skills, ambidexterity should improve, and modules such as those studied here can facilitate nondominant dexterity in a low-stress, patient-safe environment. Increasing ambidextrous proficiency as a function of arthroscopic experience should be more easily shown with more difficult virtual reality motor skills modules. The idea of increased ambidexterity with experience is simple and relatively intuitive, yet the concept has not been specifically shown in the previous literature for arthroscopy training. This concept could be exploited
Limitations This study has several limitations. First, the sample size was small. A larger sample size would have improved our ability to find significant differences between performance of experts and novices on the various modules. Second, there was no standard for how the probe and camera devices were supposed to be grasped within the hand, which could have added some data variability. Third, some participants rested their wrists on the table for support while doing the exercises, whereas others performed the module with the arms free in space. We allowed this option to be selected according to the preference of each participant, but it could have introduced some additional variability between the groups.
Conclusions This study represents a new way to assess basic arthroscopy skills using virtual reality modules developed through task deconstruction. Participants with the most arthroscopic experience performed better and were more consistent than novices on all 3 virtual reality modules. Greater arthroscopic experience correlates with more symmetry of ambidextrous performance. However, further adjustment of these modules is needed to better simulate fundamental arthroscopic skills and discriminate between experience levels.
Fig 4. Time to completion of Track A Moving Target when probe is in right hand based on experience level. Time decreases with experience level (P ¼ .004), as does consistency. Orange dots represent mean time to completion; shaded gray represents one standard deviation (SD); dashed lines represent 2 SD; and open dots represent greater than one SD from the mean.
USE OF VIRTUAL REALITY IN ARTHROSCOPY EDUCATION
References 1. Pedowitz RA, Esch J, Snyder S. Evaluation of a virtual reality simulator for arthroscopy skills development. Arthroscopy 2002;18:E29. 2. Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244: 310-314. 3. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P, Darzi A. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg 2009;96: 1086-1093. 4. Karahan M, Unalan PC, Bozkurt S, et al. Correlation of basic motor skills with arthroscopic experience. Acta Orthop Traumatol Turc 2009;43:49-53 [in Turkish]. 5. Gomoll AH, Pappas G, Forsythe B, Warner JJ. Individual skill progression on a virtual reality simulator for shoulder arthroscopy: A 3-year follow-up study. Am J Sports Med 2008;36:1139-1142. 6. Srivastava S, Youngblood PL, Rawn C, Hariri S, Heinrichs WL, Ladd AL. Initial evaluation of a shoulder arthroscopy simulator: Establishing construct validity. J Shoulder Elbow Surg 2004;13:196-205.
305
7. Panait L, Bell RL, Roberts KE, Duffy AJ. Designing and validating a customized virtual reality-based laparoscopic skills curriculum. J Surg Educ 2008;65:413-417. 8. Shetty S, Panait L, Baranoski J, et al. Construct and face validity of a virtual reality-based camera navigation curriculum. J Surg Res 2012;177:191-195. 9. Andersen C, Winding TN, Vesterby MS. Development of simulated arthroscopic skills. Acta Orthop 2011;82:90-95. 10. Aggarwal R, Grantcharov T, Moorthy K, Hance J, Darzi A. A competency-based virtual reality training curriculum for the acquisition of laparoscopic psychomotor skill. Am J Surg 2006;191:128-133. 11. Alvand A, Khan T, Al-Ali S, Jackson WF, Price AJ, Rees JL. Simple visual parameters for objective assessment of arthroscopic skill. J Bone Joint Surg Am 2012;94:e97. 12. Sinitsky DM, Fernando B, Berlingieri P. Establishing a curriculum for the acquisition of laparoscopic psychomotor skills in the virtual reality environment. Am J Surg 2012;204:367-376. 13. Gauger PG, Hauge LS, Andreatta PB, et al. Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons. Am J Surg 2010;199:72-80.
Purpose: The purpose of this study was to investigate the use and validity of virtual reality modules as part of the educational approach to mastering arthroscopy in a safe environment by assessing the ability to distinguish between experience levels. Additionally, the study aimed to evaluate whether experts have greater ambidexterity than do novices. Methods: Three virtual reality modules (Swemac/Augmented Reality Systems, Linkoping, Sweden) were created to test fundamental arthroscopic skills. Thirty participantsd10 experts consisting of faculty, 10 intermediate participants consisting of orthopaedic residents, and 10 novices consisting of medical studentsdperformed each exercise. Steady and Telescope was designed to train centering and image stability. Steady and Probe was designed to train basic triangulation. Track and Moving Target was designed to train coordinated motions of arthroscope and probe. Metrics reflecting speed, accuracy, and efficiency of motion were used to measure construct validity. Results: Steady and Probe and Track a Moving Target both exhibited construct validity, with better performance by experts and intermediate participants than by novices (P < .05), whereas Steady and Telescope did not show validity. There was an overall trend toward better ambidexterity as a function of greater surgical experience, with experts consistently more proficient than novices throughout all 3 modules. Conclusions: This study represents a new way to assess basic arthroscopy skills using virtual reality modules developed through task deconstruction. Participants with the most arthroscopic experience performed better and were more consistent than novices on all 3 virtual reality modules. Greater arthroscopic experience correlates with more symmetry of ambidextrous performance. However, further adjustment of the modules may better simulate fundamental arthroscopic skills and discriminate between experience levels. Clinical Relevance: Arthroscopy training is a critical element of orthopaedic surgery resident training. Developing techniques to safely and effectively train these skills is critical for patient safety and resident education.
A
n important aspect of orthopaedic residency training is teaching residents the motor skills required for arthroscopic surgery. The development of virtual reality simulators to teach the fundamental skills of arthroscopy may facilitate teaching and proficiency assessment before performance of these skills on patients.1 Learning basic tasks and then progressing in procedural complexity is the most logical way to train
From the Department of Orthopaedic Surgery, David Geffen School of Medicine at UCLA, Santa Monica, California, U.S.A. Supported by the Arthroscopy Association of North American Education and by academic enrichment funds provided by the David Geffen School of Medicine at UCLA. The authors report the following potential conflict of interest or source of funding: R.P. receives support from Stryker and DJ Orthopaedics. Received March 4, 2014; accepted August 15, 2014. Address correspondence to Kelsey Rose, B.A., David Geffen School of Medicine at UCLA, 334 20th St, Santa Monica, CA 90402, U.S.A. E-mail: [email protected] Ó 2015 by the Arthroscopy Association of North America 0749-8063/14184/$36.00 http://dx.doi.org/10.1016/j.arthro.2014.08.016
technical skills, and this systematic approach may be enhanced by virtual reality simulation.2,3 The Fundamentals of Arthroscopic Surgery Training (FAST) program is a collaborative effort of the Arthroscopy Association of North America, the American Academy of Orthopaedic Surgery, and the American Board of Orthopaedic Surgery. The basic objectives of the FAST program are to break down arthroscopy into its most fundamental and basic skills (task deconstruction), to teach these skills to residents in an organized and progressive fashion, and to have residents practice using simulation until they have reached a determined level of proficiency. Karahan et al.4 showed that basic motor skills are correlated with arthroscopic competency, which supports the goals of the FAST program. Performance on virtual reality simulators correlates with surgical experience.5 Those with the most surgical experience show the most rapid learning curves on virtual reality modules and require the fewest number of repetitions to reach proficiency.6-8 This is a good indicator that the modules require motor skills similar to those actually used in surgery.6 Appropriate modules
Arthroscopy: The Journal of Arthroscopic and Related Surgery, Vol 31, No 2 (February), 2015: pp 299-305
299
300
K. ROSE AND R. PEDOWITZ
not only can differentiate expertise levels (construct validity) but also are effective at improving arthroscopic skills in inexperienced surgeons.9 Andersen et al.9 showed that those who were allowed to train on the simulator between a pre- and post-test performed better than those who did not, suggesting that repetition facilitates skill development. The purpose of this study was to evaluate the use of 3 virtual reality simulator modules that were designed to teach several basic skills of arthroscopy as educational tools for arthroscopic training by comparing their ability to distinguish between groups with different levels of arthroscopy experience. In the general sense, this is part of an iterative developmental process that should improve our educational curriculum. Additionally, this study aimed to look at ambidexterity across experience levels. Intuitively, experts who have been using both hands for precise movements repeatedly for years should show better equality of performance between hands than do novice surgeons. However, to our knowledge, this has never been studied regarding arthroscopic surgery. It is hypothesized that those with the most arthroscopic experience will perform more consistently and more efficiently and show better ambidexterity on the virtual reality modules, proving that experts are more ambidextrous and that these modules are testing fundamental arthroscopic skills.
Methods Three virtual reality modules were created for the dual-haptic ArthroVision platform (Swemac/Augmented Reality Systems, Linkoping, Sweden). These tasks were defined after thorough deconstruction of arthroscopic surgical skills into their most basic forms. Participants visualized the exercises on the screen of a laptop computer connected to both haptic devices. The ergonomics mimicked the distance between the common knee arthroscopy portals (inferolateral and inferomedial), with triangulation depth selected to reflect the typical working distance from skin portal to medial meniscus. The computer software modeled the 30 arthroscope, which is the standard device for knee arthroscopy. Participants were divided into 3 groups based on experience levels: (1) experts, defined as master faculty at an Arthroscopy Association of North America Resident’s Arthroscopy Fundamental Skills Course (n ¼ 10); (2) intermediate participants, defined as orthopaedic residents in training (n ¼ 10) who were attending the same course; and (3) novices, defined as medical students without any previous training in arthroscopy (n ¼ 10). The protocol was approved by our institutional review board and all participants provided informed consent. Participants were asked to fill out a questionnaire to determine handedness and arthroscopic experience.
Each module was performed twice, and then the same module was repeated twice using the contralateral hand. Instructions were provided using the same standardized script for all participants, and up to 2 minutes of practice was allowed for each module. Randomization was used to determine which hand each participant would use first for each module. Each virtual reality module was designed to train participants on different basic arthroscopy skills. Each module involved spherical targets and a circular outline in the center of the screen to coach image centering. The first module, Steady and Telescope, trained image centering and image stability. Participants were instructed to use the arthroscope to center a highlighted yellow target and to hold it steady for 5 seconds. A contralateral probe was not used in this module. After successfully centering the target, it would turn gray and participants would find the next highlighted yellow target. This would be repeated until all targets were completed. The second module, Steady and Probe, trained basic triangulation, with the virtual arthroscope in one hand and the virtual probe in the contralateral hand. Participants were instructed to use the arthroscope to center a highlighted yellow sphere and then use the simulated probe to push smaller blue balls on the surface of the sphere into the target (Fig 1). When all the smaller balls were pushed in completely, the sphere would turn gray and a new target would appear. The third module, Track a Moving Target, trained coordination of movement of arthroscope and probe. Participants were instructed to keep the target centered
Fig 1. Steady and Probe module is shown with probe pushing in a blue ball into sphere in ArthroVision virtual reality platform.
USE OF VIRTUAL REALITY IN ARTHROSCOPY EDUCATION
with their arthroscope while pushing the target along a winding track with a probe using the contralateral hand. The target remained yellow when properly centered and would turn red to signify a deviation from center. The target could still move if not centered properly, but deviation from the center was measured. Various metrics were recorded by the computer for each of the virtual reality modules. For Steady and Telescope, the computer recorded time of target in focus, arthroscope path length, and time to completion of the module. For Steady and Probe, metrics included probe path length, arthroscope path length, and time to module completion. For Track a Moving Target, the computer measured out-of-focus time, distance deviation, center deviation, and module completion time. Distance deviation was defined as the arthroscope distance to the target minus the ideal arthroscope distance to the target (i.e. net deviation from perfect arthroscope distance). This can also be referred to as telescoping error. Center deviation was defined as the angle between the actual camera-arthroscope center line and the optimal camera-arthroscope line (i.e. deviation from perfect centering). The average of the 2 trials was used for data analysis (for each module and for each hand). The difference between dominant hand and nondominant hand was then calculated for each metric. A Kruskal-Wallis analysis was performed with a significance level of .05 and a confidence interval of 95.0 (STATA 13.0; StataCorp LP, College Station, TX). Comparison of performance across experience levels was used to assess construct validity for each module. Data outside 3 standard deviations (SD) from the mean were considered outliers and were not included in the statistical analysis.
Results All participants (n ¼ 10 per group) completed 2 trials of each module with each hand, except one novice who completed only one trial of Steady and Telescope with the dominant hand. General characteristics of the 3 groups were similar. Most participants were men. Table 1. Participant Demographics Variable Number of participants Male sex Right hand dominance No. of arthroscopic procedures performed Average no. of yr playing video games Played handdominant sport
Novice Intermediate Expert P Value 10 10 10 NA 70% 90% 90% .68 90% 100% 100% .91 0 11.5 270 .0001 (in lifetime) (per year) 11.8
80%
11
80%
7.5
80%
.48
1.00
301
Twenty-nine participants were right handed and one novice participant was left handed. All but 2 participants in each group played competitive sports that involved the use of their hands (Table 1). Construct validity was established for the Steady and Probe module and the Track a Moving Target module, based primarily on significantly shorter time to module completion as a function of greater surgical experience. For some modules, performance difference between the dominant and nondominant hands decreased with greater arthroscopy experience (in other words, better ambidextrous performance as a function of surgical experience). Steady and Telescope Performance on Steady and Telescope (Table 2) showed no statistically significant differences in performance across the study groups. There were significant differences in time to module completion between the dominant and nondominant hands (P < .05). Residents and experts were statistically more ambidextrous than novices, with less difference in time to completion between the dominant and nondominant hands compared with novices (P < .04). Although novices were always less proficient than experts across all parameters, the findings were not statistically significant, likely because of data variability, which was particularly high in the novice group. Steady and Probe In the Steady and Probe module (Table 3), there was one outlier in the expert group whose data fell more than 3 SD from the mean. Data for that participant were not used for analysis because the participant was deemed a statistical outlier (we made an a priori decision to exclude participants more than 3 SD from the mean.) When the probe was used in the left hand, time to completion was statistically significant across groups (P ¼ .02), with experts performing faster than novices (P ¼ .004). With the probe in the right hand, time to completion bordered on statistical significance (P ¼ .05), although paired comparisons showed that experts were statistically faster than novices (P ¼ .03). There was a statistically significant effect for the difference between right and left hand probe path for experts versus novices (P ¼ .05). Expert performance was generally more consistent, which is reflected by the lower standard deviations of the expert data for most variables (Fig 2). Track a Moving Target There was another outlier in the expert group (not the same individual as in the previous module) in Track a Moving Target (Table 4), whose data fell outside 3 SD from the mean, and thus the data for that individual were not used for statistical analysis based on the same a priori criteria noted previously.
302
K. ROSE AND R. PEDOWITZ
Table 2. Performance on Steady and Telescope Variable Right arthroscope time, seconds Left arthroscope time, seconds Difference in time between hands, seconds Right arthroscope path, mm Left arthroscope path, mm Difference in arthroscope path between hands, mm
Novice 73.9 20.3 75.2 23.6 13.6 13.6
Intermediate 63.7 12.6 61.8 5.4 7.6 8.1
Expert 63.6 10.9 66.7 8.4 8.0 4.7
P Value .17 .12 .04
275.2 131.4 241.2 91.9 41.1 43.3
223.1 28.3 230.7 36.7 34.6 19.7
241.8 48.2 271.3 110.5 57.4 102.8
.53 .51 .80
NOTE. Data presented as mean standard deviation.
Time to completion showed a significant difference between groups (P ¼ .003). Both intermediate participants and experts were significantly faster than the novice group in dominant hand performance (P ¼ .007 and P ¼ .003, respectively) and in nondominant hand performance (P ¼ .004 and P ¼ .003, respectively). Statistical significance was also found between groups for distance deviation when the probe was in the nondominant hand (P ¼ .02), with intermediate participants and experts showing statistically superior proficiency compared with novices (P ¼ .01 and P ¼ .03, respectively). Deviation from center showed statistical significance between groups with a left-handed probe (P ¼ .007). Experts had significantly less deviation from center than did novices (P ¼ .003). When the probe was in the right hand (P ¼ .03), intermediate participants and experts showed less deviation from center than did novices (P ¼ .03 and P ¼ .02, respectively). Intermediate participants had significantly less variability in center deviation than did novices as a function of right versus left hand on the arthroscope (P < .05). Overall, standard deviation was narrower with increasing expertise levels, showing improved consistency for left probe times (Fig 3) and right probe times (Fig 4) for experts compared with the other groups.
Discussion Virtual Reality Curriculum The development of an arthroscopic curriculum, with simulation training at its core, has been a long time
coming. Further manipulations to these modules will be required before the evaluated modules are acceptable enough for use in resident education. Adjustment of technical difficulty is easy with virtual reality simulation; it is accomplished by manipulation of skill challenges and metric tolerances that are built into the computer software. The Steady and Probe module and the Track a Moving Target module both displayed construct validity for certain parameters. Track a Moving Target was able to appropriately differentiate expertise levels for probe time for both hands and deviation from center for both hands, whereas Steady and Probe differentiated levels for probe time for both hands. Together these 2 modules suggest that time to completion may be a metric that can be used (in some cases) as a surrogate for technical proficiency; however, this remains to be evaluated. In our opinion, surgical speed should not come at the expense of technical precision and patient safety. This is a particularly important concept during the early training period. Steady and Telescope did not show construct validity overall. This educational module was designed to teach the concepts of image centering and stability, which are fundamental skills for effective and efficient arthroscopy. It is likely that this exercise would be better able to discriminate between beginners and experts if the task was harder, e.g., by using different-sized targets or by requiring a narrower margin for centering error. These are easily programmed adjustments that can be built into the virtual reality modules.
Table 3. Performance on Steady and Probe Variable Left probe time to completion, seconds Right probe time to completion, seconds Difference in time to completion between hands, seconds Left probe path, mm Right probe path, mm Difference in probe path between hands, mm Left arthroscope path, mm Right arthroscope path, mm Difference in arthroscope path between hands, mm NOTE. Data presented as mean standard deviation.
Novice 222.3 37.5 188.4 40.0 42.0 26.1 1261.9 417.0 1108.2 382.9 314.6 259.7 248.1 74.4 286.5 116.9 82.8 82.6
Intermediate 203.1 56.4 169.9 35.0 33.2 31.5 1152.2 321.3 1102.1 191.2 158.11 91.8 264.4 60.1 262.0 98.1 54.54 42.5
Expert 164.1 32.7 152.7 44.6 22.9 20.1 976.1 289.6 1011.4 270.7 117.1 64.3 243.5 39.7 243.4 59.6 39.2 26.2
P Value .02 .05 .24 .25 .70 .11 .81 .80 .61
303
USE OF VIRTUAL REALITY IN ARTHROSCOPY EDUCATION
Fig 2. Time to completion of Steady and Probe when probe is in left hand based on experience level. Time decreases with experience level (P ¼ .02), and experts were more consistent than novices based on narrower standard deviation (SD). Orange dots represent mean time to completion; shaded gray represents one SD; dashed lines represent 2 SD; and open dots represents greater than one SD from the mean.
Throughout the study, we observed decreasing standard deviation of performance metrics as a function of increased surgical experience (i.e. greater performance consistency with greater experience). Although the data did not always show a significant difference in mean values, the standard deviation was wider for novices in most cases. This can be taken as another way to measure the effectiveness of simulation to discriminate across experience levels. A wider standard deviation shows that novices are not as consistent as experts, and consistency is an important general attribute of surgical proficiency. Also interesting to note is that the mean value for novices is greater than one SD from the mean for experts as seen in all 3 graphs presented in Figures 1-3. This also shows the ability of these modules to differentiate between expertise levels. The modules assessed in this study were created and selected to assess a few of these fundamental arthroscopy skills, including triangulation, centering,
and stabilization; tracking a moving target; and coordinated movements of arthroscope and probe. The first requirement, from an educational perspective, is that the modules clearly differentiate between skill levels (construct validity). Once that is established, performance benchmarks can be developed. Those benchmarks can be used to evaluate whether a resident has reached a proficiency level that is sufficient for progression to more advanced motor skills and for performance of surgery on patients. Additionally, these modules will allow residents to build good habits early in their careers by learning the concepts sequentially and progressively building on them. Virtual reality exercises can be altered so that they are difficult enough to discriminate experience levels while still teaching the fundamental skills. Similar concepts can be applied to simulation using relatively simple physical models. With some adjustments, the modules used here all have the potential to become part of a basic motor skills curriculum. Aggarwal et al.10 showed that learning 12 basic skills rather than just the 2 most complex skills was a more efficient way to teach residents. Although it is possible to become proficient eventually, it takes a lot more repetition if one skips straight to the most complex basic skills, rather than first mastering the fundamental elements of a complex task. Although this study looked at only 3 basic skills of arthroscopy, the general goal of the FAST program is development of a comprehensive curriculum that covers all the fundamental skills defined by thorough task deconstruction. The FAST program curriculum and supporting materials are currently open access at http://www.ABOS.org. Additional studies are required to determine what parameters should be used as the best markers of motor skills proficiency. Although this study showed that time is an easy and reliable marker, it is important to see if there are better markers to assess performance quality. For example, prevalence of instrument loss and “look downs,” as well as path length have been shown to be consistent metrics of skill level and proficiency11,12 and are possible metrics to evaluate in future studies using
Table 4. Performance on Track a Moving Target Variable Time to completion, probe in left hand, seconds Time to completion, probe in right hand, seconds Difference in time between hands, seconds Distance deviation, probe left, mm Distance deviation, probe right, mm Difference in distance deviation between hands, mm Arthroscope center deviation, probe in left hand, Arthroscope center deviation, probe in right hand, Difference in center deviation between hands, NOTE. Data presented as mean standard deviation.
Novice 61.6 15.3 65.6 21.6 14.4 13.2 199.7 64.4 237.6 141.1 37.9 124.9 176.6 31.4 204.2 117.3 68.1 81.8
Intermediate 39.2 10.7 38.5 13.0 6.4 4.7 130.7 26.8 149.6 58.2 18.9 38.5 149.8 38.3 129.3 31.8 21.5 25.8
Expert 38.4 9.3 36.7 8.7 8.3 5.0 137.9 47.3 158.9 51.9 21.0 64.4 117.8 32.7 126.2 38.7 27.2 35.6
P Value .003 .004 .31 .02 .22 .84 .007 .03 .09
304
K. ROSE AND R. PEDOWITZ
as a way to measure skill improvement over time toward eventual proficiency (i.e. convergence of performance for dominant and nondominant hands as the trainee progresses in the arthroscopy motor skills curriculum).
Fig 3. Time to completion of Track A Moving Target when probe is in left hand based on experience level. Time decreases with experience level (P ¼ .003), as does consistency. Orange dots represent mean time to completion; shaded gray represents one standard deviation (SD); dashed lines represent 2 SD; and open dots represent greater than one SD from the mean.
these modules. Performance targets are also important in the learning process, because they allow trainees to have tangible goals.13 As virtual reality modules are altered and improved, construct validity should be reconfirmed, because this is a key element for rational incorporation into a generalized residency curriculum. Ambidexterity It is expected that experts at arthroscopic surgery would be more ambidextrous at arthroscopic skills than novices, because they have practice using both hands repeatedly during these precise movements. To our knowledge, this is the first study to evaluate arthroscopic ambidexterity across expertise levels. Statistically significant differences in ambidexterity between experts and novices were observed for completion time in the Steady and Telescope module. Although some comparisons did not reach statistical significance in the other modules (probably because of an inadequate number of participants), experts were generally more consistent and performed better with their nondominant hand than did novices. As novices practice their motor skills, ambidexterity should improve, and modules such as those studied here can facilitate nondominant dexterity in a low-stress, patient-safe environment. Increasing ambidextrous proficiency as a function of arthroscopic experience should be more easily shown with more difficult virtual reality motor skills modules. The idea of increased ambidexterity with experience is simple and relatively intuitive, yet the concept has not been specifically shown in the previous literature for arthroscopy training. This concept could be exploited
Limitations This study has several limitations. First, the sample size was small. A larger sample size would have improved our ability to find significant differences between performance of experts and novices on the various modules. Second, there was no standard for how the probe and camera devices were supposed to be grasped within the hand, which could have added some data variability. Third, some participants rested their wrists on the table for support while doing the exercises, whereas others performed the module with the arms free in space. We allowed this option to be selected according to the preference of each participant, but it could have introduced some additional variability between the groups.
Conclusions This study represents a new way to assess basic arthroscopy skills using virtual reality modules developed through task deconstruction. Participants with the most arthroscopic experience performed better and were more consistent than novices on all 3 virtual reality modules. Greater arthroscopic experience correlates with more symmetry of ambidextrous performance. However, further adjustment of these modules is needed to better simulate fundamental arthroscopic skills and discriminate between experience levels.
Fig 4. Time to completion of Track A Moving Target when probe is in right hand based on experience level. Time decreases with experience level (P ¼ .004), as does consistency. Orange dots represent mean time to completion; shaded gray represents one standard deviation (SD); dashed lines represent 2 SD; and open dots represent greater than one SD from the mean.
USE OF VIRTUAL REALITY IN ARTHROSCOPY EDUCATION
References 1. Pedowitz RA, Esch J, Snyder S. Evaluation of a virtual reality simulator for arthroscopy skills development. Arthroscopy 2002;18:E29. 2. Aggarwal R, Grantcharov TP, Eriksen JR, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244: 310-314. 3. Aggarwal R, Crochet P, Dias A, Misra A, Ziprin P, Darzi A. Development of a virtual reality training curriculum for laparoscopic cholecystectomy. Br J Surg 2009;96: 1086-1093. 4. Karahan M, Unalan PC, Bozkurt S, et al. Correlation of basic motor skills with arthroscopic experience. Acta Orthop Traumatol Turc 2009;43:49-53 [in Turkish]. 5. Gomoll AH, Pappas G, Forsythe B, Warner JJ. Individual skill progression on a virtual reality simulator for shoulder arthroscopy: A 3-year follow-up study. Am J Sports Med 2008;36:1139-1142. 6. Srivastava S, Youngblood PL, Rawn C, Hariri S, Heinrichs WL, Ladd AL. Initial evaluation of a shoulder arthroscopy simulator: Establishing construct validity. J Shoulder Elbow Surg 2004;13:196-205.
305
7. Panait L, Bell RL, Roberts KE, Duffy AJ. Designing and validating a customized virtual reality-based laparoscopic skills curriculum. J Surg Educ 2008;65:413-417. 8. Shetty S, Panait L, Baranoski J, et al. Construct and face validity of a virtual reality-based camera navigation curriculum. J Surg Res 2012;177:191-195. 9. Andersen C, Winding TN, Vesterby MS. Development of simulated arthroscopic skills. Acta Orthop 2011;82:90-95. 10. Aggarwal R, Grantcharov T, Moorthy K, Hance J, Darzi A. A competency-based virtual reality training curriculum for the acquisition of laparoscopic psychomotor skill. Am J Surg 2006;191:128-133. 11. Alvand A, Khan T, Al-Ali S, Jackson WF, Price AJ, Rees JL. Simple visual parameters for objective assessment of arthroscopic skill. J Bone Joint Surg Am 2012;94:e97. 12. Sinitsky DM, Fernando B, Berlingieri P. Establishing a curriculum for the acquisition of laparoscopic psychomotor skills in the virtual reality environment. Am J Surg 2012;204:367-376. 13. Gauger PG, Hauge LS, Andreatta PB, et al. Laparoscopic simulation training with proficiency targets improves practice and performance of novice surgeons. Am J Surg 2010;199:72-80.