- Email: [email protected]
Correlating virtual reality and box trainer tasks in the assessment of laparoscopic surgical skills
Meeting Papers
www. AJOG.org
Correlating virtual reality and box trainer tasks in the assessment of laparoscopic surgical skills Jordan Newmark, MD; Vani Dandolu, MD; Richard Milner, BS; Harsh Grewal, MD; Sean Harbison, MD; Enrique Hernandez, MD OBJECTIVE: The purpose of this study was to examine the correlation in the assessment of laparoscopic surgical skills in medical students with the use of a virtual reality laparoscopic trainer and a low-fidelity video box trainer with comparative tasks. STUDY DESIGN: Third-year medical students were asked to perform 3
basic skills set modules on LapSim (Surgical Science, Gothenburg, Sweden): coordination, grasping and lifting, and handling the intestines. Each task was set at the easiest level, and each student was allowed a maximum of 10 attempts to complete each task. Similarappearing tasks were chosen for comparison with the use of a standard video box trainer: pegboard, cup drop and rope pass, respectively. Laparoscopic skills were evaluated with the use of both trainers during 1 session. Pearson’s correlation coefficients were used to compare paired data on each student using statistical software. RESULTS: Forty-seven of 65 medical students were assigned to clini-
cal clerkships on-campus at Temple University School of Medicine participated in the study. All 47 students participated in the video box trainer tasks; 34 students completed both the video box trainer and
LapSim skills set. Observations that were obtained on the LapSim virtual reality system and video box trainer simulator demonstrated several correlations. The time to completion for the LapSim coordination task and the pegboard task were correlated (r ⫽ 0.507; P ⫽ .006), as were the grasping and lifting task completion time on LapSim and the comparative box trainer cup drop task completion time (r ⫽ 0.404; P ⫽ .022). When accounting for errors, the LapSim coordination task tissue damage score was correlated with the sum of all box trainer errors (r ⫽ 0.353; P ⫽ .040); the average grasping and lifting tissue damage was correlated with the total number of errors during all box trainer tasks (r ⫽ 0.374; P ⫽ .035). CONCLUSION: Overall, in evaluating laparoscopic skills, the LapSim and video box trainer were correlated positively with one another. The scoring of laparoscopic skills by both systems appears to be equivalent for the measurement of time to task completion and number of errors.
Key words: box trainer, virtual reality simulator
Cite this article as: Newmark J, Dandolu V, Milner R, Grewal H, Harbison S, Hernandez E. Correlating virtual reality and box trainer tasks in the assessment of laparoscopic surgical skills. Am J Obstet Gynecol 2007;197:546.e1-546.e4.
A
s surgical techniques become more advanced, especially in the area of minimally invasive surgery, 3-dimensional open surgical fields are being replaced by 2-dimensional video images.1 It is important for surgeons and surgical residents to incorporate these developing technologies into skill acquisition and practice improvement in an effective
manner, especially in this new “endoscopic era.”2 Traditional video box trainers and virtual reality simulators have allowed for faster acquisition of the endoscopic technical skills ex-vivo before clinical practice.2,3 Although simulators can never replicate reality completely, they are valuable in recreating important aspects of life-like surgical sit-
From the Departments of Obstetrics and Gynecology (Drs Dandolu and Hernandez) and Surgery (Drs Grewal and Harbison) and the research laboratory (Mr Milner), Temple University School of Medicine, Philadelphia, PA. Dr Newmark, a medical student at Temple University at the time of writing, is currently a resident in anesthesiology at Harvard Medical School. Presented at the annual meeting of the Council on Resident Education in Obstetrics and Gynecology and the Association of Professors of Gynecology and Obstetrics, Salt Lake City, UT, March 7-10, 2007. Received Mar. 16, 2007; revised May 27, 2007; accepted Jul. 23, 2007. Reprints: Vani Dandolu, MD, 3401 N Broad St, Temple University Hospital, Philadelphia, PA 19140: [email protected]. 0002-9378/$32.00 © 2007 Mosby, Inc. All rights reserved. doi: 10.1016/j.ajog.2007.07.026
546.e1
American Journal of Obstetrics & Gynecology NOVEMBER 2007
uations.1 Therefore, surgical simulation has become increasingly widespread and important in the teaching of surgical skills and has the potential to be a valuable tool for accomplishing certain Accreditation Council for Graduate Medical Education (ACGME) competency goals and trainee skills evaluation of surgical house staff in surgical specialties.4,5 Several laparoscopic surgery simulators exist and include computerized virtual reality simulators and traditional video box trainers.6 Although these 2 modalities may appear similar, each has its own unique advantages and disadvantages. Virtual reality simulators provide fast and precise results on many measures of skill, but the validity of these measurements is unproven. Assessment scores may not duplicate performance in the operating room; in 1 study, these simulators were unable to measure surgical errors in a valid fashion.2 Virtual reality simulators are also very expen-
Meeting Papers
www.AJOG.org sive; many cost in excess of $100,000, lack portability, and require ongoing technical support and system “upgrades.”4 Video box trainers, on the other hand, often more closely approximate real life surgical conditions and are considerably less expensive. These trainers, however, cannot automatically provide metrics for performance variables as virtual reality systems can. Trained surgical staff members are needed to evaluate skill parameters in a box trainer; therefore, metrics for a box trainer are subject to inter- and intraobserver variation. Much research in recent years has focused on translating demonstrated surgical skills in different simulation scenarios into improved patient safety and outcomes.7 It was the aim of this study to correlate the results of measuring laparoscopic surgical skills with the use of virtual reality and box trainer tasks and to compare their ability to assess similar endoscopic skills.
M ETHODS Subjects Forty-seven third-year medical students at a large university were recruited; all of the students had minimal or no experience using laparoscopic equipment. Thirty-four of the 47 students completed 3 different virtual reality basic skills modules and 3 different video box trainer tasks; all 47 students completed laparoscopic skills on the box trainer alone. The surgical simulation center at Temple University School of Medicine has 4 box trainers with video monitors attached; the LapSim unit (Surgical Science, Gothenburg, Sweden) was obtained from Immersion Medical (Gathersburg, MD) for a 2-week time period for the purpose of the study. There were technical difficulties with use of the LapSim during the initial 2 days; therefore, there is a discrepancy in the number of students who were assessed on the box trainer vs LapSim. Institutional review board approval was obtained before subject recruitment, and all participants signed an informed consent form before participation.
Surgical tasks Virtual reality tasks were demonstrated with the LapSim computerized laparoscopic surgery simulator. Three basic training modules were set at the easy level for the present study. Tasks set at the easy level have shown construct validity for all measured variables.2 Students were asked to complete the following basic LapSim tasks: coordination, lifting and grasping, and handling intestines with the use of 2 grasper handles, with artificial trocars built into LapSim. Study participants, during 1 session, repeated each task until a passing score (ⱖ70 on a 1-100 scale, scored by LapSim) was achieved. The coordination task entailed the use of alternating hands to navigate a laparoscopic camera and to touch colored balls with a probe, placing them into a target without causing damage to nearby healthy tissues. The lifting and grasping module involved the use of 1 hand to lift a tile within the field with a probe while the other hand grasped a needle and placed it under the tile, simultaneously trying to avoid damaging tissue walls. Last, the handling intestines task required students to pass a 10-cm length of small bowel from 1 hand to the other without puncturing the bowel or causing local tissue damage. The box trainer was constructed of a plastic bin that was covered by a surgical cloth with 4 trocar ports, 2 ports on each side. The bin contained the components that were needed to complete the surgical tasks. Real surgical instruments (such as graspers and laparoscopic cameras) were the tools provided; the camera was linked to a color monitor similar to those used in the operating room for laparoscopic cases. Moving rings on a pegboard, grasping beans (n ⫽ 10) and dropping them through a hole in an inverted specimen cup, and passing a length of a rope hand-to-hand were the box trainer tasks. Each student completed the pegboard and cup drop tasks twice, first with their dominant hand and then with their nondominant hand. The rope task was performed only once. The time to complete each task was recorded by a laboratory timer, and the number of errors that occurred was also recorded
(dropping rings for the pegboard task, dropping beans for the cup drop task, and touching unmarked segments of rope during the rope task). All subjects who participated in both the LapSim and box trainer tasks completed the box trainer tasks in the order of pegboard task, cup drop task, and rope task before completing LapSim modules for coordination, lifting and grasping, and handling intestines, respectively.
Statistical analysis Because each task had a unique mean value for performance variables, bivariate correlations were calculated with the Pearson’s correlation coefficient. Twotailed probability values of ⬍.05 were considered statistically significant for the present study. Data were analyzed with the use of the Statistical Package for the Social Sciences (version 14.0; SPSS Inc, Chicago, IL).
R ESULTS Mean values and SDs for key performance variables are presented in Table 1, namely the time to complete the LapSim and box trainer tasks and the number of errors made by the study participants. Mean LapSim and box trainer completion times were approximately 70-204 seconds; the mean error scores ranged from about 0.7-7.0. However, the standard SDs ranged from approximately 27-92 and 0.9-5.7, respectively. Table 2 shows Pearson correlation coefficients between a variety of surgical skills variables that were measured during LapSim and box trainer tasks. Several LapSim and box trainer tasks positively correlated with 1 another with regard to completion times and tissue damage scores/number of errors; these correlations are statistically significant. The time to completion, with the dominant hand, for the LapSim coordination task (64.9 ⫾ 37.8 seconds) and the pegboard task (143.8 ⫾ 91.8 seconds) were correlated (r ⫽ 0.507; P ⫽ .006). The grasping and lifting task completion time on LapSim (104.9 ⫾ 34.1 seconds) and the comparative box trainer cup drop task comple-
NOVEMBER 2007 American Journal of Obstetrics & Gynecology
546.e2
Meeting Papers
www.AJOG.org correlated with the total number of errors during all box trainer tasks (5.1 ⫾ 3.9; r ⫽ 0.374; P ⫽ .035. Among LapSim tasks, the average time for grasping and lifting (94.1 ⫾ 27.0 seconds) correlated with the rope passing (203.9 ⫾ 53.7 seconds) and cup drop box trainer exercise times with the nondominant hand (120.8 ⫾ 38.3 seconds; r ⫽ 0.390; P ⫽ .027; and r ⫽ 0.394; P ⫽ .026, respectively). No correlation was observed between the time to complete the handling of intestines (94.1 ⫾ 27.1 seconds) and rope passing tasks (203.9 ⫾ 53.7 seconds; r ⫽ 0.267; P ⫽ .133), although both task simulations were perceived as subjectively similar by the surgical faculty. None of the LapSim tissue damage scores correlated with box trainer times; likewise, none of the LapSim task times were found to correlate with box trainer errors.
TABLE 1
Various times to completion of laparoscopic tasks and number of errors Mean ⴞ SD
Performance variable LapSim tasks
..............................................................................................................................................................................................................................................
Coordination task
..............................................................................................................................................................................................................................................
5.61 ⫾ 5.37
Tissue damage score
..............................................................................................................................................................................................................................................
69.93 ⫾ 37.82
Completion time, 2nd attempt (sec)
..............................................................................................................................................................................................................................................
Lifting and grasping task
..............................................................................................................................................................................................................................................
Completion time (sec)
94.13 ⫾ 27.14
Tissue damage score
7.26 ⫾ 5.66
.............................................................................................................................................................................................................................................. ..............................................................................................................................................................................................................................................
Completion time, 1st attempt (sec)
104.93 ⫾ 34.11
Handling of the intestines task completion time (sec)
179.10 ⫾ 56.47
.............................................................................................................................................................................................................................................. ..............................................................................................................................................................................................................................................
Box trainer tasks
..............................................................................................................................................................................................................................................
143.83 ⫾ 91.8
Peg board task completion time, dominant hand (sec)
..............................................................................................................................................................................................................................................
Cup drop task
..............................................................................................................................................................................................................................................
120.83 ⫾ 38.3
Completion time, nondominant hand (sec)
..............................................................................................................................................................................................................................................
0.76 ⫾ 0.86
Errors, nondominant hand
..............................................................................................................................................................................................................................................
Rope task
..............................................................................................................................................................................................................................................
203.89 ⫾ 53.8
Completion time (sec)
..............................................................................................................................................................................................................................................
Errors
0.73 ⫾ 1.25
All box trainer errors
5.11 ⫾ 3.94
C OMMENT
..............................................................................................................................................................................................................................................
tion time (120.8 ⫾ 38.3 seconds) also correlated (r ⫽ 0.404; P ⫽ .022). When errors were accounted for, the LapSim coordination task tissue damage
score (5.6 ⫾ 5.4) was correlated with the sum of all box trainer errors (5.1 ⫾ 3.9; r ⫽ 0.353; P ⫽ .040). The average grasping and lifting tissue damage (7.3 ⫾ 5.7) also
Several LapSim performance measures were found to correlate positively with box trainer task completion times and the number of errors made, which suggests that each device is able to measure similar underlying principles that account for laparoscopic surgical skill. Spe-
TABLE 2
Pearson correlation coefficients (r) between LapSim and box trainer measures of laparoscopic surgical skills
LapSim
Peg board task completion time: dominant hand
Cup drop task: nondominant hand Completion time
Rope task Errors
Completion time
Errors
Total box trainer errors
Coordination task
................................................................................................................................................................................................................................................................................................................................................................................
Tissue damage score
0.114
0.003
⫺0.076
0.181
Completion time, 2nd attempt
0.507
⫺0.232
⫺0.137
⫺0.093
0.424*
0.353*
0.037
0.054
................................................................................................................................................................................................................................................................................................................................................................................ †
................................................................................................................................................................................................................................................................................................................................................................................
Lifting and grasping task
................................................................................................................................................................................................................................................................................................................................................................................
Completion time
⫺0.281
0.394*
Tissue damage score
⫺0.142
0.388*
Completion time, 1st attempt
⫺0.274
0.404*
Handling of intestines task completion time
⫺0.044
⫺0.174
0.390*
⫺0.060
⫺0.213
................................................................................................................................................................................................................................................................................................................................................................................
0.382*
0.274
0.262
0.297
⫺0.039
0.374*
................................................................................................................................................................................................................................................................................................................................................................................
⫺0.028
⫺0.083
................................................................................................................................................................................................................................................................................................................................................................................
0.191
⫺0.160
0.267
⫺0.137
⫺0.169
................................................................................................................................................................................................................................................................................................................................................................................
* P ⬍ .05. †
P ⬍ .005.
546.e3
American Journal of Obstetrics & Gynecology NOVEMBER 2007
Meeting Papers
www.AJOG.org cifically, LapSim coordination task times correlated with pegboard times and errors and the total number of errors that were counted among all box trainer tasks. The LapSim lifting and grasping module performance correlated to the cup drop task performance, because both required picking up an object and manipulating it in time and space with both hands. Interestingly, as the lifting/ grasping damage score increased, so did the amount of time that was needed to complete the cup drop box trainer task. Thus, participants who were found to take more time completing the cup drop task also created more tissue damage during the lifting/grasping module. This could explain the fact that students get stressed when mistakes are made and cause surgical tasks to take more time or that students with less laparoscopic skill take longer to complete tasks and are more susceptible to making errors. Face validity would dictate that the handling intestines and rope transfer tasks are essentially the same, because both require picking up a long, thin, rope-like structure and passing it segment by segment from 1 hand to the other. To our surprise, these tasks did not correlate. This could be explained by the fact that participants performed the rope transfer task only once and that the handling intestines task was the last task to be performed, thus students may have learned to manipulate the laparoscopic equipment by the end of the study. Also, the handling intestines task was repeated until a passing score was achieved, and the rope transfer task was performed only once. The present study has several limitations that may have impacted our results. Our study used third-year medical students as participants with minimal or no
experience using laparoscopic equipment (novices); therefore, their laparoscopic surgical skill demonstration was that of innate ability. However, as participants completed more and more tasks, some may have learned more efficiently how to perform better than other students, which is a factor that was not explored. In addition, as the students completed more tasks, first with the box trainer then with LapSim, they became more comfortable with the equipment, leading to improved performance towards the end of the study. In addition participant performance on LapSim tasks may have been enhanced because of completion of the box trainer tasks moments before. The sample size was small, and there is significant variation around our mean values, thus making our statistical analysis less robust to hypothesis testing. Because of the small sample size, we could not analyze the relationship of demographic data to surgical skills. During the box trainer tasks, completion times and error counting was done manually by third-year medical students, which subjected the data collection to human error, whereas the LapSim data were recorded by the computer system. If we were to perform a future study on the basis of this research, perhaps with a larger sample size, studying experienced surgeons or allowing participants to practice before measuring their performance would allow for a more accurate comparison between the LapSim and box trainer tasks. Future research in the area of surgical simulation and laparoscopic skills should evaluate which method (ie, virtual reality vs box trainers) of simulation is the most time- and cost-effective in the measurement and teaching of laparoscopic surgical skills. Virtual reality sim-
ulators such as LapSim are expensive; perhaps the same benefits can be achieved with box trainers, which are relatively inexpensive. Box trainers are priced at about $5000.00 for the whole unit, which includes the video monitor, light source and camera; the price for virtual reality simulators can range from $40,000-$100,000, depending on the number of modules one chooses and not including maintenance costs and upgrades. Unquestionably, simulation will continue to grow in medicine, as it has for airline pilots and military and police personnel. Research in laparoscopic surgical education with the use of virtual reality simulation must be validated by transfer of skills, and the need for cost-effective use of technology to teach these skills in the laboratory should be emphasized. f REFERENCES 1. Powers TW, Murayama KM, Toyama M, et al. Housestaff performance is improved by participation in a laparoscopic skills curriculum. Am J Surg 2002;184:626-30. 2. Aggarwal R, Grantcharov TP, Erikson JR, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244:310-4. 3. Torkington J, Smith SGT, Rees BI, Darzil A. Skill transfer from virtual reality to a real laparoscopic task. Surg Endosc 2001;15:1076-9. 4. Sidhu RS, Grober ED, Musselman LJ, Reznick RK. Assessing competency in surgery: where to begin? Surgery 2004;135:6-20. 5. Hammond L, Ketchum J, Schwartz BF. Accreditation Council on Graduate Medical Education Technical Skills Competency Compliance: urologic surgical skills. J Am Coll Surg 2005;201:454-7. 6. Cosman PH, Cregan PC, Martin CJ, Cartmill JA. Virtual reality simulators: current status in acquisition and assessment of surgical skills. ANZ J Surg 2002;72:30-4. 7. Fried GM, Feldman LS, Vassiliou MC, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg 2004;240:518-28.
NOVEMBER 2007 American Journal of Obstetrics & Gynecology
546.e4
www. AJOG.org
Correlating virtual reality and box trainer tasks in the assessment of laparoscopic surgical skills Jordan Newmark, MD; Vani Dandolu, MD; Richard Milner, BS; Harsh Grewal, MD; Sean Harbison, MD; Enrique Hernandez, MD OBJECTIVE: The purpose of this study was to examine the correlation in the assessment of laparoscopic surgical skills in medical students with the use of a virtual reality laparoscopic trainer and a low-fidelity video box trainer with comparative tasks. STUDY DESIGN: Third-year medical students were asked to perform 3
basic skills set modules on LapSim (Surgical Science, Gothenburg, Sweden): coordination, grasping and lifting, and handling the intestines. Each task was set at the easiest level, and each student was allowed a maximum of 10 attempts to complete each task. Similarappearing tasks were chosen for comparison with the use of a standard video box trainer: pegboard, cup drop and rope pass, respectively. Laparoscopic skills were evaluated with the use of both trainers during 1 session. Pearson’s correlation coefficients were used to compare paired data on each student using statistical software. RESULTS: Forty-seven of 65 medical students were assigned to clini-
cal clerkships on-campus at Temple University School of Medicine participated in the study. All 47 students participated in the video box trainer tasks; 34 students completed both the video box trainer and
LapSim skills set. Observations that were obtained on the LapSim virtual reality system and video box trainer simulator demonstrated several correlations. The time to completion for the LapSim coordination task and the pegboard task were correlated (r ⫽ 0.507; P ⫽ .006), as were the grasping and lifting task completion time on LapSim and the comparative box trainer cup drop task completion time (r ⫽ 0.404; P ⫽ .022). When accounting for errors, the LapSim coordination task tissue damage score was correlated with the sum of all box trainer errors (r ⫽ 0.353; P ⫽ .040); the average grasping and lifting tissue damage was correlated with the total number of errors during all box trainer tasks (r ⫽ 0.374; P ⫽ .035). CONCLUSION: Overall, in evaluating laparoscopic skills, the LapSim and video box trainer were correlated positively with one another. The scoring of laparoscopic skills by both systems appears to be equivalent for the measurement of time to task completion and number of errors.
Key words: box trainer, virtual reality simulator
Cite this article as: Newmark J, Dandolu V, Milner R, Grewal H, Harbison S, Hernandez E. Correlating virtual reality and box trainer tasks in the assessment of laparoscopic surgical skills. Am J Obstet Gynecol 2007;197:546.e1-546.e4.
A
s surgical techniques become more advanced, especially in the area of minimally invasive surgery, 3-dimensional open surgical fields are being replaced by 2-dimensional video images.1 It is important for surgeons and surgical residents to incorporate these developing technologies into skill acquisition and practice improvement in an effective
manner, especially in this new “endoscopic era.”2 Traditional video box trainers and virtual reality simulators have allowed for faster acquisition of the endoscopic technical skills ex-vivo before clinical practice.2,3 Although simulators can never replicate reality completely, they are valuable in recreating important aspects of life-like surgical sit-
From the Departments of Obstetrics and Gynecology (Drs Dandolu and Hernandez) and Surgery (Drs Grewal and Harbison) and the research laboratory (Mr Milner), Temple University School of Medicine, Philadelphia, PA. Dr Newmark, a medical student at Temple University at the time of writing, is currently a resident in anesthesiology at Harvard Medical School. Presented at the annual meeting of the Council on Resident Education in Obstetrics and Gynecology and the Association of Professors of Gynecology and Obstetrics, Salt Lake City, UT, March 7-10, 2007. Received Mar. 16, 2007; revised May 27, 2007; accepted Jul. 23, 2007. Reprints: Vani Dandolu, MD, 3401 N Broad St, Temple University Hospital, Philadelphia, PA 19140: [email protected]. 0002-9378/$32.00 © 2007 Mosby, Inc. All rights reserved. doi: 10.1016/j.ajog.2007.07.026
546.e1
American Journal of Obstetrics & Gynecology NOVEMBER 2007
uations.1 Therefore, surgical simulation has become increasingly widespread and important in the teaching of surgical skills and has the potential to be a valuable tool for accomplishing certain Accreditation Council for Graduate Medical Education (ACGME) competency goals and trainee skills evaluation of surgical house staff in surgical specialties.4,5 Several laparoscopic surgery simulators exist and include computerized virtual reality simulators and traditional video box trainers.6 Although these 2 modalities may appear similar, each has its own unique advantages and disadvantages. Virtual reality simulators provide fast and precise results on many measures of skill, but the validity of these measurements is unproven. Assessment scores may not duplicate performance in the operating room; in 1 study, these simulators were unable to measure surgical errors in a valid fashion.2 Virtual reality simulators are also very expen-
Meeting Papers
www.AJOG.org sive; many cost in excess of $100,000, lack portability, and require ongoing technical support and system “upgrades.”4 Video box trainers, on the other hand, often more closely approximate real life surgical conditions and are considerably less expensive. These trainers, however, cannot automatically provide metrics for performance variables as virtual reality systems can. Trained surgical staff members are needed to evaluate skill parameters in a box trainer; therefore, metrics for a box trainer are subject to inter- and intraobserver variation. Much research in recent years has focused on translating demonstrated surgical skills in different simulation scenarios into improved patient safety and outcomes.7 It was the aim of this study to correlate the results of measuring laparoscopic surgical skills with the use of virtual reality and box trainer tasks and to compare their ability to assess similar endoscopic skills.
M ETHODS Subjects Forty-seven third-year medical students at a large university were recruited; all of the students had minimal or no experience using laparoscopic equipment. Thirty-four of the 47 students completed 3 different virtual reality basic skills modules and 3 different video box trainer tasks; all 47 students completed laparoscopic skills on the box trainer alone. The surgical simulation center at Temple University School of Medicine has 4 box trainers with video monitors attached; the LapSim unit (Surgical Science, Gothenburg, Sweden) was obtained from Immersion Medical (Gathersburg, MD) for a 2-week time period for the purpose of the study. There were technical difficulties with use of the LapSim during the initial 2 days; therefore, there is a discrepancy in the number of students who were assessed on the box trainer vs LapSim. Institutional review board approval was obtained before subject recruitment, and all participants signed an informed consent form before participation.
Surgical tasks Virtual reality tasks were demonstrated with the LapSim computerized laparoscopic surgery simulator. Three basic training modules were set at the easy level for the present study. Tasks set at the easy level have shown construct validity for all measured variables.2 Students were asked to complete the following basic LapSim tasks: coordination, lifting and grasping, and handling intestines with the use of 2 grasper handles, with artificial trocars built into LapSim. Study participants, during 1 session, repeated each task until a passing score (ⱖ70 on a 1-100 scale, scored by LapSim) was achieved. The coordination task entailed the use of alternating hands to navigate a laparoscopic camera and to touch colored balls with a probe, placing them into a target without causing damage to nearby healthy tissues. The lifting and grasping module involved the use of 1 hand to lift a tile within the field with a probe while the other hand grasped a needle and placed it under the tile, simultaneously trying to avoid damaging tissue walls. Last, the handling intestines task required students to pass a 10-cm length of small bowel from 1 hand to the other without puncturing the bowel or causing local tissue damage. The box trainer was constructed of a plastic bin that was covered by a surgical cloth with 4 trocar ports, 2 ports on each side. The bin contained the components that were needed to complete the surgical tasks. Real surgical instruments (such as graspers and laparoscopic cameras) were the tools provided; the camera was linked to a color monitor similar to those used in the operating room for laparoscopic cases. Moving rings on a pegboard, grasping beans (n ⫽ 10) and dropping them through a hole in an inverted specimen cup, and passing a length of a rope hand-to-hand were the box trainer tasks. Each student completed the pegboard and cup drop tasks twice, first with their dominant hand and then with their nondominant hand. The rope task was performed only once. The time to complete each task was recorded by a laboratory timer, and the number of errors that occurred was also recorded
(dropping rings for the pegboard task, dropping beans for the cup drop task, and touching unmarked segments of rope during the rope task). All subjects who participated in both the LapSim and box trainer tasks completed the box trainer tasks in the order of pegboard task, cup drop task, and rope task before completing LapSim modules for coordination, lifting and grasping, and handling intestines, respectively.
Statistical analysis Because each task had a unique mean value for performance variables, bivariate correlations were calculated with the Pearson’s correlation coefficient. Twotailed probability values of ⬍.05 were considered statistically significant for the present study. Data were analyzed with the use of the Statistical Package for the Social Sciences (version 14.0; SPSS Inc, Chicago, IL).
R ESULTS Mean values and SDs for key performance variables are presented in Table 1, namely the time to complete the LapSim and box trainer tasks and the number of errors made by the study participants. Mean LapSim and box trainer completion times were approximately 70-204 seconds; the mean error scores ranged from about 0.7-7.0. However, the standard SDs ranged from approximately 27-92 and 0.9-5.7, respectively. Table 2 shows Pearson correlation coefficients between a variety of surgical skills variables that were measured during LapSim and box trainer tasks. Several LapSim and box trainer tasks positively correlated with 1 another with regard to completion times and tissue damage scores/number of errors; these correlations are statistically significant. The time to completion, with the dominant hand, for the LapSim coordination task (64.9 ⫾ 37.8 seconds) and the pegboard task (143.8 ⫾ 91.8 seconds) were correlated (r ⫽ 0.507; P ⫽ .006). The grasping and lifting task completion time on LapSim (104.9 ⫾ 34.1 seconds) and the comparative box trainer cup drop task comple-
NOVEMBER 2007 American Journal of Obstetrics & Gynecology
546.e2
Meeting Papers
www.AJOG.org correlated with the total number of errors during all box trainer tasks (5.1 ⫾ 3.9; r ⫽ 0.374; P ⫽ .035. Among LapSim tasks, the average time for grasping and lifting (94.1 ⫾ 27.0 seconds) correlated with the rope passing (203.9 ⫾ 53.7 seconds) and cup drop box trainer exercise times with the nondominant hand (120.8 ⫾ 38.3 seconds; r ⫽ 0.390; P ⫽ .027; and r ⫽ 0.394; P ⫽ .026, respectively). No correlation was observed between the time to complete the handling of intestines (94.1 ⫾ 27.1 seconds) and rope passing tasks (203.9 ⫾ 53.7 seconds; r ⫽ 0.267; P ⫽ .133), although both task simulations were perceived as subjectively similar by the surgical faculty. None of the LapSim tissue damage scores correlated with box trainer times; likewise, none of the LapSim task times were found to correlate with box trainer errors.
TABLE 1
Various times to completion of laparoscopic tasks and number of errors Mean ⴞ SD
Performance variable LapSim tasks
..............................................................................................................................................................................................................................................
Coordination task
..............................................................................................................................................................................................................................................
5.61 ⫾ 5.37
Tissue damage score
..............................................................................................................................................................................................................................................
69.93 ⫾ 37.82
Completion time, 2nd attempt (sec)
..............................................................................................................................................................................................................................................
Lifting and grasping task
..............................................................................................................................................................................................................................................
Completion time (sec)
94.13 ⫾ 27.14
Tissue damage score
7.26 ⫾ 5.66
.............................................................................................................................................................................................................................................. ..............................................................................................................................................................................................................................................
Completion time, 1st attempt (sec)
104.93 ⫾ 34.11
Handling of the intestines task completion time (sec)
179.10 ⫾ 56.47
.............................................................................................................................................................................................................................................. ..............................................................................................................................................................................................................................................
Box trainer tasks
..............................................................................................................................................................................................................................................
143.83 ⫾ 91.8
Peg board task completion time, dominant hand (sec)
..............................................................................................................................................................................................................................................
Cup drop task
..............................................................................................................................................................................................................................................
120.83 ⫾ 38.3
Completion time, nondominant hand (sec)
..............................................................................................................................................................................................................................................
0.76 ⫾ 0.86
Errors, nondominant hand
..............................................................................................................................................................................................................................................
Rope task
..............................................................................................................................................................................................................................................
203.89 ⫾ 53.8
Completion time (sec)
..............................................................................................................................................................................................................................................
Errors
0.73 ⫾ 1.25
All box trainer errors
5.11 ⫾ 3.94
C OMMENT
..............................................................................................................................................................................................................................................
tion time (120.8 ⫾ 38.3 seconds) also correlated (r ⫽ 0.404; P ⫽ .022). When errors were accounted for, the LapSim coordination task tissue damage
score (5.6 ⫾ 5.4) was correlated with the sum of all box trainer errors (5.1 ⫾ 3.9; r ⫽ 0.353; P ⫽ .040). The average grasping and lifting tissue damage (7.3 ⫾ 5.7) also
Several LapSim performance measures were found to correlate positively with box trainer task completion times and the number of errors made, which suggests that each device is able to measure similar underlying principles that account for laparoscopic surgical skill. Spe-
TABLE 2
Pearson correlation coefficients (r) between LapSim and box trainer measures of laparoscopic surgical skills
LapSim
Peg board task completion time: dominant hand
Cup drop task: nondominant hand Completion time
Rope task Errors
Completion time
Errors
Total box trainer errors
Coordination task
................................................................................................................................................................................................................................................................................................................................................................................
Tissue damage score
0.114
0.003
⫺0.076
0.181
Completion time, 2nd attempt
0.507
⫺0.232
⫺0.137
⫺0.093
0.424*
0.353*
0.037
0.054
................................................................................................................................................................................................................................................................................................................................................................................ †
................................................................................................................................................................................................................................................................................................................................................................................
Lifting and grasping task
................................................................................................................................................................................................................................................................................................................................................................................
Completion time
⫺0.281
0.394*
Tissue damage score
⫺0.142
0.388*
Completion time, 1st attempt
⫺0.274
0.404*
Handling of intestines task completion time
⫺0.044
⫺0.174
0.390*
⫺0.060
⫺0.213
................................................................................................................................................................................................................................................................................................................................................................................
0.382*
0.274
0.262
0.297
⫺0.039
0.374*
................................................................................................................................................................................................................................................................................................................................................................................
⫺0.028
⫺0.083
................................................................................................................................................................................................................................................................................................................................................................................
0.191
⫺0.160
0.267
⫺0.137
⫺0.169
................................................................................................................................................................................................................................................................................................................................................................................
* P ⬍ .05. †
P ⬍ .005.
546.e3
American Journal of Obstetrics & Gynecology NOVEMBER 2007
Meeting Papers
www.AJOG.org cifically, LapSim coordination task times correlated with pegboard times and errors and the total number of errors that were counted among all box trainer tasks. The LapSim lifting and grasping module performance correlated to the cup drop task performance, because both required picking up an object and manipulating it in time and space with both hands. Interestingly, as the lifting/ grasping damage score increased, so did the amount of time that was needed to complete the cup drop box trainer task. Thus, participants who were found to take more time completing the cup drop task also created more tissue damage during the lifting/grasping module. This could explain the fact that students get stressed when mistakes are made and cause surgical tasks to take more time or that students with less laparoscopic skill take longer to complete tasks and are more susceptible to making errors. Face validity would dictate that the handling intestines and rope transfer tasks are essentially the same, because both require picking up a long, thin, rope-like structure and passing it segment by segment from 1 hand to the other. To our surprise, these tasks did not correlate. This could be explained by the fact that participants performed the rope transfer task only once and that the handling intestines task was the last task to be performed, thus students may have learned to manipulate the laparoscopic equipment by the end of the study. Also, the handling intestines task was repeated until a passing score was achieved, and the rope transfer task was performed only once. The present study has several limitations that may have impacted our results. Our study used third-year medical students as participants with minimal or no
experience using laparoscopic equipment (novices); therefore, their laparoscopic surgical skill demonstration was that of innate ability. However, as participants completed more and more tasks, some may have learned more efficiently how to perform better than other students, which is a factor that was not explored. In addition, as the students completed more tasks, first with the box trainer then with LapSim, they became more comfortable with the equipment, leading to improved performance towards the end of the study. In addition participant performance on LapSim tasks may have been enhanced because of completion of the box trainer tasks moments before. The sample size was small, and there is significant variation around our mean values, thus making our statistical analysis less robust to hypothesis testing. Because of the small sample size, we could not analyze the relationship of demographic data to surgical skills. During the box trainer tasks, completion times and error counting was done manually by third-year medical students, which subjected the data collection to human error, whereas the LapSim data were recorded by the computer system. If we were to perform a future study on the basis of this research, perhaps with a larger sample size, studying experienced surgeons or allowing participants to practice before measuring their performance would allow for a more accurate comparison between the LapSim and box trainer tasks. Future research in the area of surgical simulation and laparoscopic skills should evaluate which method (ie, virtual reality vs box trainers) of simulation is the most time- and cost-effective in the measurement and teaching of laparoscopic surgical skills. Virtual reality sim-
ulators such as LapSim are expensive; perhaps the same benefits can be achieved with box trainers, which are relatively inexpensive. Box trainers are priced at about $5000.00 for the whole unit, which includes the video monitor, light source and camera; the price for virtual reality simulators can range from $40,000-$100,000, depending on the number of modules one chooses and not including maintenance costs and upgrades. Unquestionably, simulation will continue to grow in medicine, as it has for airline pilots and military and police personnel. Research in laparoscopic surgical education with the use of virtual reality simulation must be validated by transfer of skills, and the need for cost-effective use of technology to teach these skills in the laboratory should be emphasized. f REFERENCES 1. Powers TW, Murayama KM, Toyama M, et al. Housestaff performance is improved by participation in a laparoscopic skills curriculum. Am J Surg 2002;184:626-30. 2. Aggarwal R, Grantcharov TP, Erikson JR, et al. An evidence-based virtual reality training program for novice laparoscopic surgeons. Ann Surg 2006;244:310-4. 3. Torkington J, Smith SGT, Rees BI, Darzil A. Skill transfer from virtual reality to a real laparoscopic task. Surg Endosc 2001;15:1076-9. 4. Sidhu RS, Grober ED, Musselman LJ, Reznick RK. Assessing competency in surgery: where to begin? Surgery 2004;135:6-20. 5. Hammond L, Ketchum J, Schwartz BF. Accreditation Council on Graduate Medical Education Technical Skills Competency Compliance: urologic surgical skills. J Am Coll Surg 2005;201:454-7. 6. Cosman PH, Cregan PC, Martin CJ, Cartmill JA. Virtual reality simulators: current status in acquisition and assessment of surgical skills. ANZ J Surg 2002;72:30-4. 7. Fried GM, Feldman LS, Vassiliou MC, et al. Proving the value of simulation in laparoscopic surgery. Ann Surg 2004;240:518-28.
NOVEMBER 2007 American Journal of Obstetrics & Gynecology
546.e4