Virtual reality arthroscopy training simulator

Virtual reality arthroscopy training simulator

Comput. Pergamon Vol. 25. No. 2, Pp. 193-203, 1995 Copyright 0 1995 Elsevier Science Ltd Printed in Great Britain. All rights reserved OOlo-4825/95 ...

3MB Sizes 0 Downloads 101 Views

Comput.

Pergamon

Vol. 25. No. 2, Pp. 193-203, 1995 Copyright 0 1995 Elsevier Science Ltd Printed in Great Britain. All rights reserved OOlo-4825/95 $9.50+0.00

Biol. Med.

OOlO-4825(94)00038-7

VIRTUAL

REALITY

ARTHROSCOPY SIMULATOR

TRAINING

ROLF ZIEGLER, GEORG FISCHER, WOLFGANG MILLER and MARTIN G~~BEL Fraunhofer Institute for Computer Graphics, Department of Visualization and Simulation, Wilhelminenstr. 7, D-64283 Darmstadt, Germany (Received

21 September

1994; received

for publication

19 October

1994)

Abstract-This paper describes the result of the interdisciplinary cooperation of traumatologists of the Berufsgenossenschaftliche Unfallklinik (BGU) in Frankfurt am Main and a team of computer graphics scientists of the Fraunhofer Institute for Computer Graphics in Darmstadt. We have developed a highly interactive training simulator system by means of computer graphics and virtual reality techniques. VR training simulator Virtual reality (VR) in medicine Arthroscopy training simulator medicine

Computer graphics in

1. INTRODUCTION Arthroscopy has already become an irreplaceable method in diagnostics to recognize pathological changes and diseases, especially of knee joints. Traditionally, training for arthroscopic examinations is done by observing an experienced surgeon in the operating theatre before the trainee uses the arthroscope for the first time. To reinforce the surgical training, standard training systems (e.g. a synthetic replica of the knee joint) are used. The main shortcoming criticized by the surgeons is the insensitiveness of the plastic replica with regard to incorrect handling of the instruments. However, with training on synthetic knees only, the first surgical operation on the human knee is very critical. Computer simulation applying virtual reality (VR) techniques [l] offers an effective alternative for training and establishing arthroscopic techniques. This paper describes the result of the interdisciplinary cooperation of traumatologists of the Berufsgenossenschaftliche Unfallklinik (BGU) in Frankfurt am Main and a team of computer graphics scientists of the Fraunhofer Institute for Computer Graphics in Darmstadt. We analysed and discussed videos and observations of arthroscopic examinations in the operating theatre to specify the requirements for the simulator. As a result, we have developed a highly interactive training simulator system by means of computer graphics and virtual reality techniques. Our work gained from the excellent cooperation of the surgeons and radiologists to answer all our questions concerning the anatomical structure and biomechanics of the knee joint, the arthroscopy in general, and the interpretation of magnetic resonance (MR) images. This paper is structured in several parts. Section 2 describes requirements demanded by the BGU. Then, the two main development tasks are discussed in Sections 3 and 4: the 3D reconstruction of the knee joint and the 3D interaction tool. The presentation of the prototype and the feedback from the trainees are outlined in Section 5. We conclude with a discussion of the assessment and introduce our current and future work towards providing a perfect training simulator. The prototype of the VR arthroscopy training simulator was developed in only 4 months and was presented, tested and assessed during an arthroscopy training course at the “Frankfurt Sports Medicine Weekend” on 4 and 5 December 1993.

R. ZIEGLER

194

et al.

2. REQUIREMENTS Since minimally invasive surgery in general, and arthroscopical diagnosis in particular, are increasing continually, surgical training is becoming an important issue [2]. The surgeons from the BGU demand the inclusion of arthroscopy into the catalogue of training of traumatologists. Trainees of traumatherapy should have performed a specified amount of arthroscopies; hence, there is a great need for qualified training simulator systems. The surgeons of the BGU had concrete ideas concerning the features of the simulator. The main intention of the project was to develop a training simulator as an enhanced alternative to conventional training systems (i.e. a plastic replica of the knee joint). Here, a first prototype is described and the capabilities of computer graphics and virtual reality techniques are demonstrated. The following requirement list states our development priorities: l realistic three-dimensional (3D) representation of the knee joint with all relevant anatomical parts; l highly interactive system by means of VR techniques, including intuitive handling of the instruments; l real-time simulation with seamless image generation; 0 a functional user interface; l choice of different training situations (e.g. varus/valgus deformity). Section 3 describes the process of generating the 3D representation in detail, and Section 4 deals with the remaining issues, and the interaction with the training simulator system. 3. THE

VIRTUAL

KNEE

JOINT

For teaching athroscopic procedures one representation of any knee joint was necessary, which can be examined via the VR arthroscopy simulator. For instance, this could be realized by modelling a knee joint with the help of a modelling system, which takes a long time. Thus, a concept for creating a 3D representation semi-automatically was developed. This concept, based upon data of tomographic imaging modalities, is not limited to the generation of a specific virtual knee joint. Our representation of the knee joint has been created from a MRI (magnetic resonance imaging) data set. The reconstruction process was divided into seven principal steps, starting with the data acquisition and ending with the integration of the 3D representation in the VR arthroscopy simulator (see Fig. 1) [3]. This process (or parts of it) is adaptable to other applications, i.e. for simulation and visualization of respiratory airflow in the human nose [4]. The original data used in this project were a series of knee joint MR images taken with equidistant steps along the sagittal axis. The data set consisted of 64 slices, each with a resolution of 512 x 512 pixels and a slice interval of 1.9 mm. In the first step, the original MR images were re-created using image processing methods in order to accentuate the contrast between anatomical structures and the surrounding tissue. Next, segmentation methods were applied to the preprocessed images to isolate the different anatomical structures of the knee joint and to determine the contour points of each obejct. Even after performing contrast enhancement methods the edges between different tissues were not well defined in the available MRI data set. Therefore, manual intervention was required. The interslice distance in the original image sequence was much larger than the pixel size. As a consequence, interpolation between segmented slices was necessary in order to obtain an isotropic data set with equal spatial resolution in all three dimensions. An interpolation technique similar to the shape-based interpolation was applied [5]. The intermediate slices were estimated using the grey-level information in the adjacent slices under consideration of the pixel’s distance to the nearest point on the boundary of the anatomic object. For instance, the complete MRI data set interpolated via this technique

Arthroscopy

training

simulator

195

interpolation

a iii

surface reconstraction

Fig.

1. Reconstruction

process

would be transformed into a voxel representation consisting of above 90000 000 unit-sized voxels (Voxels are volume elements, similar to the pixel in 2D). At this stage of the reconstruction process, a specific isotropic data set was generated for each relevant structure, on which the subsequent surface reconstruction was based. A boundary representation of the object surface based on triangles was used. The surfaces of each object were constructed by applying the marching cubes algorithm to the segmented data [6]. This surface polygonization method generates triangles that approximate the isosurface of an object within a volume. After specification of a threshold each cube, defined by eight sampled values along the three axes, is traversed in order to find whether its corner values straddled the threshold. If tested successfully, interpolation is used to calculate the triangle vertices, where the edges are intersected by the isosurface. Up to four triangles can be generated in the interior of each cube representing the approximation of the isosurface. One problem arose in this triangulation step: owing to the high resolution of medical data sets a huge number of triangles was produced. Figure 2 shows the representation of the knee joint created from MRI data using the marching cubes triangulation technique. The surface of the relevant anatomic structures consisted of about 300000 triangles. For that reason, the system’s performance and interactivity was decreased critically. To solve this problem, in an additional processing step a reduction of the complex geometry was necessary. The goal of the decimation algorithm was to reduce the total number of triangles, while forming a good geometric approximation. Hence, a distance to plane criterion, which was under user control, was used evaluating the importance of a vertex according to its contribution to the detail [7]. By this decimation method surface models at several levels of detail could be created from a polygonal description of a given object. Finally, this decimation algorithm has been successfully applied to isosurface representations derived from volume data. We have achieved acceptable performance for the virtual knee joint consisting of nearly 20 000 polygons. Figure 3(a) shows the result of a

196

R. ZIEGLER

et al.

94% decimation. The blocky appearance of the simplified representation using a smooth shading technique instead of flat shading; see Fig. 3(b). 4. SIMULATION

OF THE

was reduced

ARTHROSCOPY

The arthroscopy in the operating theatre is divided into two parts. The surgeon pilots the instruments inserted in the human knee and gets visual feedback of their position from a monitor. The arthroscopy training simulator is implemented appropriately. The piloting of the instruments is realized as the 3D interaction, and the monitoring on the screen is implemented as the 2D user interface. In addition to the display area a functional range has been integrated (see Fig. 4) [S]. The user interacts with the system via a 2D user interface, implemented by means of OSFIMotif, an X-based window system. To allow the user to specify parameters and read status information while doing the examination, the functional area is necessary. The trainee can select the angle of the arthroscope optic to make a better demonstration or to make the first exploration easier, and can also select to examine either the left or the right knee. Therefore, the trainee can practise piloting the instruments, arthroscope and exploratory probe, in both hands. One advantage over conventional training systems is the possibility of verifying the training progress. The simulation of an arthroscopy can be recorded and played back afterwards. To activate the implemented functionality state-of-the-art interaction techniques are integrated: each function can be selected by using menus or pop-up menus with the mouse or the keyboard. The 3D interaction of the system simulates a real arthroscopy. An original exploratory probe and a replica of an arthroscope are inserted into a synthetic model of the knee through two small incisions located underneath the patella (see Fig. 5). As in common virtual reality applications [9], the simulator uses the tracking technique. Tracking [lo] means monitoring the sensors in a 3D electromagnetic field, in order to get the position and orientation of the instruments and of the tibia. The electronic unit modifies the values depending on the adjustment of the system, and transmits them via a serial interface to the computer. The tracking system used in the arthroscopy simulator consists of three tracker units, one transmitter and one electronic unit. One tracker is fixed on the exploratory probe, one on the arthroscope, and one on the tibia. The transmitter is located underneath the femur. The tracker reports continuously on their current position, with an accuracy of 0.8 mm for the position and 0.15” for the orientation. The transmission rate of the tracking system is 40 cycles per sec. If the received values are greater than a specified threshold, the display of the virtual knee joint is refreshed. The frame rate on a SGI Indigo Extreme graphics workstation is lo-16 images per sec. 5. PRESENTATION The prototype was presented at the “Frankfurt Sports Medicine Weekend” on 4 and 5 December 1994. During the weekend an arthroscopy course for surgeons took place (see Fig. 6), comprising talks and practices on conventional training systems and the training simulator. 5.1. Hardware The system was installed on a Silicon Graphics 310 Reality Engine workstation. The Polhemus 3Space FASTRAK tracking system with three trackers was connected via a serial interface cable to the workstation. The three trackers were fixed on the instruments, as described in Section 4. 5.2. Software The whole software, including the simulator software and the driver software to communicate with the FASTRACK system, was developed at our department. The

Arthroscopy

Fig.

Fig.

3. Ninety-four

2. Full resolution

training

(about

simulator

300 000 flat shaded

per cent decimation: (a) about 20 000 flat shaded smooth shaded triangles.

triangles).

triangles;

(b) about

20 000

R. ZIEGLER

Fig.

et al.

4. User interface.

transmitter

dip-switch

exploratory probe 1 I tibiatracker probe tracker plastic replica of a knee Fig.

5. Hardware

configuration

of the VR

trackerbox

I replica atiroskogtracker arthroscopy

Ofthe

simulator

NthrOSkOp

Arthroscopy

Fig.

training

6. Presentation

simulator

in Frankfurt.

199

201

Arthroscopy training simulator

software is implemented in “C” using X/Motif Graphics) in an UNIX environment.

and GL (Graphics Library from Silicon

5.3. Course notes More than 40 traumatologists, orthopaedics, students, etc., participated in the arthroscopy course. The practical training was carried out on five conventional training systems and on the training simulator. To obtain feedback from each participant the training simulator was by means of a questionnaire, referring to: 0 realistic appearance; 0 visual impression; l suitability for the training of the 2-axis coordination (monitor-instruments); l control of success; l handling; and 0 overall impression. The result was very satisfactory, although some major points were criticized. The generation and display of the images (frame rate less than eight images) was not in a smooth sequence and had “jumps”. This drawback has now been solved. Furthermore, the system did not detect any collision with objects and the trainee sometimes had problems with orientation (“inside” the femur). This effect accumulated owing to the lack of frame rate performance. The frame rate has now been doubled. Thus the collision detection drawback has been decreased, but not entirely solved (see Discussion). The main drawback of the simulator was the omission of resistance, which we call force feedback. This issue was not included on our requirements list, because it is the topic of our current research (see Discussion). Nevertheless, we had very positive assessments. The simulator is well suited for the training of the 2-axis coordination. The tracking system reacts very sensitively to each movement and thus is very good for training to pilot the instruments. With the protocol option it is possible to obtain a “documentation” of the training. 6. DISCUSSION

AND

FUTURE

WORK

Some of the drawbacks criticized during the presentation of the prototype in Frankfurt have already been solved. The generation and displaying of the images is now in realtime. The former version of the system used an inappropriate description of the objects, which has now been into a more effective description that can be processed by the graphics hardware much more quickly. Thus we could achieve a speed-up of at least 2-3fold (16-20 frames per set). Owing to this speed-up we have installed the system on a less expensive workstation: Silicon Graphics Indigo Extreme. This workstation works with a faster RISC processor than the SGI 310 but is not equipped with the fast texture engine (Reality Engine [ll]). Nevertheless, the Indigo Extreme workstation is an appropriate alternative with a high frame rate (lo-16 images per set). We are still working on increasing the performance. We are working on a new object description specific for medical data and will adapt the rendering tools accordingly. A higher speedup makes it possible to specify a more detailed representation of the knee joint in order to get a more realistic impression. An important aspect of the system is the capability to adapt it to additional joints, e.g. the hip joint (surgical planning). In the near future MR images of hip joints will be available and will be integrated into the system. Furthermore, our knee joint representation will be extended by the integration of different pathological findings. The quality of the available data determines the success or failure of all subsequent processes. Therefore, the performance of the reconstruction process can be increased by merging information of various types of medical image sequences (e.g. MRI is well suited to discrimination of soft tissues, CT to identification of bone structure). The obvious limitation of this approach is that the multiple datasets must be registered spatially.

202

R. ZIEGLER

et

al.

6.1. Force feedback

One major topic for the acceptance of the training simulator is the integration of force feedback. In cooperation with the Department of Electromechanical Construction at the Darmstadt Technical University we have completed a conceptual study. Based on the results of this study, we are working on the realization. Several problems can be addressed. For the force feedback system (FFS) we have to determine limits for the acting forces, e.g. maximum leverage or traction force. Another problem is the simulation of the properties of the material. Consequently, the characteristics of the material referring to elasticity and viscosity have to be specified. From these characteristics the FFS will determine the counteracting force that has to be exerted to the instruments. Our current work concentrates on the discussed problems, and the results have been presented at the “MEDICA” fair, 16-19 November 1994, in Dusseldorf. Besides the development of the FFS, we are working on the simulation of arthroscopical surgical techniques (e.g. resection of a patch at the meniscus). The trainee will then be able to select several training scenarios referring to different pathological findings. 7. SUMMARY The result of the interdisciplinary cooperation of traumatologists of the Berufsgenossenschaftliche Unfallklinik (BGU) in Frankfurt am Main and a team of computer graphics scientists of the Fraunhofer Institute for Computer Graphics in Darmstadt was presented. We have developed a highly interactive arthroscopy training simulator system by means of computer graphics and virtual reality techniques. This system offers an effective alternative to conventional training systems for training and establishing of arthroscopic techniques. The 3D interaction of the system simulates a real arthroscopy. An original exploratory probe and a replica of an arthroscope are inserted into a synthetic model of the knee through two small incisions located underneath the patella. The used tracking system monitors the piloting of the instruments sensitively. Thus, training becomes very effective. One additional advantage is the possibility to verify the training progress. The simulation of an arthroscopy can be recorded and thereafter played off. This paper described the requirements demanded by the BGU. The two main development tasks were discussed: the 3D reconstruction of the knee joint and the 3D interaction tool. Furthermore, the presentation of the prototype and the feedback of the trainees was outlined. Finally, the assessment of the prototype was discussed and the current and future work to provide a perfect training simulator was introduced. The prototype of the arthroscopy training simulator was developed in only 4 months and was presented, tested and assessed during an arthroscopy training course at the “Frankfurt Sports Medicine Weekend” on 4 and 5 December 1993. Acknowledgements-We

wish to thank Prof. Dr h.c. Dr-Ing. Jose L. Encarnacao for providing the environment in which this work was possible. We also thank all our colleagues and students at our lab. Without their work we would not have been able to achieve the results presented in this paper. Furthermore, we wish to thank the team from the BGU, Dr Bauer, Dr Soldner, and Jdrg Helberger, for their successful cooperation.

REFERENCES 1. M. Gobel, Ed., Virtual reality, Comput. Graphics 17, 6 (1993). 2. M. R. Satava. Virtual realitv sureical simulator. Sure. EndoscoDv. DD. 203-205. Snrinzer, Berlin (1993). 3. W. Miiller, ‘3D-Rekonstruktion von MRI-Date; zur Modellg~winnung fur’ e&n medizinischdn Ausbildungssimulator, Diploma thesis, Darmstadt Technical University, Department of Computer Science Interactive Graphics Systems Group (1994). 4. T. Friihauf, M. Gobel, H. Haase and K. Karlsson, Design of a flexible monolithic visualization system, Frontiers in Scientific Visualization, R. Earnshaw, Ed. Academic Press, London (1994). 5. G. T. Herman, J. Zheng and C. A. Bucholtz, Shape-based interpolation, IEEE Comput. Graphics Applic. 12, 69-79 (1992). 6. .W. E. Lorensen and H. E. Cline, Marching cubes: a high resolution 3D surface construction algorithm, Comput. Graphics (Proc. SZGGRAPH), 21, 163-169 (1987).

Arthroscopy training simulator

203

7. F. Schroder and P. RoBbach, Managing the Complexity of Digital Terrain Models, Computers & Graphics, Special Issue on “Modelling and Visualisation of Spatial Data in Geographic Information Systems”. Pergamon Press, Oxford (1994). 8. G. Fischer, Erstellung einer 3D-Benutzerschnittstelle fiir einen medizinischen Ausbildungssimulator, Diploma thesis, Darmstadt Technical University, Department of Computer Science Interactive Graphics Systems Group (1994). 9. P. Astheimer, W. Felger, M. Gobel, S. Miiller and R. Ziegler, Industrielle Anwendungen der Virtuellen Realitlt-Beispiele, Erfahrungen, Probleme & Zukunftsperspektiven, Tagungsband “Virtual Reality 94”, pp. 259-280, Stuttgart (1994). 10. W. Felger, How interactive visualization can benefit from multidimensional input devices, SPIELS&T Symposium on Electronic Imaging Science and Technology, Conference 1668: Visual Data Interpretation, San Jose, CA, 9-14 February (1992). 11. K. Akeley, Reality engine graphics, Comput. Graphics (Proc. SIGGRAPH), 109-116 (1993). About the Author-RoLF ZIEGLER is a staff member of the Department of Visualization and Simulation at the Fraunhofer Institute on scientific visualization, simulation and virtual reality, especially for medical applications. He is currently responsible for the development of a medical training simulator by means of virtual reality techniques including force feedback. Furthermore, he is working on a project for the integration of video into virtual reality systems. Dr Ziegler received his diploma degree in Computer Science from the Darmstadt Technical University (Germany) in 1987.

About the Author-GEORG FISCHERis a staff member at the Fraunhofer Institute for Computer Graphics (IGD) in Darmstadt (Germany). For his diploma thesis at the Department of Visualization and Simulation, he was mainly involved in the development of the first prototype of a medical training simulator by means of virtual reality. He is now working in the Department of Animation and HD Image Communication. Dr Fischer received his diploma degree in Computer Science from the Darmstadt Technical University in 1994.

the Author-WOLFGANG ~LLER is a freelancer at the Department of Visualization and Simulation at the Fraunhofer Institute for Computer Graphics (IGD) in Darmstadt. For his diploma thesis at the Department of Visualization and Simulation, he was mainly involved in the development of the first prototype of a medical training simulator by means of virtual reality. His research interests focus on image processing, especially for medical applications. Dr Miiller received his university degree required for the teaching profession in Mathematics and Sports from the Darmstadt Technical University in 1986 and his diploma degree in Computer Science in 1994. About

the Author-MARTIN G~BEL is head of the Department of Visualization and Simulation at the Fraunhofer Institute for Computer Graphics in Darmstadt and the project manager of the Fraunhofer Demonstration Centre of Virtual Reality. At IGD he is responsible for R&D preojects in interactive scientific visualization, simulation, and virtual reality. He also serves as an expert on VR to the CEC. Dr GGbel studies Computer Science at the Technical University of Darmstadt, where he received the diploma degree in 1982. He received a Ph.D. (Dr Ing.) in 1990. He is a member of the IEEE Computer Society. Eurographics, and the German Computer Society. He is actively participating in working groups on scientific visualization and virtual environments as well as imaging and visualization. Ahout