- Email: [email protected]
Haptic reproduction and interactive visualization of a beating heart for cardiovascular surgery simulation
International Journal of Medical Informatics 68 (2002) 155 /163 www.elsevier.com/locate/ijmedinf
Haptic reproduction and interactive visualization of a beating heart for cardiovascular surgery simulation M. Nakao a, , H. Oyama b, M. Komori c, T. Matsuda a, G. Sakaguchi d, M. Komeda d, T. Takahashi b b
a Graduate School of Informatics, Kyoto University, Yoshida Sakyo, Kyoto 606-8501, Japan Department of Medical Informatics, Kyoto University Hospital, Shogoin, Sakyo, Kyoto 606-8507, Japan c Computational Biomedicine, Shiga University of Medical Science, Seta, Otsu 520-2192, Japan d Cardiovascular Surgery, Kyoto University Hospital, Shogoin, Sakyo, Kyoto 606-8507, Japan
Abstract This paper aims to achieve haptic reproduction and real-time visualization of a beating heart for cardiac surgery simulation. Unlike most forgoing approaches, the authors focus on time series datasets and propose a new framework for interactive simulation of active tissues. The framework handles both detection and response of collisions between a manipulator and a beating virtual heart. Physics-based force feedback of autonomous cardiac motion is also produced based on a stress /pressure model, which is adapted to elastic objects filled with fluid. Time series datasets of an adult man were applied to an integrated simulation system with a force feedback device. The system displays multidimensional representation of a beating heart and provides a basic training environment for surgical palpation. Finally, results of measurement and medical assessment confirm the achieved quality and performance of the presented framework. # 2002 Elsevier Science Ireland Ltd. All rights reserved. Keywords: Haptic display; Palpation; Cardiology; Surgery simulation; Virtual reality
1. Introduction Computer-assisted systems in recent days play significant roles in the medical fields. Planning applications and tools are used to simulate and evaluate the results of surgical
Corresponding author
intervention. So far, some clinical uses have been reported in preoperative planning or automatic design [1,2]. Advanced virtual reality (VR) based simulations also have proven to be effective in training medical students and in optimizing surgical strategies [3]. On the other hand, many issues have been discussed to increase the overall applicability of VR simulators. One of the key problems is
1386-5056/02/$ - see front matter # 2002 Elsevier Science Ireland Ltd. All rights reserved. PII: S 1 3 8 6 - 5 0 5 6 ( 0 2 ) 0 0 0 7 3 - 4
156
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
to improve reality of modeled soft tissues. Since manipulations including navigation and surgical procedures are produced through haptic interfaces, effective force feedback schemes are also required. In order to satisfy such interests and requirements, recent work has concentrated on developing physical models [4 7] and improving surgical realism in manipulating, suturing and scalping soft tissues [8,9]. In cardiovascular surgery, computer-assisted educational systems or trainers have been proposed [10,11]. However, in comparison with other surgical fields, few studies have developed VR applications for planning and training for beating heart surgeries. This trend depends on the fact that the system has to deal with complex cardiac models as well as with fast computation. As cardiac motion has a great influence on surgical procedures during surgeries, the system not only requires reproducing the dynamic state of a beating heart, but also needs to describe interaction and force feedback between surgical instruments and tissues. It is also important that patient datasets acquired from MRI or CT can be directly applied to the system. In this paper, as a first step to construct a practical simulator for cardiovascular surgery, the authors propose a framework that enables both visualization and haptic feedback based on measured datasets. The framework handles geometrical and physical constraints of both manipulation by surgeons and of cardiac autonomous motion. Dynamic force is calculated based on a physical model that is adapted to elastic objects filled with fluid. Time-series volumetric datasets can be directly applied to a developed system with a force feedback device, which enables volume rendering and haptic reproduction. The system provides a training environment for surgical palpation. In the following chapters, /
the authors describe details of the approach and results of the simulation.
2. Simulation framework for beating heart Surgical simulators for cardiac surgeries require an interactive response. As the system has to keep refresh rate over 1000 Hz in order to satisfy human force perception, an effective and fast computational scheme is indispensable. To simulate palpation or manipulation during surgeries, the system needs to support haptic rendering between a manipulator and a reconstructed virtual heart. Once the manipulator collides with a part of the myocardium, both physics-based deformation and force feedback related to blood pressure are also required. For these requirements, in the fields of computer graphics, some collision detection techniques have been proposed [17 19]. Deformable models like mass-spring systems [5] or finite elements methods [6] are generally used to describe physical behavior of soft tissues. In the case of the construction of cardiac simulators, however, these approaches mentioned above have to handle geometrical and physical changes in the virtual heart with autonomous beating. In order to describe cardiac motion, foregoing work has reported large-scale numerical simulations that integrate complex mechanical properties of cardiac muscle [12,13]. Halperin et al defined transverse stiffness on the ventricular wall, and showed that it was proportional to the stresses in the plane of the wall [23]. W. Lin et al. achieved real time visualization of cardiac dynamics using a myocardial fibre model [14]. In spite of these results, due to the complexity of beating function, it is currently difficult to simulate its entire behavior while satisfying a haptic compatible rate. /
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
157
Fig. 1. Simulation framework for sequential dataset. Time-series 3D MRI or CT datasets and time-series pressure can be applied to the framework. Real-time volume rendering enables transparent animation of heartbeats, and physics-based force feedback is provided through the haptic interface.
On the other hand, this work focuses on a time-series 3D volumetric dataset for one cardiac cycle. Recent development of multiscan CT enables acquisition of cardiac motion as time-series geometrical information [15,16]. As sequential datasets reduce the necessity of describing physical behavior on cardiac motion, computational cost for overall simulation can be kept low. In addition, this paper proposes an advanced framework that simulates efficient interaction and force feedback for a beating virtual heart. The primary advantage of the framework is that overall methods can be applied to discrete and sequential volumetric datasets. Fig. 1 illustrates the proposed framework including dataset and algorithms. The position of a manipulated point (manipulator) is given through haptic devices. The manipulator represents the tip of the fingers or surgical instruments in the virtual environment. If the collision detection scheme detects interaction between the manipulator and the virtual heart object, both the contacted voxel and its displacement are applied to the force feedback model, which reflects cardiac motion and blood pressure. Results of simulation are visualized in real time using volume rendering.
3. Methods This chapter explains details of the proposed methods. The authors first construct a beating heart model from a measured dataset. Both haptic rendering and the force feedback scheme are also mentioned.
3.1. Constructing heart model The dataset of human body structures with a beating heart was obtained from an ECGgated 3D MRI of a normal volunteer. The whole dataset consists of time series 15 volumetric data (256 256 64 voxels) for one cardiac cycle. Each voxel has a size of 1 1 3 mm due to its resolution. Note that in recent years more detailed datasets can be clinically measured from multi-scan CT. After semi-automatic region extraction from surroundings, 3D volumes of the breast and the heart were extracted. Time series heart volumes are directly applied to fast volume rendering procedures [21], and the reconstructed beating heart is displayed as sequential animation in real time. /
/
/
/
158
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
3.2. Interference with sequential objects Interference between the manipulator and the virtual heart is described as an adapted haptic rendering scheme, which can be applied to discrete and sequential objects that consist of huge voxels. 3.2.1. Haptic rendering for voxel objects Most of the foregoing methods of collision detection are generally classified into two groups. Although geometrical methods [17] based on region partitioning provides a definite solution to voxel models, they also present some problems due to the fact that the manipulator can penetrate thin objects. On the other hand, constraint-based methods [18,19] handle an intermediate object (called a god object or a proxy), which never penetrates virtual objects. The approaches, however, are only applicable to polygonal objects and computationally more expensive than geometry-based methods. The interaction model in this work integrates the advantages of the above approaches. Fig. 2 shows a 2D outline of the proposed methods which perform constraintbased haptic rendering for voxel models. When the manipulator collides with a region of a voxel, a proxy of the manipulator is
generated and constrained on a virtual surface (voxel surface) of the contacted voxel. The proxy prevents the manipulator from penetrating the object. In 3D virtual environments, a normal vector n of the voxel surface is given by the following equation: @p @p @p ; ; (1) n 9p @x @y @z For numerical implementation, the normal vector n nx , ny , nz is: X nx ( 1)i1 pijk /
/
i;j;k 0;1
ny
X
( 1)j1 pijk
i;j;k 0;1
nz
X
( 1)k1 pijk
where the density pijk at the vertex (i, j, k) within the contacted voxel is given as the number of voxels that the vertex belongs to. The coordinate of the proxy (x, y, z) is given by solving the following equation: 10 1 0 1 0 1 0 0 nx xm x B0 1 0 ny CBy C Bym C CB C B C B (3) @0 0 1 nz A@z A @zm A nx ny nz 0 n0 l where the coordinate of the manipulator is (xm , ym , zm ), and the equation of the voxel surface is nx xny ynz z n0 0. /
Fig. 2. Constraint-based haptic rendering for voxel objects. The movement of the manipulator is constrained by the proxy, which is defined on the voxel surface of the contacted voxel.
(2)
i;j;k 0;1
/
/
/
3.2.2. Update of contacted voxel To support interaction with an object which has autonomous motion, the contacted voxel must be updated to an appropriate position reflecting its distension. In other words, the interaction model needs to assure consistency of collision response between the manipulator and the discrete object reconstructed from a sequential dataset.
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
159
Fig. 3. Interaction and force feedback scheme. (a) Update of a contacted voxel. The contacted voxel at a next time is determined by position of the contacted voxel and the voxel surface normal. (b) Calculation of a force vector. Haptic force is calculated by the current manipulator position and the voxel surface normal of the proxy.
Fig. 3(a) illustrates the proposed method by updating a contacted voxel reflecting cardiac motion. Since a beating heart approximately keeps its shape, a new contacted voxel at next discrete time can be defined using the current position of a manipulator and a voxel surface normal. In Fig. 3(a), for example, voxels that exist in the direction of a normal vector from the contacted voxel at a time t are scanned. If the wall of the myocardium is passed through, no voxel can be detected. The boundary gives the next contacted voxel at a time tDt. The significant point of the proposed method is that rotation of cardiac muscle can be simulated as the transition of the normal vector. Note that linear interpolation is required in order to provide smooth force feedback, because of the current time resolution of the measured dataset. /
3.3. Physics-based force feedback A force vector applied to a haptic device is determined using the position of the proxy and the manipulator as Fig. 3(b) illustrates. The current proxy is given by the (Eq. (3)). This situation also demonstrates that a surface of the myocardium is pushed into the position of the manipulator. In order to produce physics-based force feedback in real time,
the authors give a stress-pressure model, which is based on the Long Element Method [7]. The model is adapted to an elastic object filled with fluid, and it gives valid approximation in calculating active force. A part of the myocardium is modelled as a simple elastic element in Fig. 4. The element has the length L and the area of cross section S. A blood pressure P and an external force F affect each side of the element. When a small displacement DL occures, the physical relationship of these parameters is formulated as the following equation: F S
P K
DL L
(4)
where K denotes young modulus of the myocardium at each discrete time. Therefore, the external force is described in the (Eq. (5)). The equation theoretically reflects the same
Fig. 4. Stress-pressure model. This model defines the physical relationship in a part of myocardium.
160
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
characteristics of the stress during heart beats as a conventional numerical solution of cardiac dynamics. In addition, the model does not require a long calculation time. DL S (5) F PK L
4. Results Overall algorithms of the proposed methods are implemented on a standard PC (CPU: Pentium III 500 MHz, Memory: 256 MB). The PHANToM (SensAble Technologies.) is applied to the system as a force feedback device. The VolumePro 500 chipset (Real Time Visualization Inc.) performs real-time volume rendering and interactive response based on the ray casting method [20]. A color look-up table is also provided in order to improve realistic portrayal of a beating heart. This chapter demonstrates visual and haptic representation of the constructed system, and evaluates the performance of the proposed methods. 4.1. Visual and haptic representation Upper images in Fig. 5 show interactive visualization of a beating heart. The transparent object that is cropped by a vertical plane is effective to observe transition of 3D inner structures. Lower images demonstrate interaction and force feedback in the 3D virtual environment. A spherical object denotes the current position of the manipulator which is operated through the PHANToM. Active force caused by cardiac motion is simulated as dynamic change of a force vector shown as a line from the manipulator. In order to calculate dynamic force based on the proposed stress-pressure model, this
system applies 25 mm2 as the contacted area S, and 2.6 kPa as the constant young modulus K, which has been generally used in the literature on cardiac dynamics [13]. Timeseries blood pressure of the left ventricle measured in an adult man is also applied. The first experiment assumes surgical palpation of an autonomous beating. Therefore, the system simulates active force on the condition that the manipulator is only placed on the myocardium of the left ventricle. Fig. 6 shows transition of force from endsystole for each cardiac cycle. The system generates large force at the beginning of the diastolic phase. In the next systolic phase, since the manipulator has a distance from the myocardium, collision is not detected. Fig. 7 shows another transition of haptic force when a tip of the PHANToM is placed at a fixed position against distension of the cardiac muscle. This situation assumes a situation in diagnosis of stiffness of a myocardium. In this case, the graph has two peaks in the diastolic phase, which assures that the system adequately simulates the stress affected by the manipulator. 4.2. Evaluation The achieved reality of the developed system is evaluated qualitatively with eight surgeons who undertake surgical palpation for a cardiac muscle. The evaluation assumes surgical palpation of a myocardium, and the surgeons check the beating state of the left ventricle reproduced by the system. In order to demonstrate realism of the active force, the system gives differing haptic feedback based on the following three basic models: DL
S
(1)
stress: F K
(2)
pressure: F P S
L
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
161
Fig. 5. Haptic reproduction and interactive visualization of a beating heart.
Fig. 6. A result of force in the case of placing the manipulator on the myocardium of left ventricle.
(3)
DL stresspressure: F PK S L
To support interaction with a beating virtual heart, the proposed haptic rendering scheme is applied. As each way of holding a tip of the PHANToM is different, the area S is determined on the condition that the
Fig. 7. A result of force in the case of fixing the manipulator against distension of the myocardium.
magnitude of the force reproduced from the system is approximately the same as that from the actual beating heart. Each surgeon gives from one to five points to each model using a check sheet. The averages of the points are shown in the graph in Fig. 8, and the results assure that the proposed model produces the most realistic force in the three models.
162
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
6. Conclusions
Fig. 8. Haptic realism of the three force models. The stresspressure model can simulate realistic haptic force of heartbeats.
According to the discussion with the surgeons, the stress-pressure model also presents realistic two-step force feedback that simulates actual collision and distension which an in-vivo beating heart provides. It is thought that two peaks of the graph shown in the Fig. 7 reproduce this feature.
5. Discussion The authors presented an integrated framework that enables us to touch and feel a beating virtual heart. Although the developed system provides a basic environment to learn cardiac motion and rehearse surgical procedures with force feedback, some issues still lie in the overall approach. For instance, non-invasive acquisition of tissue elasticity like MRE (Magnetic Resonance Elasticity) technique [22] is important to obtain physical status based on patientspecific dataset. In order to simulate accurate behavior of internal organs, the system must support real-time deformation [4,6]. Although integrated cardiac models [12,13] can describe dynamic changes reflecting physical characteristics of cardiac muscles, increase of calculation cost is fatal to interactive systems. Therefore, fast and effective computational schemes are also required. Stereoscopic display and registration of visual and haptic reproduction improves overall realism. Development and implementation of glove-shaped interfaces is desired to provide more practical training.
Computer assisted simulators or planners play significant roles in learning procedures or optimizing overall strategies. In cardiovascular surgeries, however, the construction of interactive VR simulators was a challenging issue due to the difficulty of simulating complex behavior of a beating heart. This work focused on time series datasets and presented an advanced framework that achieves haptic reproduction and real-time visualization of a beating heart. Both constraint-based haptic rendering and the stresspressure model simulates interaction and force feedback considering reflecting cardiac motion. The developed system with a force feedback device and a volume rendering procedure provides a basic training environment for surgical palpation. To provide biomechanical deformation and more accurate force feedback, improvement of overall applicability of the system and quantitative evaluation of training effects are future tasks for developing more practical simulators.
Acknowledgements The authors would like to thank Tomohiro Kuroda for his helpful discussion in preparing this paper and cardiovascular surgeons at the Kyoto University Hospital for their assistance with evaluations. This work was supported by the Ministry of Education, Science, Culture of Japan, Grants-in-Aid for JSPS Fellows No. 0103889.
References [1] J. Ehrhardt, H. Handels, T. Malina, B. Strathmann, W. Plotz, S.J. Poppl, Atlas-based segmentation of bone
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
structures to support the virtual planning of hip operations, International Journal of Medical Informatics 64 (2001) 439 /447. J. Xia, X. James, H.S.I. Horace, S. Nabil, T.F.W. Helena, G. Jaime, W. Dongfeng, W.K.Y. Richie, S.B.K. Christy, T. Henk, Three-dimentional virtual-reality surgical planning and soft-tissue prediction for orthognathic surgery, IEEE Transaction of Information Technology in Biomedicine 5 (2) (2001) 97 /107. P.J. Gorman, A.H. Meier, M. Krummel, Computer assisted learning and training, Computer Aided Surgery 5 (2000) 120 /127. S. Gibson, B. Mirtich, A Survey of Deformable Modeling in Computer Graphics, MERL Technical Report, TR9719, 1997 L.P. Nedel, D. Thalmann, Real time muscle deformations using mass-spring systems, Computer Graphics International (1998) 156 /165. G. Picinbono, H. Delingette, N. Ayache, Non-linear anisotropic elasticity for real-time surgical simulation, INRIA Yearly Activity Reports, 2000. B. Remis, C.I. Ferreira, LEM-An approach for physically based soft tissue simulation suitable for haptic interaction, Proceedings of the Fifth PHANToM Users Group Workshop, 2000, pp. 26 /30. U. Kuhnapfel, H.K. Cakmak, H. Mass, Endoscopic surgery training using virtual reality and deformable tissue simulation, Elsevier Science Computers and Graphics 24 (5) (2000) 671 /682. B. Pflesser, R. Leuwer, U. Tiede, K.H. Hohne, Planning and Rehearsal of Surgical Intervention in the Volume Model, Proceedings of Medicine Meets Virtual Reality, 2000, pp. 259 /264. S.L. Dawson, S. Cotin, D. Meglan, D.W. Shaffer, M.A. Ferrell, Designing a computer-based simulator for interventional cardiology training, Catheterization and Cardiovascular Interventions 51 (2000) 522 /527. J.S. Rotnes, J. Kaasa, G. Westgaard, et al., Digital trainer developed for robotic assisted cardiac surgery, Proceedings of Medicine Meets Virtual Reality, 2001, pp. 424 /430.
163
[12] P.J. Hunter, A.D. McCulloch, et al., Modelling the mechanical properties of cardiac muscle, Progress in Biophysics and Molecular Biology 69 (1998) 289 /331. [13] M. Tokuda, K. Sekioka, T. Ueno, T. Hayashi, F. Havlicek, Numerical Simulator for Estimation of Mechanical Functions of Human Left Ventricle, JSME 58 (1992) 1100 / 1106. [14] Wei-te Lin, Richard A. Robb, Simulation and interactive multi-dimentional visualization of cardiac dynamics using a patient-specific physics-based model, Proceedings of Computer Assisted Radiology and Surgery (CARS), 2000, pp. 35 /40. [15] K. Katada, R. Kato, H. Anno, et al., Guidance with realtime CT fluoroscopy early clinical experience, Radiology 200 (1996) 851 /856. [16] M. Kachelriess, S. Ulzheimer, W.A. Kalender, ECGcorrelated imaging of the heart with subsecond multislice spiral CT, IEEE Transactions on Medical Imaging 19 (2000) 888 /901. [17] R.S. Avila, M. Sobierajski, A haptic interaction method for volume visualization, Proceedings of IEEE Visualization, 1996, pp. 197 /204. [18] C.B. Zilles, J.K. Salisbury, A constraint-based god-object method for haptic display. IROS 95 (1995) 141 /151. [19] D. Ruspini, K. Kolarov, O. Khatib, The haptic display of complex graphical environments, Proceedings of ACM SIGGRAPH, 1997, pp. 345 /52. [20] S. Parker, Y. Livnat, P. Sloan, C. Hansen, P. Shirley, Interactive ray tracing for volume visualization, IEEE Trans. Visualization and Computer Graphics 3 (1999) 238 /250. [21] H. Pfister, J. Hardenbergh, J. Knittel, H. Lauer, L. Seiler, The VolumePro Real-Time Ray-Casting System, MERL Technical Report, TR99-19, 1999. [22] M.P. Ottensmeyer1, J.K. Salisbury, In Vivo Data Acquisition Instrument for Solid Organ Mechanical Property Measurement, Proceedings of MICCAI, 2001, pp. 975 / 982 [23] H.R. Halperin, P.H. Chew, M.L. Weisfeldt, K. Sagawa, J.D. Humphrey, F.C.P. Yin, Transverse stiffness: a method for estimation of myocardial wall stress, Circ. Res. 61 (1987) 695 /703.
Haptic reproduction and interactive visualization of a beating heart for cardiovascular surgery simulation M. Nakao a, , H. Oyama b, M. Komori c, T. Matsuda a, G. Sakaguchi d, M. Komeda d, T. Takahashi b b
a Graduate School of Informatics, Kyoto University, Yoshida Sakyo, Kyoto 606-8501, Japan Department of Medical Informatics, Kyoto University Hospital, Shogoin, Sakyo, Kyoto 606-8507, Japan c Computational Biomedicine, Shiga University of Medical Science, Seta, Otsu 520-2192, Japan d Cardiovascular Surgery, Kyoto University Hospital, Shogoin, Sakyo, Kyoto 606-8507, Japan
Abstract This paper aims to achieve haptic reproduction and real-time visualization of a beating heart for cardiac surgery simulation. Unlike most forgoing approaches, the authors focus on time series datasets and propose a new framework for interactive simulation of active tissues. The framework handles both detection and response of collisions between a manipulator and a beating virtual heart. Physics-based force feedback of autonomous cardiac motion is also produced based on a stress /pressure model, which is adapted to elastic objects filled with fluid. Time series datasets of an adult man were applied to an integrated simulation system with a force feedback device. The system displays multidimensional representation of a beating heart and provides a basic training environment for surgical palpation. Finally, results of measurement and medical assessment confirm the achieved quality and performance of the presented framework. # 2002 Elsevier Science Ireland Ltd. All rights reserved. Keywords: Haptic display; Palpation; Cardiology; Surgery simulation; Virtual reality
1. Introduction Computer-assisted systems in recent days play significant roles in the medical fields. Planning applications and tools are used to simulate and evaluate the results of surgical
Corresponding author
intervention. So far, some clinical uses have been reported in preoperative planning or automatic design [1,2]. Advanced virtual reality (VR) based simulations also have proven to be effective in training medical students and in optimizing surgical strategies [3]. On the other hand, many issues have been discussed to increase the overall applicability of VR simulators. One of the key problems is
1386-5056/02/$ - see front matter # 2002 Elsevier Science Ireland Ltd. All rights reserved. PII: S 1 3 8 6 - 5 0 5 6 ( 0 2 ) 0 0 0 7 3 - 4
156
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
to improve reality of modeled soft tissues. Since manipulations including navigation and surgical procedures are produced through haptic interfaces, effective force feedback schemes are also required. In order to satisfy such interests and requirements, recent work has concentrated on developing physical models [4 7] and improving surgical realism in manipulating, suturing and scalping soft tissues [8,9]. In cardiovascular surgery, computer-assisted educational systems or trainers have been proposed [10,11]. However, in comparison with other surgical fields, few studies have developed VR applications for planning and training for beating heart surgeries. This trend depends on the fact that the system has to deal with complex cardiac models as well as with fast computation. As cardiac motion has a great influence on surgical procedures during surgeries, the system not only requires reproducing the dynamic state of a beating heart, but also needs to describe interaction and force feedback between surgical instruments and tissues. It is also important that patient datasets acquired from MRI or CT can be directly applied to the system. In this paper, as a first step to construct a practical simulator for cardiovascular surgery, the authors propose a framework that enables both visualization and haptic feedback based on measured datasets. The framework handles geometrical and physical constraints of both manipulation by surgeons and of cardiac autonomous motion. Dynamic force is calculated based on a physical model that is adapted to elastic objects filled with fluid. Time-series volumetric datasets can be directly applied to a developed system with a force feedback device, which enables volume rendering and haptic reproduction. The system provides a training environment for surgical palpation. In the following chapters, /
the authors describe details of the approach and results of the simulation.
2. Simulation framework for beating heart Surgical simulators for cardiac surgeries require an interactive response. As the system has to keep refresh rate over 1000 Hz in order to satisfy human force perception, an effective and fast computational scheme is indispensable. To simulate palpation or manipulation during surgeries, the system needs to support haptic rendering between a manipulator and a reconstructed virtual heart. Once the manipulator collides with a part of the myocardium, both physics-based deformation and force feedback related to blood pressure are also required. For these requirements, in the fields of computer graphics, some collision detection techniques have been proposed [17 19]. Deformable models like mass-spring systems [5] or finite elements methods [6] are generally used to describe physical behavior of soft tissues. In the case of the construction of cardiac simulators, however, these approaches mentioned above have to handle geometrical and physical changes in the virtual heart with autonomous beating. In order to describe cardiac motion, foregoing work has reported large-scale numerical simulations that integrate complex mechanical properties of cardiac muscle [12,13]. Halperin et al defined transverse stiffness on the ventricular wall, and showed that it was proportional to the stresses in the plane of the wall [23]. W. Lin et al. achieved real time visualization of cardiac dynamics using a myocardial fibre model [14]. In spite of these results, due to the complexity of beating function, it is currently difficult to simulate its entire behavior while satisfying a haptic compatible rate. /
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
157
Fig. 1. Simulation framework for sequential dataset. Time-series 3D MRI or CT datasets and time-series pressure can be applied to the framework. Real-time volume rendering enables transparent animation of heartbeats, and physics-based force feedback is provided through the haptic interface.
On the other hand, this work focuses on a time-series 3D volumetric dataset for one cardiac cycle. Recent development of multiscan CT enables acquisition of cardiac motion as time-series geometrical information [15,16]. As sequential datasets reduce the necessity of describing physical behavior on cardiac motion, computational cost for overall simulation can be kept low. In addition, this paper proposes an advanced framework that simulates efficient interaction and force feedback for a beating virtual heart. The primary advantage of the framework is that overall methods can be applied to discrete and sequential volumetric datasets. Fig. 1 illustrates the proposed framework including dataset and algorithms. The position of a manipulated point (manipulator) is given through haptic devices. The manipulator represents the tip of the fingers or surgical instruments in the virtual environment. If the collision detection scheme detects interaction between the manipulator and the virtual heart object, both the contacted voxel and its displacement are applied to the force feedback model, which reflects cardiac motion and blood pressure. Results of simulation are visualized in real time using volume rendering.
3. Methods This chapter explains details of the proposed methods. The authors first construct a beating heart model from a measured dataset. Both haptic rendering and the force feedback scheme are also mentioned.
3.1. Constructing heart model The dataset of human body structures with a beating heart was obtained from an ECGgated 3D MRI of a normal volunteer. The whole dataset consists of time series 15 volumetric data (256 256 64 voxels) for one cardiac cycle. Each voxel has a size of 1 1 3 mm due to its resolution. Note that in recent years more detailed datasets can be clinically measured from multi-scan CT. After semi-automatic region extraction from surroundings, 3D volumes of the breast and the heart were extracted. Time series heart volumes are directly applied to fast volume rendering procedures [21], and the reconstructed beating heart is displayed as sequential animation in real time. /
/
/
/
158
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
3.2. Interference with sequential objects Interference between the manipulator and the virtual heart is described as an adapted haptic rendering scheme, which can be applied to discrete and sequential objects that consist of huge voxels. 3.2.1. Haptic rendering for voxel objects Most of the foregoing methods of collision detection are generally classified into two groups. Although geometrical methods [17] based on region partitioning provides a definite solution to voxel models, they also present some problems due to the fact that the manipulator can penetrate thin objects. On the other hand, constraint-based methods [18,19] handle an intermediate object (called a god object or a proxy), which never penetrates virtual objects. The approaches, however, are only applicable to polygonal objects and computationally more expensive than geometry-based methods. The interaction model in this work integrates the advantages of the above approaches. Fig. 2 shows a 2D outline of the proposed methods which perform constraintbased haptic rendering for voxel models. When the manipulator collides with a region of a voxel, a proxy of the manipulator is
generated and constrained on a virtual surface (voxel surface) of the contacted voxel. The proxy prevents the manipulator from penetrating the object. In 3D virtual environments, a normal vector n of the voxel surface is given by the following equation: @p @p @p ; ; (1) n 9p @x @y @z For numerical implementation, the normal vector n nx , ny , nz is: X nx ( 1)i1 pijk /
/
i;j;k 0;1
ny
X
( 1)j1 pijk
i;j;k 0;1
nz
X
( 1)k1 pijk
where the density pijk at the vertex (i, j, k) within the contacted voxel is given as the number of voxels that the vertex belongs to. The coordinate of the proxy (x, y, z) is given by solving the following equation: 10 1 0 1 0 1 0 0 nx xm x B0 1 0 ny CBy C Bym C CB C B C B (3) @0 0 1 nz A@z A @zm A nx ny nz 0 n0 l where the coordinate of the manipulator is (xm , ym , zm ), and the equation of the voxel surface is nx xny ynz z n0 0. /
Fig. 2. Constraint-based haptic rendering for voxel objects. The movement of the manipulator is constrained by the proxy, which is defined on the voxel surface of the contacted voxel.
(2)
i;j;k 0;1
/
/
/
3.2.2. Update of contacted voxel To support interaction with an object which has autonomous motion, the contacted voxel must be updated to an appropriate position reflecting its distension. In other words, the interaction model needs to assure consistency of collision response between the manipulator and the discrete object reconstructed from a sequential dataset.
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
159
Fig. 3. Interaction and force feedback scheme. (a) Update of a contacted voxel. The contacted voxel at a next time is determined by position of the contacted voxel and the voxel surface normal. (b) Calculation of a force vector. Haptic force is calculated by the current manipulator position and the voxel surface normal of the proxy.
Fig. 3(a) illustrates the proposed method by updating a contacted voxel reflecting cardiac motion. Since a beating heart approximately keeps its shape, a new contacted voxel at next discrete time can be defined using the current position of a manipulator and a voxel surface normal. In Fig. 3(a), for example, voxels that exist in the direction of a normal vector from the contacted voxel at a time t are scanned. If the wall of the myocardium is passed through, no voxel can be detected. The boundary gives the next contacted voxel at a time tDt. The significant point of the proposed method is that rotation of cardiac muscle can be simulated as the transition of the normal vector. Note that linear interpolation is required in order to provide smooth force feedback, because of the current time resolution of the measured dataset. /
3.3. Physics-based force feedback A force vector applied to a haptic device is determined using the position of the proxy and the manipulator as Fig. 3(b) illustrates. The current proxy is given by the (Eq. (3)). This situation also demonstrates that a surface of the myocardium is pushed into the position of the manipulator. In order to produce physics-based force feedback in real time,
the authors give a stress-pressure model, which is based on the Long Element Method [7]. The model is adapted to an elastic object filled with fluid, and it gives valid approximation in calculating active force. A part of the myocardium is modelled as a simple elastic element in Fig. 4. The element has the length L and the area of cross section S. A blood pressure P and an external force F affect each side of the element. When a small displacement DL occures, the physical relationship of these parameters is formulated as the following equation: F S
P K
DL L
(4)
where K denotes young modulus of the myocardium at each discrete time. Therefore, the external force is described in the (Eq. (5)). The equation theoretically reflects the same
Fig. 4. Stress-pressure model. This model defines the physical relationship in a part of myocardium.
160
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
characteristics of the stress during heart beats as a conventional numerical solution of cardiac dynamics. In addition, the model does not require a long calculation time. DL S (5) F PK L
4. Results Overall algorithms of the proposed methods are implemented on a standard PC (CPU: Pentium III 500 MHz, Memory: 256 MB). The PHANToM (SensAble Technologies.) is applied to the system as a force feedback device. The VolumePro 500 chipset (Real Time Visualization Inc.) performs real-time volume rendering and interactive response based on the ray casting method [20]. A color look-up table is also provided in order to improve realistic portrayal of a beating heart. This chapter demonstrates visual and haptic representation of the constructed system, and evaluates the performance of the proposed methods. 4.1. Visual and haptic representation Upper images in Fig. 5 show interactive visualization of a beating heart. The transparent object that is cropped by a vertical plane is effective to observe transition of 3D inner structures. Lower images demonstrate interaction and force feedback in the 3D virtual environment. A spherical object denotes the current position of the manipulator which is operated through the PHANToM. Active force caused by cardiac motion is simulated as dynamic change of a force vector shown as a line from the manipulator. In order to calculate dynamic force based on the proposed stress-pressure model, this
system applies 25 mm2 as the contacted area S, and 2.6 kPa as the constant young modulus K, which has been generally used in the literature on cardiac dynamics [13]. Timeseries blood pressure of the left ventricle measured in an adult man is also applied. The first experiment assumes surgical palpation of an autonomous beating. Therefore, the system simulates active force on the condition that the manipulator is only placed on the myocardium of the left ventricle. Fig. 6 shows transition of force from endsystole for each cardiac cycle. The system generates large force at the beginning of the diastolic phase. In the next systolic phase, since the manipulator has a distance from the myocardium, collision is not detected. Fig. 7 shows another transition of haptic force when a tip of the PHANToM is placed at a fixed position against distension of the cardiac muscle. This situation assumes a situation in diagnosis of stiffness of a myocardium. In this case, the graph has two peaks in the diastolic phase, which assures that the system adequately simulates the stress affected by the manipulator. 4.2. Evaluation The achieved reality of the developed system is evaluated qualitatively with eight surgeons who undertake surgical palpation for a cardiac muscle. The evaluation assumes surgical palpation of a myocardium, and the surgeons check the beating state of the left ventricle reproduced by the system. In order to demonstrate realism of the active force, the system gives differing haptic feedback based on the following three basic models: DL
S
(1)
stress: F K
(2)
pressure: F P S
L
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
161
Fig. 5. Haptic reproduction and interactive visualization of a beating heart.
Fig. 6. A result of force in the case of placing the manipulator on the myocardium of left ventricle.
(3)
DL stresspressure: F PK S L
To support interaction with a beating virtual heart, the proposed haptic rendering scheme is applied. As each way of holding a tip of the PHANToM is different, the area S is determined on the condition that the
Fig. 7. A result of force in the case of fixing the manipulator against distension of the myocardium.
magnitude of the force reproduced from the system is approximately the same as that from the actual beating heart. Each surgeon gives from one to five points to each model using a check sheet. The averages of the points are shown in the graph in Fig. 8, and the results assure that the proposed model produces the most realistic force in the three models.
162
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
6. Conclusions
Fig. 8. Haptic realism of the three force models. The stresspressure model can simulate realistic haptic force of heartbeats.
According to the discussion with the surgeons, the stress-pressure model also presents realistic two-step force feedback that simulates actual collision and distension which an in-vivo beating heart provides. It is thought that two peaks of the graph shown in the Fig. 7 reproduce this feature.
5. Discussion The authors presented an integrated framework that enables us to touch and feel a beating virtual heart. Although the developed system provides a basic environment to learn cardiac motion and rehearse surgical procedures with force feedback, some issues still lie in the overall approach. For instance, non-invasive acquisition of tissue elasticity like MRE (Magnetic Resonance Elasticity) technique [22] is important to obtain physical status based on patientspecific dataset. In order to simulate accurate behavior of internal organs, the system must support real-time deformation [4,6]. Although integrated cardiac models [12,13] can describe dynamic changes reflecting physical characteristics of cardiac muscles, increase of calculation cost is fatal to interactive systems. Therefore, fast and effective computational schemes are also required. Stereoscopic display and registration of visual and haptic reproduction improves overall realism. Development and implementation of glove-shaped interfaces is desired to provide more practical training.
Computer assisted simulators or planners play significant roles in learning procedures or optimizing overall strategies. In cardiovascular surgeries, however, the construction of interactive VR simulators was a challenging issue due to the difficulty of simulating complex behavior of a beating heart. This work focused on time series datasets and presented an advanced framework that achieves haptic reproduction and real-time visualization of a beating heart. Both constraint-based haptic rendering and the stresspressure model simulates interaction and force feedback considering reflecting cardiac motion. The developed system with a force feedback device and a volume rendering procedure provides a basic training environment for surgical palpation. To provide biomechanical deformation and more accurate force feedback, improvement of overall applicability of the system and quantitative evaluation of training effects are future tasks for developing more practical simulators.
Acknowledgements The authors would like to thank Tomohiro Kuroda for his helpful discussion in preparing this paper and cardiovascular surgeons at the Kyoto University Hospital for their assistance with evaluations. This work was supported by the Ministry of Education, Science, Culture of Japan, Grants-in-Aid for JSPS Fellows No. 0103889.
References [1] J. Ehrhardt, H. Handels, T. Malina, B. Strathmann, W. Plotz, S.J. Poppl, Atlas-based segmentation of bone
M. Nakao et al. / International Journal of Medical Informatics 68 (2002) 155 /163
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
structures to support the virtual planning of hip operations, International Journal of Medical Informatics 64 (2001) 439 /447. J. Xia, X. James, H.S.I. Horace, S. Nabil, T.F.W. Helena, G. Jaime, W. Dongfeng, W.K.Y. Richie, S.B.K. Christy, T. Henk, Three-dimentional virtual-reality surgical planning and soft-tissue prediction for orthognathic surgery, IEEE Transaction of Information Technology in Biomedicine 5 (2) (2001) 97 /107. P.J. Gorman, A.H. Meier, M. Krummel, Computer assisted learning and training, Computer Aided Surgery 5 (2000) 120 /127. S. Gibson, B. Mirtich, A Survey of Deformable Modeling in Computer Graphics, MERL Technical Report, TR9719, 1997 L.P. Nedel, D. Thalmann, Real time muscle deformations using mass-spring systems, Computer Graphics International (1998) 156 /165. G. Picinbono, H. Delingette, N. Ayache, Non-linear anisotropic elasticity for real-time surgical simulation, INRIA Yearly Activity Reports, 2000. B. Remis, C.I. Ferreira, LEM-An approach for physically based soft tissue simulation suitable for haptic interaction, Proceedings of the Fifth PHANToM Users Group Workshop, 2000, pp. 26 /30. U. Kuhnapfel, H.K. Cakmak, H. Mass, Endoscopic surgery training using virtual reality and deformable tissue simulation, Elsevier Science Computers and Graphics 24 (5) (2000) 671 /682. B. Pflesser, R. Leuwer, U. Tiede, K.H. Hohne, Planning and Rehearsal of Surgical Intervention in the Volume Model, Proceedings of Medicine Meets Virtual Reality, 2000, pp. 259 /264. S.L. Dawson, S. Cotin, D. Meglan, D.W. Shaffer, M.A. Ferrell, Designing a computer-based simulator for interventional cardiology training, Catheterization and Cardiovascular Interventions 51 (2000) 522 /527. J.S. Rotnes, J. Kaasa, G. Westgaard, et al., Digital trainer developed for robotic assisted cardiac surgery, Proceedings of Medicine Meets Virtual Reality, 2001, pp. 424 /430.
163
[12] P.J. Hunter, A.D. McCulloch, et al., Modelling the mechanical properties of cardiac muscle, Progress in Biophysics and Molecular Biology 69 (1998) 289 /331. [13] M. Tokuda, K. Sekioka, T. Ueno, T. Hayashi, F. Havlicek, Numerical Simulator for Estimation of Mechanical Functions of Human Left Ventricle, JSME 58 (1992) 1100 / 1106. [14] Wei-te Lin, Richard A. Robb, Simulation and interactive multi-dimentional visualization of cardiac dynamics using a patient-specific physics-based model, Proceedings of Computer Assisted Radiology and Surgery (CARS), 2000, pp. 35 /40. [15] K. Katada, R. Kato, H. Anno, et al., Guidance with realtime CT fluoroscopy early clinical experience, Radiology 200 (1996) 851 /856. [16] M. Kachelriess, S. Ulzheimer, W.A. Kalender, ECGcorrelated imaging of the heart with subsecond multislice spiral CT, IEEE Transactions on Medical Imaging 19 (2000) 888 /901. [17] R.S. Avila, M. Sobierajski, A haptic interaction method for volume visualization, Proceedings of IEEE Visualization, 1996, pp. 197 /204. [18] C.B. Zilles, J.K. Salisbury, A constraint-based god-object method for haptic display. IROS 95 (1995) 141 /151. [19] D. Ruspini, K. Kolarov, O. Khatib, The haptic display of complex graphical environments, Proceedings of ACM SIGGRAPH, 1997, pp. 345 /52. [20] S. Parker, Y. Livnat, P. Sloan, C. Hansen, P. Shirley, Interactive ray tracing for volume visualization, IEEE Trans. Visualization and Computer Graphics 3 (1999) 238 /250. [21] H. Pfister, J. Hardenbergh, J. Knittel, H. Lauer, L. Seiler, The VolumePro Real-Time Ray-Casting System, MERL Technical Report, TR99-19, 1999. [22] M.P. Ottensmeyer1, J.K. Salisbury, In Vivo Data Acquisition Instrument for Solid Organ Mechanical Property Measurement, Proceedings of MICCAI, 2001, pp. 975 / 982 [23] H.R. Halperin, P.H. Chew, M.L. Weisfeldt, K. Sagawa, J.D. Humphrey, F.C.P. Yin, Transverse stiffness: a method for estimation of myocardial wall stress, Circ. Res. 61 (1987) 695 /703.