- Email: [email protected]
Automatic path searching for interactive navigation support within virtual medical 3D objects
International Congress Series 1256 (2003) 29 – 34
Automatic path searching for interactive navigation support within virtual medical 3D objects H. Noser *, Ch. Stern, P. Stucki MultiMedia Laboratory, Department of Information Technology, University of Zurich, Winterthurerstrasse 190, CH-8057 Zurich, Switzerland Received 14 March 2003; received in revised form 14 March 2003; accepted 20 March 2003
Abstract This article proposes the use of autonomous actors for navigation support within complex virtual medical objects reconstructed from Computed Tomography (CT) or Magnetic Resonance (MR) Imaging. Such objects are often maze-like. Therefore, users risk getting lost within them. We present synthetic vision-based actors capable of finding collision free paths from a given position to a goal point in most triangulated virtual environments containing loops and impasses. These paths are useful for guided fly-throughs when performing noninvasive diagnostic tasks. D 2003 Elsevier Science B.V. and CARS. All rights reserved. Keywords: Virtual endoscopy; Path searching; Autonomous actor; Synthetic vision; Navigation support
1. Introduction Virtual Endoscopy is a widely used method for performing screening and diagnostic activities in radiology. It involves reconstructing 3D virtual medical objects from Computed Tomography (CT) or Magnetic Resonance (MR) images using iso-surface generation or volume rendering methods, as well as interactive 3D navigation within the resulting 3D objects. However, the geometric complexity of the resulting anatomical objects makes interactive navigation in space challenging because inexperienced users often get lost within the maze-like objects. In the present work, Virtual Reality (VR) techniques and an automatic heuristic approach for guided navigation are developed for
* Corresponding author. Tel.: +41-1-635-43-16; fax: +41-1-635-68-09. E-mail addresses: [email protected] (H. Noser), [email protected] (Ch. Stern), [email protected] (P. Stucki). 0531-5131/03 D 2003 Elsevier Science B.V. and CARS. All rights reserved. doi:10.1016/S0531-5131(03)00347-9
30
H. Noser et al. / International Congress Series 1256 (2003) 29–34
supporting noninvasive diagnostic tasks, e.g. for detecting small tumors contained within the lumen of the colon. Due to the fact that cancer of the colon is the second most common malignant tumor in the western world, such a 3D navigation support is of major interest. As part of a collaborative project between the Institute of Diagnostic Radiology at the University Hospital Zurich and our laboratory, an MR- and VR-based technique combining a 3D MR data acquisition strategy with novel post-processing algorithms capable of reconstructing virtual endoscopic images was developed [1,2]. During the past years, significant progress towards the development of MR colonography as a credible screening tool has been made. However, navigation in 3D space must be considered a crucial aspect when developing a user-friendly system for this purpose. Developing, implementing, and evaluating the best means to navigate in the acquired and reconstructed 3D data for fast screening diagnosis was/is one of the key goals of this work. For fast screening, both fully guided fly-throughs as well as completely free navigation are important. Automatic fly-throughs assure that all interior parts of the organ are visited and explored while the examiner can fully concentrate on the detection of potential irregularities. Free navigation is necessary for closer examination of suspicious locations, and to adapt to specific needs of a particular examination. In addition, reasonable combinations of guidance and flexibility should be devised and evaluated because they promise improving the efficiency of the exploration process. A good path through a tubular structure such as the human colon tries to stay in the ‘‘middle’’ of the structure throughout the flight. This has been formalized in the concept of the Medial Axis Transform (MAT) in image processing and has been extended to volume processing [3]. The MAT does not consist of a single path, but of many segments connected in a complex graph structure. The MAT segments ideally include all the points of the object that lie closest to three points on the surface of the object. In the discrete case of voxels, the MAT is often approximated by skeletonization, i.e. by removing voxels from the surface of the object in a carefully controlled process. This process is difficult to implement and errorprone, in particular for 3D as it depends strongly on threshold values of voxel data. Moreover, it only accepts voxel input data sets and not surface models. Therefore, we developed a different approach for automatic path searching based on autonomous actors equipped with synthetic vision developed in computer graphics animation. Such actors are capable of finding 3D paths from start points to goal points in most virtual 3D scenes. This concept has been treated in earlier work such as Refs. [4,5]. Therefore, the next subsections describe informally how synthetic vision, visual memory, and reasoning work for autonomous actors. The focus is on its integration, some extensions, and new applications.
2. Navigation support by autonomous actors In our vision-based approach to behavioral animation, the synthetic actor perceives its environment through a small Z-buffer window in which the environment is rendered from the actor’s point of view. As an actor can access Z-buffer values, it can locate visible objects in the 3D environment. This local information is sufficient for local navigation and behavior. The concept of synthetic vision is independent of 3D world modeling. Even fractal objects or procedurally rendered worlds without 3D object database can be
H. Noser et al. / International Congress Series 1256 (2003) 29–34
31
perceived as long as they can be rendered into a vision window. Subsequently, we use synthetic vision for environment recovery and for avoiding obstacles during navigation. To behave naturally and to detect cycles and impasses in maze-like environments, actors need a visual memory that can be implemented by an occupancy space grid. The idea is that a navigating actor remembers all obstacles it sees. Each voxel occupied by an obstacle will be marked as occupied in its voxel grid. As such 3D grids tend to be huge at high resolutions, we implemented the occupancy grid as an octree, a dynamic data structure adapting itself to the complexity of the virtual scene. An interesting property of the octree is the fact that it represents a graph of a 3D environment. When considering all the empty voxels as nodes of a graph, where the neighbors are connected by edges, we can apply algorithms of graph theory directly on the octree, and it is not necessary to convert the representation of the octree into a special network suitable for path searching. Fig. 1 illustrates the building blocks of the behavior control of actors. A behavior can be composed of basic behaviors and actions. A basic action is for instance a motor procedure allowing an actor to move. To model basic behaviors, we use a nonparallel finite state automata approach. Each actor has an internal state, which changes each time step according to the currently active automaton and its sensorial input. Basic predefined behaviors are for example the automata Turn Around, L-system Control, and Path Search. The automaton Path Search shown in Fig. 1 is a complex behavior that actively searches for a path from a given position to a given goal by avoiding obstacles, and by memorizing all the obstacles it encounters during its search. This automaton has been
Fig. 1. Components of the behavior control of vision-based autonomous actors used in path searching.
32
H. Noser et al. / International Congress Series 1256 (2003) 29–34
adapted and optimized for a universal heuristic search in complex 3D environments, initially unknown for autonomous actors. A central part of the search method is the Find Path algorithm acting in the actor’s voxel memory, which has been described in an earlier publication [3]. Turn Around simply lets look around the actor’s vision system at its current position for perceiving and memorizing its neighborhood. The automaton L-system Control passes the control of the actor to a higher-level rule-based control mechanism. To control the automata behaviors of actors at the next higher level, we use a stack of command strings, which are interpreted by a special command string parser acting on the state variables of the actors. The stack-based control system allows an actor to create its own subgoals while executing the original script. With this concept, a user can compose new scenarios consisting of basic behaviors. At the highest level, actors can be scripted by rewriting rules of LWorld [6], a rewritingbased interactive animation system that includes the autonomous vision-based actors. It can read and visualize complex medical objects as surface models in 3D. Symbolic rules, as illustrated in Fig. 1, can configure actors and use automata as high-level primitives in user defined complex behaviors. As an example, we modeled a vision-based actor by rules,
Fig. 2. Software architecture for virtual endoscopy with automatic path searching for interactive navigation support.
H. Noser et al. / International Congress Series 1256 (2003) 29–34
33
Fig. 3. Found paths in two different aorta models. MRI data sets courtesy of Institute of Diagnostic Radiology, University Hospital Zurich.
capable of finding paths in complex, maze-like objects. Found paths may be exported as ASCII files and imported by other interactive applications such as MedIS-VR [1].
3. Results Fig. 2 illustrates the architecture of the system used for navigation support by automatic path searching within virtual medical organs. Automatic path finding and
Fig. 4. Found paths in a colon and in blood vessels of the brain. MRI data sets courtesy of Institute of Diagnostic Radiology, University Hospital Zurich. MRI data sets courtesy of Institute of Neuroradiology, University Hospital Zurich.
34
H. Noser et al. / International Congress Series 1256 (2003) 29–34
synthetic vision is embedded within LWorld and is configured and modeled by rules (Lsystems). Virtual Endoscopy sessions are performed in real-time employing MedIS-VR [1], a stateof-the-art application for 3D visualization and VR that combines Surface and Volume Rendering of medical data sets. To reconstruct 3D anatomical objects from MR and CT images, the well-known Marching-Cube algorithm for iso-surface generation is used. For testing the novel path searching method, the four examples of reconstructed medical organs illustrated in Figs. 3 and 4 were employed: Two aorta models a colon, and blood vessels from the brain region. In the first three organs, only a start and a goal point were necessary for the actor to find a path. In the blood vessels, however, an intermediary point was necessary because of the complex maze-like anatomical structure of the blood vessels. The lower middle image of Fig. 4 shows the voxelized visual memory of an actor with an intermediary path during the search process. The dark regions correspond to areas the actor has not seen yet. It looks for possible paths in the empty space according to its path search heuristics. The resulting paths are imported into the MedIS-VR system and can be used for guided fly-throughs facilitating the navigation task considerably.
4. Discussion Virtual Reality has shown to be a valuable tool in the field of medicine, especially when performing diagnostic tasks. Interactivity plays an important role when navigating and examining objects in 3D space. Automatic path finding facilitates navigation and the diagnostic task considerably. In this article, a new VR system based on a heuristic method for automatic path finding has been proposed. Due to its high information value, it is highly appreciated by radiologists and physicians. When considering interactive examination and automatic path searching, the aspect of user-friendliness plays a crucial role. Therefore, further research in 3D navigation and user-interface design is indispensable. It is also planned to incorporate new diagnostic activities and application fields. References [1] E. Stoll, Ch. Stern, S. Wildermuth, P. Stucki, A new filtering algorithm for medical MR- and CT-images, Journal of Digital Imaging 12 (1) (1999, February) 23. [2] Ch. Stern, S. Wildermuth, et al., Interactive definition of endoluminal aortic stent size and morphology based on virtual angioscopic rendering of 3D Magnetic Resonance Angiography (MRA), Proceedings of the Computer Aided Radiology and Surgery (CARS), 13th International Congress and Exhibition, 1999, June, pp. 176 – 180. [3] G. Yarong, et al., 3D skeleton for virtual colonoscopy, in: K.H. Ho¨hne, R. Kikinis (Eds.), Visualization in Biomedical Computing: 4th International Conference, Proceedings, Hamburg, Germany, September 22 – 25, 1996. [4] H. Noser, A Behavioral Animation System Based on L-systems and Synthetic Sensors for Actors, THE`SE No. 1609, E´cole Polytechnique Fe´de´rale de Lausanne (1997). [5] H. Noser, D. Thalmann, A rule-based interactive behavioral animation system for humanoids, IEEE Transactions on Visualization and Computer Graphics 5 (4) (1999, October – December). [6] H. Noser, Lworld: an animation system based on rewriting, Pacific Graphics 2002 Proceedings, 9 – 11 October, Beijing China, IEEE Computer Society, 2002, pp. 487 – 488.
Automatic path searching for interactive navigation support within virtual medical 3D objects H. Noser *, Ch. Stern, P. Stucki MultiMedia Laboratory, Department of Information Technology, University of Zurich, Winterthurerstrasse 190, CH-8057 Zurich, Switzerland Received 14 March 2003; received in revised form 14 March 2003; accepted 20 March 2003
Abstract This article proposes the use of autonomous actors for navigation support within complex virtual medical objects reconstructed from Computed Tomography (CT) or Magnetic Resonance (MR) Imaging. Such objects are often maze-like. Therefore, users risk getting lost within them. We present synthetic vision-based actors capable of finding collision free paths from a given position to a goal point in most triangulated virtual environments containing loops and impasses. These paths are useful for guided fly-throughs when performing noninvasive diagnostic tasks. D 2003 Elsevier Science B.V. and CARS. All rights reserved. Keywords: Virtual endoscopy; Path searching; Autonomous actor; Synthetic vision; Navigation support
1. Introduction Virtual Endoscopy is a widely used method for performing screening and diagnostic activities in radiology. It involves reconstructing 3D virtual medical objects from Computed Tomography (CT) or Magnetic Resonance (MR) images using iso-surface generation or volume rendering methods, as well as interactive 3D navigation within the resulting 3D objects. However, the geometric complexity of the resulting anatomical objects makes interactive navigation in space challenging because inexperienced users often get lost within the maze-like objects. In the present work, Virtual Reality (VR) techniques and an automatic heuristic approach for guided navigation are developed for
* Corresponding author. Tel.: +41-1-635-43-16; fax: +41-1-635-68-09. E-mail addresses: [email protected] (H. Noser), [email protected] (Ch. Stern), [email protected] (P. Stucki). 0531-5131/03 D 2003 Elsevier Science B.V. and CARS. All rights reserved. doi:10.1016/S0531-5131(03)00347-9
30
H. Noser et al. / International Congress Series 1256 (2003) 29–34
supporting noninvasive diagnostic tasks, e.g. for detecting small tumors contained within the lumen of the colon. Due to the fact that cancer of the colon is the second most common malignant tumor in the western world, such a 3D navigation support is of major interest. As part of a collaborative project between the Institute of Diagnostic Radiology at the University Hospital Zurich and our laboratory, an MR- and VR-based technique combining a 3D MR data acquisition strategy with novel post-processing algorithms capable of reconstructing virtual endoscopic images was developed [1,2]. During the past years, significant progress towards the development of MR colonography as a credible screening tool has been made. However, navigation in 3D space must be considered a crucial aspect when developing a user-friendly system for this purpose. Developing, implementing, and evaluating the best means to navigate in the acquired and reconstructed 3D data for fast screening diagnosis was/is one of the key goals of this work. For fast screening, both fully guided fly-throughs as well as completely free navigation are important. Automatic fly-throughs assure that all interior parts of the organ are visited and explored while the examiner can fully concentrate on the detection of potential irregularities. Free navigation is necessary for closer examination of suspicious locations, and to adapt to specific needs of a particular examination. In addition, reasonable combinations of guidance and flexibility should be devised and evaluated because they promise improving the efficiency of the exploration process. A good path through a tubular structure such as the human colon tries to stay in the ‘‘middle’’ of the structure throughout the flight. This has been formalized in the concept of the Medial Axis Transform (MAT) in image processing and has been extended to volume processing [3]. The MAT does not consist of a single path, but of many segments connected in a complex graph structure. The MAT segments ideally include all the points of the object that lie closest to three points on the surface of the object. In the discrete case of voxels, the MAT is often approximated by skeletonization, i.e. by removing voxels from the surface of the object in a carefully controlled process. This process is difficult to implement and errorprone, in particular for 3D as it depends strongly on threshold values of voxel data. Moreover, it only accepts voxel input data sets and not surface models. Therefore, we developed a different approach for automatic path searching based on autonomous actors equipped with synthetic vision developed in computer graphics animation. Such actors are capable of finding 3D paths from start points to goal points in most virtual 3D scenes. This concept has been treated in earlier work such as Refs. [4,5]. Therefore, the next subsections describe informally how synthetic vision, visual memory, and reasoning work for autonomous actors. The focus is on its integration, some extensions, and new applications.
2. Navigation support by autonomous actors In our vision-based approach to behavioral animation, the synthetic actor perceives its environment through a small Z-buffer window in which the environment is rendered from the actor’s point of view. As an actor can access Z-buffer values, it can locate visible objects in the 3D environment. This local information is sufficient for local navigation and behavior. The concept of synthetic vision is independent of 3D world modeling. Even fractal objects or procedurally rendered worlds without 3D object database can be
H. Noser et al. / International Congress Series 1256 (2003) 29–34
31
perceived as long as they can be rendered into a vision window. Subsequently, we use synthetic vision for environment recovery and for avoiding obstacles during navigation. To behave naturally and to detect cycles and impasses in maze-like environments, actors need a visual memory that can be implemented by an occupancy space grid. The idea is that a navigating actor remembers all obstacles it sees. Each voxel occupied by an obstacle will be marked as occupied in its voxel grid. As such 3D grids tend to be huge at high resolutions, we implemented the occupancy grid as an octree, a dynamic data structure adapting itself to the complexity of the virtual scene. An interesting property of the octree is the fact that it represents a graph of a 3D environment. When considering all the empty voxels as nodes of a graph, where the neighbors are connected by edges, we can apply algorithms of graph theory directly on the octree, and it is not necessary to convert the representation of the octree into a special network suitable for path searching. Fig. 1 illustrates the building blocks of the behavior control of actors. A behavior can be composed of basic behaviors and actions. A basic action is for instance a motor procedure allowing an actor to move. To model basic behaviors, we use a nonparallel finite state automata approach. Each actor has an internal state, which changes each time step according to the currently active automaton and its sensorial input. Basic predefined behaviors are for example the automata Turn Around, L-system Control, and Path Search. The automaton Path Search shown in Fig. 1 is a complex behavior that actively searches for a path from a given position to a given goal by avoiding obstacles, and by memorizing all the obstacles it encounters during its search. This automaton has been
Fig. 1. Components of the behavior control of vision-based autonomous actors used in path searching.
32
H. Noser et al. / International Congress Series 1256 (2003) 29–34
adapted and optimized for a universal heuristic search in complex 3D environments, initially unknown for autonomous actors. A central part of the search method is the Find Path algorithm acting in the actor’s voxel memory, which has been described in an earlier publication [3]. Turn Around simply lets look around the actor’s vision system at its current position for perceiving and memorizing its neighborhood. The automaton L-system Control passes the control of the actor to a higher-level rule-based control mechanism. To control the automata behaviors of actors at the next higher level, we use a stack of command strings, which are interpreted by a special command string parser acting on the state variables of the actors. The stack-based control system allows an actor to create its own subgoals while executing the original script. With this concept, a user can compose new scenarios consisting of basic behaviors. At the highest level, actors can be scripted by rewriting rules of LWorld [6], a rewritingbased interactive animation system that includes the autonomous vision-based actors. It can read and visualize complex medical objects as surface models in 3D. Symbolic rules, as illustrated in Fig. 1, can configure actors and use automata as high-level primitives in user defined complex behaviors. As an example, we modeled a vision-based actor by rules,
Fig. 2. Software architecture for virtual endoscopy with automatic path searching for interactive navigation support.
H. Noser et al. / International Congress Series 1256 (2003) 29–34
33
Fig. 3. Found paths in two different aorta models. MRI data sets courtesy of Institute of Diagnostic Radiology, University Hospital Zurich.
capable of finding paths in complex, maze-like objects. Found paths may be exported as ASCII files and imported by other interactive applications such as MedIS-VR [1].
3. Results Fig. 2 illustrates the architecture of the system used for navigation support by automatic path searching within virtual medical organs. Automatic path finding and
Fig. 4. Found paths in a colon and in blood vessels of the brain. MRI data sets courtesy of Institute of Diagnostic Radiology, University Hospital Zurich. MRI data sets courtesy of Institute of Neuroradiology, University Hospital Zurich.
34
H. Noser et al. / International Congress Series 1256 (2003) 29–34
synthetic vision is embedded within LWorld and is configured and modeled by rules (Lsystems). Virtual Endoscopy sessions are performed in real-time employing MedIS-VR [1], a stateof-the-art application for 3D visualization and VR that combines Surface and Volume Rendering of medical data sets. To reconstruct 3D anatomical objects from MR and CT images, the well-known Marching-Cube algorithm for iso-surface generation is used. For testing the novel path searching method, the four examples of reconstructed medical organs illustrated in Figs. 3 and 4 were employed: Two aorta models a colon, and blood vessels from the brain region. In the first three organs, only a start and a goal point were necessary for the actor to find a path. In the blood vessels, however, an intermediary point was necessary because of the complex maze-like anatomical structure of the blood vessels. The lower middle image of Fig. 4 shows the voxelized visual memory of an actor with an intermediary path during the search process. The dark regions correspond to areas the actor has not seen yet. It looks for possible paths in the empty space according to its path search heuristics. The resulting paths are imported into the MedIS-VR system and can be used for guided fly-throughs facilitating the navigation task considerably.
4. Discussion Virtual Reality has shown to be a valuable tool in the field of medicine, especially when performing diagnostic tasks. Interactivity plays an important role when navigating and examining objects in 3D space. Automatic path finding facilitates navigation and the diagnostic task considerably. In this article, a new VR system based on a heuristic method for automatic path finding has been proposed. Due to its high information value, it is highly appreciated by radiologists and physicians. When considering interactive examination and automatic path searching, the aspect of user-friendliness plays a crucial role. Therefore, further research in 3D navigation and user-interface design is indispensable. It is also planned to incorporate new diagnostic activities and application fields. References [1] E. Stoll, Ch. Stern, S. Wildermuth, P. Stucki, A new filtering algorithm for medical MR- and CT-images, Journal of Digital Imaging 12 (1) (1999, February) 23. [2] Ch. Stern, S. Wildermuth, et al., Interactive definition of endoluminal aortic stent size and morphology based on virtual angioscopic rendering of 3D Magnetic Resonance Angiography (MRA), Proceedings of the Computer Aided Radiology and Surgery (CARS), 13th International Congress and Exhibition, 1999, June, pp. 176 – 180. [3] G. Yarong, et al., 3D skeleton for virtual colonoscopy, in: K.H. Ho¨hne, R. Kikinis (Eds.), Visualization in Biomedical Computing: 4th International Conference, Proceedings, Hamburg, Germany, September 22 – 25, 1996. [4] H. Noser, A Behavioral Animation System Based on L-systems and Synthetic Sensors for Actors, THE`SE No. 1609, E´cole Polytechnique Fe´de´rale de Lausanne (1997). [5] H. Noser, D. Thalmann, A rule-based interactive behavioral animation system for humanoids, IEEE Transactions on Visualization and Computer Graphics 5 (4) (1999, October – December). [6] H. Noser, Lworld: an animation system based on rewriting, Pacific Graphics 2002 Proceedings, 9 – 11 October, Beijing China, IEEE Computer Society, 2002, pp. 487 – 488.