- Email: [email protected]
“Intelligent volumes”: a new concept for representing spatial knowledge
May 1994
Pattern Recognition Letters Pattern Recognition Letters 15 (1994) 519-526
El SEVIER
"Intelligent volumes" : a new concept for representing spatial knowledge Thomas Schiemann*, Karl Heinz Hohne, Henning Kramer, Andreas Pommert, Martin Riemer, Rainer Schubert, Ulf Tiede Institute of Mathematics and Computer Science in Medicine (IMDM), University Hospital Eppendorf, MartinistraJ3e 52, 20246 Hamburg, Germany
Received 4 August 1993 ; revised 19 October 1993
Abstract For volume rendering with many different objects there is a need of comprehensive knowledge representation schemes . This paper describes a new data structure, which links spatial knowledge from tomographic image volumes with symbolic descriptions . Its application is shown with examples of 3D anatomical atlases . Key words:
Data structure ; Knowledge representation ; Volume visualization ; 3D anatomical atlas
1 . Introduction Different fields of computer science contribute to medical imaging . Image processing and computer graphics deal with the creation and enhancement of visual presentation of medical information . Its interpretation generally remains in the hands of the viewer . Thus the representation of semantics does not play a role there . In contrast to this, computer vision tries to extract semantical information from images. Despite numerous approaches there are no general operational methods fulfilling the requirements of clinical application yet . Therefore there was no need to develop representation schemes for the extracted information . Artificial intelligence, however, is just dealing with such knowledge representation schemes in great detail, but the link to the pictorial world has mostly been ignored. When going into the field of making volume-based ' Corresponding author, Em" : [email protected]
anatomical atlases, where not much work has been done so far (Mano et al ., 1990), we encountered the need of a spatial knowledge representation, which links image volume data with symbolic knowledge . This link has not much been considered by typical applications for representation of human anatomy so far (Robinson et al ., 1993 ; Wahler-Luck et al ., 1991 ; Lemoine et al ., 1991 ; Greitz et al ., 1991) . Thus we have developed a new scheme for the combined representation of the pictorial and symbolic knowledge of image volumes, that we preliminarily call "intelligent volume" . This paper describes its structure, the tools for its generation and exploration, and (despite of remaining problems) the advantages for building and exploring 3D anatomical atlases as well as for clinical 3D imaging . 2 . Method The "intelligent volume" data structure consists of two levels (Fig . 1) . The lower level contains the spa-
0167-8655/94/$07 .00 © 1994 Elsevier Science B .V . All rights reserved SSD10167-8655(93)E0098-9
520
T. Schiemann et aL /Pattern Recognition Letters 15 (1994) 519-526
Spatial knowledge
Intensity (CT,MAl, v T, I .. .)
o
Attribute volumes (elrumere , . . .)
Symbolic knowledge
Objects with Parameters, relations text, . . .
Diagram of the "intelligent volume" data structure . The spatial knowledge representation consists of the basic intensity volume and a number of attribute volumes defining the spatial extent of objects in different domains . The symbolic knowledge representation contains a description (names, relations, visualization parameters, textual descriptions,links to microscopic images. . . .) for every object identified in the attribute volumes . Fig . 1 .
tial knowledge, which is based on intensity volumes derived from tomographic imaging techniques like computer tomography (CT), magnetic resonance imaging (MRI), or positron emission tomography (PET) . If several imaging techniques are used together (e .g . MRI and PET),these have to be spatially registered, which is not a trivial task (van den Elsen et al., 1993) . Besides the intensity volumes there are one or more attribute volumes each for a different domain of knowledge . A voxel in an attribute volume is assigned a number for the object the voxel belongs to in this domain. In the world of medical atlases some possible domains are: • anatomical structure, • functional area, • blood supply area, • radiation dose. The upper level of the data structure is a symbolic knowledge representation, which is stored in a database and contains both further descriptive knowledge on the objects and the relations between them . Examples of descriptive knowledge are : • object names in different languages, • visualization parameters, • textual descriptions, • links to reference images (e.g . histological images, radiographs) . Relations are defined between objects, in order to structure the set of objects and to facilitate navigation among the hierarchy of objects during the following exploration session . It depends on the mean-
ing of the domain, which interpretation of a relation might be useful . In the current implementation in a specific domain the semantics of all relations is the same and may be interpreted as : • part of, • next to, • supplied by, • involved in function . The symbolic knowledge of the data structure is defined in a description language, that can be understood also by non-computer scientists and can therefore easily be maintained or modified. Fig. 2 shows examples for symbolic definitions of different items in the database with obvious meanings . The declaration of the numeric value ("id="), which represents an object in the corresponding attribute volume, is the only direct link between the symbolic and the spatial knowledge representation . Apart from this link the symbolic description is completely independent of the spatial knowledge. Thus, if a symbolic description of a part of the body has been defined, this description can be used as a generic knowledge base for any data set of the same body region . New data sets are first provided with a copy of the generic knowledge base. A copy of this generic knowledge base can then be extended or modified for an individual case in order to take care of pathologies, individual variabilities, or simply missing objects. After these modifications, the knowledge base has become specific for the particular case . Defining the spatial extent of the objects is a more lengthy step and has to be carried out for every case again . For this purpose we are using a very user friendly interactive 3D segmentation and volume editing system (Hohne and Hanson, 1992 ; Schiemann et a1 ., 1992) . Automatic segmentation methods are desirable, but currently there is no approach, that would succeed sufficiently independent of the input data . We doubt, whether this goal will be reached in the near future . The expense of segmentation with our interactive system depends on the objects that shall be defined . Gross anatomy (e .g . brain) can be obtained in less than 5 minutes from a state of the art MRI volume data set . Single detailed structures (e .g . gyri or basal ganglia of the brain) need up to half an hour, and a complete atlas, in which every voxel is classified, might take up to several man month in difficult cases .
T. Schiemann et a!. I Pattern Recognition Letters 15 (1994) 519-526 Define Object "gyrus frontalis medius sin" id=67 English="left middle temporal gyrus" German="linke mittlere Schlalenvindung" data=MRI shading=brain description=gyrus-frontalis-medius .dsc End Object
521
Define Shading "Brain" SurfaceNormal=GrayLevelGradient neighbours=26 Method=Phong Ambient=0 .05 Distance=0 .3 Diffuse Fraction=0 .70 Diffuse Color="Brain" Specular Fraction=0 .25 Specular Color="White" Sharpness=l0
Define Relation Parent="lobus frontalis s n Children=
End Shading
"gyrus precentralis sin", "gyrus frontalis superior sin", gyrus frontalis medius sin's , 'gyrus frontalis inferior sin", "gyri orbitales sin", "gyrus rectus sin", substantia perforata anterior
sin'
End Relation Fig. 2 . Detail of the knowledge base showing examples
for the
In order to obtain a filled volume model it is necessary to segment all structures as solid objects . Pure surface definition results in hollow objects, whose appearance and handling during later exploration contrast reality. After segmentation of an object, its spatial extent is saved in an attribute volume with an automatically chosen id, which is assigned to every voxel contained in the object. If the object is already described in the generic knowledge base, it is sufficient to specify the object name in order to link all available symbolic knowledge (possibly modified) to the new spatial definition . If the object is still missing in the generic knowledge base, its symbolic description can be added now .
3 . Application It is the essential advantage of a data structure like the "intelligent volume", that it can be explored by arbitrarily browsing through the symbolic and the pictorial context after the data structure is filled . The information content of the "intelligent volume" can be explored in basically three ways, which are not possible with classical image data structures : • Composition of views on the basis of conditions
definition of an object, a relation, and a shading method .
to the symbolic knowledge (e .g . show all structures supplied by the carotic artery), • Extraction of symbolic knowledge in the pictorial context (e .g. anatomical annotations of object names in the actual 3D view), • Procedural extraction of knowledge by simulation of medical techniques (e .g. show all objects along the path of an endoscope) . It is a decisive feature of our approach, that the knowledge about the objects to be visualized is exclusively contained in the "intelligent volume" and not in the exploration program (Fig . 3) . Thus visualization environments of any object may be created by just filling a new "intelligent volume" without changing the visualization program . In our implementation the "intelligent volume" is explored with the program VOXEL-MAN, which offers a large set of functions for interactive creation, manipulation, and exploration of 3D images in an intuitive way. The visualization system uses volume visualization methods, which have been developed in previous projects (Tiede et al ., 1990) . More detailed information on the implementation can be found in Hohne et al . (1992) . The visualization program is separated from the specific data by the following mechanism : A user request to the visualization program is transferred to a database management system, which uses the de-
T.
522
Schiemann et at /Pattern Recognition Letters 15 (1994) 519-526
Generic knowledge base
Interactive 3D segmentation
"Intelligent Volume" specific for this case Interactive
Volume editing
visual / symbolic
Knowledge editing
exploration (VOXEL-MAN)
Intensity volume (CT,MRI, . . .) Fig. 3. Structure of the whole image volume processing system . An image volume and a suitable generic knowledge base are fed into an interactive system for spatial and symbolic definition of objects, resulting in a filled "intelligent volume" data structure . The interactive visual and symbolic exploration system VOXEGMAN can then be applied to the specific data .
scription stored in the "intelligent volume" for composing the actual case specific instructions for the visualization program . For example, the command Modify object "eyes" color="EyeGreen" on the user level would result in the command Modify object id =17,18 color=rgb= (0 .2,0 .9,0 .3 ) on the visualization level . The "intelligent volume" concept has first been applied to the creation of a 3D anatomical atlas of the human brain (VOXELMAN/brain) based on an MRI volume of a head (Hohne et al ., 1992 ; Schubert et al ., 1993) . Three attribute volumes provide knowledge on the domains of anatomical structure (currently 176 objects), blood supply area (44 objects), and functional area (43 objects) . Fig. 4 shows the net of objects describing the domain of anatomical structure . Apart from visualization parameters and textual descriptions of the objects, there are also links to histological reference images stored in the descriptive part of the "intelligent volume" . The spatial definition of all structures of the brain took several months, as every object is subdivided in great detail . Fig . 5 shows a view of the workstation screen during a typical session with VOXELMAN/brain: At the top there is a main menu bar, from which several submenus can be selected . Two 3D images have been computed, a text description of a functional area is displayed and the net of objects of the domain of anatomical structure is shown . The 3D image on the left shows how the "intelligent volume" is explored from
the pictorial context : After mouse click the list of available domains appears at the cursor position . The chosen function (in this case "Describe") is applied to the object, which has been selected from the hierarchy of objects available in this domain . Both 3D images show the advantage of the filled volume model : Dissection can be simulated and knowledge can be inquired from the appearing cut planes . The complete set of VOXELMAN tools is described in Tiede et al . (1993 ) . Several other atlases have been generated with the "intelligent volume" concept so far . An atlas of the skull with currently 90 objects obtained from CT, and an atlas of a fetus with 130 objects obtained from MRI (Fig. 6) are as well in use as the brain atlas . Atlases of the heart (from MRI) and of the upper abdomen (from spiral-CT, Fig . 7) are under development . Work is also carried out for atlases used as references for acetabular fractures (Seebode et al ., 1993) and radiotherapy treatment planning (Schiemann et al ., 1993) . One might ask, if "intelligent volumes" could be used for knowledge based segmentation purposes . In principle, the "intelligent volume" data structure could store segmentation parameters like abstract descriptions of shape and location of objects for knowledge-based segmentation approaches similar to shading parameters for the visualization program . But in contrast to the volume visualization problem there are no sufficiently proven segmentation algorithms, which could use the knowledge contained in the "intelligent volume" . The symbolic part of the "intelli-
9 i sin
Fig. 4. Net describing the dornaln of anatomical structure using the "part of" r°I'
1
domain : anatomical structure
gyrus temporalis medius sin
gyms rectus sin
gyms ortitalies sin
gyrus frontalis medius sin
gyrus frontalis inferior sin
gyms frontalis superior sin
gyrus preoentralls sin
gyrus occipitotemporalis medlalis sin
uncus hippocampi sin gyrus occipitotemporalis lateralis sin
gyrus parahippocampalis sin
T. Schiemann et al. / Pattern Recognition Letters 15 (1994) 519-526
524
V
Stow wntwfnny .-. eAf ,
a
a
Fig. 5 . View of the workstation screen during a typical session with VOXEL-MAN/brain . Several 3D images can be explored simultaneously using a mouse and tools from different submenus . Objects can be addressed by clicking onto the images (popup-menu on the left 3D image) or by selection from the net of available objects (bottom right) . In this case a text description of a functional area has been obtained from the pictorial context (bottom left) .
gent volume" is nevertheless reusable for other cases, once the objects are segmented by whatever method . Naming an object after its segmentation then automatically causes the appropriate attributes like visualization parameters or descriptions to be attached to an individual case .
4. Conclusions We have shown that the "intelligent volume" data structure is a very useful tool for representing both pictorial and symbolic knowledge in a common scheme . As both the semantics and visualization fea-
lures are contained in the "intelligent volume", many different kinds of scenes may be visualized and explored with the same visual exploration environment . This has been shown with the example of anatomical atlases using the program VOXELMAN with "intelligent volumes" describing different parts of the human body. The functionality of such atlases exceeds by far that of existing anatomical data bases or hypermedia representations of human anatomy . It is the substantial advantage of the presented approach to enable an infinite number of different possibilities to represent the models knowledge, depending on the user's choice . Hypermedia systems allow navigation
T. Schiemann et at /Pattern Recognition Letters 15 (1994) 519-526
525
of the model allows to add any kind of knowledge, which can be correlated to a point in space, e .g . the contents of already existing anatomical databases and hypermedia applications . It is needless to say, that the approach may be applied for knowledge representation and exploration of any other complex volume object .
5. Acknowledgements We are grateful to the late Prof Lierse (Dept . of Neuroanatomy), Prof . Richter (Dept . of Pediatric Radiology), and all members of our department, who have supported this work . The original MRI image sequence of the head and the spiral-CT sequence of the abdomen were kindly provided by Siemens Medical Systems (Erlangen) .
6. References
Fig. 6 . 3D anatomical atlas of a fetus obtained from an MRI data set .
Greitz, T. . C . Bohm, S. Holte and L . Eriksson (1991 ) . A computerized brain atlas : Construction, anatomical content, and some applications . J. Comput . Assist. Tomogr . 15 (I ), 26-38 . Holing, K .H., M. Romans, M. Riemer, R . Schubert, U . Tiede and W . Lierse (I992) . .A 3D anatomical atlas based on a volume model. IEEE Comp t . Graphics Appl . 12 (4) 72-78 . Hohne, K .H . and W .A. Hanson (1992) . Interactive 3Dsegmentation of MR] and CT volumes using morphological operations . J. Content. Assist . Ti mogr, 16 (2) . 285-294 . Lemoine, D-, C . Barillot, B . Gibaud and E . Pasqualini (1991) . An anatomical-based 3d registration system of multimodality and atlas data in neurosurgery . Information Processing in Medical imaging, Proc.IPMI'91 . Springer, London, 154-164 . Mano, I ., Y . Suto, M . Suzuki and M . lie (1990) . Computerized three-dimensional normal atlas . Radial. Med. 8 (2), 50-54 . Robinson, G .P, A .C. Colchester and L.D. Griffin (1993) . Modelbased recognition of anatomical objects from medical images . Information Processing in Medical Imaging, Proc . IPMI'93 .
Springer, Berlin, 197-211 . Schiemann, T . . M . Bomans, U . Tiede and K.H . Hohne (t992) . Interactive 3D-segmentation . In : R.A . Robb, Ed ., Fig. 7 . First results of an anatomical atlas of an upper abdomen based on a spiral-CT data set . on a possibly high but fixed number of predefined images and textual descriptions only . The flexibility of the data structure will enable future extensions without substantial modifications of the system's structure . The space filling characteristic
Visualization in Biomedical Computing IL Chapel Hill . .AC. Proc . SPIE 1808,376-383 .
Schiemann, T .. B. Dippold . R . Schmidt, A . Pommert, M . Riemer, R . Schubert, U . Tiede and K.H . Hohne (1993) . 31) visualization for radiotherapy treatment planning . In : FLU . Lemkc, M .L . Rhodes, C .C. Jaffe and R . Felix, Eds ., Computer Assisted Radiology, Proc. CAR'93 . Springer, Berlin, 669-675 . Schubert, R_. K.H. Hohne, A . Pommert, M . Riemer, T. Schiemann and U . Tiede (1993) . Spatial knowledge representation for
526
T. Schiemann et al. /Pattern Recognition Letters 15 (1994) 519-526
visualization of human anatomy and function . In: H .H. Barrett and A .F . Gmitro, Eds., Information Processing in Medical Imaging, Proc. IPMI'9 3 . Springer, Berlin, 168-181 . Seebode, C ., R. Schubert, A. Pommert, M. Riemer, T . Schiemann, U . Tiede, V. Wening and K .H . Kohne (1993 ) . An interactive 3D-atlas of acetabular fractures . In: H .U . Lemke, M .L. Rhodes, C .C. Jaffe and R . Felix, Eds ., Computer Assisted Radiology, Proc. CAR'93 . Springer, Berlin, 716-721 . Tiede, U ., M. Bomans, K.H . Hohne, A. Pommert, M . Riemer, T. Schiemann, R . Schubert and W . Lierse (1993) . A computerized three-dimensional atlas of the human skull and brain. Am . J. Neuroradiology 14, 551-559 .
Tiede, U,, K .H. Hohne, M . Bomans, A. Pommert, M. Riemer and G- Wiebecke (1990) . Investigation of medical 3Drendering algorithms . IEEE Compul . Graphics Appl. 10 (2), 41-53 . van den Elsen, P .A., E.-J .D . Pol and M .A. Viergever (1993) . Medical image matching - a review with classification . IEEE Engng. Med. Biol. 12 (1), 26-39 . Wahler-Luck, M ., T . Schiitz and H :J . Kretschmann (1991) . A new anatomical representation of the human visual pathways . Graefe's Arch. Clin . Exp. Ophthalmol. 229, 201-205 .
Pattern Recognition Letters Pattern Recognition Letters 15 (1994) 519-526
El SEVIER
"Intelligent volumes" : a new concept for representing spatial knowledge Thomas Schiemann*, Karl Heinz Hohne, Henning Kramer, Andreas Pommert, Martin Riemer, Rainer Schubert, Ulf Tiede Institute of Mathematics and Computer Science in Medicine (IMDM), University Hospital Eppendorf, MartinistraJ3e 52, 20246 Hamburg, Germany
Received 4 August 1993 ; revised 19 October 1993
Abstract For volume rendering with many different objects there is a need of comprehensive knowledge representation schemes . This paper describes a new data structure, which links spatial knowledge from tomographic image volumes with symbolic descriptions . Its application is shown with examples of 3D anatomical atlases . Key words:
Data structure ; Knowledge representation ; Volume visualization ; 3D anatomical atlas
1 . Introduction Different fields of computer science contribute to medical imaging . Image processing and computer graphics deal with the creation and enhancement of visual presentation of medical information . Its interpretation generally remains in the hands of the viewer . Thus the representation of semantics does not play a role there . In contrast to this, computer vision tries to extract semantical information from images. Despite numerous approaches there are no general operational methods fulfilling the requirements of clinical application yet . Therefore there was no need to develop representation schemes for the extracted information . Artificial intelligence, however, is just dealing with such knowledge representation schemes in great detail, but the link to the pictorial world has mostly been ignored. When going into the field of making volume-based ' Corresponding author, Em" : [email protected]
anatomical atlases, where not much work has been done so far (Mano et al ., 1990), we encountered the need of a spatial knowledge representation, which links image volume data with symbolic knowledge . This link has not much been considered by typical applications for representation of human anatomy so far (Robinson et al ., 1993 ; Wahler-Luck et al ., 1991 ; Lemoine et al ., 1991 ; Greitz et al ., 1991) . Thus we have developed a new scheme for the combined representation of the pictorial and symbolic knowledge of image volumes, that we preliminarily call "intelligent volume" . This paper describes its structure, the tools for its generation and exploration, and (despite of remaining problems) the advantages for building and exploring 3D anatomical atlases as well as for clinical 3D imaging . 2 . Method The "intelligent volume" data structure consists of two levels (Fig . 1) . The lower level contains the spa-
0167-8655/94/$07 .00 © 1994 Elsevier Science B .V . All rights reserved SSD10167-8655(93)E0098-9
520
T. Schiemann et aL /Pattern Recognition Letters 15 (1994) 519-526
Spatial knowledge
Intensity (CT,MAl, v T, I .. .)
o
Attribute volumes (elrumere , . . .)
Symbolic knowledge
Objects with Parameters, relations text, . . .
Diagram of the "intelligent volume" data structure . The spatial knowledge representation consists of the basic intensity volume and a number of attribute volumes defining the spatial extent of objects in different domains . The symbolic knowledge representation contains a description (names, relations, visualization parameters, textual descriptions,links to microscopic images. . . .) for every object identified in the attribute volumes . Fig . 1 .
tial knowledge, which is based on intensity volumes derived from tomographic imaging techniques like computer tomography (CT), magnetic resonance imaging (MRI), or positron emission tomography (PET) . If several imaging techniques are used together (e .g . MRI and PET),these have to be spatially registered, which is not a trivial task (van den Elsen et al., 1993) . Besides the intensity volumes there are one or more attribute volumes each for a different domain of knowledge . A voxel in an attribute volume is assigned a number for the object the voxel belongs to in this domain. In the world of medical atlases some possible domains are: • anatomical structure, • functional area, • blood supply area, • radiation dose. The upper level of the data structure is a symbolic knowledge representation, which is stored in a database and contains both further descriptive knowledge on the objects and the relations between them . Examples of descriptive knowledge are : • object names in different languages, • visualization parameters, • textual descriptions, • links to reference images (e.g . histological images, radiographs) . Relations are defined between objects, in order to structure the set of objects and to facilitate navigation among the hierarchy of objects during the following exploration session . It depends on the mean-
ing of the domain, which interpretation of a relation might be useful . In the current implementation in a specific domain the semantics of all relations is the same and may be interpreted as : • part of, • next to, • supplied by, • involved in function . The symbolic knowledge of the data structure is defined in a description language, that can be understood also by non-computer scientists and can therefore easily be maintained or modified. Fig. 2 shows examples for symbolic definitions of different items in the database with obvious meanings . The declaration of the numeric value ("id="), which represents an object in the corresponding attribute volume, is the only direct link between the symbolic and the spatial knowledge representation . Apart from this link the symbolic description is completely independent of the spatial knowledge. Thus, if a symbolic description of a part of the body has been defined, this description can be used as a generic knowledge base for any data set of the same body region . New data sets are first provided with a copy of the generic knowledge base. A copy of this generic knowledge base can then be extended or modified for an individual case in order to take care of pathologies, individual variabilities, or simply missing objects. After these modifications, the knowledge base has become specific for the particular case . Defining the spatial extent of the objects is a more lengthy step and has to be carried out for every case again . For this purpose we are using a very user friendly interactive 3D segmentation and volume editing system (Hohne and Hanson, 1992 ; Schiemann et a1 ., 1992) . Automatic segmentation methods are desirable, but currently there is no approach, that would succeed sufficiently independent of the input data . We doubt, whether this goal will be reached in the near future . The expense of segmentation with our interactive system depends on the objects that shall be defined . Gross anatomy (e .g . brain) can be obtained in less than 5 minutes from a state of the art MRI volume data set . Single detailed structures (e .g . gyri or basal ganglia of the brain) need up to half an hour, and a complete atlas, in which every voxel is classified, might take up to several man month in difficult cases .
T. Schiemann et a!. I Pattern Recognition Letters 15 (1994) 519-526 Define Object "gyrus frontalis medius sin" id=67 English="left middle temporal gyrus" German="linke mittlere Schlalenvindung" data=MRI shading=brain description=gyrus-frontalis-medius .dsc End Object
521
Define Shading "Brain" SurfaceNormal=GrayLevelGradient neighbours=26 Method=Phong Ambient=0 .05 Distance=0 .3 Diffuse Fraction=0 .70 Diffuse Color="Brain" Specular Fraction=0 .25 Specular Color="White" Sharpness=l0
Define Relation Parent="lobus frontalis s n Children=
End Shading
"gyrus precentralis sin", "gyrus frontalis superior sin", gyrus frontalis medius sin's , 'gyrus frontalis inferior sin", "gyri orbitales sin", "gyrus rectus sin", substantia perforata anterior
sin'
End Relation Fig. 2 . Detail of the knowledge base showing examples
for the
In order to obtain a filled volume model it is necessary to segment all structures as solid objects . Pure surface definition results in hollow objects, whose appearance and handling during later exploration contrast reality. After segmentation of an object, its spatial extent is saved in an attribute volume with an automatically chosen id, which is assigned to every voxel contained in the object. If the object is already described in the generic knowledge base, it is sufficient to specify the object name in order to link all available symbolic knowledge (possibly modified) to the new spatial definition . If the object is still missing in the generic knowledge base, its symbolic description can be added now .
3 . Application It is the essential advantage of a data structure like the "intelligent volume", that it can be explored by arbitrarily browsing through the symbolic and the pictorial context after the data structure is filled . The information content of the "intelligent volume" can be explored in basically three ways, which are not possible with classical image data structures : • Composition of views on the basis of conditions
definition of an object, a relation, and a shading method .
to the symbolic knowledge (e .g . show all structures supplied by the carotic artery), • Extraction of symbolic knowledge in the pictorial context (e .g. anatomical annotations of object names in the actual 3D view), • Procedural extraction of knowledge by simulation of medical techniques (e .g. show all objects along the path of an endoscope) . It is a decisive feature of our approach, that the knowledge about the objects to be visualized is exclusively contained in the "intelligent volume" and not in the exploration program (Fig . 3) . Thus visualization environments of any object may be created by just filling a new "intelligent volume" without changing the visualization program . In our implementation the "intelligent volume" is explored with the program VOXEL-MAN, which offers a large set of functions for interactive creation, manipulation, and exploration of 3D images in an intuitive way. The visualization system uses volume visualization methods, which have been developed in previous projects (Tiede et al ., 1990) . More detailed information on the implementation can be found in Hohne et al . (1992) . The visualization program is separated from the specific data by the following mechanism : A user request to the visualization program is transferred to a database management system, which uses the de-
T.
522
Schiemann et at /Pattern Recognition Letters 15 (1994) 519-526
Generic knowledge base
Interactive 3D segmentation
"Intelligent Volume" specific for this case Interactive
Volume editing
visual / symbolic
Knowledge editing
exploration (VOXEL-MAN)
Intensity volume (CT,MRI, . . .) Fig. 3. Structure of the whole image volume processing system . An image volume and a suitable generic knowledge base are fed into an interactive system for spatial and symbolic definition of objects, resulting in a filled "intelligent volume" data structure . The interactive visual and symbolic exploration system VOXEGMAN can then be applied to the specific data .
scription stored in the "intelligent volume" for composing the actual case specific instructions for the visualization program . For example, the command Modify object "eyes" color="EyeGreen" on the user level would result in the command Modify object id =17,18 color=rgb= (0 .2,0 .9,0 .3 ) on the visualization level . The "intelligent volume" concept has first been applied to the creation of a 3D anatomical atlas of the human brain (VOXELMAN/brain) based on an MRI volume of a head (Hohne et al ., 1992 ; Schubert et al ., 1993) . Three attribute volumes provide knowledge on the domains of anatomical structure (currently 176 objects), blood supply area (44 objects), and functional area (43 objects) . Fig. 4 shows the net of objects describing the domain of anatomical structure . Apart from visualization parameters and textual descriptions of the objects, there are also links to histological reference images stored in the descriptive part of the "intelligent volume" . The spatial definition of all structures of the brain took several months, as every object is subdivided in great detail . Fig . 5 shows a view of the workstation screen during a typical session with VOXELMAN/brain: At the top there is a main menu bar, from which several submenus can be selected . Two 3D images have been computed, a text description of a functional area is displayed and the net of objects of the domain of anatomical structure is shown . The 3D image on the left shows how the "intelligent volume" is explored from
the pictorial context : After mouse click the list of available domains appears at the cursor position . The chosen function (in this case "Describe") is applied to the object, which has been selected from the hierarchy of objects available in this domain . Both 3D images show the advantage of the filled volume model : Dissection can be simulated and knowledge can be inquired from the appearing cut planes . The complete set of VOXELMAN tools is described in Tiede et al . (1993 ) . Several other atlases have been generated with the "intelligent volume" concept so far . An atlas of the skull with currently 90 objects obtained from CT, and an atlas of a fetus with 130 objects obtained from MRI (Fig. 6) are as well in use as the brain atlas . Atlases of the heart (from MRI) and of the upper abdomen (from spiral-CT, Fig . 7) are under development . Work is also carried out for atlases used as references for acetabular fractures (Seebode et al ., 1993) and radiotherapy treatment planning (Schiemann et al ., 1993) . One might ask, if "intelligent volumes" could be used for knowledge based segmentation purposes . In principle, the "intelligent volume" data structure could store segmentation parameters like abstract descriptions of shape and location of objects for knowledge-based segmentation approaches similar to shading parameters for the visualization program . But in contrast to the volume visualization problem there are no sufficiently proven segmentation algorithms, which could use the knowledge contained in the "intelligent volume" . The symbolic part of the "intelli-
9 i sin
Fig. 4. Net describing the dornaln of anatomical structure using the "part of" r°I'
1
domain : anatomical structure
gyrus temporalis medius sin
gyms rectus sin
gyms ortitalies sin
gyrus frontalis medius sin
gyrus frontalis inferior sin
gyms frontalis superior sin
gyrus preoentralls sin
gyrus occipitotemporalis medlalis sin
uncus hippocampi sin gyrus occipitotemporalis lateralis sin
gyrus parahippocampalis sin
T. Schiemann et al. / Pattern Recognition Letters 15 (1994) 519-526
524
V
Stow wntwfnny .-. eAf ,
a
a
Fig. 5 . View of the workstation screen during a typical session with VOXEL-MAN/brain . Several 3D images can be explored simultaneously using a mouse and tools from different submenus . Objects can be addressed by clicking onto the images (popup-menu on the left 3D image) or by selection from the net of available objects (bottom right) . In this case a text description of a functional area has been obtained from the pictorial context (bottom left) .
gent volume" is nevertheless reusable for other cases, once the objects are segmented by whatever method . Naming an object after its segmentation then automatically causes the appropriate attributes like visualization parameters or descriptions to be attached to an individual case .
4. Conclusions We have shown that the "intelligent volume" data structure is a very useful tool for representing both pictorial and symbolic knowledge in a common scheme . As both the semantics and visualization fea-
lures are contained in the "intelligent volume", many different kinds of scenes may be visualized and explored with the same visual exploration environment . This has been shown with the example of anatomical atlases using the program VOXELMAN with "intelligent volumes" describing different parts of the human body. The functionality of such atlases exceeds by far that of existing anatomical data bases or hypermedia representations of human anatomy . It is the substantial advantage of the presented approach to enable an infinite number of different possibilities to represent the models knowledge, depending on the user's choice . Hypermedia systems allow navigation
T. Schiemann et at /Pattern Recognition Letters 15 (1994) 519-526
525
of the model allows to add any kind of knowledge, which can be correlated to a point in space, e .g . the contents of already existing anatomical databases and hypermedia applications . It is needless to say, that the approach may be applied for knowledge representation and exploration of any other complex volume object .
5. Acknowledgements We are grateful to the late Prof Lierse (Dept . of Neuroanatomy), Prof . Richter (Dept . of Pediatric Radiology), and all members of our department, who have supported this work . The original MRI image sequence of the head and the spiral-CT sequence of the abdomen were kindly provided by Siemens Medical Systems (Erlangen) .
6. References
Fig. 6 . 3D anatomical atlas of a fetus obtained from an MRI data set .
Greitz, T. . C . Bohm, S. Holte and L . Eriksson (1991 ) . A computerized brain atlas : Construction, anatomical content, and some applications . J. Comput . Assist. Tomogr . 15 (I ), 26-38 . Holing, K .H., M. Romans, M. Riemer, R . Schubert, U . Tiede and W . Lierse (I992) . .A 3D anatomical atlas based on a volume model. IEEE Comp t . Graphics Appl . 12 (4) 72-78 . Hohne, K .H . and W .A. Hanson (1992) . Interactive 3Dsegmentation of MR] and CT volumes using morphological operations . J. Content. Assist . Ti mogr, 16 (2) . 285-294 . Lemoine, D-, C . Barillot, B . Gibaud and E . Pasqualini (1991) . An anatomical-based 3d registration system of multimodality and atlas data in neurosurgery . Information Processing in Medical imaging, Proc.IPMI'91 . Springer, London, 154-164 . Mano, I ., Y . Suto, M . Suzuki and M . lie (1990) . Computerized three-dimensional normal atlas . Radial. Med. 8 (2), 50-54 . Robinson, G .P, A .C. Colchester and L.D. Griffin (1993) . Modelbased recognition of anatomical objects from medical images . Information Processing in Medical Imaging, Proc . IPMI'93 .
Springer, Berlin, 197-211 . Schiemann, T . . M . Bomans, U . Tiede and K.H . Hohne (t992) . Interactive 3D-segmentation . In : R.A . Robb, Ed ., Fig. 7 . First results of an anatomical atlas of an upper abdomen based on a spiral-CT data set . on a possibly high but fixed number of predefined images and textual descriptions only . The flexibility of the data structure will enable future extensions without substantial modifications of the system's structure . The space filling characteristic
Visualization in Biomedical Computing IL Chapel Hill . .AC. Proc . SPIE 1808,376-383 .
Schiemann, T .. B. Dippold . R . Schmidt, A . Pommert, M . Riemer, R . Schubert, U . Tiede and K.H . Hohne (1993) . 31) visualization for radiotherapy treatment planning . In : FLU . Lemkc, M .L . Rhodes, C .C. Jaffe and R . Felix, Eds ., Computer Assisted Radiology, Proc. CAR'93 . Springer, Berlin, 669-675 . Schubert, R_. K.H. Hohne, A . Pommert, M . Riemer, T. Schiemann and U . Tiede (1993) . Spatial knowledge representation for
526
T. Schiemann et al. /Pattern Recognition Letters 15 (1994) 519-526
visualization of human anatomy and function . In: H .H. Barrett and A .F . Gmitro, Eds., Information Processing in Medical Imaging, Proc. IPMI'9 3 . Springer, Berlin, 168-181 . Seebode, C ., R. Schubert, A. Pommert, M. Riemer, T . Schiemann, U . Tiede, V. Wening and K .H . Kohne (1993 ) . An interactive 3D-atlas of acetabular fractures . In: H .U . Lemke, M .L. Rhodes, C .C. Jaffe and R . Felix, Eds ., Computer Assisted Radiology, Proc. CAR'93 . Springer, Berlin, 716-721 . Tiede, U ., M. Bomans, K.H . Hohne, A. Pommert, M . Riemer, T. Schiemann, R . Schubert and W . Lierse (1993) . A computerized three-dimensional atlas of the human skull and brain. Am . J. Neuroradiology 14, 551-559 .
Tiede, U,, K .H. Hohne, M . Bomans, A. Pommert, M. Riemer and G- Wiebecke (1990) . Investigation of medical 3Drendering algorithms . IEEE Compul . Graphics Appl. 10 (2), 41-53 . van den Elsen, P .A., E.-J .D . Pol and M .A. Viergever (1993) . Medical image matching - a review with classification . IEEE Engng. Med. Biol. 12 (1), 26-39 . Wahler-Luck, M ., T . Schiitz and H :J . Kretschmann (1991) . A new anatomical representation of the human visual pathways . Graefe's Arch. Clin . Exp. Ophthalmol. 229, 201-205 .