- Email: [email protected]
Imaging and graphics in medicine: concept of an object-oriented platform for clinical research
computer methods and programs in binmedicine
ELSE‘VIER
Computer Methods and Programs in Biomedicine 48 (1995) 157- 162
Imaging and graphics in medicine: concept of an object-oriented platform for clinical research Hans-Heino
Ehricke”, Thomas G,runert”, Thomas Buckb, Jiirgen Fechterb, Uwe Kloosb, Wolfgang StraBerb, Rupert Kolb
“Department of’ Electrical “Computer Graphics Lab, University ‘Deparrnent of’ Neuror-adiology,
Engineering. of Tiibingen, Radiological
Polytechnical Scllool of Stralsund, Stralsund, Germany :~illleln?-S~/Iickard-Institute fbr Inforrnatics; Tiibingen, Gennarr? Clinic, University Hospitals of Tiibingen, Tiibingen, Ge,many
Abstract We present a workstation-based research platform with two major components. A turnkey application system provides a functionality kernel for a broad community of clinical users with an interest in digital imaging. A development toolbox allows efficient implementation of reseanzhideasand consistentintegration of new applications with the common framework of the turnkey system.. The platform is based on an elaborate object class structure describingobjectsfor imageprocessing,computergraphics,study handlingand userinterface control. Thus expertise of computer scientists familiar with this application domain is brought into the hospital and can be readily used by
clinilzal researchers. Keywork:
Image processing;
User interface
1. Introduction Due to the interdisciplinary nature of Medical Informatics (MI) the efficient communication between clinicians and computer scientists is a key element of fruitful research. However, many unsolved MI problems give evidence of the fact that researchers have not succeededin integrating the technical and medical aspects of the two underlying worlds into consistent concepts. This argu-
* Corresponding author.
ment is true in a variety of application areas including Medical Imaging and Graphics. Here computer scientists have focused, e.g., on highlycomplex graphics algorithms leading to beautiful three-dimensional presentations of anatomic objects. On the other hand their clinical usefulness has neither been evaluated nor proven. In this paper we present the concept of a software tool system which constitutes a research platform promoting the interdisciplinary exchange of ideas in Medical Imaging and Graphics. Our approach is based on a procedural model observable in this field (see Fig. 1). Typically a new
0169-260’$95/SO9.50 0 1995 Elsevier Science Ireland Ltd. All rights reserved SSDI 0169-2607(95)01677-L
158
H.-N.
Elwicke
ct cd. / Cor?~puter
Mrtltods
clinical application is triggered either by a computer scientist on the basis of a novel algorithm, or by a medical scientist who has a solution idea for a clinical problem. In order to arrive at a clinically relevant solution both need input from each other. Computer scientists need knowledge about, e.g., clinical requirements, the environment to integrate their methods and the method’s significance. On the other hand the clinical researcher is interested in available algorithms and their defciencies and advantages with respect to his problem. The primary goal of the concept proposed in this paper is to promote the process of finding the relevant overlap between solution and .application ideas, thus providing clinically relevant solutions. 2. Related work Clinical
researchers have spent a tremendous of time developing new applications from scratch on standalone systems or integrating them into modality consoles. Investigating the Medical Imaging and Graphics literature we find, besides an overwhelming variety of algorithmic ideas, a small number of proposals for clinical application systems. Principally they fall into two categories: 0 Ready-to-use application systems e Application development toolboxes amount
Fig. 1. Procedural model for medical and computer scientists.
the commu~icatiol~
between
ad
Program
in Bionzeiiicbte
48 (1995)
157-
16.2
The former offer a number of graphics and image processing methods under a more or less comfortable user interface [I]. Many of these systems are already commercially available, in most cases as software + hardware solutions (e.g. Montron Vision, Kontron, Eching; ISG Allegro, ISG Techn. Inc., Toronto). In view of clinical research their primary drawback is the lack of extensibility. New algorithms and applications cannot be consistently integrated which for a research system is art intolerable feature. Application deveiopnlent systems on the other hand, aim at the creation of new applications by quick and graphically supported assembly of available software modules [2-41. Some of them allow for the integration of self-written software modules. Since they are not dedicated specifically to the medical field they do not offer a user interface for clinical routine use. Moreover they lack a turnkey component providing base functionality which is necessary for any application. A few clinically oriented research groups have proposed application systems with a focus on user interface design and functionality [.5-81. Some of them try to achieve system extensibility by providing source code and/or a functions library for application development, but this is not the main subject of their work. 3. Concept of a medical research workstation Although our concept of a medical graphics workstation focuses on application development support, we have based it on a turnkey application system (see Fig. 2). This provides user interface, base functionality and applications, which are typically found as part of a routine system (e.g. a modality console) and serves as a common framework for application development. We are aware of the fact that especially for legal reasons a clear distinction between routine and research equipment is desirable. However, the following observations justify our concept of a research system with i~ltegrated routine features: 1. Many solution ideas expressed by clinicians focus on extensions of available methods. 2. Providing an environment similar to a routine system makes it easier to become familiar with a research platform.
H.-H.
Fig. 2. Architectural development system !ibraly.
Ehicke
et cd. / Computer
Methods
system overview: The turnkey and the are based on a common object class
3. Integrating novel applications under a common user interface increases consistency and avc’id getting lost in a variety of different interfaces. 4. The cooperation between different functions and applications within a common environment increases software reusability. The basis for the application system as well as the development tool box of the proposed platform is an elaborate object class structure. Perhaps one of the most promising characteristics of the object-orientation paradigm is its know-how transfer capability. Let us briefly explain this statement: The design of an adequate object class library structure requires a great deal of knowledge and experience in the problem domain. Thus, an object class library does not merely provide functionality as, e.g., a functions library. By the configuration of object classes, their mel:hods and attributes and the class hierarchy, a great deal of expertise with respect to the problem domain is represented. This can be directly used by an application developer. Unlike a functions library an object class structure provides an efficient mechanism for the modeling of realworld objects by a software system. Thus, a problem-adequate software structure can be designed, even by a non-expert in the application domain. We propose an object class library falling into two categories:
and Programs
in Biomrdicine
48 (1995)
157-162
159
1. Objects interfacing the turnkey application system (management objects). 2. Objects for algorithmic research and application development (image processing and graphics objects). The latter allow for an easy and quick application development by providing structure and functionality of most real-world objects in the area of Medical Imaging and Graphics. Examples are a 30-renderer for the three-dimensional visualization of volumetric datasets, a digital filter for the enhancemet of images, e.g. by noise suppression and a registrator for matching multi-modal datasets. A consistent integration of new applications into the turnkey system may be performed by use of the management objects. These provide rnethods for user interface accessas well as study handling. 4. Turnkey
imaging and graphics system
4.1. Functionality speczjication As already discussedin the previous chapter we regard the turnkey part of our platform as a means of providing base functionality and a common framework for the consistent integration of novel algorithms and applications. One of the primary goals of the system is to address a broad community of clinical clients who use digital images as an information medium. Particularly by t.he advent of new imaging m.odalities, such as Digital Subtraction Angiography (DSA), Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), the penetration of radiology departments with digital images has considerably increased. Besides radiology a variety of clinical disciplines are increasingly equipped with digital imaging devices. Examples are: ID Opthalmology (laser-scan images of the retina) @ Nuclear medicine (scintigraphs, Positron Emission Tomography) 10 Cardiology (digital ultrasound) digital @ Anatomy (scanned photographs, anatomic atlas) 10Pathology (digital microscopy) The list may be extended for example by those departments in a hospital which are traditionally radiology customers, but have an interest in post-
160
H.-H.
Elwicke
et al. / Computer
Met1lod.x
urd
processing the acquired image data for therap:y planning purposes, e.g. neurosurgery and orthopedics. Therefore, a requirements analysis for ,a medical workstation has to cover various disciplines. We performed such an analysis at the IJniversi-ty Hospitals of Tubingen. Our results document an astonishingly large overlap of the requirements of the investigated disciplines. From this we h.ave derived a funtionality specification of a universal medical graphics workstation. The focus here is not on exotic applications which are of interest only for a small group of specialists, but on a universal functionality kernel. We distinguish between five functionality categories: 1. Access: Image storage/retrieval, data compression, interpretation of file formats (esp. ACRNEMA, DI-COM), study handling, multiple image display. 2. Manipulation: Image processing operations (e.g. zoom, pan mirror, contrast/brightness adjustment, negate, filter, arithmetics). 3. Evaluation: Local/global greyvalue statistics and geometric properties (2D/3D distance, angle, profile) 4. Documentation: Image annotation and hardcopy. 5. Analysis: Advanced image processing and graphics methods (e.g. segmentation, registration. classification, multiplanar reconstruction, volume rendering). 4.2. The lightbox metaphor for user interface design Especially in Medical Informatics the success of a new system largely depends on the adequacy of its m.an-machine interface. A possible strategy in user interface design is the analysis of the way physicians deal with real-world objects which will have a representation within the software system under development. In the context of Medical Imaging and Graphics real-world objects familiar to physicians are the lightbox and X-ray films. They are by far the most common media for viewing medical images, even if those were acquired by digital devices. Why is the lightbox the most successful viewing station in medicine? Besides many other reasons we may identify some facts related to user interface design:
Pro~n~tns
in Biornrclicine
48 (1995)
157-163
1. It is quite easy to view radiographs by placing them arbitrarily to a lightbox. 2. A lightbox can hold various images of a patient and thus provide an excellent overview. 3. Switching between overview and detailed view is possible within a timeframe of milliseconds, just by head and eye movement. 4. The configuration of radiographs on a lightbox can quickly be reorganized. We used these lightbox characteristics as a guideline to user interface design of our system. An overview of the proposed UI presentation layer is given by Fig. 3. The workstation screen is used as a whole for image display. Only a small part is reserved as a menu field. Actions are triggered via mouse input. In ana.logy to the characteristics observed with the conventional counterpart of our system, the following look and feel is proposed: 1. An image or a patient study can easily be selected from the database pane11 and fixed to an arbitrary lightbox segment via a drag and drop mechanism. 7 The digital light box can simultaneously display up to 16 images from several patient studies. Unrestricted maneuvering on the lightbox panel is performed by realtime zoom and pan, thus controlling the panel section actually displayed on the screen and its spatial resolution. The logical link between images and lightbox segments can be graphically reorganized on a lightbox overview icon.
Fig. 3. User interface
presentation
layer
of the turnkey
system.
Although we have based the design of our user interface on the lightbox metaphor we have not restricted ourselves to a mere X)-medium. Many additional features, such as fast skimming through an image series, arbitrary slicing through a volume dataset or projecting a volume onto a plane, enhance the interpretation process. 5. Extension
interface and development toolbox
A Medical Imaging and Graphics workstation suitable for clinical research must contain mechanisms ‘or application development support and integration of new algorithms. Our concept of an extensible platform is based on the following observation: A mere toolsystem without a common integration framework will lead to the development of many standalone applications. This seems not very efficient because base functionality elements necessary for any application, like loading patient data, interpretation of file formats or display management are not reused. Moreover, the creation of different user interfaces will result in a great deal of learning efforts of the medical staff in order to get familiar with a new application. A third problem is the great overhead necessary for data import and export if two applications are to be combined in a certain case. For these reasons we propose a common integration framework which is given by the already described turnkey appiication system and an elaborate extension interface. Fig. 4 illustrates the mechanism of integrating a new application into the turnkey system. Usually a new application is integrated into the workstation interface as an iconized application button within the menu field. By pressing this button the user invokes the application and, e.g., a popup-menu is displayed. From the programmer’s view two object classes play an important role here: User interface control classes 0 Patient study handling classes Let us briefly explain their meaning: User interface control objects manage the behaviour and look of the common user interface. They contain methods, e.g., for enabling/disabling menu buttons, mouse drawing of a polygon within an image and displaying an image at a certain screen
Fig. 4. Principle of integrating turnkey workstation.
a new
application
into
the
location. A patient study may contain textual (elements (e.g. patient name, medical report, ,anamnesis), graphical objects, images and audio. Study handling classes provide methods for ac‘cessing and manipulating these elements and their hierarchical structure. For example image data may be accessed by an application using a get-image or get-pixel-value method. New images may be displayed by transfering the data from the application object to a socalled result study which is connected to an image display segment on the screen. In this way display management is performed by the turnkey system and the application programmer does not have to bother for this highly complex task. Besides this integration interface the application builder toolbox is another key element of our development system. Its basis is an object class library which covers a broad spectrum of Medical Imaging and Graphics functionality. Its structure represents a great deal of software development expertise in this area and therefore allows inexperienced application programmers to quickly arrive at a professional program design. The class library contains a variety of C+ + classes developed at our institute and integrates a commercial system (Imaging Applications Platform, ISG Tee., Toronto) which we found to be of good adequacy for this purpose. The library encompasses objects and methods for image file format handling, image memory management, digital filtering, pixel
163
N.-H.
Eizricke
et al. 1 Computer
Methods
and Progmrm
value statistics, raycasting, dataset registration, geometric modeling, segmentation, surface reconstruction and many more. A detailed description of the object class structure is beyond the scope of this paper. Our design was guided by the endeavour to support application development as well as algorithmic research. Especially the latter led to a sophisticated structure with many levels in the inheritance hierarchy for a number of classes, thus providing high flexibility. 6.. Comcfosions In collaboration with the University Hospitals of Tiibingen, Eye Clinic, Radiological Clinic and Surgical Clinic, an evaluation of both components of the platform was performed. Our evaluation results suggest that clinical research related to digital imaging and graphics benefits from the availability of the toolkit system in three ways: First, the path from a research idea to its implementation and evaluation is shortened. Second, novel applications are motivated by the system and the realization of complex applications be.” comes possible. Third, the communication between clinical researchers and computer scientists is supported by the platform. However, the high level of programming knowledge necessary to efficiently use our toolbox, has been an obstacle for addressing a broader community of clinical researchers. Excellent programming skills are not widely found in a clinical environment. A solution would tie a graphical programming interface which we know from commercial user interface builders or visual programming tools, such as AVS (St~~rdent Inc., Concord) and Explorer (Silicon Graphics, mountain View). We decided to
in Biomedicine
48 (1995)
157-162
design a graphical object assembly editor allowing both, user interface and functionality of an application to be visually programmed and represented as an object network on the workstation screen. References [II R.A. Robb and C. Barillot, Interactive display and analysis of 3D medical images. IEEE Trans. Med. Imag., 8(3) (1989) 217--226. [2J C. Upson, T. Faulhaber, D. Kamins, D. Laidlaw, D. Schlegel, J. Vroom, R. Gurwitz, and A. van Dam, The application visualization system: A computational environment for scientific visualization. IEEE Computer Graphics and Applications 9(4) (1989) 30-42. [3] D.S. Dyer, A dataflow toolkit for visualization. IEEE Computer Graphics and Applications IO (1990) 60-69. [4] P. Heffernan and D. Dekel, Imaging applicatiol~s platform: Concept to implementation. In R.A. Robb. editor. Visualization in Biomedical Computing. pp. 495 -509. Bellingham 1992. SPIE. [S] B.K.T. Ho, 0. Ratib and S.C. Horii, PACS workstation design. Computerized Medical Imaging and Graphics 15(3) (1991)
147-155.
K.-H. Englmeier, T. Hilbertz, and U. Fink. Computer-assisted interaction with and manipulation of multi-modal images in medicine. In Medical Informatics Europe 1993, pp. 347, 354. [7] M. Staemmler. R. Brill, K. Becker, K.-H. Polkerts and K. Gersonde, in SUNRISE: a software system for medical imaging analysis, eds. H.U. Lemke. M.L. Rhodes, CC. Jaffe and R. Felix, Computer Assisted Radiology 1989, pp. 671-677, Berlin, Heidelberg, New York, 1989. Springer. [8] M. Dahm, K. Giaser, H. Jansen-Dittmer, A. Keizers, P. Krueger, ct. Meyer-~br~ht. K. ~~nker-Ka~ipp, H. Rudolf, C. Schilling, R. Sieslack, W. Winkler, B. Wein and R. Giinther, in Radiologists start designing their digital workplace: prototyping with a digital image workstation in the radiology, eds. H.U. Lemke, M.L. Rhodes, C.C. Jaffe and R. Felix, Computer Assisted Radiology 9 1, pp. 699-704, Berlin, Heidelberg, New York, 1991. Springer. [B]
ELSE‘VIER
Computer Methods and Programs in Biomedicine 48 (1995) 157- 162
Imaging and graphics in medicine: concept of an object-oriented platform for clinical research Hans-Heino
Ehricke”, Thomas G,runert”, Thomas Buckb, Jiirgen Fechterb, Uwe Kloosb, Wolfgang StraBerb, Rupert Kolb
“Department of’ Electrical “Computer Graphics Lab, University ‘Deparrnent of’ Neuror-adiology,
Engineering. of Tiibingen, Radiological
Polytechnical Scllool of Stralsund, Stralsund, Germany :~illleln?-S~/Iickard-Institute fbr Inforrnatics; Tiibingen, Gennarr? Clinic, University Hospitals of Tiibingen, Tiibingen, Ge,many
Abstract We present a workstation-based research platform with two major components. A turnkey application system provides a functionality kernel for a broad community of clinical users with an interest in digital imaging. A development toolbox allows efficient implementation of reseanzhideasand consistentintegration of new applications with the common framework of the turnkey system.. The platform is based on an elaborate object class structure describingobjectsfor imageprocessing,computergraphics,study handlingand userinterface control. Thus expertise of computer scientists familiar with this application domain is brought into the hospital and can be readily used by
clinilzal researchers. Keywork:
Image processing;
User interface
1. Introduction Due to the interdisciplinary nature of Medical Informatics (MI) the efficient communication between clinicians and computer scientists is a key element of fruitful research. However, many unsolved MI problems give evidence of the fact that researchers have not succeededin integrating the technical and medical aspects of the two underlying worlds into consistent concepts. This argu-
* Corresponding author.
ment is true in a variety of application areas including Medical Imaging and Graphics. Here computer scientists have focused, e.g., on highlycomplex graphics algorithms leading to beautiful three-dimensional presentations of anatomic objects. On the other hand their clinical usefulness has neither been evaluated nor proven. In this paper we present the concept of a software tool system which constitutes a research platform promoting the interdisciplinary exchange of ideas in Medical Imaging and Graphics. Our approach is based on a procedural model observable in this field (see Fig. 1). Typically a new
0169-260’$95/SO9.50 0 1995 Elsevier Science Ireland Ltd. All rights reserved SSDI 0169-2607(95)01677-L
158
H.-N.
Elwicke
ct cd. / Cor?~puter
Mrtltods
clinical application is triggered either by a computer scientist on the basis of a novel algorithm, or by a medical scientist who has a solution idea for a clinical problem. In order to arrive at a clinically relevant solution both need input from each other. Computer scientists need knowledge about, e.g., clinical requirements, the environment to integrate their methods and the method’s significance. On the other hand the clinical researcher is interested in available algorithms and their defciencies and advantages with respect to his problem. The primary goal of the concept proposed in this paper is to promote the process of finding the relevant overlap between solution and .application ideas, thus providing clinically relevant solutions. 2. Related work Clinical
researchers have spent a tremendous of time developing new applications from scratch on standalone systems or integrating them into modality consoles. Investigating the Medical Imaging and Graphics literature we find, besides an overwhelming variety of algorithmic ideas, a small number of proposals for clinical application systems. Principally they fall into two categories: 0 Ready-to-use application systems e Application development toolboxes amount
Fig. 1. Procedural model for medical and computer scientists.
the commu~icatiol~
between
ad
Program
in Bionzeiiicbte
48 (1995)
157-
16.2
The former offer a number of graphics and image processing methods under a more or less comfortable user interface [I]. Many of these systems are already commercially available, in most cases as software + hardware solutions (e.g. Montron Vision, Kontron, Eching; ISG Allegro, ISG Techn. Inc., Toronto). In view of clinical research their primary drawback is the lack of extensibility. New algorithms and applications cannot be consistently integrated which for a research system is art intolerable feature. Application deveiopnlent systems on the other hand, aim at the creation of new applications by quick and graphically supported assembly of available software modules [2-41. Some of them allow for the integration of self-written software modules. Since they are not dedicated specifically to the medical field they do not offer a user interface for clinical routine use. Moreover they lack a turnkey component providing base functionality which is necessary for any application. A few clinically oriented research groups have proposed application systems with a focus on user interface design and functionality [.5-81. Some of them try to achieve system extensibility by providing source code and/or a functions library for application development, but this is not the main subject of their work. 3. Concept of a medical research workstation Although our concept of a medical graphics workstation focuses on application development support, we have based it on a turnkey application system (see Fig. 2). This provides user interface, base functionality and applications, which are typically found as part of a routine system (e.g. a modality console) and serves as a common framework for application development. We are aware of the fact that especially for legal reasons a clear distinction between routine and research equipment is desirable. However, the following observations justify our concept of a research system with i~ltegrated routine features: 1. Many solution ideas expressed by clinicians focus on extensions of available methods. 2. Providing an environment similar to a routine system makes it easier to become familiar with a research platform.
H.-H.
Fig. 2. Architectural development system !ibraly.
Ehicke
et cd. / Computer
Methods
system overview: The turnkey and the are based on a common object class
3. Integrating novel applications under a common user interface increases consistency and avc’id getting lost in a variety of different interfaces. 4. The cooperation between different functions and applications within a common environment increases software reusability. The basis for the application system as well as the development tool box of the proposed platform is an elaborate object class structure. Perhaps one of the most promising characteristics of the object-orientation paradigm is its know-how transfer capability. Let us briefly explain this statement: The design of an adequate object class library structure requires a great deal of knowledge and experience in the problem domain. Thus, an object class library does not merely provide functionality as, e.g., a functions library. By the configuration of object classes, their mel:hods and attributes and the class hierarchy, a great deal of expertise with respect to the problem domain is represented. This can be directly used by an application developer. Unlike a functions library an object class structure provides an efficient mechanism for the modeling of realworld objects by a software system. Thus, a problem-adequate software structure can be designed, even by a non-expert in the application domain. We propose an object class library falling into two categories:
and Programs
in Biomrdicine
48 (1995)
157-162
159
1. Objects interfacing the turnkey application system (management objects). 2. Objects for algorithmic research and application development (image processing and graphics objects). The latter allow for an easy and quick application development by providing structure and functionality of most real-world objects in the area of Medical Imaging and Graphics. Examples are a 30-renderer for the three-dimensional visualization of volumetric datasets, a digital filter for the enhancemet of images, e.g. by noise suppression and a registrator for matching multi-modal datasets. A consistent integration of new applications into the turnkey system may be performed by use of the management objects. These provide rnethods for user interface accessas well as study handling. 4. Turnkey
imaging and graphics system
4.1. Functionality speczjication As already discussedin the previous chapter we regard the turnkey part of our platform as a means of providing base functionality and a common framework for the consistent integration of novel algorithms and applications. One of the primary goals of the system is to address a broad community of clinical clients who use digital images as an information medium. Particularly by t.he advent of new imaging m.odalities, such as Digital Subtraction Angiography (DSA), Computed Tomography (CT) and Magnetic Resonance Imaging (MRI), the penetration of radiology departments with digital images has considerably increased. Besides radiology a variety of clinical disciplines are increasingly equipped with digital imaging devices. Examples are: ID Opthalmology (laser-scan images of the retina) @ Nuclear medicine (scintigraphs, Positron Emission Tomography) 10 Cardiology (digital ultrasound) digital @ Anatomy (scanned photographs, anatomic atlas) 10Pathology (digital microscopy) The list may be extended for example by those departments in a hospital which are traditionally radiology customers, but have an interest in post-
160
H.-H.
Elwicke
et al. / Computer
Met1lod.x
urd
processing the acquired image data for therap:y planning purposes, e.g. neurosurgery and orthopedics. Therefore, a requirements analysis for ,a medical workstation has to cover various disciplines. We performed such an analysis at the IJniversi-ty Hospitals of Tubingen. Our results document an astonishingly large overlap of the requirements of the investigated disciplines. From this we h.ave derived a funtionality specification of a universal medical graphics workstation. The focus here is not on exotic applications which are of interest only for a small group of specialists, but on a universal functionality kernel. We distinguish between five functionality categories: 1. Access: Image storage/retrieval, data compression, interpretation of file formats (esp. ACRNEMA, DI-COM), study handling, multiple image display. 2. Manipulation: Image processing operations (e.g. zoom, pan mirror, contrast/brightness adjustment, negate, filter, arithmetics). 3. Evaluation: Local/global greyvalue statistics and geometric properties (2D/3D distance, angle, profile) 4. Documentation: Image annotation and hardcopy. 5. Analysis: Advanced image processing and graphics methods (e.g. segmentation, registration. classification, multiplanar reconstruction, volume rendering). 4.2. The lightbox metaphor for user interface design Especially in Medical Informatics the success of a new system largely depends on the adequacy of its m.an-machine interface. A possible strategy in user interface design is the analysis of the way physicians deal with real-world objects which will have a representation within the software system under development. In the context of Medical Imaging and Graphics real-world objects familiar to physicians are the lightbox and X-ray films. They are by far the most common media for viewing medical images, even if those were acquired by digital devices. Why is the lightbox the most successful viewing station in medicine? Besides many other reasons we may identify some facts related to user interface design:
Pro~n~tns
in Biornrclicine
48 (1995)
157-163
1. It is quite easy to view radiographs by placing them arbitrarily to a lightbox. 2. A lightbox can hold various images of a patient and thus provide an excellent overview. 3. Switching between overview and detailed view is possible within a timeframe of milliseconds, just by head and eye movement. 4. The configuration of radiographs on a lightbox can quickly be reorganized. We used these lightbox characteristics as a guideline to user interface design of our system. An overview of the proposed UI presentation layer is given by Fig. 3. The workstation screen is used as a whole for image display. Only a small part is reserved as a menu field. Actions are triggered via mouse input. In ana.logy to the characteristics observed with the conventional counterpart of our system, the following look and feel is proposed: 1. An image or a patient study can easily be selected from the database pane11 and fixed to an arbitrary lightbox segment via a drag and drop mechanism. 7 The digital light box can simultaneously display up to 16 images from several patient studies. Unrestricted maneuvering on the lightbox panel is performed by realtime zoom and pan, thus controlling the panel section actually displayed on the screen and its spatial resolution. The logical link between images and lightbox segments can be graphically reorganized on a lightbox overview icon.
Fig. 3. User interface
presentation
layer
of the turnkey
system.
Although we have based the design of our user interface on the lightbox metaphor we have not restricted ourselves to a mere X)-medium. Many additional features, such as fast skimming through an image series, arbitrary slicing through a volume dataset or projecting a volume onto a plane, enhance the interpretation process. 5. Extension
interface and development toolbox
A Medical Imaging and Graphics workstation suitable for clinical research must contain mechanisms ‘or application development support and integration of new algorithms. Our concept of an extensible platform is based on the following observation: A mere toolsystem without a common integration framework will lead to the development of many standalone applications. This seems not very efficient because base functionality elements necessary for any application, like loading patient data, interpretation of file formats or display management are not reused. Moreover, the creation of different user interfaces will result in a great deal of learning efforts of the medical staff in order to get familiar with a new application. A third problem is the great overhead necessary for data import and export if two applications are to be combined in a certain case. For these reasons we propose a common integration framework which is given by the already described turnkey appiication system and an elaborate extension interface. Fig. 4 illustrates the mechanism of integrating a new application into the turnkey system. Usually a new application is integrated into the workstation interface as an iconized application button within the menu field. By pressing this button the user invokes the application and, e.g., a popup-menu is displayed. From the programmer’s view two object classes play an important role here: User interface control classes 0 Patient study handling classes Let us briefly explain their meaning: User interface control objects manage the behaviour and look of the common user interface. They contain methods, e.g., for enabling/disabling menu buttons, mouse drawing of a polygon within an image and displaying an image at a certain screen
Fig. 4. Principle of integrating turnkey workstation.
a new
application
into
the
location. A patient study may contain textual (elements (e.g. patient name, medical report, ,anamnesis), graphical objects, images and audio. Study handling classes provide methods for ac‘cessing and manipulating these elements and their hierarchical structure. For example image data may be accessed by an application using a get-image or get-pixel-value method. New images may be displayed by transfering the data from the application object to a socalled result study which is connected to an image display segment on the screen. In this way display management is performed by the turnkey system and the application programmer does not have to bother for this highly complex task. Besides this integration interface the application builder toolbox is another key element of our development system. Its basis is an object class library which covers a broad spectrum of Medical Imaging and Graphics functionality. Its structure represents a great deal of software development expertise in this area and therefore allows inexperienced application programmers to quickly arrive at a professional program design. The class library contains a variety of C+ + classes developed at our institute and integrates a commercial system (Imaging Applications Platform, ISG Tee., Toronto) which we found to be of good adequacy for this purpose. The library encompasses objects and methods for image file format handling, image memory management, digital filtering, pixel
163
N.-H.
Eizricke
et al. 1 Computer
Methods
and Progmrm
value statistics, raycasting, dataset registration, geometric modeling, segmentation, surface reconstruction and many more. A detailed description of the object class structure is beyond the scope of this paper. Our design was guided by the endeavour to support application development as well as algorithmic research. Especially the latter led to a sophisticated structure with many levels in the inheritance hierarchy for a number of classes, thus providing high flexibility. 6.. Comcfosions In collaboration with the University Hospitals of Tiibingen, Eye Clinic, Radiological Clinic and Surgical Clinic, an evaluation of both components of the platform was performed. Our evaluation results suggest that clinical research related to digital imaging and graphics benefits from the availability of the toolkit system in three ways: First, the path from a research idea to its implementation and evaluation is shortened. Second, novel applications are motivated by the system and the realization of complex applications be.” comes possible. Third, the communication between clinical researchers and computer scientists is supported by the platform. However, the high level of programming knowledge necessary to efficiently use our toolbox, has been an obstacle for addressing a broader community of clinical researchers. Excellent programming skills are not widely found in a clinical environment. A solution would tie a graphical programming interface which we know from commercial user interface builders or visual programming tools, such as AVS (St~~rdent Inc., Concord) and Explorer (Silicon Graphics, mountain View). We decided to
in Biomedicine
48 (1995)
157-162
design a graphical object assembly editor allowing both, user interface and functionality of an application to be visually programmed and represented as an object network on the workstation screen. References [II R.A. Robb and C. Barillot, Interactive display and analysis of 3D medical images. IEEE Trans. Med. Imag., 8(3) (1989) 217--226. [2J C. Upson, T. Faulhaber, D. Kamins, D. Laidlaw, D. Schlegel, J. Vroom, R. Gurwitz, and A. van Dam, The application visualization system: A computational environment for scientific visualization. IEEE Computer Graphics and Applications 9(4) (1989) 30-42. [3] D.S. Dyer, A dataflow toolkit for visualization. IEEE Computer Graphics and Applications IO (1990) 60-69. [4] P. Heffernan and D. Dekel, Imaging applicatiol~s platform: Concept to implementation. In R.A. Robb. editor. Visualization in Biomedical Computing. pp. 495 -509. Bellingham 1992. SPIE. [S] B.K.T. Ho, 0. Ratib and S.C. Horii, PACS workstation design. Computerized Medical Imaging and Graphics 15(3) (1991)
147-155.
K.-H. Englmeier, T. Hilbertz, and U. Fink. Computer-assisted interaction with and manipulation of multi-modal images in medicine. In Medical Informatics Europe 1993, pp. 347, 354. [7] M. Staemmler. R. Brill, K. Becker, K.-H. Polkerts and K. Gersonde, in SUNRISE: a software system for medical imaging analysis, eds. H.U. Lemke. M.L. Rhodes, CC. Jaffe and R. Felix, Computer Assisted Radiology 1989, pp. 671-677, Berlin, Heidelberg, New York, 1989. Springer. [8] M. Dahm, K. Giaser, H. Jansen-Dittmer, A. Keizers, P. Krueger, ct. Meyer-~br~ht. K. ~~nker-Ka~ipp, H. Rudolf, C. Schilling, R. Sieslack, W. Winkler, B. Wein and R. Giinther, in Radiologists start designing their digital workplace: prototyping with a digital image workstation in the radiology, eds. H.U. Lemke, M.L. Rhodes, C.C. Jaffe and R. Felix, Computer Assisted Radiology 9 1, pp. 699-704, Berlin, Heidelberg, New York, 1991. Springer. [B]