- Email: [email protected]
High-Performance Telemedicine Information Management
Journal of Parallel and Distributed Computing 56, 235250 (1999) Article ID jpdc.1998.1520, available online at http:www.idealibrary.com on
High-Performance Telemedicine Information Management Hong-Mei Chen* , 1 and David Y. Y. Yun *Advanced Information Management Solutions (AIMS) Laboratory 6 Image Information Processing Laboratory 6 Department of Decision Sciences, College of Business Administration; Department of Electrical Engineering, University of Hawaii, Honolulu, 96822 Hawaii E-mail: hmchendscience.cba.hawaii.edu, dyunwiliki.eng.hawaii.edu
Received February 1, 1998; revised November 12, 1998; accepted November 18, 1998
The rapid advances in high performance global communication have accelerated cooperative image-based medical services to a new frontier. Traditional image-based medical services such as radiology and diagnostic consultation can now fully utilize multimedia technologies to provide novel services, including remote cooperative medical triage, distributed virtual simulation of operations, as well as cross-country collaborative medical research and training. Fast (efficient) and easy (flexible) retrieval of relevant images remains a critical requirement for the provision of remote medical services. This paper describes the database system requirements and presents a system architecture of a distributed multimedia database system, MISSION-DBS, which has been designed to fulfill the goals of Project MISSION (medical imaging support via satellite integrated optical network)an experimental high performance gigabit satellite communication network linking remote supercomputing power, medical image databases, and 3D visualization capabilities, in addition to medical expertise anywhere and anytime around the globe. The MISSION-DBS design employs a synergistic fusion of techniques in distributed databases (DDB) and artificial intelligence (AI) for storing, migrating, accessing, and exploring images. The efficient storage and retrieval of voluminous image information is achieved by integrating DDB modeling and AI techniques for image processing while the flexible retrieval mechanisms are accomplished by combining attribute-based and contentbased retrievals. 1999 Academic Press, Inc. Key Words: global PACS; medical image database management; telemedicine; multimedia; ACTS; distributed databases; AI; semantic data modeling; image retrieval; cooperative medical services. 1
Corresponding author.
235
0743-731599 30.00 Copyright 1999 by Academic Press All rights of reproduction in any form reserved.
236
CHEN AND YUN
1. INTRODUCTION
The technology required to realize the total computerization of medical image acquisition, processing, storage, transmission, and manipulation has been actively pursued for the past two decades in a well-established research field known as picture archiving and communication systems (PACS). As PACS progresses from a novel research concept to commercially viable systems, research focus in both academia and industry has shifted from providing ``totally digital radiology'' to providing image information management solutions for hospitals. New requirements continue to arise for PACS database system design as new medical services and functions are added in response to technological advances in multidimensional imaging as well as multimedia communications and computing [10]. The recent advances in high performance global communications aim at providing network bandwidth capacity and transfer rate, necessary for the remote access and utilization of image-based medical information in a cooperative context (e.g., transcending the barriers of time, distance, and geography). The launch of the National information infrastructure (NII) initiative in February of 1993 has accelerated the development of advanced communication technology, together with the large-scale integration of diverse, heterogeneous imaging, computing, and visualization capabilities. As an in-orbit tested for new technologies and applications, NASA's ACTS (advanced communication technology satellite) was launched into orbit by the Space Shuttle DISCOVERY on September 12, 1993 and achieved its intended geosynchronous station at 100% W longitude on September 28, 1993 [10]. The Project MISSION (medical imaging support via satellite integrated optical network), funded by ARPA (DoD's advanced research project agency) and coordinated by the authors at the University of Hawaii, is a feasibility study that experiments with the use of the ACTS gigabit satellite linkage, combined with supercomputing and graphic visualization power to validate such novel medical applications as distributed radiation treatment planning and remote medical triage support. These services have made fast (efficient) and easy (flexible) image management even more critical due to the increasing demands from real-time remote display, quick response time, and cross-organizational data sharing. MISSION database system (MISSION-DBS) is an experimental distributed multimedia database designed for the management and utilization of image data in support of novel cooperative medical services. Traditionally, the challenge of managing (e.g., storing, searching, and retrieving) voluminous image data has been tackled separately from two distinct perspectives: logical image level (e.g., information query, image analysis, and understanding) and physical image level (e.g., digitalization, encoding, compression, segmentation, enhancement, and restoration). Nevertheless, efficient and effective image retrieval must rely on the complimentary usage of suitable techniques at both levels [8]. Information flows from the very top of the image representation pyramid to facilitate the lower-level processing; welldesigned physical representation will facility fast retrieval of intended logical objects (see Fig. 1). Such cooperation is particularly important for meeting high performance requirements in cooperative remote medical services. MISSION-DBS is developed, based on information management solutions that integrate techniques at
MISSION-DBS
FIG. 1.
237
Image information management pyramid and presentation levels.
both the logical and physical image level, exploiting the duality of logical presentation of data (e.g., structure and semantics of data) and physical presentation of data (e.g., geometric models, segmented objects, picture functions, and signal). In the next section, we will first discuss the requirements for a medical database system in the context of Project MISSION. Based on the requirements, appropriate approaches will be identified and integrated into MISSION-DBS. The system architecture of MISSION-DBS is presented in Section 4. The design of MISSIONDBS employs a synergistic fusion of techniques in distributed databases (DDB) and AI for storing, migrating, accessing, and exploring, images. Issues of unified semantics modeling for MlSSION-DBS are discussed in Section 5. Section 6 concludes this paper with remarks on the development of MISSION-DBS and its implications for future health care education and research in a high-speed global communication environment.
2. REQUIREMENTS FOR MISSION-DBS As ACTS provides three simultaneous simplex channels for high data rate communications, three geographically separate sites covered by three different antennas on the ACTS satellite have been teamed together to conduct these medical imaging experiments: Georgetown University Medical Center (GUMC) in Washington, DC is located in the eastern sector of the ACTS satellite, Ohio Supercomputing Center (OSC) at Ohio State University in Columbus, OH is in the western sector, while the University of Hawaii at Manoa (UH) requires the use of the unique steerable antenna of ACTS. Critical support to this project also comes from NASA (providing free ACTS time), the State of Hawaii (HDR acquisition and administrative assistance), and the Army's MDIS (medical diagnostic information support) program. Hawaii's Tripler Army Medical Center (TAMC) will not only be providing medical data and clinical expertise but also a home for the HDR earth station. As the hub of medical services in the Pacific and a key participant in the MDIS
238
CHEN AND YUN
program, TAMC is a natural partner for Project MISSION. The scalable supercomputer (presently a 120-processor IBM SP2 capable of 40 giga-flops) funded by the Air Force and installed at Hawaii's Maui Research and Technology Park (MRTP) is also a natural partner to support the computational requirements of the experiments. Thus, TAMC and MRTP, led by the UH and linked with a fiber optic ATM network, become the Hawaii-based tripartite team that collectively and individually complements the satellite-linked team of GUMC, OSC, and UH in terms of both functionality and expertise. The medical service selected for Project MISSION is the distributed remote treatment planning session. Radiation treatment planning is an ideal application for validating the usage of high-speed networks in cooperative medicine, due to the scarcity of oncology expertise, the complexity of optimization procedures, and its requirement of high-speed computing power and communication bandwidth for volume visualization. Radiation therapy has been the main treatment for cancer for decades. The idea behind radiation therapy is to deliver a tumorcidal dose of radiation to a tumor volume while minimizing doses to surrounding normal tissues. The results of the treatments depend greatly on the planning. In order to administer tumorcidal doses to deep-seated tumors, it is necessary for the radiation to be crossfired from a number of different angles which are aligned to intersect inside the tumor. Treatment planning is the process developed to assure that these beams indeed intersect inside the tumor and avoid irradiating critical normal tissues such as the spinal cord or kidney. The design of MISSION-DBS in this context considers two levels of medical image transactions: local PACS (LPACS) for a single hospital and global PACS (GPACS) for linkages with external hospitals or other resources. An envisioned GPACS health care environment consists of networked LPACS among regional, national, or even international hospitals [10, 33]. As a result, the design of MISSION-DBS will build upon existing infrastructure of LPACS and bc tailored to the requirements of cooperative remote medical services in a GPACS environment. 2.1. Local PACS Requirements The complexity of medical information storage and retrieval lies in the large volume, the diverse types of image data from different modalities, and the close relationship among multimedia data in PACS. In general, medical images contain two-dimensional information of varied sizes. There are also three-dimensional spatially reconstructed images (3D volume) from computed tomography (CT) or magnetic resonance imaging (MRI) and dynamic or time-serial images from cineangiogram (CINE) or ultrasound (US). Medical images are interpreted (read) by radiologists, resulting in associated diagnostic reports (keyed-in textual data or voice-activated input) and graphic annotations called overlays (marks such as arrows, circles, lines on images to denote areas of abnormality). Furthermore, there are complex temporal, anatomical, and modality relationships between new and old image sets. There are very few deletes, no updates, but frequent routine and ad hoc retrievals on image data. In addition, performancewide, radiologists typically require that responses to image retrievals be completed
MISSION-DBS
239
within 2 s [29]. The typical peak load traffic rate of radiology services (e.g., one image reading per 8 s) imposes high throughput requirements on the design of image database systems. In a clinical environment, data availability and system reliability are so essential that diagnostic transactions should not be affected by system malfunctions, such as system down time [29]. Moreover, special considerations must be given to the interdependence of PACS with exiting information systems such as hospital information systems (HIS) and radiology information systems (RIS) [30]. 2.2. Performance Requirements for Cooperative Remote Medical Services Cooperative image-based medical services have long been used in situations of medical consultation, medical triage, education, and research [30]. Traditional image sharing in cooperative medical services was manual, in-person, time consuming, and ineffective. The advancements of communications, computation, and visual display technologies have significantly improved the feasibility and effectiveness of remote cooperative image-based medical services across a network [33]. The multimedia interactive conferencing capability is now being utilized in teleradiology to enable cooperation with remotely located experts to interactively analyze a patient's situation by viewing the images in various perspectives. In triage situations, high speed communications allow rapid access to vital information about the injured and requires real-time reference to similar patient cases for confirming diagnosis. This requires ad hoc content-based retrieval capability. For medical research and educational purposes, there is also a great need to reference data by the image content to retrieve images from a large medical data bank or image archives [26]. Limited by current communication bandwidth, image management for remote services can only be done off-line. With the availability of a high performance network, the medical image management for cooperative remote services encounter greater requirements in handling a large image volume (multimedia, 3D), real time response time, and complex queries that involve content-based retrievals. For instance, the essence of the remote distributed treatment planning experimented here is to achieve diagnostic efficiency and effectiveness through the sharing of image information, medical experts and imaging and computing equipment (see Fig. 2 for the context). Conventional treatment planning is performed on 2D slices of CT images. This is not only inaccurate but is also incapable of restricting radiation dosage to the desired region. In response, 3D conformal treatment planning has just recently emerged. Its accuracy is due to the full consideration of the threedimensional structure of a patient's anatomy. Furthermore, by planning 3D beams (as opposed to co-planer beams in the 2D planning) we can expect to deliver doses conformally on the tumor. Therefore, the database is required to reconstruct a 3D data volume, based on image sets from different modalities (possibly from different locations) in a clinically acceptable speed. Three-dimensional planning also requires a high performance 3D graphics computer to allow physicians to view the anatomy, draw beams, and evaluate dose distributions. It then requires a high speed CPU to compute and even optimize dose distributions based on the beam and anatomy information. The ideal goal,
240
CHEN AND YUN
FIG. 2.
Cooperative remote image-based medical services.
clinically, is for the computations to be near real time so that the iterative optimization can be efficiently executed. High speed communication provides a solution for remote access to these expensive facilities. Most importantly, it requires appropriate image management to prepare the necessary volumetric dataset efficiently to avoid any weak link in the communications.
3. MEDICAL INFORMATION MANAGEMENT APPROACHES To address the performance requirements for supporting efficient and effective cooperative remote medical services, we have considered the following information management approaches at both the logical level and the physical image levels.
3.1. Distributed Database Approaches Distributed database design (DDB) has been shown to better meet the various performance requirements than a central database design in the clinical environment of LPACS [28, 29, 31]. For cooperative medical services, the distributed architecture which allows parallel retrieval of images for a transaction will mitigate the communication bottlenecks encountered due to the speed disparity between the data storage component and the (high performance) data communication component. Under such a design, multimedia data can be stored in different physical media to enable parallel data retrievals, thereby reducing retrieval response time [6, 7, 15, 34]. In addition, a distributed design is also more dependable in that it can continue information service in the face of failures of individual sites or partial communication links. Moreover, cost economy is gained since a distributed design can make use of local workstations and their hierarchical data storage systems [7, 34].
MISSION-DBS
241
3.2. Knowledge-Based Approach for Pipeline (Prefetch) Operations To enhance the performance of a DDB, knowledge-based augmentation has been proposed. For example, IRES [32] is an initial application of expert system technology to computerize radiologists' knowledge to identify relevant images for a radiological exam reading. This allows image prefetch, as well as migration among data storage hierarchy and different modalities in different network locations to expedite real-time image retrieval response time. Such augmentation is particularly desirable for remote cooperative services as the medical specialist requesting data may not be familiar with the storage structure of the radiological data at the remote site. In addition to preserving expertise of experienced radiologists and improving the response time of the database system, it will also increase diagnostic accuracy by providing relevant information automatically. 3.3. Image Retrieval Approaches Most current PACS database implementation adopts attribute-based retrieval, utilizing a relational approach which treats an image as a BLOB (binary large object) (e.g., physical picture) and uses an image identifier and related attributes (e.g., logical picture) to retrieve images [8, 37]. In the most common case, the raw images (physical pictures) were processed by an image processing module and image descriptors were obtained. These image descriptors were then handled by a RDBMS (relational distributed Database management system) and used for indexing the image. Although the DDB approach, coupled with knowledge-based image prefetch, will allow high performance image retrieval, the employment of only attribute-based image retrieval limits flexibility for user query formulation in complicated diagnostic cases, as well as in ad hoc retrieval situations. The inflexibility is in two aspects: first, due to query language restriction and, second, because it is difficult to exhaust all the possible descriptions of objects and their (anatomical, spatial) relationships on a certain image by attributes. The limitations are most felt in cases when user queries contain attributes that are not predefined in the databases. To allow more flexibility in image retrieval, techniques used for document bases in traditional information retrieval (IR) field [3, 36] have been proposed for image retrieval in that the retrieval of images are mediated via the retrieval of textual description (in natural language or keywords) of images [14]. This full-text-based or IR-based approach is often referred as pseudo-content-based retrieval. In a medical environment, the diagnostic report(s) that accompanies each image is a perfect intermediate. Various models and concept-based [14] techniques that utilize application semantics and user-feedback have been shown to have better recall and precision rate in image retrieval than free text description. Although the IR-based approach can offer much flexibility in query formulation, there are still needs for content-based (or pictorial query-by-example) query capabilities [26, 39, 41]. This is because sometimes images are hard to comprehend, interpret, or describe as a picture is worth a thousand words. For instance, if a doctor encounters a difficult case and needs some supporting evidence to make his diagnosis, the content-based capability would allow him to consult
242
CHEN AND YUN
similar images associated with a confirmed diagnosis by presenting just the images at hand. This capability is particularly important for medical consultation, research, and education [26]. In a content-based approach, features of images are analyzed and extracted utilizing AI or neural network (NN) techniques, such as object recognition and pattern recognition, the retrieval of certain images are then based on the similarity measures between the query objects and the stored objects [39]. 3.4. Image Processing The provision of content-based retrieval is the most difficult as it requires sophisticated, intelligent processing at the physical picture level to ``understand'' the image [5]. A critical step to ``understaned'' images is to decompose a picture into meaningful partsto separate objects from background and to distinguish among objects after an image is properly digitized and coded (e.g., with gray level). Segmentation techniques include simple pattern recognition by thresholding, boundary detection, contour following, etc. [8]. Many advanced pattern recognition techniques employ a neural network (NN) approach. It then requires domain knowledge to associate a segmented object with a meaning. In addition, the 3D data visualization is another that requires intelligent image processing, such as the segmentation of 2D slices for reconstructing a 3D volume, and the projection of a 3D object to a 2D screen. For supporting the goals of MISSION-DBS, we have developed symbolic encoding techniques [26], pursued an advanced neural network feature extractor [24, 25], and explored the parallel execution of 3D volume rendering code [44]. These techniques will be integrated into a ``toolset'' in our MISSION-DBS. 3.5. Poly-Paradigmic Approach to Content-Based Retrieval We have investigated the integration of image management at a logical and physical image level for effective support of cooperative remote medical services. Our initial research resulted in a poly-paradigmic approach for achieving efficient content-based retrieval. In our proposed approach, the symbolic coding of images, advanced NN feature extractors and semantic data modeling are utilized to reduce the content understanding problem to one that is similar to a textual database retrieval problem [25]. Image features are extracted by an advanced NN feature extractor and stored in a metadatabase. Application semantics will be associated with each feature. The symbolic encoding technique is applied to the image feature which will be stored as an image tag or ``signature.'' In cases of pictorial query, users can present an image as an ``example image.'' Features in the example image will be extracted, coded, and then compared with prestored ``signatures.'' Searching for a pattern image in the user query can then be automated as a computer operation that simply matches the higher level abstract signatures of the feature with those of the stored images. We have investigated the use of DDB abstraction mechanisms (generalization and aggregation) used for data fragmentation for classifying ``signatures'' symbolically coded. As each feature is associated with data semantics or concepts, attributes-based or concept-based retrieval can be executed
MISSION-DBS
FIG. 3.
243
Poly-paradigmic approach for content-based retrieval.
using the same conceptual data model. This approach represents savings in modeling efforts and will reduce inconsistency among the data semantics modeled. We have developed a NN-based holographic approach utilizing only pixel presentation for attention-modulated content-based retrieval [24, 25]. It will be used for complementing our poly-paradigmic approach shown in Fig. 3.
4. MISSION-DBS ARCHITECTURE To meet the information management requirements of PACS in general and cooperative image-based medical service in particular, such techniques as DDB design, knowledge-based ``prefetch,'' and ``pipeline,'' content-based image retrieval, and intelligent image processing techniques must be brought together. Attributebased, IR-based, and content-based retrieval techniques should all be incorporated because no single image retrieval approach satisfies all the requirements of medical information, management. The recent commercial development in DBMS has been encouraging toward developing intelligent access to medical image information. The commercial RDBMSs, such as Oracle V.8 Sybase System 11, are all capable of handling distributed image data. Full-text DBMSs (e.g., TOPIC, TextDBMS) have new improvements in full text retrieval. Commercial products for simple contentbased retrieval have been announced, such as IBM's QBIC (Query By Image Content). There has also been a rapid development of parallel database servers [4]. Figure 4 depicts the system architecture of MISSION-DBS, considering the special characteristics and requirements of medical image management discussed above. MISSION-DBS resides in a global distributed environment with a high performance communication network. The architecture combines the efficiency of the distributed database technology, the flexibility of a knowledge-based approach, and the power of AI techniques to address medical multimedia information management requirements. Many research results from previous PACS research (e.g., hierarchical storage management and emerging technology, such as parallel query processors) are integrated into the architecture. The system architecture defines interfaces among various technological building blocks and thus serves as a research framework to guide our development effort. Our current research effort concentrated on the modules of the shaded areas.
244
CHEN AND YUN
A plausible design for MISSION-DBS could be to ``loosely couple'' the modules for each technology identified. However, as common application semantics underlies the design and modeling of DDB and AI techniques, unified semantic modeling is therefore examined to enable the integration of modules for both user query formulation at the logical image level and for image encoding, processing and automatic image indexing at the physical image level. The initial results in combining the DDB abstraction mechanism for feature indexing have led lo our adoption of a ``tightly coupled'' architecture for MlSSION-DBS that will permit quick and intelligent access to medical images. As a result, the distributed database catalog system is integrated with knowledge-based components in the Metadatabase (see Fig. 4). The system components are described below and the integrated modeling issues will be discussed in the section that follows. 4.1. Graphical User Interface (GUI) The graphical user interface of MISSION-DBS resides in the PACS diagnostic or viewing workstations [19, 25], where multimedia objects can be created, processed, manipulated, searched, browsed, and retrieved. Interactive relational query language, natural language-based, as well as pictorial query language should be provided for different types of database queries. Multimedia input devices and an interactive image processing language for voice input of diagnostic reports, drawing overlays on images and performing pictorial query-by-example are provided here. Flexible image viewing capabilities (e.g., zooming, image browsing, 3D image
FIG. 4.
MISSION-DBS system architecture.
MISSION-DBS
245
motion viewing such as US or CINE) are included. Multimedia conferencing video inputoutput is part of the GUI for remote consultation. In addition, the following modules are important to the performance of medical image information retrieval and manipulation: Dialogue manager. This module will conduct dialogueconversation with users to clarify incomplete or ambiguous information requests and assist users in general multimedia information manipulation tasks and system usage. Tutorials and help facilities should be available. Spatial analyzer. This module will allow flexible Spatial analysis on images to identify objects or diagnosis of abnormality in an image. For example, the size of a tumor and its position in relations to other organs. Conceptkeyword manager. This module will maintain (createupdate) concepts, keywords and a thesaurus to assist users in formulating multimedia database queries and also allow automatic indexing of multimedia objects. Latent query register. Preprocessed queries for routine readings are stored and can be selected by users to speed up query performance. Query-by-picture example manager. This module will handle the ``interpretation'' of the query image, make calls to metadatabase to find matches in the image feature catalogue. CADx. CADx software for detecting abnormality or patterns that are easily missed by trained eyes is highly needed, such as for lung nodules [19, 44]. Volume renderer. This module will allow fast and parallel 3D volumetric data rendering and viewing. Remote consultation multimedia IO manager. This module manages multimedia input and output for interactive remote consultation sessions. 4.2. Metadatabase The metadatabase stores the following data and knowledge to facilitate intelligent access and manipulation of distributed medical information: DDB schemata. This module includes conceptual schema, logical schema, fragmentation schema, allocation schema, local mapping, access method description, and patterns of querytransaction activities to allow query processing, such as quick search and access to local data. Information such as usage statistics, protection, and access authorization are also stored to maintain data security and integrity. Operational 6 contextual 6 usage knowledge. This module is for use in ``reasoning'' about user queries, image understanding and formulate search strategies. It contains application domain knowledge and medical expert rules as well as grammatical rules of natural language. It (1) facilitates users in formulating multimedia database queries; (2) facilitates natural language (IR-based) query processing; (3) facilitates image object recognition and understanding (contentbased); (4) facilitates multimedia objects classification and indexing; and (5) enhances
246
CHEN AND YUN
distributed database performance by automatic image migration and ``prefetching'' to expedite image retrieval. Thesaurus and concepts. This contains language rules for resolving data semantics conflicts among local databases and to facilitate full-text user queries. Image feature catalogue. This module stores preprocessed image objects to facilitate content-based retrieval by matching stored objects with requested objects presented by the user. 4.3. Distributed Database Management System (DDBMS) The DDBMS hides the multimedia data distribution from the users and provides data fragmentation, allocation, and replication transparency to users [4, 14]. It manages all the complicated distributed operation problems for the users, including distributed query processing [17], distributed concurrencyrecovery control, metadatabase management, distributed security control [31], and determines physical data storage structure, format, indices, and search strategies [28]. The following modules are essential components in DDBMS: SQL 6 full-text processor 6 content-based query processors. These are three submodules. Each query processor decomposes a user request of multimedia information to subtransactions for local DBMSs or specialized DBMS components (each of them handles certain medium type), performs all relevant IO operations, and then integrates the results of these subtransactions. The SQL processor understands SQL. The full-text processor utilizes the thesaurus and concepts in the metadatabase for query processing, while content-based query processors utilize the image processing toolset. For facilitating content-based image retrieval, the goal of the module is to automatically extract from a medical image the symbolic descriptions that are meaningful or relevant to user queries. A knowledge base that contains descriptions (or symbolic coding that is purely numeric or vector-based) of the objects or of generalized object classes will be constructed first to enable content-based image database query. The rule-based inference mechanism and an interactive image-processing language are then used to help DDBMS retrieve images. Concurrencyrecovery controller. This module ensures serializability and recovery when transactions are executed concurrently or in the event of system failures [29]. Communication manager. This module handles data format and transfer functions according to the underlying multimedia communication protocols (gateway and router functions) [6, 44]. Parallel query processor. This module performs parallel query to the database using a back-end machine that offloads IO bound functions from DDBMS. For instance, a query can contain several predicates for different medium types or different image features and each predicate will be searched in parallel. The parallel machine can improve the system's reliability through complexity reduction and enhances the system's performance through specialization, increased parallelism, and processing power [15].
MISSION-DBS
247
Local data Manager. Each local data manager is responsible for executing individual subtransaction for each local database. It contains a local DBMS, a file manager for each medium that manages storage spaces and a migration manager that moves medical multimedia data up or down within a storage hierarchy [29]. 4.4. Image Processing Toolset The image processor consists of pattern-recognition and image-processing routines that are called by the query processor. These routines are designed to process images and extract spatial and structural information from images of different modality types. The afore-mentioned intelligent image processing routines are integrated into this module [24, 25]. 4.5. Storage Subsystem The storage subsystem stores multimedia data according to different data types (structured data, text, vector-based overlays, 2D and 3D images) and data fragments according to the logical data organization and transaction patterns. These data are managed by local DBMSs specialized to their respective medium. Moreover, since data retrievals exhibit a temporal correlation such that newly generated images are retrieved more frequently and older ones less frequently, it is cost-effective to have a hierarchical storage architecture, where older images are stored in a lower-cost but slower storage system (e.g., optical disks) and more recent images in a more expensive but faster storage system (e.g., magnetic disks). RAID is considered for better reliability and read-only performance [2]. 5. INTEGRATED SEMANTICS MODELING ISSUES It is clearly revealed from the system architecture that the fusion of research results and techniques from different research fields is essential to enable efficient, as well as intelligent, management of medical multimedia information. While the modeling, design, and implementation of each component in the system architecture already poses challenges for each component's distinct research field, the development of integrated modeling for MISSION-DBS merits immediate attention. Common application semantics underlies the modeling and design of a distributed database (DDB) and AI-based techniques for text and image understanding in MISSION DBS. A uniform approach to model semantics would be an important step toward simplifying the interfaces among these components. In particular, the knowledge representation would be facilitated by a single semantic formalism that could represent concept-based query semantics, as well as logic database design [20, 35]. The integrated modeling of common application knowledge (rules, facts, or data) will have an advantage in reducing duplicated modeling efforts and increasing the efficiency of metadata management. In short, the unified modeling constructs and abstraction mechanisms (1) must represent medical multimedia data and usage knowledge easily and effectively; (2) must allow transformation from conceptual level to logical level (including data distribution)
248
CHEN AND YUN
and physical level representation for implementation so that multimedia data can be easily stored, indexed, searched, and retrieved; (3) must support distributed multimedia information access (e.g., attribute-based, text-based, and content-based retrievals), and (4) should facilitate multimedia data and knowledge integration and automatic image object classification and indexing. We are extending a DDB design methodology [9, 11] for meeting these requirements. 6. CONCLUSION Technological fusion of distributed database and artificial intelligence (knowledge-base augmentation and intelligent image processing) is necessary to meet the special requirements of total medical image management for cooperative remote medical services, taking full advantage of high performance communication capabilities for delivering novel cooperative remote medical services. Our initial results in integrating image management at the logical and physical levels is through the development of the ploy-paradigmic approach to content-based retrieval. The MISSION-DBS architecture reflects the integration of various approaches necessary for effective image management. The MlSSION-DBS system architecture also serves as a framework for our continued research and development effort in providing total medical image management for future health care in a GPACS environment. We have focused on developing the capabilities of high performance image retrieval and effective content-based retrieval. The research results are expected to directly benefit our experimental remote image-based medical services (e.g., distributed treatment planning) in our MISSION project and will be useful for other future novel cooperative remote medical services. Particularly, the knowledge-based retrieval strategies and content-base retrieval capabilities will be very desirable for the medical education and research on a global scale. It is an objectiveand, in our opinion, will eventually become a requirement for the PACS database system to be able to manage (e.g., store, process, retrieve and manipulate) voluminous 2D or 3D image data in the same manner (e.g., speed and flexibility) as for traditional alphanumerical data. Although current contentbased image retrieval performance is still far from ideal, our research effort to integrate DDB and AI techniques shows promise toward this objective. ACKNOWLEDGMENT This research is supported by a grant from DOD ARPA (Advanced Research Project Agency) (Award HJ1500-3175 0562AAP*DAR3A634).
REFERENCES 1. J. R. Bach, S. Paul, and R. Jain, A visual information management system for the interactive retrieval of faces, IEEE Trans. Knowledge Data Eng. 5, 4 (August 1993), 619628. 2. K. Bachus, P. Houston, and E. Longsworth, Right as RAID, Corporate Comput. 2, 5 (May 1993), 6164.
MISSION-DBS
249
3. E. Bertino, F. Rabitti, and G. Simon, Query processing in a multimedia document system, ACM Trans. Office Inform. Systems 6, 1 (January 1988), 141. 4. R. Bloor, The coming of parallel servers, DBMS 7, 5 (May 1994), 1416. 5. J. Brolio, B. A. Draper, J. R. Beveridge, and A. R. Hanson, ISR: A database for symbolic processing in computer vision, IEEE Comput. (December 1989), 2230. 6. G. Bucci, R. Detti, V. Pasqui, and S. Nativi, Sharing multimedia data over a client-server network, IEEE Multimedia 1, 3 (Fall 1994), 4455. 7. S. Ceri and G. Pelagatti, ``Distributed Databases: Principles and Systems,'' McGrawHill, New York, 1984. 8. S.-K. Chang, ``Principles of Pictorial Information Systems Design,'' PrenticeHall, Englewood Cliffs, NJ, 1989. 9. H.-M. Chen Garcia, ``A Semantics-Based Methodology for Integrated Distributed Database Design: Toward Combined Logical and Fragmentation Design and Design Automation,'' Ph.D. dissertation, University of Arizona, (1992). 10. H.-M. Chen Garcia and O. R. Liu Sheng, Medical multi-media information management: A research framework, in ``Proceedings of the 1993 Pan Pacific Conference on Information Systems, Kaoshsiung, Taiwan, R.O.C., May 30June 1, 1993,'' pp. 3947. 11. H.-M. Chen Garcia and O. R. Liu Sheng, An entity-relationship-based methodology for distributed database design: An integrated approach toward combined logical and fragmentation design, in ``Proceedings of the 11th International Conference on the Entity Relationship Approach, Karlsruhe, Germany, October 79, 1992.'' 12. F. Chen, M. Hearst, J. Kupiec, J. Pedersen, and L. Wilcox, Metadata for mixed-media access, SIGMOD RECORD 23, 4 (December 1994), 6471. 14. T.-S. Chua, H.-K. Pung, G.-J. Lu, and H.-S. Jong, A concept-based image retrieval system, in ``Proceedings of 27th Hawaiian International Conference on System Sciences, Maui, Hawaii, Jan. 46, 1994.'' 15. S. Ghandeharizadeh and L. Ramos, Continuous retrieval of multimedia data using parallelism, IEEE Trans. Knowledge Data Eng. 5, 4 (August 1993), 658669. 16. M. Goldberg and J. Herscovitz, Image organization and Navigation using dynamic icons for a radiology workstation, in ``Proceedings of IMAC '91, April 1013, Kyoto, Japan, IEEE, 1991,'' pp. 371377. 17. H. Edelstein, Distributed databases, DBMS (September 1990), 3648. 18. L. R. Hevner and A. Rao, Distributed data allocation strategies, in ``Advances in Computers'' (M. C. Yovits, Ed.), pp. 121155, Academic Press, New York, 1988. 19. M. Hosoba et al., PACS workstation for computer assisted image diagnosis, in ``Proceedings SPIE Medical Imaging Conference IV,'' February 49, 1990, Newport Beach, CA. 20. R. Hull and R. King, Semantic database modeling: Survey, applications, and research issues, ACM Comput. Surv. 19, 3 (September 1987), 201260. 21. R. Kasturi, R. Fernandez, M. L. Amlani, and W.-C. Feng, Map data processing in geographic information systems, IEEE Comput. (December 1989), 1021. 22. M. A. Ketabchi and V. Berzins, Modeling and managing CAD databases, IEEE Comput. (February 1987), 93102. 23. J. I. Khan and D. Y. Y. Yun, Chaotic Vectors and a proposal for multidimensional associative network, in ``Proceedings IS6TSPIE Electronic Imaging: Science 6 Technology,'' Feb. 610, 1994, San Jose, CA. 24. J. I. Khan and D. Y. Y. Yun, Feature-based contraction of sparse holographic associative memory, in ``Proceedings World Congress on Neural Networks, San Diego, June 59, 1994,'' pp. IV2634. 25. P. Kim, H. W. Park, and D. R. Haynor, Requirements for PACS workstations, in ``Proceedings of IMAC '91, April 1013, Kyoto, Japan, 1991,'' pp. 3642.
250
CHEN AND YUN
26. P. Kofakis, A. Karmirantzos, Y. Kavaklis, E. Petrakis, and S. Orphanoudakis, Image archiving by content: An object-oriented approach, in ``Proceedings SPIE Medical Imaging Conference IV, February 49, Newport Beach, CA, 1990.'' 28. O. R. Liu Sheng, C. P. Wei, and H.-M. Chen Garcia, Design and implementation of a distributed PACS database system, in ``SPIE Proceedings of Medical Imaging Conference VIII, February 49, 1992, Newport Beach, CA.'' 29. O. R. Liu Sheng and H.-M. Chen Garcia, The design of medical image databases: A distributed approach, in ``IEEE Proceedings of International Phoenix Conference on Computers and Communications, Scottsdale, AZ, March 2123, 1990,'' pp. 288295. 30. O. R. Liu Sheng and H.-M. Chen Garcia, Information management in hospitals: An integrated approach, in ``IEEE Proceedings of International Phoenix Conference on Computers and Communications, Scottsdale, AZ, March 2123, 1990,'' pp. 296303. 31. O. R. Liu Sheng, H.-M. Chen Garcia and C.-P. Wei, Distributed database design and modeling for PACS, in ``SPIE Proceedings of Medical Imaging Conference IV, February 49, 1990, Newport Beach, CA,'' pp. 256269. 32. O. R. Liu Sheng, H.-C. Wang and H.-M. Chen Garcia, IRESImage retrieval expert system, in ``SPIE Proceedings of Medical Imaging Conference IV, February 49, 1990, Newport Beach, CA,'' pp. 832841. 33. R. Martinez, Y. Alsafadi, and J. Kim, Design of multimedia global PACS distributed computing environment, in ``IEEE Proceedings of hawaiian International Conference on Systems Sciences (HICSS '28), 1995.'' 34. T. M. Ozsu and P. Valduriez, Distributed database systems: Where are we now?, IEEE Comput. (August 1991), 6878. 35. J. Peckham and F. Maryanski, Semantic data models, ACM Comput. Surv. 20, 3 (September 1988), 154189. 36. G. Salton, ``Automatic Text Processing: The Transformation, Analysis and Retrieval of Information by Computer,'' AddisonWesley, Reading, MA, 1989. 37. G. N. M. Shdhakar, A. Karmouch, and N. D. Georganas, Design and performance evaluation considerations of a multimedia medical database, IEEE Transactions on Knowledge and Data Engineering 5, 5 (October 1993), 888894. 38. J. M. Smith and D. C. P. Smith, Database abstraction: aggregation and generalization, ACM Transactions on Database System 20, 2 (June 1977), 105133. 39. Stephen W. Smoliar and HongJiang Zhang, Content-Based Video Indexing and Retrieval of Facial Images, IEEE Multimedia 1, 2 (Summer 1994), pp. 6272. 40. J. F. Sowa, Knowledge representation in databases, expert systems and natural languages, (R. A. Meersman, Z. Shi, and C.-H. Kung, Eds.), Artificial Intelligence in Databases and Information Systems, Elsevier Science, pp. 1741, North-Holland, 1990. 41. J. K. Wu, Y. H. Ang, P. Lam, H. H. Loh, and A. Narasimhalu, Desai, Inference and retrieval of facial images, Multimedia Systems 2, 1 (1994), 114. 42. Atsuo Yoshitaka, Setsuko Kishida, Masahito Hirakawa, and Tadao Ichikawa, Knowledge-assisted content-based retrieval for multimedia databases, IEEE Multimedia 1, 4 (Winter 1994), 1221. 43. D. Y. Y. Yun, Symbolic labeling for image management (SLIM), working paper, University of Hawaii, 1993. 44. D. Y. Y. Yun and H.-M. Garcia, High performance medical image communications via ACTS-HDR, in ``Medical Imaging '95, San diego, Feb. 26Mar. 28, 1995.''
High-Performance Telemedicine Information Management Hong-Mei Chen* , 1 and David Y. Y. Yun *Advanced Information Management Solutions (AIMS) Laboratory 6 Image Information Processing Laboratory 6 Department of Decision Sciences, College of Business Administration; Department of Electrical Engineering, University of Hawaii, Honolulu, 96822 Hawaii E-mail: hmchendscience.cba.hawaii.edu, dyunwiliki.eng.hawaii.edu
Received February 1, 1998; revised November 12, 1998; accepted November 18, 1998
The rapid advances in high performance global communication have accelerated cooperative image-based medical services to a new frontier. Traditional image-based medical services such as radiology and diagnostic consultation can now fully utilize multimedia technologies to provide novel services, including remote cooperative medical triage, distributed virtual simulation of operations, as well as cross-country collaborative medical research and training. Fast (efficient) and easy (flexible) retrieval of relevant images remains a critical requirement for the provision of remote medical services. This paper describes the database system requirements and presents a system architecture of a distributed multimedia database system, MISSION-DBS, which has been designed to fulfill the goals of Project MISSION (medical imaging support via satellite integrated optical network)an experimental high performance gigabit satellite communication network linking remote supercomputing power, medical image databases, and 3D visualization capabilities, in addition to medical expertise anywhere and anytime around the globe. The MISSION-DBS design employs a synergistic fusion of techniques in distributed databases (DDB) and artificial intelligence (AI) for storing, migrating, accessing, and exploring images. The efficient storage and retrieval of voluminous image information is achieved by integrating DDB modeling and AI techniques for image processing while the flexible retrieval mechanisms are accomplished by combining attribute-based and contentbased retrievals. 1999 Academic Press, Inc. Key Words: global PACS; medical image database management; telemedicine; multimedia; ACTS; distributed databases; AI; semantic data modeling; image retrieval; cooperative medical services. 1
Corresponding author.
235
0743-731599 30.00 Copyright 1999 by Academic Press All rights of reproduction in any form reserved.
236
CHEN AND YUN
1. INTRODUCTION
The technology required to realize the total computerization of medical image acquisition, processing, storage, transmission, and manipulation has been actively pursued for the past two decades in a well-established research field known as picture archiving and communication systems (PACS). As PACS progresses from a novel research concept to commercially viable systems, research focus in both academia and industry has shifted from providing ``totally digital radiology'' to providing image information management solutions for hospitals. New requirements continue to arise for PACS database system design as new medical services and functions are added in response to technological advances in multidimensional imaging as well as multimedia communications and computing [10]. The recent advances in high performance global communications aim at providing network bandwidth capacity and transfer rate, necessary for the remote access and utilization of image-based medical information in a cooperative context (e.g., transcending the barriers of time, distance, and geography). The launch of the National information infrastructure (NII) initiative in February of 1993 has accelerated the development of advanced communication technology, together with the large-scale integration of diverse, heterogeneous imaging, computing, and visualization capabilities. As an in-orbit tested for new technologies and applications, NASA's ACTS (advanced communication technology satellite) was launched into orbit by the Space Shuttle DISCOVERY on September 12, 1993 and achieved its intended geosynchronous station at 100% W longitude on September 28, 1993 [10]. The Project MISSION (medical imaging support via satellite integrated optical network), funded by ARPA (DoD's advanced research project agency) and coordinated by the authors at the University of Hawaii, is a feasibility study that experiments with the use of the ACTS gigabit satellite linkage, combined with supercomputing and graphic visualization power to validate such novel medical applications as distributed radiation treatment planning and remote medical triage support. These services have made fast (efficient) and easy (flexible) image management even more critical due to the increasing demands from real-time remote display, quick response time, and cross-organizational data sharing. MISSION database system (MISSION-DBS) is an experimental distributed multimedia database designed for the management and utilization of image data in support of novel cooperative medical services. Traditionally, the challenge of managing (e.g., storing, searching, and retrieving) voluminous image data has been tackled separately from two distinct perspectives: logical image level (e.g., information query, image analysis, and understanding) and physical image level (e.g., digitalization, encoding, compression, segmentation, enhancement, and restoration). Nevertheless, efficient and effective image retrieval must rely on the complimentary usage of suitable techniques at both levels [8]. Information flows from the very top of the image representation pyramid to facilitate the lower-level processing; welldesigned physical representation will facility fast retrieval of intended logical objects (see Fig. 1). Such cooperation is particularly important for meeting high performance requirements in cooperative remote medical services. MISSION-DBS is developed, based on information management solutions that integrate techniques at
MISSION-DBS
FIG. 1.
237
Image information management pyramid and presentation levels.
both the logical and physical image level, exploiting the duality of logical presentation of data (e.g., structure and semantics of data) and physical presentation of data (e.g., geometric models, segmented objects, picture functions, and signal). In the next section, we will first discuss the requirements for a medical database system in the context of Project MISSION. Based on the requirements, appropriate approaches will be identified and integrated into MISSION-DBS. The system architecture of MISSION-DBS is presented in Section 4. The design of MISSIONDBS employs a synergistic fusion of techniques in distributed databases (DDB) and AI for storing, migrating, accessing, and exploring, images. Issues of unified semantics modeling for MlSSION-DBS are discussed in Section 5. Section 6 concludes this paper with remarks on the development of MISSION-DBS and its implications for future health care education and research in a high-speed global communication environment.
2. REQUIREMENTS FOR MISSION-DBS As ACTS provides three simultaneous simplex channels for high data rate communications, three geographically separate sites covered by three different antennas on the ACTS satellite have been teamed together to conduct these medical imaging experiments: Georgetown University Medical Center (GUMC) in Washington, DC is located in the eastern sector of the ACTS satellite, Ohio Supercomputing Center (OSC) at Ohio State University in Columbus, OH is in the western sector, while the University of Hawaii at Manoa (UH) requires the use of the unique steerable antenna of ACTS. Critical support to this project also comes from NASA (providing free ACTS time), the State of Hawaii (HDR acquisition and administrative assistance), and the Army's MDIS (medical diagnostic information support) program. Hawaii's Tripler Army Medical Center (TAMC) will not only be providing medical data and clinical expertise but also a home for the HDR earth station. As the hub of medical services in the Pacific and a key participant in the MDIS
238
CHEN AND YUN
program, TAMC is a natural partner for Project MISSION. The scalable supercomputer (presently a 120-processor IBM SP2 capable of 40 giga-flops) funded by the Air Force and installed at Hawaii's Maui Research and Technology Park (MRTP) is also a natural partner to support the computational requirements of the experiments. Thus, TAMC and MRTP, led by the UH and linked with a fiber optic ATM network, become the Hawaii-based tripartite team that collectively and individually complements the satellite-linked team of GUMC, OSC, and UH in terms of both functionality and expertise. The medical service selected for Project MISSION is the distributed remote treatment planning session. Radiation treatment planning is an ideal application for validating the usage of high-speed networks in cooperative medicine, due to the scarcity of oncology expertise, the complexity of optimization procedures, and its requirement of high-speed computing power and communication bandwidth for volume visualization. Radiation therapy has been the main treatment for cancer for decades. The idea behind radiation therapy is to deliver a tumorcidal dose of radiation to a tumor volume while minimizing doses to surrounding normal tissues. The results of the treatments depend greatly on the planning. In order to administer tumorcidal doses to deep-seated tumors, it is necessary for the radiation to be crossfired from a number of different angles which are aligned to intersect inside the tumor. Treatment planning is the process developed to assure that these beams indeed intersect inside the tumor and avoid irradiating critical normal tissues such as the spinal cord or kidney. The design of MISSION-DBS in this context considers two levels of medical image transactions: local PACS (LPACS) for a single hospital and global PACS (GPACS) for linkages with external hospitals or other resources. An envisioned GPACS health care environment consists of networked LPACS among regional, national, or even international hospitals [10, 33]. As a result, the design of MISSION-DBS will build upon existing infrastructure of LPACS and bc tailored to the requirements of cooperative remote medical services in a GPACS environment. 2.1. Local PACS Requirements The complexity of medical information storage and retrieval lies in the large volume, the diverse types of image data from different modalities, and the close relationship among multimedia data in PACS. In general, medical images contain two-dimensional information of varied sizes. There are also three-dimensional spatially reconstructed images (3D volume) from computed tomography (CT) or magnetic resonance imaging (MRI) and dynamic or time-serial images from cineangiogram (CINE) or ultrasound (US). Medical images are interpreted (read) by radiologists, resulting in associated diagnostic reports (keyed-in textual data or voice-activated input) and graphic annotations called overlays (marks such as arrows, circles, lines on images to denote areas of abnormality). Furthermore, there are complex temporal, anatomical, and modality relationships between new and old image sets. There are very few deletes, no updates, but frequent routine and ad hoc retrievals on image data. In addition, performancewide, radiologists typically require that responses to image retrievals be completed
MISSION-DBS
239
within 2 s [29]. The typical peak load traffic rate of radiology services (e.g., one image reading per 8 s) imposes high throughput requirements on the design of image database systems. In a clinical environment, data availability and system reliability are so essential that diagnostic transactions should not be affected by system malfunctions, such as system down time [29]. Moreover, special considerations must be given to the interdependence of PACS with exiting information systems such as hospital information systems (HIS) and radiology information systems (RIS) [30]. 2.2. Performance Requirements for Cooperative Remote Medical Services Cooperative image-based medical services have long been used in situations of medical consultation, medical triage, education, and research [30]. Traditional image sharing in cooperative medical services was manual, in-person, time consuming, and ineffective. The advancements of communications, computation, and visual display technologies have significantly improved the feasibility and effectiveness of remote cooperative image-based medical services across a network [33]. The multimedia interactive conferencing capability is now being utilized in teleradiology to enable cooperation with remotely located experts to interactively analyze a patient's situation by viewing the images in various perspectives. In triage situations, high speed communications allow rapid access to vital information about the injured and requires real-time reference to similar patient cases for confirming diagnosis. This requires ad hoc content-based retrieval capability. For medical research and educational purposes, there is also a great need to reference data by the image content to retrieve images from a large medical data bank or image archives [26]. Limited by current communication bandwidth, image management for remote services can only be done off-line. With the availability of a high performance network, the medical image management for cooperative remote services encounter greater requirements in handling a large image volume (multimedia, 3D), real time response time, and complex queries that involve content-based retrievals. For instance, the essence of the remote distributed treatment planning experimented here is to achieve diagnostic efficiency and effectiveness through the sharing of image information, medical experts and imaging and computing equipment (see Fig. 2 for the context). Conventional treatment planning is performed on 2D slices of CT images. This is not only inaccurate but is also incapable of restricting radiation dosage to the desired region. In response, 3D conformal treatment planning has just recently emerged. Its accuracy is due to the full consideration of the threedimensional structure of a patient's anatomy. Furthermore, by planning 3D beams (as opposed to co-planer beams in the 2D planning) we can expect to deliver doses conformally on the tumor. Therefore, the database is required to reconstruct a 3D data volume, based on image sets from different modalities (possibly from different locations) in a clinically acceptable speed. Three-dimensional planning also requires a high performance 3D graphics computer to allow physicians to view the anatomy, draw beams, and evaluate dose distributions. It then requires a high speed CPU to compute and even optimize dose distributions based on the beam and anatomy information. The ideal goal,
240
CHEN AND YUN
FIG. 2.
Cooperative remote image-based medical services.
clinically, is for the computations to be near real time so that the iterative optimization can be efficiently executed. High speed communication provides a solution for remote access to these expensive facilities. Most importantly, it requires appropriate image management to prepare the necessary volumetric dataset efficiently to avoid any weak link in the communications.
3. MEDICAL INFORMATION MANAGEMENT APPROACHES To address the performance requirements for supporting efficient and effective cooperative remote medical services, we have considered the following information management approaches at both the logical level and the physical image levels.
3.1. Distributed Database Approaches Distributed database design (DDB) has been shown to better meet the various performance requirements than a central database design in the clinical environment of LPACS [28, 29, 31]. For cooperative medical services, the distributed architecture which allows parallel retrieval of images for a transaction will mitigate the communication bottlenecks encountered due to the speed disparity between the data storage component and the (high performance) data communication component. Under such a design, multimedia data can be stored in different physical media to enable parallel data retrievals, thereby reducing retrieval response time [6, 7, 15, 34]. In addition, a distributed design is also more dependable in that it can continue information service in the face of failures of individual sites or partial communication links. Moreover, cost economy is gained since a distributed design can make use of local workstations and their hierarchical data storage systems [7, 34].
MISSION-DBS
241
3.2. Knowledge-Based Approach for Pipeline (Prefetch) Operations To enhance the performance of a DDB, knowledge-based augmentation has been proposed. For example, IRES [32] is an initial application of expert system technology to computerize radiologists' knowledge to identify relevant images for a radiological exam reading. This allows image prefetch, as well as migration among data storage hierarchy and different modalities in different network locations to expedite real-time image retrieval response time. Such augmentation is particularly desirable for remote cooperative services as the medical specialist requesting data may not be familiar with the storage structure of the radiological data at the remote site. In addition to preserving expertise of experienced radiologists and improving the response time of the database system, it will also increase diagnostic accuracy by providing relevant information automatically. 3.3. Image Retrieval Approaches Most current PACS database implementation adopts attribute-based retrieval, utilizing a relational approach which treats an image as a BLOB (binary large object) (e.g., physical picture) and uses an image identifier and related attributes (e.g., logical picture) to retrieve images [8, 37]. In the most common case, the raw images (physical pictures) were processed by an image processing module and image descriptors were obtained. These image descriptors were then handled by a RDBMS (relational distributed Database management system) and used for indexing the image. Although the DDB approach, coupled with knowledge-based image prefetch, will allow high performance image retrieval, the employment of only attribute-based image retrieval limits flexibility for user query formulation in complicated diagnostic cases, as well as in ad hoc retrieval situations. The inflexibility is in two aspects: first, due to query language restriction and, second, because it is difficult to exhaust all the possible descriptions of objects and their (anatomical, spatial) relationships on a certain image by attributes. The limitations are most felt in cases when user queries contain attributes that are not predefined in the databases. To allow more flexibility in image retrieval, techniques used for document bases in traditional information retrieval (IR) field [3, 36] have been proposed for image retrieval in that the retrieval of images are mediated via the retrieval of textual description (in natural language or keywords) of images [14]. This full-text-based or IR-based approach is often referred as pseudo-content-based retrieval. In a medical environment, the diagnostic report(s) that accompanies each image is a perfect intermediate. Various models and concept-based [14] techniques that utilize application semantics and user-feedback have been shown to have better recall and precision rate in image retrieval than free text description. Although the IR-based approach can offer much flexibility in query formulation, there are still needs for content-based (or pictorial query-by-example) query capabilities [26, 39, 41]. This is because sometimes images are hard to comprehend, interpret, or describe as a picture is worth a thousand words. For instance, if a doctor encounters a difficult case and needs some supporting evidence to make his diagnosis, the content-based capability would allow him to consult
242
CHEN AND YUN
similar images associated with a confirmed diagnosis by presenting just the images at hand. This capability is particularly important for medical consultation, research, and education [26]. In a content-based approach, features of images are analyzed and extracted utilizing AI or neural network (NN) techniques, such as object recognition and pattern recognition, the retrieval of certain images are then based on the similarity measures between the query objects and the stored objects [39]. 3.4. Image Processing The provision of content-based retrieval is the most difficult as it requires sophisticated, intelligent processing at the physical picture level to ``understand'' the image [5]. A critical step to ``understaned'' images is to decompose a picture into meaningful partsto separate objects from background and to distinguish among objects after an image is properly digitized and coded (e.g., with gray level). Segmentation techniques include simple pattern recognition by thresholding, boundary detection, contour following, etc. [8]. Many advanced pattern recognition techniques employ a neural network (NN) approach. It then requires domain knowledge to associate a segmented object with a meaning. In addition, the 3D data visualization is another that requires intelligent image processing, such as the segmentation of 2D slices for reconstructing a 3D volume, and the projection of a 3D object to a 2D screen. For supporting the goals of MISSION-DBS, we have developed symbolic encoding techniques [26], pursued an advanced neural network feature extractor [24, 25], and explored the parallel execution of 3D volume rendering code [44]. These techniques will be integrated into a ``toolset'' in our MISSION-DBS. 3.5. Poly-Paradigmic Approach to Content-Based Retrieval We have investigated the integration of image management at a logical and physical image level for effective support of cooperative remote medical services. Our initial research resulted in a poly-paradigmic approach for achieving efficient content-based retrieval. In our proposed approach, the symbolic coding of images, advanced NN feature extractors and semantic data modeling are utilized to reduce the content understanding problem to one that is similar to a textual database retrieval problem [25]. Image features are extracted by an advanced NN feature extractor and stored in a metadatabase. Application semantics will be associated with each feature. The symbolic encoding technique is applied to the image feature which will be stored as an image tag or ``signature.'' In cases of pictorial query, users can present an image as an ``example image.'' Features in the example image will be extracted, coded, and then compared with prestored ``signatures.'' Searching for a pattern image in the user query can then be automated as a computer operation that simply matches the higher level abstract signatures of the feature with those of the stored images. We have investigated the use of DDB abstraction mechanisms (generalization and aggregation) used for data fragmentation for classifying ``signatures'' symbolically coded. As each feature is associated with data semantics or concepts, attributes-based or concept-based retrieval can be executed
MISSION-DBS
FIG. 3.
243
Poly-paradigmic approach for content-based retrieval.
using the same conceptual data model. This approach represents savings in modeling efforts and will reduce inconsistency among the data semantics modeled. We have developed a NN-based holographic approach utilizing only pixel presentation for attention-modulated content-based retrieval [24, 25]. It will be used for complementing our poly-paradigmic approach shown in Fig. 3.
4. MISSION-DBS ARCHITECTURE To meet the information management requirements of PACS in general and cooperative image-based medical service in particular, such techniques as DDB design, knowledge-based ``prefetch,'' and ``pipeline,'' content-based image retrieval, and intelligent image processing techniques must be brought together. Attributebased, IR-based, and content-based retrieval techniques should all be incorporated because no single image retrieval approach satisfies all the requirements of medical information, management. The recent commercial development in DBMS has been encouraging toward developing intelligent access to medical image information. The commercial RDBMSs, such as Oracle V.8 Sybase System 11, are all capable of handling distributed image data. Full-text DBMSs (e.g., TOPIC, TextDBMS) have new improvements in full text retrieval. Commercial products for simple contentbased retrieval have been announced, such as IBM's QBIC (Query By Image Content). There has also been a rapid development of parallel database servers [4]. Figure 4 depicts the system architecture of MISSION-DBS, considering the special characteristics and requirements of medical image management discussed above. MISSION-DBS resides in a global distributed environment with a high performance communication network. The architecture combines the efficiency of the distributed database technology, the flexibility of a knowledge-based approach, and the power of AI techniques to address medical multimedia information management requirements. Many research results from previous PACS research (e.g., hierarchical storage management and emerging technology, such as parallel query processors) are integrated into the architecture. The system architecture defines interfaces among various technological building blocks and thus serves as a research framework to guide our development effort. Our current research effort concentrated on the modules of the shaded areas.
244
CHEN AND YUN
A plausible design for MISSION-DBS could be to ``loosely couple'' the modules for each technology identified. However, as common application semantics underlies the design and modeling of DDB and AI techniques, unified semantic modeling is therefore examined to enable the integration of modules for both user query formulation at the logical image level and for image encoding, processing and automatic image indexing at the physical image level. The initial results in combining the DDB abstraction mechanism for feature indexing have led lo our adoption of a ``tightly coupled'' architecture for MlSSION-DBS that will permit quick and intelligent access to medical images. As a result, the distributed database catalog system is integrated with knowledge-based components in the Metadatabase (see Fig. 4). The system components are described below and the integrated modeling issues will be discussed in the section that follows. 4.1. Graphical User Interface (GUI) The graphical user interface of MISSION-DBS resides in the PACS diagnostic or viewing workstations [19, 25], where multimedia objects can be created, processed, manipulated, searched, browsed, and retrieved. Interactive relational query language, natural language-based, as well as pictorial query language should be provided for different types of database queries. Multimedia input devices and an interactive image processing language for voice input of diagnostic reports, drawing overlays on images and performing pictorial query-by-example are provided here. Flexible image viewing capabilities (e.g., zooming, image browsing, 3D image
FIG. 4.
MISSION-DBS system architecture.
MISSION-DBS
245
motion viewing such as US or CINE) are included. Multimedia conferencing video inputoutput is part of the GUI for remote consultation. In addition, the following modules are important to the performance of medical image information retrieval and manipulation: Dialogue manager. This module will conduct dialogueconversation with users to clarify incomplete or ambiguous information requests and assist users in general multimedia information manipulation tasks and system usage. Tutorials and help facilities should be available. Spatial analyzer. This module will allow flexible Spatial analysis on images to identify objects or diagnosis of abnormality in an image. For example, the size of a tumor and its position in relations to other organs. Conceptkeyword manager. This module will maintain (createupdate) concepts, keywords and a thesaurus to assist users in formulating multimedia database queries and also allow automatic indexing of multimedia objects. Latent query register. Preprocessed queries for routine readings are stored and can be selected by users to speed up query performance. Query-by-picture example manager. This module will handle the ``interpretation'' of the query image, make calls to metadatabase to find matches in the image feature catalogue. CADx. CADx software for detecting abnormality or patterns that are easily missed by trained eyes is highly needed, such as for lung nodules [19, 44]. Volume renderer. This module will allow fast and parallel 3D volumetric data rendering and viewing. Remote consultation multimedia IO manager. This module manages multimedia input and output for interactive remote consultation sessions. 4.2. Metadatabase The metadatabase stores the following data and knowledge to facilitate intelligent access and manipulation of distributed medical information: DDB schemata. This module includes conceptual schema, logical schema, fragmentation schema, allocation schema, local mapping, access method description, and patterns of querytransaction activities to allow query processing, such as quick search and access to local data. Information such as usage statistics, protection, and access authorization are also stored to maintain data security and integrity. Operational 6 contextual 6 usage knowledge. This module is for use in ``reasoning'' about user queries, image understanding and formulate search strategies. It contains application domain knowledge and medical expert rules as well as grammatical rules of natural language. It (1) facilitates users in formulating multimedia database queries; (2) facilitates natural language (IR-based) query processing; (3) facilitates image object recognition and understanding (contentbased); (4) facilitates multimedia objects classification and indexing; and (5) enhances
246
CHEN AND YUN
distributed database performance by automatic image migration and ``prefetching'' to expedite image retrieval. Thesaurus and concepts. This contains language rules for resolving data semantics conflicts among local databases and to facilitate full-text user queries. Image feature catalogue. This module stores preprocessed image objects to facilitate content-based retrieval by matching stored objects with requested objects presented by the user. 4.3. Distributed Database Management System (DDBMS) The DDBMS hides the multimedia data distribution from the users and provides data fragmentation, allocation, and replication transparency to users [4, 14]. It manages all the complicated distributed operation problems for the users, including distributed query processing [17], distributed concurrencyrecovery control, metadatabase management, distributed security control [31], and determines physical data storage structure, format, indices, and search strategies [28]. The following modules are essential components in DDBMS: SQL 6 full-text processor 6 content-based query processors. These are three submodules. Each query processor decomposes a user request of multimedia information to subtransactions for local DBMSs or specialized DBMS components (each of them handles certain medium type), performs all relevant IO operations, and then integrates the results of these subtransactions. The SQL processor understands SQL. The full-text processor utilizes the thesaurus and concepts in the metadatabase for query processing, while content-based query processors utilize the image processing toolset. For facilitating content-based image retrieval, the goal of the module is to automatically extract from a medical image the symbolic descriptions that are meaningful or relevant to user queries. A knowledge base that contains descriptions (or symbolic coding that is purely numeric or vector-based) of the objects or of generalized object classes will be constructed first to enable content-based image database query. The rule-based inference mechanism and an interactive image-processing language are then used to help DDBMS retrieve images. Concurrencyrecovery controller. This module ensures serializability and recovery when transactions are executed concurrently or in the event of system failures [29]. Communication manager. This module handles data format and transfer functions according to the underlying multimedia communication protocols (gateway and router functions) [6, 44]. Parallel query processor. This module performs parallel query to the database using a back-end machine that offloads IO bound functions from DDBMS. For instance, a query can contain several predicates for different medium types or different image features and each predicate will be searched in parallel. The parallel machine can improve the system's reliability through complexity reduction and enhances the system's performance through specialization, increased parallelism, and processing power [15].
MISSION-DBS
247
Local data Manager. Each local data manager is responsible for executing individual subtransaction for each local database. It contains a local DBMS, a file manager for each medium that manages storage spaces and a migration manager that moves medical multimedia data up or down within a storage hierarchy [29]. 4.4. Image Processing Toolset The image processor consists of pattern-recognition and image-processing routines that are called by the query processor. These routines are designed to process images and extract spatial and structural information from images of different modality types. The afore-mentioned intelligent image processing routines are integrated into this module [24, 25]. 4.5. Storage Subsystem The storage subsystem stores multimedia data according to different data types (structured data, text, vector-based overlays, 2D and 3D images) and data fragments according to the logical data organization and transaction patterns. These data are managed by local DBMSs specialized to their respective medium. Moreover, since data retrievals exhibit a temporal correlation such that newly generated images are retrieved more frequently and older ones less frequently, it is cost-effective to have a hierarchical storage architecture, where older images are stored in a lower-cost but slower storage system (e.g., optical disks) and more recent images in a more expensive but faster storage system (e.g., magnetic disks). RAID is considered for better reliability and read-only performance [2]. 5. INTEGRATED SEMANTICS MODELING ISSUES It is clearly revealed from the system architecture that the fusion of research results and techniques from different research fields is essential to enable efficient, as well as intelligent, management of medical multimedia information. While the modeling, design, and implementation of each component in the system architecture already poses challenges for each component's distinct research field, the development of integrated modeling for MISSION-DBS merits immediate attention. Common application semantics underlies the modeling and design of a distributed database (DDB) and AI-based techniques for text and image understanding in MISSION DBS. A uniform approach to model semantics would be an important step toward simplifying the interfaces among these components. In particular, the knowledge representation would be facilitated by a single semantic formalism that could represent concept-based query semantics, as well as logic database design [20, 35]. The integrated modeling of common application knowledge (rules, facts, or data) will have an advantage in reducing duplicated modeling efforts and increasing the efficiency of metadata management. In short, the unified modeling constructs and abstraction mechanisms (1) must represent medical multimedia data and usage knowledge easily and effectively; (2) must allow transformation from conceptual level to logical level (including data distribution)
248
CHEN AND YUN
and physical level representation for implementation so that multimedia data can be easily stored, indexed, searched, and retrieved; (3) must support distributed multimedia information access (e.g., attribute-based, text-based, and content-based retrievals), and (4) should facilitate multimedia data and knowledge integration and automatic image object classification and indexing. We are extending a DDB design methodology [9, 11] for meeting these requirements. 6. CONCLUSION Technological fusion of distributed database and artificial intelligence (knowledge-base augmentation and intelligent image processing) is necessary to meet the special requirements of total medical image management for cooperative remote medical services, taking full advantage of high performance communication capabilities for delivering novel cooperative remote medical services. Our initial results in integrating image management at the logical and physical levels is through the development of the ploy-paradigmic approach to content-based retrieval. The MISSION-DBS architecture reflects the integration of various approaches necessary for effective image management. The MlSSION-DBS system architecture also serves as a framework for our continued research and development effort in providing total medical image management for future health care in a GPACS environment. We have focused on developing the capabilities of high performance image retrieval and effective content-based retrieval. The research results are expected to directly benefit our experimental remote image-based medical services (e.g., distributed treatment planning) in our MISSION project and will be useful for other future novel cooperative remote medical services. Particularly, the knowledge-based retrieval strategies and content-base retrieval capabilities will be very desirable for the medical education and research on a global scale. It is an objectiveand, in our opinion, will eventually become a requirement for the PACS database system to be able to manage (e.g., store, process, retrieve and manipulate) voluminous 2D or 3D image data in the same manner (e.g., speed and flexibility) as for traditional alphanumerical data. Although current contentbased image retrieval performance is still far from ideal, our research effort to integrate DDB and AI techniques shows promise toward this objective. ACKNOWLEDGMENT This research is supported by a grant from DOD ARPA (Advanced Research Project Agency) (Award HJ1500-3175 0562AAP*DAR3A634).
REFERENCES 1. J. R. Bach, S. Paul, and R. Jain, A visual information management system for the interactive retrieval of faces, IEEE Trans. Knowledge Data Eng. 5, 4 (August 1993), 619628. 2. K. Bachus, P. Houston, and E. Longsworth, Right as RAID, Corporate Comput. 2, 5 (May 1993), 6164.
MISSION-DBS
249
3. E. Bertino, F. Rabitti, and G. Simon, Query processing in a multimedia document system, ACM Trans. Office Inform. Systems 6, 1 (January 1988), 141. 4. R. Bloor, The coming of parallel servers, DBMS 7, 5 (May 1994), 1416. 5. J. Brolio, B. A. Draper, J. R. Beveridge, and A. R. Hanson, ISR: A database for symbolic processing in computer vision, IEEE Comput. (December 1989), 2230. 6. G. Bucci, R. Detti, V. Pasqui, and S. Nativi, Sharing multimedia data over a client-server network, IEEE Multimedia 1, 3 (Fall 1994), 4455. 7. S. Ceri and G. Pelagatti, ``Distributed Databases: Principles and Systems,'' McGrawHill, New York, 1984. 8. S.-K. Chang, ``Principles of Pictorial Information Systems Design,'' PrenticeHall, Englewood Cliffs, NJ, 1989. 9. H.-M. Chen Garcia, ``A Semantics-Based Methodology for Integrated Distributed Database Design: Toward Combined Logical and Fragmentation Design and Design Automation,'' Ph.D. dissertation, University of Arizona, (1992). 10. H.-M. Chen Garcia and O. R. Liu Sheng, Medical multi-media information management: A research framework, in ``Proceedings of the 1993 Pan Pacific Conference on Information Systems, Kaoshsiung, Taiwan, R.O.C., May 30June 1, 1993,'' pp. 3947. 11. H.-M. Chen Garcia and O. R. Liu Sheng, An entity-relationship-based methodology for distributed database design: An integrated approach toward combined logical and fragmentation design, in ``Proceedings of the 11th International Conference on the Entity Relationship Approach, Karlsruhe, Germany, October 79, 1992.'' 12. F. Chen, M. Hearst, J. Kupiec, J. Pedersen, and L. Wilcox, Metadata for mixed-media access, SIGMOD RECORD 23, 4 (December 1994), 6471. 14. T.-S. Chua, H.-K. Pung, G.-J. Lu, and H.-S. Jong, A concept-based image retrieval system, in ``Proceedings of 27th Hawaiian International Conference on System Sciences, Maui, Hawaii, Jan. 46, 1994.'' 15. S. Ghandeharizadeh and L. Ramos, Continuous retrieval of multimedia data using parallelism, IEEE Trans. Knowledge Data Eng. 5, 4 (August 1993), 658669. 16. M. Goldberg and J. Herscovitz, Image organization and Navigation using dynamic icons for a radiology workstation, in ``Proceedings of IMAC '91, April 1013, Kyoto, Japan, IEEE, 1991,'' pp. 371377. 17. H. Edelstein, Distributed databases, DBMS (September 1990), 3648. 18. L. R. Hevner and A. Rao, Distributed data allocation strategies, in ``Advances in Computers'' (M. C. Yovits, Ed.), pp. 121155, Academic Press, New York, 1988. 19. M. Hosoba et al., PACS workstation for computer assisted image diagnosis, in ``Proceedings SPIE Medical Imaging Conference IV,'' February 49, 1990, Newport Beach, CA. 20. R. Hull and R. King, Semantic database modeling: Survey, applications, and research issues, ACM Comput. Surv. 19, 3 (September 1987), 201260. 21. R. Kasturi, R. Fernandez, M. L. Amlani, and W.-C. Feng, Map data processing in geographic information systems, IEEE Comput. (December 1989), 1021. 22. M. A. Ketabchi and V. Berzins, Modeling and managing CAD databases, IEEE Comput. (February 1987), 93102. 23. J. I. Khan and D. Y. Y. Yun, Chaotic Vectors and a proposal for multidimensional associative network, in ``Proceedings IS6TSPIE Electronic Imaging: Science 6 Technology,'' Feb. 610, 1994, San Jose, CA. 24. J. I. Khan and D. Y. Y. Yun, Feature-based contraction of sparse holographic associative memory, in ``Proceedings World Congress on Neural Networks, San Diego, June 59, 1994,'' pp. IV2634. 25. P. Kim, H. W. Park, and D. R. Haynor, Requirements for PACS workstations, in ``Proceedings of IMAC '91, April 1013, Kyoto, Japan, 1991,'' pp. 3642.
250
CHEN AND YUN
26. P. Kofakis, A. Karmirantzos, Y. Kavaklis, E. Petrakis, and S. Orphanoudakis, Image archiving by content: An object-oriented approach, in ``Proceedings SPIE Medical Imaging Conference IV, February 49, Newport Beach, CA, 1990.'' 28. O. R. Liu Sheng, C. P. Wei, and H.-M. Chen Garcia, Design and implementation of a distributed PACS database system, in ``SPIE Proceedings of Medical Imaging Conference VIII, February 49, 1992, Newport Beach, CA.'' 29. O. R. Liu Sheng and H.-M. Chen Garcia, The design of medical image databases: A distributed approach, in ``IEEE Proceedings of International Phoenix Conference on Computers and Communications, Scottsdale, AZ, March 2123, 1990,'' pp. 288295. 30. O. R. Liu Sheng and H.-M. Chen Garcia, Information management in hospitals: An integrated approach, in ``IEEE Proceedings of International Phoenix Conference on Computers and Communications, Scottsdale, AZ, March 2123, 1990,'' pp. 296303. 31. O. R. Liu Sheng, H.-M. Chen Garcia and C.-P. Wei, Distributed database design and modeling for PACS, in ``SPIE Proceedings of Medical Imaging Conference IV, February 49, 1990, Newport Beach, CA,'' pp. 256269. 32. O. R. Liu Sheng, H.-C. Wang and H.-M. Chen Garcia, IRESImage retrieval expert system, in ``SPIE Proceedings of Medical Imaging Conference IV, February 49, 1990, Newport Beach, CA,'' pp. 832841. 33. R. Martinez, Y. Alsafadi, and J. Kim, Design of multimedia global PACS distributed computing environment, in ``IEEE Proceedings of hawaiian International Conference on Systems Sciences (HICSS '28), 1995.'' 34. T. M. Ozsu and P. Valduriez, Distributed database systems: Where are we now?, IEEE Comput. (August 1991), 6878. 35. J. Peckham and F. Maryanski, Semantic data models, ACM Comput. Surv. 20, 3 (September 1988), 154189. 36. G. Salton, ``Automatic Text Processing: The Transformation, Analysis and Retrieval of Information by Computer,'' AddisonWesley, Reading, MA, 1989. 37. G. N. M. Shdhakar, A. Karmouch, and N. D. Georganas, Design and performance evaluation considerations of a multimedia medical database, IEEE Transactions on Knowledge and Data Engineering 5, 5 (October 1993), 888894. 38. J. M. Smith and D. C. P. Smith, Database abstraction: aggregation and generalization, ACM Transactions on Database System 20, 2 (June 1977), 105133. 39. Stephen W. Smoliar and HongJiang Zhang, Content-Based Video Indexing and Retrieval of Facial Images, IEEE Multimedia 1, 2 (Summer 1994), pp. 6272. 40. J. F. Sowa, Knowledge representation in databases, expert systems and natural languages, (R. A. Meersman, Z. Shi, and C.-H. Kung, Eds.), Artificial Intelligence in Databases and Information Systems, Elsevier Science, pp. 1741, North-Holland, 1990. 41. J. K. Wu, Y. H. Ang, P. Lam, H. H. Loh, and A. Narasimhalu, Desai, Inference and retrieval of facial images, Multimedia Systems 2, 1 (1994), 114. 42. Atsuo Yoshitaka, Setsuko Kishida, Masahito Hirakawa, and Tadao Ichikawa, Knowledge-assisted content-based retrieval for multimedia databases, IEEE Multimedia 1, 4 (Winter 1994), 1221. 43. D. Y. Y. Yun, Symbolic labeling for image management (SLIM), working paper, University of Hawaii, 1993. 44. D. Y. Y. Yun and H.-M. Garcia, High performance medical image communications via ACTS-HDR, in ``Medical Imaging '95, San diego, Feb. 26Mar. 28, 1995.''