Intelligent image content semantic description for cardiac 3D visualisations

Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

Contents lists available at ScienceDirect

Engineering Applications of Artificial Intelligence journal homepage: www.elsevier.com/locate/engappai

Intelligent image content semantic description for cardiac 3D visualisations Miros"aw Trzupek, Marek R. Ogiela n, Ryszard Tadeusiewicz AGH University of Science and Technology, Institute of Automatics, 30 Mickiewicza Ave, 30-059 Krakow, Poland

a r t i c l e i n f o

a b s t r a c t

Available online 12 June 2011

In these days of the rapid development of diagnostic equipment with increasingly sophisticated technology it is necessary to put more emphasis on implementing processes for the computer support of medical diagnostics, which are more and more often used to automate diagnostic procedures carried out in healthcare. Current research shows that a significant part of diagnostic imaging, including e.g. of coronary arteries, is still difficult to automatically assess using computer analysis techniques aimed at extracting information having semantic meaning. This mainly applies to images that show many structures simultaneously, as well as 3D images. In this context, this publication presents new capabilities to formulate semantic descriptions of 3D structures of coronary vascularisation using graph formalisms. The proposed syntactic semantic description makes it possible to intelligently model the examined structure and then to automatically find the locations of significant stenoses in coronary arteries and identify their morphometric diagnostic parameters. In this research, images originating from diagnostic examinations with 64-slice spiral computed tomography were used. & 2011 Elsevier Ltd. All rights reserved.

Keywords: Image semantic description Syntactic pattern analysis Medical image understanding Spatial modelling of coronary vessels Computer-aided diagnosis (CAD)

1. Introduction Heart and circulatory diseases are among the most serious and frequent threats to life in developed countries and at the same time constitute a crucial diagnostic problem of the 21st century. Every year, over 19 million people globally suffer sudden, severe coronary incidents (Yusuf et al., 2001). The increasing number of hospitalised patients drives the search for tools capable of helping the physician take therapeutic decisions (second-opinion assistance). This is particularly significant e.g. in the case of screening a selected part of the population, where the physician analysing a large set of diagnostic image data might, because of fatigue leading to poor concentration, miss a detail that could turn out to be medically significant in the process of formulating the diagnosis, the prognosis or the current treatment, and this may bring about significant consequences for the continued correct hospitalisation of the patient. What is more, the impressive progress in apparatuses for acquiring medical images means that new generations of diagnostic apparatuses enter the market. As a result, the images analysed so far, familiar to the physician, are replaced with other, better ones. Although the latter contain more information, they are unfortunately unfamiliar. In the present situation it is also increasingly difficult to be an expert who could efficiently assess all the currently available types of medical images. The reason is that the wide range of diagnostic

n

Corresponding author. Tel.: þ48 12 617 38 54; fax: þ 48 12 634 15 68. E-mail addresses: [email protected] (M. Trzupek), [email protected] (M.R. Ogiela), [email protected] (R. Tadeusiewicz). 0952-1976/$ - see front matter & 2011 Elsevier Ltd. All rights reserved. doi:10.1016/j.engappai.2011.05.005

apparatuses for visualising the same organ makes it possible to present the organ in various visual forms depending on the technique used to acquire the image. All these and many other circumstances have led the authors of this publication to carry out wide-ranging research to find new solutions that could help develop intelligent systems for medical diagnostic support (CAD—computer-aided diagnosis) and also significantly contribute to solving this very important problem. The impressive technological progress in medical image diagnostics and the wide opportunities for 3D visualisation of human organs have significantly improved the efficiency of medical diagnostic tasks. The 3D reconstructions of examined medical structures obtained by rendering make it possible to truly represent the selected organ (including the changes in its texture), allowing its external and internal morphology to be observed precisely (Lewandowski et al., 2007). Such high technologies of image processing are today used in almost all types of diagnostic examinations based on digital technologies and of surgical jobs performed with the use of medical robots. As a result, it has become possible to identify a greater number of qualitative parameters of the examined structure (e.g. in the case of coronary arteries), which may be significant for making the correct diagnosis, and which could not be identified if the examination was made using a conventional method (2D imaging; Katritsis et al., 2008; Sirol et al., 2009). Diagnostic examinations of coronary vascularisation obtained from various image modalities (conventional angiography, intra-vascular USG, computed tomography or magnetic resonance) constitute complementary examinations (Bob Meijboom et al., 2008) and depending on the type of complaint the patient suffers, the physician decides which of the diagnostic apparatuses is to be

M. Trzupek et al. / Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

used. However, it is obvious that all these achievements in the visualisation technology field offer limited opportunities for automating the interpretation of the diagnostic images acquired. This is mainly due to the difficulties that informatics encounters in formally describing and modelling complex thought processes taking place in the human mind which enable the semantic interpretation of analysed medical images.

2. Semantic image understanding technique The number of IT tools developed to improve the capability of visual assessment of a given diagnostic image is high, and their operation can be considered satisfactory. However, there are no IT tools to be used for the automatic support of diagnostic or decision-making processes of physicians when they analyse an image and interpret it. Such intelligent IT systems supporting the thought processes of a physician analysing complex cases simply have not been developed yet. There are, however, tools that support the work of a diagnostician by making quantitative measurements of pathologies (e.g. stenoses in the case of coronary arteries) depicted in the image (Yin Wang and Liatsis, 2009; Oncel et al., 2007), which obviously makes his/her work easier, but only understanding the essence of the disease process allows the appropriate diagnosis to be made and the correct therapy to be prescribed. The reason for this deficit is a series of scientific and technical problems encountered by developers of intelligent diagnostic systems. One of the main difficulties in developing universal, intelligent systems for medical image diagnostics is the huge variety of images, both healthy and pathological, which have to be taken into account when intelligently supporting physicians interpreting them. In particular, the aforementioned varied shapes of morphological elements make it difficult to create a universal pattern defining the model shape of a healthy organ, or a pathological one. Yet the input of precisely such a pattern is required by a computer, using automatic image recognition technologies already well known and frequently used, as the IT technologies used are to a large extent based on the intuition of measuring the similarity of the case currently analysed to that abstract pattern. These technologies frequently fail if there are unexpected changes to the shapes of analysed organs caused by the disease process or individual variability. For this reason it is necessary to use those advanced artificial intelligence techniques and computational intelligence techniques that can generalise the recorded image patterns. What is particularly important is to use intelligent description methods that would ignore individual characteristics of the patient examined and characteristics dependant on the specific form of the disease unit considered, while at the same time making it possible to locate significant morphology changes and also to interpret and determine their diagnostic significance. Such methods, aimed at focusing the image description on diagnostically significant properties to the maximum extent can then be used in intelligent computer-aided diagnostics systems (CAD). The last dozen or so years have mainly seen attempts to detect significant coronary artery stenoses (Yin Wang and Liatsis, 2009; Oncel et al., 2007) and coronary artery diseases using various techniques, e.g. neural networks (Kurgan et al., 2001), waveletbased fuzzy neural networks (Akay et al., 1994), Bayesian methods (Cios et al., 1989). In this context one cannot find solutions in the form of intelligent systems that could imitate the thought processes taking place in the mind of the diagnosing physicians and thus generate a specific diagnosis, and not just a quantitative assessment of pathologies present. The problem presented is complicated. Such a system cannot be developed using classical methods of the traditional automatic recognition and

1411

classification (Meyer-Baese, 2003)—it is necessary to make use of a new technique of their automatic understanding leading to formulating semantic descriptions of analysed images. Methods presented here refer to earlier publications by the authors and use the concept of the automatic understanding of medical images (Tadeusiewicz and Ogiela, 2004). The authors’ success in developing tools for automatically understanding flat images (Tadeusiewicz and Ogiela, 2004; Ogiela and Tadeusiewicz, 2002, 2008) has encouraged them to propose similar methods for 3D representations of biological structures provided by modern imaging systems. Considerations presented herein will particularly apply to 3D visualisations of coronary vessels, which will form the foundation of the detailed part of this publication, but the formulation of the problem itself and the solution methodology outlined here can have much wider application. At this stage we should note that we will consider a given computer interpretation of the 3D image analysed to exhaust the criteria of ‘‘automatic understanding’’ if this interpretation suggests directly what the 3D structure of the analysed organ is, what the spatial relationship of this structure to other organs and body parts is, and what the consequences of this structure and this relationship are. Taking into account the needs of the diagnostics and the therapy, it is expected that the automatic understanding process should be capable of indicating in what place, in what way and with what consequences the disease process has disturbed the 3D topology of the structure examined. Based on the understanding of a 3D medical image thus defined, we can give the physician far more and far more valuable premises for his/her therapeutic decisions than we could if we were using the traditional image recognition paradigm ending in a decision on the name of the diagnostic unit recognised by reference to an a-priori adopted taxonomy of diseases considered. Adjective ‘‘semantic’’ is currently used in various context, e.g. in popular term ‘‘semantic web’’ and also in many other contexts. Therefore before discussion of our method of cognitive interpretation of coronary arteries medical visualisation we must first define, how we understand these adjective in proposed here term ‘‘semantic analysis of medical images’’. Especially we must compare terms semantic web and semantic analysis of images. Semantic web is now a popular and a very general idea, proposed for intelligent information retrieval models. It is particularly effective for bag-of-words document representation, when user needs particular information, which is hidden in text, but the exact form of such information is not known and not predictable. Therefore in semantic web central point of consideration is concentration on relations between words and phrases, which can be different in form, but can have the same meaning. In semantic webs classic (most widely used) idea of the tool applied for this purpose is so called ‘‘ontology’’. Shortly speaking ‘‘ontology’’ is computer representation of our knowledge about relations between words and phrases—mostly represented in the form of edge labelled directed graphs. This relatively simple general idea can be enriched with many smart details and many useful tools, therefore ten years after introduction of ‘‘semantic web’’ and ‘‘ontology based approach’’ they are still the basis for many proposed and discussed detailed solutions. Recent advances in semantic web technologies have resulted in methods and tools that allow creating and managing domain knowledge. They influence the way and form of representing documents in the memory of computers, approaches to analyse documents, techniques to mine and retrieve, etc. There are three main differences between mentioned above idea of ‘‘semantic web’’ and discussed in this paper idea ‘‘semantic analysis of medical images’’. First and most evident difference is related to the form of taken into account information. Despite

1412

M. Trzupek et al. / Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

many efforts dedicated to generalisation of the main idea of ‘‘ontology’’, etc. for multimedia data form, semantic web technologies are until now dedicated mainly to text documents. Intelligent and content oriented searching for video, voice and speech raises new challenging problems to semantic web retrieval systems. On contrary to this situation our approach leading to semantic analysis of medical images is dedicated for information in the pictorial form and absolutely ignore every texts (if any exist in considered system). Second difference is related to the form of knowledge used for intelligent problem solving in both considered methodologies. Semantic web methodology is based on the outer relations between whole pieces of information (words and phrases). Ontologies are collections of facts and rules describing relation between such items (words and phrases) where internal structure of information, which is taken into account, is not used (words and phrases are treated as ‘‘atoms’’—terms without internal structure). On contrary to this approach in our propositions the internal structure of the analysed data structures (images) is the most important factor taken into account during analysis, data description and final interpretation. Prior to analysis and extraction of semantic content of the image we must perform image segmentation. As a result of this segmentation we obtain many types of image elements (called sometimes graphical primitives), which are ‘‘nouns’’ for linguistic descriptions of image content in graph grammars designed by us specially for these purposes. Simultaneously with segmentation of considered image for image primitives also special kind of relations between such elements are detected. Such relations can be used as ‘‘verbs’’ in our linguistic description of the image content. Off course both graphical ‘‘nouns’’ and ‘‘verbs’’ in considered description are goal-oriented. It means we must first select this type of graphical primitives and this type of graphical relations, which are important from the point of view of further image interpretation. For example in medical applications graphical primitives are connected with particular anatomical structures, and the relations represents connections between such structures, their dimensions, distances, angles, shapes, etc. Of course general draft of our idea given in previous paragraph is the simplified one. In practical allocation of mentioned above idea we can enrich this model taking into account some different classes (semantic categories) of graphical primitives used as a ‘‘bricks’’ for building of image content description, as well we can differentiate more complicated relations between selected elements, which can be treated not only as a ‘‘verbs’’ in used language, but also as ‘‘adjectives’’ or ‘‘adverbs’’. Taking into account all mentioned possibilities we can use graph language as powerful tool for image content description, what is the most important part of our proposition. Third difference (between ‘‘semantic web’’ approach and ‘‘semantic analysis of the images’’) is related to the way of use of discovered content of considered information. In semantic web approach structural contents extracted from the ‘‘bag-of-words’’ texts representations are used for information classification, retrieval and also association. Semantic analysis of the images

(especially medical ones) can have more ambitious purposes. We try extract semantic meaning of the image for diagnostic and therapeutic purposes, therefore we must establish relation between semantic content of considered image and resources of medical knowledge collected in system as an equivalent of ‘‘internal wisdom’’. Using ‘‘cognitive resonance’’ method we can take into account relations between content of particular image and based on knowledge and experience expectations, which must be satisfied if considered image can be understood as evidence for some particular medical (or other) conclusion. The techniques of the semantic analysis of 3D medical images of coronary vessels proposed by the authors refer to a new concept of ‘‘cognitive resonance’’ (Tadeusiewicz and Ogiela, 2004; Ogiela and Tadeusiewicz, 2008), which is a key element of the image understanding process. Unlike in the classical image recognition scheme (Meyer-Baese, 2003) where we are dealing with a one-way information flow (Fig. 1), the new technique of image understanding features a two-way information flow (Fig. 2) and consists in using the interference between the knowledge collected in the set of productions in the form of graph grammatical rules and the stream of data coming from the system analysing the examined image (morphometric and structural data obtained for visualisations analysed one after another). The knowledge collected in the set of productions in the form of graph grammatical rules results from a-priori knowledge of the diagnostic and prognostic significance of specified elements of the examined structure found in the considered class of medical images. These expectations are a kind of specific hypotheses about the semantic interpretation of the image content. As a result of the interference leading to cognitive resonance, certain coincidences gain in validity and thus confirm one of the possible hypotheses while others become less valid. If no hypothesis is confirmed, then we receive feedback that either we are dealing with an unusual case that has not been considered when developing the knowledge base in the form of graph grammatical rules, or we are dealing with a diagnostic image which, for some reason, was not acquired correctly and thus contains artefacts that hinder its further correct interpretation. A material advantage of the methodology, from the perspective of bio-cybernetic mechanisms, is the deep analogy of the operation of the presented structural analysis model to the cognitive interpretation mechanisms proceeding in the human brain, known as knowledge-based perception and reasoning, which is another premise confirming the legitimacy of selecting the proposed methodology and the research necessary for its further development.

3. Graph-based formalisms in intelligent semantic description and analysis of coronary vessels This section presents methods for making semantic models of 3D reconstructions of coronary arteries and interpretation capabilities based on these models.

Fig. 1. Traditional methods of medical image recognition are characterised by a one-way scheme of data flow.

M. Trzupek et al. / Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

1413

Fig. 2. Medical image understanding technology is characterised by a two-way flow of information with the ‘‘cognitive resonance’’ mechanism.

3.1. Preliminary analysis of digital medical images of 3D reconstructions of coronary vessels Research work was conducted on images from diagnostic examinations made using 64-slice spiral computed tomography in the form of animations saved as AVI (MPEG4) files with the 512  512 pixel format. The analysed images were acquired with a SOMATOM tomograph (Get the Entire Picture, 2004), which offers a number of opportunities to acquire data and make 3D reconstructions. Here it should be noted that our work omits the preprocessing stage, as coronary arteries are extracted using dedicated software integrated with the tomograph. In this software, predefined procedures have been implemented, which allow the vascularisation to be quickly extracted from the visible structures of the cardiac muscle. Thus high quality images showing coronary arteries of the examined patients without ancillary elements were acquired. Consequently, this pre-processing stage was omitted, which allowed us to focus directly on the arteries examined. Since image data has been saved in the form of animations showing coronary vessels in various projections, for the further analysis we should select the appropriate projection which will show the examined coronary vessels in the most transparent form most convenient for describing and interpreting. In the clinical practice, this operation is done manually by the operator, who uses his/her own criteria to select the appropriate projection, which shows the coronary vessels including their possible lesions. In our research we have attempted to automate the procedure of finding such a projection using selected geometric transformations during image processing. Using the fact that the spatial layout of an object can be determined by projecting it onto the axes of the Cartesian coordinate system, values of horizontal Feret diameters (Russ, 2011), which are a measure of the horizontal extent of the diagnosed coronary artery tree, are calculated for every subsequent animation frame during the image rotation (Fig. 3). Determining these values is not particularly difficult and consists in calculating the difference between the maximum and the minimum coordinate of all points belonging to the projection of the analysed image on the X-axis. The projection for which the horizontal Feret diameter is the greatest is selected for further analyses, as this visualisation shows both the right and the left coronary artery in the most convenient take. In a small number of analysed images, regardless of selecting the projection with the longest horizontal Feret diameter, vessels may obscure one another in space, which causes a problem at subsequent stages of the analysis. The best method to avoid this would be to

Fmax Fig. 3. The projection of the coronary arteries with the longest Feret diameter, obtained from an animation stored in the MPEG4 format.

use advanced techniques for determining mutually corresponding elements for every subsequent animation frame based on the geometric relations in 3D space. 3.2. 3D modelling of coronary arteries using a description based on IE graphs (indexed edge-unambiguous graphs) As the structure of coronary vessels is characterised by three basic types of artery distribution on the heart surface, the proposed methods should include three basic cases: balanced artery distribution, right artery dominant and left artery dominant (Faergeman, 2003). For the purposes of this article, in further considerations we will focus on the balanced distribution of coronary arteries which is the most frequent type seen by diagnosticians (60–70% of all cases). To help represent the examined structure of coronary vascularisation with a graph, it is necessary to define primary components of the analysed image and their spatial relations (Tanaka, 1995), which will serve to extract and suitably represent the morphological characteristics significant for understanding the pathology shown in the image. It is therefore necessary to identify individual coronary arteries and their mutual spatial relations. To ease this process, the projection selected for analysing was skeletonised. This made it possible to obtain the centre lines of

1414

M. Trzupek et al. / Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

Fig. 4. Coronary vascularisation projection and its skeleton produced using the Pavlidis skeletonising algorithm.

examined arteries. These lines are equidistant from their external edges and one unit wide (Fig. 4). This gives us the skeleton of the given artery, which is much smaller than the artery itself, but fully reflects its topological structure. Of several skeletonising algorithms used to analyse medical images, the Pavlidis skeletonising algorithm (Pavlidis, 1982) turned out to be one of the best. It facilitates generating regular, continuous skeletons with a central location and one unit width. It also leaves the fewest apparent side branches in the skeleton and the lines generated during the analysis are only negligibly shortened at their ends. Skeletonising is aimed only at making it possible to find branching points in the vascularisation structures and then to introduce an unambiguous linguistic description of individual coronary arteries and their branches. Lesions will be detected in a representation defined in this way, even though their morphometric parameters have to be determined based on a pattern showing the appropriate vessel, and not just its skeleton. Unambiguous linguistic description refers to the possibility of defining an unambiguous graph representation for the analysed image. Such representation is a prerequisite for further semantic analysis of visible artery patterns, and forms the elements of the special graph language, generated by the proposed in the article graph grammar. The centre lines of analysed arteries produced by skeletonising them is then searched for informative points, i.e. points where artery sections intersect or end. These points will constitute the vertices of a graph modelling the spatial structure of the coronary vessels of the heart. The next step is labelling them by giving each located informative point the appropriate label from the set of vertex labels. In the case of terminal points (leaves of a graph modelling the coronary vascularisation), the set of vertex labels comprises abbreviated names of arteries found in coronary vascularisation. They have been defined as follows: a) for the left coronary artery: LCA—left coronary artery LAD—anterior interventricular branch (left anterior descending) CX—circumflex branch L—lateral branch LM—left marginal branch b) for the right coronary artery: RCA—right coronary artery RM—right marginal branch PI—posterior interventricular branch RP—right posterolateral branch If a given informative point is a branching point, then the vertex will be labelled with the concatenation of names of the vertex labels of arteries which begin at this point. This way, all

Fig. 5. Procedure of identifying spatial relations between individual coronary arteries.

initial and final points of coronary vessels as well as all points where main vessels branch or change into lower level vessels have been determined and labelled as appropriate. After this operation, the coronary vascularisation tree is divided into sections, which constitute the edges of a graph modelling the examined coronary arteries. This makes it possible to formulate a description in the form of edge labels, which determine the mutual spatial relations between the primary components, i.e. between subsequent arteries shown in the analysed image. These labels have been identified according to the following system. Mutual spatial relations that may occur between elements of the vascular structure represented by a graph are described by the set of edge. The elements of this set have been defined by introducing the appropriate spatial relations: vertical—defined by the set of labels {a, b,y, m} and horizontal—defined by the set of labels {1, 2,y, 24} on a hypothetical sphere surrounding the heart muscle. These labels designate individual final intervals, each of which has the angular spread of 151. Then, depending on the location, terminal edge labels are assigned to all branches identified by the beginnings and ends of the appropriate sections of coronary arteries. The presented methodology draws upon the method of determining the location of a point on the surface of our planet in the system of geographic coordinates, where a similar cartographic projection is used to make topographic maps. The use of the presented methodology to determine spatial relations for the analysed projection is shown below (Fig. 5). To determine the appropriate label for the vector W, its beginning should be placed at the zero point of the coordinate system, and then its terminal point location should be established. For this purpose, two angles have been defined: the azimuth angle

M. Trzupek et al. / Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

A to identify the location of the given point as a result of rotating around the vertical axis and the elevation angle E, which identifies the elevation of a given point above the horizon (Fig. 5). This representation of mutual spatial relations between the analysed arteries yields a convenient access to and a unanimous description of all elements of the vascular structure. At subsequent analysis stages, this description will be correctly formalised using embedding transformation-preserved production-ordered k-left nodes unambiguous (ETPL(k)) graph grammar defined in Flasin´ski (1993) and Skomorowski (2000), supporting the search for stenoses in the lumen of arteries forming parts of the coronary vascularisation. ETPL(k) grammar generates IE (indexed edgeunambiguous) graphs (Flasin´ski, 1993; Skomorowski, 2000), which can unambiguously represent 3D structures of heart muscle vascularisation visualised in images acquired during diagnostic examinations with the use of spiral computed tomography. Before we define the representation of the analysed image in the form of IE graphs, we have to introduce the following order relationship in the set of ! edge labels: 1r2r3ryr24 and a r b r g ryr m. This way, we index all vertices according to the r relationship in the set of edge labels which connect the main vertex marked 1 to the adjacent vertices and we index in the ascending order (i¼ 2, 3, y, n). After this operation every vertex of the graph is unambiguously assigned the appropriate index, which will later be used when syntactically analysing the graph representations. IE graphs generated using the presented methodology, modelling the analysed coronary vascularisation, are presented in Fig. 6. When graphs shown in Fig. 6 are represented by their characteristic descriptions, they look as follows: For the right coronary artery

For the left coronary artery

ST1

RCA2

PI3

RM4

ST1

LCA2

CX3

LAD4

1 17m 2

2 4l, 10o 3, 4

1 14x 4

– – –

1 7Z 2

2 13n, 15l 3, 4

1 12e 4

– – –

The graph structure created in this way will form elements of a graph language defining the spatial topology of the heart muscle vascularisation including its possible morphological changes. Formulating a linguistic description for the purpose of determining the semantics of the lesions searched for and identifying (locating) pathological stenoses will support the computer analysis of the structure obtained in order to automatically detect the number of stenoses, their location, type (concentric or eccentric) and extent. For IE graphs defined as above, in order to locate the place where stenoses occur in the case of a balanced artery distribution, the graph grammar may take the following form: a) for the right coronary artery: GR ¼ ðS, D, G, P, ZÞ S ¼ {ST, RCA, RM, PI, C_Right} is a finite, non-empty set of node labels D ¼ {ST, RCA, RM, PI} is a set of terminal node labels G ¼ {17m, 4l, 10o, 14x} is a finite, non-empty set of edge labels Start graph Z and set of productions P are shown in Fig. 7. b) for the left coronary artery: GL ¼ ðS, D, G, P, ZÞ S ¼ {ST, LCA, LAD, CX, C_Left} D ¼ {ST, LCA, LAD, CX} G ¼ {7Z, 13n, 15l, 12e} Start graph Z and set of productions P are shown in Fig. 8.

This way, we have defined a mechanism in the form of ETPL(k) graph grammars, which create a certain linguistic representation of each analysed image in the form of IE graphs. The set of all representations of images generated by this grammar is treated as a certain language. Consequently, we can build a syntax

Fig. 7. Start graph Z and set of productions for grammar GR.

Fig. 6. The representation of the right and the left coronary artery using IE graphs.

1415

Fig. 8. Start graph Z and set of productions for grammar GL.

1416

M. Trzupek et al. / Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

analyser based on the proposed graph grammar which will recognise elements of this language. The syntax analyser is the proper programme that will recognise the changes looked for in the lumen of coronary arteries.

3.3. Detecting lesions and constructing the syntactic analyser The most important stage, which corresponds to the recognition procedure for the pathologies found, is the implementation of a syntactic analyser, which would allow the analysis to be carried out using ‘‘cognitive resonance’’, which is key to understanding the image. This is the hardest part of the entire recognition process, in particular for grammars, which describe the set of rules of a language using graph grammars. One of the procedures of algorithms for parsing IE graphs for ETPL(k) graph grammars is the comparison of descriptions of characteristic subsequent vertices of the analysed IE graph and the derived IE graph, which is to lead to generating the analysed graph. A one-pass generation-type parser carries out the syntactic analysis, at every step examining the characteristic description of the given vertex. If the vertex of the analysed and the derived graphs is terminal, then their characteristic descriptions are examined. However, if the vertex of the derived graph is non-terminal, then a production is searched for, after the application of which the characteristic descriptions are consistent. The numbers of productions used

during the parsing form the basis for classifying the recognised structure. This methodology makes use of theoretical aspects of conducting the syntactic analysis for ETPL(k) grammars, described in Flasin´ski (1993) and Skomorowski (2000). Due to the fact that in obtained during diagnostic examination visualisations of coronary vascularisation, we can distinguish three different types of topologies, characteristic for these vessels, therefore, for each of the three types of topology, we can proposed appropriate type of ETPL(k) graph grammars. Each grammar generates IE graphs language, modelling particular types of coronary vascularisation. This representation was then subjected to a detailed analysis, to find the places of morphological changes indicating occurrence of pathologies. This operation consists of several stages, and uses, among others context-free sequential grammars, used successfully by the authors in previous publications for the detection of lesions in coronary planar images (Ogiela and Tadeusiewicz, 2002). Next steps in the analysis of the example coronary artery are shown in figure (Fig. 9). Artery with the vertices ST1–RCA2 represented by the edge 17m of the IE graph has been subjected to the operation of the straightening transformation (Tadeusiewicz and Ogiela, 2004), which allow to obtain the width diagram of the analysed artery, while preserving all its properties, including potential changes in morphology. In addition, such representation allows to determine the nature of the narrowing (concentric or eccentric). Concentric stenoses occur on a cross-section as a uniform stricture of the

Fig. 9. Next steps in the analysis and recognition of morphological changes occurring on the example of the right coronary artery.

M. Trzupek et al. / Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

whole artery (this symptom is characteristic for a stable disturbance of heart rhythm), whereas eccentric stenoses occur only on one vascular wall (an unstable angina pectoris; Faergeman, 2003). Analysis of morphological changes was conducting based on the obtained width diagrams, and using context-free attributed grammars (Tadeusiewicz and Ogiela, 2004). As a result of carried out operations were obtained profile of the analysed coronary artery with marked areas of existing pathology, together with the determination of the numerical values of their advancement level (Fig. 9). Methodology presented above was implemented sequentially to the individual sections of coronary vascularisation represented by the particular edges of the introduced graph representation. Because of practical applications, the graph grammar should facilitate effective parsing. This has been achieved using ETPL(k) grammars. For such grammars, there are deterministic automata supporting effective parsing with the computational complexity of O(n2) (Flasin´ski, 1993). This allows us to develop very productive analysers that make it possible to verify the graph representations analysed to check if they constitute elements of the language defined by the graph grammar introduced. A significant benefit of using ETPL(k) graph grammars is the possibility of extending them to the form of probabilistic grammars (Skomorowski, 2000).

4. Results and discussion In order to determine the operating efficiency of the proposed methods, a set of test data, namely visualisations obtained during diagnostic examinations using 64-slice spiral computed tomography was used. This set included 75 complete reconstructions of coronary vascularisation obtained during diagnostic examinations of various patients, mainly suffering from coronary heart disease at different progression stages. The test data also included visualisations previously used to construct the grammar and the syntactic analyser. However, to avoid analysing identical images, from the same sequences we selected frames that were several frames later than the projections used to construct the set of grammatical rules, and we used these later frames for the analysis. Due to the different types of topologies of coronary vascularisation, the set of analysed images is shown in Table 1. Structure of the coronary vascularisation was determined by a diagnostician at the stage of acquisition of image data. This distinction was intended to obtain additional information about the importance of providing health risks of the patient depending on the place where pathology occurs (e.g. stenosis occurring in the left coronary artery will be a greater threat to the health of the patient having left artery dominant structure in comparison to the right artery dominant structure). The above set of image data was used to determine the percentage efficiency of correct recognitions of the stenoses present, using the methodology proposed here. The recognition consists in identifying the locations of stenoses, their number, extent and type (concentric or eccentric). For the image data included in the experiment, 84.5% of recognitions were correct. This value is the percentage proportion of the number of images in which the occurring stenoses were correctly located, measured and properly interpreted to the number of all analysed images included in the experimental data

1417

set. No indication of major differences in the effectiveness evaluation, depending on the structure of the coronary vascularisation is noticed. There were, however, problems in the qualifying stage of the image to the appropriate of the defined graph grammars which were created for different types of vascular topology. This was the case with the images on which the structure of coronary vascularisation contained no significant features belonging to one of three distinguished sets. The following figure (Fig. 10) shows the CT image of coronary vascularisation, together with a record describing the pathological changes occurring in the right coronary artery. In order to assess whether the size of the stenosis was correctly measured, we used comparative values from the syngo Vessel View software forming part of the HeartView CI suite (Get the Entire Picture, 2004). This programme is used in everyday clinical practice where examinations are made with the SOMATOM Sensation Cardiac 64 tomograph. In order to confirm or reject the regularity of the stenosis type determination (concentric or eccentric) shown in the examined image, we decided to use a visual assessment, because the aforementioned programs did not have this functionality implemented. As the set of test data was small the results obtained are very promising, and this effectiveness is due, among other things, to the strong generalising properties of the algorithms applied. Further research on improving the presented analysis techniques of lesions occurring in the morphology of coronary vessels might bring about a further improvement in the effectiveness and the future standardisation of these methods, obviously after they have first been tested on a much more numerous image data set (i.e. containing a hundred or more cases). Results obtained in the research conducted show that graph languages for describing shape features can be effectively used to describe 3D reconstructions of coronary vessels and also to formulate semantic meaning descriptions of lesions found in these reconstructions. Such formalisms, due to their significant descriptive power (characteristic especially for graph grammars) can create models of both examined vessels, whose morphology shows no lesions and those with visible lesions bearing witness to early or more advanced stages of the ischaemic heart disease. An additional advantage of the discussed image languages, which use

Table 1 Number of images with particular coronary arteries topologies. Balanced artery distribution

Right artery dominant

Left artery dominant

36

24

15

Fig. 10. The result of the analysis in the search for pathological changes occurring in the CT image of coronary arteries.

1418

M. Trzupek et al. / Engineering Applications of Artificial Intelligence 24 (2011) 1410–1418

advanced formalisms of mathematical linguistics, translation theory and graph theory, is that they automatically identify significant informative points on the image which indicate the presence of lesions. In addition, by introducing the appropriate spatial relations into the coronary vessel reconstruction, it is possible to reproduce their biological role, namely the blood distribution within the whole coronary circulation system, which also facilitates locating and determining the progression stage of lesions. All of this makes up a process of automatically understanding the examined 3D structure, which allows us to provide the physician with far more and far more valuable premises for his/her therapeutic decisions than we could if we were using the traditional image recognition paradigm.

5. Conclusions The methodology of intelligent modelling and semantic description of coronary arteries employing techniques of automatic image understanding, has several significant advantages over classical image recognition algorithms. For many of the considered images it would be very difficult to define a representative vector of image features, which is required in the classical approach based on decision theory methods. This supports the claim that a certain type of images, containing information in the structural form, can be extremely difficult or even impossible for classifying based on selected features in the numerical format. Such images should be described using a certain generalised model, including the component elements of a given structure together with relations occurring between its elements. It is therefore necessary to analyse the image both in the classification sense and in the descriptive one. The classification consists in finding a certain similarity, but this operation has generalising properties and together with the semantic information obtained during the analysis makes it possible to recognise an unlimited number of classes. One drawback of the presented methodology is the frequent lack of a grammar defined for a specific application and the need to build it based on a large set of test data. Consequently, it is necessary to perform a series of deductions to identify the optimum set of derivation rules, which will form the grammar generating the language searched for. This may turn out to be difficult as there is an ambiguity between the languages and the grammars created for them, which often generate a broader language. In addition, building the grammar itself is only half of the job, as there are further difficulties involved in developing the software translator of the analysed linguistic expressions. This entire process depends on the type of grammar proposed, but the ease of developing the grammar is inversely proportional to the parsing job. For some types of grammars (particularly graph ones) it actually turns out that the computational complexity of the parsing problem is, for the overwhelming majority of graph grammar classes, an NP-complete problem. The ongoing research is not solving problems related to automating the process of generating new grammars for cases not included in the present language. In those cases it will be necessary to define a grammar taking this new case into account. The processes of creating new grammars and enriching existing ones with new description rules will be undertaken in further directions of research of the presented methods. Another planned element of further research is to focus on using linguistic artificial intelligence methods to create additional, effective mechanisms, which can be used for indexing and quickly finding specialised image data in medical databases (Kumar et al., 2008; Sonntag and ¨ Moller, 2009). This searching will use semantic keys and will help find cases meeting specified substantive conditions related to ¨ image contents (Moller et al., 2009; Rubin et al., 2008). This can

significantly contribute to solving at least some of the problems of intelligently archiving this type of data and finding semantic image data fulfilling semantic criteria defined using example image patterns in medical multimedia databases, e.g. picture archiving and communication systems (PACS).

Acknowledgements This work has been supported by the National Science Centre, Republic of Poland, under project number N N516 478940. References Akay, Y.M., Akay, M., Welkowitz, W., Kostis, J., 1994. Noninvasive detection of coronary artery disease using wavelet based fuzzy neural networks. IEEE EMB Magazine 13 (5), 761–764. Bob Meijboom, W., Van Mieghem, Carlos A.G., van Pelt, Niels, et al., 2008. Comprehensive assessment of coronary artery stenoses: computed tomography coronary angiography versus conventional coronary angiography and correlation with fractional flow reserve in patients with stable angina. Journal of the American College of Cardiology 52 (8), 636–643. Cios, K.J., Goodenday, L.S., Wedding, D.K., 1989. A Bayesian approach for dealing with uncertainties in detection of coronary artery stenosis using a knowledgebased system. IEEE Engineering in Medicine and Biology Magazine 8 (4), 53–58. Faergeman, O., 2003. Coronary Artery Disease. Elsevier. Flasin´ski, M., 1993. On the parsing of deterministic graph languages for syntactic pattern recognition. Pattern Recognition 26, 1–16. Get the Entire Picture, 2004. SOMATOM Sensation Cardiac 64 Brochure, Siemens medical. Katritsis, D.G., Pantos, I., Efstathopoulos, E.P., et al., 2008. Three-dimensional analysis of the left anterior descending coronary artery: comparison with conventional coronary angiograms. Coronary Artery Disease 19 (4), 265–270. Kurgan, L.A., Cios, K.J., Tadeusiewicz, R., Ogiela, M., Goodenday, L.S., 2001. Knowledge discovery approach to automated cardiac SPECT diagnosis. Artificial Intelligence in Medicine 23 (No 2), 149–169. Kumar, V.S., Narayanan, S., Kurc, T., Kong, J., Gurcan, M.N., Saltz, J.H., 2008. Analysis and semantic querying in large biomedical image datasets. Computer 41 (4), 52–59. Lewandowski, P., Tomczyk, A., Szczepaniak, P.S., 2007. Visualization of 3-D Objects in medicine—selected technical aspects for physicians. Journal of Medical Informatics and Technologies 11, 59–67. Meyer-Baese, A., 2003. Pattern Recognition in Medical Imaging. Elsevier-Academic Press. ¨ Moller M., Regel S., Sintek M., 2009. RadSem: semantic annotation and retrieval for medical images. In: Proceedings of the 6th Annual European Semantic Web Conference (ESWC). Oncel, D., Oncel, G., Tastan, A., Tamci, B., 2007. Detection of significant coronary artery stenosis with 64-section MDCT angiography. European Journal of Radiology 62 (3), 394–405. Ogiela, M.R., Tadeusiewicz, R., 26, 2002. Syntactic reasoning and pattern recognition for analysis of coronary artery images. Artificial Intelligence in Medicine, 145–159. Ogiela, M.R., Tadeusiewicz, R., 2008. Modern Computational Intelligence Methods for the Interpretation of Medical Images. Springer-Verlag, Berlin Heidelberg. Pavlidis, T., 1982. Algorithms for graphics and image processing. Computer Science Press, Rockville. Russ, J.C., 2011. The Image Processing Handbook. CRC Press, Boca Raton. Rubin D., Mongkolwat P., Kleper V., Supekar K., Channin D., 2008. Medical imaging on the semantic web: annotation and image markup. In: Proceedings of the AAAI Spring Symposium Series, Semantic Scientific Knowledge Integration, Stanford, USA. Sirol, M., Sanz, J., Henry, P., et al., 2009. Evaluation of 64-slice MDCT in the real world of cardiology: a comparison with conventional coronary angiography. Archives of Cardiovascular Diseases 102 (5), 433–439. Skomorowski, M., 2000. A Syntactic-Statistical Approach to Recognition of Distorted Patterns. Jagiellonian University Press, Krakow. ¨ Sonntag D., Moller M., 2009. Unifying semantic annotation and querying in biomedical images repositories. In: Proceedings of the International Conference on Knowledge Management and Information Sharing (KMIS), IC3K. Tadeusiewicz, R., Ogiela, M.R., 2004. Medical Image Understanding Technology. Springer Verlag, Berlin-Heidelberg. Tanaka, E., 1995. Theoretical aspects of syntactic pattern recognition. Pattern Recognition 28, 1053–1061. Yusuf, S., Reddy, S., Ounpuu, S., Anand, S., 2001. Global burden of cardiovascular diseases, Part I. General considerations, the epidemiologic transition, risk factors, and impact of urbanization. Circulation 104, 2746–2753. Yin Wang, P. Liatsis, A., 2009. Fully automated framework for segmentation and stenosis quantification of coronary arteries in 3D CTA imaging, DeSe. In: Proceedings of the Second International Conference on Developments in eSystems Engineering, pp. 136–140.