- Email: [email protected]
Automatic interpretation of medical image sequences
Pattern Recognition Letters 8 (1988) 87 102 North-Holland
September 1988
Automatic interpretation of medical image sequences G.
SAGERER
Lehrstuhl fiir lnformatik 5 ( Mustererkennung), Universitiit Erlangen-Niirnberg, Martensstr. 3, D-8520 Erlangen, FRG Received 30 June 1988
Abstract: This paper describes both a general framework for expert systems in the area of image understanding, called ERNEST, and its application to the diagnostic interpretation of scintigraphic image sequences of the human heart, the system DIss. The shell offers a framework to represent declarative knowledge in an associative network, allow the attachment of arbitrary procedural knowledge, and provide a problem-independent control algorithm. The input to the system DISS is an image sequence which is first preprocessed and segmented to obtain the left ventricle and its sectors. The preprocessing and the detection of the contours of the ventricle and the sectors are integrated into the knowledge based analysis of the images. This proceeds from the image sequence via the contour segments, the objects, and their motility to diagnostic interpretations of the image sequence. Experiments showed that the system successfully completes analysis and in particular does not make wrong diagnostic suggestions. Key words: Image understanding, knowledge representation, semantic networks, control, scintigraphic image sequences, fuzzy sets, segmentation, diagnostic image interpretation.
1. I n t r o d u c t i o n
Medical imaging has become an important source of diagnostic information, in particular since new and powerful imaging devices like X-ray and MR tomograms, scintigraphic images, and ultrasound tomograms have become available. The diagnostic evaluation of these images requires experts having the relevant competence in the field. Due to the large amount of medical images and their complexity it is highly desirable to support the human expert by adequate automatic expert systems. This opens the possibility for fast and reproducible preevaluation of images by an expert system and for close scrutiny of difficult cases by a human expert - perhaps also by using additional sources of diagnostic information. A number of expert systems has been described for various task domains in the past, for example diagnosis of bacterial infections [2], oil exploration [4], machine configuration [3], signal interpretation [14], or in medical image understanding [12, 16, 20]. Common to those systems is that they have perfor-
mance comparable to a human expert in a reasonably complex area, that they rely on explicitly represented knowledge about the task domain, and that they can handle imprecise and uncertain situations. Some systems, e.g. [2] require that the situation is described to the system by a human operator, some systems, e.g. [12, 16, 20], extract the required input information from an appropriate sensor device. Apparently it is the latter type of system which is adequate in the area of expert systems for the evaluation of medical images. The purpose of expert systems for medical image analysis is the automatic computation of a symbolic description of an image, or an image sequence. The content of the symbolic description must correspond to the requirements of a concrete application and provide a description on an adequate level of abstraction and with adequate amount of details. For example, in a medical application like X-ray images, diagnostic descriptions are to be extracted out of an image and have to be represented in a format which is useful and understandable to a medical doctor. Complex applications require the ex-
0167-8655/88/$3.50 O 1988, Elsevier Science Publishers B.V. (North-Holland)
87
Volume8. Number2
PATTERN RECOGNITIONLETTERS
plicit representation of extensive task-specific knowledge about objects and events to be recognized, their relations to each other, and about the properties and interpretation of the task domain. Furthermore, there is a need for control algorithms to use the stored knowledge during the analysis process. The efficiency of an image analysis system therefore depends on both the knowledge base and the control algorithm. Besides the two modules knowledge base and control, a database for the intermediate results and also methods for preprocessing and segmentation of images are necessary [i 1]. In many approaches the system is completed by an explanation module which allows the user to check system activities and to evaluate final and intermediate results after an analysis process has been finished. Up to this point the structure of an image analysis system is quite similar to a so called expert system [7]. The differences are mainly the input data and therefore also the data transformation processes. While expert systems usually transform symbolic data into other symbolic data, expert systems applied in medical imaging have to extract symbolic descriptions out of image matrices. It must also be considered that images are noisy, segmentation errors may occur, and so on. That is the reason why image analysis systems have to take into account certainty values of data they manipulate. In the following the system shell ERNEST(ERlangen semantic NEtwork System and Tools) for knowledge based pattern understanding and the system DIss (Diagnostic Interpretation of Scintigraphic image Sequences of the human heart) will be presented. The input data for this system are ECG triggered gated blood pool studies [19]. For the realization of D1ss the ERNEST environment is used. As indicated by the name knowledge representation is based on semantic networks. Generally spoken, such networks are labeled directed graphs where conceptions like objects, events, or diagnostic terms are represented by nodes and relations between such conceptions by links [6]. The network approach of ERNEST also allows the integration of complex procedures like parsers into nodes of the network. This capability is widely used in DISS. Therefore, the presentation of the procedural knowledge in Dlss will be one main point of this paper. 88
September 1988
2. The system shell ERNEST 2.1. Overview
According to our experience semantic networks with the capability to handle also procedural knowledge are an adequate way to build up knowledge bases for image (and speech) analysis systems and furthermore to integrate a complete knowledge based system into such a structure [13]. The idea is that the semantic network with its nodes and links is a kind of skeleton specifying and representing the overall structure of the declarative knowledge base. Attached to the nodes is local task-specific procedural knowledge. The declarative and procedural knowledge is activated by a problem independent control algorithm. Our definition of a semantic network language described in the next sections is mainly influenced by KL-ONE [I] and PSN [10], but it has the following extensions making it particularly suited for pattern analysis and understanding: (1) The introduction of the CONCRETE link in order to represent different levels of abstraction (or different conceptual systems). (2) The statement of precise rules for creating instances, thereby introducing a precise pragmatics of the network. (3) The formulation of problem-independent control algorithms. (4) The definition of sets of modality to distinguish obligatory and optional parts and/or concretes. (5) The possibility to represent context dependences. By the network language and the associated control algorithms a complete system shell for knowledge based pattern analysis is given. This shell ERNESt is realized in the programming language C in an UNIX environment. Besides the network language and the control algorithms, tools for manual and automatic knowledge acquisition and for the explanation of system resources and results are available respectively under development [9]. 2.2. Nodes in the ERNESTnetwork
Two different types of nodes exist in the network. First of all, a CONCEeTgives the intensional descrip-
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
tion of an object, an event, or classes of objects or events. The intensional description subsumes all properties of the conception to be modeled. Examples of such concepts are general ones like '3D-object', 'movement', or 'disease', or more special ones like 'heart', 'fast filling period', or 'enlarged left ventricle'. The second type, called INSTANCE, represents a hypothesis that a real world object or event satisfies the definition of a concept. If, for example, some region of an image is interpreted to be a 'heart' during the analysis process, an instance of the concept 'heart' is created referring this region; this interpretation may be true or not. Therefore, one real object may be referred to by more than one instance, but each instance is associated with exactly one concept. The concepts form the declarative part of the knowledge base, while their extensional sets, the instances, are the intermediate and final results. Viewed as data structures, both are identical, except that instances hold results while concepts subsume descriptions or definitions. Figure 1 shows the structure of a concept node and the most relevant substructures which are used to build up a concept. Additionally, a third type of nodes, called MODIFIED CONCEPT, can be used. It allows the propagating of constraints in the network language. By this type concepts can be modified with respect to a given set of instances [18]. 2.3. Links in the ERNEST network
Relationships between concepts (and also between instances) can be built up by three different types of edges or links. They are denoted by SPECIALIZATION, PART, and CONCRETE. Each of these links is described in a concept, see Figure 1. Whereas a concept can only have one unique generalization which is another concept, it may consist of an arbitrary number of parts and/or concretes. The links define a hierarchy along three independent axes in the network. Both the part and concrete relationships are described by a complex slot, see Figure 1. If, e.g., a concept A has concept B as 'part', B is in the 'domain' slot of the 'part' in A. For all three relationships also the inverse is defined by SPECIALIZATION-OF, PART-OF, and CONCRETE-OF, respectively. In the following, first the semantics of the
September 1988
relationships will be discussed, and after that the different items inside the part and concrete definitions. SPECIALIZATION combines a concept to another one which represents a subclass compared to the concept itself, e.g. 'normally beating heart' is a specialization of 'heart' and 'heart' of 'object'. As in KL-ONE all properties that are parts, concretes, attributes, and relations defined in a concept A are inherited to all concepts which are specializations of A, but they may be explicitely modified. Such modifications are to be described in special facets of the different slots. The restrictions and the rules for such modifications are comparable to those defined in KL-ONE. Contrary to other approaches of associative networks we distinguish two different 'part' relationships. PARTS of a concept must meet the restriction that they are parts in a natural sense. For example, parts of a car are wheels, parts of a special movement can only be other special movements, not objects. Therefore, we can denote this by: a concept and its parts must have the same level (or degree) of abstraction. By the relationship CONCRETE concepts of different abstraction levels can be combined (e.g. the levels 'object', 'motion', and 'diagnosis'). If a concept A has a concrete B, this fact can be expressed by: to talk about A requires B. Therefore, an object can be 'concrete' of a movement, or a contour line of a car. All slots in a concept are uniquely defined by their ROLE which describes the functional role of the slot inside the concept. For example, the fact that one car passes another one has to be modeled in the following way: a concept 'pass' has two concretes. Both have the domain 'car'. One of the partslots has the role 'passes', the other one 'is passed'. The item MODIFIED explains modifications of an inherited slot. The value can be No, if the slot is not defined in a more general concept, or YES, if a slot with identical role has been modified compared to the more general concept, and therefore been overlayed by a new definition. If also the role has changed, the old one has to be referred to by this facet. Finally, the facet CONTEXT DEPENDING which is only defined for parts gives information about the context dependency of the referred part. That is, the conception defined by this part depends on a larger context, i.e., the concept referring the part. 89
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
concept
September 1988
attribute description lexl
name of concepl
4 integer
domoin
*priorities*
5 inlegel
seleclion
inlormolion
text
modifies
speciolizoiion ol
.~
link to concept
speciolizolion context of pod ol
port
--~
*odiocency dependenl,
Y[S or NO
local oltribule
lisle ol attribute descriptions
*reslriclion, deloull value
text
role
lisl ol concepts
domoin
list ot relations lisl ol links to concepts
conlexl depending
,concrete*
lisl ol link descriptions
modifies
Y[S or NO
YES or NO or role
list ol attribute descriptions
dimension
2 integer
list ol relations
*reslriction*
procedure
liSl OI identifications
deloull volue
procedure --~
procedure ---~ hsI ot values
link description
*concrele o1.
inslonce
2x2 integer
dimension
lisl ot links Io concepts
list ol attribute descriptions
*judgemenl,
---.r,. Y[S or NO or role
list o1 concepts
attribute
,identilicotion*
2 values
---e. procedure
list ol modality descriptions
*analysis relolion*
lisl of instances
---~ value
relation text
role modality description obligolory
lest ol relolion ---e,. list of roles c. list of roles
oplionol inherent odjocency coherent
,
~. list ol roles r~ adjacency
---~
YES of NO
value type number value
t:, lype ~, integer t., value array
)rocedure no~
*orgumenl* *inverse procedure,
*odiocen~ dependent, defoullvalue
procedure --~
YES o~ NO list of values
adjacency dimension
integer
roles
lisl ol roles
diagonal
t, integer vector
matrix
t~ bilmotrix
identification *palM*
list ol roles
*path2,
lisl ol roles
text lisl o1 arguments name of procedure
Figure 1. Data structures building up a concept in ERNEST. 90
type
eompulotion ol value
modolity
*onolysis poromeler,
m~
lisl ol links to concepts
---~ lislof link descriptions
structural relolion
texl
role
*degree,
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
The conception '(chair-)leg', for example, depends on the context 'chair'. It is usually not possible on the level of pixel data to recognize a concept 'chairleg' without the knowledge that it is a part of a 'chair'. Contrary, a wheel can be modeled and recognized in an image without knowing if it is part of a car, or a van and so on. Different parts or different concretes can be combined to MODALITYDESCRIPTIONS. Each such set describes which combinations of such links are sufficient for the concept (OBLIGATORY SET), and for each set the OPTIONAL and INHERENT parts which may complete the obligatory set. For example, in a scintigraphic image of the heart the 'heart' is obligatory, but due to varying imaging conditons 'liver' or 'milt' may be visible and are optional components of such an image.
2.4. Concept depending attributes, structures, and judgements ATTRIBUTEScharacterize properties of a concept which are essential with respect to the task domain. In most cases such attributes are physical properties, like the volume or the color of an object, or start and end time or velocity vector of a movement (e.g. of the diastolic phase). But also textual explanations of diagnostic interpretations may be attributes of concepts. However, they must not be used to handle properties which describe relationships to another attribute or special relationships between properties of parts. Attributes are also uniquely defined by their ROLE. Additional facets give the DOMAIN, a SELECTIONfor the domain, and PREFERENCE values. As mentioned above, attributes are inherited from a more general to a more special concept. But it is also possible to MODIFY an attribute in the same way as a modification can be done for parts. Substantial for the definition of an attribute is the procedure for the computation of a value of an instance. Such a procedure can be a simple multiplication (e.g. in order to compute the area of a rectangle), but also a parser combined with a g r a m m a r (e.g. to match a certain motional behaviour of the left ventricle), or a set of rules (e.g. rules are used to infer diagnostic terms in the example system described below). There is no necessity for such procedures 1:o give a unique result. Based on the same input data they may build up different competing
September 1988
results. Each is stored in one instance, and therefore competing instances are created, each having one unique value for all its attributes. Relations which must hold between different attributes and/or attributes of parts, are considered to be structural properties and are tested by RELATION. The semantics of these slots, which are also uniquely defined by their role, is: a relation tests the arguments due to the fact substantial for the concept. For example, there may be spatial relations like 'the left ventrile is adjacent to the right ventricle' or temporal relations like 'motion of the basal segment is synchronous to motion of the left ventricle'. The results of structural tests are essential for an instance to be more or less valid for the concept. Other informations which may help to judge the quality of an instance are the restriction facets in the different slots and the quality of the instances referred to by part relationships. Out of these facts the arguments for computation of a certainty factor can be chosen to check the quality of an instance by the JUDGEMENT.
2.5. The procedural semantics of the ERNEST network language The main idea to use knowledge stored in an associative network as described above is concentrated in the 'basic rule for instantiation'. This rule completes the definition of the network scheme by a third step. Besides the syntax and the (declarative) semantics of the formalism, the use of stored knowledge and therefore a procedural semantics or the pragmatics of the language is defined. This can be done without respect to a task domain. The basic rule for instantiation is: If for a concept A with respect to one obligatory set of the modality of A, instances for those concepts exist, which are referred to by the following slots in A or slots inherited to A without modifications: 'concretes' AND 'parts' with facet 'context dependent' equal to ' N o ' AND one of 'context' if this slot is not equal to ' N o ' . Then build up partial instances ivp(A) of A as follows: create empty instances of A, 91
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
- c o n n e c t these instances to those instances, which are mentioned in the condition of the rule, -calculate the 'attribute'-values for all ivp(A), with respect to the 'arguments', - test the 'restrictions', estimate the quality of ivp(A), by 'JUDGEMENT' with respect to the 'arguments' referring yet known instances and 'restrictions'. Bij this rule partial instances are built up with calculated attribute values and tested restrictions, and an estimated quality. As mentioned in subsection 2.4, for structural tests attributes out of all parts may be used. Therefore, an additional rule, the rule for completing an instance, is required: If an instance ivp(A) of a concept A exists AND instances for all those concepts exist, which are referred to by the part slots in A, with respect to the obligatory set of the modality used for building up ivp(A) Then build up iv(A) out ofivp(A) as follows: - connect iv(A) to the instances referred to by the condition of this rule and not yet connected to ivp(A), -test the structures defined in A with respect to the arguments, -calculate the quality of iv(A), by JUDGEMENT with respect to the arguments. If a goal concept for an analysis process is known, recursive application of these two rules resuits in a search path for the goal concept. By competing instances, generated for a concept, this path is splitted into a search tree. Based on the qualities of instances qualities for concepts restricted to one path of the tree can be estimated. This yields quality functions for the nodes of the search tree, which are used for the A*-Algorithm [15, 16]. If the goal concept is known, these rules together with the A*-AIgortihm form the skeleton for different control strategies. But they are also one complete strategy. In the described medical application the goal concept is chosen by the user. Furthermore, it is also possible to extract automatically potential goal concepts [5] and additional rules are defined to construct modified concepts and instances out of modified concepts [18]. 92
September 1988
2.6. One control algorithm of ERNEST Different tools exist in ERNEST for building up control algorithms for special applications. But also complete problem independent control modules are realized. The underlying algorithms of all these modules are based on the procedural semantics of the network language and on the A*-Algorithm. They differ in their strategy and their complexity. The first one is goal directed. After a top-down expansion of a selected goal concept out of the knowledge base, instances are created bottom-up with respect to the rules of subsection 2.4 [16]. This control module is used for the application which will be discussed in the next section. A second one offers a bidirectional strategy [5], while the third one has also the capability to propagate constraints [l 8]. Due to the topic of this paper only the first one will be presented in the following. For analyzing an image sequence we assume that the goal, i.e. a concept selected by the user to be instantiated, is known. In general this goal concept will be the general concept 'complete interpretation'. If the rule is applied to the goal concept, a number of instances of other concepts are required. Recursively, the rule is applied to these concepts and so on until concepts are detected which refer no other ones in the premise of the rule. Those concepts form the interface between the image segmentation and the knowledge based analysis. The result of this evaluation process is a path of concepts starting with the goal and ending in concepts having an empty premise. Now the instantiation process can be done vice versa starting at the bottom of this path and ending with an instance of the goal concept which shows the result of the analysis. But one has to take into account that the results of calculating attributes are not unique. This means, that in one application of the rule concurrent instances with different values for attributes may be created. This is necessary because the restriction to one result with a local point of view to one instance may yield incorrect results for the overall analysis. Thus the path described is expanded into a tree during the instantiation process. If, e.g. in the application of scintigraphic image sequences, all instances would be created the tree would consist of more than 106 nodes. In order to
PATTERN RECOGNITION LETTERS
Volume 8, Number 2
September 1988
sp
complete Interpretation
cj
dla~nostlc terms for the Interpretatlon ot the local motility of
t interpretation sp of the local motility oi' objects
interpretation of local
moll Ity
complete Interpretation In dla~nostlc terms
objects
anatomical motility phases of Lhe lefL
CO~
con
con
con
ventricle
:on sp ,,[motility
I
sp
Ico Ico
motility
phases [
basic ~ moll fly
motility /1 ~hases
of objects
basic motilities of anatomical
basic motilities
objects
1
description in form and proportions
sp
terms about form and proportions
\hCOn COil
sp
obJect
I
anatomical ) objects
con
enddtastollc description of anatomical objects
imase sequence
S
Input sequence description
Figure 2. Network overview of the knowledge base of the system DIss. 93
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
reduce the effort of analyzing an image sequence we use the well known A*-algorithm [15]. The judgement of a nove vi of the tree depends on the judgement values of those instances created so far on the path from the start node to vi and on the judgement functions of the concepts not created on this path but necessary to create an instance of the goal concept. For this estimation a recursive function is used which depends on the arguments mentioned above [16].
3. The knowledge base of the system DISS 3.1. The declarative knowledge in DISS
Concepts as shown in Figure l make up the nodes of the knowledge base. They are linked to each other, to quantitative attributes and relations, and to procedures as indicated in Figure 1. The concrete and the specialization hierarchy of the whole network is shown in Figure 2. It integrates knowledge about objects, motility, and medical evidence. Since there are about 150 concepts and 400 links for specializations, concretes and parts, Figure 2 can only give a condensed view of the general structure of the network. We now turn to a short description of the 9 levels of the network. Level one and two is built up by the concepts INPUT SEQUENCE DESCRIPTION respectively IMAGE SEQUENCE. INPUT SEQUENCE DESCRIPTION describes questions to the user about the scintigram sequence to be analyzed, that are the spatial and the time resolution and the name of the sequence. With this information an instance of IMAGE SEQUENCE can be created. Such instances cover both the original image sequence and the sequence after spatial median filtering. At the conceptual system which is characterized by OBJECT (level 3), concepts for the objects HEART, LEFT VENTRICLE, RIGHT VENTRICLE, and four anatomical segments of the left ventricle are defined. In the used projection LAO 45, these are the septal (SE), the inferio-apical (IA), the postero-lateral (PL), and the basal (BA) segment. The part relationships between these concepts are obvious. Furthermore, all these concepts, except HEART, are described context dependent from the concept they are part from. Among the main attributes are the con94
September 1988
tours and the volume curve of the objects. The fourth level only describes form and size of the left ventricle and its segments. This may be used on level 9 to infer statements like 'left ventricle enlarged'. BASIC MOTILITYand MOTILITYare the most general concepts of the conceptual systems defining the levels 5 and 6. Both are specialized for a cycle and its phases of contraction, expansion, and stagnation. Those concepts are further specialized for motility descriptions of the left ventricle and its segments. One attribute of those concepts which belong to the conceptual system characterized by BASIC MOTILITY is the change of area between two consecutive images. Furthermore, the direction and the strength of motility is obtained from the center of gravity of objects in consecutive images. Based on area changes the three phases of motility are characterized by fuzzy membership functions according to Figure 3. Using a set of anatomically possible volume curves the best fitting of motion cycles is selected. The set of anatomically possible cycle types is specified and represented by the regular expressions S*C + S*E + S ' E * and S +
where C is Contraction E is Expansion and S is Stagnation. Based on these expressions different cycle types are described. These types are generated out of the expressions above by splitting up the first expression into regular expressions which only use the + operator. The resulting types are shown in Figure 4. The concepts at the 6th conceptual system (MoTILITY)are defined by similar terms. The main difference is that level 5 (BAsIc MOTILITY)always relates
1.0
e-2" 0.5
J
I
I i
I I
Figure 3. Example of a fuzzy-membership-function.
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
1/
+ + + + + +
CSESES
4++++ CESES
September 1988
+ + + + +
÷ + + +
if+÷
C S E S E
C S E S
C S E
+ + + +
CESE
÷ + +
CES
++
CE
+ S
Figure 4. The regular expressionswhich charecterizecycletypes.
two consecutive images i and i + 1, whereas level 6 describes larger entities, e.g., a contraction phase for images 1 to 7. Because of this description in larger entities the concepts standing for contraction, stagnation, and expansion in this conceptual system are defined to be context dependent from the corresponding cycles. At this level no types are predefined. The cycle description is a verification of the description at the former level. Out of this verified cycle description the phases are extracted. Contrary, all concepts which describe basic motilities are independent from a larger context. Several competing alternative descriptions of the motility of anatomical objects may arise during these steps. Level 7 gives a description of the motility of the left ventricle in medical terms. The general statement is that an anatomical interpretation of the motility of the left ventricle during a heart beat consists of systolic and a diastolic phase, and these phases are further subdivided into pre-ejection period, ejection period, endsystolic stagnation, fast filling period, iso volume expansion, and slow filling period. Attributes are start time, end time, and ejection fractions. The result of these interpretations is a unique segmentation of the cycle into the described phases out of one interpretation at the former level. The last two conceptual systems relate motility phases, form, and proportions to statements about medical evidence. On level 8 local diagnostic motility descriptions are derived, e.g. hypokinetic postero-lateral segment or akinetic basal segment. For this purpose, the anatomically motivated segmentation of the left ventricle cycle is used together with the descriptions of level 6 to get the interpretations for the four segments and the left ventricle. On the
top level the local diagnostic interpretations are combined with each other and with the description of form and proportions. This allows to model the complete diagnostic interpretations. Interpretations like ANEURYSMAare inferred but also local interpretations can be modified. For example, if a posterolateral hypokinesis and an inferio-apical akinesis are confirmed with high certainty, it should be checked whether the first one is only a side effect of the second one, because both segments are mechanically coupled. If motility deficiency of the posterolateral segment is strongest in the vicinity to the inferio-apical segment only the inferio-apical akinesis will be maintained.
3.2. The procedural knowledge in DIss A large amount of procedural knowledge is attached to the network which builds up the knowledge base of the system. In fact, the ratio of procedural to declarative knowledge is about 10 to 1 concerning the required storage space. Therefore, a full description is impossible here and probably not necessary because a good deal is fairly straightforward, for example, computation of areas, centers of gravity, lengths of contours, axes of an ellipse approximating the left ventricle, and so on. The most complex and most sophisticated algorithms are the detection of contours, the two stage interpretation of the volume curves of the left ventricle and its four segments, and the inferences about medical evidence. In order to describe the analysis of an image sequence from the procedural point of view we use one example of an analysis process. The intermediate results which will be shown in Figure 5 for 95
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
a
(a) Scintigraphic image sequence after 7*7 median filtering.
(b) Contours of the left ventricle and its four segments. Figure 5. Intermediate results of an analysis process.
oreo
September 1988
the different algorithms are attribute values in those instances which are associated with the optimal path in the search tree. Low level methods are referred in the three lowest levels of the network. With respect to the answers of the user, which are stored in an instance of the concept INPUT SEQUENCE DESCRIPTION, the input image sequence is transferred into an instance of the concept IMAGE SEQUENCE and immediately median filtered when creating this instance, Figure 5a. Segmentation is done by creating instances of the concepts which belong to the conceptual system OBJECT. Detection of the heart and the left ventricle respectively is done by the following steps: Step 1. Find a point (xo, Yo) belonging to the heart respectively to the left ventricle. This point is difficult and crucial, but is solved successfully by automatic procedures. Step 2. Transform the original image into a polar coordinate system having as its origin the above found point. Step 3. Apply a one-dimensional edge detector based on the first derivative to the image computed in step 2. Step 4. Detect ramp edges (for the border of the heart and the upper, lower, and left border of the left ventricle) and valleys (the septum) in the image of step 3. Two edge images are constructed one for the heart and one for the left ventricle.
oreo
100 250
70 200
i
i
i
i
i
i
i
i
J
i
D
J
12 imoge number c) Volume Curve of the left Ventricle (LV)
i
i
~
i
J
r
i
i
i
12 imooe number d) Volume Curve of the Postero lalerol Segment (PL)
Figure 5 (continued). (c)-(d). 96
i
D
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
Step 5. Find an optimal path in both images of step 4 belonging to an optimal contour by using dynamic programming. Limit the search to closed contours with restricted change of tangent direction. Step 6. Transform the two paths of step 5 back to the original coordinate system. The closed contour of the left ventricle is partitioned into the four anatomical segments septal, in-
I S
S
S
S
C
E
E
S
E
S
S
S
e) Basic MotilityDescriptionof the LV Cycle
E
[CS] [CS] [CS] [CS] [CS] E
ferio-apical, postero-lateral, and basal by using a priori knowledge about expected contour directions. In order to obtain a better localization of motion impairments a second partitioning is performed. It takes the center of gravity of the left ventricle in the first image of the sequence and computes 12 sectors of equal angles. An example of the result of segmentation for the left ventricle is shown in Figure 5b. The borders are marked in each of the
[ S
C
C
C
E
E
S
S
S
S
S
S
S
S
s]
f) Basic Motility Description of the PL Cycle
S
E
S
S [CS] I
S
C [C-]
C [ES][ES]
E [S-]
S
h) Verification of f)
g) Verification of e)
[CS]
0.91
September 1988
E
I
0.41
S
E
S
S
C
C
i
I
0.58 0.66 0.75
0,91
0.66
0.08 0.16 0.25 0.33
ef=26%
IES]
E
I
0.50 0.66
ef=25%
i) Motility Paths of the LV
j) Motility Paths of the PL Segment
Diastole
Systole
PEP 0.75
FFP
EP 0.91
0.41
IVE SFP 0.58 0.66 0.75
k) Anatomically Motivated ]erms for the Motility Paths at the LV Figure 5 (continued). (e) (k). 97
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
12 images of a sequence. Tests on image sequences gave the result that on the type of images provided by our medical partners a sufficiently reliable segmentation is achieved by the above procedure (about 90 %correlation with the hand segmentation by a medical doctor). Additional details are avail-
September 1988
able in [11, 16]. Each of these procedures are bound to the concepts for which they detect the boundary. Therefore the resulting contours are attribute values in corresponding instances. Figure 5b shows an image sequence with the contours of the left ventricle and four segments IA, PL, BA, SE. Additional
ef=0.26
el=O,25
u(WEAK,LV,EF;0,26) = 0.41
u(AKIN,PL,EF;O,25)= 1.0
t (LV;IS],CYCLE)=0.75
t (PL;(S],CYCLE) = 0.75
u(WEAK,LV,SA; t (LV;{S],CYCLE)) = 0.33
u(AKIN,PL,SA; t(PL;{S],CYCLE))= 1.0 - t (PL;[C],PEP)= 1.0 ~ t(PL;{E],EP)=O.66
I) Parameters for the Calculation of the CF of the Diagnostic Description LV_WEAK
t (PL;[C],FFP)= 1.0 t (PL;[E],IVE)=0.66 t (PL;iC],SFP) = 1 .O t(PL;{SI,EP)=O.66 t(PL;{C,E],EP) = 0.33 t(PL;IS],FFP)=0.5 t(PL;[C,E],FFP)=0.O t (PL;IS),SFP)= 1.0 ~ t(PL;{C,E),SFP)= 1.0
m) Parameters for the Calculation of the CF of the Diagnostic Description Akinetic Molitility at the P L - S e g m e n t
The left ventricle viewed as one
The p o s i e r o - l o t e r a l Segment
object satisfies the motility
satisfies the motility
interpretation "weak motility"
interpretation "ekinesis" with
with certainty 0.33
certainty 0.5
n) Short Textual Description of the Result of I)
o) Short Textual Description at the Result of m)
In the p o s t e r o - l o t e r o l area of the left ventricle was on okinesis detected with certainty 0.33
p) Short Textual Descripption of the Complete Diagnostic Interpretation of the Image Sequence
Figure 5 (continued). (1) (p). 98
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
attributes in the concepts of this level are the volume curves of the objects. Figure 5c shows the volume curve of the left ventricle and Figure 5d that of the postero-lateral segment. Both are calculated out of the contours in Figure 5b. Interpretation of a heart cycle for each of the objects left ventricle (LV), IA, PL, BA, and SE, is a two phase process executed on the levels 5 and 6 of the knowledge base. The basic idea is to use optimal search techniques [11, 15]. Image to image fuzzy membership values for stagnation, expansion, and contraction are used as negative cost functions. Based on these costs, dynamic programming is used to find the optimal fit between the volume curve of each object and each cycle type of Figure 4. Examples of resulting descriptions are shown in Figures 5e and 5f for a LV and a PL segment respectively. At the next level (MOTILITY) cycles are verified by introducing a refined alphabet, which subsumes also symbols for the overlapping of different motility terms and for gaps inside one phase. These somewhat relaxed interpretations are generated in order to allow some flexibility in later processing steps. Results of such cycle descriptions are shown in Figures 5g and 5h. Overlapping symbols are denoted by the two elementary terms and gaps by a phase symbol and ' - ' . This multilevel parsing procedure was presented in more detail in [17]. Out of these descriptions the motility phases are calculated with normalized start and end points (in the time range 0.0 to 1.0), Figures 5i, 5j. Because most diagnostic terms can only be evaluated with respect to the global motility behaviour of the left ventricle, an intermediate level, called anatomical motility phases of the LV, is introduced. Here, the cycle of the LV is divided into the medical terms SYSTOLE, DIASTOLE, and in a second level into Pre-Ejection-Period, Ejection-Period, Fast-Filling-Period, Iso-Volume-Expansion, and Slow-Filling-Period. This segmentation is based on the cycle description of the LV at the former level. For the calculation of each phase patterns in the former cycle descriptions are searched and scored with special fuzzy membership functions, which reflect a priori knowledge about the start time, the duration, and the ratio between minimal and maximal area of the LV in the corresponding phase. Such ratios, which are also called ejection fractions
September 1988
(el), are themselves parameters for the diagnostic evaluation at the both top levels in the network. The segmentation of a left ventricle cycle in these terms together with the corresponding start and end times is shown in Figure 5k. Presently the system knows about 45 individual diagnostic interpretations which are represented by concepts at the levels 4, 8, and 9. These are of the type 'left ventricle is normally beating', 'the left ventricle is deformed', or 'the basal segment of the left ventricle is nearly motionless'. The certainty of each statement is measured by a certainty factor CF which is computed by fuzzy algebra. For example, given a rule of inference of the form If (not A) or (B and C) then D the certainty factor of D is computed from those of A, B, and C by CF(D) : = max(l - CF(A), min(CF(B),CF(C))) or in short hand notation CF(D) : = (not CF(A)) or (CF(B) and CF(C)). For the very top level, i.e. for COMPLETE INTERPRETATIONS, A, B, C, and D in the equations above stand for names of concepts in the knowledge base. Since there is an obvious one-to-one correspondence between the rule for deriving a conclusion and the CF of this conclusion, it is sufficient to state only the equation for the CF. The rules for complete interpretations at level 9 use concepts on level 8 LOCAL MOTILITY DESCRIPTIONS and/or level 4 DESCRIPTIONS IN FORM AND PROPORTIONS as arguments. One of the complete interpretations which is only based on local motility descriptions is akinetic motility (AKIN). The CF is given by CF(AKtN): = CF(LV WEAK) and (CF(AKIN IA) or CF(AKIN__PL) or CF(AKIN BA) or CF(AKIN SE) ). That means an akinetic motility is stated if at least one of the four segments shows akinetic motility and the global left ventricle has a weak motility. Another more complicated diagnostic interpretation is the kongestive kardiomypathy (KKMY). It is characterized by nearly akinetic inferio-apical and postero-lateral segments (AKIN_IA, AKIN PL) 99
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
combined with very weak motility of the total left ventricle (LV VERY WEAK) as well as an enlarged left ventricle (LV ENLA). A rest of motility is just in the septal or in the basal segment (SE, BA). That means, at least one of these both segments shows normal (NORM) or hypokinetic motility (HvPo). Therefore, this diagnostic interpretation is stated by CF(KKMY) : = CF(LV_VERY_WEAK) and CF(LV ENLA)and CF(IA_AKIN) and CF(PL_AKIN) and (CF(BA NORM) or CF(BA HYPO) or CF(SE_NoRM) or CF(BA HvPo)). Similar rules are available for 6 more diagnoses on level 9. Things are somewhat more complicated on level 8 because on this level descriptions have to be derived from motility phases. At this level the motility of individual segments of the left ventricle is diagnostically interpreted. This is done with the help of two functions t and u defined below. Let P be an element of the anatomical motility phases which are conceptually defined at level 6, that is P c {CYCLE, SYSTOLE,DIASTOLE, PEP, EP, FFP, IVE, SFP}. On level 5, compare Figures 5i, j, for each segment s sE {IA, PL, BA, SE}. The behaviour during one heart cycle is represented by possibly overlapping phases C, E, or S. Let B be defined by B~{C, E, S}. The function t(B, s, P) gives the percentage of motility phases of B in a segment s during the anatomical motility phase P. For example, t({E,S}, PL, FFP) = 0.8 means that the postero-lateral segment expands or stagnates during 80 % of the fast-filling-period. Let
u(d,s,k,i,x) be a fuzzy membership function of similar shape as shown in Figure 3, where 100
September 1988
d ~ (NoRMal, HvPokinetic, AKINetic, DvsKinetic, PHAsEshifted) are diagnostic terms, s is a segment as defined above, k is the criterion used, e.g. ejection fraction of the cycle (EF) or amount of stagnation (SA), and x is the independent variable for which u is to be evaluated. For example, u(AKIN, IA, SA, t({S},IA,CYCLE)) is the certainty factor for akinesis of the inferio-apical segment when the amount of stagnation is considered and the independent variable is the result of the application of the function t. The functions u are derived from general medical knowledge [19]. With the functions u and t the CF's for individual diagnostic terms are determined. As an example, the CF of akinesis in a segment s is considered. A segment is termed akinetic if it shows nearly no motility. That is, the segment does not contract or expand during each phase of the left ventricle but it stagnates. These are the main criteria in the following rule. Further criteria are necessary to distinguish akinetic behaviour from a phaseshifted, a hypokinetic, and a dyskinetic one. CF(s,AKIN) = u(AKIN,s,EF,ef) and
u(AKIN,s,SA,t({S},s, CYCLE)) and (not t({K},s,PEP)) and (not t({E},s,EP)) and (not t({K},s,FFP)) and (not t({K},s,SFP)) and (not t({E},s,IVE)) and (t({S},s,EP) or (not t({K,E},s,EP)) ) and (t({S},s,FFP) or (not t({K,E},s,FFP)) ) and (t({S},s,SFP) or (not t({K,E},s,SFP)) ), where ef is the ratio between minimal and maximal area of the cycle. For the postero-lateral segment the values for all these functions are shown in Figure 5m. Applying the rule, results for the example of Figure 5 in a CF of 0.5 for the term postero-lateral akinesis, Figure 5o. With a similar rule also based on the functions t and u, a weak motility is stated for the left ventricle, Figures 51, 5n. Finally, at the very top level the CF for akinesis, see the rule or CF(AKIN) above, is calculated and the result of the analysis process as indicated in Figure 5p is reached.
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
4. Results and conclusion The paper presented an expert system which successfully describes scintigraphic image sequences of the beating heart. The first operational version of the realized system has about 4MB of code. Tests on about 20 image sequences having validated diagnostic evaluations by medical doctors showed that no wrong descriptions were given by the system; however, in about 10% of the cases the system could not give any description. The system is an example of a successful combination of low level image processing techniques with advanced knowledge based processing and inference. Quite different types of knowledge (e.g. rules for medical diagnosis, regular expressions for typical heart cycles) were integrated in a complex associative network structure, and quite different types of procedural knowledge (e.g. for computation of contour length, of certainty factors, or of optimal heart cycles) were attached to the concepts of the semantic network. The semantic network thus proved to be a flexible and powerful approach to integrating a complex system into a homogeneous structure. Besides the described modules, the system covers tools for explanation and knowledge acquisition. Especially in a medical environment the user needs a lot of information about automatically derived diagnostic interpretations in order to be confident in the way the system solves a given problem. The developed tools for explanation are based on the a priori knowledge of the system, the generated instances, and the search space of the analysis process to be explained [8]. The main tool for knowledge acquisition in the described application is a special editor for the network. Having prepared the relevant knowledge the editor offers a special human interface to store the knowledge in the network by observing all restrictions of the special formalism. One network node is a special questionnaire filled with default values, or values obtained by the user. This questionnaire is the human view of the complex data structures stored in the knowledge base [9]. Our future work in the area of knowledge based systems for image understanding will make further use of associative networks and extend the facilities for knowledge acquisition and explanation of system resources and activities.
September 1988
Acknowledgement The work reported in this paper was supported by the German Research Foundation (DFG). The system DISS was developed in cooperation with H. Feistel and F. Wolf of the Department for Nuclear Medicine of the University Erlangen. Additionally, I want to thank D.P. Pretschner from the University of Hildesheim for many fruitful discussions concerning the system DIss. The development of the system shell ERNESTwas also supported by the Siemens AG, Erlangen. A lot of work for the realization of this shell was done by S. Schr6der and F. Kummert.
References [1] Brachman, R.J. (1979). On the epistemological status of semantic networks. In: N.V. Findler, Ed., Associative Networks. Academic Press, New York. [2] Buchanan, B.G. and Shortliffe, E.H. (1984). Rule-Based Expert Systems. Addison-Wesley, Reading, MA. [3] McDermott, J. (1981) RI: The formative years. AI Magazine 2, 21 29. [4] Duda, R.O., et al. (1978). Development of the PROSPECTOR Consultant System for Mineral Exploration. Rept. Artificial Intelligence Center, SRI Int., Menlo Park, CA. [5] Eichhorn, W. and Niemann, H. (1986). A bidirectional control strategy in a hierarchical knowledge structure. Proc. 8th Int. Cot1[~ on Pattern Recognition ( I C P R ) , Paris, 181 183. [6] Findler, N.V., Ed. (1979). Associative Networks. Academic Press, New York. [7] Hayes-Roth, F., Waterman, D.A. and Lenat, D.B., Eds. (1982). Building Expert Systems. McGraw Hill, New York. [8] Hofmann, I., Giimlich, R. and Niemann, H. (1986). A human interface for control of an image processing system. Proc. 8th ICPR, Paris, 1986, 1256-1258. [9] Kummert, F., Niemann, H., Sagerer, G. and Schroeder, S., Werkzeuge zur modellgesteuerten Bildanalyse und Wissensakquisition. In: M. Paul, Ed., GI-17. Jahrestagung. Springer, Berlin, 556-570. [io] Levesque, H. and Mylopoulos, J. (1979). A procedural semantics for semantic networks. In: N.V. Findler, Ed. Associative Networks. Academic Press, New York, 93- 121. [11] Niemann, H. (1981). Pattern Analysis. Springer, Berlin. B21 Niemann, H., Bunke, H., Hofmann, I., Sagerer, G., Wolf, F. and Feistel, H. (1985). A knowledge based system for analysis of gated blood pool studies. IEEE Trans. Pattern Anal. Machine Intell. 7, 246 259. [13] Niemann, H. (1985). A homogeneous architecture for knowledge based image understanding. Proc. 2. Conf Art# ficial Intefligence Applications. Miami, EL, 1985, 88 93. 101
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
[14] Nii, H.P. et al. (1982). Signal-to-symbol transformation: HASP/SlAP case study. AI Magazine 3, 23 35. [15] Nilsson, N.J. (1982). Principles of Artificial Intelligence. Springer, Berlin. [16] Sagerer, G. (1985). Darstellung und Nutzung von Expertenwissen fiir ein Bildanalysesystem. Springer, Berlin. [17] Sagerer, G. (1986). A multilevel parsing procedure based on dynamic programming for noisy input strings. Proc. EUSIPCO-86. North-Holland Amsterdam, 517-520. [18] Sagerer, G. and Kummert, F. (1988). Knowledge based sys-
102
September 1988
tems for speech understanding. In: Niemann, H., Lang, M. and Sagerer, G., Eds. Recent Advances in Speech Understanding and Dialog Systems. Springer, Berlin. [19] Schicha, H. and Emrich, D. (1984). Nuklearmedizin in der kardiologischen Praxis: Methoden Ergebnisse Indikation. GIT-Verlag, Ernst Giebeler, Darmstadt. [20] Stansfield, S.A. (1986). ANGY: A rule-based expert system for automatic segmentation of coronary vessels from digital subtracted angiograms. 1EEE Trans. Pattern Anal. Machine Intell., 188 199.
September 1988
Automatic interpretation of medical image sequences G.
SAGERER
Lehrstuhl fiir lnformatik 5 ( Mustererkennung), Universitiit Erlangen-Niirnberg, Martensstr. 3, D-8520 Erlangen, FRG Received 30 June 1988
Abstract: This paper describes both a general framework for expert systems in the area of image understanding, called ERNEST, and its application to the diagnostic interpretation of scintigraphic image sequences of the human heart, the system DIss. The shell offers a framework to represent declarative knowledge in an associative network, allow the attachment of arbitrary procedural knowledge, and provide a problem-independent control algorithm. The input to the system DISS is an image sequence which is first preprocessed and segmented to obtain the left ventricle and its sectors. The preprocessing and the detection of the contours of the ventricle and the sectors are integrated into the knowledge based analysis of the images. This proceeds from the image sequence via the contour segments, the objects, and their motility to diagnostic interpretations of the image sequence. Experiments showed that the system successfully completes analysis and in particular does not make wrong diagnostic suggestions. Key words: Image understanding, knowledge representation, semantic networks, control, scintigraphic image sequences, fuzzy sets, segmentation, diagnostic image interpretation.
1. I n t r o d u c t i o n
Medical imaging has become an important source of diagnostic information, in particular since new and powerful imaging devices like X-ray and MR tomograms, scintigraphic images, and ultrasound tomograms have become available. The diagnostic evaluation of these images requires experts having the relevant competence in the field. Due to the large amount of medical images and their complexity it is highly desirable to support the human expert by adequate automatic expert systems. This opens the possibility for fast and reproducible preevaluation of images by an expert system and for close scrutiny of difficult cases by a human expert - perhaps also by using additional sources of diagnostic information. A number of expert systems has been described for various task domains in the past, for example diagnosis of bacterial infections [2], oil exploration [4], machine configuration [3], signal interpretation [14], or in medical image understanding [12, 16, 20]. Common to those systems is that they have perfor-
mance comparable to a human expert in a reasonably complex area, that they rely on explicitly represented knowledge about the task domain, and that they can handle imprecise and uncertain situations. Some systems, e.g. [2] require that the situation is described to the system by a human operator, some systems, e.g. [12, 16, 20], extract the required input information from an appropriate sensor device. Apparently it is the latter type of system which is adequate in the area of expert systems for the evaluation of medical images. The purpose of expert systems for medical image analysis is the automatic computation of a symbolic description of an image, or an image sequence. The content of the symbolic description must correspond to the requirements of a concrete application and provide a description on an adequate level of abstraction and with adequate amount of details. For example, in a medical application like X-ray images, diagnostic descriptions are to be extracted out of an image and have to be represented in a format which is useful and understandable to a medical doctor. Complex applications require the ex-
0167-8655/88/$3.50 O 1988, Elsevier Science Publishers B.V. (North-Holland)
87
Volume8. Number2
PATTERN RECOGNITIONLETTERS
plicit representation of extensive task-specific knowledge about objects and events to be recognized, their relations to each other, and about the properties and interpretation of the task domain. Furthermore, there is a need for control algorithms to use the stored knowledge during the analysis process. The efficiency of an image analysis system therefore depends on both the knowledge base and the control algorithm. Besides the two modules knowledge base and control, a database for the intermediate results and also methods for preprocessing and segmentation of images are necessary [i 1]. In many approaches the system is completed by an explanation module which allows the user to check system activities and to evaluate final and intermediate results after an analysis process has been finished. Up to this point the structure of an image analysis system is quite similar to a so called expert system [7]. The differences are mainly the input data and therefore also the data transformation processes. While expert systems usually transform symbolic data into other symbolic data, expert systems applied in medical imaging have to extract symbolic descriptions out of image matrices. It must also be considered that images are noisy, segmentation errors may occur, and so on. That is the reason why image analysis systems have to take into account certainty values of data they manipulate. In the following the system shell ERNEST(ERlangen semantic NEtwork System and Tools) for knowledge based pattern understanding and the system DIss (Diagnostic Interpretation of Scintigraphic image Sequences of the human heart) will be presented. The input data for this system are ECG triggered gated blood pool studies [19]. For the realization of D1ss the ERNEST environment is used. As indicated by the name knowledge representation is based on semantic networks. Generally spoken, such networks are labeled directed graphs where conceptions like objects, events, or diagnostic terms are represented by nodes and relations between such conceptions by links [6]. The network approach of ERNEST also allows the integration of complex procedures like parsers into nodes of the network. This capability is widely used in DISS. Therefore, the presentation of the procedural knowledge in Dlss will be one main point of this paper. 88
September 1988
2. The system shell ERNEST 2.1. Overview
According to our experience semantic networks with the capability to handle also procedural knowledge are an adequate way to build up knowledge bases for image (and speech) analysis systems and furthermore to integrate a complete knowledge based system into such a structure [13]. The idea is that the semantic network with its nodes and links is a kind of skeleton specifying and representing the overall structure of the declarative knowledge base. Attached to the nodes is local task-specific procedural knowledge. The declarative and procedural knowledge is activated by a problem independent control algorithm. Our definition of a semantic network language described in the next sections is mainly influenced by KL-ONE [I] and PSN [10], but it has the following extensions making it particularly suited for pattern analysis and understanding: (1) The introduction of the CONCRETE link in order to represent different levels of abstraction (or different conceptual systems). (2) The statement of precise rules for creating instances, thereby introducing a precise pragmatics of the network. (3) The formulation of problem-independent control algorithms. (4) The definition of sets of modality to distinguish obligatory and optional parts and/or concretes. (5) The possibility to represent context dependences. By the network language and the associated control algorithms a complete system shell for knowledge based pattern analysis is given. This shell ERNESt is realized in the programming language C in an UNIX environment. Besides the network language and the control algorithms, tools for manual and automatic knowledge acquisition and for the explanation of system resources and results are available respectively under development [9]. 2.2. Nodes in the ERNESTnetwork
Two different types of nodes exist in the network. First of all, a CONCEeTgives the intensional descrip-
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
tion of an object, an event, or classes of objects or events. The intensional description subsumes all properties of the conception to be modeled. Examples of such concepts are general ones like '3D-object', 'movement', or 'disease', or more special ones like 'heart', 'fast filling period', or 'enlarged left ventricle'. The second type, called INSTANCE, represents a hypothesis that a real world object or event satisfies the definition of a concept. If, for example, some region of an image is interpreted to be a 'heart' during the analysis process, an instance of the concept 'heart' is created referring this region; this interpretation may be true or not. Therefore, one real object may be referred to by more than one instance, but each instance is associated with exactly one concept. The concepts form the declarative part of the knowledge base, while their extensional sets, the instances, are the intermediate and final results. Viewed as data structures, both are identical, except that instances hold results while concepts subsume descriptions or definitions. Figure 1 shows the structure of a concept node and the most relevant substructures which are used to build up a concept. Additionally, a third type of nodes, called MODIFIED CONCEPT, can be used. It allows the propagating of constraints in the network language. By this type concepts can be modified with respect to a given set of instances [18]. 2.3. Links in the ERNEST network
Relationships between concepts (and also between instances) can be built up by three different types of edges or links. They are denoted by SPECIALIZATION, PART, and CONCRETE. Each of these links is described in a concept, see Figure 1. Whereas a concept can only have one unique generalization which is another concept, it may consist of an arbitrary number of parts and/or concretes. The links define a hierarchy along three independent axes in the network. Both the part and concrete relationships are described by a complex slot, see Figure 1. If, e.g., a concept A has concept B as 'part', B is in the 'domain' slot of the 'part' in A. For all three relationships also the inverse is defined by SPECIALIZATION-OF, PART-OF, and CONCRETE-OF, respectively. In the following, first the semantics of the
September 1988
relationships will be discussed, and after that the different items inside the part and concrete definitions. SPECIALIZATION combines a concept to another one which represents a subclass compared to the concept itself, e.g. 'normally beating heart' is a specialization of 'heart' and 'heart' of 'object'. As in KL-ONE all properties that are parts, concretes, attributes, and relations defined in a concept A are inherited to all concepts which are specializations of A, but they may be explicitely modified. Such modifications are to be described in special facets of the different slots. The restrictions and the rules for such modifications are comparable to those defined in KL-ONE. Contrary to other approaches of associative networks we distinguish two different 'part' relationships. PARTS of a concept must meet the restriction that they are parts in a natural sense. For example, parts of a car are wheels, parts of a special movement can only be other special movements, not objects. Therefore, we can denote this by: a concept and its parts must have the same level (or degree) of abstraction. By the relationship CONCRETE concepts of different abstraction levels can be combined (e.g. the levels 'object', 'motion', and 'diagnosis'). If a concept A has a concrete B, this fact can be expressed by: to talk about A requires B. Therefore, an object can be 'concrete' of a movement, or a contour line of a car. All slots in a concept are uniquely defined by their ROLE which describes the functional role of the slot inside the concept. For example, the fact that one car passes another one has to be modeled in the following way: a concept 'pass' has two concretes. Both have the domain 'car'. One of the partslots has the role 'passes', the other one 'is passed'. The item MODIFIED explains modifications of an inherited slot. The value can be No, if the slot is not defined in a more general concept, or YES, if a slot with identical role has been modified compared to the more general concept, and therefore been overlayed by a new definition. If also the role has changed, the old one has to be referred to by this facet. Finally, the facet CONTEXT DEPENDING which is only defined for parts gives information about the context dependency of the referred part. That is, the conception defined by this part depends on a larger context, i.e., the concept referring the part. 89
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
concept
September 1988
attribute description lexl
name of concepl
4 integer
domoin
*priorities*
5 inlegel
seleclion
inlormolion
text
modifies
speciolizoiion ol
.~
link to concept
speciolizolion context of pod ol
port
--~
*odiocency dependenl,
Y[S or NO
local oltribule
lisle ol attribute descriptions
*reslriclion, deloull value
text
role
lisl ol concepts
domoin
list ot relations lisl ol links to concepts
conlexl depending
,concrete*
lisl ol link descriptions
modifies
Y[S or NO
YES or NO or role
list ol attribute descriptions
dimension
2 integer
list ol relations
*reslriction*
procedure
liSl OI identifications
deloull volue
procedure --~
procedure ---~ hsI ot values
link description
*concrele o1.
inslonce
2x2 integer
dimension
lisl ot links Io concepts
list ol attribute descriptions
*judgemenl,
---.r,. Y[S or NO or role
list o1 concepts
attribute
,identilicotion*
2 values
---e. procedure
list ol modality descriptions
*analysis relolion*
lisl of instances
---~ value
relation text
role modality description obligolory
lest ol relolion ---e,. list of roles c. list of roles
oplionol inherent odjocency coherent
,
~. list ol roles r~ adjacency
---~
YES of NO
value type number value
t:, lype ~, integer t., value array
)rocedure no~
*orgumenl* *inverse procedure,
*odiocen~ dependent, defoullvalue
procedure --~
YES o~ NO list of values
adjacency dimension
integer
roles
lisl ol roles
diagonal
t, integer vector
matrix
t~ bilmotrix
identification *palM*
list ol roles
*path2,
lisl ol roles
text lisl o1 arguments name of procedure
Figure 1. Data structures building up a concept in ERNEST. 90
type
eompulotion ol value
modolity
*onolysis poromeler,
m~
lisl ol links to concepts
---~ lislof link descriptions
structural relolion
texl
role
*degree,
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
The conception '(chair-)leg', for example, depends on the context 'chair'. It is usually not possible on the level of pixel data to recognize a concept 'chairleg' without the knowledge that it is a part of a 'chair'. Contrary, a wheel can be modeled and recognized in an image without knowing if it is part of a car, or a van and so on. Different parts or different concretes can be combined to MODALITYDESCRIPTIONS. Each such set describes which combinations of such links are sufficient for the concept (OBLIGATORY SET), and for each set the OPTIONAL and INHERENT parts which may complete the obligatory set. For example, in a scintigraphic image of the heart the 'heart' is obligatory, but due to varying imaging conditons 'liver' or 'milt' may be visible and are optional components of such an image.
2.4. Concept depending attributes, structures, and judgements ATTRIBUTEScharacterize properties of a concept which are essential with respect to the task domain. In most cases such attributes are physical properties, like the volume or the color of an object, or start and end time or velocity vector of a movement (e.g. of the diastolic phase). But also textual explanations of diagnostic interpretations may be attributes of concepts. However, they must not be used to handle properties which describe relationships to another attribute or special relationships between properties of parts. Attributes are also uniquely defined by their ROLE. Additional facets give the DOMAIN, a SELECTIONfor the domain, and PREFERENCE values. As mentioned above, attributes are inherited from a more general to a more special concept. But it is also possible to MODIFY an attribute in the same way as a modification can be done for parts. Substantial for the definition of an attribute is the procedure for the computation of a value of an instance. Such a procedure can be a simple multiplication (e.g. in order to compute the area of a rectangle), but also a parser combined with a g r a m m a r (e.g. to match a certain motional behaviour of the left ventricle), or a set of rules (e.g. rules are used to infer diagnostic terms in the example system described below). There is no necessity for such procedures 1:o give a unique result. Based on the same input data they may build up different competing
September 1988
results. Each is stored in one instance, and therefore competing instances are created, each having one unique value for all its attributes. Relations which must hold between different attributes and/or attributes of parts, are considered to be structural properties and are tested by RELATION. The semantics of these slots, which are also uniquely defined by their role, is: a relation tests the arguments due to the fact substantial for the concept. For example, there may be spatial relations like 'the left ventrile is adjacent to the right ventricle' or temporal relations like 'motion of the basal segment is synchronous to motion of the left ventricle'. The results of structural tests are essential for an instance to be more or less valid for the concept. Other informations which may help to judge the quality of an instance are the restriction facets in the different slots and the quality of the instances referred to by part relationships. Out of these facts the arguments for computation of a certainty factor can be chosen to check the quality of an instance by the JUDGEMENT.
2.5. The procedural semantics of the ERNEST network language The main idea to use knowledge stored in an associative network as described above is concentrated in the 'basic rule for instantiation'. This rule completes the definition of the network scheme by a third step. Besides the syntax and the (declarative) semantics of the formalism, the use of stored knowledge and therefore a procedural semantics or the pragmatics of the language is defined. This can be done without respect to a task domain. The basic rule for instantiation is: If for a concept A with respect to one obligatory set of the modality of A, instances for those concepts exist, which are referred to by the following slots in A or slots inherited to A without modifications: 'concretes' AND 'parts' with facet 'context dependent' equal to ' N o ' AND one of 'context' if this slot is not equal to ' N o ' . Then build up partial instances ivp(A) of A as follows: create empty instances of A, 91
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
- c o n n e c t these instances to those instances, which are mentioned in the condition of the rule, -calculate the 'attribute'-values for all ivp(A), with respect to the 'arguments', - test the 'restrictions', estimate the quality of ivp(A), by 'JUDGEMENT' with respect to the 'arguments' referring yet known instances and 'restrictions'. Bij this rule partial instances are built up with calculated attribute values and tested restrictions, and an estimated quality. As mentioned in subsection 2.4, for structural tests attributes out of all parts may be used. Therefore, an additional rule, the rule for completing an instance, is required: If an instance ivp(A) of a concept A exists AND instances for all those concepts exist, which are referred to by the part slots in A, with respect to the obligatory set of the modality used for building up ivp(A) Then build up iv(A) out ofivp(A) as follows: - connect iv(A) to the instances referred to by the condition of this rule and not yet connected to ivp(A), -test the structures defined in A with respect to the arguments, -calculate the quality of iv(A), by JUDGEMENT with respect to the arguments. If a goal concept for an analysis process is known, recursive application of these two rules resuits in a search path for the goal concept. By competing instances, generated for a concept, this path is splitted into a search tree. Based on the qualities of instances qualities for concepts restricted to one path of the tree can be estimated. This yields quality functions for the nodes of the search tree, which are used for the A*-Algorithm [15, 16]. If the goal concept is known, these rules together with the A*-AIgortihm form the skeleton for different control strategies. But they are also one complete strategy. In the described medical application the goal concept is chosen by the user. Furthermore, it is also possible to extract automatically potential goal concepts [5] and additional rules are defined to construct modified concepts and instances out of modified concepts [18]. 92
September 1988
2.6. One control algorithm of ERNEST Different tools exist in ERNEST for building up control algorithms for special applications. But also complete problem independent control modules are realized. The underlying algorithms of all these modules are based on the procedural semantics of the network language and on the A*-Algorithm. They differ in their strategy and their complexity. The first one is goal directed. After a top-down expansion of a selected goal concept out of the knowledge base, instances are created bottom-up with respect to the rules of subsection 2.4 [16]. This control module is used for the application which will be discussed in the next section. A second one offers a bidirectional strategy [5], while the third one has also the capability to propagate constraints [l 8]. Due to the topic of this paper only the first one will be presented in the following. For analyzing an image sequence we assume that the goal, i.e. a concept selected by the user to be instantiated, is known. In general this goal concept will be the general concept 'complete interpretation'. If the rule is applied to the goal concept, a number of instances of other concepts are required. Recursively, the rule is applied to these concepts and so on until concepts are detected which refer no other ones in the premise of the rule. Those concepts form the interface between the image segmentation and the knowledge based analysis. The result of this evaluation process is a path of concepts starting with the goal and ending in concepts having an empty premise. Now the instantiation process can be done vice versa starting at the bottom of this path and ending with an instance of the goal concept which shows the result of the analysis. But one has to take into account that the results of calculating attributes are not unique. This means, that in one application of the rule concurrent instances with different values for attributes may be created. This is necessary because the restriction to one result with a local point of view to one instance may yield incorrect results for the overall analysis. Thus the path described is expanded into a tree during the instantiation process. If, e.g. in the application of scintigraphic image sequences, all instances would be created the tree would consist of more than 106 nodes. In order to
PATTERN RECOGNITION LETTERS
Volume 8, Number 2
September 1988
sp
complete Interpretation
cj
dla~nostlc terms for the Interpretatlon ot the local motility of
t interpretation sp of the local motility oi' objects
interpretation of local
moll Ity
complete Interpretation In dla~nostlc terms
objects
anatomical motility phases of Lhe lefL
CO~
con
con
con
ventricle
:on sp ,,[motility
I
sp
Ico Ico
motility
phases [
basic ~ moll fly
motility /1 ~hases
of objects
basic motilities of anatomical
basic motilities
objects
1
description in form and proportions
sp
terms about form and proportions
\hCOn COil
sp
obJect
I
anatomical ) objects
con
enddtastollc description of anatomical objects
imase sequence
S
Input sequence description
Figure 2. Network overview of the knowledge base of the system DIss. 93
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
reduce the effort of analyzing an image sequence we use the well known A*-algorithm [15]. The judgement of a nove vi of the tree depends on the judgement values of those instances created so far on the path from the start node to vi and on the judgement functions of the concepts not created on this path but necessary to create an instance of the goal concept. For this estimation a recursive function is used which depends on the arguments mentioned above [16].
3. The knowledge base of the system DISS 3.1. The declarative knowledge in DISS
Concepts as shown in Figure l make up the nodes of the knowledge base. They are linked to each other, to quantitative attributes and relations, and to procedures as indicated in Figure 1. The concrete and the specialization hierarchy of the whole network is shown in Figure 2. It integrates knowledge about objects, motility, and medical evidence. Since there are about 150 concepts and 400 links for specializations, concretes and parts, Figure 2 can only give a condensed view of the general structure of the network. We now turn to a short description of the 9 levels of the network. Level one and two is built up by the concepts INPUT SEQUENCE DESCRIPTION respectively IMAGE SEQUENCE. INPUT SEQUENCE DESCRIPTION describes questions to the user about the scintigram sequence to be analyzed, that are the spatial and the time resolution and the name of the sequence. With this information an instance of IMAGE SEQUENCE can be created. Such instances cover both the original image sequence and the sequence after spatial median filtering. At the conceptual system which is characterized by OBJECT (level 3), concepts for the objects HEART, LEFT VENTRICLE, RIGHT VENTRICLE, and four anatomical segments of the left ventricle are defined. In the used projection LAO 45, these are the septal (SE), the inferio-apical (IA), the postero-lateral (PL), and the basal (BA) segment. The part relationships between these concepts are obvious. Furthermore, all these concepts, except HEART, are described context dependent from the concept they are part from. Among the main attributes are the con94
September 1988
tours and the volume curve of the objects. The fourth level only describes form and size of the left ventricle and its segments. This may be used on level 9 to infer statements like 'left ventricle enlarged'. BASIC MOTILITYand MOTILITYare the most general concepts of the conceptual systems defining the levels 5 and 6. Both are specialized for a cycle and its phases of contraction, expansion, and stagnation. Those concepts are further specialized for motility descriptions of the left ventricle and its segments. One attribute of those concepts which belong to the conceptual system characterized by BASIC MOTILITY is the change of area between two consecutive images. Furthermore, the direction and the strength of motility is obtained from the center of gravity of objects in consecutive images. Based on area changes the three phases of motility are characterized by fuzzy membership functions according to Figure 3. Using a set of anatomically possible volume curves the best fitting of motion cycles is selected. The set of anatomically possible cycle types is specified and represented by the regular expressions S*C + S*E + S ' E * and S +
where C is Contraction E is Expansion and S is Stagnation. Based on these expressions different cycle types are described. These types are generated out of the expressions above by splitting up the first expression into regular expressions which only use the + operator. The resulting types are shown in Figure 4. The concepts at the 6th conceptual system (MoTILITY)are defined by similar terms. The main difference is that level 5 (BAsIc MOTILITY)always relates
1.0
e-2" 0.5
J
I
I i
I I
Figure 3. Example of a fuzzy-membership-function.
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
1/
+ + + + + +
CSESES
4++++ CESES
September 1988
+ + + + +
÷ + + +
if+÷
C S E S E
C S E S
C S E
+ + + +
CESE
÷ + +
CES
++
CE
+ S
Figure 4. The regular expressionswhich charecterizecycletypes.
two consecutive images i and i + 1, whereas level 6 describes larger entities, e.g., a contraction phase for images 1 to 7. Because of this description in larger entities the concepts standing for contraction, stagnation, and expansion in this conceptual system are defined to be context dependent from the corresponding cycles. At this level no types are predefined. The cycle description is a verification of the description at the former level. Out of this verified cycle description the phases are extracted. Contrary, all concepts which describe basic motilities are independent from a larger context. Several competing alternative descriptions of the motility of anatomical objects may arise during these steps. Level 7 gives a description of the motility of the left ventricle in medical terms. The general statement is that an anatomical interpretation of the motility of the left ventricle during a heart beat consists of systolic and a diastolic phase, and these phases are further subdivided into pre-ejection period, ejection period, endsystolic stagnation, fast filling period, iso volume expansion, and slow filling period. Attributes are start time, end time, and ejection fractions. The result of these interpretations is a unique segmentation of the cycle into the described phases out of one interpretation at the former level. The last two conceptual systems relate motility phases, form, and proportions to statements about medical evidence. On level 8 local diagnostic motility descriptions are derived, e.g. hypokinetic postero-lateral segment or akinetic basal segment. For this purpose, the anatomically motivated segmentation of the left ventricle cycle is used together with the descriptions of level 6 to get the interpretations for the four segments and the left ventricle. On the
top level the local diagnostic interpretations are combined with each other and with the description of form and proportions. This allows to model the complete diagnostic interpretations. Interpretations like ANEURYSMAare inferred but also local interpretations can be modified. For example, if a posterolateral hypokinesis and an inferio-apical akinesis are confirmed with high certainty, it should be checked whether the first one is only a side effect of the second one, because both segments are mechanically coupled. If motility deficiency of the posterolateral segment is strongest in the vicinity to the inferio-apical segment only the inferio-apical akinesis will be maintained.
3.2. The procedural knowledge in DIss A large amount of procedural knowledge is attached to the network which builds up the knowledge base of the system. In fact, the ratio of procedural to declarative knowledge is about 10 to 1 concerning the required storage space. Therefore, a full description is impossible here and probably not necessary because a good deal is fairly straightforward, for example, computation of areas, centers of gravity, lengths of contours, axes of an ellipse approximating the left ventricle, and so on. The most complex and most sophisticated algorithms are the detection of contours, the two stage interpretation of the volume curves of the left ventricle and its four segments, and the inferences about medical evidence. In order to describe the analysis of an image sequence from the procedural point of view we use one example of an analysis process. The intermediate results which will be shown in Figure 5 for 95
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
a
(a) Scintigraphic image sequence after 7*7 median filtering.
(b) Contours of the left ventricle and its four segments. Figure 5. Intermediate results of an analysis process.
oreo
September 1988
the different algorithms are attribute values in those instances which are associated with the optimal path in the search tree. Low level methods are referred in the three lowest levels of the network. With respect to the answers of the user, which are stored in an instance of the concept INPUT SEQUENCE DESCRIPTION, the input image sequence is transferred into an instance of the concept IMAGE SEQUENCE and immediately median filtered when creating this instance, Figure 5a. Segmentation is done by creating instances of the concepts which belong to the conceptual system OBJECT. Detection of the heart and the left ventricle respectively is done by the following steps: Step 1. Find a point (xo, Yo) belonging to the heart respectively to the left ventricle. This point is difficult and crucial, but is solved successfully by automatic procedures. Step 2. Transform the original image into a polar coordinate system having as its origin the above found point. Step 3. Apply a one-dimensional edge detector based on the first derivative to the image computed in step 2. Step 4. Detect ramp edges (for the border of the heart and the upper, lower, and left border of the left ventricle) and valleys (the septum) in the image of step 3. Two edge images are constructed one for the heart and one for the left ventricle.
oreo
100 250
70 200
i
i
i
i
i
i
i
i
J
i
D
J
12 imoge number c) Volume Curve of the left Ventricle (LV)
i
i
~
i
J
r
i
i
i
12 imooe number d) Volume Curve of the Postero lalerol Segment (PL)
Figure 5 (continued). (c)-(d). 96
i
D
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
Step 5. Find an optimal path in both images of step 4 belonging to an optimal contour by using dynamic programming. Limit the search to closed contours with restricted change of tangent direction. Step 6. Transform the two paths of step 5 back to the original coordinate system. The closed contour of the left ventricle is partitioned into the four anatomical segments septal, in-
I S
S
S
S
C
E
E
S
E
S
S
S
e) Basic MotilityDescriptionof the LV Cycle
E
[CS] [CS] [CS] [CS] [CS] E
ferio-apical, postero-lateral, and basal by using a priori knowledge about expected contour directions. In order to obtain a better localization of motion impairments a second partitioning is performed. It takes the center of gravity of the left ventricle in the first image of the sequence and computes 12 sectors of equal angles. An example of the result of segmentation for the left ventricle is shown in Figure 5b. The borders are marked in each of the
[ S
C
C
C
E
E
S
S
S
S
S
S
S
S
s]
f) Basic Motility Description of the PL Cycle
S
E
S
S [CS] I
S
C [C-]
C [ES][ES]
E [S-]
S
h) Verification of f)
g) Verification of e)
[CS]
0.91
September 1988
E
I
0.41
S
E
S
S
C
C
i
I
0.58 0.66 0.75
0,91
0.66
0.08 0.16 0.25 0.33
ef=26%
IES]
E
I
0.50 0.66
ef=25%
i) Motility Paths of the LV
j) Motility Paths of the PL Segment
Diastole
Systole
PEP 0.75
FFP
EP 0.91
0.41
IVE SFP 0.58 0.66 0.75
k) Anatomically Motivated ]erms for the Motility Paths at the LV Figure 5 (continued). (e) (k). 97
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
12 images of a sequence. Tests on image sequences gave the result that on the type of images provided by our medical partners a sufficiently reliable segmentation is achieved by the above procedure (about 90 %correlation with the hand segmentation by a medical doctor). Additional details are avail-
September 1988
able in [11, 16]. Each of these procedures are bound to the concepts for which they detect the boundary. Therefore the resulting contours are attribute values in corresponding instances. Figure 5b shows an image sequence with the contours of the left ventricle and four segments IA, PL, BA, SE. Additional
ef=0.26
el=O,25
u(WEAK,LV,EF;0,26) = 0.41
u(AKIN,PL,EF;O,25)= 1.0
t (LV;IS],CYCLE)=0.75
t (PL;(S],CYCLE) = 0.75
u(WEAK,LV,SA; t (LV;{S],CYCLE)) = 0.33
u(AKIN,PL,SA; t(PL;{S],CYCLE))= 1.0 - t (PL;[C],PEP)= 1.0 ~ t(PL;{E],EP)=O.66
I) Parameters for the Calculation of the CF of the Diagnostic Description LV_WEAK
t (PL;[C],FFP)= 1.0 t (PL;[E],IVE)=0.66 t (PL;iC],SFP) = 1 .O t(PL;{SI,EP)=O.66 t(PL;{C,E],EP) = 0.33 t(PL;IS],FFP)=0.5 t(PL;[C,E],FFP)=0.O t (PL;IS),SFP)= 1.0 ~ t(PL;{C,E),SFP)= 1.0
m) Parameters for the Calculation of the CF of the Diagnostic Description Akinetic Molitility at the P L - S e g m e n t
The left ventricle viewed as one
The p o s i e r o - l o t e r a l Segment
object satisfies the motility
satisfies the motility
interpretation "weak motility"
interpretation "ekinesis" with
with certainty 0.33
certainty 0.5
n) Short Textual Description of the Result of I)
o) Short Textual Description at the Result of m)
In the p o s t e r o - l o t e r o l area of the left ventricle was on okinesis detected with certainty 0.33
p) Short Textual Descripption of the Complete Diagnostic Interpretation of the Image Sequence
Figure 5 (continued). (1) (p). 98
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
attributes in the concepts of this level are the volume curves of the objects. Figure 5c shows the volume curve of the left ventricle and Figure 5d that of the postero-lateral segment. Both are calculated out of the contours in Figure 5b. Interpretation of a heart cycle for each of the objects left ventricle (LV), IA, PL, BA, and SE, is a two phase process executed on the levels 5 and 6 of the knowledge base. The basic idea is to use optimal search techniques [11, 15]. Image to image fuzzy membership values for stagnation, expansion, and contraction are used as negative cost functions. Based on these costs, dynamic programming is used to find the optimal fit between the volume curve of each object and each cycle type of Figure 4. Examples of resulting descriptions are shown in Figures 5e and 5f for a LV and a PL segment respectively. At the next level (MOTILITY) cycles are verified by introducing a refined alphabet, which subsumes also symbols for the overlapping of different motility terms and for gaps inside one phase. These somewhat relaxed interpretations are generated in order to allow some flexibility in later processing steps. Results of such cycle descriptions are shown in Figures 5g and 5h. Overlapping symbols are denoted by the two elementary terms and gaps by a phase symbol and ' - ' . This multilevel parsing procedure was presented in more detail in [17]. Out of these descriptions the motility phases are calculated with normalized start and end points (in the time range 0.0 to 1.0), Figures 5i, 5j. Because most diagnostic terms can only be evaluated with respect to the global motility behaviour of the left ventricle, an intermediate level, called anatomical motility phases of the LV, is introduced. Here, the cycle of the LV is divided into the medical terms SYSTOLE, DIASTOLE, and in a second level into Pre-Ejection-Period, Ejection-Period, Fast-Filling-Period, Iso-Volume-Expansion, and Slow-Filling-Period. This segmentation is based on the cycle description of the LV at the former level. For the calculation of each phase patterns in the former cycle descriptions are searched and scored with special fuzzy membership functions, which reflect a priori knowledge about the start time, the duration, and the ratio between minimal and maximal area of the LV in the corresponding phase. Such ratios, which are also called ejection fractions
September 1988
(el), are themselves parameters for the diagnostic evaluation at the both top levels in the network. The segmentation of a left ventricle cycle in these terms together with the corresponding start and end times is shown in Figure 5k. Presently the system knows about 45 individual diagnostic interpretations which are represented by concepts at the levels 4, 8, and 9. These are of the type 'left ventricle is normally beating', 'the left ventricle is deformed', or 'the basal segment of the left ventricle is nearly motionless'. The certainty of each statement is measured by a certainty factor CF which is computed by fuzzy algebra. For example, given a rule of inference of the form If (not A) or (B and C) then D the certainty factor of D is computed from those of A, B, and C by CF(D) : = max(l - CF(A), min(CF(B),CF(C))) or in short hand notation CF(D) : = (not CF(A)) or (CF(B) and CF(C)). For the very top level, i.e. for COMPLETE INTERPRETATIONS, A, B, C, and D in the equations above stand for names of concepts in the knowledge base. Since there is an obvious one-to-one correspondence between the rule for deriving a conclusion and the CF of this conclusion, it is sufficient to state only the equation for the CF. The rules for complete interpretations at level 9 use concepts on level 8 LOCAL MOTILITY DESCRIPTIONS and/or level 4 DESCRIPTIONS IN FORM AND PROPORTIONS as arguments. One of the complete interpretations which is only based on local motility descriptions is akinetic motility (AKIN). The CF is given by CF(AKtN): = CF(LV WEAK) and (CF(AKIN IA) or CF(AKIN__PL) or CF(AKIN BA) or CF(AKIN SE) ). That means an akinetic motility is stated if at least one of the four segments shows akinetic motility and the global left ventricle has a weak motility. Another more complicated diagnostic interpretation is the kongestive kardiomypathy (KKMY). It is characterized by nearly akinetic inferio-apical and postero-lateral segments (AKIN_IA, AKIN PL) 99
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
combined with very weak motility of the total left ventricle (LV VERY WEAK) as well as an enlarged left ventricle (LV ENLA). A rest of motility is just in the septal or in the basal segment (SE, BA). That means, at least one of these both segments shows normal (NORM) or hypokinetic motility (HvPo). Therefore, this diagnostic interpretation is stated by CF(KKMY) : = CF(LV_VERY_WEAK) and CF(LV ENLA)and CF(IA_AKIN) and CF(PL_AKIN) and (CF(BA NORM) or CF(BA HYPO) or CF(SE_NoRM) or CF(BA HvPo)). Similar rules are available for 6 more diagnoses on level 9. Things are somewhat more complicated on level 8 because on this level descriptions have to be derived from motility phases. At this level the motility of individual segments of the left ventricle is diagnostically interpreted. This is done with the help of two functions t and u defined below. Let P be an element of the anatomical motility phases which are conceptually defined at level 6, that is P c {CYCLE, SYSTOLE,DIASTOLE, PEP, EP, FFP, IVE, SFP}. On level 5, compare Figures 5i, j, for each segment s sE {IA, PL, BA, SE}. The behaviour during one heart cycle is represented by possibly overlapping phases C, E, or S. Let B be defined by B~{C, E, S}. The function t(B, s, P) gives the percentage of motility phases of B in a segment s during the anatomical motility phase P. For example, t({E,S}, PL, FFP) = 0.8 means that the postero-lateral segment expands or stagnates during 80 % of the fast-filling-period. Let
u(d,s,k,i,x) be a fuzzy membership function of similar shape as shown in Figure 3, where 100
September 1988
d ~ (NoRMal, HvPokinetic, AKINetic, DvsKinetic, PHAsEshifted) are diagnostic terms, s is a segment as defined above, k is the criterion used, e.g. ejection fraction of the cycle (EF) or amount of stagnation (SA), and x is the independent variable for which u is to be evaluated. For example, u(AKIN, IA, SA, t({S},IA,CYCLE)) is the certainty factor for akinesis of the inferio-apical segment when the amount of stagnation is considered and the independent variable is the result of the application of the function t. The functions u are derived from general medical knowledge [19]. With the functions u and t the CF's for individual diagnostic terms are determined. As an example, the CF of akinesis in a segment s is considered. A segment is termed akinetic if it shows nearly no motility. That is, the segment does not contract or expand during each phase of the left ventricle but it stagnates. These are the main criteria in the following rule. Further criteria are necessary to distinguish akinetic behaviour from a phaseshifted, a hypokinetic, and a dyskinetic one. CF(s,AKIN) = u(AKIN,s,EF,ef) and
u(AKIN,s,SA,t({S},s, CYCLE)) and (not t({K},s,PEP)) and (not t({E},s,EP)) and (not t({K},s,FFP)) and (not t({K},s,SFP)) and (not t({E},s,IVE)) and (t({S},s,EP) or (not t({K,E},s,EP)) ) and (t({S},s,FFP) or (not t({K,E},s,FFP)) ) and (t({S},s,SFP) or (not t({K,E},s,SFP)) ), where ef is the ratio between minimal and maximal area of the cycle. For the postero-lateral segment the values for all these functions are shown in Figure 5m. Applying the rule, results for the example of Figure 5 in a CF of 0.5 for the term postero-lateral akinesis, Figure 5o. With a similar rule also based on the functions t and u, a weak motility is stated for the left ventricle, Figures 51, 5n. Finally, at the very top level the CF for akinesis, see the rule or CF(AKIN) above, is calculated and the result of the analysis process as indicated in Figure 5p is reached.
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
4. Results and conclusion The paper presented an expert system which successfully describes scintigraphic image sequences of the beating heart. The first operational version of the realized system has about 4MB of code. Tests on about 20 image sequences having validated diagnostic evaluations by medical doctors showed that no wrong descriptions were given by the system; however, in about 10% of the cases the system could not give any description. The system is an example of a successful combination of low level image processing techniques with advanced knowledge based processing and inference. Quite different types of knowledge (e.g. rules for medical diagnosis, regular expressions for typical heart cycles) were integrated in a complex associative network structure, and quite different types of procedural knowledge (e.g. for computation of contour length, of certainty factors, or of optimal heart cycles) were attached to the concepts of the semantic network. The semantic network thus proved to be a flexible and powerful approach to integrating a complex system into a homogeneous structure. Besides the described modules, the system covers tools for explanation and knowledge acquisition. Especially in a medical environment the user needs a lot of information about automatically derived diagnostic interpretations in order to be confident in the way the system solves a given problem. The developed tools for explanation are based on the a priori knowledge of the system, the generated instances, and the search space of the analysis process to be explained [8]. The main tool for knowledge acquisition in the described application is a special editor for the network. Having prepared the relevant knowledge the editor offers a special human interface to store the knowledge in the network by observing all restrictions of the special formalism. One network node is a special questionnaire filled with default values, or values obtained by the user. This questionnaire is the human view of the complex data structures stored in the knowledge base [9]. Our future work in the area of knowledge based systems for image understanding will make further use of associative networks and extend the facilities for knowledge acquisition and explanation of system resources and activities.
September 1988
Acknowledgement The work reported in this paper was supported by the German Research Foundation (DFG). The system DISS was developed in cooperation with H. Feistel and F. Wolf of the Department for Nuclear Medicine of the University Erlangen. Additionally, I want to thank D.P. Pretschner from the University of Hildesheim for many fruitful discussions concerning the system DIss. The development of the system shell ERNESTwas also supported by the Siemens AG, Erlangen. A lot of work for the realization of this shell was done by S. Schr6der and F. Kummert.
References [1] Brachman, R.J. (1979). On the epistemological status of semantic networks. In: N.V. Findler, Ed., Associative Networks. Academic Press, New York. [2] Buchanan, B.G. and Shortliffe, E.H. (1984). Rule-Based Expert Systems. Addison-Wesley, Reading, MA. [3] McDermott, J. (1981) RI: The formative years. AI Magazine 2, 21 29. [4] Duda, R.O., et al. (1978). Development of the PROSPECTOR Consultant System for Mineral Exploration. Rept. Artificial Intelligence Center, SRI Int., Menlo Park, CA. [5] Eichhorn, W. and Niemann, H. (1986). A bidirectional control strategy in a hierarchical knowledge structure. Proc. 8th Int. Cot1[~ on Pattern Recognition ( I C P R ) , Paris, 181 183. [6] Findler, N.V., Ed. (1979). Associative Networks. Academic Press, New York. [7] Hayes-Roth, F., Waterman, D.A. and Lenat, D.B., Eds. (1982). Building Expert Systems. McGraw Hill, New York. [8] Hofmann, I., Giimlich, R. and Niemann, H. (1986). A human interface for control of an image processing system. Proc. 8th ICPR, Paris, 1986, 1256-1258. [9] Kummert, F., Niemann, H., Sagerer, G. and Schroeder, S., Werkzeuge zur modellgesteuerten Bildanalyse und Wissensakquisition. In: M. Paul, Ed., GI-17. Jahrestagung. Springer, Berlin, 556-570. [io] Levesque, H. and Mylopoulos, J. (1979). A procedural semantics for semantic networks. In: N.V. Findler, Ed. Associative Networks. Academic Press, New York, 93- 121. [11] Niemann, H. (1981). Pattern Analysis. Springer, Berlin. B21 Niemann, H., Bunke, H., Hofmann, I., Sagerer, G., Wolf, F. and Feistel, H. (1985). A knowledge based system for analysis of gated blood pool studies. IEEE Trans. Pattern Anal. Machine Intell. 7, 246 259. [13] Niemann, H. (1985). A homogeneous architecture for knowledge based image understanding. Proc. 2. Conf Art# ficial Intefligence Applications. Miami, EL, 1985, 88 93. 101
Volume 8, Number 2
PATTERN RECOGNITION LETTERS
[14] Nii, H.P. et al. (1982). Signal-to-symbol transformation: HASP/SlAP case study. AI Magazine 3, 23 35. [15] Nilsson, N.J. (1982). Principles of Artificial Intelligence. Springer, Berlin. [16] Sagerer, G. (1985). Darstellung und Nutzung von Expertenwissen fiir ein Bildanalysesystem. Springer, Berlin. [17] Sagerer, G. (1986). A multilevel parsing procedure based on dynamic programming for noisy input strings. Proc. EUSIPCO-86. North-Holland Amsterdam, 517-520. [18] Sagerer, G. and Kummert, F. (1988). Knowledge based sys-
102
September 1988
tems for speech understanding. In: Niemann, H., Lang, M. and Sagerer, G., Eds. Recent Advances in Speech Understanding and Dialog Systems. Springer, Berlin. [19] Schicha, H. and Emrich, D. (1984). Nuklearmedizin in der kardiologischen Praxis: Methoden Ergebnisse Indikation. GIT-Verlag, Ernst Giebeler, Darmstadt. [20] Stansfield, S.A. (1986). ANGY: A rule-based expert system for automatic segmentation of coronary vessels from digital subtracted angiograms. 1EEE Trans. Pattern Anal. Machine Intell., 188 199.