- Email: [email protected]
On information fusion to improve segmentation of MRI sequences
Information Fusion 3 (2002) 103–117 www.elsevier.com/locate/inffus
On information fusion to improve segmentation of MRI sequences A. Taleb-Ahmed *, L. Gautier Laboratoire d’Analyse des Syst emes du Littoral, 40 rue F. Buisson, Bat A, B.P. 649-62228, Calais Cedex, France Received 29 November 2000; received in revised form 6 December 2001; accepted 7 December 2001
Abstract The paper deals with the problem of segmentation of MRI sequences of vertebrae, in the form of images of their multiple slices, using the Dempster–Shafer theory. This leads to the study of 3-D deformations of the scoliosis. The motivation comes from the inadequacy of the existing techniques based on X-ray image analysis. Such analysis cannot deal with, on the one hand, the complex anatomical structures (‘‘scoliotic rachis’’), and the spongy tissue peri-rachidian, and, on the other hand, the choice of slices and the problem of the residue irradiation present in each examination. The main contributions of the paper are: • • • • • •
New architecture for the fusion of MRI data sets. A novel method to exploit the information contained in MRI sequence. Model for knowledge representation adapted to specificity of information available (Dempster–Shafer theory). Choice of the discriminating parameters for the statistical expertise. Construction of the belief functions. Choice of the decision criterion.
Starting from segmentation by active contour (snake) [Deformable contour: modelling, extraction, detection and classification, Ph.D. Thesis, Wisconsin–Madison University, 1994; Proceedings of the 15th International Conference on Pattern Recognition, vol. 4, Barcelona, 2000, p. 17; Int. J. Comput. Vision 1 (3) (1987) 211], we upgrade it in an attempt to present the doctor with a degree of belief concerning their membership of the contour of the vertebra. We illustrate the proposed fusion architecture by application to actual MRI sequences of the vertebrae, and include perhaps the first example of 3-D reconstruction of the lumbar rachis starting from the results obtained during fusion. 2002 Elsevier Science B.V. All rights reserved. Keywords: Architecture of global fusion; Belief function; Information fusion; Statistical training; MRI image; Dempster–Shafer theory; Snake; Decision criterion; Segmentation of vertebra
1. Introduction A scoliosis [1–3,6] is a 3-D deformation of lumbar rachis. It is also a disease of the rachis during the growth, whose dynamics needs to be known for medical diagnostics. It is a 3-D deformation of a part of or of the complete the spinal column (cf. Figs. 1(a) and (b)). It involves a torsion of one or more vertebrae on themselves, and causes a deformation of the thorax, abdomen, and para vertebral zones. This torsion results in several curves and ‘non-curves’ in 3-D space. The noncurves compensate for the total balance of the rachis to maintain the horizontally of the glance. In these deformations, we observe the following: *
Corresponding author. E-mail address: [email protected] (A. Taleb-Ahmed).
• a frontal component, the side deviation, whereas the vertebrae are normally stacked vertically; • a sagittal component responsible for a lordosis of a vertebra compared to the other because the length of the deformation is more significant in front than behind; and • a transverse component with rotation of the vertebra around a vertical axis bringing the vertebral body towards convexity. The study of scoliosis requires the specification of a system of axes. By convention, we define the standard orthogonal system of Cartesian co-ordinates with origin O: OX forward, OY to the left and OZ upward. This enables us to define and quantify the position and the orientation of the vertebrae. The deformation of the rachis is a torsion, i.e., a simultaneous movement in each plane. Note that we should not confuse torsion with the
1566-2535/02/$ - see front matter 2002 Elsevier Science B.V. All rights reserved. PII: S 1 5 6 6 - 2 5 3 5 ( 0 2 ) 0 0 0 5 2 - 0
104
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
Fig. 1. Rachis lumbar: (a) healthy rachis; (b) scoliotic rachis.
rotation, which is an one-way movement in the transverse plan, vertebra by vertebra. We analyse the 3-D vertebral deformations using the MRI sequence in order not only to evaluate the modifications of the position relating one vertebra to the other in translation and rotation, but also for a better understanding of the biomechanics and the physiopathology of the scoliosis. Godillon [23] study of these deformations from X-ray projections, and propose a technique to visualize a geometrical 3-D model of the rachis on a graphic screen. In this context, see [25] for a computerized decisionmaking preoperative system for orthopedic surgery of the scoliosis, and [24] for a system of measurement of 3D vertebral displacements during surgery. In the latter, segmentation of the vertebrae is not the main objective but only visualization in 3-D. The papers of [26,27] deal with the segmentation of the vertebrae using a spherical or elliptic model.
A disadvantage of the X-ray-based methods is the problem of irradiation and protection against it. In fact, radiation examinations are repeatedly conducted on children, teenagers, and young patients until the end of their growth. The carcinogenic risks are significant involving (i) the breast and thyroid or (ii) genetics by chromosomal transfer. This is not the case if we carry out an examination by using MRI [9,22], which is not irradiant. A recent paper [28] presents a data processing method of segmentation of image based on the use of models in which information for segmentation is automatically obtained starting from training in the form of examples of segmentation. The training consists in building two models: one represents the form to be obtained and the other the area where the object is located. For segmentation, it uses active contours whose external term of energy takes into account the area delimited at the time of the training. The method was tested in MRI on 2-D slices. Fusion of information from the MRI sequence is expected to yield results that constitute an improvement over those obtained from standard segmentation procedures applied to the individual images. And such a fusion can be accomplished in many ways. The classical architectures meant for this purpose (cf. Figs. 2(a) and (b)) correspond to single/multisensor information fusion. The latter requires several physical sensors that includes the case of a single sensor with different assumptions (as in medical imaging) or geometric observation positions (as in stereovision). The multiple images in the MRI sequence could be treated as though multiple sensors have generated them. In this paper, we present the first stage of the work on the segmentation of the vertebrae using the MRI sequences. The examination of a scoliosis by MRI requires two orthogonal incidences (sagittal and coronal), each incidence consisting of a sequence of 12 slices. The information is then available at different levels: (i) low (grey value distribution), (ii) contour (relationship be-
Fig. 2. Fusion system architecture: (a) monosensor; (b) multisensor.
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
tween the contours of vertebra), and (iii) volume (MRI sequence). It has been found that an independent examination of these levels does not help in (i) the extraction of valuable information from the segmentation points using snakes, thereby implying that we cannot quantify with certainty the 3-D vertebral deformations, and (ii) the minimization of uncertainty and the inaccuracy due to the sensor in MRI data acquisition. The method that we propose in this paper for segmentation of MRI sequences differs significantly from the methods of the literature in the following respects: • Application of active contours (‘snake’) to improve the result obtained from the segmentation. • Exploitation of information of MRI sequence by data fusion. • Architecture of fusion. • Presentation of the final results with a degree of belief for each point of the snake. • Future prospects of this work, especially involving realistic 3-D visualization. 2. Data acquisition and image sequence analysis The MRI sensor is rated at 0.2 T, and outputs an image sequence of sagittal parallel jointed slices of the spinal column. The protocol of acquisition is T 1; the thickness of the slice is 4 mm; and the images are of size 512 512 pixels, 0.7031 mm/pixel, and each pixel having been quantized to 4096 grey levels. Segmentation of the MRI sequence consists in extracting the relevant entities from it, and is crucial to the phases of analysis and interpretation. From an initial segmentation, we seek a segmentation which represents, as well as possible, the anatomical contour of the vertebra, in order to propose to the doctors image pixels really forming part of the vertebra, and to highlight the locations on which we cannot base our inferences. The main idea is to improve the knowledge of the actual objects observed, starting from only partial knowledge and deformation of the objects. The slices of sequence MRI generally present imperfections (partial volume, noise, etc.) so that it is not always possible to define exactly the anatomical contour of vertebra. The partial volume appears when the thickness of the slice is too significant with respect to the organ is studied, thereby involving an attenuation or a local vanishing of certain edges of the vertebrae. We propose to use the maximum of information that is present in the MRI sequence, in order to extract additional and redundant information, thereby allowing us to confirm or cancel the decision on the membership of each point in the initial segmentation. We invoke the fusion of information which consists in combining information resulting from several sources in order to improve the decision-making: (i) raw data
105
from the physical sensor, (ii) pre-processed data, and (iii) a priori knowledge which could be physical, statistical or expert knowledge, relating to the same object. The methodology is based on the modelling of knowledge using the theory of the evidence [14,15] with a view to improve the segmentation by snake. It has the advantage of introducing the concept of doubt among the various data analysed. Moreover, it provides a better representation of certainty regarding the computed segmentation, and makes it possible to combine very heterogeneous data [29–31]. We first define the framework of discernment and the specification of information of inputs. Then we present how knowledge is modelled in the form of distributions of mass. After describing the proposed architecture of fusion, we evaluate the performance of the fusion algorithm. Finally, we provide some results obtained from real images along with an example of 3-D reconstruction of lumbar rachis. 2.1. Problem formulation As mentioned above, the MRI image sequence relates to sagittal parallel jointed slices of the spinal column. Each slice consists of several vertebrae. From the acquired series of parallel slices of the spinal column, one of the problems is to determine the pixels belonging to the cortex (cf. Fig. 3). Each slice consists of K objects representing K distinct organs (or bodies) defined by H ¼ fskin; spongy matter cortex; body; air; vertebral; muscle; fat; fluid:g
ð1Þ
Many of these objects are not separable from each other, caused by the principle of operation of the MRI
Fig. 3. Cortex in MRI image.
106
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
sensor, giving rise to partial volume, noise, artifacts of edges, and distortion. However, partial volume and the noise are the only factors affecting the segmentation. To model a zone made up of a mixture of several different objects, one adds a new element to H of (1) to get H0 ¼ fskin; spongy matter cortex; body; air; vertebral; muscle; fat; fluid; miscellaneousg: ð2Þ The main problems addressed in the paper can be formulated as follows: • How to develop a strategy to decide whether each pixel belongs to the cortex? • What is the estimate of the confidence that we can associate with that decision?
3. Proposed technique One of the difficulties in the analysis of an MRI sequence is the extremely large number of image pixels. However, it turns out that we can limit the number of these pixels by pre-processing and by focusing on the area of interest. We define points of interest as those on the contour of the vertebra. On each parallel slice of the spinal column, we carry out segmentation by active contour, commonly called snake [7,8,11–13]. For the snake, the usual segmentation criterion is of the form: Esnake ðvÞ ¼
Z
1
½Eint ðvðsÞÞ þ Eext ðvðsÞÞ ds;
ð3Þ
0
where Eint ðvðsÞÞ is an internal energy term that represents constraints on the object shape and Eext ðvðsÞÞ is the external energy term that is associated with the image data. The external energy is usually computed from the magnitude of the image intensity gradient. Here, it is parameterized so that the criterion of minimization of energy results in a convergence to the zone of hyposignal which is characteristic of belonging to cortex (or cortical osseous) (cf. Fig. 3). The cortex part is extracted from the images of all the slices, and is sampled with the same number M of points Qi . The results obtained are very satisfactory when the phenomenon of partial volume is negligible, but not satisfactory when the snake diverges (cf. Fig. 4). In order to solve the problem of divergence, we resort to data fusion for combining all the information in the sequence, at the levels of (i) the pixel (grey level distribution), and (ii) the contour (relation between successive slices). Further, we invoke Dempster–Shafer’s theory of evidence in order to minimize the uncertainty (on the exact position of the contour of the vertebra) caused, amongst other things, by partial volume and noise.
Fig. 4. Divergent snakes.
3.1. A priori knowledge In order to be able to determine if a point belongs to the cortex or not, we need to determine properties of the cortex based on the parameters measured. We can now exploit physical, statistical or human expertise: 1. Weak signal (hypo-signal) or low level of grey. This knowledge is deduced from the physical laws that govern the sensor and from the effects on the various anatomical structures.
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
107
5. A small distance between two matching points on two consecutive slices means that both are parts of the cortex. 6. On three consecutive slices, if the two distances between the three matching points are significant, then the middle point is not a part of cortex. 3.2. Modeling of knowledge: distributions of mass
Fig. 5. Expertise picture.
2. A statistical study is carried out on a sample set of 50 MRI slices. For example, for the image slice of Fig. 5, we study all the possible situations that we can associate with an unspecified Qi pixel. Table 1 presents, for each one of these situations, the minimal value of the intensity of grey level and the value of the standard deviation of grey level (denoted by r), calculated by starting from a neighbourhood of 3 3 and 5 5. It is found that calculation based on a neighbourhood of 3 3 is sufficient. From Table 1, we note that there is very little variation of the computed values for both neighbourhood 3 3 and 5 5. Further, human and physical expertise’s make it possible to conclude that the contour of the vertebra has a thickness lower than 3 mm. 3. The inter-slice distance is known by the structure of sensor and, according to experts, the dimension of vertebra is high compared to this thickness. 4. A significant distance between two matching points on two consecutive slices means that at least one of the two is not a part of cortex.
The evidence theory developed by Dempster [15] and better formalized by Shafer [14] enables us to represent both uncertainty and imprecision with two functions, plausibility and credibility (or belief) [17,18]. This is one of the main advantages of Dempster–Shafer approach (DS) [34,35]. Indeed, it leads to very flexible and rich modelling, able to fit large class functions, occurring in particular in image imaging. A few examples of situations where DS or evidence theory may be successfully used are: • When a source differentiates two classes and another not: Dempster–Shafer allows to deal with hesitation or ambiguity between two classes. • In the case of partial volume effects in MRI (often at the border of two classes) it can also be taken into account by assigning masses to union of the two classes mixed in considered area. • In cases where knowledge of source reliability is available only for some classes: it can be taken into account by modifying accordingly the masses assigned to these classes and by introducing ignorance. • In cases where a priori information has to be introduced: even if it is not represented in a probabilistic manner. It can be taken into account if it induces a way to assign masses, in particular to compound hypothesis. • When a source provides information concerning only a few of several classes: for instance brain Positron Emission Tomography (PET) images under some conditions allow for the detection of the brain surface but not of the head surface.
Table 1 Statistical training Neighborhood 33 Min. of grey level A B C D E F G
59 782 775 629 285 40 387
55 r 287 107 86 275 103 80 166
Min. of grey level 59 782 707 460 198 40 373
r 330 98 101 341 101 91 206
Cortex Vertebral body Vertebral body Contour between two biological entities Muscle Air Contour between two biological entities
108
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
The plausibility and credibility functions are based on the definition of the frame of discernment H constituted of k exclusives hypothesis, the k classes i.e., a set of elementary events. H ¼ fH1 ; H2 ; . . . ; Hk g:
ð4Þ
The frame of discernment H is composed of exclusive and exhaustive hypothesis (the solution of the problem Hi 2 H). A referential 2H represents the set of all subsets S of H: 2H ¼ fS=S Hg ¼ f;; fH1 g; fH2 g; . . . ; fHk g; fH1 ; H2 g; . . . ; Hg:
ð5Þ
A subset S that contains several assumptions is called a composed proposition. If a subset contains only one assumption Hi , it is called singleton (single) proposition. Plausibility and credibility can be expressed with a unique function, the mass function. Mass, plausibility and credibility, which are defined on 2H , characterize the likelihood on any subset of H. It can be shown that there exists a bisection between these three functions. The mass function is defined on 2H 2 ½0; 1 by mð;Þ ¼ 0; X mðSÞ ¼ 1:
ð6Þ ð7Þ
S H
The propositions of 2H that have a mass not equal to zero are named focal elements. The value mðSÞ represents the degree of belief (or evidential) support that a specific element of H belongs to the set S, but not to a particular subset of S. Each expert (or each source of information) defines a mass assignment according to its opinion about the situation. To obtain a final decision it is necessary to combine the opinion of the different experts. This combination is realized with the Dempster’s orthogonal sum. Let mð1Þ ; . . . ; mðdÞ be the masses associated to d independent information sources defined on the same frame of discernment H. It is then possible to combine them according the Dempster–Shafer orthogonal combination rule. This rule results in m ¼ m1 m2 md
For each point (for example called Qi ) of contours of each slices (for example the slice noted Q), we have measurement and a priori knowledge. It is now a question of modelling this information in order to be able to combine them with an aim of making a decision on the membership of each point Qi to cortex. Modeling requires the definition of a distribution of mass of evidence depending on each measurement and based on knowledge. The functions of masses make it possible to give knowledge which we have the various assumptions of the frame of discernment. The data which we measured and calculated for each snaxel are: • the grey level; • the average and the standard deviation of grey level in a neighbourhood; • the distance between two snaxels of two consecutive slices. 3.2.1. Minimum of grey level intensity According to the statistical study made 3 3 neighbourhood, the minimum of the intensity of grey level is low (in Table 1 values inferior to 100) for the cortex one, the air and the fluid. The proposal Sgrey level is thus defined by Sgrey level 2 2H ; Sgrey level ¼ fcortex; air; fluidg; Sgrey level ¼ 2 Sgrey level : We propose a distribution of mass mgrey level as presented in Fig. 6(a).
ð8Þ
with the operator of Dempster–Shafer combination. The combination is possible only when sources are independent and when basic mass assignments are defined on the same referential H. K ¼ mð;Þ 6¼ 0 is a measure of conflict between the different sources. After the combination of the different sources of information, each pixel is defined as belonging to a given class according to a certain criteria: the maximum plausibility, the maximum of credibility or the mass of evidence. We chose, in this paper, the measure of the mass of evidence.
ð9Þ
H
Fig. 6. Mass functions.
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
3.2.2. Maximum of standard deviation The statistical study indicates that the maximum of standard deviation (in Table 1 values superior to 200) 3 3 neighbourhood corresponds to the cortex and to the mixture between two bodies. One thus defines
( mPQ ðSPQ Þ ¼
mPQ ðSPQ Þ ¼ Sr 2 2H ; Sr ¼ fcortex; mixesg;
dðPi ; Qi Þ 2 ½e::b ;
0
else
One proposes a distribution of mass mr as presented in Fig. 6(b). 3.2.3. Inter-slice distance The distance between two matching snaxels of consecutive slices gives information on the joint membership to cortex of these two points. The definitions of the assumptions (hypothesis) are more complex in this case, because an assumption does not take its value in 2h that according to other parameters such as it is indicated in a priori knowledge described in Section 3.1 at items 4, 5 and 6. Let three matching points Pi , Qi and Ri of the slices P, Q and R. We propose the frame of discernment hPQ ¼ fsPQ ; sPQ g concerning the slices P and Q such as SPQ corresponds to the assumption, where two points Pi and Qi are of comparable nature (thus both belonging to cortical osseous), and hQR ¼ fsQR ; sQR g concerning the slices Q and R. We seek to define when Qi belongs to cortical osseous. We call SQ this assumption. According to a priori knowledge, we can draw up the following truth table of combination hypothesis (cf. Table 2). We propose then the distributions of masses mPQ and mQR concerning frames of discernment hPQ and hQR , where mPQ is defined by the following equation:
Table 2 The truth table of combining hypothesis SPQ
SPQ
HPQ
SQ SQ SQ
SQ SQ SQ
SQ SQ HQ
8dðPi ; Qi Þ;
1 egjdðPi ;Qi Þbj ;
dðPi ; Qi Þ 2 ½b::1½;
0
else: ð11Þ
ð10Þ
Sr ¼ 2H Sr :
SQR SQR HQR
1 egjdðPi ;Qi Þbj ;
gjdðPi ;Qi Þbj
mPQ ðHPQ Þ ¼ e (
109
• dðPi ; Qi Þ is the distance between detected contours of the two consecutive slices (P ; Q), • b represents the tolerance which the expert associates with the value dðPi ; Qi Þ. When dðPi ; Qi Þ ¼ b, the doubt is maximum between the two assumptions, • e represents the inter-slice distance, • the coefficient g makes it possible to tolerate a greater inaccuracy on the geometrical resemblance of two consecutive contours. This distribution of mass is represented Fig. 7. It is noted that when the distance dðPi ; Qi Þ is low (inferior to b) then both points are of comparable nature and thus they belong to the cortex. If the distance dðPi ; Qi Þ is high (superior to b) then the both points are of different nature and one of them does not belong to the cortex. The distribution mQR has the same form exactly. 3.3. Architecture of fusion As mentioned above, the combination of distributions of masses requires that those be defined on the same frame of discernment which is not the case for the functions defined in Section 3.2.3 in Eq. (11). Indeed, information obtained starting from these two frames of discernment hPQ and hQR gives total information on the membership of these two points Pi Qi or Qi Ri . Either they belong both to the cortex or one of both does not belong to the cortex. Unfortunately we cannot know which of the two points belongs (or not belongs) to the cortex. On the other hand information obtained starting from the minimum of the intensity of grey level and maximum of standard deviation experts, directly give information on the belonging of the point to the cortex or not. Then, both space of discernment defined for the two expertise’s
Fig. 7. Representation of the functions of masses.
110
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
Fig. 8. Fusion structure.
(minimum of the intensity of grey level, maximum of standard deviation and inter-slice distance) are thus not the same. We have two distributions of masses mgrey level and mr defined on 2hQi and two distributions of masses mPi Qi and mQi Ri defined on 2hPi Qi and 2hQi Ri . If we combine mPi Qi and mQi Ri by using the truth table (Table 2), the result mPi Qi Ri takes, then its values in the frame of discernment 2hQi . We thus propose the structure of fusion presented in Fig. 8. After having carried out the segmentation of each slice by snake contour, one carries out the pairing of the snaxels. We start by combined mPQ and mQR . We obtain the following equations starting from the logical table: mPQR ¼ mPQ mQR ; mPQR ðSQ Þ ¼ mPQ ðSPQ Þ mQR ðSRQ Þ þ mPQ ðSPQ Þ mQR ðSRQ Þ
means that a certain quantity of mass can be affected by calculation to the empty set. This mass corresponds to the conflict between the sources, it is noted K: K ¼ mQ ð;Þ; It makes it possible to highlight a defect concerning the sources of information and if its value is high (higher than the precision of numerical calculations) it must lead to a discussion on the validity of the results but in no case to be neglected. Our fusion architecture is given Fig. 9 in which the model fusion system exploits the output of the fictive sensors and the snake in order to obtain a better knowledge of the points of the snake which belong or not to the vertebra.
þ mPQ ðSPQ Þ mQR ðHRQ Þ þ mPQ ðSPQ Þ mQR ðSRQ Þ þ mPQ ðHPQ Þ mQR ðSRQ Þ; mPQR ðSQ Þ ¼ mPQ ðSPQ Þ mQR ðSRQ Þ þ mPQ ðHPQ Þ mQR ðSRQ Þ þ mPQ ðSPQ Þ mQR ðHRQ Þ; mPQR ðHQ Þ ¼ mPQ ðHPQ Þ mQR ðHRQ Þ; ð12Þ
where SQ corresponds to the assumption cortex. mPQR is defined on 2h same manner as mgreylevel and mr . It is thus possible to combine them again to define a distribution of mass mQ taking into account all information of which we lay out on the point Qi : mQ ¼ mPQR mgreylevel mr :
ð13Þ
It should be noted that during fusion, no ‘‘standardization’’ distribution of mass is carried out. That
Fig. 9. Our fusion architecture.
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
111
Fig. 10. Active contours from segmentation step: (a) coupe P; (b) coupe Q; (c) coupe R.
4. Experimental results At the end of the step of global fusion, a proposal for a segmentation is suggested to the doctor. That consists in giving to the doctor for each point of the snake the degree of belief calculated and visualizing the result as in Figs. 11(a)–(c). The degree of belief gives an opinion for snaxel on the fact that it belongs to the cortex (S proposal, or not S proposal), on the fact that we cannot decide (X proposal), or if a conflict between sources of information was detected (K high value). Our synoptic of global fusion was tested, on NEC Worksation, Pentium II, 266 MHz, on real data. The time required for a sequence MRI including the steps of segmentation by snake and statistical training and fusion algorithm is 10–20 s. In Figs. 11(a)–(c), we give an example corresponding to the real slices of Figs. 10(a)–(c) (in these figures the crosses represent the points resulting from the segmentation by snake). In Figs. 11(a)–(c), the points represented by • squares correspond to a bad decision of segmentation;
• rounds correspond to a good decision; • the ‘+’ sign represents the doubt; • the crosses represent the conflict between the experts. In Figs. 12(a)–(c), we represent in x-coordinate the index of the snaxels and in ordinate the amplitude of measurements of the evidence and the conflict: • The features in double thickness represent the coefficient of conflict between the experts K. • Those in full feature relate to the points belonging to the cortex. • The dotted lines are associated with the points not belonging to the cortex. • And finally the discontinuous feature represents the doubt. The goal of Figs. 12(a)–(c) is to show, in detail, for each point of snake: (i) the existence and the evolution of the conflict at the time of the decisions, (ii) and, when the conflict is null, the decision without ambiguity on the membership or not of the point to the cortex.
112
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
Fig. 11. Global fusion: (a) slice P; (b) slice Q; (c) slice R.
We note a high value of K for certain points, for example the subscripted points from 1 to 8. If we refer to Fig. 13, we note that these points correspond to situation A. This situation corresponds to the presence of a partial volume. From the point of view of experts, there is a conflict between the experts of fusion on the levels of the pixels and contours. We will consider in detail some points (the point noted 5, the point noted 25 and the point noted 45) (cf. Figs. 11 and 13). We gathered the values of Tables 3 and 4, where the exponent 1 corresponds to the expertise on the level of the pixels and the exponent 2 corresponds to the expertise on the level of contours.
belongs to the cortex with a degree of confidence of 79% (4th row). We have, after total fusion, a conflict of 60% (10th row). Thus one of the experts is inevitably mistaken. Indeed, let us suppose that the hypothesis of our frame of discernment are correct and that the source of information is not defective. Then in our case, we can say if there is a conflict it means that one of the two experts gives an distorted answer. For the slice R, expert 1 affirms that the point does not belong to the cortex with a degree of confidence of 96% (2nd row) whereas expert 2 affirms that the point belongs to the cortex with a degree of confidence of 100% (4th row). There is after global fusion a conflict of 96% (10th row). Inevitably one of the experts is mistaken.
Consider the point of index 5 of Figs. 11 and 13. For the slice P, the total decision is in agreement with the opinions of the experts on the level of the pixels (2nd row) and the level of contours (5th row). For the slice Q, expert 1 affirms that the point does not belong to the cortex with a degree of confidence of 76% (2nd row) whereas expert 2 affirms that the point
Consider the point of index 25 of Figs. 11 and 13, we make the same study that for the point of index 5. For the slice P expert 1 affirms that the point does not belong to the cortex with a degree of confidence of 49% (2nd row) with an uncertainty of 51% (3rd row) whereas expert 2 has a total uncertainty (6th row). There is after
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
113
Fig. 12. Decision criteria on the various points of contour active: (a) slice P; (b) slice Q; (c) slice R.
global fusion a carry forward of the decision of expert 1 (9th row).
For the slice Q, we have after global fusion a conflict of 57% (10th row), Inevitably one of the experts is mistaken. For the slice R, we have a conflict of 74% (10th row). Finally consider the point of index 45 of Figs. 11 and 13. Table 3 Summary of the examples of results of decision-making (part one) Point no. 5
1
Fig. 13. Projection in a same plan of the three slices P, Q and R.
m ðSx Þ m1 ðSx Þ m1 ðXx Þ m2 ðSx Þ m2 ðSx Þ m2 ðXx Þ m1;2 ðSx Þ m1;2 ðSx Þ m1;2 ðXx Þ K 1;2
Point no. 25
P
Q
R
P
Q
R
0 0.90 0.10 0 0.79 0.21 0 0.98 0.02 0
0 0.76 0.24 0.79 0 0.21 0.19 0.16 0.05 0.60
0 0.96 0.04 1 0 0 0.04 0 0 0.96
0 0.49 0.51 0 0 1 0 0.49 0.51 0
0 0.57 0.43 1 0 0 0.43 0 0 0.57
0 0.74 0.26 1 0 0 0.26 0 0 0.74
114
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
Table 4 Summary of the examples of results of decision-making (part two) Point no. 45
m1 ðSx Þ m1 ðSx Þ m1 ðXx Þ m2 ðSx Þ m2 ðSx Þ m2 ðXx Þ m1;2 ðSx Þ m1;2 ðSx Þ m1;2 ðXx Þ K 1;2
P
Q
R
0 0.43 0.57 1 0 0 0.57 0 0 0.43
0.81 0 0.19 1 0 0 1 0 0 0
0 0.14 0.86 1 0 0 0.86 0 0 0.14
1st row 2nd row 3rd row 4th row 5th row 6th row 7th row 8th row 9th row 10th row
For the slice P we have a conflict of 43% (10th row). For the slice Q, expert 1 affirms that the point belongs to the cortex with a degree of confidence of 81% (1st row) whereas expert 2 affirms that the point belongs to the cortex with a degree of confidence of 100% (4th row). There is after global fusion a total agreement of the decisions (7th row). For the slice R, we have after global fusion, a conflict of 14% (10th row) because expert 1 doubts for 86% (3rd row) on the membership of the point to the cortex, while expert 2 affirms that the point belongs to the cortex with a degree of confidence of 100% (4th row). 4.1. Evaluation of the decision The evaluation of a system of perception is always complex and requires having a ‘‘reference’’. We have chosen to ask an expert (the doctor) to carry out the classification of the 50 manually snaxels obtained by the snake. The results are then compared with those obtained with the architecture of fusion proposed when the decision is made starting from the criterion of maximum of evidence mass. Thus we will be able to compare the decisions we made, concerning each point of the snake, with resulting from our process of fusion that we proposed. This evaluation will be used as a test of quality of our architecture of fusion. Labeling of the points of contour by an human operator. With the exit of the segmentation by snake, the human operator (the doctor) finds: • For the slice P, 45 points belonging to the cortex and 5 points not belonging to the cortex, on a total of 50 initial points. • For the slice Q, 40 points belonging to the cortex and 10 points not belonging to the cortex, on a total of 50 initial points. • For the slice R, 41 points belonging to the cortex and 9 points not belonging to the cortex, on a total of 50 initial points.
Labeling of the points of contour after global fusion. After the stage of decision, based on the criterion of the measurement of the mass of the evidence we find: • For the slice P, 28 points belonging to the cortex, 3 points not belonging to the cortex, 3 points for which the expertise cannot conclude and 16 points where there is a conflict between the experts. • For the slice Q, 21 points belonging to the cortex and 6 points not belonging to the cortex, 1 dubious point and 23 points presenting a conflict. • For the slice R, 19 points belonging to the cortex and 4 points not belonging to the cortex, 1 dubious point and 26 points presenting a conflict. Analysis by human operator of labelling after fusion. • For the slice P, on the 28 points which are classified as belonging to the cortex, there is no error of decision. On the other hand, on the 3 points that are classified as not belonging to the cortex there is 1 error of decision. • For the slice Q, on the 21 points which are classified as belonging to the cortex, there is no error of decision. On the other hand on the 6 points which are classified as not belonging to the cortex, there is 1 error of decision. • For the slice R, on the 19 points which are classified as belonging to the cortex, there is no error of decision. On the 4 points which are classified as not belonging to the cortex, there is no error of decision. 4.2. Calculation of the rate of good decision We chose to evaluate the rate (noted T) of good decision, on the one hand for the points belonging to the cortex, on the other hand for the points not belonging to the cortex. This rate corresponds to the number (or percentage) of good decision, concerning the membership or not of a snaxel to the cortex, delivered by our architecture of fusion. This maximum rate is given in the form of the following report/ratio: Tl ¼
number of good decisions number of bad good decisions ; the number of good decisions
where l 2 S; S. We have for the three slices P ; Q; R an average rate of good decision of: • TS ¼
ð28 þ 21 þ 19Þ 0 ¼ 100% 28 þ 21 þ 19
for the points belonging to the cortex,
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
•
115
6. Conclusions and prospects ð3 þ 6 þ 4Þ 2 ¼ 84:6% T ¼ 3þ6þ4 S
for the points not belonging to the cortex. We note, on the one hand, that the number of points for which it is possible to make a decision as belonging to (or not belonging to) the cortex is low. Indeed we have 65 points out of the initial 150 where there is a conflict between the experts and for which it is impossible to make a decision. In addition, there is no error of classification on the points belonging to the cortex. The system is thus relatively ‘‘careful’’, not affirming the assumption of membership if this one is false. 5. Comparison with literature To the best of our knowledge, there are no results in the literature similar to ours using the same approach and the same architecture of fusion to improve the segmentation by snake from MRI sequence of vertebrae. The comparison which we can carry out relates to the work of [28], because it also deals with 2-D slices MRI sequence (but there is no fusion strategies used). It provides a methodology for fully automated model-based image segmentation. All the information necessary to perform image segmentation is automatically obtained from a training set that provides segmentation examples form. The training set is used to construct two models representing the objects: shape model based on Hough transform and border appearance model. The border appearance model can be used to automatically compute cost functions used in segmentation criteria of image segmentation methods, for examples like dynamic programming or for snakes. The segmentation task required for a total of 55 images, 15 images were selected for training, each with example segmentation of a single vertebra. Each training image contained manually outlined border with 50 landmarks. For a HP Worksation, Pentium II, 300 MHz the time required: • For the shape model, 1–8 s. The time depends on the number of training images (3–21) as well as on the number of landmarks defined on every image (41–51). • For border appearance model, 30 s–3 min. The time depends on the number of training data and the number of expected border patterns. The segmentation was compared with manually defined border, the errors of approximate location is 2:3 0:8 pixels for the rms (root mean square) criterion. Finally, we do not know at the end of the segmentation if the found points belong or not with certainty to the vertebra required and the extension to three dimensions is not considered yet.
This paper provides a methodology for fully automated 2-D segmentation of vertebra from MRI sequence. We have shown that it is possible, thanks to the fusion of data, to improve classification of the points resulting from the segmentation by snake and to quantify by a degree of belief the membership of each point to the cortex. We also showed as the representation of knowledge using the theory of evidence of Dempster– Shafer was well adapted to combine information available to different levels: on the level of the pixels and the level of contours. Dempster–Shafer’s theory has the advantage of introducing the concept of doubt between the various studied data and allows combining in an unspecified order the data available to the various levels. With regard to the prospects, we can underline several points. It would be interesting, for the fusion on the level of the pixels, not to take as only value (or parameter) the grey value distribution of the images, but to integrate others of them, such as for example the space context, or the parameters of textures. Each of these parameters can bring new information and it is very easy to combine all this knowledge if they are expressed in the same formalism. It remains, nevertheless, to evaluate knowledge and to model the sets of masses by taking into account the space context. The spatial context represents an additional source of information. To express that constraint of spatial homogeneity, we can use the Markov models. Moreover, the assumption used within the framework of fusion on the level of contours which stipulates that the distance between the matching snaxels of two consecutive slices must be close to the inter-slice distance e, can be supplemented by other information a priori, in particular on the form of contours to be obtained. It would be also interesting to evaluate, on the one hand, the contribution of all new parameter in fusion scheme, on the level of the pixels as on the level of contours, in terms of cost of calculation (this parameter can provoke in the global level fusion a combinative explosion) and in addition the real contribution of information which can bring this parameter, in order to determine precisely which are those which really make it possible to differentiate the initial assumptions. Indeed, it is not necessary to multiply the number of parameters to be combined. It would also be necessary to make a comparative study on the influence of the functions of masses on the results of fusion. Indeed, the value of the masses is obtained directly starting from the functions of masses. Any modification of these functions involves modifications on the decision-making, distributed or not. We showed that the model of global fusion proposed reveals the conflict between the sources, and thus makes it possible to question the reliability of at least one of
116
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
providing MR sequences images for experiments and validation. Siemens and ‘‘le Conseil general du Nord pas de Calais’’ for the financing. Special thanks to J.M. Folzan.
References
Fig. 14. Examples of 3-D visualization of the spinal column.
these sources. It would be interesting to determine by a system of rules which source is not reliable and thus through a loop of feedback, to inform the system about the non-reliability of the defective source. Lastly, we limited ourselves in this work to only one MRI sequence, but the extension of our study on several incidences is absolutely possible. We can, in this case, tackle the problem directly starting from 3-D data MRI. Finally, it is possible, starting from contours obtained of each vertebra, from the global fusion processing, to calculate the relative 3-D deformations between the vertebrae and to give a representation in 3-D of the column (cf. Fig. 14). We have to carry out this pilot study on the basis of a certain number of simplifying assumptions namely: • The point obtained at the end of the global fusion have all be validated by the doctor. • Whatever the size of snake which approximates the vertebra, we consider that each snaxel have the same weighting for the computation of the center of gravity of the vertebra. • The computation of the center of gravity is not carried out compared with the volume of the vertebra but through the points of the different snakes. • The cubic splines use for the computation of axis principal of column is the one proposed by the software Matlab. • The angles computed between two unspecified vertebrae of the column are obtained by derivation of the spline function. Acknowledgements The authors express thanks to the Department of Radiology of the Institut Calot de Berck-sur Mer for
[1] H. Leclet, P.M. Delforge, Imagerie des scolioses: quand et comment, Rev. Imag. Med. (5) (1993) 773–782. [2] H. Leclet, P.Y. Plais, P.M. Delforge, Scoliose de l’enfant et de l’adolescent. Des lesions anatomiques a l’imagerie pratique, J. Francßais Radiol. (1992) 73–82. [3] H. Leclet, Artifact in MRI of the spine with or without implant, Eur. Spine J. (1994) 240–245. [6] P. Maroteaux, Les maladies osseuses de l’enfant, Flammarion, Paris, 1995. [7] K.F. Lai, Deformable contour: modeling, extraction, detection and classification, Ph.D. Thesis Wisconsin–Madison University, 1994. [8] T. Abe, Y. Matsuzawa, Multiple active contour models with application to region extraction, in: 15th International Conference on Pattern Recognition, Barcelona, vol. 4, 2000, p. 17. [9] A. Dromigny-Badin, Fusion d’images par la theorie de l’evidence en vue d’applications medicales et industrielles, Ph.D. Thesis, Institut Nationnal des Sciences Appliquees de Lyon, 1998. [11] R. Terzopoulos, A. Witkin, M. Kass, Symmetry-seeking models for 3D object reconstruction, Int. J. Comput. Vision 1 (3) (1987) 211–221. [12] M. Kunt, Traitement numerique des images, Presses Polytechniques et Universitaires Romandes, 1993. [13] J.P. Coquerez, S. Philippe, Analyse d’images: filtrage et segmentation, Masson, Paris, 1995. [14] G. Shafer, A Mathematical Theory of Evidence, Princeton University Press, Princeton, NJ, 1976. [15] A.P. Dempster, A class of random convex polytopes, Ann. Math. Statist. (1972) 250–272. [17] T. Denoeux, A k-nearest neighbor classification rule based on Dempster–Shafer theory, IEEE Trans. Syst., Mans Cybern. 25 (5, 25) (1995) 804–813. [18] I. Bloch, Information combination operators for data fusion : a comparative review with classification, IEEE Trans. Syst., Man Cybern. 26 (1) (1996) 52–67. [22] M. Brejl, M. Sonka, Medical Image segmentation, automated design of border detection criteria from examples, J. Electron. Imag. 8 (1) (1999) 54–64. [23] A. Godillon, Modelisation geometrique et cinematqiue 3D du rachis scoliotique pour la realisation d’un outil d’aide au medecin, Ph.D. Thesis, Universite de Valenciennes, 1991. [24] F. Dufour, Systeme de mesure 3D des deplacements vertebraux en cours de chirurgie, Ph.D. Thesis, Universite de Valenciennes, 1999. [25] S. Chusseau, Systeme d’aide a la decision pre-operatoire en chirurgie orthopedique de la scoliose, Ph.D. Thesis, Universite de Valenciennes, 1999. [26] T.N. Huynh, G. Maurais, Development of a vertebral endplate 3D reconstruction technique, IEEE. Trans. Med. Imag. 11 (2) (1997) 689–696. [27] G. Lefaix, A. Rameee, 3D modeling and deformation analysis of the vertebra with spherical harmonics, IEEE. EMBS 2 (1997) 422– 425. [28] M. Brejl, M. Sonka, Object localization and border detection criteria design in edge-based image segmentation: automated learning from examples, IEEE Trans. Med. Imag. 19 (10) (2000) 973– 983. [29] L. Gautier, A. Taleb-Ahmed, M. Rombaut, J.G. Postaire, H. Leclet, Proved segmentation from pictures sequence by evidence
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117 theory, application MRI pictures, in: The 2nd International Conference on Information, Fusion’99, 1999. [30] L. Gautier, A. Taleb-Ahmed, M. Rombaut, J.G. Postaire, H. Leclet, Belief function in low level data fusion: application in RMI, in: The 3rd International Conference on Information, Fusion’2000, 2000, p. TUC5-3 (see also p. TUC5-8). [31] A. Taleb-Ahmed, L. Gautier, M. Rombaut, J.G. Postaire, H. Leclet, Information fusion: application for improve segmentation
117
from MRI sequence images, in: The 4th International Conference on Information, Fusion’2001, 2001. [34] D. Dubois, H. Prade, Possibility theory in information fusion, in: IEEE 3rd International Conference on Information Fusion, Fusion’2000, 2000, p. PS-6. [35] P. Smets, Data fusion in the transferable belief model, in: IEEE 3rd International Conference on Information Fusion, Fusion’2000, 2000, p. PS-21.
On information fusion to improve segmentation of MRI sequences A. Taleb-Ahmed *, L. Gautier Laboratoire d’Analyse des Syst emes du Littoral, 40 rue F. Buisson, Bat A, B.P. 649-62228, Calais Cedex, France Received 29 November 2000; received in revised form 6 December 2001; accepted 7 December 2001
Abstract The paper deals with the problem of segmentation of MRI sequences of vertebrae, in the form of images of their multiple slices, using the Dempster–Shafer theory. This leads to the study of 3-D deformations of the scoliosis. The motivation comes from the inadequacy of the existing techniques based on X-ray image analysis. Such analysis cannot deal with, on the one hand, the complex anatomical structures (‘‘scoliotic rachis’’), and the spongy tissue peri-rachidian, and, on the other hand, the choice of slices and the problem of the residue irradiation present in each examination. The main contributions of the paper are: • • • • • •
New architecture for the fusion of MRI data sets. A novel method to exploit the information contained in MRI sequence. Model for knowledge representation adapted to specificity of information available (Dempster–Shafer theory). Choice of the discriminating parameters for the statistical expertise. Construction of the belief functions. Choice of the decision criterion.
Starting from segmentation by active contour (snake) [Deformable contour: modelling, extraction, detection and classification, Ph.D. Thesis, Wisconsin–Madison University, 1994; Proceedings of the 15th International Conference on Pattern Recognition, vol. 4, Barcelona, 2000, p. 17; Int. J. Comput. Vision 1 (3) (1987) 211], we upgrade it in an attempt to present the doctor with a degree of belief concerning their membership of the contour of the vertebra. We illustrate the proposed fusion architecture by application to actual MRI sequences of the vertebrae, and include perhaps the first example of 3-D reconstruction of the lumbar rachis starting from the results obtained during fusion. 2002 Elsevier Science B.V. All rights reserved. Keywords: Architecture of global fusion; Belief function; Information fusion; Statistical training; MRI image; Dempster–Shafer theory; Snake; Decision criterion; Segmentation of vertebra
1. Introduction A scoliosis [1–3,6] is a 3-D deformation of lumbar rachis. It is also a disease of the rachis during the growth, whose dynamics needs to be known for medical diagnostics. It is a 3-D deformation of a part of or of the complete the spinal column (cf. Figs. 1(a) and (b)). It involves a torsion of one or more vertebrae on themselves, and causes a deformation of the thorax, abdomen, and para vertebral zones. This torsion results in several curves and ‘non-curves’ in 3-D space. The noncurves compensate for the total balance of the rachis to maintain the horizontally of the glance. In these deformations, we observe the following: *
Corresponding author. E-mail address: [email protected] (A. Taleb-Ahmed).
• a frontal component, the side deviation, whereas the vertebrae are normally stacked vertically; • a sagittal component responsible for a lordosis of a vertebra compared to the other because the length of the deformation is more significant in front than behind; and • a transverse component with rotation of the vertebra around a vertical axis bringing the vertebral body towards convexity. The study of scoliosis requires the specification of a system of axes. By convention, we define the standard orthogonal system of Cartesian co-ordinates with origin O: OX forward, OY to the left and OZ upward. This enables us to define and quantify the position and the orientation of the vertebrae. The deformation of the rachis is a torsion, i.e., a simultaneous movement in each plane. Note that we should not confuse torsion with the
1566-2535/02/$ - see front matter 2002 Elsevier Science B.V. All rights reserved. PII: S 1 5 6 6 - 2 5 3 5 ( 0 2 ) 0 0 0 5 2 - 0
104
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
Fig. 1. Rachis lumbar: (a) healthy rachis; (b) scoliotic rachis.
rotation, which is an one-way movement in the transverse plan, vertebra by vertebra. We analyse the 3-D vertebral deformations using the MRI sequence in order not only to evaluate the modifications of the position relating one vertebra to the other in translation and rotation, but also for a better understanding of the biomechanics and the physiopathology of the scoliosis. Godillon [23] study of these deformations from X-ray projections, and propose a technique to visualize a geometrical 3-D model of the rachis on a graphic screen. In this context, see [25] for a computerized decisionmaking preoperative system for orthopedic surgery of the scoliosis, and [24] for a system of measurement of 3D vertebral displacements during surgery. In the latter, segmentation of the vertebrae is not the main objective but only visualization in 3-D. The papers of [26,27] deal with the segmentation of the vertebrae using a spherical or elliptic model.
A disadvantage of the X-ray-based methods is the problem of irradiation and protection against it. In fact, radiation examinations are repeatedly conducted on children, teenagers, and young patients until the end of their growth. The carcinogenic risks are significant involving (i) the breast and thyroid or (ii) genetics by chromosomal transfer. This is not the case if we carry out an examination by using MRI [9,22], which is not irradiant. A recent paper [28] presents a data processing method of segmentation of image based on the use of models in which information for segmentation is automatically obtained starting from training in the form of examples of segmentation. The training consists in building two models: one represents the form to be obtained and the other the area where the object is located. For segmentation, it uses active contours whose external term of energy takes into account the area delimited at the time of the training. The method was tested in MRI on 2-D slices. Fusion of information from the MRI sequence is expected to yield results that constitute an improvement over those obtained from standard segmentation procedures applied to the individual images. And such a fusion can be accomplished in many ways. The classical architectures meant for this purpose (cf. Figs. 2(a) and (b)) correspond to single/multisensor information fusion. The latter requires several physical sensors that includes the case of a single sensor with different assumptions (as in medical imaging) or geometric observation positions (as in stereovision). The multiple images in the MRI sequence could be treated as though multiple sensors have generated them. In this paper, we present the first stage of the work on the segmentation of the vertebrae using the MRI sequences. The examination of a scoliosis by MRI requires two orthogonal incidences (sagittal and coronal), each incidence consisting of a sequence of 12 slices. The information is then available at different levels: (i) low (grey value distribution), (ii) contour (relationship be-
Fig. 2. Fusion system architecture: (a) monosensor; (b) multisensor.
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
tween the contours of vertebra), and (iii) volume (MRI sequence). It has been found that an independent examination of these levels does not help in (i) the extraction of valuable information from the segmentation points using snakes, thereby implying that we cannot quantify with certainty the 3-D vertebral deformations, and (ii) the minimization of uncertainty and the inaccuracy due to the sensor in MRI data acquisition. The method that we propose in this paper for segmentation of MRI sequences differs significantly from the methods of the literature in the following respects: • Application of active contours (‘snake’) to improve the result obtained from the segmentation. • Exploitation of information of MRI sequence by data fusion. • Architecture of fusion. • Presentation of the final results with a degree of belief for each point of the snake. • Future prospects of this work, especially involving realistic 3-D visualization. 2. Data acquisition and image sequence analysis The MRI sensor is rated at 0.2 T, and outputs an image sequence of sagittal parallel jointed slices of the spinal column. The protocol of acquisition is T 1; the thickness of the slice is 4 mm; and the images are of size 512 512 pixels, 0.7031 mm/pixel, and each pixel having been quantized to 4096 grey levels. Segmentation of the MRI sequence consists in extracting the relevant entities from it, and is crucial to the phases of analysis and interpretation. From an initial segmentation, we seek a segmentation which represents, as well as possible, the anatomical contour of the vertebra, in order to propose to the doctors image pixels really forming part of the vertebra, and to highlight the locations on which we cannot base our inferences. The main idea is to improve the knowledge of the actual objects observed, starting from only partial knowledge and deformation of the objects. The slices of sequence MRI generally present imperfections (partial volume, noise, etc.) so that it is not always possible to define exactly the anatomical contour of vertebra. The partial volume appears when the thickness of the slice is too significant with respect to the organ is studied, thereby involving an attenuation or a local vanishing of certain edges of the vertebrae. We propose to use the maximum of information that is present in the MRI sequence, in order to extract additional and redundant information, thereby allowing us to confirm or cancel the decision on the membership of each point in the initial segmentation. We invoke the fusion of information which consists in combining information resulting from several sources in order to improve the decision-making: (i) raw data
105
from the physical sensor, (ii) pre-processed data, and (iii) a priori knowledge which could be physical, statistical or expert knowledge, relating to the same object. The methodology is based on the modelling of knowledge using the theory of the evidence [14,15] with a view to improve the segmentation by snake. It has the advantage of introducing the concept of doubt among the various data analysed. Moreover, it provides a better representation of certainty regarding the computed segmentation, and makes it possible to combine very heterogeneous data [29–31]. We first define the framework of discernment and the specification of information of inputs. Then we present how knowledge is modelled in the form of distributions of mass. After describing the proposed architecture of fusion, we evaluate the performance of the fusion algorithm. Finally, we provide some results obtained from real images along with an example of 3-D reconstruction of lumbar rachis. 2.1. Problem formulation As mentioned above, the MRI image sequence relates to sagittal parallel jointed slices of the spinal column. Each slice consists of several vertebrae. From the acquired series of parallel slices of the spinal column, one of the problems is to determine the pixels belonging to the cortex (cf. Fig. 3). Each slice consists of K objects representing K distinct organs (or bodies) defined by H ¼ fskin; spongy matter cortex; body; air; vertebral; muscle; fat; fluid:g
ð1Þ
Many of these objects are not separable from each other, caused by the principle of operation of the MRI
Fig. 3. Cortex in MRI image.
106
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
sensor, giving rise to partial volume, noise, artifacts of edges, and distortion. However, partial volume and the noise are the only factors affecting the segmentation. To model a zone made up of a mixture of several different objects, one adds a new element to H of (1) to get H0 ¼ fskin; spongy matter cortex; body; air; vertebral; muscle; fat; fluid; miscellaneousg: ð2Þ The main problems addressed in the paper can be formulated as follows: • How to develop a strategy to decide whether each pixel belongs to the cortex? • What is the estimate of the confidence that we can associate with that decision?
3. Proposed technique One of the difficulties in the analysis of an MRI sequence is the extremely large number of image pixels. However, it turns out that we can limit the number of these pixels by pre-processing and by focusing on the area of interest. We define points of interest as those on the contour of the vertebra. On each parallel slice of the spinal column, we carry out segmentation by active contour, commonly called snake [7,8,11–13]. For the snake, the usual segmentation criterion is of the form: Esnake ðvÞ ¼
Z
1
½Eint ðvðsÞÞ þ Eext ðvðsÞÞ ds;
ð3Þ
0
where Eint ðvðsÞÞ is an internal energy term that represents constraints on the object shape and Eext ðvðsÞÞ is the external energy term that is associated with the image data. The external energy is usually computed from the magnitude of the image intensity gradient. Here, it is parameterized so that the criterion of minimization of energy results in a convergence to the zone of hyposignal which is characteristic of belonging to cortex (or cortical osseous) (cf. Fig. 3). The cortex part is extracted from the images of all the slices, and is sampled with the same number M of points Qi . The results obtained are very satisfactory when the phenomenon of partial volume is negligible, but not satisfactory when the snake diverges (cf. Fig. 4). In order to solve the problem of divergence, we resort to data fusion for combining all the information in the sequence, at the levels of (i) the pixel (grey level distribution), and (ii) the contour (relation between successive slices). Further, we invoke Dempster–Shafer’s theory of evidence in order to minimize the uncertainty (on the exact position of the contour of the vertebra) caused, amongst other things, by partial volume and noise.
Fig. 4. Divergent snakes.
3.1. A priori knowledge In order to be able to determine if a point belongs to the cortex or not, we need to determine properties of the cortex based on the parameters measured. We can now exploit physical, statistical or human expertise: 1. Weak signal (hypo-signal) or low level of grey. This knowledge is deduced from the physical laws that govern the sensor and from the effects on the various anatomical structures.
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
107
5. A small distance between two matching points on two consecutive slices means that both are parts of the cortex. 6. On three consecutive slices, if the two distances between the three matching points are significant, then the middle point is not a part of cortex. 3.2. Modeling of knowledge: distributions of mass
Fig. 5. Expertise picture.
2. A statistical study is carried out on a sample set of 50 MRI slices. For example, for the image slice of Fig. 5, we study all the possible situations that we can associate with an unspecified Qi pixel. Table 1 presents, for each one of these situations, the minimal value of the intensity of grey level and the value of the standard deviation of grey level (denoted by r), calculated by starting from a neighbourhood of 3 3 and 5 5. It is found that calculation based on a neighbourhood of 3 3 is sufficient. From Table 1, we note that there is very little variation of the computed values for both neighbourhood 3 3 and 5 5. Further, human and physical expertise’s make it possible to conclude that the contour of the vertebra has a thickness lower than 3 mm. 3. The inter-slice distance is known by the structure of sensor and, according to experts, the dimension of vertebra is high compared to this thickness. 4. A significant distance between two matching points on two consecutive slices means that at least one of the two is not a part of cortex.
The evidence theory developed by Dempster [15] and better formalized by Shafer [14] enables us to represent both uncertainty and imprecision with two functions, plausibility and credibility (or belief) [17,18]. This is one of the main advantages of Dempster–Shafer approach (DS) [34,35]. Indeed, it leads to very flexible and rich modelling, able to fit large class functions, occurring in particular in image imaging. A few examples of situations where DS or evidence theory may be successfully used are: • When a source differentiates two classes and another not: Dempster–Shafer allows to deal with hesitation or ambiguity between two classes. • In the case of partial volume effects in MRI (often at the border of two classes) it can also be taken into account by assigning masses to union of the two classes mixed in considered area. • In cases where knowledge of source reliability is available only for some classes: it can be taken into account by modifying accordingly the masses assigned to these classes and by introducing ignorance. • In cases where a priori information has to be introduced: even if it is not represented in a probabilistic manner. It can be taken into account if it induces a way to assign masses, in particular to compound hypothesis. • When a source provides information concerning only a few of several classes: for instance brain Positron Emission Tomography (PET) images under some conditions allow for the detection of the brain surface but not of the head surface.
Table 1 Statistical training Neighborhood 33 Min. of grey level A B C D E F G
59 782 775 629 285 40 387
55 r 287 107 86 275 103 80 166
Min. of grey level 59 782 707 460 198 40 373
r 330 98 101 341 101 91 206
Cortex Vertebral body Vertebral body Contour between two biological entities Muscle Air Contour between two biological entities
108
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
The plausibility and credibility functions are based on the definition of the frame of discernment H constituted of k exclusives hypothesis, the k classes i.e., a set of elementary events. H ¼ fH1 ; H2 ; . . . ; Hk g:
ð4Þ
The frame of discernment H is composed of exclusive and exhaustive hypothesis (the solution of the problem Hi 2 H). A referential 2H represents the set of all subsets S of H: 2H ¼ fS=S Hg ¼ f;; fH1 g; fH2 g; . . . ; fHk g; fH1 ; H2 g; . . . ; Hg:
ð5Þ
A subset S that contains several assumptions is called a composed proposition. If a subset contains only one assumption Hi , it is called singleton (single) proposition. Plausibility and credibility can be expressed with a unique function, the mass function. Mass, plausibility and credibility, which are defined on 2H , characterize the likelihood on any subset of H. It can be shown that there exists a bisection between these three functions. The mass function is defined on 2H 2 ½0; 1 by mð;Þ ¼ 0; X mðSÞ ¼ 1:
ð6Þ ð7Þ
S H
The propositions of 2H that have a mass not equal to zero are named focal elements. The value mðSÞ represents the degree of belief (or evidential) support that a specific element of H belongs to the set S, but not to a particular subset of S. Each expert (or each source of information) defines a mass assignment according to its opinion about the situation. To obtain a final decision it is necessary to combine the opinion of the different experts. This combination is realized with the Dempster’s orthogonal sum. Let mð1Þ ; . . . ; mðdÞ be the masses associated to d independent information sources defined on the same frame of discernment H. It is then possible to combine them according the Dempster–Shafer orthogonal combination rule. This rule results in m ¼ m1 m2 md
For each point (for example called Qi ) of contours of each slices (for example the slice noted Q), we have measurement and a priori knowledge. It is now a question of modelling this information in order to be able to combine them with an aim of making a decision on the membership of each point Qi to cortex. Modeling requires the definition of a distribution of mass of evidence depending on each measurement and based on knowledge. The functions of masses make it possible to give knowledge which we have the various assumptions of the frame of discernment. The data which we measured and calculated for each snaxel are: • the grey level; • the average and the standard deviation of grey level in a neighbourhood; • the distance between two snaxels of two consecutive slices. 3.2.1. Minimum of grey level intensity According to the statistical study made 3 3 neighbourhood, the minimum of the intensity of grey level is low (in Table 1 values inferior to 100) for the cortex one, the air and the fluid. The proposal Sgrey level is thus defined by Sgrey level 2 2H ; Sgrey level ¼ fcortex; air; fluidg; Sgrey level ¼ 2 Sgrey level : We propose a distribution of mass mgrey level as presented in Fig. 6(a).
ð8Þ
with the operator of Dempster–Shafer combination. The combination is possible only when sources are independent and when basic mass assignments are defined on the same referential H. K ¼ mð;Þ 6¼ 0 is a measure of conflict between the different sources. After the combination of the different sources of information, each pixel is defined as belonging to a given class according to a certain criteria: the maximum plausibility, the maximum of credibility or the mass of evidence. We chose, in this paper, the measure of the mass of evidence.
ð9Þ
H
Fig. 6. Mass functions.
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
3.2.2. Maximum of standard deviation The statistical study indicates that the maximum of standard deviation (in Table 1 values superior to 200) 3 3 neighbourhood corresponds to the cortex and to the mixture between two bodies. One thus defines
( mPQ ðSPQ Þ ¼
mPQ ðSPQ Þ ¼ Sr 2 2H ; Sr ¼ fcortex; mixesg;
dðPi ; Qi Þ 2 ½e::b ;
0
else
One proposes a distribution of mass mr as presented in Fig. 6(b). 3.2.3. Inter-slice distance The distance between two matching snaxels of consecutive slices gives information on the joint membership to cortex of these two points. The definitions of the assumptions (hypothesis) are more complex in this case, because an assumption does not take its value in 2h that according to other parameters such as it is indicated in a priori knowledge described in Section 3.1 at items 4, 5 and 6. Let three matching points Pi , Qi and Ri of the slices P, Q and R. We propose the frame of discernment hPQ ¼ fsPQ ; sPQ g concerning the slices P and Q such as SPQ corresponds to the assumption, where two points Pi and Qi are of comparable nature (thus both belonging to cortical osseous), and hQR ¼ fsQR ; sQR g concerning the slices Q and R. We seek to define when Qi belongs to cortical osseous. We call SQ this assumption. According to a priori knowledge, we can draw up the following truth table of combination hypothesis (cf. Table 2). We propose then the distributions of masses mPQ and mQR concerning frames of discernment hPQ and hQR , where mPQ is defined by the following equation:
Table 2 The truth table of combining hypothesis SPQ
SPQ
HPQ
SQ SQ SQ
SQ SQ SQ
SQ SQ HQ
8dðPi ; Qi Þ;
1 egjdðPi ;Qi Þbj ;
dðPi ; Qi Þ 2 ½b::1½;
0
else: ð11Þ
ð10Þ
Sr ¼ 2H Sr :
SQR SQR HQR
1 egjdðPi ;Qi Þbj ;
gjdðPi ;Qi Þbj
mPQ ðHPQ Þ ¼ e (
109
• dðPi ; Qi Þ is the distance between detected contours of the two consecutive slices (P ; Q), • b represents the tolerance which the expert associates with the value dðPi ; Qi Þ. When dðPi ; Qi Þ ¼ b, the doubt is maximum between the two assumptions, • e represents the inter-slice distance, • the coefficient g makes it possible to tolerate a greater inaccuracy on the geometrical resemblance of two consecutive contours. This distribution of mass is represented Fig. 7. It is noted that when the distance dðPi ; Qi Þ is low (inferior to b) then both points are of comparable nature and thus they belong to the cortex. If the distance dðPi ; Qi Þ is high (superior to b) then the both points are of different nature and one of them does not belong to the cortex. The distribution mQR has the same form exactly. 3.3. Architecture of fusion As mentioned above, the combination of distributions of masses requires that those be defined on the same frame of discernment which is not the case for the functions defined in Section 3.2.3 in Eq. (11). Indeed, information obtained starting from these two frames of discernment hPQ and hQR gives total information on the membership of these two points Pi Qi or Qi Ri . Either they belong both to the cortex or one of both does not belong to the cortex. Unfortunately we cannot know which of the two points belongs (or not belongs) to the cortex. On the other hand information obtained starting from the minimum of the intensity of grey level and maximum of standard deviation experts, directly give information on the belonging of the point to the cortex or not. Then, both space of discernment defined for the two expertise’s
Fig. 7. Representation of the functions of masses.
110
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
Fig. 8. Fusion structure.
(minimum of the intensity of grey level, maximum of standard deviation and inter-slice distance) are thus not the same. We have two distributions of masses mgrey level and mr defined on 2hQi and two distributions of masses mPi Qi and mQi Ri defined on 2hPi Qi and 2hQi Ri . If we combine mPi Qi and mQi Ri by using the truth table (Table 2), the result mPi Qi Ri takes, then its values in the frame of discernment 2hQi . We thus propose the structure of fusion presented in Fig. 8. After having carried out the segmentation of each slice by snake contour, one carries out the pairing of the snaxels. We start by combined mPQ and mQR . We obtain the following equations starting from the logical table: mPQR ¼ mPQ mQR ; mPQR ðSQ Þ ¼ mPQ ðSPQ Þ mQR ðSRQ Þ þ mPQ ðSPQ Þ mQR ðSRQ Þ
means that a certain quantity of mass can be affected by calculation to the empty set. This mass corresponds to the conflict between the sources, it is noted K: K ¼ mQ ð;Þ; It makes it possible to highlight a defect concerning the sources of information and if its value is high (higher than the precision of numerical calculations) it must lead to a discussion on the validity of the results but in no case to be neglected. Our fusion architecture is given Fig. 9 in which the model fusion system exploits the output of the fictive sensors and the snake in order to obtain a better knowledge of the points of the snake which belong or not to the vertebra.
þ mPQ ðSPQ Þ mQR ðHRQ Þ þ mPQ ðSPQ Þ mQR ðSRQ Þ þ mPQ ðHPQ Þ mQR ðSRQ Þ; mPQR ðSQ Þ ¼ mPQ ðSPQ Þ mQR ðSRQ Þ þ mPQ ðHPQ Þ mQR ðSRQ Þ þ mPQ ðSPQ Þ mQR ðHRQ Þ; mPQR ðHQ Þ ¼ mPQ ðHPQ Þ mQR ðHRQ Þ; ð12Þ
where SQ corresponds to the assumption cortex. mPQR is defined on 2h same manner as mgreylevel and mr . It is thus possible to combine them again to define a distribution of mass mQ taking into account all information of which we lay out on the point Qi : mQ ¼ mPQR mgreylevel mr :
ð13Þ
It should be noted that during fusion, no ‘‘standardization’’ distribution of mass is carried out. That
Fig. 9. Our fusion architecture.
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
111
Fig. 10. Active contours from segmentation step: (a) coupe P; (b) coupe Q; (c) coupe R.
4. Experimental results At the end of the step of global fusion, a proposal for a segmentation is suggested to the doctor. That consists in giving to the doctor for each point of the snake the degree of belief calculated and visualizing the result as in Figs. 11(a)–(c). The degree of belief gives an opinion for snaxel on the fact that it belongs to the cortex (S proposal, or not S proposal), on the fact that we cannot decide (X proposal), or if a conflict between sources of information was detected (K high value). Our synoptic of global fusion was tested, on NEC Worksation, Pentium II, 266 MHz, on real data. The time required for a sequence MRI including the steps of segmentation by snake and statistical training and fusion algorithm is 10–20 s. In Figs. 11(a)–(c), we give an example corresponding to the real slices of Figs. 10(a)–(c) (in these figures the crosses represent the points resulting from the segmentation by snake). In Figs. 11(a)–(c), the points represented by • squares correspond to a bad decision of segmentation;
• rounds correspond to a good decision; • the ‘+’ sign represents the doubt; • the crosses represent the conflict between the experts. In Figs. 12(a)–(c), we represent in x-coordinate the index of the snaxels and in ordinate the amplitude of measurements of the evidence and the conflict: • The features in double thickness represent the coefficient of conflict between the experts K. • Those in full feature relate to the points belonging to the cortex. • The dotted lines are associated with the points not belonging to the cortex. • And finally the discontinuous feature represents the doubt. The goal of Figs. 12(a)–(c) is to show, in detail, for each point of snake: (i) the existence and the evolution of the conflict at the time of the decisions, (ii) and, when the conflict is null, the decision without ambiguity on the membership or not of the point to the cortex.
112
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
Fig. 11. Global fusion: (a) slice P; (b) slice Q; (c) slice R.
We note a high value of K for certain points, for example the subscripted points from 1 to 8. If we refer to Fig. 13, we note that these points correspond to situation A. This situation corresponds to the presence of a partial volume. From the point of view of experts, there is a conflict between the experts of fusion on the levels of the pixels and contours. We will consider in detail some points (the point noted 5, the point noted 25 and the point noted 45) (cf. Figs. 11 and 13). We gathered the values of Tables 3 and 4, where the exponent 1 corresponds to the expertise on the level of the pixels and the exponent 2 corresponds to the expertise on the level of contours.
belongs to the cortex with a degree of confidence of 79% (4th row). We have, after total fusion, a conflict of 60% (10th row). Thus one of the experts is inevitably mistaken. Indeed, let us suppose that the hypothesis of our frame of discernment are correct and that the source of information is not defective. Then in our case, we can say if there is a conflict it means that one of the two experts gives an distorted answer. For the slice R, expert 1 affirms that the point does not belong to the cortex with a degree of confidence of 96% (2nd row) whereas expert 2 affirms that the point belongs to the cortex with a degree of confidence of 100% (4th row). There is after global fusion a conflict of 96% (10th row). Inevitably one of the experts is mistaken.
Consider the point of index 5 of Figs. 11 and 13. For the slice P, the total decision is in agreement with the opinions of the experts on the level of the pixels (2nd row) and the level of contours (5th row). For the slice Q, expert 1 affirms that the point does not belong to the cortex with a degree of confidence of 76% (2nd row) whereas expert 2 affirms that the point
Consider the point of index 25 of Figs. 11 and 13, we make the same study that for the point of index 5. For the slice P expert 1 affirms that the point does not belong to the cortex with a degree of confidence of 49% (2nd row) with an uncertainty of 51% (3rd row) whereas expert 2 has a total uncertainty (6th row). There is after
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
113
Fig. 12. Decision criteria on the various points of contour active: (a) slice P; (b) slice Q; (c) slice R.
global fusion a carry forward of the decision of expert 1 (9th row).
For the slice Q, we have after global fusion a conflict of 57% (10th row), Inevitably one of the experts is mistaken. For the slice R, we have a conflict of 74% (10th row). Finally consider the point of index 45 of Figs. 11 and 13. Table 3 Summary of the examples of results of decision-making (part one) Point no. 5
1
Fig. 13. Projection in a same plan of the three slices P, Q and R.
m ðSx Þ m1 ðSx Þ m1 ðXx Þ m2 ðSx Þ m2 ðSx Þ m2 ðXx Þ m1;2 ðSx Þ m1;2 ðSx Þ m1;2 ðXx Þ K 1;2
Point no. 25
P
Q
R
P
Q
R
0 0.90 0.10 0 0.79 0.21 0 0.98 0.02 0
0 0.76 0.24 0.79 0 0.21 0.19 0.16 0.05 0.60
0 0.96 0.04 1 0 0 0.04 0 0 0.96
0 0.49 0.51 0 0 1 0 0.49 0.51 0
0 0.57 0.43 1 0 0 0.43 0 0 0.57
0 0.74 0.26 1 0 0 0.26 0 0 0.74
114
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
Table 4 Summary of the examples of results of decision-making (part two) Point no. 45
m1 ðSx Þ m1 ðSx Þ m1 ðXx Þ m2 ðSx Þ m2 ðSx Þ m2 ðXx Þ m1;2 ðSx Þ m1;2 ðSx Þ m1;2 ðXx Þ K 1;2
P
Q
R
0 0.43 0.57 1 0 0 0.57 0 0 0.43
0.81 0 0.19 1 0 0 1 0 0 0
0 0.14 0.86 1 0 0 0.86 0 0 0.14
1st row 2nd row 3rd row 4th row 5th row 6th row 7th row 8th row 9th row 10th row
For the slice P we have a conflict of 43% (10th row). For the slice Q, expert 1 affirms that the point belongs to the cortex with a degree of confidence of 81% (1st row) whereas expert 2 affirms that the point belongs to the cortex with a degree of confidence of 100% (4th row). There is after global fusion a total agreement of the decisions (7th row). For the slice R, we have after global fusion, a conflict of 14% (10th row) because expert 1 doubts for 86% (3rd row) on the membership of the point to the cortex, while expert 2 affirms that the point belongs to the cortex with a degree of confidence of 100% (4th row). 4.1. Evaluation of the decision The evaluation of a system of perception is always complex and requires having a ‘‘reference’’. We have chosen to ask an expert (the doctor) to carry out the classification of the 50 manually snaxels obtained by the snake. The results are then compared with those obtained with the architecture of fusion proposed when the decision is made starting from the criterion of maximum of evidence mass. Thus we will be able to compare the decisions we made, concerning each point of the snake, with resulting from our process of fusion that we proposed. This evaluation will be used as a test of quality of our architecture of fusion. Labeling of the points of contour by an human operator. With the exit of the segmentation by snake, the human operator (the doctor) finds: • For the slice P, 45 points belonging to the cortex and 5 points not belonging to the cortex, on a total of 50 initial points. • For the slice Q, 40 points belonging to the cortex and 10 points not belonging to the cortex, on a total of 50 initial points. • For the slice R, 41 points belonging to the cortex and 9 points not belonging to the cortex, on a total of 50 initial points.
Labeling of the points of contour after global fusion. After the stage of decision, based on the criterion of the measurement of the mass of the evidence we find: • For the slice P, 28 points belonging to the cortex, 3 points not belonging to the cortex, 3 points for which the expertise cannot conclude and 16 points where there is a conflict between the experts. • For the slice Q, 21 points belonging to the cortex and 6 points not belonging to the cortex, 1 dubious point and 23 points presenting a conflict. • For the slice R, 19 points belonging to the cortex and 4 points not belonging to the cortex, 1 dubious point and 26 points presenting a conflict. Analysis by human operator of labelling after fusion. • For the slice P, on the 28 points which are classified as belonging to the cortex, there is no error of decision. On the other hand, on the 3 points that are classified as not belonging to the cortex there is 1 error of decision. • For the slice Q, on the 21 points which are classified as belonging to the cortex, there is no error of decision. On the other hand on the 6 points which are classified as not belonging to the cortex, there is 1 error of decision. • For the slice R, on the 19 points which are classified as belonging to the cortex, there is no error of decision. On the 4 points which are classified as not belonging to the cortex, there is no error of decision. 4.2. Calculation of the rate of good decision We chose to evaluate the rate (noted T) of good decision, on the one hand for the points belonging to the cortex, on the other hand for the points not belonging to the cortex. This rate corresponds to the number (or percentage) of good decision, concerning the membership or not of a snaxel to the cortex, delivered by our architecture of fusion. This maximum rate is given in the form of the following report/ratio: Tl ¼
number of good decisions number of bad good decisions ; the number of good decisions
where l 2 S; S. We have for the three slices P ; Q; R an average rate of good decision of: • TS ¼
ð28 þ 21 þ 19Þ 0 ¼ 100% 28 þ 21 þ 19
for the points belonging to the cortex,
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
•
115
6. Conclusions and prospects ð3 þ 6 þ 4Þ 2 ¼ 84:6% T ¼ 3þ6þ4 S
for the points not belonging to the cortex. We note, on the one hand, that the number of points for which it is possible to make a decision as belonging to (or not belonging to) the cortex is low. Indeed we have 65 points out of the initial 150 where there is a conflict between the experts and for which it is impossible to make a decision. In addition, there is no error of classification on the points belonging to the cortex. The system is thus relatively ‘‘careful’’, not affirming the assumption of membership if this one is false. 5. Comparison with literature To the best of our knowledge, there are no results in the literature similar to ours using the same approach and the same architecture of fusion to improve the segmentation by snake from MRI sequence of vertebrae. The comparison which we can carry out relates to the work of [28], because it also deals with 2-D slices MRI sequence (but there is no fusion strategies used). It provides a methodology for fully automated model-based image segmentation. All the information necessary to perform image segmentation is automatically obtained from a training set that provides segmentation examples form. The training set is used to construct two models representing the objects: shape model based on Hough transform and border appearance model. The border appearance model can be used to automatically compute cost functions used in segmentation criteria of image segmentation methods, for examples like dynamic programming or for snakes. The segmentation task required for a total of 55 images, 15 images were selected for training, each with example segmentation of a single vertebra. Each training image contained manually outlined border with 50 landmarks. For a HP Worksation, Pentium II, 300 MHz the time required: • For the shape model, 1–8 s. The time depends on the number of training images (3–21) as well as on the number of landmarks defined on every image (41–51). • For border appearance model, 30 s–3 min. The time depends on the number of training data and the number of expected border patterns. The segmentation was compared with manually defined border, the errors of approximate location is 2:3 0:8 pixels for the rms (root mean square) criterion. Finally, we do not know at the end of the segmentation if the found points belong or not with certainty to the vertebra required and the extension to three dimensions is not considered yet.
This paper provides a methodology for fully automated 2-D segmentation of vertebra from MRI sequence. We have shown that it is possible, thanks to the fusion of data, to improve classification of the points resulting from the segmentation by snake and to quantify by a degree of belief the membership of each point to the cortex. We also showed as the representation of knowledge using the theory of evidence of Dempster– Shafer was well adapted to combine information available to different levels: on the level of the pixels and the level of contours. Dempster–Shafer’s theory has the advantage of introducing the concept of doubt between the various studied data and allows combining in an unspecified order the data available to the various levels. With regard to the prospects, we can underline several points. It would be interesting, for the fusion on the level of the pixels, not to take as only value (or parameter) the grey value distribution of the images, but to integrate others of them, such as for example the space context, or the parameters of textures. Each of these parameters can bring new information and it is very easy to combine all this knowledge if they are expressed in the same formalism. It remains, nevertheless, to evaluate knowledge and to model the sets of masses by taking into account the space context. The spatial context represents an additional source of information. To express that constraint of spatial homogeneity, we can use the Markov models. Moreover, the assumption used within the framework of fusion on the level of contours which stipulates that the distance between the matching snaxels of two consecutive slices must be close to the inter-slice distance e, can be supplemented by other information a priori, in particular on the form of contours to be obtained. It would be also interesting to evaluate, on the one hand, the contribution of all new parameter in fusion scheme, on the level of the pixels as on the level of contours, in terms of cost of calculation (this parameter can provoke in the global level fusion a combinative explosion) and in addition the real contribution of information which can bring this parameter, in order to determine precisely which are those which really make it possible to differentiate the initial assumptions. Indeed, it is not necessary to multiply the number of parameters to be combined. It would also be necessary to make a comparative study on the influence of the functions of masses on the results of fusion. Indeed, the value of the masses is obtained directly starting from the functions of masses. Any modification of these functions involves modifications on the decision-making, distributed or not. We showed that the model of global fusion proposed reveals the conflict between the sources, and thus makes it possible to question the reliability of at least one of
116
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117
providing MR sequences images for experiments and validation. Siemens and ‘‘le Conseil general du Nord pas de Calais’’ for the financing. Special thanks to J.M. Folzan.
References
Fig. 14. Examples of 3-D visualization of the spinal column.
these sources. It would be interesting to determine by a system of rules which source is not reliable and thus through a loop of feedback, to inform the system about the non-reliability of the defective source. Lastly, we limited ourselves in this work to only one MRI sequence, but the extension of our study on several incidences is absolutely possible. We can, in this case, tackle the problem directly starting from 3-D data MRI. Finally, it is possible, starting from contours obtained of each vertebra, from the global fusion processing, to calculate the relative 3-D deformations between the vertebrae and to give a representation in 3-D of the column (cf. Fig. 14). We have to carry out this pilot study on the basis of a certain number of simplifying assumptions namely: • The point obtained at the end of the global fusion have all be validated by the doctor. • Whatever the size of snake which approximates the vertebra, we consider that each snaxel have the same weighting for the computation of the center of gravity of the vertebra. • The computation of the center of gravity is not carried out compared with the volume of the vertebra but through the points of the different snakes. • The cubic splines use for the computation of axis principal of column is the one proposed by the software Matlab. • The angles computed between two unspecified vertebrae of the column are obtained by derivation of the spline function. Acknowledgements The authors express thanks to the Department of Radiology of the Institut Calot de Berck-sur Mer for
[1] H. Leclet, P.M. Delforge, Imagerie des scolioses: quand et comment, Rev. Imag. Med. (5) (1993) 773–782. [2] H. Leclet, P.Y. Plais, P.M. Delforge, Scoliose de l’enfant et de l’adolescent. Des lesions anatomiques a l’imagerie pratique, J. Francßais Radiol. (1992) 73–82. [3] H. Leclet, Artifact in MRI of the spine with or without implant, Eur. Spine J. (1994) 240–245. [6] P. Maroteaux, Les maladies osseuses de l’enfant, Flammarion, Paris, 1995. [7] K.F. Lai, Deformable contour: modeling, extraction, detection and classification, Ph.D. Thesis Wisconsin–Madison University, 1994. [8] T. Abe, Y. Matsuzawa, Multiple active contour models with application to region extraction, in: 15th International Conference on Pattern Recognition, Barcelona, vol. 4, 2000, p. 17. [9] A. Dromigny-Badin, Fusion d’images par la theorie de l’evidence en vue d’applications medicales et industrielles, Ph.D. Thesis, Institut Nationnal des Sciences Appliquees de Lyon, 1998. [11] R. Terzopoulos, A. Witkin, M. Kass, Symmetry-seeking models for 3D object reconstruction, Int. J. Comput. Vision 1 (3) (1987) 211–221. [12] M. Kunt, Traitement numerique des images, Presses Polytechniques et Universitaires Romandes, 1993. [13] J.P. Coquerez, S. Philippe, Analyse d’images: filtrage et segmentation, Masson, Paris, 1995. [14] G. Shafer, A Mathematical Theory of Evidence, Princeton University Press, Princeton, NJ, 1976. [15] A.P. Dempster, A class of random convex polytopes, Ann. Math. Statist. (1972) 250–272. [17] T. Denoeux, A k-nearest neighbor classification rule based on Dempster–Shafer theory, IEEE Trans. Syst., Mans Cybern. 25 (5, 25) (1995) 804–813. [18] I. Bloch, Information combination operators for data fusion : a comparative review with classification, IEEE Trans. Syst., Man Cybern. 26 (1) (1996) 52–67. [22] M. Brejl, M. Sonka, Medical Image segmentation, automated design of border detection criteria from examples, J. Electron. Imag. 8 (1) (1999) 54–64. [23] A. Godillon, Modelisation geometrique et cinematqiue 3D du rachis scoliotique pour la realisation d’un outil d’aide au medecin, Ph.D. Thesis, Universite de Valenciennes, 1991. [24] F. Dufour, Systeme de mesure 3D des deplacements vertebraux en cours de chirurgie, Ph.D. Thesis, Universite de Valenciennes, 1999. [25] S. Chusseau, Systeme d’aide a la decision pre-operatoire en chirurgie orthopedique de la scoliose, Ph.D. Thesis, Universite de Valenciennes, 1999. [26] T.N. Huynh, G. Maurais, Development of a vertebral endplate 3D reconstruction technique, IEEE. Trans. Med. Imag. 11 (2) (1997) 689–696. [27] G. Lefaix, A. Rameee, 3D modeling and deformation analysis of the vertebra with spherical harmonics, IEEE. EMBS 2 (1997) 422– 425. [28] M. Brejl, M. Sonka, Object localization and border detection criteria design in edge-based image segmentation: automated learning from examples, IEEE Trans. Med. Imag. 19 (10) (2000) 973– 983. [29] L. Gautier, A. Taleb-Ahmed, M. Rombaut, J.G. Postaire, H. Leclet, Proved segmentation from pictures sequence by evidence
A. Taleb-Ahmed, L. Gautier / Information Fusion 3 (2002) 103–117 theory, application MRI pictures, in: The 2nd International Conference on Information, Fusion’99, 1999. [30] L. Gautier, A. Taleb-Ahmed, M. Rombaut, J.G. Postaire, H. Leclet, Belief function in low level data fusion: application in RMI, in: The 3rd International Conference on Information, Fusion’2000, 2000, p. TUC5-3 (see also p. TUC5-8). [31] A. Taleb-Ahmed, L. Gautier, M. Rombaut, J.G. Postaire, H. Leclet, Information fusion: application for improve segmentation
117
from MRI sequence images, in: The 4th International Conference on Information, Fusion’2001, 2001. [34] D. Dubois, H. Prade, Possibility theory in information fusion, in: IEEE 3rd International Conference on Information Fusion, Fusion’2000, 2000, p. PS-6. [35] P. Smets, Data fusion in the transferable belief model, in: IEEE 3rd International Conference on Information Fusion, Fusion’2000, 2000, p. PS-21.