- Email: [email protected]
An intelligent generalized system for tissue classification on MR images by integrating qualitative medical knowledge
Biomedical Signal Processing and Control 6 (2011) 21–26
Contents lists available at ScienceDirect
Biomedical Signal Processing and Control journal homepage: www.elsevier.com/locate/bspc
An intelligent generalized system for tissue classification on MR images by integrating qualitative medical knowledge H. Kang a , A. Pinti a,∗ , A. Taleb-Ahmed a , X. Zeng b a b
Université de Valenciennes, LAMIH-CNRS, Le Mont Houy, 59313 Valenciennes, France ENSAIT, 2 allée Louise et Victor Champier, 59100 Roubaix, France
a r t i c l e
i n f o
Article history: Received 14 December 2009 Received in revised form 5 June 2010 Accepted 27 July 2010 Available online 21 August 2010 Keywords: Fuzzy C-Means MRI Tissue classification Image segmentation Medical knowledge
a b s t r a c t In the diagnosis using MRI images, image segmentation techniques play a key role. Existing segmentation methods are generally based on basic image features such as grey level and texture. However, these methods cannot effectively identify physical significance of segmented objects from image because these basic image features such as grey level cannot take into consideration specialized medical knowledge, which is important when doctors study them manually using their own vision and experience. To deal with this problem, many tissue classification systems have been developed by integrating specific medical knowledge. All of these systems focus on specific applications and cannot be normalized and structured. Therefore, adaption of such systems to other contexts is rather difficult. In this paper, we propose an intelligent generalized tissue classification system which combines both the Fuzzy C-Means algorithm and the qualitative medical knowledge on geometric properties of different tissues. In this system, a general geometric model is proposed for formalizing non-structured and non-normalized medical knowledge from various medical images. This system has been successfully applied to the classification of human thigh, crus, arm, forearm, and brain in MRI images. © 2010 Elsevier Ltd. All rights reserved.
1. Introduction Magnetic Resonance Imaging (MRI) allows us to explore living organs in a non-invasive way. The analysis of MRI images can be used in the quantification of human body composition, such as the quantification of muscle/fat ratio [1], and assessment for variation of body fat content [2]. All this work has important medical significance in human nutrition and muscle physiology, in the study of pathologic consequences of obesity, and in the study of diseases of muscle [3]. In the diagnosis using MRI, image segmentation techniques play a key role. It aims at partitioning an image into a number of non-overlapped and constituent regions which are homogeneous with respect to some basic image characteristics such as grey level or texture [4]. Many clustering methods such as Fuzzy C-means [4–12] have been proposed for image segmentation. However, these segmentation methods cannot identify physical significance of segmented objects from image. This is because basic image features that they used cannot take into consideration specialized medical knowledge, which is crucial when doctors study them manually using their own vision and experience. Therefore, it is
∗ Corresponding author. Tel.: +33 327511429; fax: +33 327511316. E-mail addresses: [email protected] (A. Pinti), [email protected] (X. Zeng). 1746-8094/$ – see front matter © 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.bspc.2010.07.005
necessary to integrate our a priori knowledge on medical image analysis in order to interpret segmented objects or classes. In this context, various knowledge-based tissue classification systems have been proposed, such as the analysis for brain tissues [13–16], the analysis for bone composition [17], and the diagnosis for chest [18]. In [13], the authors proposed a knowledge-based approach for tissue labelling of 2D MRI images of human brain. The system consists of two components: a clustering algorithm and an expert system. MRI images are initially segmented by a FCM-based clustering algorithm [19], and then the expert system locates a landmark tissue or cluster and analyzes it by matching it with a model or searching in it for an expected feature. The location and analysis of landmark tissue are repeated until a tumour is found or all tissues are labelled. The system accurately labels all tissues in normal slices and provides a partial labelling of the tissues in abnormal slices. In [17], the authors proposed an image analysis system for quantitative bone image analysis which uses the spatial knowledge such as the relationship between canals and osteons, expected shape of osteons, and the content of osteons. The use of knowledge about bone cross-section characteristics and microradiographs enable us to obtain more accurate and more consistent quantitative analysis results with respect to the algorithms using simple clustering techniques for segmentation. In [18], the authors developed an architecture for automatic segmentation of chest CT data that isolates the right and left lungs, central tracheobronchial tree, chest wall, and mediastinum. Anatomical knowledge includ-
22
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
The middle level is the Rules Generation. It includes the Geometric Models for formalizing medical knowledge, and the rules for splitting each class obtained by Fuzzy C-Means algorithm into tissues and giving significance to each class. The high level is the Rules Control Strategy Generation. We propose two principles to define the priorities for these rules in order to optimize their application. The rules with higher priorities should be applied earlier than those with lower priorities. Using this system, we can effectively label tissues and give significance to segmented classes from image which can be used in further medical analysis. 2. The proposed system 2.1. Low level: Knowledge Acquisition The low level includes a Knowledge Base module and an Interface module. The Knowledge Base is composed of a number of features used for classifying each tissue. The knowledge is acquired through three channels [20]. The first is a documented discussion with a doctor which aides in deciding what are the important anatomical features that need to be modelled. The second is the use of anatomical text books and neuro-anatomical atlases. And the third is an informal discussion with neurologists and neuroradiologists. These channels allow to record any anomalies in the representation of the anatomy of tissues. The Interface module is an online questionnaire. The experts are invited to fill this questionnaire or we search for the answers from the anatomical text books. In this system, the knowledge is expressed using linguistic description. From all the answers collected in the questionnaire, we concentrate on six features, of which the Knowledge Base module is composed: (a) relative positions between tissues. (b) Neighboring relation of each tissue. (c) Ranking order of all tissues according to their areas. (d) Number of connected components of certain tissues. (e) Tissues having similar grey level. (f) Shape of certain tissue. 2.2. Middle level: Rules Generation Fig. 1. The three-level structure for tissue classification with knowledge integration.
ing expected volume, shape, relative position provides feature constraints that guide the segmentation process. Knowledge is represented at a high level using an explicit anatomical model. The model is stored in a frame-based semantic network and anatomical variability is integrated using fuzzy sets. The use of knowledge offers improvements over non-guided segmentation in terms of automation, robustness and variety of anatomical structures that can be identified. Nevertheless, all of these systems focus on specific applications and are not normalized and structured. So they lack of certainty and precision when being applied in other contexts. Therefore, a more generalized automatic tissue classification system is needed for integrating medical knowledge in a flexible way and being adapted to different applications.In this paper, we propose an intelligent generalized tissue classification system which combines both the Fuzzy C-Means algorithm [6] and the qualitative medical knowledge on geometric properties of different tissues for further improving obtained segmentation results. We choose a cluster-based method instead of an edge-based method for image segmentation. This is because cluster analysis can offer important domain knowledge, and clusters have better semantic meanings than edges regarding knowledge-based analysis [13]. There are three levels in this system (see Fig. 1). The low level is the Knowledge Acquisition, including a Knowledge Base module and an Interface module. The user friendly interface is constructed so that medical knowledge can be integrated into this system in an interactive way.
There are two parts in middle level, the geometric models and the rules. In the geometric models, we formalize the above six features and then store them in a predefined data structure, i.e. a 2-dimensional array. Based on these formalized features, we set up in the rules module one rule for each feature in order to split each class obtained by Fuzzy C-Means algorithm [6] into tissues and giving significance to each class. In the geometric models, assuming there exist n tissues on a specific medical image denoted as T1 , . . ., Tn , we formalize the knowledge on geometric properties of tissues as follows. (a) The set of relative positions between tissues.
RP = {A, Ti , Tj |i, j ∈ {1, . . . , n} and Ti = / Tj } where A is an attribute between two tissues and A ∈ {above, below, left, right, above left, below left, above right, below right, inside, outside}. For example, “” means that Tissue 1 is above Tissue 2. (b) The matrix of neighbors for tissues. It is a n × n matrix denoted as NT = {ntij }. i, j ∈ {1, . . ., n}. ntij is the element in the position (i, j) of the matrix. If ntij = 1, then Ti and Tj are neighbors. If ntij = 0, then Ti and Tj are not neighbors.
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
23
(c) The list of tissues with one connected component ranked in the decreasing order of their areas.
classified, the value in the corresponding position in this array is “1”.
LTA = (Ti1 , Ti2 , . . ., Tiq ) where Area(Tij ) > Area(Tik ) for j < k, i, j ∈ {1, . . ., n}. Tiq is the qth tissue in the ith list.
(i) Already classified tissues.
(d) The set of numbers of connected components in tissues. NC = { | i ∈ {1, . . ., n}}where nci is the number of connected components in tissue Ti . (e) The set of groups of tissues having similar grey levels. GT = {GTi |i ∈ {1, . . ., p}| p is number of groups of grey levels}; and Grey level(Tj ) = Grey level(Tk ) for any two tissues Tj , Tk ∈ GTi (j = / k). GTi is the ith group of tissues having similar grey levels. (f) The set of shapes for the tissues with one connected component ST = { |i ∈ {1, . . ., n} and Si ∈ {triangle, circle, rectangle, etc.} After having formalized all the six features, we store them in a predefined data structure, i.e. a 2-dimensional array. So we have six corresponding arrays: (1) RelativePosition. (2) NeighboringRelation. (3) RankingOrderofArea. (4) NumberofConnectedComponents. (5) SimilarGreyLevel. (6) ShapeofTissue. We now take the feature “The set of groups of tissues having similar grey levels” as an example to show how we store them. Name of array: SimilarGreyLevel. The data structure is a n × 2 array in the following form: Tissue 1
(g) Segmentation result by FCM algorithm. Name of array: SegmentationResult. The data structure is a n × n array, with the same size as the original image. Before we run the FCM algorithm, we predefine the number of classes for image segmentation. When the segmentation is finished, each pixel in the original image is labeled a specific class. The array “SegmentationResult” is just used to store the corresponding class label for each pixel in the original image. We assign a specific value to each class. For example, we assign the value “1” to class 1, “2” to class 2, etc. So each pixel in the original image will be assigned a specific value according to its class label, this value is stored in the corresponding position in array “SegmentationResult”. For example, the pixel in the position (3,5) of the original image belongs to class 2, so “2” is stored in the position (3,5) of the array “SegmentationResult”. (h) Classification state for each tissue. Name of array: ClassificationState. The data structure is a 1 × n array, n is the number of tissues in the original image. It is in the following form: State of Tissue 2
State of Tissue 3
(Rule 1) Background extraction. It is easy to extract and label the background from an image because it is an object with only one connected component linking all the four bounds of the image.
Tissue 2
which means Tissue 1 and Tissue 2 have similar grey levels. There are three more arrays to store the “segmentation result by FCM algorithm”, the “classification state for each tissue”, and the “already classified tissues”, respectively.
State of Tissue 1
Name of array: ClassifiedTissues. The data structure is a n × n array, with the same size as the original image. We initialize each element in this array as “255”. Before classifying tissues, we assign a specific value to each tissue in order to denote them. For example, we assign the value “1” to muscle, “2” to adipose tissue, etc. When a specific tissue is classified, all the pixels belonging to this tissue are assigned the corresponding label value. Then, for all these pixels, their label value is stored in corresponding positions in this array. When applying the Fuzzy C-Means algorithm for image segmentation, we often obtain several tissues in one segmented class and several classes corresponding to one tissue. This is because the feature used in this algorithm is based on grey levels. A knowledge guided intelligent control procedure is used for splitting obtained classes of pixels denoted as C1 , C2 , . . ., Cp into tissues. The predefined number of classes for the Fuzzy C-Means algorithm is p (we define this number by our a priori knowledge on images), i.e. the number of groups each including tissues of similar grey levels. In this case, we obtain classes each having similar grey levels. For splitting each class into tissues and giving significance to each class, we define the following rules according to the previous geometric models. These rules permit to label tissues from classes.
...
State of Tissue n
For example, if Tissue n is not classified yet, the value in the corresponding position in this array is “0”. If Tissue n is already
(Rule 2) Relative positions of tissues. For any labelled tissue Ti , if there is only one unlabelled tissue Tj so that A, Ti , Tj ∈ RP (A can be any relation attribute), and if Tj does not have similar grey levels with any other tissues, then we can find the class corresponding to Tj by comparing positions of pixels and label it. (Rule 3) Neighbors of tissues. For each unlabelled tissue Ti , if its neighboring tissues have been all labelled, then we can find and label Ti using connectivity analysis related to these neighboring tissues. (Rule 4) The tissues with one connected component ranked in the decreasing order of their areas. If any group of similar grey levels GTi has only one unlabelled tissue, then we can rank all the unlabelled classes with one connected component according to their areas and make equivalence between them and all ranked unlabelled tissues with one connected component. (Rule 5) Numbers of connected components in tissues. Given a specific number of connected objects, if the number of unlabelled tissues is 1, and if the number of unlabelled classes is 1, then this class is equivalent to the related tissue. (Rule 6) Groups of tissues having similar grey levels.
24
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
For each group of tissues GTi corresponding to one class obtained by the Fuzzy C-Means algorithm, if only one tissue in it has not been labelled, then we can easily separate and label it by deducing the pixels corresponding to labelled tissues from this class. (Rule 7) Shapes of the tissues with one connected component. For each shape, if the number of unlabelled tissues in ST is 1, and if the number of corresponding unlabelled classes with one connected component is 1, then this class is equivalent to the related tissue. We now take Rule 6 as an example to show how the rules are applied. The system first checks the array “SimilarGreyLevel”. If there is a group of tissues having similar grey levels, then the system will check the array “ClassificationState”. If only one tissue in this group has not been labelled, then the system will check the array “SegmentationResult” and the array “ClassifiedTissues” to label this tissue.
Fig. 2. One MRI image of human crus.
2.3. High level: Rules Control Strategy Generation When we run our system, we first use the clustering algorithm to obtain image segmentation. Next, we can perform the classification by iteratively applying the above seven rules. In order to optimize the application of these rules, we define three priorities for them. The rules with higher priorities will be applied before those with lower priorities. Two principles are proposed to define the priorities. (a) The priority of a rule can be defined according to the number of the corresponding premises. Rules with fewer number of premises have higher priority than those with bigger number of premises. Then, we have: (1) Rule 1 has the highest priority, because it does not have any premise, (2) Rules 2, 5, and 7 have the lowest priority, because each of them has two premises, (3) the priority of Rules 3, 4, 6 is medium, because each of them has one premise. (b) The priority of a rule can be defined according to the number of arrays involved. Rules with fewer number of arrays involved have higher priority than those with bigger number of arrays involved. For example, for applying the Rule 6, four array need to be checked, namely, “SimilarGreyLevel”, “ClassificationState”, “SegmentationResult”, “ClassifiedTissues”. So we have: (1) Rule 1 has the highest priority, (2) Rules 2, 3, and 4 have the lowest priority, and (3) the priority of Rules 5, 6, and 7 is medium.
Fig. 3. Histogram of a MRI image of crus.
cortical bone and background belong to one class C1 , adipose tissue and spongy bone to another class C2 , and muscle corresponds to C3 . We use the first principle to define the priorities of rules. So we first use Rule 1 which is with the highest priority to separate the background from C1 . Then we recursively apply the Rule 3, Rule 4 and Rule 6 which are with medium priority for separating new tissues from the obtained classes. We use Rule 6 (groups of tissues having similar grey levels) to label cortical bone from the remaining pixels in C1 with respect to the background. Next, we apply the Rule 3 (neighbors of tissues) with respect to the cortical bone for determining pixels of spongy bone from C2 . The remaining pixels of
3. Validation of our system Our system has been successfully applied to the tissue classification of human arm, forearm, thigh, crus and brain. We take the classification of crus as an example to show this system. Fig. 2 shows a MRI image of crus. It is in axial plane, T1-weighted, obtained from a system of Philips Intera of 1.5 T. The size of voxel is 0.93 mm × 0.93 mm × 1.0 mm, and the coding is 12 bits. In addition, the signal has been analyzed by spin echo technique in order to minimize artifacts. We can see that there are four tissues and one background [21]. The histogram shows that grey level ranges overlap for cortical bone and background, and for spongy bone and adipose tissue (see Fig. 3). Therefore, using Fuzzy C-Means algorithm with 5 classes results in bad segmentation (Fig. 4). When applying the proposed system, we run the Fuzzy C-Means algorithm with 3 classes because only three grey levels can be observed (p = 3). Then, we obtain from the clustering result that
Fig. 4. Segmentation by Fuzzy C-Means with five classes.
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
25
Fig. 5. Classification result of crus. (a) The classification by a medical expert. (b) The classification by system A. (c) The classification by the proposed system.
C2 can be then identified as adipose tissue using Rule 6 with respect to the spongy bone. Using Rule 3 with respect to cortical bone and adipose tissue, we can label C3 as muscle. Using our proposed intelligent system, we effectively separate the four tissues and the background on the image of crus. Then we reconstruct a synthetic image using the obtained five classes with following grey levels: background: 20; adipose tissue: 250; spongy bone: 150; cortical bone: 200; muscle: 60 (Fig. 5c). We can see the quality in Fig. 5c is much higher than the one in Fig. 4. Fig. 5b shows the synthetic image generated from the identified tissues obtained by using a specific tissue classification system A [3]. In this system, the original image of thigh is first segmented into three classes by a FCM-based clustering algorithm [22]. By using the knowledge on the ranking order of tissue volumes, only three tissues (muscle, adipose tissue and spongy bone) can be identified. However, the integration of medical knowledge in this system is a manual procedure. But the proposed system is completely automatic. Next, we do quantitative comparison of the classification by the system A, our system and a reference classification (see Fig. 5a) provided by a medical expert. The Performance Criterion for the tissue Tj related to the ith method (PCTij ) is given as follows: PCTij =
Aij ∩ Arefj Aij ∪ Arefj
(1)
where Aij is the set of pixels belonging to the jth class obtained by the ith method, and Arefj is the set of pixels belonging to jth class in the reference classification result. For all identified tissues, the Performance Criterion for evaluating the ith method (PCi ) is given by:
NuT j=1 (Aij ∩ Arefj ) PCi = NuT NuT ( j=1 Aij ) ∪ ( j=1 Arefj )
(2)
Table 1 Quantitative comparison of classification.
The system A The proposed
Adipose tissue
Cortical bone
Spongy bone
Muscle
86.91% 93.18%
× 81.96%
82.98% 88.07%
97.07% 98.05%
Table 2 Quantitative comparison of overall classification performance. Overall performance (PCi )
The system A
The proposed
92.79%
98.06%
where NuT is the number of tissues in the image. Tables 1 and 2 show the corresponding quantitative comparison. 4. Conclusion In this paper, we present a generalized automatic system for tissue classification which can be adapted to different parts of human body. In this system, a general geometric model is proposed for formalizing non-structured and non-normalized medical knowledge from various medical images. An intelligent control procedure is developed for transforming the medical knowledge into several rules in order to improve the quality of the segmentation by Fuzzy C-Means. Two principles are proposed to define the priorities for these rules in order to optimize their application. One issue needs to be considered is that the geometric features used in this system may be sensitive to geometric transformations like rotation or flipping. We propose to normalize images before we run this system. In the future, we will work on the quantification of the muscle/fat ratio, assessment of the muscle/fat temporal variation, and measurement of the volume of muscle, fat and bone in human legs based on the obtained results. We will also validate our system on
26
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
other parts of human body. Furthermore, we will make more efforts for exploring other common tissue features and add new rules to our system so that our system can be more generalized. We will also endeavor to apply new methods to formalize the tissue geometric feature, such as the Fuzzy Logic. In addition, we will try to propose new principles to define the priorities of rules in order to optimize their applications. Conflict of interest None. Acknowledgements The present research work has been supported by International Campus on Safety and Intermodality in Transportation, the Nord-Pas-de-Calais Region France, the European Community, the Regional Delegation for Research and Technology, the Ministry of Higher Education and Research, and the National Center for Scientific Research. The authors gratefully acknowledge the support of these institutions. References [1] A. Colin, E. Erbland, C. Datin, J.Y. Boire, A. Veyre, M. Zanca, Automatic muscle/fat quantification on MR images, in: Proceedings of the 17th International Conference of the IEEE Engineering in Medicine and Biology, 1, 1995, pp. 479–480. [2] G.Z. Yang, S. Myerson, F. Chabat, D.J. Pennell, D.N. Firmin, Automatic MRI adipose tissue mapping using overlapping mosaics, magnetic resonance materials in physics, Biology and Medicine 14 (2002) 39–44. [3] V. Barra, J.V. Boire, Segmentation of fat and muscle from MR images of the thigh by a possibilistic clustering algorithm, Computer Methods and Programs in Biomedicine 68 (2002) 185–193. [4] D. Zhang, S. Chen, A novel kernelized fuzzy C-means algorithm with application in medical image segmentation, Artificial Intelligence in Medicine 32 (2004) 37–50. [5] V. Roullier, C. Cavaro-Menard, G. Calmon, C. Aube, Fuzzy algorithms: application to adipose tissue quantification on MR images, Biomedical Signal Processing and Control 2 (2007) 239–247. [6] M.N. Ahmed, S.M. Yamany, N. Mohamed, A.A. Farag, T. Moriarty, A modified fuzzy c-means algorithm for bias field estimation and segmentation of MRI data, IEEE Transactions on Medical Imaging 21 (3) (2002) 193–200.
[7] S. Chen, D. Zhang, Robust image segmentation using FCM with spatial constraints based on new kernel-induced distance measure, IEEE transactions on system, Man, and Cybernetics-Part B: Cybernetics 34 (4) (2004) 1907–1916. [8] W. Hesheng, F. Baowei, A modified fuzzy C-means classification method using a multiscale diffusion filtering scheme, Medical Image Analysis 13 (2009) 193–202. [9] W. Jianzhong, K. Jun, L. Yinghua, Q. Miao, Z. Baoxue, A modified FCM algorithm for MRI brain image segmentation using both local and non-local spatial constraints, Computerized Medical Imaging and Graphics 32 (2008) 685–698. [10] H. Kang, Contribution to automatic corporal tissue classification by integrating qualitative medical knowledge: application to the analysis of musculo skeletal diseases and disabilities from MRI sequences, Ph.D. Thesis, Université de Valenciennes et du Hainaut-Cambrésis, France, 2009. [11] X. Li, T. Zhang, Z. Qu, Image segmentation using fuzzy clustering with spatial constraints based on markov random field via Bayesian theory, special section on Signal Processing for Audio and Visual Systems and its Implementations, IEICE Transactions Fundamentals E91–A (3) (2008) 723–728. [12] J. Valente de Oliveira, W. Pedrycz, Advances in Fuzzy Clustering and its Applications, John Wiley and Sons Ltd., England, 2007. [13] C. Li, D.B. Goldgof, L.O. Hall, Knowledge-based classification and tissue labeling of MR images of human brain, IEEE Transactions on Medical Imaging 12 (4) (1993) 740–750. [14] H. Li, R. Deklerck, J. Cornelis, Integration of multiple knowledge sources in a system for brain CT-scan interpretation based on the blackboard model, in: Proceedings of the 10th Conference on Artificial Intelligence for Applications, 1994, pp. 336–343. [15] H. Li, R. Deklerck, B.D. Cuyper, A. Hermanus, E. Nysser, J. Cornelis, Object recognition in brain CT-scans: knowledge-based fusion of data from multiple feature extractors, IEEE Transactions on Medical Imaging 14 (2) (1995) 212–229. [16] C. Wenhung, C. Youyin, C. Chienwen, L. Shenghuang, S. Yenyui, T. Siny, Improving segmentation accuracy for magnetic resonance imaging using a boosted decision tree, Journal of Neuroscience Methods 175 (2008) 206–217. [17] Z.Q. Liu, H.L. Liew, J.G. Clement, C.D.L. Thomas, Bone image segmentation, IEEE Transactions on Biomedical Engineering 46 (5) (1999) 565–570. [18] M.S. Brown, M.F. McNitt-Gray, N.J. Mankovich, J.G. Goldin, J. Hiller, L.S. Wilson, D.R. Aberle, Method for segmenting chest CT image data using an anatomical model: preliminary results, IEEE Transactions on Medical Imaging 16 (6) (1997) 828–839. [19] R.L. Cannon, J.V. Dave, J.C. Bezdek, Efficient implementation of the fuzzy c-means clustering algorithms, IEEE Transactions on Pattern Analysis and Machine Intelligence 8 (2) (1986) 248–255. [20] G.P. Robinson, A.C.F. Colchester, L.D. Griffin, Model-based recognition of anatomical objects from medical images, Image and Vision Computing 12 (8) (1994) 499–507. [21] P. Hédoux, A. Pinti, A. Taleb-Ahmed, Pixel classification method using statistical attributes of distributed imagelets, Journal Européen des Systèmes Automatisés 43 (3) (2009) 337–350. [22] R. Krishnapuram, J.M. Keller, A possibilistic approach to clustering, IEEE Transactions on Fuzzy Systems 1 (2) (1993) 98–110.
Contents lists available at ScienceDirect
Biomedical Signal Processing and Control journal homepage: www.elsevier.com/locate/bspc
An intelligent generalized system for tissue classification on MR images by integrating qualitative medical knowledge H. Kang a , A. Pinti a,∗ , A. Taleb-Ahmed a , X. Zeng b a b
Université de Valenciennes, LAMIH-CNRS, Le Mont Houy, 59313 Valenciennes, France ENSAIT, 2 allée Louise et Victor Champier, 59100 Roubaix, France
a r t i c l e
i n f o
Article history: Received 14 December 2009 Received in revised form 5 June 2010 Accepted 27 July 2010 Available online 21 August 2010 Keywords: Fuzzy C-Means MRI Tissue classification Image segmentation Medical knowledge
a b s t r a c t In the diagnosis using MRI images, image segmentation techniques play a key role. Existing segmentation methods are generally based on basic image features such as grey level and texture. However, these methods cannot effectively identify physical significance of segmented objects from image because these basic image features such as grey level cannot take into consideration specialized medical knowledge, which is important when doctors study them manually using their own vision and experience. To deal with this problem, many tissue classification systems have been developed by integrating specific medical knowledge. All of these systems focus on specific applications and cannot be normalized and structured. Therefore, adaption of such systems to other contexts is rather difficult. In this paper, we propose an intelligent generalized tissue classification system which combines both the Fuzzy C-Means algorithm and the qualitative medical knowledge on geometric properties of different tissues. In this system, a general geometric model is proposed for formalizing non-structured and non-normalized medical knowledge from various medical images. This system has been successfully applied to the classification of human thigh, crus, arm, forearm, and brain in MRI images. © 2010 Elsevier Ltd. All rights reserved.
1. Introduction Magnetic Resonance Imaging (MRI) allows us to explore living organs in a non-invasive way. The analysis of MRI images can be used in the quantification of human body composition, such as the quantification of muscle/fat ratio [1], and assessment for variation of body fat content [2]. All this work has important medical significance in human nutrition and muscle physiology, in the study of pathologic consequences of obesity, and in the study of diseases of muscle [3]. In the diagnosis using MRI, image segmentation techniques play a key role. It aims at partitioning an image into a number of non-overlapped and constituent regions which are homogeneous with respect to some basic image characteristics such as grey level or texture [4]. Many clustering methods such as Fuzzy C-means [4–12] have been proposed for image segmentation. However, these segmentation methods cannot identify physical significance of segmented objects from image. This is because basic image features that they used cannot take into consideration specialized medical knowledge, which is crucial when doctors study them manually using their own vision and experience. Therefore, it is
∗ Corresponding author. Tel.: +33 327511429; fax: +33 327511316. E-mail addresses: [email protected] (A. Pinti), [email protected] (X. Zeng). 1746-8094/$ – see front matter © 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.bspc.2010.07.005
necessary to integrate our a priori knowledge on medical image analysis in order to interpret segmented objects or classes. In this context, various knowledge-based tissue classification systems have been proposed, such as the analysis for brain tissues [13–16], the analysis for bone composition [17], and the diagnosis for chest [18]. In [13], the authors proposed a knowledge-based approach for tissue labelling of 2D MRI images of human brain. The system consists of two components: a clustering algorithm and an expert system. MRI images are initially segmented by a FCM-based clustering algorithm [19], and then the expert system locates a landmark tissue or cluster and analyzes it by matching it with a model or searching in it for an expected feature. The location and analysis of landmark tissue are repeated until a tumour is found or all tissues are labelled. The system accurately labels all tissues in normal slices and provides a partial labelling of the tissues in abnormal slices. In [17], the authors proposed an image analysis system for quantitative bone image analysis which uses the spatial knowledge such as the relationship between canals and osteons, expected shape of osteons, and the content of osteons. The use of knowledge about bone cross-section characteristics and microradiographs enable us to obtain more accurate and more consistent quantitative analysis results with respect to the algorithms using simple clustering techniques for segmentation. In [18], the authors developed an architecture for automatic segmentation of chest CT data that isolates the right and left lungs, central tracheobronchial tree, chest wall, and mediastinum. Anatomical knowledge includ-
22
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
The middle level is the Rules Generation. It includes the Geometric Models for formalizing medical knowledge, and the rules for splitting each class obtained by Fuzzy C-Means algorithm into tissues and giving significance to each class. The high level is the Rules Control Strategy Generation. We propose two principles to define the priorities for these rules in order to optimize their application. The rules with higher priorities should be applied earlier than those with lower priorities. Using this system, we can effectively label tissues and give significance to segmented classes from image which can be used in further medical analysis. 2. The proposed system 2.1. Low level: Knowledge Acquisition The low level includes a Knowledge Base module and an Interface module. The Knowledge Base is composed of a number of features used for classifying each tissue. The knowledge is acquired through three channels [20]. The first is a documented discussion with a doctor which aides in deciding what are the important anatomical features that need to be modelled. The second is the use of anatomical text books and neuro-anatomical atlases. And the third is an informal discussion with neurologists and neuroradiologists. These channels allow to record any anomalies in the representation of the anatomy of tissues. The Interface module is an online questionnaire. The experts are invited to fill this questionnaire or we search for the answers from the anatomical text books. In this system, the knowledge is expressed using linguistic description. From all the answers collected in the questionnaire, we concentrate on six features, of which the Knowledge Base module is composed: (a) relative positions between tissues. (b) Neighboring relation of each tissue. (c) Ranking order of all tissues according to their areas. (d) Number of connected components of certain tissues. (e) Tissues having similar grey level. (f) Shape of certain tissue. 2.2. Middle level: Rules Generation Fig. 1. The three-level structure for tissue classification with knowledge integration.
ing expected volume, shape, relative position provides feature constraints that guide the segmentation process. Knowledge is represented at a high level using an explicit anatomical model. The model is stored in a frame-based semantic network and anatomical variability is integrated using fuzzy sets. The use of knowledge offers improvements over non-guided segmentation in terms of automation, robustness and variety of anatomical structures that can be identified. Nevertheless, all of these systems focus on specific applications and are not normalized and structured. So they lack of certainty and precision when being applied in other contexts. Therefore, a more generalized automatic tissue classification system is needed for integrating medical knowledge in a flexible way and being adapted to different applications.In this paper, we propose an intelligent generalized tissue classification system which combines both the Fuzzy C-Means algorithm [6] and the qualitative medical knowledge on geometric properties of different tissues for further improving obtained segmentation results. We choose a cluster-based method instead of an edge-based method for image segmentation. This is because cluster analysis can offer important domain knowledge, and clusters have better semantic meanings than edges regarding knowledge-based analysis [13]. There are three levels in this system (see Fig. 1). The low level is the Knowledge Acquisition, including a Knowledge Base module and an Interface module. The user friendly interface is constructed so that medical knowledge can be integrated into this system in an interactive way.
There are two parts in middle level, the geometric models and the rules. In the geometric models, we formalize the above six features and then store them in a predefined data structure, i.e. a 2-dimensional array. Based on these formalized features, we set up in the rules module one rule for each feature in order to split each class obtained by Fuzzy C-Means algorithm [6] into tissues and giving significance to each class. In the geometric models, assuming there exist n tissues on a specific medical image denoted as T1 , . . ., Tn , we formalize the knowledge on geometric properties of tissues as follows. (a) The set of relative positions between tissues.
RP = {A, Ti , Tj |i, j ∈ {1, . . . , n} and Ti = / Tj } where A is an attribute between two tissues and A ∈ {above, below, left, right, above left, below left, above right, below right, inside, outside}. For example, “
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
23
(c) The list of tissues with one connected component ranked in the decreasing order of their areas.
classified, the value in the corresponding position in this array is “1”.
LTA = (Ti1 , Ti2 , . . ., Tiq ) where Area(Tij ) > Area(Tik ) for j < k, i, j ∈ {1, . . ., n}. Tiq is the qth tissue in the ith list.
(i) Already classified tissues.
(d) The set of numbers of connected components in tissues. NC = {
(g) Segmentation result by FCM algorithm. Name of array: SegmentationResult. The data structure is a n × n array, with the same size as the original image. Before we run the FCM algorithm, we predefine the number of classes for image segmentation. When the segmentation is finished, each pixel in the original image is labeled a specific class. The array “SegmentationResult” is just used to store the corresponding class label for each pixel in the original image. We assign a specific value to each class. For example, we assign the value “1” to class 1, “2” to class 2, etc. So each pixel in the original image will be assigned a specific value according to its class label, this value is stored in the corresponding position in array “SegmentationResult”. For example, the pixel in the position (3,5) of the original image belongs to class 2, so “2” is stored in the position (3,5) of the array “SegmentationResult”. (h) Classification state for each tissue. Name of array: ClassificationState. The data structure is a 1 × n array, n is the number of tissues in the original image. It is in the following form: State of Tissue 2
State of Tissue 3
(Rule 1) Background extraction. It is easy to extract and label the background from an image because it is an object with only one connected component linking all the four bounds of the image.
Tissue 2
which means Tissue 1 and Tissue 2 have similar grey levels. There are three more arrays to store the “segmentation result by FCM algorithm”, the “classification state for each tissue”, and the “already classified tissues”, respectively.
State of Tissue 1
Name of array: ClassifiedTissues. The data structure is a n × n array, with the same size as the original image. We initialize each element in this array as “255”. Before classifying tissues, we assign a specific value to each tissue in order to denote them. For example, we assign the value “1” to muscle, “2” to adipose tissue, etc. When a specific tissue is classified, all the pixels belonging to this tissue are assigned the corresponding label value. Then, for all these pixels, their label value is stored in corresponding positions in this array. When applying the Fuzzy C-Means algorithm for image segmentation, we often obtain several tissues in one segmented class and several classes corresponding to one tissue. This is because the feature used in this algorithm is based on grey levels. A knowledge guided intelligent control procedure is used for splitting obtained classes of pixels denoted as C1 , C2 , . . ., Cp into tissues. The predefined number of classes for the Fuzzy C-Means algorithm is p (we define this number by our a priori knowledge on images), i.e. the number of groups each including tissues of similar grey levels. In this case, we obtain classes each having similar grey levels. For splitting each class into tissues and giving significance to each class, we define the following rules according to the previous geometric models. These rules permit to label tissues from classes.
...
State of Tissue n
For example, if Tissue n is not classified yet, the value in the corresponding position in this array is “0”. If Tissue n is already
(Rule 2) Relative positions of tissues. For any labelled tissue Ti , if there is only one unlabelled tissue Tj so that A, Ti , Tj ∈ RP (A can be any relation attribute), and if Tj does not have similar grey levels with any other tissues, then we can find the class corresponding to Tj by comparing positions of pixels and label it. (Rule 3) Neighbors of tissues. For each unlabelled tissue Ti , if its neighboring tissues have been all labelled, then we can find and label Ti using connectivity analysis related to these neighboring tissues. (Rule 4) The tissues with one connected component ranked in the decreasing order of their areas. If any group of similar grey levels GTi has only one unlabelled tissue, then we can rank all the unlabelled classes with one connected component according to their areas and make equivalence between them and all ranked unlabelled tissues with one connected component. (Rule 5) Numbers of connected components in tissues. Given a specific number of connected objects, if the number of unlabelled tissues is 1, and if the number of unlabelled classes is 1, then this class is equivalent to the related tissue. (Rule 6) Groups of tissues having similar grey levels.
24
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
For each group of tissues GTi corresponding to one class obtained by the Fuzzy C-Means algorithm, if only one tissue in it has not been labelled, then we can easily separate and label it by deducing the pixels corresponding to labelled tissues from this class. (Rule 7) Shapes of the tissues with one connected component. For each shape, if the number of unlabelled tissues in ST is 1, and if the number of corresponding unlabelled classes with one connected component is 1, then this class is equivalent to the related tissue. We now take Rule 6 as an example to show how the rules are applied. The system first checks the array “SimilarGreyLevel”. If there is a group of tissues having similar grey levels, then the system will check the array “ClassificationState”. If only one tissue in this group has not been labelled, then the system will check the array “SegmentationResult” and the array “ClassifiedTissues” to label this tissue.
Fig. 2. One MRI image of human crus.
2.3. High level: Rules Control Strategy Generation When we run our system, we first use the clustering algorithm to obtain image segmentation. Next, we can perform the classification by iteratively applying the above seven rules. In order to optimize the application of these rules, we define three priorities for them. The rules with higher priorities will be applied before those with lower priorities. Two principles are proposed to define the priorities. (a) The priority of a rule can be defined according to the number of the corresponding premises. Rules with fewer number of premises have higher priority than those with bigger number of premises. Then, we have: (1) Rule 1 has the highest priority, because it does not have any premise, (2) Rules 2, 5, and 7 have the lowest priority, because each of them has two premises, (3) the priority of Rules 3, 4, 6 is medium, because each of them has one premise. (b) The priority of a rule can be defined according to the number of arrays involved. Rules with fewer number of arrays involved have higher priority than those with bigger number of arrays involved. For example, for applying the Rule 6, four array need to be checked, namely, “SimilarGreyLevel”, “ClassificationState”, “SegmentationResult”, “ClassifiedTissues”. So we have: (1) Rule 1 has the highest priority, (2) Rules 2, 3, and 4 have the lowest priority, and (3) the priority of Rules 5, 6, and 7 is medium.
Fig. 3. Histogram of a MRI image of crus.
cortical bone and background belong to one class C1 , adipose tissue and spongy bone to another class C2 , and muscle corresponds to C3 . We use the first principle to define the priorities of rules. So we first use Rule 1 which is with the highest priority to separate the background from C1 . Then we recursively apply the Rule 3, Rule 4 and Rule 6 which are with medium priority for separating new tissues from the obtained classes. We use Rule 6 (groups of tissues having similar grey levels) to label cortical bone from the remaining pixels in C1 with respect to the background. Next, we apply the Rule 3 (neighbors of tissues) with respect to the cortical bone for determining pixels of spongy bone from C2 . The remaining pixels of
3. Validation of our system Our system has been successfully applied to the tissue classification of human arm, forearm, thigh, crus and brain. We take the classification of crus as an example to show this system. Fig. 2 shows a MRI image of crus. It is in axial plane, T1-weighted, obtained from a system of Philips Intera of 1.5 T. The size of voxel is 0.93 mm × 0.93 mm × 1.0 mm, and the coding is 12 bits. In addition, the signal has been analyzed by spin echo technique in order to minimize artifacts. We can see that there are four tissues and one background [21]. The histogram shows that grey level ranges overlap for cortical bone and background, and for spongy bone and adipose tissue (see Fig. 3). Therefore, using Fuzzy C-Means algorithm with 5 classes results in bad segmentation (Fig. 4). When applying the proposed system, we run the Fuzzy C-Means algorithm with 3 classes because only three grey levels can be observed (p = 3). Then, we obtain from the clustering result that
Fig. 4. Segmentation by Fuzzy C-Means with five classes.
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
25
Fig. 5. Classification result of crus. (a) The classification by a medical expert. (b) The classification by system A. (c) The classification by the proposed system.
C2 can be then identified as adipose tissue using Rule 6 with respect to the spongy bone. Using Rule 3 with respect to cortical bone and adipose tissue, we can label C3 as muscle. Using our proposed intelligent system, we effectively separate the four tissues and the background on the image of crus. Then we reconstruct a synthetic image using the obtained five classes with following grey levels: background: 20; adipose tissue: 250; spongy bone: 150; cortical bone: 200; muscle: 60 (Fig. 5c). We can see the quality in Fig. 5c is much higher than the one in Fig. 4. Fig. 5b shows the synthetic image generated from the identified tissues obtained by using a specific tissue classification system A [3]. In this system, the original image of thigh is first segmented into three classes by a FCM-based clustering algorithm [22]. By using the knowledge on the ranking order of tissue volumes, only three tissues (muscle, adipose tissue and spongy bone) can be identified. However, the integration of medical knowledge in this system is a manual procedure. But the proposed system is completely automatic. Next, we do quantitative comparison of the classification by the system A, our system and a reference classification (see Fig. 5a) provided by a medical expert. The Performance Criterion for the tissue Tj related to the ith method (PCTij ) is given as follows: PCTij =
Aij ∩ Arefj Aij ∪ Arefj
(1)
where Aij is the set of pixels belonging to the jth class obtained by the ith method, and Arefj is the set of pixels belonging to jth class in the reference classification result. For all identified tissues, the Performance Criterion for evaluating the ith method (PCi ) is given by:
NuT j=1 (Aij ∩ Arefj ) PCi = NuT NuT ( j=1 Aij ) ∪ ( j=1 Arefj )
(2)
Table 1 Quantitative comparison of classification.
The system A The proposed
Adipose tissue
Cortical bone
Spongy bone
Muscle
86.91% 93.18%
× 81.96%
82.98% 88.07%
97.07% 98.05%
Table 2 Quantitative comparison of overall classification performance. Overall performance (PCi )
The system A
The proposed
92.79%
98.06%
where NuT is the number of tissues in the image. Tables 1 and 2 show the corresponding quantitative comparison. 4. Conclusion In this paper, we present a generalized automatic system for tissue classification which can be adapted to different parts of human body. In this system, a general geometric model is proposed for formalizing non-structured and non-normalized medical knowledge from various medical images. An intelligent control procedure is developed for transforming the medical knowledge into several rules in order to improve the quality of the segmentation by Fuzzy C-Means. Two principles are proposed to define the priorities for these rules in order to optimize their application. One issue needs to be considered is that the geometric features used in this system may be sensitive to geometric transformations like rotation or flipping. We propose to normalize images before we run this system. In the future, we will work on the quantification of the muscle/fat ratio, assessment of the muscle/fat temporal variation, and measurement of the volume of muscle, fat and bone in human legs based on the obtained results. We will also validate our system on
26
H. Kang et al. / Biomedical Signal Processing and Control 6 (2011) 21–26
other parts of human body. Furthermore, we will make more efforts for exploring other common tissue features and add new rules to our system so that our system can be more generalized. We will also endeavor to apply new methods to formalize the tissue geometric feature, such as the Fuzzy Logic. In addition, we will try to propose new principles to define the priorities of rules in order to optimize their applications. Conflict of interest None. Acknowledgements The present research work has been supported by International Campus on Safety and Intermodality in Transportation, the Nord-Pas-de-Calais Region France, the European Community, the Regional Delegation for Research and Technology, the Ministry of Higher Education and Research, and the National Center for Scientific Research. The authors gratefully acknowledge the support of these institutions. References [1] A. Colin, E. Erbland, C. Datin, J.Y. Boire, A. Veyre, M. Zanca, Automatic muscle/fat quantification on MR images, in: Proceedings of the 17th International Conference of the IEEE Engineering in Medicine and Biology, 1, 1995, pp. 479–480. [2] G.Z. Yang, S. Myerson, F. Chabat, D.J. Pennell, D.N. Firmin, Automatic MRI adipose tissue mapping using overlapping mosaics, magnetic resonance materials in physics, Biology and Medicine 14 (2002) 39–44. [3] V. Barra, J.V. Boire, Segmentation of fat and muscle from MR images of the thigh by a possibilistic clustering algorithm, Computer Methods and Programs in Biomedicine 68 (2002) 185–193. [4] D. Zhang, S. Chen, A novel kernelized fuzzy C-means algorithm with application in medical image segmentation, Artificial Intelligence in Medicine 32 (2004) 37–50. [5] V. Roullier, C. Cavaro-Menard, G. Calmon, C. Aube, Fuzzy algorithms: application to adipose tissue quantification on MR images, Biomedical Signal Processing and Control 2 (2007) 239–247. [6] M.N. Ahmed, S.M. Yamany, N. Mohamed, A.A. Farag, T. Moriarty, A modified fuzzy c-means algorithm for bias field estimation and segmentation of MRI data, IEEE Transactions on Medical Imaging 21 (3) (2002) 193–200.
[7] S. Chen, D. Zhang, Robust image segmentation using FCM with spatial constraints based on new kernel-induced distance measure, IEEE transactions on system, Man, and Cybernetics-Part B: Cybernetics 34 (4) (2004) 1907–1916. [8] W. Hesheng, F. Baowei, A modified fuzzy C-means classification method using a multiscale diffusion filtering scheme, Medical Image Analysis 13 (2009) 193–202. [9] W. Jianzhong, K. Jun, L. Yinghua, Q. Miao, Z. Baoxue, A modified FCM algorithm for MRI brain image segmentation using both local and non-local spatial constraints, Computerized Medical Imaging and Graphics 32 (2008) 685–698. [10] H. Kang, Contribution to automatic corporal tissue classification by integrating qualitative medical knowledge: application to the analysis of musculo skeletal diseases and disabilities from MRI sequences, Ph.D. Thesis, Université de Valenciennes et du Hainaut-Cambrésis, France, 2009. [11] X. Li, T. Zhang, Z. Qu, Image segmentation using fuzzy clustering with spatial constraints based on markov random field via Bayesian theory, special section on Signal Processing for Audio and Visual Systems and its Implementations, IEICE Transactions Fundamentals E91–A (3) (2008) 723–728. [12] J. Valente de Oliveira, W. Pedrycz, Advances in Fuzzy Clustering and its Applications, John Wiley and Sons Ltd., England, 2007. [13] C. Li, D.B. Goldgof, L.O. Hall, Knowledge-based classification and tissue labeling of MR images of human brain, IEEE Transactions on Medical Imaging 12 (4) (1993) 740–750. [14] H. Li, R. Deklerck, J. Cornelis, Integration of multiple knowledge sources in a system for brain CT-scan interpretation based on the blackboard model, in: Proceedings of the 10th Conference on Artificial Intelligence for Applications, 1994, pp. 336–343. [15] H. Li, R. Deklerck, B.D. Cuyper, A. Hermanus, E. Nysser, J. Cornelis, Object recognition in brain CT-scans: knowledge-based fusion of data from multiple feature extractors, IEEE Transactions on Medical Imaging 14 (2) (1995) 212–229. [16] C. Wenhung, C. Youyin, C. Chienwen, L. Shenghuang, S. Yenyui, T. Siny, Improving segmentation accuracy for magnetic resonance imaging using a boosted decision tree, Journal of Neuroscience Methods 175 (2008) 206–217. [17] Z.Q. Liu, H.L. Liew, J.G. Clement, C.D.L. Thomas, Bone image segmentation, IEEE Transactions on Biomedical Engineering 46 (5) (1999) 565–570. [18] M.S. Brown, M.F. McNitt-Gray, N.J. Mankovich, J.G. Goldin, J. Hiller, L.S. Wilson, D.R. Aberle, Method for segmenting chest CT image data using an anatomical model: preliminary results, IEEE Transactions on Medical Imaging 16 (6) (1997) 828–839. [19] R.L. Cannon, J.V. Dave, J.C. Bezdek, Efficient implementation of the fuzzy c-means clustering algorithms, IEEE Transactions on Pattern Analysis and Machine Intelligence 8 (2) (1986) 248–255. [20] G.P. Robinson, A.C.F. Colchester, L.D. Griffin, Model-based recognition of anatomical objects from medical images, Image and Vision Computing 12 (8) (1994) 499–507. [21] P. Hédoux, A. Pinti, A. Taleb-Ahmed, Pixel classification method using statistical attributes of distributed imagelets, Journal Européen des Systèmes Automatisés 43 (3) (2009) 337–350. [22] R. Krishnapuram, J.M. Keller, A possibilistic approach to clustering, IEEE Transactions on Fuzzy Systems 1 (2) (1993) 98–110.