- Email: [email protected]
Tissue counter analysis of benign common nevi and malignant melanoma
International Journal of Medical Informatics 69 (2003) 17 /28 www.elsevier.com/locate/ijmedinf
Tissue counter analysis of benign common nevi and malignant melanoma M. Wiltgen a,, A. Gerger b, J. Smolle b a
b
Institute of Medical Informatics, Statistics and Documentation, University of Graz, Engelgasse 13, A-8010 Graz, Austria Department of Dermatology, Division of Analytical-Morphological Dermatology, University of Graz, Auenbruggerplatz 8, A-8036 Graz, Austria Accepted 10 September 2002
Abstract Objective: The aim of this study was to evaluate the applicability of tissue counter analysis to the interpretation of skin images. Method: Digital images from microscopic views of benign common nevi and malignant melanoma were classified by the use of features extracted from histogram and co-occurrence matrix. Eighty cases were sampled and split into a training set and a test set. The images were dissected in square elements and the different features were calculated for each element. The classification was done by classification and regression trees (CART) analysis. In the CART procedure, the square elements were split into disjunctive nodes, which were characterized by a relevant subset of the features. The classification results were indicated in the original image in order to evaluate the performance of the procedure. Results: For the learning set and the test set there is a significant difference between benign nevi and malignant melanoma without overlap. Discriminant analysis based on the percentage of ‘malignant elements’ facilitated a correct classification of all cases. Discussion: Since no image segmentation was needed, problems related to this task were avoided. Though wrong classification of individual elements is unavoidable to some degree, tissue counter analysis shows a good discrimination between benign common nevi and malignant melanoma. Conclusion: In conclusion, tissue counter analysis may be a useful method for the interpretation of melanocytic skin tumors. # 2002 Elsevier Science Ireland Ltd. All rights reserved. Keywords: Medical image processing; Tissue counter analysis; Computer assisted diagnosis; Benign common nevi; Malignant melanoma
1. Introduction
are well separated and can be clearly defined [1 3]. In histological tissue, the structures are mostly arranged in a variety of patterns and the segmentation of different structures, such as cells, nuclei, cytoplasm, vessels etc., is difficult, case dependent and cannot be done in a general approach. This is one reason why /
Automatic medical image analysis is successful when the structures, e.g. blood cells, Corresponding author E-mail address: [email protected] (M. Wiltgen).
1386-5056/02/$ - see front matter # 2002 Elsevier Science Ireland Ltd. All rights reserved. PII: S 1 3 8 6 - 5 0 5 6 ( 0 2 ) 0 0 0 4 9 - 7
18
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
compact tissue structures generally resist a fully automatic analysis [4]. We concentrated our attention on the automatic analysis of benign common nevi and malignant melanoma lesions. In this study, we tested the applicability of tissue counter analysis to the diagnostic discrimination of a set of melanoma and nevi [5]. The images of the test set show a broad variation in patterns in several cases. To enable the analysis of tissue structures, without being limited by the selection and detection of structures of interest, the images are divided into square elements of equal size. In this way, a priori definition and segmentation of the structures, which is the crucial point in automatic classification, was avoided. Features are calculated by extracting the digital information in each square element. The features from all the elements in the set of images were submitted as data set to a classification and regression trees (CART) analysis [6]. By the CART analysis the elements were automatically separated into more or less homogeneous nodes corresponding to different classes of structures.
showing the largest vertical tumor diameter was used for further analysis. One part of the study set was used as learning set, the other part as test set. In each case, a representative image from the tumor bulk was selected from the hematoxylin and eosin stained section. The images were taken using a Zeiss Axioskop bright field microscope (Zeiss, Oberkochen, Germany) mounted with a Sony 3-chip video camera (Sony, Tokyo, Japan), connected to a KS 400 system (Zeiss Vision, Hallbermos, Germany). All images were RGB color images with 512 512 pixels and 8 bits/pixel in each channel. The resolution was 1.3 mm/pixel. The software for image analysis was developed with the ‘INTERACTIVE DATA LANGUAGE’ software tool (IDL 5.4, Research Systems, Boulder, CO). /
3. Method The tissue counter analysis consists of three steps: the feature extraction, the classification and the verification. 3.1. Feature extraction
2. Materials
The images are divided in 256 square elements (measure masks) of 32 32 pixels. For each square element texture features are calculated. In this study, texture features were derived from the histogram and the co-occurrence matrix [7]. The histogram features describe mean value, standard deviation, skewness, kurtosis and entropy of the grey level distribution. For the calculation of the histogram the probability density function h(gi ) of the grey levels gi , satisfying the condition: X h(gi ) 1 /
All biopsies were routinely embedded in paraffin and cut at a section thickness of 4 mm. The slides were routinely stained with hematoxylin and eosin using a fully automated staining device (DRF, Fakura, Japan). The specimens were randomly sampled from the Dermatopathology files of the Department of Dermatology, University of Graz, Austria. All cases were diagnosed by two pathologists. In total, we sampled 40 biopsies of benign common nevi and 40 biopsies of malignant melanoma. For the study set only tumors with a vertical thickness exceeding 1 mm were used. In each tumor, the slide
i
was used.
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
The matrix elements of the co-occurrence matrix are normalized by: aij wij X X i
aij
j
The variable aij is the number of different grey level combinations (gi , gj ) between two adjacent pixels (the actually considered pixel and its right neighbor pixel) in the square element. From these matrix elements several features, such as average values, moments, variances, correlations and entropies, were calculated. For the calculation of the features, the following sums of rows and columns: X X wij Py (j) wij Px (i) j
i
and the sums of diagonals: XX wij Pxy (k) i
Pxy (k)
j ijk
XX i
wij
j ½ij½k
are calculated from the elements of the cooccurrence matrix, together with the means and standard deviations: X X gi Px (i) my gj Py (j) mx i
sx
j
X (gi mx )2 Px (i) i
sy
X
(gj my )2 Py (j)
j
These auxiliary variables describe the distribution of the matrix elements in the cooccurrence matrix and are used for the definition of features. From the histogram 15 features (5 per RGB channel, Table 1) were used and from the co-occurrence matrix 36 features (12 per RGB channel, Table 2).
19
Table 1 Features based on the histogram, that means the grey level distribution: h (gi ) Mean value in the RGB channel S.D. in the RGB channel Skewness in the RGB channel Kurtosis in the RGB channel Entropy of the grey levels in the RGB channel
mB ai gi h(gi )/ F1 ai (gi mB )2 h(gi )/ 3 /F2 ai (gi mB ) h(gi )/ 4 /F3 ai (gi mB ) h(gi )/ /F4 (1)ai h(gi )log2 (h(gi ))/ / /
The features and the corresponding RGB channels, which are relevant in the CART analysis, for the discrimination of the benign common nevi and the malignant melanoma are highlighted.
3.2. Classification In this study, the classification was done by CART analysis (Salford Systems, San Diego, USA). From the 80 cases of the study set 40 cases (20 images of benign common nevi and 20 images of malignant melanoma) were randomly selected and used as learning set. The 5120 elements from the 20 images of the benign common nevi and the 5120 elements of the 20 images of the malignant melanoma were used as data input for the CART analysis. In the CART analysis the set of elements are split into more or less homogeneous terminal nodes, which are assigned to classes. The splitting rules and the relevant features used in the analysis to split the set of elements into different classes were used to verify the performance of the method.
3.3. Verification The remaining 40 cases of the study set were used as test set. By use of the splitting rules and the relevant features (which is a subset of the 51 features) the square elements corresponding to the different classes were indicated in the original images. The elements of the respective class were highlighted by drawing squares in the overlay of the original image. The misclassified tissue structures
20
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
Table 2 Features based on the co-occurrence matrix Angular second moment in the RGB channel Difference moment in the RGB channel
/
F5 ai aj w2ij/
were taken into account for the 40 cases of the learning set, the percentage of tissue elements classified as ‘malignant elements was 7.39 4.9% (range: 1.6 22.7%) for the cases of benign common nevi and 92.196.6% (range: 74.6 99.82%) for the cases of malignant melanoma. For the 40 cases of the test set (by considering again all the nodes from the CART tree), the percentage of tissue elements classified as ‘malignant elements’ was 19.89 9.8% (range: 8.6 38.3%) for the cases of benign common nevi and 77.8918.0% (range: 44.9 98.8%) for the cases of malignant melanoma (Mann Whitney-U-test: P B 0.001; Fig. 1). Discriminant analysis based on the percentage of ‘malignant elements’ facilitated a correct classification of all cases (sensitivity 100%, specificity 100%). When the percentage of elements suggestive for malignancy in each case was evaluated, it turned out that a threshold level of 42% provides a correct classification of nevi and melanoma. From the 51 features a subset of features was really relevant in the CART analysis for the discrimination of the elements. To discuss the relevance of particular image analysis features, we concentrate our interest on the main terminal nodes (nodes containing more than 5% of all elements). The rules, which split the set of square elements into disjunctive nodes, use a subset of features, which are relevant for a successful splitting. The relevant features for the discrimination of the benign common nevi and the malignant melanoma are highlighted in Tables 1 and 2. The tissue of benign common nevi appears faded rose red and homogeneous, the nuclei are small and spread out widely. The tissue of malignant melanoma appears dark with high contrast. The properties, enabling the discrimination between the different tissues, are described by the relevant features. These features reflect in some sense how the tissue /
/
/
F6 ak k2 Pxy (k)/
/
/
Correlation in the RGB channel
/
F7
1 ða a (g g )w mx my Þ/ sx sy i j i j ij
Sum of squares in the RGB /F8 ai aj (gi mB )2 wij/ channel
/
/
Inverse difference moment in the RGB channel
/
F9 ak
Pxy (k) / 1 k2
/
/
/
Summed average in the RGB channel Sum variance in the RGB channel Entropy in the RGB channel Sum Entropy in the RGB channel Difference entropy in the RGB channel
/
F10 ak kPxy (k)/
/
F11 ak (kF10 )2 Pxy (k)/
/
F12 (1)ai aj wij log2 (wij )/
/
F13 (1)ak Pxy (k)log2 (Pxy (k))/
/
F14 (1)ak Pxy (k)log2 (Px4 (k))/
F12 Ixy a Information measure of /F15 / max[Ix ; Iy ] correlation 1 RGB channel Information measure of /F16 correlation 2 RGB channel
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffia (1exp(2(I?xy Ixy )))/
The features and the corresponding RGB channels, which are relevant in the CART analysis are highlighted. a Ix /(/1)/ai Px (i)log2 (Px (i)); Iy /(/1)/aj Py (j)log2 (Py (j)); Ixy /(/1)/ai aj wij log2 (Px (i)Py (j));/ ? /(/1)/ai aj Px (i)Py (j)log2 (Px (i)Py (j)):/ Ixy
could be gradually faded out by a slider, which facilitated visual control of the analysis results. For diagnostic assessment, only the percentage of elements suggestive for malignancy in each case was used.
4. Results The classification results of the CART analysis are given in Table 3. When all nodes
/
/
/
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
21
Table 3 Classification results of the CART analysis (element-by-element analysis) in the learning set Class
Total number of elements
Classification rate (%)
Number of elements related to Number of elements related to class 1 class 2
1, Benign common nevi 2, Malignant melanoma
5120
92 734
4748
372
5120
92 090
405
4715
appears to the human observer and the rules take into account how the human observer would classify the tissue structures in the microscopic images. The histogram skewness of the images of benign common nevi shows negative values, the variance is low and the mean value lies in the range of high grey levels (Fig. 2). The distribution of the elements in the co-occurrence matrix is concentrated in the range of high values and the variance of the element distribution is low. The histogram
skewness of the images of malignant melanoma shows positive values, the variance is higher than in the case of benign common nevi and the mean value lies in the range of low grey levels (Fig. 3). The distribution of the elements in the co-occurrence matrix is concentrated in the range of lower grey values, the variance of the element distribution is high and the correlation low. The percentage elements classified as common nevi and classified as malignant mela-
Fig. 1. Box plots of all cases of the learning set (left plot) and the test set (right plot), showing the percentage of ‘malignant elements’ in each case. For both sets there is a significant difference between benign nevi and malignant melanoma without overlap. Discriminant analysis based on the percentage of ‘malignant elements’ facilitated a correct classification of all cases (sensitivity/100%, specificity/ 100%).
22
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
Fig. 2. The figure shows the histogram and co-occurrence matrix (as a plot) of the blue channel from a square element in an image of benign common nevi. The histogram has a skewness with a negative value, a low variance and the mean value lies in the range of high grey levels. The distribution of the elements in the co-occurrence matrix is concentrated in the range of high values with a low variance.
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
23
Fig. 3. The figure shows the histogram and co-occurrence matrix (as a plot) of the green channel from a square element in an image of malignant melanoma. The histogram skewness has a positive value, a higher variance as in the case of benign common nevi and the mean value lies in the range of low grey levels. The distribution of the elements in the co-occurrence matrix is concentrated in the range of lower values with a high variance.
24
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
Fig. 4. Percentages of benign common nevi and malignant melanoma elements per case for the 20 cases of malignant melanoma in the learning set.
noma was calculated for each case in the learning and test set. As examples the percentages of elements for the cases with malignant melanoma in the learning set are shown in Fig. 4 and the percentages of elements for the cases with benign common nevi in the test set are shown in Fig. 5. The output results were indicated in the corresponding image in order to judge the performance of the analysis. The benign common nevi elements and the malignant melanoma elements, respectively, were highlighted in the corresponding image. In the image, incorrectly classified elements can be gradually faded out with a slider. This enables
one to distinguish better between the different tissue structures. Fig. 6 illustrates the case for a good classification of malignant melanoma. Fig. 7 illustrates a case with a bad classification result for benign common nevi. This case clearly shows that a high percentage of nontumor structures (e.g. hair follicles) falsifies the result.
5. Discussion There is an increasing number of melanocytic tumors. In the fight against skin cancer, researchers have high hopes in improved
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
25
Fig. 5. Percentages of benign common nevi and malignant melanoma elements per case for the 20 cases of benign common nevi in the test set.
provisional screening methods, such as optimized computer aided diagnostic methods. The texture features obtained from histogram and co-occurrence matrix for each square element contain sufficient information to facilitate the differentiation between malignant melanoma and benign common nevi elements. In contrast to other methods in medical image analysis no previous definition of relevant features is necessary. Therefore, problems related to image segmentation are avoided. The classification by CART analysis enables the determination of the relevant features and the classification rules. The
implementation of the rules allows an indication of the square elements in the corresponding image, so that the performance of the procedure can be evaluated. The results from this study show several limitations. First, the diagnostic groups were clear-cut with no doubtful cases. Second, the analysis was limited to tumors with a vertical thickness of at least 1-mm, which excludes very thin and early lesions. Third, analysis was focused on representative areas interactively selected by an observer. Further, as illustrated in Fig. 7, elements which are mainly concentrated at the border of two components, for
26
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
Fig. 6. Classification results for malignant melanoma. This figure illustrates the case where elements in a microscopic image of malignant melanoma are correctly interpreted. For the indication only the main terminal nodes, with the highest percentage of recognized structures, are taken into account. The elements, which are classified as malignant melanoma, are highlighted by drawing squares in the overlay of the original image. With the slider the incorrectly classified tissue structures can be gradually faded out.
example tissue and hair follicle, may be misclassified and, therefore, influence diagnostic accuracy. Thus the results do not suggest tissue counter analysis as a tool for automatic analysis and diagnosis of skin tumors but should illustrate the potential feasibility of the algorithm. Nevertheless, the indication of the classified square elements superimposed on the image offers the human observer the opportunity to
obtain additional (non-visual) information about morphology. The method of highlighting the classified tissue areas and the gradually fading out of misclassified areas may be a diagnostic aid. Therefore, beside automatic analysis of skin images, tissue counter analysis offers possibilities in computer aided diagnosis. In order to judge and improve the performance of tissue counter analysis of melano-
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
27
Fig. 7. Erroneous classification results for benign common nevi. For the indication only the main terminal nodes, with the highest percentage of interpreted structures, are considered. This figure illustrates as an example the case where elements in a microscopic image of benign common nevi are erroneously interpreted as malignant melanoma elements. These elements are highlighted by drawing squares in the overlay of the original image. It is noticeable that these elements are mainly concentrated along the dermoepidermal junction, blood vessels and around hair follicles, which, in this case, are not correctly interpreted and thereby falsify the result.
cytic skin tumors further studies are needed. The properties of the considered tissue, e.g. homogeneous areas in benign common nevi versus more structures in malignant melanoma, may encourage testing discrimination with features based on spectral properties of Fourier and wavelet transform. In conclusion, this study shows that tissue counter analysis allows discrimination be-
tween malignant melanoma and benign common nevi.
Acknowledgements This study was supported by the ‘Jubi¨ sterreichischen Nationalla¨umsfond der O bank’, project number 9297.
28
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
References [1] H. Harms, H.M. Aus, M. Haucke, U. Gunzer, Segmentation of stained blood cell images measured at high scanning density with high magnification and high numerical aperture optics, Cytometry 7 (1992) 522 /531. [2] W. Stolz, T. Vogth, M. Landthaler, S. Hempfer, P. Bingler, W. Abmayr, Differentiation between maligant melanomas and benign melanocytic nevi by computerized DNA cytometry of imprint specimens, J. Cutaneous Pathol. 21 (1994) 7 /15. [3] P.W. Hamilton, P.H. Bartels, R. Montironi, et al., Automated histometry in quantitative prostate pathology, Anal. Quant. Cytol. Histol. 20 (1998) 443 /460.
[4] J. Smolle, Computer recognition of skin structures using discriminant and cluster analysis, Skin Res. Technol. 6 (2000) 58 /63. [5] J. Smolle, Diagnostic assessment of cutaneous melanoma and common nevi using tissue counter analysis, Anal. Quant. Cytol. Histol. 22 (4) (2000) 299 /306. [6] L. Breiman, J.H. Friedman, R.A. Olshen, C.J. Stone, Classification and Regression Trees, Chapman & Hall, New York, London, 1993. [7] M. Wiltgen, Digitale Bildverarbeitung in der Medizin. Grundlagen der Verarbeitung bildhafter Informationen fu¨r die Interpretation und Kommunikation in der Medizin Shaker Verlag, Aachen, 1999.
Tissue counter analysis of benign common nevi and malignant melanoma M. Wiltgen a,, A. Gerger b, J. Smolle b a
b
Institute of Medical Informatics, Statistics and Documentation, University of Graz, Engelgasse 13, A-8010 Graz, Austria Department of Dermatology, Division of Analytical-Morphological Dermatology, University of Graz, Auenbruggerplatz 8, A-8036 Graz, Austria Accepted 10 September 2002
Abstract Objective: The aim of this study was to evaluate the applicability of tissue counter analysis to the interpretation of skin images. Method: Digital images from microscopic views of benign common nevi and malignant melanoma were classified by the use of features extracted from histogram and co-occurrence matrix. Eighty cases were sampled and split into a training set and a test set. The images were dissected in square elements and the different features were calculated for each element. The classification was done by classification and regression trees (CART) analysis. In the CART procedure, the square elements were split into disjunctive nodes, which were characterized by a relevant subset of the features. The classification results were indicated in the original image in order to evaluate the performance of the procedure. Results: For the learning set and the test set there is a significant difference between benign nevi and malignant melanoma without overlap. Discriminant analysis based on the percentage of ‘malignant elements’ facilitated a correct classification of all cases. Discussion: Since no image segmentation was needed, problems related to this task were avoided. Though wrong classification of individual elements is unavoidable to some degree, tissue counter analysis shows a good discrimination between benign common nevi and malignant melanoma. Conclusion: In conclusion, tissue counter analysis may be a useful method for the interpretation of melanocytic skin tumors. # 2002 Elsevier Science Ireland Ltd. All rights reserved. Keywords: Medical image processing; Tissue counter analysis; Computer assisted diagnosis; Benign common nevi; Malignant melanoma
1. Introduction
are well separated and can be clearly defined [1 3]. In histological tissue, the structures are mostly arranged in a variety of patterns and the segmentation of different structures, such as cells, nuclei, cytoplasm, vessels etc., is difficult, case dependent and cannot be done in a general approach. This is one reason why /
Automatic medical image analysis is successful when the structures, e.g. blood cells, Corresponding author E-mail address: [email protected] (M. Wiltgen).
1386-5056/02/$ - see front matter # 2002 Elsevier Science Ireland Ltd. All rights reserved. PII: S 1 3 8 6 - 5 0 5 6 ( 0 2 ) 0 0 0 4 9 - 7
18
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
compact tissue structures generally resist a fully automatic analysis [4]. We concentrated our attention on the automatic analysis of benign common nevi and malignant melanoma lesions. In this study, we tested the applicability of tissue counter analysis to the diagnostic discrimination of a set of melanoma and nevi [5]. The images of the test set show a broad variation in patterns in several cases. To enable the analysis of tissue structures, without being limited by the selection and detection of structures of interest, the images are divided into square elements of equal size. In this way, a priori definition and segmentation of the structures, which is the crucial point in automatic classification, was avoided. Features are calculated by extracting the digital information in each square element. The features from all the elements in the set of images were submitted as data set to a classification and regression trees (CART) analysis [6]. By the CART analysis the elements were automatically separated into more or less homogeneous nodes corresponding to different classes of structures.
showing the largest vertical tumor diameter was used for further analysis. One part of the study set was used as learning set, the other part as test set. In each case, a representative image from the tumor bulk was selected from the hematoxylin and eosin stained section. The images were taken using a Zeiss Axioskop bright field microscope (Zeiss, Oberkochen, Germany) mounted with a Sony 3-chip video camera (Sony, Tokyo, Japan), connected to a KS 400 system (Zeiss Vision, Hallbermos, Germany). All images were RGB color images with 512 512 pixels and 8 bits/pixel in each channel. The resolution was 1.3 mm/pixel. The software for image analysis was developed with the ‘INTERACTIVE DATA LANGUAGE’ software tool (IDL 5.4, Research Systems, Boulder, CO). /
3. Method The tissue counter analysis consists of three steps: the feature extraction, the classification and the verification. 3.1. Feature extraction
2. Materials
The images are divided in 256 square elements (measure masks) of 32 32 pixels. For each square element texture features are calculated. In this study, texture features were derived from the histogram and the co-occurrence matrix [7]. The histogram features describe mean value, standard deviation, skewness, kurtosis and entropy of the grey level distribution. For the calculation of the histogram the probability density function h(gi ) of the grey levels gi , satisfying the condition: X h(gi ) 1 /
All biopsies were routinely embedded in paraffin and cut at a section thickness of 4 mm. The slides were routinely stained with hematoxylin and eosin using a fully automated staining device (DRF, Fakura, Japan). The specimens were randomly sampled from the Dermatopathology files of the Department of Dermatology, University of Graz, Austria. All cases were diagnosed by two pathologists. In total, we sampled 40 biopsies of benign common nevi and 40 biopsies of malignant melanoma. For the study set only tumors with a vertical thickness exceeding 1 mm were used. In each tumor, the slide
i
was used.
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
The matrix elements of the co-occurrence matrix are normalized by: aij wij X X i
aij
j
The variable aij is the number of different grey level combinations (gi , gj ) between two adjacent pixels (the actually considered pixel and its right neighbor pixel) in the square element. From these matrix elements several features, such as average values, moments, variances, correlations and entropies, were calculated. For the calculation of the features, the following sums of rows and columns: X X wij Py (j) wij Px (i) j
i
and the sums of diagonals: XX wij Pxy (k) i
Pxy (k)
j ijk
XX i
wij
j ½ij½k
are calculated from the elements of the cooccurrence matrix, together with the means and standard deviations: X X gi Px (i) my gj Py (j) mx i
sx
j
X (gi mx )2 Px (i) i
sy
X
(gj my )2 Py (j)
j
These auxiliary variables describe the distribution of the matrix elements in the cooccurrence matrix and are used for the definition of features. From the histogram 15 features (5 per RGB channel, Table 1) were used and from the co-occurrence matrix 36 features (12 per RGB channel, Table 2).
19
Table 1 Features based on the histogram, that means the grey level distribution: h (gi ) Mean value in the RGB channel S.D. in the RGB channel Skewness in the RGB channel Kurtosis in the RGB channel Entropy of the grey levels in the RGB channel
mB ai gi h(gi )/ F1 ai (gi mB )2 h(gi )/ 3 /F2 ai (gi mB ) h(gi )/ 4 /F3 ai (gi mB ) h(gi )/ /F4 (1)ai h(gi )log2 (h(gi ))/ / /
The features and the corresponding RGB channels, which are relevant in the CART analysis, for the discrimination of the benign common nevi and the malignant melanoma are highlighted.
3.2. Classification In this study, the classification was done by CART analysis (Salford Systems, San Diego, USA). From the 80 cases of the study set 40 cases (20 images of benign common nevi and 20 images of malignant melanoma) were randomly selected and used as learning set. The 5120 elements from the 20 images of the benign common nevi and the 5120 elements of the 20 images of the malignant melanoma were used as data input for the CART analysis. In the CART analysis the set of elements are split into more or less homogeneous terminal nodes, which are assigned to classes. The splitting rules and the relevant features used in the analysis to split the set of elements into different classes were used to verify the performance of the method.
3.3. Verification The remaining 40 cases of the study set were used as test set. By use of the splitting rules and the relevant features (which is a subset of the 51 features) the square elements corresponding to the different classes were indicated in the original images. The elements of the respective class were highlighted by drawing squares in the overlay of the original image. The misclassified tissue structures
20
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
Table 2 Features based on the co-occurrence matrix Angular second moment in the RGB channel Difference moment in the RGB channel
/
F5 ai aj w2ij/
were taken into account for the 40 cases of the learning set, the percentage of tissue elements classified as ‘malignant elements was 7.39 4.9% (range: 1.6 22.7%) for the cases of benign common nevi and 92.196.6% (range: 74.6 99.82%) for the cases of malignant melanoma. For the 40 cases of the test set (by considering again all the nodes from the CART tree), the percentage of tissue elements classified as ‘malignant elements’ was 19.89 9.8% (range: 8.6 38.3%) for the cases of benign common nevi and 77.8918.0% (range: 44.9 98.8%) for the cases of malignant melanoma (Mann Whitney-U-test: P B 0.001; Fig. 1). Discriminant analysis based on the percentage of ‘malignant elements’ facilitated a correct classification of all cases (sensitivity 100%, specificity 100%). When the percentage of elements suggestive for malignancy in each case was evaluated, it turned out that a threshold level of 42% provides a correct classification of nevi and melanoma. From the 51 features a subset of features was really relevant in the CART analysis for the discrimination of the elements. To discuss the relevance of particular image analysis features, we concentrate our interest on the main terminal nodes (nodes containing more than 5% of all elements). The rules, which split the set of square elements into disjunctive nodes, use a subset of features, which are relevant for a successful splitting. The relevant features for the discrimination of the benign common nevi and the malignant melanoma are highlighted in Tables 1 and 2. The tissue of benign common nevi appears faded rose red and homogeneous, the nuclei are small and spread out widely. The tissue of malignant melanoma appears dark with high contrast. The properties, enabling the discrimination between the different tissues, are described by the relevant features. These features reflect in some sense how the tissue /
/
/
F6 ak k2 Pxy (k)/
/
/
Correlation in the RGB channel
/
F7
1 ða a (g g )w mx my Þ/ sx sy i j i j ij
Sum of squares in the RGB /F8 ai aj (gi mB )2 wij/ channel
/
/
Inverse difference moment in the RGB channel
/
F9 ak
Pxy (k) / 1 k2
/
/
/
Summed average in the RGB channel Sum variance in the RGB channel Entropy in the RGB channel Sum Entropy in the RGB channel Difference entropy in the RGB channel
/
F10 ak kPxy (k)/
/
F11 ak (kF10 )2 Pxy (k)/
/
F12 (1)ai aj wij log2 (wij )/
/
F13 (1)ak Pxy (k)log2 (Pxy (k))/
/
F14 (1)ak Pxy (k)log2 (Px4 (k))/
F12 Ixy a Information measure of /F15 / max[Ix ; Iy ] correlation 1 RGB channel Information measure of /F16 correlation 2 RGB channel
pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffia (1exp(2(I?xy Ixy )))/
The features and the corresponding RGB channels, which are relevant in the CART analysis are highlighted. a Ix /(/1)/ai Px (i)log2 (Px (i)); Iy /(/1)/aj Py (j)log2 (Py (j)); Ixy /(/1)/ai aj wij log2 (Px (i)Py (j));/ ? /(/1)/ai aj Px (i)Py (j)log2 (Px (i)Py (j)):/ Ixy
could be gradually faded out by a slider, which facilitated visual control of the analysis results. For diagnostic assessment, only the percentage of elements suggestive for malignancy in each case was used.
4. Results The classification results of the CART analysis are given in Table 3. When all nodes
/
/
/
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
21
Table 3 Classification results of the CART analysis (element-by-element analysis) in the learning set Class
Total number of elements
Classification rate (%)
Number of elements related to Number of elements related to class 1 class 2
1, Benign common nevi 2, Malignant melanoma
5120
92 734
4748
372
5120
92 090
405
4715
appears to the human observer and the rules take into account how the human observer would classify the tissue structures in the microscopic images. The histogram skewness of the images of benign common nevi shows negative values, the variance is low and the mean value lies in the range of high grey levels (Fig. 2). The distribution of the elements in the co-occurrence matrix is concentrated in the range of high values and the variance of the element distribution is low. The histogram
skewness of the images of malignant melanoma shows positive values, the variance is higher than in the case of benign common nevi and the mean value lies in the range of low grey levels (Fig. 3). The distribution of the elements in the co-occurrence matrix is concentrated in the range of lower grey values, the variance of the element distribution is high and the correlation low. The percentage elements classified as common nevi and classified as malignant mela-
Fig. 1. Box plots of all cases of the learning set (left plot) and the test set (right plot), showing the percentage of ‘malignant elements’ in each case. For both sets there is a significant difference between benign nevi and malignant melanoma without overlap. Discriminant analysis based on the percentage of ‘malignant elements’ facilitated a correct classification of all cases (sensitivity/100%, specificity/ 100%).
22
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
Fig. 2. The figure shows the histogram and co-occurrence matrix (as a plot) of the blue channel from a square element in an image of benign common nevi. The histogram has a skewness with a negative value, a low variance and the mean value lies in the range of high grey levels. The distribution of the elements in the co-occurrence matrix is concentrated in the range of high values with a low variance.
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
23
Fig. 3. The figure shows the histogram and co-occurrence matrix (as a plot) of the green channel from a square element in an image of malignant melanoma. The histogram skewness has a positive value, a higher variance as in the case of benign common nevi and the mean value lies in the range of low grey levels. The distribution of the elements in the co-occurrence matrix is concentrated in the range of lower values with a high variance.
24
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
Fig. 4. Percentages of benign common nevi and malignant melanoma elements per case for the 20 cases of malignant melanoma in the learning set.
noma was calculated for each case in the learning and test set. As examples the percentages of elements for the cases with malignant melanoma in the learning set are shown in Fig. 4 and the percentages of elements for the cases with benign common nevi in the test set are shown in Fig. 5. The output results were indicated in the corresponding image in order to judge the performance of the analysis. The benign common nevi elements and the malignant melanoma elements, respectively, were highlighted in the corresponding image. In the image, incorrectly classified elements can be gradually faded out with a slider. This enables
one to distinguish better between the different tissue structures. Fig. 6 illustrates the case for a good classification of malignant melanoma. Fig. 7 illustrates a case with a bad classification result for benign common nevi. This case clearly shows that a high percentage of nontumor structures (e.g. hair follicles) falsifies the result.
5. Discussion There is an increasing number of melanocytic tumors. In the fight against skin cancer, researchers have high hopes in improved
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
25
Fig. 5. Percentages of benign common nevi and malignant melanoma elements per case for the 20 cases of benign common nevi in the test set.
provisional screening methods, such as optimized computer aided diagnostic methods. The texture features obtained from histogram and co-occurrence matrix for each square element contain sufficient information to facilitate the differentiation between malignant melanoma and benign common nevi elements. In contrast to other methods in medical image analysis no previous definition of relevant features is necessary. Therefore, problems related to image segmentation are avoided. The classification by CART analysis enables the determination of the relevant features and the classification rules. The
implementation of the rules allows an indication of the square elements in the corresponding image, so that the performance of the procedure can be evaluated. The results from this study show several limitations. First, the diagnostic groups were clear-cut with no doubtful cases. Second, the analysis was limited to tumors with a vertical thickness of at least 1-mm, which excludes very thin and early lesions. Third, analysis was focused on representative areas interactively selected by an observer. Further, as illustrated in Fig. 7, elements which are mainly concentrated at the border of two components, for
26
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
Fig. 6. Classification results for malignant melanoma. This figure illustrates the case where elements in a microscopic image of malignant melanoma are correctly interpreted. For the indication only the main terminal nodes, with the highest percentage of recognized structures, are taken into account. The elements, which are classified as malignant melanoma, are highlighted by drawing squares in the overlay of the original image. With the slider the incorrectly classified tissue structures can be gradually faded out.
example tissue and hair follicle, may be misclassified and, therefore, influence diagnostic accuracy. Thus the results do not suggest tissue counter analysis as a tool for automatic analysis and diagnosis of skin tumors but should illustrate the potential feasibility of the algorithm. Nevertheless, the indication of the classified square elements superimposed on the image offers the human observer the opportunity to
obtain additional (non-visual) information about morphology. The method of highlighting the classified tissue areas and the gradually fading out of misclassified areas may be a diagnostic aid. Therefore, beside automatic analysis of skin images, tissue counter analysis offers possibilities in computer aided diagnosis. In order to judge and improve the performance of tissue counter analysis of melano-
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
27
Fig. 7. Erroneous classification results for benign common nevi. For the indication only the main terminal nodes, with the highest percentage of interpreted structures, are considered. This figure illustrates as an example the case where elements in a microscopic image of benign common nevi are erroneously interpreted as malignant melanoma elements. These elements are highlighted by drawing squares in the overlay of the original image. It is noticeable that these elements are mainly concentrated along the dermoepidermal junction, blood vessels and around hair follicles, which, in this case, are not correctly interpreted and thereby falsify the result.
cytic skin tumors further studies are needed. The properties of the considered tissue, e.g. homogeneous areas in benign common nevi versus more structures in malignant melanoma, may encourage testing discrimination with features based on spectral properties of Fourier and wavelet transform. In conclusion, this study shows that tissue counter analysis allows discrimination be-
tween malignant melanoma and benign common nevi.
Acknowledgements This study was supported by the ‘Jubi¨ sterreichischen Nationalla¨umsfond der O bank’, project number 9297.
28
M. Wiltgen et al. / International Journal of Medical Informatics 69 (2003) 17 /28
References [1] H. Harms, H.M. Aus, M. Haucke, U. Gunzer, Segmentation of stained blood cell images measured at high scanning density with high magnification and high numerical aperture optics, Cytometry 7 (1992) 522 /531. [2] W. Stolz, T. Vogth, M. Landthaler, S. Hempfer, P. Bingler, W. Abmayr, Differentiation between maligant melanomas and benign melanocytic nevi by computerized DNA cytometry of imprint specimens, J. Cutaneous Pathol. 21 (1994) 7 /15. [3] P.W. Hamilton, P.H. Bartels, R. Montironi, et al., Automated histometry in quantitative prostate pathology, Anal. Quant. Cytol. Histol. 20 (1998) 443 /460.
[4] J. Smolle, Computer recognition of skin structures using discriminant and cluster analysis, Skin Res. Technol. 6 (2000) 58 /63. [5] J. Smolle, Diagnostic assessment of cutaneous melanoma and common nevi using tissue counter analysis, Anal. Quant. Cytol. Histol. 22 (4) (2000) 299 /306. [6] L. Breiman, J.H. Friedman, R.A. Olshen, C.J. Stone, Classification and Regression Trees, Chapman & Hall, New York, London, 1993. [7] M. Wiltgen, Digitale Bildverarbeitung in der Medizin. Grundlagen der Verarbeitung bildhafter Informationen fu¨r die Interpretation und Kommunikation in der Medizin Shaker Verlag, Aachen, 1999.