Color texture classification by integrative Co-occurrence matrices

Pattern Recognition 37 (2004) 965 – 976 www.elsevier.com/locate/patcog

Color texture classi"cation by integrative Co-occurrence matrices Christoph Palm1 Institute of Medical Informatics, Aachen University of Technology, D-52057 Aachen, Germany Received 26 November 2002; received in revised form 4 August 2003; accepted 18 September 2003

Abstract Integrative Co-occurrence matrices are introduced as novel features for color texture classi"cation. The extended Co-occurrence notation allows the comparison between integrative and parallel color texture concepts. The information pro"t of the new matrices is shown quantitatively using the Kolmogorov distance and by extensive classi"cation experiments on two datasets. Applying them to the RGB and the LUV color space the combined color and intensity textures are studied and the existence of intensity independent pure color patterns is demonstrated. The results are compared with two baselines: gray-scale texture analysis and color histogram analysis. The novel features improve the classi"cation results up to 20% and 32% for the "rst and second baseline, respectively. ? 2003 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. Keywords: Color texture; Co-occurrence matrix; Integrative features; Kolmogorov distance; Image classi"cation

1. Introduction Texture analysis is one of the main topics in the "eld of digital image processing. Texture is described as a pattern with some kind of regularity. The approaches of mathematical modeling are grouped into structural, statistical and signal theoretic methods [1]. Structural methods are based on a more or less deterministic arrangement of textural elements (texels). They are mainly used in industrial quality control, where arti"cal patterns are regular and di?erences between model and reality indicate failures [2,3]. Statistical methods de"ne textures as stochastic processes and characterize them by a few statistical features. Most relevant statistical approaches are Co-occurrence matrices [4], Markov random "elds [5] and autocorrelation methods [6]. Signal theoretic approaches focus on periodic pattern resulting in peaks in the spatial frequency domain, e.g. Gabor "ltering [7,8] and wavelet decomposition [9]. 1 Christoph Palm is now with the Aixplain AG, Monheimsallee 22, D-52062 Aachen, Germany. E-mail address: [email protected] (C. Palm).

Beside texture, color is an important issue not only in human vision but in digital image processing where its impact is still rising. In contrast to intensity, coded as scalar gray values, color is a vectorial feature assigned to each pixel in a color image. The mathematical di?erence between scalars and vectors for gray and color values, respectively, demands a careful transfer of methods from the gray-scale to the color domain. Although the use of color for texture analysis is shown to be advantageous, the integration of color and texture is still exceptional. We di?erentiate between parallel, sequential and integrative methods (Section 2). Several studies favor the statistical texture modeling and in particular Co-occurrence matrices in the gray-scale domain [10]. Nevertheless, just few approaches were made to transfer Co-occurrence matrices to color images [35]. In this paper, we study existing approaches for color texture classi"cation by Co-occurrence features and propose novel matrices and features to allow the exploitation of the color information. After the introduction of general color texture concepts (Section 2) and details of the most relevant color spaces (Section 3), we shortly repeat the concept of

0031-3203/$30.00 ? 2003 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved. doi:10.1016/j.patcog.2003.09.010

966

C. Palm / Pattern Recognition 37 (2004) 965 – 976

Co-occurrence matrices for gray-scale images (Section 4). This concept is extended to the color domain (Section 5) de"ning single- and multi-channel matrices, respectively. The description of the experimental setup (Section 6) is followed by the results and a detailed discussion (Section 7).

feature analysis color histogram feature analysis

color image

grayscale image

2. Combining color and texture We group the approaches combining color and texture into parallel, sequential and integrative approaches. The parallel concept for color texture analysis separates the processing of both phenomena. Color is measured globally according to the histogram ignoring local neighboring pixels. Texture is characterized by the relationship of the intensities of neighboring pixels ignoring their color. The results of both analyses are combined subsequently to a feature vector (Fig. 1, top). The parallel approach is advantageous, because the known methods of gray-scale texture analysis as well as pixel based color analysis can be applied directly. Up to now, parallel algorithms are most commonly used for image retrieval applications with color and texture as separated discriminative features [11]. However, the view on texture as a pure intensity based structure is very simpli"ed and disregards, for instance, colored texture primitives with constant intensity. Sequential approaches use color analysis as a "rst step of a process chain. Clustering the color histogram, a partitioning of the image is obtained. These consecutive numbered segments contain color information, but they are just represented by their indices and, hence, by scalars. Subsequently, the index pattern is processed as gray-scale texture (Fig. 1, middle). The sequential approach is related to the structural texture model, where the pattern is composed by (color) primitives. Hauta-Kasari et al. used it for industrial quality control [12], Song et al. especially for defect detection in granite images [13], and Kukkonen et al. for inspection of ceramic tiles [14]. These examples show the usefulness of the sequential approach. Nevertheless, it is based on the segmentation procedure, which depends on several parameters and, therefore, gives no reproducible results. Additionally, the assumption of colored texture primitives cannot be generalized. The third way of combining the analysis of color and texture is called integrative, because the information dependency between both image features is taken into account. Integrative methods can be further di?erentiated into single- and multi-channel strategies (Fig. 1, bottom). The single-channel methods apply the gray-scale texture analysis on each color channel separately. The color information is used indirectly restricting the intensity pattern to the wavelength interval associated with the color channel. The advantage of the single channel approach is the easy adaption of known methods based on the gray-scale domain. Recently, some of these were compared with the paral-

segmentation color image

color histogram

feature analysis label image

feature analysis feature analysis

single channel

feature analysis color channels feature analysis

color image

multi channel

feature analysis

feature analysis

Fig. 1. Top: Illustration of the parallel concept for color texture analysis. Textural features and histogram features are strictly separated. Middle: Illustration of the sequential concept for color texture analysis. After the color segmentation of the histogram the texture features are determined on basis of the segment indices. Bottom: Illustration of integrative single- and multi-channel color texture analysis. Both methods take color and texture into account.

lel concept and showed signi"cantly improved results [15]. Integrative multi-channel texture analysis handles two (or more) channels simultaneously. These approaches have already been proposed for well known textural features like Markov random "elds [16], wavelet [17], Gabor "lters [18– 21], and autocorrelation features [22], showing encouraging results. Up to now, Co-occurrence matrices are just adapted for sequential [12,14] and integrative single-channel color texture analysis [15]. The idea of multi-channel Co-occurrence matrices was "rstly addressed by Rosenfeld et al. in 1982, but found to be computationally cumbersome [23]. In this paper, we "rstly extend the Co-occurrence approach to

C. Palm / Pattern Recognition 37 (2004) 965 – 976

integrative multi-channel color texture features. Additionally, we introduce an information measure to describe the supplementary information content of the multi-channel in comparison to the single-channel approach. The extensive experimental evaluation of the features in a classi"cation task takes two color spaces into consideration. The results show the advantages of the multi-channel approach and demonstrate the existence of non-intensity based pure color textures.

967

d

3. Color spaces To study the relation of intensity textures, pure color textures and combinations of both, respectively, the representation of intensity within the color space is important. The RGB (red, green, blue) color space is a projection of the cumulated intensity of reIected light within a de"ned range of wavelength to the axes of a three-dimensional space. Certainly, the RGB values are not only dependent on the reIectance properties of the observed pattern but on the illuminant and the recording characteristics of the camera. The problems of color constancy and device independency are studied in detail elsewhere [22,24] and are not topic of this paper. Because of the wavelength-dependent intensity cumulation each color band is inherently a combination of color and intensity. Additionally, a strong correlation between the color channels is reported [25]. De-correlating the RGB color bands and separating color and intensity yields the LUV color space [26]:     L R      U  = HLUV ·  G      V B √ √   √ R 2 2 2   1    2 −1 −1  (1) =√  G : 6 √ √ B 3 − 3 0 Tan et al. reported improving results for color histogram analysis using LUV [26]. Independently from the idea of mathematical de-correlation, a similar color space is de"ned modeling human opponent color vision [27]. Additionally, the LUV space is strongly related to the complex color space since the color planes spanned by UV and the complex colors are equal [28]. With RGB and LUV we study color spaces where intensity and color are combined or separated, respectively.

4. Gray-scale Co-occurrence matrices In contrast to histograms, Co-occurrence matrices (CMs) characterize the relationship between the values of

Fig. 2. Illustration of the concept of gray-scale Co-occurrence matrices. The Co-occurrence matrices within one octant are combined to one mean meatix to be more robust for long distances d.

neighboring pixels [4]. Therefore, they represent a second order statistical measurement. We denote the values of a gray-scale image f as w ∈ {0; : : : ; W − 1} with f(p) = w. The pixel position is given by p = (m; n) with m; n ∈ Z. The distance vector d is noted in polar coordinates (d; %) with discrete length and orientation d; % ∈ N, respectively, computed from linear coordinates applying the polar transform and an appropriate truncation. The probability Pr of two values w and wK co-occurring with pixel positions related by d, de"nes the cell entry (w1 ; w2 ) of the CM Cd : K Cd (w; w) := Pr(f(p1 ) = w ∧ f(p2 ) =w| K |p1 − p2 | = d):

(2)

Hence, Cd is a symmetric two-dimensional neighborhood histogram, which depends on d. To study the e?ect of changing d to the classi"cation results, we extend Cd to long-distance CMs. Fig. 2 visualizes the idea to build the mean CM for each octant of a discrete circle approximation. The single CMs within the octant are summed up and normalized keeping d constant. Because of the symmetric de"nition of CM only four mean CMs remain. Nevertheless, with (W × W ) they are high dimensional. To reduce the large amount of data represented by CMs, two methods are applicable: • quantize the image into few values as a preprocessing step • extract special features to describe CM data as a post-processing method In this work we applied the last approach and used the matrix features introduced by Haralick et al. [4]. Unfortunately, some of the 14 features show redundancies, but it is not yet clear, which of them can be ignored. Therefore, we selected eight features (homogeneity, contrast, correlation, variance, inverse di?erence moment, entropy, correlation I, correlation

C. Palm / Pattern Recognition 37 (2004) 965 – 976

Distance

0.2

0.2

0

Image Number

30

5.1. Single-channel Co-occurrence matrices 5.1.1. Background For the de"nition of SCMs, we assume a limited K-dimensional Cartesian space like RGB and LUV with K = 3. A SCM SCkd corresponds to Cd applied to the kth color channel fk of the image f = (f1 ; : : : ; fK )T : Pr(wk ; wK k )

0

1

1

0.8

0.8

0.2

According to our categorization (Section 2), in the following two integrative extensions of gray-scale CMs to color CMs are proposed. The single-channel Co-occurrence matrices (SCMs) consist of gray-scale CMs successively applied to separated color channels. The correlation between textures of di?erent color channels is captured by novel multi-channel Co-occurrence matrices (MCMs).

Image Number

30

0.2 0

Image Number

30

0

Image Number

30

Fig. 3. Top left: Kolmogorov distance D(SCdk ) for single-channel Co-occurrence matrices with k = R (white), k = G (black), and k = B (gray). The image numbers follow the ascending order of D(SCdG ). Bottom left: D(SCdk ) with k = L (black) k = U (gray), and k = V (white). The image numbers folk ;k low the ascending order of D(SCdL ). Top right: D(MCd1 2 ) for multi-channel Co-occurrence matrices with (k1 ; k2 ) = (R; G) (white), (k1 ; k2 ) = (R; B) (black), and (k1 ; k2 ) = (G; B) (gray). The image numbers follow the ascending order of D(MCdR; B ). Bottom k ;k right: D(MCd1 2 ) with (k1 ; k2 ) = (U; V ) (black) (k1 ; k2 ) = (L; U ) (gray), and (k1 ; k2 ) = (L; V ) (white). The image numbers follow the ascending order of D(MCdU; V ).

corresponding SCkd by summing up the columns: Pr(wK k ) =

Wk −1



SCkd (w; w): K

(4)

w=0 K

SCkd (w; w) K

:= Pr(fk (p1 ) = w ∧ fk (p2 ) = w| K |p1 − p2 | = d):

1

0.8

Distance

5. Integrative color Co-occurrence matrices

:=

1 0.8

Distance

II), that build the set union used in [10,15,29]. They are well distributed over the four feature groups de"ned in [10]. The Cartesian product for the selected Haralick features generates a (8 × 4)-space for the rotationally variant CM features. To be rotationally invariant, the mean and the variance of the orientation-dependent features are determined separately. Overall, the gray-scale CM features (GCFs) are vectors of size 8 × 2.

Distance

968

(3)

The joint probability Pr(wk ; wK k ) indicates the adjacency of w and wK on the same channel k. The corresponding rotational invariant single-channel Co-occurrence features (SCFs) consist of K feature vectors SCFk , where SCFk is analog to GCF according to k: SCF = SCF1 × · · · × SCFK . Following Section 3, RGB combines intensity as well as color information. Therefore, RGB-based SCFs contain combined color and intensity texture information. Recently, Drimbarean et al. showed RGB-based SCF-like features to outperform GCFs [15]. In contrast to RGB, LUV separates intensity and color strictly. Hence, SCF1 represents pure intensity texture and equals GCF. In contrast, pure color textures are modeled by SCF2 and SCF3 according to the channels U and V , respectively. 5.1.2. Information measurement The quanti"cation of the information pro"t of CMs in comparison to common histograms is based on the Kolmogorov distance [30], which measures the di?erence between two probability distributions. The one-dimensional histogram Pr(wk ) for each channel k is derived from the

In the case of stochastic independency of d-adjacent values,  k ; wK k ) is estimated by the multithe joint probability Pr(w plicative combination of the univariate densities Pr(wk ) and Pr(wK k ):  k ; wK k ) = Pr(wk ) · Pr(wK k ): Pr(w

(5) k

k

In contrast, the actually measured density Pr(w ; wK ) is related to the conditional probability Pr(wk |wK k ): Pr(wk ; wK k ) = Pr(wk |wK k ) · Pr(wK k ):

(6)

The Kolmogorov distance D(SCkd ) as di?erence between  k ; wK k ) and Pr(wk ; wK k ) is equal to the di?erence between Pr(w random adjacency and the actual adjacency of w and w. K Therefore, it is a measure of stochastic dependency and, consequently, a measure of information gain of the CM in contrast to the univariate histogram: D(SCkd ) =

Wk −1 Wk −1 1    k ; wK k )|: |Pr(wk ; wK k ) − Pr(w 2 w=0


(7)

w =0

With the help of D(SCkd ), the general information pro"t of CMs speci"c to the color channels is determined. Fig. 3 (left) shows the Kolmogorov distance values of 30 images of the VisTex database (see Section 6) corresponding to RGB and LUV.

C. Palm / Pattern Recognition 37 (2004) 965 – 976

According to RGB, D(SCkd ) ranges from 0:2 to 0:8. Compared to the baseline of zero (no information pro"t), this range shows the bene"t of texture analysis by CM in contrast to pure histogram analysis. Obviously, the Kolmogorov distances of R, G and B are correlated. Nevertheless, for single textures di?erences up to 0:2 between two of them have to be recognized. This indicates the non-redundancy of channel-speci"c texture measures in the RGB color space. Similar to RGB, D(SCkd ) for LUV range from 0:1 to 0:8. Especially the curve according to L show noticeable parallels to the RGB related curves. This indicates the strong intensity relation of the RGB color channels. In contrast to RGB, only the Kolmogorov distances for U and V are strongly correlated, but they show signi"cant di?erences to the L values. Hence, the information pro"t of pure color texture analysis is di?erent from the information pro"t of intensity-based texture analysis. Most frequently, with values between 0.2 and 0.7 the information pro"t by intensity texture recognition is higher than that for pure color texture analysis. Nevertheless, comparing to the zero baseline pure color texture features are proved to provide additional information of substance for color texture analysis.

5.2.1. Background The SCMs ignore the correlation between color channels, although it may provide additional textural information [16]. Taking this into account, we introduce multi-channel Co-occurrence matrices (MCMs) MCkd1 ; k2 , which count the pairwise occurrence of values in di?erent bands k1 and k2 with k1 = k2 : Pr(wk1 ; wK k2 )

5.2.2. Information measurement In Section 5.1.2 we introduced the Kolmogorov distance as a measure of the information gain using Co-occurrence matrices instead of histograms. After de"nition of MCMs, this measure is now extended to quantify the relation of color and texture for each image objectively. In analogy to Eq. (5), the adjacency of two values w and wK across two color channels, k1 and k2 , is a multiplicative combination of the color histogram distribution Pr 0 (wk1 ; w˜ k2 ) and the channel-speci"c distribution Pr(w˜ k2 ; wK k2 ). The product is normalized by Pr(w˜ k2 ) and summed up over all possible w˜ k2 :  k1 ; wK k2 ) Pr(w =



For MCMs the symmetry is lost. Since (MCkd1 ; k2 )T = (MCkd2 ; k1 ) is valid obviously, the mean matrix of (MCkd1 ; k2 ) and (MCkd2 ; k1 ) is symmetric again, which results in K 2 − K di?erent mean matrices. Again, Haralick-features are applied on the mean MCMs to extract 16·(K 2 −K)=2 rotational invariant multi-channel Co-occurrence features (MCFs). The interpretation of MCF in the LUV color space has to di?erentiate between intensity dependent and independent parts, respectively. Whereas the MCF for (k1 ; k2 ) = (U; V ) represents pure color neighborhood, the L-related MCF (k1 ; k2 ) = (L; U ) and (k1 ; k2 ) = (L; V ) are intensity dependent and characterize the correlation of color and intensity with regard to texture. Assuming the spatial distance vector d = 0, the representation of MCM in Eq. (8) is related to the two-dimensional color histogram approximation, because no spatial adjacency but channel adjacency is regarded. For this, histogram features HCF’s can be extracted directly from the MCM. The

(9)

Pr 0 (wk1 ; wK k2 )  w˜

(8)

1 · Pr 0 (wk1 ; w˜ k2 ) · Pr(w˜ k2 ; wK k2 ): Pr(w˜ k2 )

Note, that Eq. (9) assumes the stochastic independency of Pr 0 (wk1 ; w˜ k2 ) and Pr(w˜ k2 ; wK k2 ). The parallel color texture concept (Section 2) is based on this independency. The es k1 ; wK k2 ) in Eq. (9) is computable timated distribution Pr(w k1 ; k 2 k using SCd and MC0 . For the actual distribution Pr(wk1 ; wK k2 ) conditional probabilities are needed:

=

:= MCkd1 ; k2 (w; w) K := Pr(fk1 (p1 ) = w ∧ fk2 (p2 ) = w| K |p1 − p2 | = d):

color histogram distribution is here noted as Pr 0 (wk1 ; wK k2 ). Color histograms are frequently used to characterize the image color globally without considering local texture information. Hence, MCM provide the opportunity to study the e?ect of texture and pure color analysis in one uni"ed framework.

w∈W ˜

5.2. Multi-channel Co-occurrence matrices

969

1 · Pr(wk1 ; w˜ k2 |w˜ k2 ; wK k2 ) · Pr(w˜ k2 ; wK k2 ): Pr(w˜ k2 ) (10)

The Kolmogorov distance D(MCkd1 ; k2 ) quanti"es the information pro"t of multi-channel color texture analysis in contrast to the parallel concept by separated analysis of color and texture: D(MCkd1 ; k2 ) W −1 W −1

k1 k2 1    k1 ; wK k2 )| |Pr(wk1 ; wK k2 ) − Pr(w = 2 w=0 w=0 K

W −1 W −1 W −1

k1 k2 k2 1    Pr(w˜ k2 ; wK k2 ) = 2 w=0 w=0 Pr(w˜ k2 ) K w=0 ˜

· (|Pr 0 (wk1 ; w˜ k2 ) − Pr 0 (wk1 ; w˜ k2 |w˜ k2 ; wK k2 )|): D(MCkd1 ; k2 )

(11)

reaches zero, if either the texture image is non-colored, (identical red, green and blue channel), or the color image is non-textured (each pixel value has the same neighbor). In each case, MCMs provide no additional information to SCMs and histograms, respectively.

970

C. Palm / Pattern Recognition 37 (2004) 965 – 976

Fig. 3 (right) shows D(MCkd1 ; k2 ) for the natural color textures of the VisTex database (see Section 6) for RGB and LUV, respectively. According to RGB, the values range from 0.1 to 0.7. Considering the baseline of zero for stochastic independency of color and texture, this range indicates clearly the high information pro"t using the novel multi-channel Co-occurrence features in contrast to any parallel method. The similarity of D(MCkd1 ; k2 ) for di?erent color channels regarding to each image separately allows their interpretation as a global color texture feature rather than a channel-speci"c feature. In contrast to RGB, the curves for di?erent channel combinations of LUV vary signi"cantly. Whereas D(MCkd1 ; k2 ) for the intensity independent combination (k1 ; k2 ) = (U; V ) reaches values in [0:7; 1:0], the distances according to the combinations of color and intensity channels are mostly less than 0.3. Therefore, the information pro"t analyzing intensity independent color textures is very high and the existence of pure intensity independent color texture obvious.

6. Experimental evaluation Studying the Kolmogorov distance, we already showed the existence of intensity independent color textures as well as the usefulness of the integrative color texture analysis. These results will be con"rmed by experimental evaluation.

Fig. 4. Images of the VisTex database (left–right, up–down): Bark0, Bark4, Bark6, Bark8, Bark9, Brick1, Brick4, Brick5, Fabric0, Fabric4, Fabric7, Fabric9, Fabric11, Fabric13, Fabric16, Fabric17, Fabric18, Food0, Food2, Food5, Food8, Grass1, Sand0, Stone4, Tile1, Tile3, Tile7, Water6, Wood1, Wood2 [31].

6.1. Experimental setup To allow the generalization of the classi"cation results, two di?erent image sets are used. From the VisTex dataset [31] of MIT we choose 30 images of size 512×512 (Fig. 4), each of which contains one natural colored texture. One texture class is built splitting one image into 64 non-overlapping sub-images of size 64 × 64. Therefore, inter-class variation corresponds to local variation of the full-size images. This dataset is here addressed by DS 1 . The second dataset called DS 2 contains images of natural barks [32], built up in the context of the PhD thesis of Lakmann [33]. The images consist of one type of bark and background. Each image shows a di?erent tree, which is previously classi"ed into one of six classes (Fig. 5). Therefore, the inter-class variation reIects the natural surface aberration of trees of the same kind. Each class consists of 68 images of size 384 × 256 yielding a collection of 408 images. Since we apply an image classi"cation not a texture segmentation, the image border is excluded de"ning a Region-of-Interest of "xed size 300 × 200 located at the image center. For classi"cation, the leaving-one-out method is used to guarantee strict separation of test and training set in combination with the maximization of the number of training images. A 5-Nearest-Neighbor classi"er using the Euclidian vector distance was employed. To achieve a comparable range of values for di?erent features, a normalization

Fig. 5. Images of the BarkTex database, one example for each class (left–right, up–down): birch, beech, spruce, pine, oak, robinia [32].

C. Palm / Pattern Recognition 37 (2004) 965 – 976

971

Table 1 The notation de"ned in this table will be needed for understanding the following tables of classi"cation results Space

Feature vector

Abbreviation

gray RGB RGB RGB LUV LUV

GCF SCF MCF HCF SCF MCF

m(L) m(R), m(G), m(B) m(RG), m(RB), m(GB) m(RG0 ), m(RB0 ), m(GB0 ) m(L), m(U ), m(V ) m(LU ), m(LV ), m(UV )

RGB RGB RGB RGB RGB RGB RGB RGB RGB RGB LUV LUV LUV

m(R) × m(G) × m(B) m(RG) × m(RB) × m(GB) m(RGBSC ) × m(L) m(RGBMC ) × m(L) m(RG0 ) × m(RB0 ) × m(GB0 ) m(RGB0 ) × m(L) m(RGB0 ) × m(RGBSC ) m(RGB0 ) × m(RGBMC ) m(RGBSC ) × m(RGBMC ) m(RGB0 ) × m(RGBSC ) × m(RGBMC ) m(U ) × m(V ) × m(UV ) m(LU ) × m(LV ) × m(L) m(UVII ) × m(LUVID )

m(RGBSC ) m(RGBMC ) m(RGBSC ; L) m(RGBMC ; L) m(RGB0 ) m(RGB0 ; L) m(RGB0; SC ) m(RGB0; MC ) m(RGBSC; MC ) m(RGB0; SC; MC ) m(UVII ) m(LUVID ) m(LUVII; ID )

E 16 16 16 8 16 16 48 48 64 64 24 72 72 96 96 120 48 48 96

The gray-scale Co-occurrence features, single-channel Co-occurrence features, multi-channel Co-occurrence features and histogram Co-occurrence features are here noted as the feature vectors GCF, SCF, MCF and HCF, respectively. The abbreviations of the feature vectors m make clear, which color channels are involved. These vectors are combined by Cartesian product ×. The indices MC, SC and 0 indicate multi-channel Co-occurrence, single-channel Co-occurrence and histogram features, respectively, in the RGB color space. For LUV, the feature vectors are combined to intensity independent (II) and intensity dependent vectors (ID). E symbolizes the dimension of the feature space.

according to variance e and mean e of each dimension e is done separately exploiting the training data TR: m e − e me = (12) e with e =

1  me ; M m∈TR

e =

 1 (me − e )2 : M − 1 m∈TR

(13)

The eth element of an E-dimensional feature vector m and the corresponding normalized element is denoted with me and me , respectively. The optimal parameter d of a CM depends on the resolution of the texture. Zucker et al. developed a method to determine this optimal distance [34]. However, this optimum refers to a distinct texture and one single color channel. In best case, it can be generalized to the entire texture class. In our classi"cation experiments several classes are involved. Hence, an optimal distance for all color channels and all classes cannot be determined. Consequently, we studied the color textures according to several distances. The following experiments are performed with d = 0 (histogram) and d = 1; 5; 10; 15; 20. We combined the previously mentioned SCFs and MCFs and compared them to pure gray-scale texture and pure color histogram classi"cation results. To give a clear reference to

the feature vectors m used in the experiments, the abbreviations of the single- and combined feature vectors are listed in Table 1. Note, that the combinations in LUV are grouped into intensity independent (II) and intensity dependent (ID) whereas in RGB they are grouped into SCFs (SC) and MCFs (MC). 6.2. Results The classi"cation results for DS 1 and DS 2 are listed in Tables 2 and 3, respectively. The CM-representation of the color histogram allows the direct comparison of the parallel and the integrative color texture concept (Section 2). Rows 1–3 of both tables show the results of gray-scale texture features m(L) and the color histogram, described by its Haralick features m(RGB0 ). Obviously, the importance of color and intensity texture is quite di?erent for the two datasets. Whereas the HCFs for DS 1 performs with 0:938 better than the GCFs with 0:855, the texture features show better discrimination ePciency with 0:787 compared to 0:654 for m(RGB0 ) according to DS 2 . For both datasets the parallel concept combining the two single features outperform with 0.977 and 0.838 for DS 1 and DS 2 , respectively, the results of the single features.

972

C. Palm / Pattern Recognition 37 (2004) 965 – 976 Table 2 Classi"cation results for dataset DS 1 according to di?erent color texture feature combinations DS 1 0

1

5

10

15

20

1 2 3

m(L) m(RGB0 ) m(RGB0 ; L)

— 0.938 —

0.855 — 0.977

0.652 — 0.945

0.544 — 0.929

0.531 — 0.910

0.518 — 0.928

4 5 6

m(R) m(G) m(B)

— — —

0.868 0.869 0.896

0.771 0.779 0.819

0.726 0.697 0.738

0.702 0.691 0.720

0.691 0.677 0.704

7 8 9

m(RGBSC ) m(RGBMC ) m(RGBSC; MC )

— — —

0.951 0.960 0.971

0.884 0.870 0.903

0.826 0.862 0.887

0.811 0.850 0.864

0.793 0.864 0.869

10 11 12

m(RGB0; SC ) m(RGB0; MC ) m(RGB0; SC; MC )

— — —

0.974 0.972 0.977

0.951 0.933 0.946

0.928 0.920 0.928

0.906 0.925 0.917

0.913 0.926 0.920

13 14 15

m(RGBSC ; L) m(RGBMC ; L) m(RGBSC; MC ; L)

— — —

0.946 0.966 0.968

0.868 0.894 0.902

0.821 0.865 0.905

0.807 0.864 0.876

0.804 0.882 0.879

16 17 18

m(U ) m(V ) m(UV )

— — —

0.886 0.856 0.891

0.754 0.719 0.799

0.666 0.592 0.795

0.609 0.536 0.794

0.595 0.532 0.661

19 20 21

m(UVII ) m(LUVID ) m(LUVII; ID )

— — —

0.969 0.948 0.986

0.910 0.895 0.949

0.867 0.870 0.923

0.840 0.852 0.916

0.843 0.863 0.922

The di?erent results in one row refer to CM distances from zero (color histogram) up to 20. The feature combinations are divided into semantically interrelated blocks: parallel color texture concept (Rows 1–3), integrative color texture concept in the RGB color space (Rows 4 –9), combination of integrative features with pure color and pure intensity texture features, respectively (Rows 10 –15) integrative color texture concept in the LUV color space (Rows 16 –21).

The results for SCF of the RGB color space are presented in Rows 4 – 6. The best results are obtained by m(B) with d = 1 and correctness 0.896 and by m(B) with d = 5 and correctness 0.794 for DS 1 and DS 2 , respectively. A general superior performance of the B channel is not plausible and depends on the image dataset. All single results for both datasets are superior to the corresponding pure gray-scale texture result. These classi"cation rates for m(R), m(G), m(B) are improved signi"cantly (unidirectional Binomial test statistic, p ¡ 0:05) by combining them to m(RGBSC ) (Row 7). According to m(RGBSC ) we achieved a correctness of 0.951 and 0.828 for DS 1 and DS 2 , respectively. The corresponding combination of MCFs to m(RGBMC ) yield with 0.960 better and with 0:795 even worth results for DS 1 and DS 2 . Combining m(RGBSC ) and m(RGBMC ) (Row 8), the dimension of the feature space rises to 96 and, hence, the complexity of classi"cation task rises, too. Nevertheless, the results improved to 0.971 and 0.841, respectively.

The results of Rows 10 –12 in Tables 2 and 3 allow the evaluation of possible advantages combining RGB-related integrative color texture features with the pure color histogram features. For DS 1 the combination of SCF, MCF and both with histogram features showed slightly increasing results, respectively. The overall combination m(RGB0; SC; MC ) performed best with a correctness of 0.977. According to DS 2 the combination of SCF with histogram features improve the performance of SCF with a correctness of 0.841. In contrast, the performance of the overall combination decreased to 0.808. However, the feature space dimension and, therefore, the complexity of the classi"cation problem, rose to 120 for this combination (see Table 1). All results of texture feature combination with the histogram features outperform the pure histogram features. Rows 13–15 show the classi"cation results combining RGB-based SCFs and MCFs with GCFs. For both datasets all color texture combinations performed better than the pure

C. Palm / Pattern Recognition 37 (2004) 965 – 976

973

Table 3 Classi"cation results of dataset DS 2 according to di?erent color texture feature combinations DS 2 0

1

5

10

15

20

1 2 3

m(L) m(RGB0 ) m(RGB0 ; L)

— 0.654 —

0.721 — 0.831

0.787 — 0.838

0.742 — 0.814

0.694 — 0.765

0.637 — 0.767

4 5 6

m(R) m(G) m(B)

— — —

0.723 0.745 0.740

0.775 0.777 0.794

0.708 0.740 0.750

0.664 0.689 0.703

0.639 0.627 0.708

7 8 9

m(RGBSC ) m(RGBMC ) m(RGBSC; MC )

— — —

0.787 0.807 0.809

0.828 0.795 0.841

0.784 0.747 0.789

0.708 0.705 0.767

0.666 0.738 0.746

10 11 12

m(RGB0; SC ) m(RGB0; MC ) m(RGB0; SC; MC )

— — —

0.826 0.812 0.824

0.841 0.798 0.808

0.806 0.759 0.813

0.794 0.764 0.803

0.779 0.746 0.795

13 14 15

m(RGBSC ; L) m(RGBMC ; L) m(RGBSC; MC ; L)

— — —

0.777 0.809 0.804

0.824 0.797 0.847

0.784 0.767 0.772

0.703 0.794 0.762

0.668 0.733 0.768

16 17 18

m(U ) m(V ) m(UV )

— — —

0.658 0.653 0.532

0.776 0.614 0.548

0.714 0.597 0.484

0.646 0.585 0.548

0.595 0.548 0.453

19 20 21

m(UVII ) m(LUVID ) m(LUVII; ID )

— — —

0.756 0.786 0.863

0.807 0.740 0.806

0.759 0.694 0.836

0.778 0.729 0.856

0.729 0.673 0.786

The notation corresponds to that of Tables 1 and 2.

gray-scale texture features. The combined features show similar results as the corresponding single features in Rows 7–9. Therefore, no tendency can be remarked which indicates the usefulness of these combinations. For both datasets the best result of this block is achieved by the overall combination m(RGBSC; MC ; L). It yields a correctness of 0:968 and 0:847 for DS 1 and DS 2 , respectively. The results related to the intensity independent single feature vectors of the LUV color space are listed in Rows 16 –18. For DS 1 the best performance is achieved with 0:891 by m(UV ) and d = 1. On the other hand, for DS 2 the best correctness of 0:776 is achieved by m(U ) and d = 5. Therefore, both are below the respective maxima according to the single RGB features (Rows 4 – 6). For both datasets the results are enhanced by combining the single features to m(UVII ) (Row 19). With 0:969 and 0:807 for DS 1 and DS 2 , respectively, the improvement is signi"cant (p ¡ 0:05). The intensity independent texture features are in both cases superior to the intensity dependent features (Row 20). Note, that m(L) is inherently integrated into m(LUVID ) as SCF1 of the LUV color space. The best results are obtained by the combination of both, intensity independent and intensity dependent features (Row

21). These are superior not only to other combinations of the RGB color space but to the parallel color texture concept m(RGB0 ; L) as well. The results 0:986 and 0:863 for DS 1 and DS 2 , respectively, refer both to d = 1. Taking d into account, a strong tendency to an optimum distance for each dataset can be stated. All superior results for each experiment block are found for d = 1 and (except one) for d = 5 according to DS 1 and DS 2 , respectively.

7. Discussion In our experiments, gray-scale texture and color histogram analysis were compared with novel color texture features. Additionally, two di?erent concepts of color texture analysis, parallel and integrative, were evaluated. This was possible for the "rst time, because in the framework of CMs color histograms and textural features are both well described. The two image datasets used are rather di?erent in content, color texture appearance as well as discrimination capability of color histogram and gray-scale texture. Nevertheless, the tendencies of the results are quite similar. Therefore,

974

C. Palm / Pattern Recognition 37 (2004) 965 – 976

it is possible to generalize them to large application "elds dealing with natural color textures. It has been shown, that color information improve the results of gray-scale texture features [14]. Our experiments support these results. The combination of color histograms and gray-scale texture features shows a high supplementation ability. The results of this parallel color texture concept are just slightly below the best results we achieved by integrative color texture features. Whereas the parallel concept is most commonly applied in the color image processing community, this paper introduced novel integrative color texture features to learn more about the aspects of color texture and develop new methods to model them. According to SCMs, we realize noticeable di?erences between the single-channel results. This indicates di?erences in the textures of di?erent color channels and relativizes their correlation. This observation supports our evaluation of Kolmogorov distances in Section 5. Additionally, the channel-speci"c features supplement each other resulting in a signi"cant better performance for the combination features. This enhancement is noticeable in particular considering the increase of dimensionality of the feature space. Because the gray-scale texture results are below all single-channel results, the transform from color to gray values is lossy with respect to textural features. The novel method of MCMs improve the results of the SCMs taking the correlation between color channels into account. With the help of the Kolmogorov distance, we demonstrated that the multi-channel modeling is not equivalently realized by the single-channel features in combination with the color histogram. Therefore, MCMs measure new color textural features. The low redundancy of both integrative methods is also shown by the experimental evaluation. The combination of m(RGBSC ) and m(RGBMC ) leads to increasing correctness although the feature space dimensionality rises by a factor of two. The combination of the SCFs with the histogram features show rather similar results to the parallel color texture concept. This could be expected since the only di?erence is the already stated wavelength dependency of colored texture. Unfortunately, the information pro"t of texture modeling subsequently on the color channels is over-compensated by the disproportionate extension of the feature space. Nevertheless, color histogram and single-channel texture analysis supplement each other. Therefore, the classi"cation correctness rises in comparison to both single components. On the other hand, the combination of m(RGBSC ) with the GCFs yields worse results according to both components. Again, this shows the high redundancy of GCFs and SCFs. Studying the redundancy of MCFs with HCFs and GCFs, respectively, in both cases better results in comparison with the single components can be stated. Therefore, a new kind of texture information is measurable by the multi-channel analysis. In addition to RGB, the experimental evaluation take the LUV color space into account. Although the methods

of single- and multi-channel color texture analysis are the same, the interpretation of the results has to be adapted. Because of the two intensity independent channels U and V we achieve three intensity dependent and three intensity independent matrices. Obviously, the intensity independent features model a completely di?erent kind of texture. Although the RGB related color texture features consider both color and texture, the texture is again intensity based, even though this intensity is measured on a small wavelength interval. In contrast, the MCMs related to U and V characterize the Co-occurrence of pure colors. Because the three single feature vectors m(U ), m(V ) and m(UV ) represent di?erent parts of the color plane, they supplement each other and, hence, their combination yields to a signi"cant better classi"cation result. Additionally, the results to the intensity independent features are better than that of the intensity dependent. The novel pure color texture features in comparison with pure intensity texture features show the overall best performance. Again, this proves the fact low redundancy between these two concepts. 8. Conclusion In this paper, we presented novel integrative single- and multi-channel Co-occurrence matrices for color texture analysis and put intensity independent color textures in perspective. The multi-channel approach integrates color histograms into the Co-occurrence notation and allows the comparison of parallel and integrative approaches on the same level for the "rst time. Additionally, the Kolmogorov distance is recommended to measure quantitatively the information gain of the multi-channel approach. The advantages of the integrative methods were demonstrated by the Kolmogorov distance as well as by the results of the experimental evaluation. Nevertheless, because of the disproportional increase of feature space dimensionality, the best results according to RGB are achieved by the parallel concept of separate analysis of color by histogram and texture by intensity pattern analysis. However, the integrative color features based on the LUV color space show not only the existence of intensity independent color texture but its discriminative power. In combination with the common intensity dependent pattern, colored textures are more adequately described. Summary Color image analysis and texture classi"cation are two basic tasks in pattern recognition. The dependencies of both are still not clear and, hence, several concepts dealing with colored textures exist. The objective of this paper is to strengthen the basis for the discussion in the "eld of color texture recognition. To achieve this goal, we treated

C. Palm / Pattern Recognition 37 (2004) 965 – 976

"ve main "elds within the framework of Co-occurrence matrices: • Categorization of the concepts for color texture feature extraction. The identi"cation of these concepts enables an overview over the relevant literature and allows the comparison between these approaches. Two of them, the parallel and the integrative method, have been compared within the experimental section. • Extension of Co-occurrence matrices to multi-channel color texture characterization. Since Co-occurrence matrices are one of the most frequently used methods for gray-scale texture analysis, multi-channel matrices enlarges the toolbox of color texture analysis methods signi"cantly. • Application of Kolmogorov distance to the novel singleand multi-channel matrices. The Kolmogorov distance serves as a quantitative measure to determine the dependency of intensity textures and colors. Therefore, the discussion about this dependency in the scienti"c community can be adjudged objectively for each image separately. • Application of integrative Co-occurrence matrices to the RGB as well as the LUV color space. Since the LUV space separates one intensity channel L and two color channels U and V, Co-occurrence matrices according to U and V provide intensity independent and, hence, pure color texture features. This features are used to prove the existence of intensity independent color textures. • Classi"cation experiments on two large image databases. The large number of experimental data allows the generalization of the conclusions drawn on the basis of the results. In particular, the parallel and the integrative concept are compared in the framework of Co-occurrence matrices. Two main conclusions have to be emphasized. Firstly, the parallel concept of intensity texture analysis and separately color histogram analysis is suitable for color image classi"cation. Secondly, the existence of pure color textures as well as their supplementation ability with respect to intensity dependent textures features is proven. Therefore, the integrative color texture features introduced in this paper enable the characterization of intensity independent color textures for the "rst time. Using these novel features in combination with gray-scale texture and color histogram features the classi"cation correctness is increased up to 32%.

References [1] T.R. Reed, J.M.H. Du Buf, A review of recent texture segmentation and feature extraction techniques, CVGIP— Image Understanding 57 (3) (1993) 359–372.

975

[2] I. Tu"s, Automated fabric inspection based on a structural texture analysis method, in: Recent Issues in Pattern Analysis and Recognition, Springer, Berlin (1989) 377–390. [3] A. Bodnarova, M. Bennamoun, K.K. Kubik, Suitability analysis of techniques for Iaw detection in textiles using texture analysis, Pattern Anal. Appl. 3 (2000) 254–266. [4] R. Haralick, K. Shanmugam, I. Dinstein, Texture features for image classi"cation, IEEE Trans. Syst. Man Cybernet. 3 (1973) 610–621. [5] Lei Wang, Jun Liu, Texture classi"cation using multi-resolution markov random "eld models, Pattern Recognition Lett. 20 (2) (1999) 171–182. [6] M. PietikSainen, T. Ojala, Z. Xu, Rotation-invariant texture classi"cation using feature distributions, Pattern Recognition 33 (2000) 43–52. [7] A.K. Jain, F. Farrokhnia, Un-supervised texture segmentation using Gabor "lters, Pattern Recognition 24 (12) (1991) 1167–1186. [8] T. Randen, J.H. HusHy, Multichannel "ltering for image texture segmentation, Opt. Eng. 33 (8) (1994) 2617–2625. [9] Jing-Wein Wang, Chin-Hsing Chen, Wei-Ming Chien, Chih-Ming Tsai, Texture classi"cation using non-separable two-dimensional wavelets, Pattern Recognition Lett. 19 (13) (1998) 1225–1234. [10] C. Gotlieb, H. Kreyszig, Texture descriptors based on Co-occurrence matrices, Comput. Vision Graph. Image Process. 51 (1990) 47–52. [11] K. Messer, J. Kittler, A region-based image database system using colour and texture, Pattern Recognition Lett. 20 (1999) 1323–1330. [12] M. Hauta-Kasari, J. Parkkinen, T. Jaaskelainen, R. Lenz, Multi-spectral texture segmentation based on the spectral Co-occurrence matrix, Pattern Anal. Appl. 2 (4) (1999) 275–284. [13] K.Y. Song, J. Kittler, M. Petrou, Defect detection in random colour textures, Image Vision Comput. 14 (1996) 667–683. [14] S. Kukkonen, H. KSalviSainen, J. Parkkinen, Color features for quality control in ceramic tile industry, Opt. Eng. 40 (2) (2001) 170–177. [15] A. Drimbarean, P.F. Whelan, Experiments in colour texture analysis, Pattern Recognition Lett. 22 (2001) 1161–1167. [16] P.-H. Suen, G. Healey, Modeling and classifying color textures using random "elds in a random environment, Pattern Recognition 32 (1999) 1009–1017. [17] G. Van de Wouwer, P. Scheunders, S. Livens, D. Van Dyck, Wavelet correlation signatures for color texture characterization, Pattern Recognition 32 (1999) 443–451. [18] A.K. Jain, G. Healey, A multi-scale representation including opponent color features for texture recognition, IEEE Trans. Image Process. 7 (1) (1998) 124–128. [19] C. Palm, D. Keysers, T.M. Lehmann, K. Spitzer, Gabor "ltering of complex hue/saturation images for color texture classi"cation, Joint Conference on Information Science 2, Atlantic City, NJ, USA (2000) 45 – 49. [20] G. Paschos, Perceptually uniform color spaces for color texture analysis: an empirical evaluation, IEEE Trans. Image Process. 10 (6) (2001) 932–937.

976

C. Palm / Pattern Recognition 37 (2004) 965 – 976

[21] C. Palm, T.M. Lehmann, Classi"cation of color textures by Gabor "ltering, Mach. Graph. and Vision 11 (2/3) (2002) 195–219. [22] G. Healey, L. Wang, Illumination-invariant recognition of texture in color images, J. Opt. Soc. Am. A 12 (9) (1995) 1877–1883. [23] A. Rosenfeld, C.-Y. Wang, A.Y. Wu, Multispectral texture, IEEE Trans. Syst. Man and Cybernet. SMC-12 (1) (1982) 79–84. [24] T.M. Lehmann, C. Palm, Local line search for illuminant estimation in real world color scenes, J. Opt. Soc. Am. A 18 (11) (2001) 2679–2691. [25] B.A. Wandell, Foundations of Vision Science, Sinauer Ass., Sunderland, MA, USA, 1995. [26] T.S.C. Tan, J. Kittler, Colour texture analysis using colour histogram, IEE Proc. Vision Image Signal Process. 141 (6) (1994) 403–412. [27] M. Mirmehdi, M. Petrou, Segmentation of color textures, IEEE Trans. Pattern Anal. Mach. Intell. 22 (2) (2000) 142–159. [28] A. McCabe, T. Caelli, G. West, A. Reeves, Theory of spatiochromatic image encoding and feature extraction, J. Opt. Soc. Am. A 17 (10) (2000) 1744–1754.

[29] A. Bradley, P. Jackaway, B. Lovell, Classi"cation in scale-space: applications to texture analysis, 14th International Conference on Information Processing in Medical Imaging, Ile de Berder, France (1995) 375 –376. [30] A.N. Kolmogorov, Three approaches to the quantitative de"nition of information, Int. J. Comput. Math. 2 (1968) 157–168. [31] Database VisTex of color textures of MIT, USA: ftp://whitechapel.media.mit.edu/pub/VisTex. [32] Database of color textured barks of University of Koblenz, Germany: ftp://ftphost.uni-koblenz.de/outgoing/vision/ Lakmann/BarkTex. [33] R. Lakmann, Statistische Modellierung von Farbtexturen, Ph.D. Thesis, University Koblenz-Landau, FSolbach, Koblenz, 1998 (in German). [34] S.W. Zucker, D. Terzopoulos, Finding structure in Co-occurrence matrices for texture analysis, Comput. Graph. Image Process. 12 (1980) 286–308. [35] J.F.R. Ilgner, C. Palm, A.G. SchSutz, K. Spitzer, M. Westhofen, T.M. Lehmann, Colour texture analysis for quantitative laryngoscopy, Acta Otolaryngol. 123 (2003) 730–734.

About the Author—CHRISTOPH PALM studied Computer Science at the Aachen University of Technology (RWTH), Germany, and received his Diploma degree in 1997. Between 1997 and 2001 he was Ph.D. student at the Institute of Medical Informatics at the University Hospital of the RWTH. During his Ph.D. research, his research interests covered all aspects of pattern recognition, texture classi"cation and color image processing. He is co-editor of a proceedings volume and author of several scienti"c conference and journal papers. He is peer reviewer of the Optical Society of America. In 2001 Christoph Palm joined the Aixplain AG, Aachen. Beside image processing, his research focus is now enlarged to information retrieval and statistical machine translation.