- Email: [email protected]
Representing the color aspect of texture images
Pattern Recognition Letters 15 (1994) 191-197 North-Holland
February 1994
PATREC 1160
Representing the color aspect of texture images J. Scharcanski a, J.K. Hovis b and H.C. Shen a a Pattern Analysis & Machine Intelligence Lab, Department of Systems Design Engineering, University of Waterloo, Waterloc Ontario, Canada N2L 3G I b School of Optometry, University of Waterloo, Waterloo, Ontario, Canada, N2L 3G1
Received 3 October 1992 Revised 17 March 1993
Abstract Scharcanski, J., J.K. Hovis and H.C. Shen, Representing the color aspect of texture images, Pattern Recogmtion Letters 15 (1994) 191-197. The problem of representing the color aspect of texture images is analyzed in this paper. We propose that a color codebook be used to represent the color aspect of texture images. This color codebook is built adaptively from the problem prototypes and allows texture discrimination based on the color aspect by using a simple similarity measure.
Keywords. Texture analysis, computer vision, color image, analysis, neural networks, vector quantiTation.
I. Introduction
Real visual textures are in fact colored textures. They m a y present the same colors a n d different structural patterns, different colors in the same structural patterns, or different colors and different structural patterns. Therefore, a texture description should include both the color and structural aspects. Various m e t h o d s have been p r o p o s e d to represent the structural aspect o f visual textures, but, surprisingly, representation o f the color aspect has not received as much attention [ 5 ]. In this paper, we describe a technique to represent the color aspect o f texture images, as one c o m p o n e n t o f a full texture representation. In previous work, characteristic colors have been p r o p o s e d as the feature set representing the color aspect o f visual textures [4]. This set contains the Correspondence to: J. Scharcanski, Pattern Analysis & Machine Intelligence Lab, Department of Systems Design Engineering, University of Waterloo, Waterloo, Ontario, Canada N2L 3G 1.
smallest n u m b e r o f colors satisfying some criteria and is o b t a i n e d by a lateral inhibition network. However, the set o f characteristic colors is image dependent, and distinct images o f the same texture m a y present different characteristic colors. This occurs because colors are noisy, a n d color cluster centroids m a y not corres p o n d from sample to sample. The consequences o f this fact m a y be disastrous for color texture classification a n d discrimination, because it is not trivial to compare texture representations when the feature sets are particular to each image. Therefore, in order to obtain consistent texture classification results, some organization o f the occurring colors is necessary. In this work, we describe how a color codebook can be used as a s t a n d a r d to represent the colors o f a texture. This color codebook is o b t a i n e d a d a p t i v e l y based on the texture prototypes o f the problem. Before describing the color codebook, we will overview how the characteristic colors set is obtained.
0167-8655/94/$07.00 © 1994 - - Elsevier Science B.V. All rights reserved SSDI 01 6 7 - 8 6 5 5 ( 9 3 ) E 0 0 2 0 - O
191
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
2. S e l e c t i n g a characteristic colors set
Usually, colors are fuzzy in color space [3]. This imprecision results in ambiguous representations for computer classification and discrimination purposes. In order to minimize the ambiguity in color texture representation, we use a representative colors approach. With this approach, the smallest representative color set (according to established criteria) defines a feature set for computer color texture representation. The elements of this feature set are the characteristic colors of the texture. These colors are obtained by a parallel distributed approach, and, to each color, we assign a measure of importance [ 4 ]. This parallel distributed approach consists of a network of competitive linear units. A color is assigned to each network unit, and the network units are organized according to the colors they represent. We assume that a unit un is an immediate neighbor of a unit Urn if they correspond to colors that are immediate neighbors in the RGB color space. Each network unit un is associated with an occurring color xn in the input pattern, and its activation, In, is the relative frequency ofxn: N
In =f(Xn) = f ( x n ) / ~ f(xp)
(1)
p=l
where f ( x n ) is the frequency of occurrence of xn in the input image, and N is the number of occurring colors. Thus, the initial state of the network depends on the input color distribution. Starting with this initial state, the network achieves a final (steady) state by taking into account the lateral interactions of units. A network unit un (with activation In) is inhibited by all units that correspond to colors within a distance Rb in the color space. The variable Rb is defined as 2 ~"~-b, where bmaxis the maximum number of bits available for each channel (i.e., red, green and blue), and b is the number of bits effectively used by each channel. Usually, bm~xis pre-defined, and b is a color resolution parameter selected for a particular application. The unit lateral interactions are described by: U
I ~ - v ~= Iprnv I" = f
lfln-.-en
=1
.
t ~
,
0
otherwise
(2)
where Ig denotes the new unit activation level and 192
February 1994
U= 2 3b The inhibition coefficient rnp of the neighboring units equals r for those units within a distance Rb of Un, and is 0 otherwise. The threshold ~ determines when un is in steady state. We have adopted En=~, where e is in (0, 1]. If the frequency distribution of the occurring colors is multimodal, and the peaks are well separated so that their mutual lateral inhibition can be neglected, the network computation converges to high activation units (i.e., the peaks of the frequency distribution). This computation may be interpreted as a feature selection process, where, given the constraints, the smallest feature set is chosen in order to minimize the information loss. If entropy is expressed as Norig
H = Z f(xn) logf(xn) n=l
for a given number of colors Nong, Ho,~ is the entropy of the original color set, and Hch,r is the entropy of the characteristic colors set, then the difference between Ho~ig and nchar is minimized when the Nchar colors with higher relative frequency f(xn) are chosen. As a consequence, the color space is partitioned into mutually exclusive regions with centroids corresponding to distribution peaks. However, in general the mutual inhibition of nearby peaks is non-negligible. In this case, the network adapts to the local shape of the distribution, and the computation converges to local representative colors. A color is selected as characteristic because of its particular location, regional dominance, or both. The number of selected characteristic colors (Nchar) depends on the network parameter setting and the initial distribution. Whether Nchar is the optimal set to minimize information loss in the general case has not yet been determined. After the network has achieved steady state, a set of representative colors has been selected for a certain input pattern. Each color is associated with one of the remaining active units Um ( m = 1, 2 ..... V and V~< U). These are the texture characteristic colors Ym. Given this color set, the input color texture image is quantized and a final image is obtained. That is, each color xn at an input image position (p, q) is mapped to a characteristic color Yk which is its nearest neighbor in the RGB space (i.e., Yk minimizes d(xn, Ym), Vm). The measure of importance MI(ym) of each char-
Volume 15, Number 2
PATTERN RECOGNITIONLETTERS
acteristic color corresponds to the color relative frequency f ( y , , ) in the quantized image. Therefore characteristic colors that are more frequent, are more important in the texture color aspect.
3. Texture discrimination based on the color aspect
A variety of features can be used to describe the color aspect of a texture (e.g. associating colors with the textural elements). In this work, the color aspect is described in terms of the characteristic colors and their respective measure of importance. Our approach is based on a problem-specific color codebook as a standard representation. The use of this color codebook requires two phases. Initially, the color codebook is built adaptively by estimating the expectations of the characteristic colors presented by the problem prototypes. In the next phase, the colors of an input texture image are described in terms of the color codebook. Using the codebook description and a simple similarity measure, textures can be discriminated based on the color aspect.
prototypes using the following rules. Clusters that overlap totally are assigned to the same prototype. Clusters that do not overlap are assigned to distinct
prototypes. Clusters that overlap partially are those that occur within a distance of Ro units from each other in the color space. If Roe (0, Rb], then there is high representation uncertainty, because the occurring cluster centers are within a distance of Rb units from each other and, by definition, belong to the same cluster. This uncertainty is resolved by taking the weighted average of the cluster centers, lfRo > Rb, the clusters are assigned to distinct codebook prototypes. Basically, we have described a smoothing approach for uncertainty reduction in color texture representation [ 1 ]. This approach is justified if we consider that colors in the color space are formed by the addition of noise ~/i to a color prototype c, = (ri, gi, b,). Under this assumption, a characteristic color y~ may be expressed as: yi=ci"~'?]i
(3)
.
It is assumed that within the neighborhood of every color c~ the noise has zero average value. By averaging M different noisy samples we obtain:
3. I. Building a color codebook
1 M --
In assigning colors to classes, we can use either a supervised or an unsupervised approach. The supervised approach requires that labeled patterns (i.e., colors) are presented so that the classifier may learn the mapping between the input and the classification space. However, this approach may lead to complex classification problems due to the fact that colors tolerance volumes are non-uniformly distributed in the color space; particularly, if the color samples are labeled according to the human color perception. Because of these potential classification problems, we chose a mechanism that segregates the input patterns (i.e., colors) into clusters in an unsupervised manner. This unsupervised method creates a color codebook by identifying several prototypes, or exemplars, that can serve as color cluster centers. The input patterns are the characteristic colors of the texture samples and their respective measures of importance. The prototypes in the color codebook are expectations calculated dynamically based on the presented texture samples. This method assigns color clusters to the codebook
February 1994
Yi=
~ Mj=I
Yi.
(4)
Then it follows that
E{~)=c,
(5)
and a~=
a~,
(6)
2 is the varwhere E{~,} is the expected value o f ~ , ay7 iance ofyi and a~, is the variance ofq~. Equations (5) and (6) indicate that as M increases, the variability decreases and y,. will approach c,. In the actual building of the color codebook, the prototypes are obtained sequentially as the texture samples are presented. The characteristic colors y~ of the first sample presented are taken as codebook prototypes c~. For the subsequent samples the following procedure is executed. 1. Compute the characteristic colors and their respective measure of importance. 193
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
2. For each characteristic colory~, that is the nearest neighbor of the codebook prototype ck: (a) if d(yi, Ck) <
Crkr-.~ ~jYjr X f(Yjr) "bYir x f(Yir) ~jf(Yjr) +f(Yir) ' c'~
=
Y.J(Yjb ) + f(Y,b ) where (Yjr, Yjg, Yjb) is denoting the characteristic colors yj previously assigned to the prototype Ck; (b) ifd(y~, Ck) >Rb then includey~ as a newprototype in the color codebook.
3.2. Texture discrimination If a similarity measure is available, then textures can be discriminated based on the color aspect. According to the color codebook approach, the occurring colors in a texture image are mapped to the codebook prototypes using the nearest neighbor criterion (given a color tolerance). The measure of importance associated with each one of the codebook prototypes corresponds to its relative frequency of occurrence in the texture image. Therefore, a texture of index i is represented in terms of this codebook by:
C~={(cj,f(cy)) [ j = 1, ..., L}
codebook. The color codebook may be considered as a vector whose elementsf(cj) assume values in the range [0, Nimg], and Z j f ( c j ) =Nirng, where Nimg is the number of pixels of the input image. Two texture representations (i.e., codebooks) may then be compared by using the angle 0 between two vectors Cm and C. as a simple similarity measure. Because all vector elements are positive, Oe [0, n / 2 ] , so that we can use the following similarity measure p.,.:
194
(Cm'C.) [ 0 , 1] IlCmll'llC. II~
Proper representation of color tolerance volumes for each of the codebook prototypes is not trivial. Initially, we do not know much about the actual color distribution in color space and assume that the volumes of tolerance are spheroids of radius R b. Based on this assumption, the codebook prototypes are calculated as centroids of overlapping volumes of tolerance. At this stage, we prefer not to assume any predefined volumes of tolerance for the codebook prototypes. Instead, the actual distribution of the occurring colors in the vicinity of the codebook prototypes is used. Note that an occurring color xi is considered to be in the vicinity of a codebook prototype ck, if the nearest neighbor ofxi is Ck. A representation of the occurring colors distribution, Dk, in the vicinity of a codebook prototype Ck should be unbiased, and not assume a priori any particular probability distribution (e.g. gaussian) 1. Such a representation may be obtained based on the covariance matrix [Mk] of the distribution Dk, with the color ck as the mean. The matrix [Mk] is then defined as [ 31:
(7)
where f ( c j ) is the frequency of the color prototype cj, and L is the number of color prototypes in the
x
If the value of pro, is high, the two textures (m and n) are similar in terms of the color aspect. This approach may be used to assign textures to classes based only on the color aspect.
'
C'kb = 2jYjb x f(Yjb ) q-Yib x f(Yib )
P m ' = 1 - -2c o s -1
where IICII is the vector length, and (Cm" Cn) is the scalar product of the two vectors.
3.3. Color tolerance volumes
~jYjg × f(Yjg) + Y~gx f(Yig)
ZJ(yjA + f(y,~)
February 1994
(8)
F
Mk=ICG~ LC~
~ C~
CGBI a~_.]
(9)
where a 2 is the variance of the variable X, and Cxris the covariance of the variables X a n d Y, where X a n d Yare R, G or B. Note that Cxr= Crx. Unbiased estimates about the distribution Dk can be obtained directly from the covariance matrix [Dk] by a number of linear combinations of the variables R, G and B. Let us consider the following linear combinations: R, G, B, R+G, R+B, G + B a n d R + G + B . Preliminary work shows that occurring colors in digital images are not adequately modelled by gaussian distributions (personal communication of Dr. Ed Jernigan, Dept. of Systems Design Eng., University of Waterloo, Canada).
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
These combinations may be considered as variables related to the distribution Dk, whose variances are expressed in terms of the covariance matrix elements [2]:
a2+o=a~ +a~ + 2Cgo, a ~ + , = a ~ + a2 + 2CR,, a~+e = a~ + a~ + 2Coe, 2
2
2
2
¢7R+G+ B = C7g + ¢7G + ~7B
+2CR~ +2CR, + 2 C o , ,
(10)
and the means are given respectively by:/tR,/zo,/tB, /tR+~; (=~R+/~o), /IR+B (=#g+lt,), ltC+, ( = J Z G - J - I ~ B ) , ,UR + G + B ( = ~ R " { - t l 2 G " ~ J ~ n ) .
Although the linear combinations of R, G and B provide an estimate of the occurring colors distribution Dk in the vicinity of a codebook prototype ck, we need a criterion to decide whether an input color x~ can be assigned to Ck. This criterion should only use the information that is provided by the Dk estimate. Tchebyschev's inequality 2 provides a criterion and is expressed by:
1 P( I X - u x l >~nXax) <~--~
(1 1 )
where P( ) is a probability measure, and ne~ +. Therefore, we can establish a number of standard deviations n × ax such that the error probability ( 1/ n 2) is small and consider an input color x~ (r~, g~, b,) as belonging to the vicinity of Ck if the following inequalities are true:
Ir~--l~gl <~n×aR , Ig~-/tol < n × a ~ , Ibi-/L. I 4 n X a B , I (r,+g~)-/IR+cl ~nX~R+o, I(r,+bi)--#g+.l~nXag+8,
[ (gi + bi) - Ito+B I <
(12)
2 This inequality does not assume any distribution beforehand. However in some practical situations assumptions about the distribution may be necessary to improve the power of the statistical criterion. These assumptions may be justified based on the provided estimates.
February 1994
Using this approach we can decide whether an input color xi which is in the vicinity of a certain codebook prototype Ck, belongs to the volume of tolerance. This decision criterion is unbiased because no previous assumptions are made about the local distribution of the occurring colors. Note that the various combinations of the R, G and B channels resemble some psychophysical quantities associated with colors, such as the luminance ( R + G + B ) , the individual responses of each channel (R, G, and B), the blue-yellow ( R + B ) and the red-green ( R + G ) dimensions. Also note that we can introduce unexpected events within the color aspect representation by including a dummy color in the color codebook (c~:+~). Therefore, ifxh does not belong to the vicinity of cn, Vn, n = 1, ..., L, then the input color Xh at the image position (p, q), which was not expected considering the color prototypes, is assigned to the dummy color (Xh~CL+ ~). This procedure might be useful in evaluating the similarity between an input pattern and the stored class prototypes.
4. Experimental results In this section, we demonstrate that textures can also be compared in terms of their color composition, irrespective of their structure. A set of texture images with resolution 256 × 256 was used. The following parameter settings were used in these experiments: b = 4 (i.e., a palette with 4096 colors) and R b = 16. For the network simulation, the parameters were chosen as r = 10 -3 and e = 10 -6. Figure 1 shows the set of texture images used. Textures wpl, wp2, wp3 and wp4 contained 13, 11, 15 and 24 characteristic colors respectively. The codebook obtained from these four textures contained 45 color prototypes. Table 1 lists the similarity values p,~, for the color aspect representation of the four textures (i.e., the codebook vectors Ci, i= 1, 2, 3, 4). The values in Table 1 show that, although the patterns are structurally different, there is a high similarity between textures wp I and wp4 with regard to the color aspect. This high similarity occurs because wpl and wp4 agree in the blue colors, which dominate in these textures. Also, wp3 has a slightly higher similarity with wpl than with wp4. This last result may be explained 195
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
a
b
a
February 1994
Figure 2. Location of the color mismatches between wp3 and wp3defective. (a) wp3-defective. (b) Color mismatches.
defective was then compared with the color codebook. The number of standard deviations was set to n = 3.2,
d Figure 1. Color texture images. (a) wpl. (b) wp2. (c) wp3. (d)wp4. Table 1 Texture similarity based on the color aspect (Pmn) Image
wp 1
wp2
wp3
wp4
wpl wp2 wp3
1.00 0.02 0.49 0.68
0.02 1.00 0.10 0.06
0.49 0.10 1.00 0.41
0.68 0.06 0.41 1.00
wp4
by observing that wp3 agrees with wp4 in the blue and red colors, but agrees with wp1 only in the blue colors. However, because wp3 has a slightly higher content of blue than of red, and because wpl is predominantly blue, wp3 and wpl were found to be more similar. The similarity values in Table I correspond with the perception that the color composition of images wpl and wp4 are most similar. We also conducted an experiment to demonstrate an application of the volumes of tolerance in color representation and discrimination. In this experiment, a color codebook was built from wp3, and a defective version of wp3 (wp3-defective) was generated by introducing a rectangular patch in the image (see Figure 2). Note that its color is close to the average color of the texture. The color composition of wp3196
so that the probability of misassignment by each one of the functions (R, G, B, R+G, R+B, G+B and R+G+B) is less than 0.1. We say that a color mismatch occurred when an input color does not fit in the color tolerance volume of its nearest neighbor in the color codebook. The color mismatches occurring between wp3 and wp3-defective are displayed, pixel by pixel, in Figure 2. Except for some scattered digitization noise, the mismatches shown in Figure 2 occur at the specific location where the patch is located. When the number of standard deviations n is set higher (e.g. n = 4 ), there is excessive color tolerance and the patch is missed. When n is set lower (e.g. n = 2 ), the digitization noise becomes more noticeable, and the patch remains detected.
5. Concluding remarks
A color codebook approach was proposed for the representation of the texture color aspect. The usage of this color codebook requires two steps. Initially, the codebook is built adaptively given the problem characteristic colors. These colors represent cluster centers in color space, and cluster overlapping is reduced to have a consistent representation. Next, the obtained color codebook is used as a standard to represent the color aspect of texture images. The experimental results showed that this method can be effective in evaluating the similarity between textures in terms of the color aspect. In particular, the assignment of volumes of tolerance to the codebook prototypes allows one to attribute (or reject)
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
textures to classes based on their color composition. It m a y be used for inspection o f texture patterns based on the color aspect. The codebook can be easily a d a p t e d to other color representations.
Acknowledgement We t h a n k Prof. Marcelo Alencar, D e p a r t m e n t o f Electrical Engineering, U n i v e r s i t y o f Paraiba, Camp i n a G r a n d e , Brazil, for his useful suggestions. This work is partially s u p p o r t e d by the Brazilian Research and D e v e l o p m e n t Council ( C N P q ) .
February 1994
References [1] Gonzalez, R.C. and P. Wintz (1987). Digital Image Processing. Addison-Wesley, Reading, MA. [2] Papoulis, A. ( 1991 ). Probability, Random Variables and Stochastic Processes. McGraw-Hill, New York. [3] Pienkovsky, A.E.K. (1989). Artificial Colour Perception using Fuzzy Techniques in Digital Image Processing. Verlag TUV Rheinland. [4] Scharcanski, J., H.C. Shen and A.P. Alves da Silva (1992). Characteristic colors for color texture analysis: a parallel distributed approach. Vision Interface 92, Vancouver. Canada, May 1992. [5 ] Tomita, F. and S. Tsuji (1990). Computer Analysis of Visual Textures. Kluwer Academic Publishers, Dordrecht.
197
February 1994
PATREC 1160
Representing the color aspect of texture images J. Scharcanski a, J.K. Hovis b and H.C. Shen a a Pattern Analysis & Machine Intelligence Lab, Department of Systems Design Engineering, University of Waterloo, Waterloc Ontario, Canada N2L 3G I b School of Optometry, University of Waterloo, Waterloo, Ontario, Canada, N2L 3G1
Received 3 October 1992 Revised 17 March 1993
Abstract Scharcanski, J., J.K. Hovis and H.C. Shen, Representing the color aspect of texture images, Pattern Recogmtion Letters 15 (1994) 191-197. The problem of representing the color aspect of texture images is analyzed in this paper. We propose that a color codebook be used to represent the color aspect of texture images. This color codebook is built adaptively from the problem prototypes and allows texture discrimination based on the color aspect by using a simple similarity measure.
Keywords. Texture analysis, computer vision, color image, analysis, neural networks, vector quantiTation.
I. Introduction
Real visual textures are in fact colored textures. They m a y present the same colors a n d different structural patterns, different colors in the same structural patterns, or different colors and different structural patterns. Therefore, a texture description should include both the color and structural aspects. Various m e t h o d s have been p r o p o s e d to represent the structural aspect o f visual textures, but, surprisingly, representation o f the color aspect has not received as much attention [ 5 ]. In this paper, we describe a technique to represent the color aspect o f texture images, as one c o m p o n e n t o f a full texture representation. In previous work, characteristic colors have been p r o p o s e d as the feature set representing the color aspect o f visual textures [4]. This set contains the Correspondence to: J. Scharcanski, Pattern Analysis & Machine Intelligence Lab, Department of Systems Design Engineering, University of Waterloo, Waterloo, Ontario, Canada N2L 3G 1.
smallest n u m b e r o f colors satisfying some criteria and is o b t a i n e d by a lateral inhibition network. However, the set o f characteristic colors is image dependent, and distinct images o f the same texture m a y present different characteristic colors. This occurs because colors are noisy, a n d color cluster centroids m a y not corres p o n d from sample to sample. The consequences o f this fact m a y be disastrous for color texture classification a n d discrimination, because it is not trivial to compare texture representations when the feature sets are particular to each image. Therefore, in order to obtain consistent texture classification results, some organization o f the occurring colors is necessary. In this work, we describe how a color codebook can be used as a s t a n d a r d to represent the colors o f a texture. This color codebook is o b t a i n e d a d a p t i v e l y based on the texture prototypes o f the problem. Before describing the color codebook, we will overview how the characteristic colors set is obtained.
0167-8655/94/$07.00 © 1994 - - Elsevier Science B.V. All rights reserved SSDI 01 6 7 - 8 6 5 5 ( 9 3 ) E 0 0 2 0 - O
191
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
2. S e l e c t i n g a characteristic colors set
Usually, colors are fuzzy in color space [3]. This imprecision results in ambiguous representations for computer classification and discrimination purposes. In order to minimize the ambiguity in color texture representation, we use a representative colors approach. With this approach, the smallest representative color set (according to established criteria) defines a feature set for computer color texture representation. The elements of this feature set are the characteristic colors of the texture. These colors are obtained by a parallel distributed approach, and, to each color, we assign a measure of importance [ 4 ]. This parallel distributed approach consists of a network of competitive linear units. A color is assigned to each network unit, and the network units are organized according to the colors they represent. We assume that a unit un is an immediate neighbor of a unit Urn if they correspond to colors that are immediate neighbors in the RGB color space. Each network unit un is associated with an occurring color xn in the input pattern, and its activation, In, is the relative frequency ofxn: N
In =f(Xn) = f ( x n ) / ~ f(xp)
(1)
p=l
where f ( x n ) is the frequency of occurrence of xn in the input image, and N is the number of occurring colors. Thus, the initial state of the network depends on the input color distribution. Starting with this initial state, the network achieves a final (steady) state by taking into account the lateral interactions of units. A network unit un (with activation In) is inhibited by all units that correspond to colors within a distance Rb in the color space. The variable Rb is defined as 2 ~"~-b, where bmaxis the maximum number of bits available for each channel (i.e., red, green and blue), and b is the number of bits effectively used by each channel. Usually, bm~xis pre-defined, and b is a color resolution parameter selected for a particular application. The unit lateral interactions are described by: U
I ~ - v ~= Iprnv I" = f
lfln-.-en
=1
.
t ~
,
0
otherwise
(2)
where Ig denotes the new unit activation level and 192
February 1994
U= 2 3b The inhibition coefficient rnp of the neighboring units equals r for those units within a distance Rb of Un, and is 0 otherwise. The threshold ~ determines when un is in steady state. We have adopted En=~, where e is in (0, 1]. If the frequency distribution of the occurring colors is multimodal, and the peaks are well separated so that their mutual lateral inhibition can be neglected, the network computation converges to high activation units (i.e., the peaks of the frequency distribution). This computation may be interpreted as a feature selection process, where, given the constraints, the smallest feature set is chosen in order to minimize the information loss. If entropy is expressed as Norig
H = Z f(xn) logf(xn) n=l
for a given number of colors Nong, Ho,~ is the entropy of the original color set, and Hch,r is the entropy of the characteristic colors set, then the difference between Ho~ig and nchar is minimized when the Nchar colors with higher relative frequency f(xn) are chosen. As a consequence, the color space is partitioned into mutually exclusive regions with centroids corresponding to distribution peaks. However, in general the mutual inhibition of nearby peaks is non-negligible. In this case, the network adapts to the local shape of the distribution, and the computation converges to local representative colors. A color is selected as characteristic because of its particular location, regional dominance, or both. The number of selected characteristic colors (Nchar) depends on the network parameter setting and the initial distribution. Whether Nchar is the optimal set to minimize information loss in the general case has not yet been determined. After the network has achieved steady state, a set of representative colors has been selected for a certain input pattern. Each color is associated with one of the remaining active units Um ( m = 1, 2 ..... V and V~< U). These are the texture characteristic colors Ym. Given this color set, the input color texture image is quantized and a final image is obtained. That is, each color xn at an input image position (p, q) is mapped to a characteristic color Yk which is its nearest neighbor in the RGB space (i.e., Yk minimizes d(xn, Ym), Vm). The measure of importance MI(ym) of each char-
Volume 15, Number 2
PATTERN RECOGNITIONLETTERS
acteristic color corresponds to the color relative frequency f ( y , , ) in the quantized image. Therefore characteristic colors that are more frequent, are more important in the texture color aspect.
3. Texture discrimination based on the color aspect
A variety of features can be used to describe the color aspect of a texture (e.g. associating colors with the textural elements). In this work, the color aspect is described in terms of the characteristic colors and their respective measure of importance. Our approach is based on a problem-specific color codebook as a standard representation. The use of this color codebook requires two phases. Initially, the color codebook is built adaptively by estimating the expectations of the characteristic colors presented by the problem prototypes. In the next phase, the colors of an input texture image are described in terms of the color codebook. Using the codebook description and a simple similarity measure, textures can be discriminated based on the color aspect.
prototypes using the following rules. Clusters that overlap totally are assigned to the same prototype. Clusters that do not overlap are assigned to distinct
prototypes. Clusters that overlap partially are those that occur within a distance of Ro units from each other in the color space. If Roe (0, Rb], then there is high representation uncertainty, because the occurring cluster centers are within a distance of Rb units from each other and, by definition, belong to the same cluster. This uncertainty is resolved by taking the weighted average of the cluster centers, lfRo > Rb, the clusters are assigned to distinct codebook prototypes. Basically, we have described a smoothing approach for uncertainty reduction in color texture representation [ 1 ]. This approach is justified if we consider that colors in the color space are formed by the addition of noise ~/i to a color prototype c, = (ri, gi, b,). Under this assumption, a characteristic color y~ may be expressed as: yi=ci"~'?]i
(3)
.
It is assumed that within the neighborhood of every color c~ the noise has zero average value. By averaging M different noisy samples we obtain:
3. I. Building a color codebook
1 M --
In assigning colors to classes, we can use either a supervised or an unsupervised approach. The supervised approach requires that labeled patterns (i.e., colors) are presented so that the classifier may learn the mapping between the input and the classification space. However, this approach may lead to complex classification problems due to the fact that colors tolerance volumes are non-uniformly distributed in the color space; particularly, if the color samples are labeled according to the human color perception. Because of these potential classification problems, we chose a mechanism that segregates the input patterns (i.e., colors) into clusters in an unsupervised manner. This unsupervised method creates a color codebook by identifying several prototypes, or exemplars, that can serve as color cluster centers. The input patterns are the characteristic colors of the texture samples and their respective measures of importance. The prototypes in the color codebook are expectations calculated dynamically based on the presented texture samples. This method assigns color clusters to the codebook
February 1994
Yi=
~ Mj=I
Yi.
(4)
Then it follows that
E{~)=c,
(5)
and a~=
a~,
(6)
2 is the varwhere E{~,} is the expected value o f ~ , ay7 iance ofyi and a~, is the variance ofq~. Equations (5) and (6) indicate that as M increases, the variability decreases and y,. will approach c,. In the actual building of the color codebook, the prototypes are obtained sequentially as the texture samples are presented. The characteristic colors y~ of the first sample presented are taken as codebook prototypes c~. For the subsequent samples the following procedure is executed. 1. Compute the characteristic colors and their respective measure of importance. 193
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
2. For each characteristic colory~, that is the nearest neighbor of the codebook prototype ck: (a) if d(yi, Ck) <
Crkr-.~ ~jYjr X f(Yjr) "bYir x f(Yir) ~jf(Yjr) +f(Yir) ' c'~
=
Y.J(Yjb ) + f(Y,b ) where (Yjr, Yjg, Yjb) is denoting the characteristic colors yj previously assigned to the prototype Ck; (b) ifd(y~, Ck) >Rb then includey~ as a newprototype in the color codebook.
3.2. Texture discrimination If a similarity measure is available, then textures can be discriminated based on the color aspect. According to the color codebook approach, the occurring colors in a texture image are mapped to the codebook prototypes using the nearest neighbor criterion (given a color tolerance). The measure of importance associated with each one of the codebook prototypes corresponds to its relative frequency of occurrence in the texture image. Therefore, a texture of index i is represented in terms of this codebook by:
C~={(cj,f(cy)) [ j = 1, ..., L}
codebook. The color codebook may be considered as a vector whose elementsf(cj) assume values in the range [0, Nimg], and Z j f ( c j ) =Nirng, where Nimg is the number of pixels of the input image. Two texture representations (i.e., codebooks) may then be compared by using the angle 0 between two vectors Cm and C. as a simple similarity measure. Because all vector elements are positive, Oe [0, n / 2 ] , so that we can use the following similarity measure p.,.:
194
(Cm'C.) [ 0 , 1] IlCmll'llC. II~
Proper representation of color tolerance volumes for each of the codebook prototypes is not trivial. Initially, we do not know much about the actual color distribution in color space and assume that the volumes of tolerance are spheroids of radius R b. Based on this assumption, the codebook prototypes are calculated as centroids of overlapping volumes of tolerance. At this stage, we prefer not to assume any predefined volumes of tolerance for the codebook prototypes. Instead, the actual distribution of the occurring colors in the vicinity of the codebook prototypes is used. Note that an occurring color xi is considered to be in the vicinity of a codebook prototype ck, if the nearest neighbor ofxi is Ck. A representation of the occurring colors distribution, Dk, in the vicinity of a codebook prototype Ck should be unbiased, and not assume a priori any particular probability distribution (e.g. gaussian) 1. Such a representation may be obtained based on the covariance matrix [Mk] of the distribution Dk, with the color ck as the mean. The matrix [Mk] is then defined as [ 31:
(7)
where f ( c j ) is the frequency of the color prototype cj, and L is the number of color prototypes in the
x
If the value of pro, is high, the two textures (m and n) are similar in terms of the color aspect. This approach may be used to assign textures to classes based only on the color aspect.
'
C'kb = 2jYjb x f(Yjb ) q-Yib x f(Yib )
P m ' = 1 - -2c o s -1
where IICII is the vector length, and (Cm" Cn) is the scalar product of the two vectors.
3.3. Color tolerance volumes
~jYjg × f(Yjg) + Y~gx f(Yig)
ZJ(yjA + f(y,~)
February 1994
(8)
F
Mk=ICG~ LC~
~ C~
CGBI a~_.]
(9)
where a 2 is the variance of the variable X, and Cxris the covariance of the variables X a n d Y, where X a n d Yare R, G or B. Note that Cxr= Crx. Unbiased estimates about the distribution Dk can be obtained directly from the covariance matrix [Dk] by a number of linear combinations of the variables R, G and B. Let us consider the following linear combinations: R, G, B, R+G, R+B, G + B a n d R + G + B . Preliminary work shows that occurring colors in digital images are not adequately modelled by gaussian distributions (personal communication of Dr. Ed Jernigan, Dept. of Systems Design Eng., University of Waterloo, Canada).
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
These combinations may be considered as variables related to the distribution Dk, whose variances are expressed in terms of the covariance matrix elements [2]:
a2+o=a~ +a~ + 2Cgo, a ~ + , = a ~ + a2 + 2CR,, a~+e = a~ + a~ + 2Coe, 2
2
2
2
¢7R+G+ B = C7g + ¢7G + ~7B
+2CR~ +2CR, + 2 C o , ,
(10)
and the means are given respectively by:/tR,/zo,/tB, /tR+~; (=~R+/~o), /IR+B (=#g+lt,), ltC+, ( = J Z G - J - I ~ B ) , ,UR + G + B ( = ~ R " { - t l 2 G " ~ J ~ n ) .
Although the linear combinations of R, G and B provide an estimate of the occurring colors distribution Dk in the vicinity of a codebook prototype ck, we need a criterion to decide whether an input color x~ can be assigned to Ck. This criterion should only use the information that is provided by the Dk estimate. Tchebyschev's inequality 2 provides a criterion and is expressed by:
1 P( I X - u x l >~nXax) <~--~
(1 1 )
where P( ) is a probability measure, and ne~ +. Therefore, we can establish a number of standard deviations n × ax such that the error probability ( 1/ n 2) is small and consider an input color x~ (r~, g~, b,) as belonging to the vicinity of Ck if the following inequalities are true:
Ir~--l~gl <~n×aR , Ig~-/tol < n × a ~ , Ibi-/L. I 4 n X a B , I (r,+g~)-/IR+cl ~nX~R+o, I(r,+bi)--#g+.l~nXag+8,
[ (gi + bi) - Ito+B I <
(12)
2 This inequality does not assume any distribution beforehand. However in some practical situations assumptions about the distribution may be necessary to improve the power of the statistical criterion. These assumptions may be justified based on the provided estimates.
February 1994
Using this approach we can decide whether an input color xi which is in the vicinity of a certain codebook prototype Ck, belongs to the volume of tolerance. This decision criterion is unbiased because no previous assumptions are made about the local distribution of the occurring colors. Note that the various combinations of the R, G and B channels resemble some psychophysical quantities associated with colors, such as the luminance ( R + G + B ) , the individual responses of each channel (R, G, and B), the blue-yellow ( R + B ) and the red-green ( R + G ) dimensions. Also note that we can introduce unexpected events within the color aspect representation by including a dummy color in the color codebook (c~:+~). Therefore, ifxh does not belong to the vicinity of cn, Vn, n = 1, ..., L, then the input color Xh at the image position (p, q), which was not expected considering the color prototypes, is assigned to the dummy color (Xh~CL+ ~). This procedure might be useful in evaluating the similarity between an input pattern and the stored class prototypes.
4. Experimental results In this section, we demonstrate that textures can also be compared in terms of their color composition, irrespective of their structure. A set of texture images with resolution 256 × 256 was used. The following parameter settings were used in these experiments: b = 4 (i.e., a palette with 4096 colors) and R b = 16. For the network simulation, the parameters were chosen as r = 10 -3 and e = 10 -6. Figure 1 shows the set of texture images used. Textures wpl, wp2, wp3 and wp4 contained 13, 11, 15 and 24 characteristic colors respectively. The codebook obtained from these four textures contained 45 color prototypes. Table 1 lists the similarity values p,~, for the color aspect representation of the four textures (i.e., the codebook vectors Ci, i= 1, 2, 3, 4). The values in Table 1 show that, although the patterns are structurally different, there is a high similarity between textures wp I and wp4 with regard to the color aspect. This high similarity occurs because wpl and wp4 agree in the blue colors, which dominate in these textures. Also, wp3 has a slightly higher similarity with wpl than with wp4. This last result may be explained 195
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
a
b
a
February 1994
Figure 2. Location of the color mismatches between wp3 and wp3defective. (a) wp3-defective. (b) Color mismatches.
defective was then compared with the color codebook. The number of standard deviations was set to n = 3.2,
d Figure 1. Color texture images. (a) wpl. (b) wp2. (c) wp3. (d)wp4. Table 1 Texture similarity based on the color aspect (Pmn) Image
wp 1
wp2
wp3
wp4
wpl wp2 wp3
1.00 0.02 0.49 0.68
0.02 1.00 0.10 0.06
0.49 0.10 1.00 0.41
0.68 0.06 0.41 1.00
wp4
by observing that wp3 agrees with wp4 in the blue and red colors, but agrees with wp1 only in the blue colors. However, because wp3 has a slightly higher content of blue than of red, and because wpl is predominantly blue, wp3 and wpl were found to be more similar. The similarity values in Table I correspond with the perception that the color composition of images wpl and wp4 are most similar. We also conducted an experiment to demonstrate an application of the volumes of tolerance in color representation and discrimination. In this experiment, a color codebook was built from wp3, and a defective version of wp3 (wp3-defective) was generated by introducing a rectangular patch in the image (see Figure 2). Note that its color is close to the average color of the texture. The color composition of wp3196
so that the probability of misassignment by each one of the functions (R, G, B, R+G, R+B, G+B and R+G+B) is less than 0.1. We say that a color mismatch occurred when an input color does not fit in the color tolerance volume of its nearest neighbor in the color codebook. The color mismatches occurring between wp3 and wp3-defective are displayed, pixel by pixel, in Figure 2. Except for some scattered digitization noise, the mismatches shown in Figure 2 occur at the specific location where the patch is located. When the number of standard deviations n is set higher (e.g. n = 4 ), there is excessive color tolerance and the patch is missed. When n is set lower (e.g. n = 2 ), the digitization noise becomes more noticeable, and the patch remains detected.
5. Concluding remarks
A color codebook approach was proposed for the representation of the texture color aspect. The usage of this color codebook requires two steps. Initially, the codebook is built adaptively given the problem characteristic colors. These colors represent cluster centers in color space, and cluster overlapping is reduced to have a consistent representation. Next, the obtained color codebook is used as a standard to represent the color aspect of texture images. The experimental results showed that this method can be effective in evaluating the similarity between textures in terms of the color aspect. In particular, the assignment of volumes of tolerance to the codebook prototypes allows one to attribute (or reject)
Volume 15, Number 2
PATTERN RECOGNITION LETTERS
textures to classes based on their color composition. It m a y be used for inspection o f texture patterns based on the color aspect. The codebook can be easily a d a p t e d to other color representations.
Acknowledgement We t h a n k Prof. Marcelo Alencar, D e p a r t m e n t o f Electrical Engineering, U n i v e r s i t y o f Paraiba, Camp i n a G r a n d e , Brazil, for his useful suggestions. This work is partially s u p p o r t e d by the Brazilian Research and D e v e l o p m e n t Council ( C N P q ) .
February 1994
References [1] Gonzalez, R.C. and P. Wintz (1987). Digital Image Processing. Addison-Wesley, Reading, MA. [2] Papoulis, A. ( 1991 ). Probability, Random Variables and Stochastic Processes. McGraw-Hill, New York. [3] Pienkovsky, A.E.K. (1989). Artificial Colour Perception using Fuzzy Techniques in Digital Image Processing. Verlag TUV Rheinland. [4] Scharcanski, J., H.C. Shen and A.P. Alves da Silva (1992). Characteristic colors for color texture analysis: a parallel distributed approach. Vision Interface 92, Vancouver. Canada, May 1992. [5 ] Tomita, F. and S. Tsuji (1990). Computer Analysis of Visual Textures. Kluwer Academic Publishers, Dordrecht.
197