Local multiple patterns based multiresolution gray-scale and rotation invariant texture classification

Information Sciences 187 (2012) 93–108

Contents lists available at SciVerse ScienceDirect

Information Sciences journal homepage: www.elsevier.com/locate/ins

Local multiple patterns based multiresolution gray-scale and rotation invariant texture classification Changren Zhu ⇑, Runsheng Wang ATR National Lab., National University of Defense Technology, China

a r t i c l e

i n f o

Article history: Received 18 May 2010 Received in revised form 20 April 2011 Accepted 16 October 2011 Available online 25 October 2011 Keywords: Local multiple patterns (LMP) ‘‘Uniform’’ LMP Texture analysis Rotation invariance Multiresolution analysis

a b s t r a c t The local binary pattern (LBP) is a robust but computationally simple approach in texture analysis. However, LBP does not have enough information to discriminate among multiple patterns due to its binary patterns only comprising of 0s and 1s. Thus, a multi-resolution gray-scale and rotation invariant texture classification based on local multiple patterns (LMP) is proposed in the paper. The LMP extends binary patterns to multiple patterns, which can preserve more structural information, and be more suitable for image analysis, including the analysis of flat image areas, in which case LBP code is often a random value and thus unsuitable. In addition, several extensions to the LMP, including multi-resolution analysis with the ‘‘uniform’’ LMP with rotation invariance, are presented. The ‘‘uniform’’ LMP can be regarded as a gray-scale and rotation invariant description in various applications and its discrete occurrence histogram over a region of image is proved to be a powerful invariant texture feature. Experimental results of multi-resolution texture classification with support vector machine show that the proposed method obtains overall better performance than other common methods in several aspects of space-resolution and gray-scale variation, rotation, and noise. Ó 2011 Elsevier Inc. All rights reserved.

1. Introduction Analysis of two-dimensional textures is a fundamental research topic in the area of computer vision and has many potential applications, for example, in industrial surface inspection, remote sensing, and biomedical image analysis. However, it is difficult to describe texture accurately. One major problem is that textures in real world are often not uniform due to variations in orientation, scale, or other visual appearance. These factors produce great within-class variability. There are many publications [1–58] discussing the texture analysis and classification, but there exist only a limited number of examples of successful exploitation of texture. In addition, most proposed texture measures has high degree of computational complexity, as Randen and Husoy [42] concluded in their extensive comparative study involving dozens of different spatial filtering methods: ‘‘A very useful direction for future research is therefore the development of powerful texture measures that can be extracted and classified with a low-computational complexity.’’ The local binary pattern (LBP) operator, first introduced by Ojala et al. [36], is a robust but theoretically and computationally simple approach. It brings together the separate statistical and structural approaches to texture analysis, thereby opening a door for the analysis of both stochastic microtextures and deterministic macrotextures simultaneously. Additionally, it has shown excellent performance in many comparative studies, in terms of both speed and discrimination performance. The original LBP operator [36] labels each 3  3-neighborhood pixel in each pixel of an image with a binary number by comparing gray values between the pixel and its each neighborhood pixel. The neighborhood pixel labels form the LBP code ⇑ Corresponding author. Tel.: +86 731 4575724; fax: +86 731 4575791. E-mail addresses: [email protected] (C. Zhu), [email protected] (R. Wang). 0020-0255/$ - see front matter Ó 2011 Elsevier Inc. All rights reserved. doi:10.1016/j.ins.2011.10.014

94

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

of the pixel. The LBP code characterizes local image texture patterns around central pixel, describing its structural information. The histogram of the codes can be used as a texture descriptor, describing its statistical information. Later the original LBP operator [36] has been extended in several ways, such as neighborhoods with different sizes [35], multi-resolution [27], uniform patterns [35], etc. The extended LBP operator can be used for rotation-invariant texture classification, and has a wide application, such as texture analysis and classification [21–42], face detection and recognition [1,2,6,10,12,16,19,20,22,45,56,36], image retrieval [43], etc. However, the binary patterns comprising of 0s and 1s, adopted by the LBP, do not have enough information to discriminate between multiple patterns in actual images. Especially, the LBP is unsuitable for the analysis of flat image areas, where all pixels in a small neighbor have nearly equal gray values. The LBP code bit in flat image areas might be a random value of 0 or 1. The case is especially true when there is a little noise, as similarly described about the drawback of LBP in [3,14]. It results in serious unreliability of the extracted feature. This weakness cannot be overcome by the original LBP-based methods. Later, some improvements of LBP were developed thereafter. On the one hand, a special processing was done on the zone around the center pixel value. For example, the reference value was shifted by adding a given threshold to make LBP less sensitive to noise [15] (denoted as LBP_HP). On the other hand, an extension of LBP called Local Ternary Patterns (LTP_TT), which bears some similarity to the texture spectrum (TS) technique [13], was proposed by Tan and Triggs [44] and had already been used for facial expression recognition by Gritti et al. [12]. In LTP_TT, neighboring pixel values in a zone of width ±t around the center pixel value ic were quantized to zeros, values above this quantized to +1 and values below it to 1. Thus, LTP_TT uses a ternary code instead of a binary code. Then a coding scheme was adopted to splits each ternary pattern into its positive and negative parts, subsequently treating these as two separate channels of LBP descriptors. LTP_TT was less sensitive to noise in near-flat regions. In this paper, focusing on rotation and gray-scale invariant texture classification, a new generalized feature description called ‘‘local multiple patterns’’ (LMP) is proposed. At first, our approach is to generalize LBP, TS [13] and LTP_TT to LMP. The basic idea of LMP is to extend the LBP’s binary patterns to multiple patterns. With multiple patterns, LMP can preserve more information to discriminate between different patterns. Moreover, with the definition of a special zone around the center pixel value, LMP aims to be more robust for image analysis, including the analysis in flat image areas, in which case the original LBP often fails. Furthermore, our approach is to select the certain LMP, called ‘‘uniform’’ LMP, which are fundamental rotation-invariant properties of local image texture and whose occurrence histogram is proven to be a very powerful texture feature. Multiresolution analysis is carried out by combining several neighbors with different sizes or shapes. In addition, our proposed method is robust in terms of gray-scale variation because the global illumination change results in only little variation of the local relative gray-scale. The performance of the proposed approach is demonstrated with some experiments. The proposed texture operator is proved to be an excellent gray-scale and rotation invariant texture descriptor. The paper is organized as follows. Section 2 gives a brief analysis of the LBP operator and its extensions, which are the basis of the LMP. Section 3 presents the definition and the derivation of the LMP. Section 4 gives several extensions of the LMP. Section 5 describes the multiresolution gray-scale and rotation invariant texture classification method based on the LMP. Section 6 shows several experiments. Finally discussion and conclusion are made in Section 7. 2. Analysis of local binary patterns (LBP) 2.1. Fundaments of LBP The derivation of the LBP presented by Ojala et al. [35] is described briefly as follows. The texture T is defined as the joint distribution of the gray levels of P + 1(P > 0) image pixels:

T ¼ tðg c ; g 0 ; . . . ; g P1 Þ;

ð1Þ

where gc is the gray value of the center pixel in a local neighborhood. gp(p = 0, . . . , P  1) corresponds to the gray values of P equally spaced pixels in a circle of radius R(R > 0). Fig. 1 illustrates three circularly symmetric neighbor sets for different values of P and R [24]. To achieve invariance with respect to any monotonic transformation of the gray scale, the signs of the differences are only considered as:

T  tðsðg 0  g c Þ; . . . ; sðg P1  g c ÞÞ;  where sðxÞ ¼

1 if x >¼ 0 0 if x < 0:

ð2Þ ð3Þ

Then, a weight 2p is assigned to each sign s(gp  gc) to transform the differences in a neighborhood into a unique LBP code. The code characterizes the local image texture:

LBPP;R ¼

P 1 X p¼0

sðg p  g c Þ2p :

ð4Þ

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

95

Fig. 1. Circularly symmetric neighbor sets.

Fig. 2. Some examples of LBP patterns.

The local texture can thus be approximately described with a 2P -bin discrete distribution of LBP codes:

T  tðLBP P;R Þ:

ð5Þ

2.2. Analysis of LBP The LBP operator is a powerful, but simple nonparametric descriptor of local texture patterns, and has wide applications. However, the LBP adopts binary patterns, which have not enough information to discriminate between multiple patterns. (i) LBP code cannot discriminate the two pixels in situation when one is with gray level near but below the center pixel, and the other is with gray level far smaller than the center pixel. It is also true about the two pixels with gray value above (or equal to) the central pixel. Fig. 2 shows some examples. In example 1, gray levels of 13 and 10, which are far larger than 6, are thresholded into 1. 6, which is near or equal to 6, is also thresholded into 1. On the other hand, gray levels of 1, 2, 3, and 5 are all thresholded to 0. It makes LBP unable to distinguish some texture patterns. (ii) LBP is unsuitable for image analysis of flat image areas, which are often important parts of an image, such as face images, remote-sensing images, etc. Example 2 and 3 in Fig. 2 show two typical samples in a flat image area, where all gray values of pixels in a small neighbor are nearly equal. The gray value of every pixel fluctuates near the gray value of the central pixel. This is especially true when there is a little noise in the flat image area, as shown in example 2 and 3. All gray values of the pixels are near 166, and they satisfy a homogeneous distribution. Thus, the thresholded bit pattern is a random value of 0 and 1. This causes serious instabilities of its LBP feature. At this time, LBP is unsuitable for image analysis. Its weaknesses result from the only binary patterns: 0 and 1. To overcome the shortcomings stated above, we introduce local multiple patterns (LMP). It is described in details in Section 3. 3. Introduction of local multiple patterns (LMP) This section shows how the LMP is derived as follows. Firstly, LMP extends LBP from binary patterns to multiple patterns to be more robust for image analysis, including the analysis of flat image areas. We add middle patterns between 0 and 1. Thus, LMP can describe more information than LBP.

96

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

Suppose: gc corresponds to the gray value of the center pixel of a local neighborhood. gp(p = 0, . . . , P  1) correspond to the gray values of P equally spaced pixels on a circle of radius R(R > 0) that form a circularly symmetric set of neighbors. Thmin 6 (gp  gc) 6 Thmax, where Thmax and Thmin are the maximum and minimum values of gp  gc, respectively. Thmax is usually equal to the negative of Thmin.They are 255, and 255 in the 8-bits image, respectively. N corresponds to the number of divided patterns, which is ordinarily an odd number to ensure a zone across zero to be suitable for the analysis in flat image area. It needs (N  1) 0 thresholds to separate the interval [Thmin, Thmax] into N parts. The thresholds are set from small to large as Th1, . . . , ThN1. Like the LBP, local texture T is also defined as

T ¼ tðg c ; g 0  g c ; . . . ; g P1  g c Þ:

ð6Þ

An approximate form T is defined as

T  tðsðg 0  g c Þ; . . . ; sðg P1  g c ÞÞ:

ð7Þ

By setting x ¼ g p  g c ;

ð8Þ

s(x) in LMP is defined as

8 N  1 if x >¼ ThN1 ; > > > > > > < N  2 if ThN2 6 x < ThN1 ; sðxÞ ¼  > > >1 if Th1 6 x < Th2 ; > > > : 0 if x < Th1 :

ð9Þ

The thresholds, Th1, . . . , ThN1, can be defined according to corresponding application. As a typical example, the interval is simply equally divided. The LMP is defined as follows:

Thu ¼ ðThmax  Thmin Þ=N; Thi ¼ Thmin þ ðThmax  Thmin Þ=N i ¼ Thmin þ Thu  i;

ð10Þ i ¼ 1; 2; . . . ; N  1;

8 N  1 if x P Thmin þ ðN  1Þ  Thu ; > > > > > > < N  2 if Thmin þ ðN  2Þ  Thu 6 x < Thmin þ ðN  1Þ  Thu ; sðxÞ ¼  > > >1 if Thmin þ Thu 6 x < Thmin þ 2Thu ; > > > : 0 if x < Thmin þ Thu :

ð11Þ

ð12Þ

If N is an even number, a value should be added to or subtracted from Thi defined in (11) to ensure a zone across zero to be suitable for the analysis in the flat image area. In the LMP, the thresholded pattern can be described with P-bit number, every bit number being an integer in the range of [0, N  1]. The text unit [17,18] with a different set of thresholds is similar to the particular example when N equals 5. When N equals 3, the LMP is simplified into local triple patterns (LTP). In general local triple patterns, s(x) is defined as

8 > < 2 if x P Thmin þ Th2 ; sðxÞ ¼ 1 if Thmin þ Th1 6 x < Thmin þ Th2 ; > : 0 if x < Thmin þ Th1 :

ð13Þ

Specially, if it is simply equally divided, s(x) in the LTP is defined as

8 > < 2 if x P Thmin þ 2Thu ; sðxÞ ¼ 1 if Thmin þ Thu 6 x < Thmin þ 2Thu ; > : 0 if x < Thmin þ Thu :

ð14Þ

On the other hand, if only the interval across zero is divided into a new part, s(x) in the LTP is defined as

8 > < 2 if x > Th0 ; sðxÞ ¼ 1 if  Th0 6 x 6 Th0 ; > : 0 if x < Th0 :

ð15Þ

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

97

Fig. 3. Calculating the LTP code.

Fig. 4. Calculating the LTP codes.

Here Th0 is a positive threshold, which is determined according to corresponding application. If LTP is used simply to reduce the noise, Th0 can be a small value. In the LTP, the thresholded pattern can be described with P-bit ternary number, which every bit number is an integer in a range of [0, 2]. Secondly, the LMP’s neighbors can have different sizes and shapes. The notation LMPP,R describes the LMP with a circularly symmetric set of neighbors which are formed by P equally spaced pixels on a circle of radius R (R > 0). Fig. 1 illustrates three circularly symmetric neighbor sets for different values of P and R. Thirdly, the LMP and the LBP have different bit weights. In the LBP, the weights are 1, 2, 4, 8, . . . , 128, . . ., which a binomial weight 2p is assigned to each sign s(gp  gc), transforming the differences in a neighborhood into a unique LBP code. In the LMP, a weight Np is assigned to each sign s(gp  gc). In the LTP, a trinomial weight 3p, for example, 1, 3, 9, 27 , . . ., is assigned to each sign s(gp  gc). Fig. 3 shows how the LTP code is derived. It adopts s(x) defined in (15). Here Th0 is set to 1. It is an example of LTP8,1. We compute again the LTP codes of the latter two examples in Fig. 2. Example 2 and example 3 in Fig. 2 are two typical image samples in a flat image area where there is a little noise. It is shown in Fig. 4. Here s(x) defined in (15) is adopted and Th0 equals 2. The LTP thresholded patterns of example 2 and example 3 are all 11111111. Thus, their LTP codes are all 3280. However, due to a little noise, their LBP thresholded patterns are different and thus their LBP codes are different. It can be easily concluded from comparison of Figs. 2 and 4 that the LTP has much higher performance than the LBP in terms of noise reduction. The LTP is similar to Local Ternary Pattern [12,44] (denoted as LTP_TT). The LTP_TT uses s(x) similar to (15), but uses a different coding scheme that splits each ternary pattern into its positive and negative parts, subsequently treating these as two separate channels of LBP descriptors. On the other hand, if Th0 in (15) is equal to zero, the LTP is simplified into texture spectrum technique [13]. In addition, it can be easily found from above that LBP is a particular example of LMP. When N equals 2, and s(x) is defined as (3), the LMP is simplified into the LBP. 4. Extensions of local multiple patterns (LMP) Like the LBP, the LMP also has some extensions. 1) The LMP operator can be extended to combine neighbors with different sizes or different shapes. For example, the neighbors with different R, such as LMP8,1 + LMP8,2 + LMP8,3, are combined to extract features for texture classification. 2) The LMP can also be extended to multi-resolution LMP with Gaussian lowpass filtering to improve the performance like [27]. 3) Rotation invariant LMP The rotation invariant LMP code is derived in a similar way of the rotation invariant LBP. To remove the effect of rotation, each LMP code must be rotated back to a reference position, making all rotated versions of a pattern code the same. This transformation is defined as follows:

98

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

Fig. 5. Some unique rotation invariant patterns (top row) that can occur in the circularly symmetric neighbor set of LTPri8;R , and some patterns that produce the same unique rotation invariant patterns.

  LMPriP;R ¼ min RORðLMP P;R ; iÞ ji ¼ 0; 1; . . . ; P  1 ;

ð16Þ

where the superscript ri stands for ‘‘rotation invariant’’. The function ROR (y, i) circularly shifts the P-bit number y (with every bit number being an integer in the range of [0, N  1])i times to the right (jij < P). In short, the rotation invariant code LMPriP;R is produced by circularly rotating the code LMPP,R until its minimum value is attained. When N equals 3, LMPriP;R is simplified into LTPriP;R . Fig. 5 shows some unique rotation invariant patterns (top row) that can occur in the circularly symmetric neighbor set of LTP ri8;R , and some patterns that produce the same unique rotation invariant patterns. 4) The ‘‘uniform’’ LMP Another extension is so-called uniform patterns. In the LMP, a local pattern is called ‘‘uniform’’ if it meets the following conditions: when the P-bit pattern string is considered circular, a) it contains at most N bitwise transitions; and b) every bit n1in the bitwise transitions can occur at most two times. Here n1 is any integer in the range [0, N  1]. The notation LMPu2 P;R represents the uniform LMP operator in a (P, R) neighborhood. The superscript u2 stands for the use of only uniform patterns, with all remaining patterns labeled with a single label. When N equals 3, LMPu2 P;R is simplified into LTPu2 P;R . On the other hand, in LMP a local pattern is called ‘‘uniform-0’’ if it contains at most two bitwise transitions from n1 to n2 or vice versa when the P-bit pattern string is considered circular. Here n1 and n2 are any integers in the range [0, N  1]. It can be easily concluded that the ‘‘uniform-0’’ patterns are a subset of the ‘‘uniform’’ patterns. The denotation LMPu0 P;R represents ‘‘uniform-0’’ LMP operator in a (P, R) neighborhood. Superscript u0 stands for using only ‘‘uniform-0’’ patterns and labeling all remaining patterns with a single label. 5) The ‘‘uniform’’ LMP with rotation invariance When ‘‘uniform’’ LMP codes are rotated to their minimum values, the rotation-invariant ‘‘uniform’’ LMP codes LMPriu2 P;R are derived. The codes can be defined as follows:

n   o u2 LMPriu2 P;R ¼ min ROR LMP P;R ; i j i ¼ 0; 1; . . . ; P  1 :

ð17Þ

riu2 riu2 When N equals 3, LMPriu2 P;R is simplified into LTP P;R . Fig. 6 shows most of the unique rotation invariant patterns of LTP 8;R . The first 9 rows contain all of the 66 ‘‘uniform’’ patterns and the numbers inside them correspond to their unique LTPriu2 8;R codes. The remaining patterns are all labeled with ‘‘66’’. Similarly, when the ‘‘uniform-0’’ LMP codes are rotated to their minimum values, the rotation-invariant ‘‘uniform-0’’ LMP codes LMPriu0 P;R are derived. In Fig. 6, the first three rows contain all of the 24 ‘‘uniform-0’’ patterns and the numbers inside them correspond to their unique LTP riu0 8;R codes. The remaining patterns are all labeled with ‘‘24’’. riu2 How to compute the LTP riu2 P;R code simply? The LTP P;R code can be easily extracted by counting the number of ‘0’, ‘1’, and ‘2’ riu2 in the LTP bit code string. The LTP 8;1 is selected as an example to depict the extraction method as follows:

1) Count the number of ‘0’, ‘1’, and ‘2’ in the LTP bit code string, denoted as m0, m1, and m2, respectively; 2) Count the bit change number in the LTP bit code string, denoted as mc; 3) Obtain LTPriu2 8;1 code:

If mc is no more than 2, then If m2 is zero, then the LTPriu2 8;1 code is m1; else if m1 is zero, then the LTP riu2 8;1 code is m2 + 8; else if m0 is zero, then the LTP riu2 8;1 code is m2 + 16;

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

99

Fig. 6. Most of the unique rotation invariant patterns that can occur in the circularly symmetric neighbor set of LTP riu2 8;R . The first 9 rows contain all of the 66 ‘‘uniform’’ patterns and the numbers inside them correspond to their unique LTPriu2 8;R codes. The remaining patterns are all labeled with ‘‘66’’.

else if mc is no more than 3 if the bit sequence is 2–1–0 If m2 equals 1, then the LTP riu2 8;1 code is m1 + 23; else If m2 equals 2, then the LTP riu2 8;1 code is m1 + 29; else If m2 equals 3, then the LTP riu2 8;1 code is m1 + 34; else If m2 equals 4, then the LTP riu2 8;1 code is m1 + 38; else If m2 equals 5, then the LTP riu2 8;1 code is m1 + 41; else If m2 equals 6, then the LTP riu2 8;1 code is m1 + 43; else the LTPriu2 8;1 code is 66; else // if the bit sequence is 1–2-0 If m1 equals 1, then the LTP riu2 8;1 code is m2 + 44; else If m1 equals 2, then the LTP riu2 8;1 code is m2 + 50; else If m1 equals 3, then the LTP riu2 8;1 code is m2 + 55; else If m1 equals 4, then the LTP riu2 8;1 code is m2 + 59; else If m1 equals 5, then the LTP riu2 8;1 code is m2 + 62; else If m1 equals 6, then the LTP riu2 8;1 code is m2 + 64; else the LTPriu2 8;1 code is 66; else the LTPriu2 8;1 code is 66;

100

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

riu0 It can be seen from above that the extraction of the LTPriu2 P;R code is simple and fast. Also, the LTP P;R code can be easily extracted by only considering the case that mc is no more than 2. Moreover, the joint distribution of LMP and local variance, denoted as LMPriu2 P;R =VARP;R , can be introduced as a rotation invariant description of texture in terms of texture patterns and their strength.

5. Multiresolution gray-scale and rotation invariant classification based on local multiple patterns Like LBP, the LMP can be widely applied to many fields, such as texture analysis, remote sensing image classification, face detection and recognition, etc. Here we only describe how to analyze and classify the texture with LMP. 5.1. Gray-scale and rotation invariant feature extraction When the illumination varies, gray value in a given local neighbor shifts simultaneously with a little distortion. The relative value of gray in a given local neighbor can be basically invariant, and thus the LMP code is invariant. It is easily seen from above that the rotation-invariant ‘‘uniform’’ LMP codes LMPriu2 P;R can be regarded as gray-scale and rotation invariant micro-features for texture classification. When N P 3, the range of the LMP code is larger than that of the LBP. Thus, the LMP maybe require more memory capacity than the LBP. When N equals 3, the LMP is simplified into LTP, which may be often a tradeoff between complexity and performance. Thus, the rotation-invariant ‘‘uniform’’ LTP codes LTP riu2 P;R will be often adopted for rotation-invariant texture classification. The feature extraction consists of the following two steps. First, the LMP codes are extracted pixel by pixel. N, s(x), and neighbors (P, R) first need to be determined according to corresponding application. For example, When N equals 3, P equals 8, and s(x) defined as (15) is adopted, the LTP riu2 8;R codes are defined. Then, reliable features are extracted. The discrete occurrence histogram of the ‘‘uniform’’ patterns computed over an image or a region of image is adopted as a powerful texture feature. By computing the occurrence histogram, we effectively combine structural and statistical approaches: the local multiple pattern detects microstructures (e.g., edges, lines, spots, flat areas) whose underlying distribution is estimated by the histogram. 5.2. Multiresolution analysis We have presented general rotation-invariant operators LMPriu2 P;R for characterizing the spatial pattern and the contrast VARP,R of local image texture using a circularly symmetric neighbor set of P pixels placed on a circle of radius R. Parameter P controls the quantization of the angular space, whereas R determines the spatial resolution of the operator. On the one hand, multiresolution analysis generally needs to compute multiple LMP codes in several neighbors with different sizes or different shapes. By altering P and R, we can realize operators for any quantization of the angular space and for any spatial resolution. Multiresolution analysis can be accomplished by combining the information provided by multiple riu2 riu2 operators of varying (P, R). For example, LTP riu2 8;1 ; LTP 16;2 , and LTP 24;3 can be combined to analyze the texture. On the other hand, multiresolution analysis can be implemented by scaling the source image and apply the operation at different scales. Here the scaling refers to subsampling with interpolation or low-pass filtering. For example, the Gaussian filters with different parameter r are adopted as low-pass filter. However, it needs more computational complexity. 5.3. Classification In the classification phase, many classifiers, such as k-nearest classifier, BP neural network, Support Vector Machines (SVM), etc., can be adopted. In our approach SVM [7,47,48] is adopted in our experiments due to its high performance in many applications. 6. Experiments The performance of our approach has been demonstrated with following gray-scale and rotation invariant texture riu0 analysis. Our experiments included two sections. In the first section, when N is set to 3, the LTP riu2 P;R and LTP P;R codes with s (x) defined as (15) are adopted. To be competent for multiple different spatial resolutions and different angular resolutions, riu2 riu2 we have implemented LTP riu2 8;1 ; LTP 16;2 ; LTP 24;3 and their combinations in our experiments. Also, we have realized riu0 riu0 riu2 riu2 LTPriu0 ; LTP ; LTP and their combinations. In addition, LBPriu2 8;1 16;2 24;3 8;1 ; LBP 16;2 ; LBP 24;3 and their combinations [35], improved LBP [15] (LBP_HP), TS [13] and LTP_TT [12,44] were implemented for the performance comparison. In LBP_HP and LTP_TT methods, the threshold parameter was equal to that of LTP. Furthermore, two other typical good textural feature extraction methods, cooccurrence matrix (CM) [8], a good classical method, and wavelet transform feature (WT) [21], a good recent method, were implemented for the performance comparison. In CM method, the rotation-invariant features based on the isotropic combination of distance 1 and 2, including energy, entropy, correlation, contrast and inverse difference

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

101

moment, which had best performance in its experiment, was adopted for performance comparison. In WT method, db4 wavelet transform with three levels was selected for implementation. riu0 In the second section, when N is more than 3, equal to 4 or 5, the LMPriu0 4 P;R and LMP 5 P;R codes were implemented to describe the performance of LMP beyond LTP. In our experiments, to evaluate the performance of the LMP directly, no preprocessing and postprocessing steps were done. That is to say, all features in the following experiments were extracted directly from source images. In addition, because Libsvm developed by Chih-Jen Lin et al. [7] is a simple, easy-to-use, and efficient software for SVM classification and regression, in our experiments, the C-SVM classification with polynomial kernel was used with the parameters: degree = 3, gamma = 40, coef0 = 0, penalty parameter C = 8, stop criteria eps = 1.000000e  007. And the classification strategy with one against all was adopted. 6.1. Texture image data setup On the one hand, our experiments mainly involved the following texture image data sets from Outex [37]: 1) Set1: Outex_TC_00010; 2) Set2: Outex_TC_00012:000; 3) Set3: Outex_TC_00012:001. Outex provides a large collection of textures and ready-made test suites for different types of texture analysis problems. In Outex, each texture available at the site is captured using three different simulated illuminants provided in the light source, denoted as ‘‘horizon’’, ‘‘inca’’, and ‘‘tl84’’. Each texture is captured using six spatial resolutions (100, 120, 300, 360, 500, and 600 dpi) and nine rotation angles (0°, 5°, 10°, 15°, 30°, 45°, 60°, 75°, and 90°). Hence, 162 images are captured from each texture. Outex_TC_00010 and Outex_TC_00012 have a natural tactile dimension and natural appearance of local intensity distortions caused by the tactile dimension. This image data presents a very realistic and challenging problem for illumination and rotation invariant texture analysis. Outex_TC_00010 [37] (Set1): It is used for rotation invariant texture classification. The classifier is trained with the reference textures (20 samples of illuminant ‘‘inca’’ and angle 0°in each texture class), while the 160 samples of the same illuminant ‘‘inca’’ but the other eight other rotation angles in each texture class, are used for testing the classifier. Hence, in this suite, there are 480 (24  20) models and 3840 (24  20  8) validation samples in total. Outex_TC_00012 [37] (Set2 and Set3): It is used for rotation and illuminant invariant texture classification. The classifier is trained with the reference textures (20 samples of illuminant ‘‘inca’’ and angle 0°in each texture class) and tested with all samples captured using illuminant ‘‘tl84’’ (problem 000, Set2) and ‘‘horizon’’ (problem 001, Set3). Hence, in both problems, there are 480 (24  20) models and 4320 (24  20  9) validation samples in total. On the other hand, Brodatz [4,58] (Set4) and KTH-TIPS [11,5] (Set5) were adopted for performance comparison. They are described in details in the subSection 6 in the section of experiments. 6.2. The statistics of LMP Here, we take a close look at the certain LMP, called ‘‘uniform’’ LMP, which are found to be the fundamental properties of local image texture. 8 In the case of LTP riu2 8;1 , we choose 66 ‘‘uniform’’ patterns out of the 3 possible patterns, merging the remaining patterns under the ‘‘non-uniform’’ label. Similarly, LTP riu2 considers only 258 out of 316 possible patterns, and LTP riu2 16;2 24;3 considers only riu0 24 578 out of 3 possible patterns. Similarly, LTP 8;1 chooses 24 ‘‘uniform-0’’ patterns out of the 38 possible patterns, LTP riu0 16;2 24 considers only 48 out of 316 possible patterns, and LTPriu0 possible patterns. 24;3 considers only 72 out of 3 Fig. 7 shows the distributions of different patterns of a typical example. Although there is a peak at the ‘‘non-uniform’’ label, in the experiments of all three image data sets, the 66 ‘‘uniform’’ patterns of LTPriu2 8;1 contributed from 66.2 percent up to 90.2 percent of the total pattern data, averaging 79.5 percent. As expected, the ‘‘uniform’’ patterns of LTP riu2 16;2 contributed a smaller proportion of the image data, from 48.0 percent up to 62.2 percent of the total pattern data, averaging 57.5 percent. The ‘‘uniform’’ patterns of LTPriu2 24;3 contributed a even smaller proportion of the image data, from 38.0 percent up to 46.3 percent of the total pattern data, averaging 43.0 percent.

riu2 riu2 Fig. 7. (a) Is a typical original image. (b), (c) and (d) are the distributions of different patterns in LTP riu2 8;1 ; LTP 16;2 , and LTP 24;3 , respectively.

102

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

Similarly, the contribution of the ‘‘uniform-0’’ patterns of LTP riu0 8;1 ranged from 37.5 percent to 45.5 percent, averaging 42.2 percent. The contribution of the ‘‘uniform-0’’ patterns of LTP riu0 16;2 ranged from 32.3 percent to 39.6 percent, averaging 35.8 percent. The contribution of the ‘‘uniform-0’’ patterns of LTPriu0 24;3 ranged from 27.8 percent to 38.8 percent, averaging 32.5 percent. Although being a minority of all possible patterns, the ‘‘uniform’’ patterns contribute to a large portion of all possible patterns, and then appear to be fundamental properties of local image texture.

6.3. Rotation-invariant texture classification with different threshold This experiment shows the classification performance change with different threshold of the LTP. This experiment was done on 5 subsets of Set1 and obtained the average performance. Every subset had 11 categories which were randomly selected from Set1. In every suite, there were 220 (11  20) models and 1760 (11  20  8) validation samples in total. Table 1 shows the classification performance comparison of the LMP with different thresholds. The values in the table are corresponding average classification accuracies. When the threshold Th0 changed from 1 to 20, the classification accuracy of riu2 riu2 riu0 riu0 riu0 LTPriu2 8;1 þ LTP 16;2 þ LTP 24;3 ranged from 99.09% to 100.00%, and that of LTP 8;1 þ LTP 16;2 þ LTP 24;3 ranged from 98.98% to 99.89%, riu2 riu2 riu2 all of which had a high performance. Similarly, the accuracies of LTP 8;1 ; LTP 16;2 , and LTP 24;3 ranged from 83.69% to 98.47%, from 96.42% to 99.94%, and from 98.13% to 99.83%, respectively. It can be easily concluded that the classification performance of the LMP is consistently reliable in terms of the threshold, and very close to the highest value at the threshold of 2 or 3. Thus, the threshold Th0 was set to 2 for simplicity in our following experiments.

6.4. Gray-scale and rotation invariant texture classification on different sets In this experiment, the threshold Th0 equals 2. The experiment was done on all three texture image data sets. Table 2.1 shows the performance comparison of different LTPs, different LBPs, TS [13], CM [8], and WT [21]. The classification accurariu2 riu2 riu0 riu0 cies on Set1, Set2, and Set3 of LTP riu2 8;1 þ LTP 16;2 þ LTP 24;3 achieved 97.26%, 92.89%, and 91.85%, and those of LTP 8;1 þ LTP 16;2 þ riu2 riu2 riu2 LTPriu0 achieved 96.30%, 91.44%, and 91.44%, all of which were better than those of LBP þ LBP þ LBP , respectively. 24;3 8;1 16;2 24;3 riu2 riu2 riu2 riu2 riu2 Similarly, the classification accuracies of LTP riu2 8;1 ; LTP 16;2 , and LTP 24;3 were better than those of LBP 8;1 ; LBP 16;2 , and LBP 24;3 , respectively. It is easily found that the performance of LBP_HP was better than that of the corresponding LBP due to the processing of the pixel value near the center pixel value. riu2 riu2 Most classification accuracies of LTP TT riu2 8;1 ; LTP TT 16;2 ; LTP TT 24;3 , and their combinations were worse than those of the corriu2 responding LTPP;R operator maybe due to the different coding scheme, whereas better than those of the corresponding LBPriu2 P;R operator due to the processing of noise reduction. riu2 riu2 riu2 riu2 It is easily seen from Table 2.1 that LTPriu2 8;1 þ LTP 16;2 þ LTP 24;3 or LTP 16;2 þ LTP 24;3 can achieve the highest classification performance. It is easily concluded that the LMP is competent for the texture classification in variant illumination case despite a little fall in performance from Set1 to Set2 and Set3.

Table 1 The classification accuracies of the LTP in different thresholds (%). Th0

LTP riu2 8;1 þ16;2 þ24;3

LTPriu0 8;1 þ16;2 þ24;3

LTPriu2 8;1

LTPriu2 16;2

LTP riu2 24;3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

99.26 100 99.32 100 99.77 99.49 99.09 99.32 99.15 99.60 99.20 98.92 99.09 99.66 99.77 99.72 99.77 99.72 99.83 99.83

99.72 99.89 99.72 99.83 99.77 99.60 99.38 99.49 99.26 99.49 99.38 99.15 98.98 99.55 99.72 99.72 99.72 99.77 99.83 99.83

97.84 98.47 98.24 95.63 95.17 92.78 92.78 95.00 94.49 95.00 95.00 94.55 92.24 93.01 91.76 92.90 89.43 86.76 84.77 83.69

98.53 98.13 99.94 99.38 99.55 98.75 99.83 99.32 96.42 97.50 97.78 98.69 98.92 99.49 99.38 99.77 99.43 99.20 99.32 99.66

98.13 99.26 98.30 99.38 99.38 98.64 99.55 99.70 99.32 99.32 99.43 99.20 99.49 99.60 99.66 99.66 99.60 99.60 99.83 99.66

103

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108 Table 2.1 The classification accuracy comparison of different LTP and LBP on three data sets (%). Mode

Set1

Set2

Set3

Average

LTPriu2 8;1 LTPriu2 16;2 LTPriu2 24;3 LTPriu2 8;1 þ16;2 LTPriu2 8;1 þ24;3

84.32

70.35

71.11

75.26

93.83

90.93

90.23

91.66

96.77

93.96

93.82

94.85

93.98

87.13

87.29

89.47

93.88

86.81

86.48

89.06

LTPriu2 16;2 þ24;3 LTPriu2 8;1 þ16;2 þ24;3

96.15

94.86

94.72

95.24

97.27

92.89

91.85

94.00

LTPriu0 8;1 þ16;2 þ24;3

94.30

91.44

91.44

92.39

LBPriu2 8;1

77.32

64.45

62.43

68.07

LBPriu2 16;2

86.43

80.07

78.36

81.62

LBPriu2 24;3 LBPriu2 8;1 þ16;2 þ24;3 LBP HP riu2 8;1 LBP HP riu2 16;2 LBP HP riu2 24;3 LBP HP riu2 8;1 þ16;2 þ24;3

94.17

85.67

84.12

87.99

93.31

89.40

89.47

90.73

80.81

64.84

62.64

69.43

89.56

83.87

82.25

85.23

95.60

88.89

88.22

90.90

97.01

89.86

88.70

91.86

TS8,1 LTP TT riu2 8;1

55.96 83.91

57.62 68.80

60.32 69.56

57.97 74.09

LTP TT riu2 16;2

94.17

90.86

91.67

92.23

LTP TT riu2 24;3

97.68

93.13

92.36

94.39

LTP TT riu2 8;1 þ16;2

94.11

89.95

88.80

90.95

TT riu2 8;1 þ24;3 TT riu2 16;2 þ24;3

96.54

92.11

90.56

93.07

95.91

92.36

91.09

93.12

LTP TT riu2 8;1 þ16;2 þ24;3

95.70

91.30

89.42

92.14

WT [21] CM [8]

62.62 81.84

56.06 64.98

55.25 65.30

57.98 70.71

LTP LTP

TS [13] obtained the worst performance, averaging 57.97%. In addition, TS has two drawbacks: too long feature length, and sensitivity to noise due to no special processing of the pixel value around the center pixel value. Thus, TS [13] was not selected for performance comparison in our following experiments. In addition, WT [21], a good recent method of texture analysis, got almost the worst performance, averaging 57.98%. The reason may be that the principal directions of texture images in three data sets are not dominant, and the orientation adjustment cannot improve the performance a lot. And CM [8], a good classical method of texture analysis, achieved 70.71%, much worse than all of multiple resolution LTPs and LBPs, which were all more than 89%. Thus, WT [21], and CM [8] were also not selected for performance comparison in our following experiments. On the other hand, we drew a comparison between the time consumption of the LTP and the LBP. In this experiment, the image data Set1 was adopted. In Set1, every image size is 128  128 pixels. The experiment was done on a PC with P4-CPU 2.0G, 1024 MB memory. Table 2.2 shows the average time consumption of their feature extraction on every image. It is noted that the time consumption of LBP_HP [15] is the same as that of the corresponding LBP. Due to a longer code, the average riu2 riu2 riu0 riu0 riu0 riu2 time consumption of LTP riu2 8;1 þ LTP 16;2 þ LTP 24;3 and LTP 8;1 þ LTP 16;2 þ LTP 24;3 were a little more than that of LBP 8;1 þ riu2 LBPriu2 þ LBP . However, the better classification performance of the LMP given above balances a little higher time 16;2 24;3 requirements. riu2 riu2 For the usage of two separate LBP coding, the average time consumption of LTP TT riu2 8;1 þ LTP TT 16;2 þ LTP TT 24;3 is almost riu2 riu2 riu2 riu2 riu2 two times that of LBPriu2 þ LBP þ LBP , and thus by far more than that of LTP þ LTP þ LTP . 8;1 16;2 24;3 8;1 16;2 24;3

Table 2.2 The time consumption comparison of different LTP and LBP. Mode

LTPriu2 8;1

LTPriu2 16;2

LTP riu2 24;3

LTPriu2 8;1 þ16;2 þ24;3

LTP riu0 8;1 þ16;2 þ24;3

LBPriu2 8;1 þ16;2 þ24;3

LTP riu2 8;1 þ16;2

LTPriu2 8;1 þ24;3

Average timeconsumption (s) Mode

0.010155

0.013141

0.017400

0.038777

0.038564

0.031036

0.021518

0.025850

LTPriu2 16;2 þ24;3

LBPriu2 8;1

LBP riu2 16;2

LBPriu2 24;3

LTP TT riu2 8;1 þ16;2 þ24;3

WT [21]

CM [8]

Average timeconsumption (s)

0.030468

0.007105

0.010582

0.013568

0.050141

0.007425

0.003900

104

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

6.5. Texture classification with random turbulence noise This experiment aims to evaluate the performance comparison of different LTPs and LBPs when different kinds of noise are introduced. Noise setup: here random turbulence noise was added to the three data sets. The random turbulence noise is zero mean, magnitude Vn. If a pixel is xi, then xi + (1)kVn ? xi, here k is a random integer. It is easily seen from above that when Vn equals 0, it means that there is no noise. The noise was not added to the source images in training steps, while the noise was added to all source image data in test steps. In this experiment, the variance Vn ranged from 1 to 5, and the threshold Th0 was equal to 2. Tables 3.1, 3.2, and 3.3 shows the classification performance of LTP, LBP, LBP_HP and the LTP_TT in different noises on Set1, Set2, and Set3, respectively. Regarding the classification performance, the LTP outperforms LTP_TT and LBP_HP, while the last two surpass LBP due to the special processing of the pixel value around the center pixel value, whereas all of the performance decline with increasing noise magnitude. In addition, there is a large drop of all classification performance when the noise magnitude Vn varies from 2 to 3, which is larger than the threshold Th0. Moreover, the classification performance of the LTP drops more slowly than that of the LBP with the increasing small noise magnitude. It is especially true for test sets Set2 and Set 3, in which there is variation in illumination. For example, when noise magnitude Vn changes from 0 to 1, the classification accuracy of LTP riu2 8;1 þ16;2 þ24;3 on Set3 increases from 91.85%% to 93.06%, whereas that of LBPriu2 8;1 þ16;2 þ24;3 drops from 89.47% to 72.36%. It may be due to that the noise reduction process with a special zone defined to describe the interval across zero in the LTP partially counteracts the side effects of the variation in illumination. Thus, the LMP is more robust for noise reduction than the LBP. In addition, the LTPs with different thresholds, ranging from 1 to 10, in different noise, were analyzed on Set1, the results shown in Table 3.4. When Th0 is smaller than the noise magnitude Vn, the accuracies of the LTPs are relatively low and varied irregularly because of the noise. On the other hand, when Th0 is larger than the noise magnitude Vn, the accuracies of the LTPs are relatively high and stable because of the process of noise reduction. Thus, the threshold Th0 is generally set to a value larger than the noise magnitude Vn.

Table 3.1 The classification accuracies of different LTP and LBP in different noise on Set1 (%). Noise

LTPriu2 8;1þ

LTP riu0 8;1þ

LBPriu2 8;1þ

LBP HP riu2 8;1þ

LTP TT riu2 8;1þ

magnitudeVn

16,2 + 24,3

16,2 + 24,3

16,2 + 24,3

+16,2+ 24,3

+16,2 + 24,3

1 2 3 4 5

97.03 95.89 89.22 77.53 65.49

93.98 90.76 78.36 69.53 60.18

80.03 69.84 58.13 47.19 35.94

96.30 92.99 84.87 76.69 62.00

95.94 93.93 85.70 80.36 61.12

Table 3.2 The classification accuracies of different LTP and LBP in different noise on Set2 (%). Noise

LTPriu2 8;1þ

LTPriu0 8;1þ

LBPriu2 8;1þ

LBP HP riu2 8;1þ

LTP TT riu2 8;1þ

magnitudeVn

16,2 + 24,3

16,2 + 24,3

16,2 + 24,3

+16,2 + 24,3

+16,2 + 24,3

1 2 3 4 5

92.48 90.53 81.55 69.56 55.63

91.57 87.04 72.41 64.24 54.10

73.40 62.29 51.88 42.89 31.27

91.11 85.51 74.86 66.32 52.06

91.04 86.60 78.13 67.36 52.75

Table 3.3 The classification accuracies of different LTP and LBP in different noise on Set3 (%). Noise

LTPriu2 8;1þ

LTP riu0 8;1þ

LBPriu2 8;1þ

LBP HP riu2 8;1þ

LTP TT riu2 8;1þ

magnitude Vn

16,2 + 24,3

16,2 + 24,3

16,2 + 24,3

+16,2 + 24,3

+16,2 + 24,3

1 2 3 4 5

93.06 91.37 81.90 67.25 52.57

92.13 87.38 73.70 62.18 51.56

72.36 60.39 50.11 40.16 29.38

89.81 84.19 75.56 65.88 51.39

89.00 85.12 78.13 65.65 50.93

105

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108 Table 3.4 The classification accuracies of the LTP with different thresholds in different noise on Set1 (%). Th0

Mode

LTPriu2 8;1þ16;2þ24;3

NoiseVn

1

3

5

1

3

5

92.17 96.93 97.24 96.09 95.86 95.13 97.16 97.86 97.34 96.88

78.02 89.11 84.87 79.69 84.14 84.77 89.79 91.04 92.61 92.50

53.93 65.03 65.83 62.58 58.05 55.68 55.99 56.04 58.54 59.74

89.40 94.17 94.09 94.43 94.20 94.82 96.33 96.85 97.16 96.72

78.88 77.97 80.03 77.92 83.36 85.83 91.02 91.51 92.40 92.68

53.83 60.70 61.33 58.05 55.03 52.86 55.49 54.66 57.71 61.35

1 2 3 4 5 6 7 8 9 10

LTPriu0 8;1þ16;2þ24;3

6.6. Gray-scale and rotation invariant texture classification with LMP when N > 3 6.6.1. Texture classification with LMP (N > 3) on Set1 This experiment was to evaluate the performance of the LMP when N is more than 3. In this experiment, for simplicity, the riu0 ‘‘uniform-0’’ LMPs with an N of 4 or an N of 5, denoted as LMPriu0 4 P;R and LMP 5 P;R , respectively, were implemented to evaluate the performance of LMP beyond LTP on Set1. Moreover, with different thresholds ranging from 1 to 9, they were implemented to evaluate the performance. riu0 riu0 riu0 riu0 It can be seen from Table 4 that the accuracies of LMPriu0 4 8;1 þ LMP 4 16;2 þ LMP 4 24;3 and LMP 4 16;2 þ LMP 4 24;3 are all more than 93.46%, and they increase with the increasing thresholds, and can achieve the highest classification performance of 97.21% and 98.65%, respectively. Their performances are relatively consistently stable with different thresholds. Similar riu0 riu0 riu0 riu0 conclusion can be drawn about LMPriu0 5 8;1 þ LMP 5 16;2 þ LMP 5 24;3 and LMP 5 16;2 þ LMP 5 24;3 . Moreover, it can be easily found from the comparison of the Tables 2.1 and 4 that the classification accuracy of LMP is not always monotonously improved with the increasing N, while all of the accuracies of LMPs are relatively higher and better than those of LBPs. On the other hand, the average time consumption of the feature extraction on every image of LMPriu0 4 8;1 þ riu0 riu0 riu0 riu0 LMPriu0 4 16;2 þ LMP 4 24;3 and LMP 5 8;1 þ LMP 5 16;2 þ LMP 5 24;3 were 0.036325 s and 0.0379 s, respectively. They were close riu0 riu0 to that of LTPriu0 8;1 þ LTP 16;2 þ LTP 5 24;3 , shown in Table 2.2. It can be concluded that the time consumption of LMP is not riu2 riu2 increasing rapidly with the increasing N, and they are just a little more than that of LBPriu2 8;1 þ LBP 16;2 þ LBP 24;3 . However, the better classification performance of the LMP given above balances a little higher time requirements. In addition, LMP4 and LMP5 with different thresholds and different noise were analyzed on Set1, with results shown in Table 5. A conclusion similar to that of Table 3.4 about threshold Th0, the noise magnitude Vn, and the performance of the LMP can be drawn. When Th0 is larger than the noise magnitude Vn, the accuracies of the LMPs are relatively high and stable due to the definition of a special zone around the center pixel value to process noise. Thus, the threshold Th0 is generally set to a value larger than the noise magnitude Vn. Moreover, it can be concluded from the comparison of Tables 5 and 3.1 that in most case in a noise condition the LMP with a proper threshold can obtain better performance than LBP, LBP_HP, and LTP_TT. And LMPriu0 4 16;2 þ24;3 or LMPriu0 5 16;2 þ24;3 can achieve the best performance. It is further concluded from above that the LMP is robust to noise. 6.6.2. Texture classification on Set4 and Set5 Some other well-known texture datasets were selected for performance comparison. 1) Set4 [4,58]: this set consisted of 13 Brodatz texture classes, and each texture class included seven size 512⁄512 images rotated by different angles (0°, 30°, 60°, 90°, 120°, 150°, and 200°). The training set included one sample with angle 0 per texture class. All of the set was adopted as the test set. It was used for rotation invariant texture classification. 2) Set5: KTH-TIPS (Textures of materials in the real world under varying Illumination, Pose and Scale) [11,5]. It was used in the following three ways. Set5–1: the training set included one sample with scale 1, frontal pose and frontal illumination direction. Set5–2: the training set included one sample with frontal pose and frontal illumination direction in every scale. Set5–3: the training set included all samples with scale

Table 4 The classification accuracy comparison of the LMP when N > 3 on Set1 (%). Th0

LMPriu0 4 8;1 þ16;2 þ24;3

LMPriu0 4 16;2 þ24;3

LMP riu0 5 8;1 þ16;2 þ24;3

LMP riu0 5 16;2 þ24;3

1 3 5 7 9

94.71 94.24 95.10 96.33 97.21

93.46 94.77 96.20 98.20 98.65

94.87 94.24 95.10 96.38 97.19

93.31 94.90 96.09 98.36 98.65

106

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

Table 5 The classification accuracy comparison of the LMP when N > 3 with different thresholds in different noise on Set1 (%). Mode Th0Vn

1 3 5 7 9

LMPriu0 4 8;1 þ16;2 þ24;3

LMP riu0 4 16;2 þ24;3

LMP riu0 5 8;1 þ16;2 þ24;3

LMPriu0 5 16;2 þ24;3

1

3

5

1

3

5

1

3

5

1

3

5

90.70 92.76 94.32 96.30 97.42

81.30 79.40 85.86 90.60 93.20

50.86 64.01 57.34 53.44 56.69

91.93 94.77 96.07 98.31 98.80

86.95 91.82 93.28 96.25 98.59

79.30 78.10 80.49 82.01 87.63

90.49 92.79 94.43 96.25 97.5

80.78 78.98 86.17 90.31 93.49

50.96 64.04 57.58 53.85 55.73

91.80 94.79 95.99 98.31 98.78

87.08 91.67 93.28 96.22 98.31

78.83 77.84 80.44 82.08 87.37

Table 6 The classification accuracy comparison on Set4 and Set5 (%). Dataset

Set4 Set5–1 Set5–2 Set5–3

LBPriu2

LTP_TTriu2

LTPriu2 8,1 + 16,2 + 24,38,1

LMP riu0 4 16,2 + 24,3

LMPriu0 5 8,1 + 16,2 + 24,3

LMPriu0 5 16,2 + 24,3

CM[8]

8,1 + 16,2 + 24,3

LMP riu0 4 +16,2 + 24,3

WT[21]

8,1 + 16,2 + 24,3 93.41 15.31 79.88 39.01

100 16.79 68.89 40.12

100 17.53 70.62 47.04

100 16.79 71.11 48.89

100 18.15 69.26 47.16

100 17.04 69.01 48.15

100 18.40 67.41 44.20

70.33 13.09 52.35 28.52

94.51 9.88 48.40 26.30

Table 7 The classification accuracy comparison of the combined features with VARP,R on Set1 (%). Feature

LBPriu2/VAR8,1

LBPriu2/VAR16,2

Accuracy Feature

93.58 LTPriu2/ VAR8,1 + 16,2 + 24,3 97.06

97.40

Accuracy

LBPriu2/VAR24,3

LBPriu2/ VAR8,1 + 16,2 + 24,3

LTPriu2/ VAR8,1

LTPriu2/ VAR16,2

LTPriu2/ VAR24,3

95.39

95.76 LMP riu0 5 =VAR8;1 þ16;2 þ24;3

96.51 VAR8,1

94.35 VAR16,2

95.83 VAR24,3

93.18

LMPriu0 4 =VAR8;1 þ16;2 þ24;3

97.40

84.71

83.20

77.58

1. All of the above test sets were the whole Set5. Set5–1 was used for scale, rotation and illumination invariant texture classification, Set5–2 was used for rotation and illumination invariant texture classification, and Set5–3 was used for scale invariant texture classification. The experimental results are shown in Table 6. It is easily concluded that the accuracies of LMP (N > =3) are relatively higher whereas the accuracies of WT [21] and CM [8] on all sets are relatively low. And the accuracies of LMP (N > =3) are higher than the corresponding ones of LTP_TT, WT [21] and CM [8]. Also they are higher than those of LBP, except on Set5–2. Furthermore, the relative lower classification accuracies of all features on Set5–1 and Set5–3 show the demand for better features for texture analysis. 6.6.3. Texture classification with LMPriu2 P;R =VARP;R Finally, the combined features with local variance VARP,R were analyzed, and the experimental results are shown in Table 7. It can be found from the comparison of Tables 7, 4, and 2.1 that the combined features with VARP,R generally improved the performance, except (24,3). The addition of the poorly performing VAR24,3 hampered the excellent performance of LTP riu2 24;3 , riu2 riu2 and LTPriu2 8;1þ16;2þ24;3 . In addition, as expected, LMP P;R =VARP;R ðN >¼ 3Þ generally outperforms the corresponding LBP P;R =VARP;R . 7. Discussion and conclusion The local multiple patterns (LMP) operator, proved to be a theoretically and computationally simple yet efficient texture analysis approach, has been presented in this paper. The basic idea of the LMP is to extend binary patterns to multiple patterns to be robust for the image analysis. The ‘‘uniform’’ LMP were shown as a fundamental property of texture as they provide a vast majority of local texture patterns, corresponding to texture microstructures such as spots, edges, curves, flat areas. By estimating the macro-distributions of these microstructures, we can combine structural and statistical texture analysis. The discrete occurrence histogram of the ‘‘uniform’’ LMP patterns computed over an image or a region of image has been shown to be a very powerful texture feature. In comparison with the LBP, the LMP has many advantages. We can draw the following conclusions from the description above. 1) The LMP extends binary patterns to multiple patterns and preserves more structural information than the LBP. The LMP can be more robust for any structural analysis, including the analysis of flat image areas, for which the LBP is unsuitable.

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

107

2) The LMP has a big advantage over the LBP with regard to noise reduction because in the LMP a special zone is defined to describe the interval across zero to overcome the noise. 3) Experimental results show that despite a little more time consumption, the LMP has better classification performance than the LBP in all terms of gray-scale and rotation invariant classification, as well as noise reduction. In addition, the following conclusions can be drawn from theoretical and experimental analysis. 1) It can easily be seen from the experiments that the classification performance of the LMP is relatively consistently better and more reliable with the different threshold Th0. Thus, the threshold Th0 is often set to 2 for simplicity. If there is noise in the image, the threshold Th0 is generally set to a value larger than the noise magnitude Vn. 2) To solve multiresolution analysis, it generally needs to combine multiple LMP operators with several neighbors in different sizes or different shapes. Experimental results involving three different spatial resolutions showed that multiresolution analysis is helpful for better performance. 3) Excellent experimental results, having been achieved in several true problems of rotation invariant texture classification on three well-known texture data sets, showed that our proposed method was robust in terms of gray-scale variation, e.g. resulting from the illumination change. 4) The LMP can be considered as the generalization of LBPs, TS [13], and LTP_TT [12,44]. Experimental results showed that LMPs ( N > =3) overall outperform other methods, such as LBP, LBP_HP, TS, LTP_TT, CM [8], and WT [21]. All of the above show that the presented ‘‘uniform’’ LMP is proved to be a very powerful tool for multiresolution grayscale and rotation invariant texture analysis. The further work will focus on the analysis of LMP subset, and the application of the LMP in face detection, remote sensing, etc. Acknowledgment The authors thank the Prof. S.W. Lu from Memorial University of Newfoundland in Canada, the Editors and the anonymous reviewers for their valuable comments. The authors also would like to thank Dr. H. Zhou and Dr. P. Zhong from National University of Defense Technology in China for their helpful discussion about the LBP operator. References [1] T. Ahonen, A. Hadid, M. Pietikäinen, Face Recognition with Local Binary Patterns, in: 8th European Conf. on Computer Vision- ECCV 2004, ser, Lecture Notes in Computer Science, vol. 3021, Springer, 2004, pp. 469–481. [2] T. Ahonen, A. Hadid, M. Pietikäinen, Face description with local binary patterns: application to face recognition, IEEE Trans. Pattern Anal. Mach. Intell. 28 (12) (2006) 2037–2041. [3] T. Ahonen, M. Pietikäinen, Soft histograms for local binary patterns, in: Proceedings of Finnish Signal Processing Symposium (FINSIG 2007), Oulu, Finland, 2007, pp.1–4. [4] P. Brodatz, Texture: A Photographic Album for Artists and Designers, Dover, New York, 1966. [5] B. Caputo, E. Hayman, M.J. Fritz, J.-O. Eklundh, Classifying materials in the real world, Image Vision Comput. 28 (1) (2010) 150–163. [6] F. Cardinaux, C. Sanderson, and S. Bengio, Face Verification Using Adapted Generative Models, in: IEEE Conf. Automatic Face and Gesture Recognition (AFGR), 2004, pp. 825–830. [7] C.C. Chang, C.J. Lin, LIBSVM: a Library for Support Vector Machines. . [8] L. Davis, S. Johns, K. Aggarwal, Texture analysis using generalized cooccurrence matrices, IEEE Trans. Pattern Anal. Mach. Intell. n1 (1979) 251–259. [9] X. Feng, J. Cui, M. Pietikäinen, A. Hadid, Real time facial expression recognition using local binary patterns and linear programming, in: Advances in Artificial Intelligence, MICAI 2005 Proceedings, Lecture Notes in Computer Science 3789, 2005, pp. 328–336. [10] X. Feng, M. Pietikäinen, A. Hadid, Facial expression recognition with local binary patterns and linear programming, Pattern Recogn. Image Anal. 15 (2) (2005) 546–548. [11] E. Hayman, B. Caputo, M.J. Fritz, J.-O. Eklund, On the significance of real-world conditions for material classification, in: 8th European Conf. on Computer Vision – ECCV 2004, Lecture Notes in Computer Science, vol. 4, Springer, Prague, 2004, pp. 253–266. [12] T. Gritti, C. Shan, V. Jeanne, and R. Braspenning, Local features based facial expression recognition with face registration errors, in: Proceedings of IEEE Int. Conf. Automatic Face and Gesture Recognition (FG), 2008, pp. 1–8. [13] D.C. He, L. Wang, Texture unit, texture spectrum, and texture analysis, IEEE Trans. Geosci. remote sens. 28 (4) (1990) 509–512. [14] L.H. He, C.R. Zou, L. Zhao, D. Hu, An enhanced LBP feature based on facial expression recognition, in: Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference, Shanghai, China, September 1–4, 2005, pp. 3300–3303. [15] M. Heikkilä, M. Pietikäinen, A texture-based method for modeling the background and detecting moving objects, IEEE Trans. Pattern Anal. Mach. Intell. 28 (4) (2006) 657–662. [16] G. Heusch, Y. Rodriguez, S. Marcel, Local Binary Patterns as an Image Preprocessing for Face Authentication, in: The 7th International Conference on Automatic Face and Gesture Recognition (FGR’06), 2006, pp. 9–14. [17] G.W. Jiji, L. Ganesan, S.S. Ganesh, Unsupervised texture classification, J. Theor. Appl. Inf. Technol. 5 (4) (2009) 373–381. [18] G.W. Jiji, L. Ganesan, A new approach for unsupervised segmentation, Appl. Soft Comput. 10 (3) (2010) 689–693. [19] B.J. Jun, T.W. Kim, D.J. Kim, A compact local binary pattern using maximization of mutual information for face analysis, Pattern Recogn. 44 (3) (2011) 532–543. [20] H.L. Jin, Q.S. Liu, H.Q. Lu, X.F. Tong, Face Detection using Improved LBP under Bayesian Framework, in: The Third International Conference on Image and Graphics (ICIG’04), 2004, pp. 306–309. [21] K.J. Khouzani, S.Z. Hamid, Radon transform orientation estimation for rotation invariant texture analysis, IEEE Trans. Pattern Anal. Mach. Intell. 27 (6) (2005) 1004–1008. [22] X. Li, W.M. Hu, Z.F. Zhang, H.Z. Wang, Heat kernel based local binary pattern for face representation, IEEE Signal Process. Lett. 17 (3) (2010) 308–311. [23] S. Lazebnik, C. Schmid, J. Ponce, A sparse texture representation using local affine regions, IEEE Trans. Pattern Anal. Mach. Intell. 27 (8) (2005) 1265– 1278.

108

C. Zhu, R. Wang / Information Sciences 187 (2012) 93–108

[24] T. Mäenpää, The Local Binary Pattern Approach to Texture Analysis—Extensions and Applications, Academic Dissertation, Infotech Oulu and Department of Electrical and Information Engineering, Springer, Finland, 2003. [25] T. Mäenpää, M. Pietikäinen, T. Ojala, Texture classification by multi-predicate local binary pattern operators, in: Proceedings of 15th International Conference on Pattern Recognition, Barcelona, Spain, 2000, pp. 951–954. [26] T. Mäenpää, T. Ojala, M. Pietikäinen, M. Soriano, Robust texture classification by subsets of local binary patterns, in: 15th International Conference of Pattern Recognition, Barcelona, Spain, vol. 3, 2000, pp. 947–950. [27] T. Mäenpää, M. Pietikäinen, Multi-scale binary patterns for texture analysis, in: Proceedings of 13th Scandinavian Conference on Image Analysis, G oteborg, Sweden, 2003. ¨ [28] G. Martens, C. Poppe, P. Lambert, R. Van De Walle, Unsupervised texture segmentation and labeling using biologically inspired features, in: IEEE 10th Workshop on Multimedia Signal Processing, MMSP 2008, pp 159–164 [29] G. Martens, C. Poppe, P. Lambert, R. Van De Walle, Noise- and compression-robust biological features for texture classification, Visual Comput. (2010) 1–8. [30] G. Martens, C. Poppe, P. Lambert, R. Van De Walle, Perceptual-based textures for scene labeling: A bottom-up and a top-down approach, in: 2010 5th International Conference on Future Information Technology, 2010. [31] M. Petrou, P. Garcia-Sevilla, Image Processing: Dealing with texture, Wiley, 2006. [32] M. Pietikäinen, A. Hadid, Texture features in facial image analysis, in: Advances in Biometric Person Authentication, IWBRS 2005 Proceedings, Lecture Notes in Computer Science 3781, 2005, pp. 1-=8. [33] M. Pietikäinen, T. Ojala, Z. Xu, Rotation-invariant texture classification using feature distributions, Pattern Recogn. 33 (2000) 43–52. [34] M. Pietikäinen, Image analysis with local binary patterns, in: Image Analysis, SCIA 2005 Proceedings, Lecture Notes in Computer Science, vol. 3540, Springer, 2005, pp. 115–118. plenary presentation. [35] T. Ojala, M. Pietikäinen, T. Mäenpää, Multiresolution Gray-Scale and Rotation Invariant Texture Classification with Local Binary patterns, IEEE Trans. Pattern Anal. Mach. Intell. 24 (7) (2002) 971–987. [36] T. Ojala, M. Pietikäinen, D. Harwood, A comparative study of texture measures with classification based on featured distributions, Pattern Recogn. 29 (1) (1996) 51–59. [37] T. Ojala, T. Mäenpää, M. Pietikäinen, J. Viertola, J. Kyll ¨onen, S. Huovinen, Outex - A New framework for empirical evaluation of texture analysis algorithms, in: Proceedings of 16th Inter. Conf. on Pattern Recognition, Qu’ebec City, Canada, 2002, pp. 701–706. [38] T. Ojala, M. Pietikäinen, Unsupervised texture classification using feature distributions, Pattern Recogn. 32 (1999) 477–486. [39] T. Ojala, M. Pietikäinen, T. Mäenpää, Gray scale and rotation invariant texture classification with Local Binary Patterns, in: Proceedings of Sixth European Conference on Computer Vision, Dublin, Ireland, 2000, pp. 404–420. [40] T. Ojala, K. Valkealahti, E. Oja, M. Pietikäinen, Texture discrimination with multidimensional distributions of signed gray level differences, Pattern Recogn. 34 (2001) 727–739. [41] T. Ojala, M. Pietikäinen and T. Mäenpää. A Generalized Local Binary Pattern Operator for Multiresolution Gray Scale and Rotation Invariant Texture Classification, , Machine Vision and Media Processing Unit, Infotech Oulu, University of Oulu, Finland. [42] T. Randen, J.H. Husoy, Filtering for Texture Classification: A Comparative Study, IEEE Trans. Pattern Anal. Mach. Intell. 21 (1999) 291–310. [43] V. Takala, T. Ahonen, and M. Pietik¨ ainen, Block-based methods for image retrieval using local binary patterns, in: Proceedings of 14th Scandinavian Conference on Image Analysis (SCIA), Joensuu, Finland, 2005, pp. 882–891. [44] X. Tan and B. Triggs, Enhanced local texture feature sets for face recognition under difficult lighting conditions, in Proceedings of Analysis and Modeling of Faces and Gestures (AMFG), 2007, pp. 168–182. [45] H.L. Tang. Y.F. Sun, B.C.Yin, Y. Ge, Face recognition based on Haar LBP histogram, in: ICACTE 2010 – 2010 3rd International Conference on Advanced Computer Theory and Engineering 6 (2010) 6235–6238. [46] M. Turtinen, M. Pietikäinen, O. Silven, Visual characterization of paper using isomap and local binary patterns, IEICE Trans. Inf. Syst. E89-D (7) (2006) 2076–2083. [47] V.N. Vapnik, The Nature of Statistical Learning Theory, Springer-Verlag, 1995. [48] V.N. Vapnik, Statistical learning theory, John Wiley & Sons, New York, 1998. [49] M. Varma, A. Zisserman, A statistical approach to texture classification from single images, Int. J. Comput. Vision 62 (1–2) (2005) 61–81. [50] H.Z. Wang, A new rotation invariant Gabor features for texture image retrieval, J. Inf. Comput. Sci. 6 (2) (2009) 561–568. [51] Groissboeck Werner, Lughofer Edwin, Thumfart Stefan, Associating visual textures with human perceptions using genetic algorithms, Inf. Sci. 180 (11) (2010) 2065–2084. [52] J. Zhang, M. Marszalek, S. Lazebnik, C. Schmid, Local features and kernels for classification of texture and object categories: a comprehensive study, Int. J. Comput. Vision 73 (2) (2007) 213–238. [53] X.D. Zhang, N.H. Younan, C.G. O’Hara, Wavelet domain statistical hyperspectral soil texture classification, IEEE Trans. Geosci. Remote Sens. 43 (3) (2005) 615–618. [54] G. Zhang, X. Huang, S.Z. Li, Y. Wang, X. Wu, Boosting local binary pattern (LBP)-based face recognition, in: Chinese Conference on Biometric Recognition, SINOBIOMETRICS 2004, 2004. [55] W. Zhang, S. Shan,W. Gao, X. Chen, H. Zhang, Local gabor binary pattern histogram sequence (LGBPHS): A novel non-statistical model for face representation and recognition, in: International Conference on Computer Vision, ICCV’2005, pp. 786–791. [56] Y. Zhang, T.L. Chai, C.C. Hung. Local binary patterns for face recognition under varying variations. In: ACM International Conference 2010, 6th Annual Cyber Security and Information Intelligence Research Workshop: Cyber Security and Information Intelligence Challenges and Strategies, CSIIRW10. [57] H. Zhou, R.S. Wang, C. Wang, A novel extended local binary pattern operator for texture analysis, Inf. Sci. 178 (22) (2008) 4314–4325. [58] The USC-SIPI Image Database. .