- Email: [email protected]
Color face recognition by PCA-like approach
Neurocomputing 152 (2015) 231–235
Contents lists available at ScienceDirect
Neurocomputing journal homepage: www.elsevier.com/locate/neucom
Color face recognition by PCA-like approach Xinguang Xiang a,n, Jing Yang b, Qiuping Chen a a b
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, PR China School of Arts, Chongqing University, Chongqing 400044, PR China
art ic l e i nf o
a b s t r a c t
Article history: Received 18 June 2014 Received in revised form 25 October 2014 Accepted 28 October 2014 Available online 21 November 2014
In this paper, a novel technique aimed to make full use of the color cues is proposed to improve the accuracy of color face recognition based on principal component analysis. Principal component analysis (PCA) has been an important method in the field of face recognition since the very early stage. Later, two-dimensional principal component analysis (2DPCA) was developed to improve the accuracy of PCA. However, the color information is omitted since the images need to be transformed into a greyscale version before applying both of the two methods. In order to exploit the color information to recognize faces, we propose a novel technique which utilizes color images matrix-representation model based on the framework of PCA for color face recognition. Furthermore, a color 2DPCA (C2DPCA) method is devised to combine the spatial and color information for color face recognition. Experiment results show that our proposed methods can achieve higher accuracy than regular PCA methods. & 2014 Elsevier B.V. All rights reserved.
Keywords: Face recognition Principal component analysis Eigenface Color cues
1. Introduction Face recognition has received significant attention and still been a hot research topic in the field of image analysis and understanding during the past twenty years, since its important role in various applications, such as identification, video surveillance, and social security. Principal component analysis (PCA), which is widely used in the area of pattern recognition and computer vision, is a classical feature extraction and data representation technique. Sirovich and Kirby [1] argued that any face image could be reconstructed approximately as a weighted sum of a small collection of images that define a facial basis. Based on their theory, Turk and Pentland [2] presented the well-known Eigenface method for face recognition in 1991. From then on, PCA attracted lots of attention and has become one of the most successful approaches in face recognition [3–5]. In recent years, some more sophisticated techniques [6–15] are proposed for pattern recognition, and can be utilized for face recognition. For example, multimodal learning methods [6–9] are proposed to integrate multiple features for clustering or classification. Multigraph learning and hyper-graph analysis [10–12] are employed for object recognition. Also, some recognition schemes which utilized in image annotation [13–15] are interesting and enlightening. Yang et al. [16] proposed a novel technique named twodimensional principal component analysis (2DPCA), which extends
n
Corresponding author. E-mail addresses: [email protected] (X. Xiang), [email protected] (J. Yang), [email protected] (Q. Chen). http://dx.doi.org/10.1016/j.neucom.2014.10.074 0925-2312/& 2014 Elsevier B.V. All rights reserved.
the traditional PCA method and does not need to transform the face image matrix into a vector for preprocessing. This method demonstrated better performance as a result of lower dimension and the utilizing of the spatial information of images. However, both methods cannot be directly applied to the color images. The color images need to be transformed into greyscale versions first. Obviously, the color information is lost during the transformation. Some former research has also evinced that color cues contribute in recognizing faces, especially when shape cues of the images are degraded [17]. Several works [18,19] have noticed the importance of color information. Torres et al. [18] extended traditional PCA to color face recognition by using the R, G, B color channel, respectively. Face recognition is conducted on each color channel and the final result comes from the fusion of the three color channels. Yang et al. [19] presented a general discriminant model for color face recognition. The model uses a set of color component combination coefficients to convert three color channels of the face image into one channel to represent color face images in the form of onedimensional vector for recognition. We believe that all the three components of each pixel can benefit the task of face recognition and there indeed exists some inner correlations among them. However, most previous works make use of the three color channels respectively which would omit the inner relationships between channels. In this paper, two novel color face recognition method are proposed based on the framework of PCA and 2DPCA. First, based on the framework of PCA, we propose a novel scheme employing a color images matrixrepresentation model for color face recognition. Furthermore, we define the concept of color value which regards the three channels as a whole, and by providing the basic operations on color values,
232
X. Xiang et al. / Neurocomputing 152 (2015) 231–235
we devise the C2DPCA method applying the existing framework of 2DPCA directly to color face images. The rest of the paper is organized as follows. The proposed color face recognition method which combines the framework of PCA will be introduced in Section 2. In Section 3, the concept of color value will be formally introduced, and we will demonstrate the rationality of this model by looking further into the approach we discussed in the former section. Then, our final method that combines the color value concept with the framework of 2DPCA will be presented. Section 4 will show the experimental results of the color face recognition on the Georgia Tech (GATech) color face databases [20]. Conclusion is drawn in Section 5.
Then we get d feature vectors Y 1 ; …; Y d . Right after that, a nearest ðiÞ ðiÞ neighbor classifier is used for classification. Let Y ðiÞ 1 ; Y 2 ; …; Y d denote ðjÞ ðjÞ the feature vectors obtained from image Ai , and Y ðjÞ 1 ; Y 2 ; …; Y d denote the feature vectors obtained from image Aj , then we define the difference between image Ai and Aj as follows: d
Y ðjÞ jj2 dðAi ; Aj Þ ¼ ∑ jjY ðiÞ k k
ð4Þ
k¼1
Y ðjÞ jj2 denotes the Euclidean distance between the two where jjY ðiÞ k k feature vectors Y ðiÞ and Y ðjÞ . When given a test sample A, if k k dðA; Al Þ ¼ minj dðA; Aj Þ, then we classify A into the same class that Al belongs to.
2. Color face recognition based on PCA We call this method as ‘based on PCA’ because the presentation of a color image is very similar to what PCA does, both of the methods flatten the pixels of an image as a row. For convenience, we will call this method as color principal component analysis (CPCA) in the following. CPCA can be comprehended in two ways, one is the matrix-representation model, and the other is the model which introduces the concept of ‘color value’. The latter will be discussed in detail in the next section. 2.1. Matrix-representation model We present a color image as a m n matrix, where m is the number of color channels (usually 3), and n is the number of pixels. The color image is flattened like what the PCA approach does, and each column of the matrix is a color vector that represents one image pixel. By this way, every color face image can be presented as a two-dimensional matrix. Let A denote the image matrix, and X denote an n-dimensional unitary column vector. Our idea is to project image A onto X by the following transformation: Y ¼ AX:
ð1Þ
Thus, an m-dimensional projected vector Y is obtained, and it is called the projected feature vector of image A. The key problem is how to define a projection vector X as a good one. The theory of 2DPCA [16] has already solved this problem, it has been proved that the best projection vector X is the eigenvector of Gt corresponding to the largest eigenvalue, where Gt is defined as follows: h i 1 M ∑ ðA AÞT ðAj AÞ: Gt ¼ E ðA EAÞT ðA EAÞ ¼ Mj¼1 j
ð2Þ
Obviously, only one optimal projection vector is not enough, we often choose a set of projection vectors which subject to the orthonormal constraints. In fact, the optimal projection vectors X 1 ; …; X d are the orthonormal eigenvectors of Gt corresponding to the first d largest eigenvalues.
Now we are able to handle color faces with CPCA approach. But there is still one flaw leaving behind. In CPCA approach, although the color information is utilized, the spatial information is destroyed when we flatten the pixels of an image as a row. In order to preserve the spatial information, we proposed a 2DPCA based color face recognition method, which is called as C2DPCA (abbreviation of color 2DPCA) for convenience in the following. However, it seems hard to explain C2DPCA in matrix form, because the color image with spatial cues is three-dimensional which cannot even be presented in regular matrix form. Thus, the key factors of our proposed C2DPCA method are as follows:
The three components of a color pixel are regarded as a whole
item, and the color pixel calculated or processed as a basic unit with the proposed concept of color value. We define the basic operations of color value, thus the color values can be regarded as normal scalar values for computing.
With the proposed concept of color value and its operations, we establish the color value representation model for color images. Furthermore, color face recognition can be implemented by applying the framework of 2DPCA based on the new basic computing elements (color values). 3.1. Color value representation model In the previous section, we represent the color image problem in two-dimensional matrix form and finally we solve it with the theory of 2DCPA [16]. However, the image matrix form in CPCA does not have the intuitive meaning like PCA in which every element of the image vector is definitely a value of one pixel. Our original idea is that, we want to regard the three components of a color pixel as a whole, and to calculate or process the color pixel as a basic unit. Let’s rewrite the image matrix in CPCA as below: A ¼ ½c1 ; c1 ; …; cn T
2.2. Classification method First, the optimal projection vectors X 1 ; …; X d are utilized for feature extraction. Let Y k ¼ AX k ;
3. Color face recognition based on 2DPCA
k ¼ 1; 2; …; d:
ð3Þ
ð5Þ
here we regard the color image as a column ‘vector’, and every element of this ‘vector’ is a ‘value’ of the pixel—a color vector indeed. We call this ‘vector’ as composite vector, and this ‘value’ as color value. Then we are going to apply the conventional PCA on
Classifier Color face images
Color Value Representation Model
Color value vector
2DPCA
projection vectors
Fig. 1. The flowchart of proposed C2DPCA scheme for color face recognition.
Recognition
X. Xiang et al. / Neurocomputing 152 (2015) 231–235
233
Fig. 2. Sample images from the GATech color face database.
these image composite vectors. That is, we need to find a projection vector X to project A to a color value Y ¼ AX. And the projection should maximize the total scatter of the projected color values Y. Assume that there are M samples, and A denotes the average of all the composite vectors. Then we define the sample ‘matrix’ as: h iT ð6Þ D ¼ ðA1 AÞT ; ðA2 AÞT ; …; ðAM AÞT where every row of D represents a centered sample. It should be noted that D is not a normal matrix, and it is a composite matrix since the basic element of D is not a scalar value but a color value. The theory of conventional PCA [21] tells us, the optimal projection vector X can be obtained by calculating the eigenvector of DT D. The problem is how we calculate DT D, since the basic element of D is now a color value. Our key idea is to define the basic operations of color value such as addition and substraction. Once the basic operations are defined for color values, the color values can be regarded as normal scalar values for computing. Since there are only three kinds of basic operations in the process of calculating DT D, thus, we only define the following three operations for color value:
Addition: the sum of two color values is the sum of these two color vectors.
Substraction: the difference of two color values is the difference of these two color vectors.
Multiplication: the product of two color values is the inner product of these two color vectors. It’s intuitively to understand the definition of addition and substraction, the only doubt is the definition of multiplication. In the process of calculating DT D in conventional PCA, the operation of multiplication is intrinsically used to calculate the covariance of two features, and multiplication reflects the correlation of two centered feature value. If the result of multiplication is zero, they are independent; otherwise they are positively or negatively
correlated. It’s wonderful that inner product has the same property. If the inner product of two centered feature vector is zero, they are independent (they are mutually perpendicular), otherwise they are positively or negatively correlated. With the above definitions, we can now calculate A and DT D easily, just regard the color value as a basic unit while computing. And what’s more, after the calculation, we find that: M
DT D ¼ ∑ ðAj AÞT ðAj AÞT ¼ M Gt j¼1
ð7Þ
That means, the eigenvectors of DT D are the same with Gt (see Section 2), and we will get the same projection vectors by this way. It again evinced the rationality of our color value model. 3.2. Color 2DPCA With our color value model, we can finally give our C2DPCA approach, which is shown as Fig. 1. As mentioned above, we want to preserve both the color information and the spatial information. However, the color face with both color and spatial cues could not even be presented as regular two-dimensional matrix. That’s why the color value model is introduced. With that model, all we need to do is just to present the color image as a composite matrix whose basic element is color value. By integrating the algorithm of 2DPCA with our color value model, we can get the optimal projection vectors. After that, a nearest neighbor classifier is also applied, and the details are the same as Yang’s paper [16], however the basic computing elements become color values. It should be noted that the computational complexity of our proposed algorithm is similar to 2DPCA, since the computing process is similar, while the basic computing element is different. 4. Experimental results To evaluate the performance of our color face recognition method (CPCA, C2DPCA), we conduct our experiments on the
234
X. Xiang et al. / Neurocomputing 152 (2015) 231–235
recognition methods are tested in our experiments, and they are PCA scheme, 2DPCA scheme, our proposed CPCA scheme and our proposed C2DPCA scheme. In our experiment, all these face images were resized to the resolution of 33 44, thus the number of pixels in each face image is n ¼ 33 44. The RGB color space is used, so the parameter m is 3. Besides, there are two other parameters. One is the number of images to be utilized for training. We use k to denote how many images per individual are used for training, so the total number of training sets is 50 k, and the remaining images are used for testing. The other parameter is the number of principal eigenvectors d. Meanwhile, gray-level face images are obtained from the corresponding color face images, and they are used to test the conventional PCA and 2DPCA. Several sets of experiments are carried. Fig. 3 shows the experimental results on k¼ 7, 10, and 13. From these results, we can see that our proposed C2DPCA can achieve the highest accuracy among the four face recognition schemes when we adjust the parameter d under the same k. That is not surprising, because C2DPCA approach utilizes both the color and spatial information. The next is 2DPCA, then CPCA and PCA. From the results, we can also see that C2DPCA performs better than 2DPCA, and CPCA performs better than PCA for the utilization of color cues. It should be noted the other phenomenon that C2DPCA and 2DPCA achieve their highest accuracy before CPCA and PCA on the dimension axis. That is because the largest number of principal eigenvectors is 33 44 for CPCA and PCA, while it is 33 for C2DPCA and 2DPCA. In other words, the eigenvector of C2DPCA and 2DPCA can have stronger expressive power because of the utilization of spatial cues. Experimental results show that our proposed color value model can greatly benefit the performance of conventional PCA and 2DPCA, and our final proposed C2DPCA approach can achieve the best performance among these schemes.
5. Conclusion A novel color value model for color face recognition is proposed in this paper. By integrating with conventional PCA and 2DPCA, respectively, the CPCA and C2DPCA approach are proposed. We explained our color value model’s rationality by the comparison with matrix-representation model. Experimental results show that C2DPCA can achieve the best performance among the four approaches by utilizing both the spatial and color cues. Our C2DPCA and CPCA approach also perform better than 2DPCA and PCA respectively for the using of color cue.
Acknowledgements This work was supported in part by the Natural Science Foundation of China under Grant 61301106 and 61327013, the Natural Science Foundation of Jiangsu Province under Grant BK2012397, Research Fund for the Doctoral Program of Higher Education of China under Grant no. 20123219120024, the Fundamental Research Funds for the Central Universities under Grant no. 30920130121003, the Jiangsu Key Laboratory of Image and Video Understanding for Social Safety (Nanjing University of Science and Technology) under Grant no. 30920130122006. Fig. 3. Comparison results under various k.
GATech color face database [20]. The GATech images are composed of color images of 50 individuals with 15 views per individual, and with no specific order in their viewing direction. Fig. 2 shows some sample images from GATech color face database. Four face
References [1] M. Kirby, L. Sirovich, Application of the karhunenloeve procedure for the characterization of human faces, IEEE Trans. Pattern Anal. Mach. Intell. 12 (1) (1990) 103–108. [2] M. Turk, A. Pentland, Eigenfaces for recognition, J. Cogn. Neurosci. 3 (1) (1991) 71–86.
X. Xiang et al. / Neurocomputing 152 (2015) 231–235
[3] M.A. Grudin, On internal representations in face recognition systems, Pattern Recognit. 33 (7) (2000) 1161–1177. [4] A. Pentland, Looking at people: sensing for ubiquitous and wearable computing, IEEE Trans. Pattern Anal. Mach. Intell. 22 (1) (2000) 107–119. [5] D. Valentin, H. Abdi, A.J. O’Toole, G.W. Cottrell, Connectionist models of face processing: a survey, Pattern Recognit. 27 (9) (1994) 1209–1230. [6] J.u.n. Yu, Meng Wang, Dacheng Tao, Semi-supervised multiview distance metric learning for cartoon synthesis, IEEE Trans. Image Processing 21 (11) (2012) 4636–4648. [7] J.u.n. Yu, Yong Rui, Dacheng Tao, Click prediction for web image reranking using multimodal sparse coding, IEEE Trans. Image Processing 23 (5) (2014) 2019–2032. [8] J.u.n. Yu, Dongquan Liu, Dacheng Tao, Hock Soon Seah, On combining multiview features for cartoon character retrieval and clip synthesis, IEEE Trans. Syst. Man Cybern. Part B Cybern. 42 (5) (2012) 1413–1427. [9] Y.u.e. Gao, Meng Wang, Zheng-Jun Zha, Jialie Shen, Xuelong Li, Xindong Wu, Visual–textual joint relevance learning for tag-based social image search, IEEE Trans. Image Processing 22 (1) (2013) 363–376. [10] Meng Wang, Xian-Sheng Hua, Richang Hong, Jinhui Tang, Guo-Jun Qi, Y.a.n. Song, Unified video annotation via multigraph learning, IEEE Trans. Circuits Syst. Video Technol. 19 (5) (2009) 733–746. [11] Meng Wang, Xian-Sheng Hua, Jinhui Tang, Richang Hong, Beyond distance measurement: constructing neighborhood similarity for video annotation, IEEE Trans. Multimedia 11 (3) (2009) 465–476. [12] Y.u.e. Gao, Meng Wang, Dacheng Tao, Rongrong Ji, Qionghai Dai, 3D object retrieval and recognition with hypergraph analysis, IEEE Trans. Image Processing 21 (9) (2012) 4290–4303. [13] Jinhui Tang, Zheng-Jun Zha, Dacheng Tao, Tat-Seng Chua, Semantic-gap oriented active learning for multi-label image annotation, IEEE Trans. Image Processing 21 (4) (2012) 2354–2360. [14] Jinhui Tang, Richang Hong, Shuicheng Yan, Tat-Seng Chua, Guo-Jun Qi, Ramesh Jain, Image annotation by kNN-sparse graph-based label propagation over noisily-tagged web images, ACM Trans. Intell. Syst. Technol. 2 (2) (2011) 111–126. [15] Jinhui Tang, Qiang Chen, Meng Wang, Shuicheng Yan, Tat-Seng Chua, Ramesh Jain, Towards optimizing human labeling for interactive image tagging, ACM Trans. Multimedia Comput. Commun. Appl. 9 (4) (2013) 29. [16] J. Yang, D. Zhang, A.F. Frangi, J.Y. Yang, Two dimensional PCA: a new approach to appearance-based face representation and recognition, IEEE Trans. Pattern Anal. Mach. Intell. 26 (1) (2004) 131–137. [17] A. Yip, P. Sinha, Role of color in face recognition, J. Vis. 2 (7) (2002) (article 596). [18] L. Torres, J.Y. Reutter, L. Lorente, The importance of the color information in face recognition, in: IEEE International Conference on Image Processing, vol. 3,1999, pp. 627–631. [19] J. Yang, C. Liu, A general discriminant model for color face recognition, in: IEEE 11th International Conference on Computer Vision, 2007, pp. 1–6. [20] 〈http://www.anefian.com/research/face reco.htm〉. [21] S. Watanabe, Karhunen-loeve expansion and factor analysis, theoretical remarks and applications, in: Proceedings of Fourth Prague Conference Information Theory, 1965.
235 Xinguang Xiang received the B.A., M.S. and Ph.D. degrees in computer science from the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin, China, in 2005, 2007, and 2011, respectively. He is currently with the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. His current research interests include face recognition, image analysis and understanding, video processing, and video coding.
Jing Yang he has been pursuing the M.S. degree from the School of Arts, Chongqing University, Chongqing, China, since 2012. His current research interests include image analysis and understanding.
Qiuping Chen received the B.A. degree from the School of Computer Science and Technology, Xuzhou Institute of Technology, China, in 2012. Since 2012, he has been pursuing the M.S. degree from the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. His current research interests include face recognition, image analysis and understanding.
Contents lists available at ScienceDirect
Neurocomputing journal homepage: www.elsevier.com/locate/neucom
Color face recognition by PCA-like approach Xinguang Xiang a,n, Jing Yang b, Qiuping Chen a a b
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, PR China School of Arts, Chongqing University, Chongqing 400044, PR China
art ic l e i nf o
a b s t r a c t
Article history: Received 18 June 2014 Received in revised form 25 October 2014 Accepted 28 October 2014 Available online 21 November 2014
In this paper, a novel technique aimed to make full use of the color cues is proposed to improve the accuracy of color face recognition based on principal component analysis. Principal component analysis (PCA) has been an important method in the field of face recognition since the very early stage. Later, two-dimensional principal component analysis (2DPCA) was developed to improve the accuracy of PCA. However, the color information is omitted since the images need to be transformed into a greyscale version before applying both of the two methods. In order to exploit the color information to recognize faces, we propose a novel technique which utilizes color images matrix-representation model based on the framework of PCA for color face recognition. Furthermore, a color 2DPCA (C2DPCA) method is devised to combine the spatial and color information for color face recognition. Experiment results show that our proposed methods can achieve higher accuracy than regular PCA methods. & 2014 Elsevier B.V. All rights reserved.
Keywords: Face recognition Principal component analysis Eigenface Color cues
1. Introduction Face recognition has received significant attention and still been a hot research topic in the field of image analysis and understanding during the past twenty years, since its important role in various applications, such as identification, video surveillance, and social security. Principal component analysis (PCA), which is widely used in the area of pattern recognition and computer vision, is a classical feature extraction and data representation technique. Sirovich and Kirby [1] argued that any face image could be reconstructed approximately as a weighted sum of a small collection of images that define a facial basis. Based on their theory, Turk and Pentland [2] presented the well-known Eigenface method for face recognition in 1991. From then on, PCA attracted lots of attention and has become one of the most successful approaches in face recognition [3–5]. In recent years, some more sophisticated techniques [6–15] are proposed for pattern recognition, and can be utilized for face recognition. For example, multimodal learning methods [6–9] are proposed to integrate multiple features for clustering or classification. Multigraph learning and hyper-graph analysis [10–12] are employed for object recognition. Also, some recognition schemes which utilized in image annotation [13–15] are interesting and enlightening. Yang et al. [16] proposed a novel technique named twodimensional principal component analysis (2DPCA), which extends
n
Corresponding author. E-mail addresses: [email protected] (X. Xiang), [email protected] (J. Yang), [email protected] (Q. Chen). http://dx.doi.org/10.1016/j.neucom.2014.10.074 0925-2312/& 2014 Elsevier B.V. All rights reserved.
the traditional PCA method and does not need to transform the face image matrix into a vector for preprocessing. This method demonstrated better performance as a result of lower dimension and the utilizing of the spatial information of images. However, both methods cannot be directly applied to the color images. The color images need to be transformed into greyscale versions first. Obviously, the color information is lost during the transformation. Some former research has also evinced that color cues contribute in recognizing faces, especially when shape cues of the images are degraded [17]. Several works [18,19] have noticed the importance of color information. Torres et al. [18] extended traditional PCA to color face recognition by using the R, G, B color channel, respectively. Face recognition is conducted on each color channel and the final result comes from the fusion of the three color channels. Yang et al. [19] presented a general discriminant model for color face recognition. The model uses a set of color component combination coefficients to convert three color channels of the face image into one channel to represent color face images in the form of onedimensional vector for recognition. We believe that all the three components of each pixel can benefit the task of face recognition and there indeed exists some inner correlations among them. However, most previous works make use of the three color channels respectively which would omit the inner relationships between channels. In this paper, two novel color face recognition method are proposed based on the framework of PCA and 2DPCA. First, based on the framework of PCA, we propose a novel scheme employing a color images matrixrepresentation model for color face recognition. Furthermore, we define the concept of color value which regards the three channels as a whole, and by providing the basic operations on color values,
232
X. Xiang et al. / Neurocomputing 152 (2015) 231–235
we devise the C2DPCA method applying the existing framework of 2DPCA directly to color face images. The rest of the paper is organized as follows. The proposed color face recognition method which combines the framework of PCA will be introduced in Section 2. In Section 3, the concept of color value will be formally introduced, and we will demonstrate the rationality of this model by looking further into the approach we discussed in the former section. Then, our final method that combines the color value concept with the framework of 2DPCA will be presented. Section 4 will show the experimental results of the color face recognition on the Georgia Tech (GATech) color face databases [20]. Conclusion is drawn in Section 5.
Then we get d feature vectors Y 1 ; …; Y d . Right after that, a nearest ðiÞ ðiÞ neighbor classifier is used for classification. Let Y ðiÞ 1 ; Y 2 ; …; Y d denote ðjÞ ðjÞ the feature vectors obtained from image Ai , and Y ðjÞ 1 ; Y 2 ; …; Y d denote the feature vectors obtained from image Aj , then we define the difference between image Ai and Aj as follows: d
Y ðjÞ jj2 dðAi ; Aj Þ ¼ ∑ jjY ðiÞ k k
ð4Þ
k¼1
Y ðjÞ jj2 denotes the Euclidean distance between the two where jjY ðiÞ k k feature vectors Y ðiÞ and Y ðjÞ . When given a test sample A, if k k dðA; Al Þ ¼ minj dðA; Aj Þ, then we classify A into the same class that Al belongs to.
2. Color face recognition based on PCA We call this method as ‘based on PCA’ because the presentation of a color image is very similar to what PCA does, both of the methods flatten the pixels of an image as a row. For convenience, we will call this method as color principal component analysis (CPCA) in the following. CPCA can be comprehended in two ways, one is the matrix-representation model, and the other is the model which introduces the concept of ‘color value’. The latter will be discussed in detail in the next section. 2.1. Matrix-representation model We present a color image as a m n matrix, where m is the number of color channels (usually 3), and n is the number of pixels. The color image is flattened like what the PCA approach does, and each column of the matrix is a color vector that represents one image pixel. By this way, every color face image can be presented as a two-dimensional matrix. Let A denote the image matrix, and X denote an n-dimensional unitary column vector. Our idea is to project image A onto X by the following transformation: Y ¼ AX:
ð1Þ
Thus, an m-dimensional projected vector Y is obtained, and it is called the projected feature vector of image A. The key problem is how to define a projection vector X as a good one. The theory of 2DPCA [16] has already solved this problem, it has been proved that the best projection vector X is the eigenvector of Gt corresponding to the largest eigenvalue, where Gt is defined as follows: h i 1 M ∑ ðA AÞT ðAj AÞ: Gt ¼ E ðA EAÞT ðA EAÞ ¼ Mj¼1 j
ð2Þ
Obviously, only one optimal projection vector is not enough, we often choose a set of projection vectors which subject to the orthonormal constraints. In fact, the optimal projection vectors X 1 ; …; X d are the orthonormal eigenvectors of Gt corresponding to the first d largest eigenvalues.
Now we are able to handle color faces with CPCA approach. But there is still one flaw leaving behind. In CPCA approach, although the color information is utilized, the spatial information is destroyed when we flatten the pixels of an image as a row. In order to preserve the spatial information, we proposed a 2DPCA based color face recognition method, which is called as C2DPCA (abbreviation of color 2DPCA) for convenience in the following. However, it seems hard to explain C2DPCA in matrix form, because the color image with spatial cues is three-dimensional which cannot even be presented in regular matrix form. Thus, the key factors of our proposed C2DPCA method are as follows:
The three components of a color pixel are regarded as a whole
item, and the color pixel calculated or processed as a basic unit with the proposed concept of color value. We define the basic operations of color value, thus the color values can be regarded as normal scalar values for computing.
With the proposed concept of color value and its operations, we establish the color value representation model for color images. Furthermore, color face recognition can be implemented by applying the framework of 2DPCA based on the new basic computing elements (color values). 3.1. Color value representation model In the previous section, we represent the color image problem in two-dimensional matrix form and finally we solve it with the theory of 2DCPA [16]. However, the image matrix form in CPCA does not have the intuitive meaning like PCA in which every element of the image vector is definitely a value of one pixel. Our original idea is that, we want to regard the three components of a color pixel as a whole, and to calculate or process the color pixel as a basic unit. Let’s rewrite the image matrix in CPCA as below: A ¼ ½c1 ; c1 ; …; cn T
2.2. Classification method First, the optimal projection vectors X 1 ; …; X d are utilized for feature extraction. Let Y k ¼ AX k ;
3. Color face recognition based on 2DPCA
k ¼ 1; 2; …; d:
ð3Þ
ð5Þ
here we regard the color image as a column ‘vector’, and every element of this ‘vector’ is a ‘value’ of the pixel—a color vector indeed. We call this ‘vector’ as composite vector, and this ‘value’ as color value. Then we are going to apply the conventional PCA on
Classifier Color face images
Color Value Representation Model
Color value vector
2DPCA
projection vectors
Fig. 1. The flowchart of proposed C2DPCA scheme for color face recognition.
Recognition
X. Xiang et al. / Neurocomputing 152 (2015) 231–235
233
Fig. 2. Sample images from the GATech color face database.
these image composite vectors. That is, we need to find a projection vector X to project A to a color value Y ¼ AX. And the projection should maximize the total scatter of the projected color values Y. Assume that there are M samples, and A denotes the average of all the composite vectors. Then we define the sample ‘matrix’ as: h iT ð6Þ D ¼ ðA1 AÞT ; ðA2 AÞT ; …; ðAM AÞT where every row of D represents a centered sample. It should be noted that D is not a normal matrix, and it is a composite matrix since the basic element of D is not a scalar value but a color value. The theory of conventional PCA [21] tells us, the optimal projection vector X can be obtained by calculating the eigenvector of DT D. The problem is how we calculate DT D, since the basic element of D is now a color value. Our key idea is to define the basic operations of color value such as addition and substraction. Once the basic operations are defined for color values, the color values can be regarded as normal scalar values for computing. Since there are only three kinds of basic operations in the process of calculating DT D, thus, we only define the following three operations for color value:
Addition: the sum of two color values is the sum of these two color vectors.
Substraction: the difference of two color values is the difference of these two color vectors.
Multiplication: the product of two color values is the inner product of these two color vectors. It’s intuitively to understand the definition of addition and substraction, the only doubt is the definition of multiplication. In the process of calculating DT D in conventional PCA, the operation of multiplication is intrinsically used to calculate the covariance of two features, and multiplication reflects the correlation of two centered feature value. If the result of multiplication is zero, they are independent; otherwise they are positively or negatively
correlated. It’s wonderful that inner product has the same property. If the inner product of two centered feature vector is zero, they are independent (they are mutually perpendicular), otherwise they are positively or negatively correlated. With the above definitions, we can now calculate A and DT D easily, just regard the color value as a basic unit while computing. And what’s more, after the calculation, we find that: M
DT D ¼ ∑ ðAj AÞT ðAj AÞT ¼ M Gt j¼1
ð7Þ
That means, the eigenvectors of DT D are the same with Gt (see Section 2), and we will get the same projection vectors by this way. It again evinced the rationality of our color value model. 3.2. Color 2DPCA With our color value model, we can finally give our C2DPCA approach, which is shown as Fig. 1. As mentioned above, we want to preserve both the color information and the spatial information. However, the color face with both color and spatial cues could not even be presented as regular two-dimensional matrix. That’s why the color value model is introduced. With that model, all we need to do is just to present the color image as a composite matrix whose basic element is color value. By integrating the algorithm of 2DPCA with our color value model, we can get the optimal projection vectors. After that, a nearest neighbor classifier is also applied, and the details are the same as Yang’s paper [16], however the basic computing elements become color values. It should be noted that the computational complexity of our proposed algorithm is similar to 2DPCA, since the computing process is similar, while the basic computing element is different. 4. Experimental results To evaluate the performance of our color face recognition method (CPCA, C2DPCA), we conduct our experiments on the
234
X. Xiang et al. / Neurocomputing 152 (2015) 231–235
recognition methods are tested in our experiments, and they are PCA scheme, 2DPCA scheme, our proposed CPCA scheme and our proposed C2DPCA scheme. In our experiment, all these face images were resized to the resolution of 33 44, thus the number of pixels in each face image is n ¼ 33 44. The RGB color space is used, so the parameter m is 3. Besides, there are two other parameters. One is the number of images to be utilized for training. We use k to denote how many images per individual are used for training, so the total number of training sets is 50 k, and the remaining images are used for testing. The other parameter is the number of principal eigenvectors d. Meanwhile, gray-level face images are obtained from the corresponding color face images, and they are used to test the conventional PCA and 2DPCA. Several sets of experiments are carried. Fig. 3 shows the experimental results on k¼ 7, 10, and 13. From these results, we can see that our proposed C2DPCA can achieve the highest accuracy among the four face recognition schemes when we adjust the parameter d under the same k. That is not surprising, because C2DPCA approach utilizes both the color and spatial information. The next is 2DPCA, then CPCA and PCA. From the results, we can also see that C2DPCA performs better than 2DPCA, and CPCA performs better than PCA for the utilization of color cues. It should be noted the other phenomenon that C2DPCA and 2DPCA achieve their highest accuracy before CPCA and PCA on the dimension axis. That is because the largest number of principal eigenvectors is 33 44 for CPCA and PCA, while it is 33 for C2DPCA and 2DPCA. In other words, the eigenvector of C2DPCA and 2DPCA can have stronger expressive power because of the utilization of spatial cues. Experimental results show that our proposed color value model can greatly benefit the performance of conventional PCA and 2DPCA, and our final proposed C2DPCA approach can achieve the best performance among these schemes.
5. Conclusion A novel color value model for color face recognition is proposed in this paper. By integrating with conventional PCA and 2DPCA, respectively, the CPCA and C2DPCA approach are proposed. We explained our color value model’s rationality by the comparison with matrix-representation model. Experimental results show that C2DPCA can achieve the best performance among the four approaches by utilizing both the spatial and color cues. Our C2DPCA and CPCA approach also perform better than 2DPCA and PCA respectively for the using of color cue.
Acknowledgements This work was supported in part by the Natural Science Foundation of China under Grant 61301106 and 61327013, the Natural Science Foundation of Jiangsu Province under Grant BK2012397, Research Fund for the Doctoral Program of Higher Education of China under Grant no. 20123219120024, the Fundamental Research Funds for the Central Universities under Grant no. 30920130121003, the Jiangsu Key Laboratory of Image and Video Understanding for Social Safety (Nanjing University of Science and Technology) under Grant no. 30920130122006. Fig. 3. Comparison results under various k.
GATech color face database [20]. The GATech images are composed of color images of 50 individuals with 15 views per individual, and with no specific order in their viewing direction. Fig. 2 shows some sample images from GATech color face database. Four face
References [1] M. Kirby, L. Sirovich, Application of the karhunenloeve procedure for the characterization of human faces, IEEE Trans. Pattern Anal. Mach. Intell. 12 (1) (1990) 103–108. [2] M. Turk, A. Pentland, Eigenfaces for recognition, J. Cogn. Neurosci. 3 (1) (1991) 71–86.
X. Xiang et al. / Neurocomputing 152 (2015) 231–235
[3] M.A. Grudin, On internal representations in face recognition systems, Pattern Recognit. 33 (7) (2000) 1161–1177. [4] A. Pentland, Looking at people: sensing for ubiquitous and wearable computing, IEEE Trans. Pattern Anal. Mach. Intell. 22 (1) (2000) 107–119. [5] D. Valentin, H. Abdi, A.J. O’Toole, G.W. Cottrell, Connectionist models of face processing: a survey, Pattern Recognit. 27 (9) (1994) 1209–1230. [6] J.u.n. Yu, Meng Wang, Dacheng Tao, Semi-supervised multiview distance metric learning for cartoon synthesis, IEEE Trans. Image Processing 21 (11) (2012) 4636–4648. [7] J.u.n. Yu, Yong Rui, Dacheng Tao, Click prediction for web image reranking using multimodal sparse coding, IEEE Trans. Image Processing 23 (5) (2014) 2019–2032. [8] J.u.n. Yu, Dongquan Liu, Dacheng Tao, Hock Soon Seah, On combining multiview features for cartoon character retrieval and clip synthesis, IEEE Trans. Syst. Man Cybern. Part B Cybern. 42 (5) (2012) 1413–1427. [9] Y.u.e. Gao, Meng Wang, Zheng-Jun Zha, Jialie Shen, Xuelong Li, Xindong Wu, Visual–textual joint relevance learning for tag-based social image search, IEEE Trans. Image Processing 22 (1) (2013) 363–376. [10] Meng Wang, Xian-Sheng Hua, Richang Hong, Jinhui Tang, Guo-Jun Qi, Y.a.n. Song, Unified video annotation via multigraph learning, IEEE Trans. Circuits Syst. Video Technol. 19 (5) (2009) 733–746. [11] Meng Wang, Xian-Sheng Hua, Jinhui Tang, Richang Hong, Beyond distance measurement: constructing neighborhood similarity for video annotation, IEEE Trans. Multimedia 11 (3) (2009) 465–476. [12] Y.u.e. Gao, Meng Wang, Dacheng Tao, Rongrong Ji, Qionghai Dai, 3D object retrieval and recognition with hypergraph analysis, IEEE Trans. Image Processing 21 (9) (2012) 4290–4303. [13] Jinhui Tang, Zheng-Jun Zha, Dacheng Tao, Tat-Seng Chua, Semantic-gap oriented active learning for multi-label image annotation, IEEE Trans. Image Processing 21 (4) (2012) 2354–2360. [14] Jinhui Tang, Richang Hong, Shuicheng Yan, Tat-Seng Chua, Guo-Jun Qi, Ramesh Jain, Image annotation by kNN-sparse graph-based label propagation over noisily-tagged web images, ACM Trans. Intell. Syst. Technol. 2 (2) (2011) 111–126. [15] Jinhui Tang, Qiang Chen, Meng Wang, Shuicheng Yan, Tat-Seng Chua, Ramesh Jain, Towards optimizing human labeling for interactive image tagging, ACM Trans. Multimedia Comput. Commun. Appl. 9 (4) (2013) 29. [16] J. Yang, D. Zhang, A.F. Frangi, J.Y. Yang, Two dimensional PCA: a new approach to appearance-based face representation and recognition, IEEE Trans. Pattern Anal. Mach. Intell. 26 (1) (2004) 131–137. [17] A. Yip, P. Sinha, Role of color in face recognition, J. Vis. 2 (7) (2002) (article 596). [18] L. Torres, J.Y. Reutter, L. Lorente, The importance of the color information in face recognition, in: IEEE International Conference on Image Processing, vol. 3,1999, pp. 627–631. [19] J. Yang, C. Liu, A general discriminant model for color face recognition, in: IEEE 11th International Conference on Computer Vision, 2007, pp. 1–6. [20] 〈http://www.anefian.com/research/face reco.htm〉. [21] S. Watanabe, Karhunen-loeve expansion and factor analysis, theoretical remarks and applications, in: Proceedings of Fourth Prague Conference Information Theory, 1965.
235 Xinguang Xiang received the B.A., M.S. and Ph.D. degrees in computer science from the Department of Computer Science and Technology, Harbin Institute of Technology, Harbin, China, in 2005, 2007, and 2011, respectively. He is currently with the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. His current research interests include face recognition, image analysis and understanding, video processing, and video coding.
Jing Yang he has been pursuing the M.S. degree from the School of Arts, Chongqing University, Chongqing, China, since 2012. His current research interests include image analysis and understanding.
Qiuping Chen received the B.A. degree from the School of Computer Science and Technology, Xuzhou Institute of Technology, China, in 2012. Since 2012, he has been pursuing the M.S. degree from the School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, China. His current research interests include face recognition, image analysis and understanding.