- Email: [email protected]
Group-specific face verification using soft biometrics
ARTICLE IN PRESS Journal of Visual Languages and Computing 20 (2009) 101–109
Contents lists available at ScienceDirect
Journal of Visual Languages and Computing journal homepage: www.elsevier.com/locate/jvlc
Group-specific face verification using soft biometrics Gian Luca Marcialis , Fabio Roli, Daniele Muntoni Department of Electrical and Electronic Engineering, University of Cagliari, Piazza d’Armi, 09123 Cagliari, Italy
a r t i c l e in fo
Keywords: Biometrics Soft biometrics Face verification
abstract Soft biometrics have been recently proposed for improving the verification performance of biometric recognition systems. Examples of soft biometrics are skin, eyes, hair colour, height, and ethnicity. Some of them are often cheaper than ‘‘hard’’, standard biometrics (e.g., face and fingerprints) to extract. They exhibit a low discriminant power for recognizing persons, but can add some evidences about the personal identity, and can be useful for a particular set of users. In particular, it is possible to argue that users with a certain high discriminant soft biometric can be better recognized. Identifying such users could be useful in exploiting soft biometrics at the best, as deriving an appropriate methodology for embedding soft-biometric information into the score computed by the main biometric. In this paper, we propose a group-specific algorithm to exploit soft-biometric information in a biometric verification system. Our proposal is exemplified using hair colour and ethnicity as soft biometrics and face as biometric. Hair colour and information about ethnicity can be easily extracted from face images, and used only for a small number of users with highly discriminant hair colour or ethnicity. We show by experiments that for those users, hair colour or ethnicity strongly contributes to reduce the false rejection rate without a significant impact on the false acceptance rate, whilst the performance does not change for other users. & 2009 Elsevier Ltd. All rights reserved.
1. Introduction Biometrics received increasing attention in the last decade [1–3]. They are physiological or behavioural characteristics of human, as the fingerprint, the face, the voice, the iris, and the gait. Since biometrics are unique from person to person, cannot be forgotten and are very difficult to steal or reproduce, they have been proposed for personal verification instead of (or coupled with) passwords and PINs. Although performances of biometric verification systems are promising, they have to handle the large variations of the acquisition environment and the physiological and emotional states of the subject to be recognized. In particular, the time required for verifying
Corresponding author.
E-mail addresses: [email protected] (G.L. Marcialis), [email protected] (F. Roli), [email protected] (D. Muntoni). 1045-926X/$ - see front matter & 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.jvlc.2009.01.005
the person identity strictly depends on the system performance. As an example, if the percentage of genuine users’ rejection false reject rate (FRR) is too high, some users could be often wrongly rejected and they could be constrained to further access trials, that is, to further submissions of their biometric. Hence, the time required for accessing to the system could increase, with an impact on the verification time and, in the worst case, on the acceptability of the system. Some recent works suggested that using soft biometrics can allow increasing the verification performance [4–7]. Soft biometrics are physiological or behavioural human characteristics with a low discriminant power. Examples of soft biometrics are gender, eye colour, hair colour, skin colour, ethnicity, height, and weight. They are not unique from person to person; therefore they cannot be individually used for personal verification. However, the extraction of many of them is cheaper than that of standard biometrics (e.g., face and fingerprints), and can
ARTICLE IN PRESS 102
G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
give some evidences about the subject’s identity. In some cases, biometric and soft-biometrics extraction can have almost the same cost (for example, ethnicity and hair colour). On the basis of this concept, a simple framework has been proposed to integrate the ancillary information with the main biometric system in Refs. [4–7], and applied in identification mode: briefly, the posterior probability of each identity given the biometric score is combined with the correspondent posterior probability given the softbiometric ‘‘scores’’ by weighted averaging. As pointed out in [4–7], this approach exhibits some drawbacks: estimation of weights (no method is proposed, and soft biometrics must be weighted much less than the biometric score due to their low discriminant power) and estimation of posterior probabilities of soft biometrics, which are not permanent and reliable, since they need to be updated continuously. Therefore, we propose an alternative framework, which avoids the problem of evaluating the posterior probabilities related to soft biometrics, and exploits some hypotheses that can be drawn around them. Due to their low discriminant power, it can be argued that a certain soft biometric can be useful only for a limited set of users (this is mentioned in [7], but not exploited in the authors’ framework). For example, Latin people rarely exhibit blond hair. Worth noting, hair colour has been recently proposed as biometric in [8] for person identification. Experiments reported on the AR data set showed very low recognition accuracy, and a slight improvement by integrating hair colour and face as biometrics has been shown in [8]. Despite these results, hair colour could have a good impact on improving the verification performance, if considered as ‘‘soft’’ biometric, especially for users exhibiting highly discriminant hair colour, but it cannot bring information to others. Accordingly, hair colour (in general any soft biometrics) should be considered as able to add further evidences only if the claimed identity is in the group of users mentioned above. Moreover, the relevance of the hair colour is high if this group is much smaller than the overall user population. In other words, hair colour does not exhibit intrinsic discriminant properties, but is informative if it is possible to point out, from the particular user population, some peculiarities in the hair colour belonging to a small group of users only. In the following, we will refer to those users as ‘‘minority group’’. This point has not yet been investigated in the literature, and is addressed in the framework proposed in this paper. For its validation, we adopt a face verification system using hair colour and ethnicity as soft biometrics. We selected the face biometric for two reasons. Firstly, it has been investigated and used widely [9]. Secondly, hair colour does not require additional hardware because it can be extracted directly from the true-colour face image. Therefore, it is cheap to automatically extract from face images, although it has limited discriminant power. The face verification system is based on wellknown techniques [10–12]. Experiments are carried out on the AR data set for hair colour and on the Notre Dame data set for ethnicity. The requirements of this data set fit well with our hypothesis. In fact, it is made up of Latin
people prevalently [13]. On the other hand, Notre Dame can be easily partitioned according to the ethnicity. Our experiments confirm the effectiveness of our framework and show the benefits of using hair colour by reducing the false rejection rate without a significant impact on the false acceptance rate, whilst the performance does not vary for other users. This paper is organized as follows. Section 2 describes our proposal. Section 3 reports its validation by describing the system adopted in our experiments and related results on the AR data set. Section 4 concludes the paper.
2. The proposed framework In this section, we derive a simple framework that shows the combination of the information extracted from a generic soft biometric with the one extracted from the main biometric according to a proposed group-specific approach. In this paper, we show the framework only in the case of one soft biometric, but it can be easily generalized to more soft biometrics. The main characteristic of the proposed framework is that it is not based on the ‘‘fusion’’ of the main biometric score with a correspondent ‘‘soft-biometric’’ score as [4–7]. In other words, we do not express the information captured by soft biometrics by a score, but as an additional evidence aimed to increase the probability of the class, given the main biometric. Therefore, we derive the expression of the final score integrating biometric and soft-biometric information, as an expression of the probability of a certain genuine user given the biometric features and the related soft-biometric feature, as a function of the probability of the user with the only evidence of its biometric, which is viewed as the main biometric score. The feature related to the soft biometric is interpreted as further evidence, which increases the above probability with respect to the initial biometric score, under the hypothesis that the genuine user can be inserted into a small group with highly discriminant soft-biometric feature, exemplified by hair colour in this paper. Let o be a generic identity in the user population U ¼ {o1, y, oM}, xface be the information extracted from the given biometric (e.g., the face feature vector), and xhair, be the feature vector related to the hair colour (in general, to soft biometric). Relationships among face and hair colour can be modelled as a Bayesian network as shown in Fig. 1. This figure points out that the client posterior probability can be viewed as a function of the probability of xhair membership to a sort of ‘‘minority group’’ C (defined in the following) and by the current value of feature vector xface. Given the evidence on the client, information coming from face and hair colour is independent, as can be reasonably supposed [4–8]. If the face matching score is interpreted as P(o|xface), the final posterior probability, under the hypothesis behind the minority group definition, can be derived from the relationships among other events. First of all, we define formally the concept of ‘‘minority group’’. If we suppose that the soft-biometric feature
ARTICLE IN PRESS G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
103
regard to this issue, we can consider the relationship
xhair
Pðojxface Þ ¼ Pðojxface ; xhair 2 CÞPðxhair 2 CÞ þ Pðojxface ; xhair eCÞPðxhair eCÞ
xhair ∈C
(4)
Where we hypothesized the independence of hair colour with respect to the face information: P(xhairAC|xface) ¼ P(xhairAC). Eq. (4) can be written as follows
xface
Pðojxface Þ ¼ Pðojxface ; xhair 2 CÞp þ Pðojxface ; xhair eCÞð1 pÞ
ω
(5)
Now we can analyse the following ratio
Fig. 1. The Bayesian network modelling the proposed framework. In this network, two events are emphasized by two nodes: the simple evidence of having an xhair value, and the fact that xhair is localized into the ‘‘minority group’’ C.
Pðojxface ; xhair 2 CÞ Pðxhair 2 Cjxface ; oÞ Pðxhair eCjxface Þ ¼ Pðojxface ; xhair eCÞ Pðxhair eCjxface ; oÞ Pðxhair 2 Cjxface Þ (6) On the basis of Bayesian network in Fig. 1, we obtain
pðxhair Þ ¼
N X
pðxhair jxhair 2 C i ÞPðxhair 2 C i Þ
Pðxhair 2 C; xface joÞ pðxface joÞ Pðxhair 2 CjoÞpðxface joÞ ¼ pðxface joÞ
Pðxhair 2 Cjxface ; oÞ ¼
space can be partitioned into N clusters, with N5|U|: (1)
i¼1
The cluster Ci is defined as ‘‘minority cluster’’ if and only if Pðxhair 2 C i Þ ¼ pi 51 ( 9O U
8o 2 O;
Pðxhair 2 C i joÞ40:5
8oeO;
Pðxhair 2 C i joÞ ¼ pi
(2.1) (2.2)
Pðojxface ; xhair 2 CÞ po 1 p ¼ L41 ¼ Pðojxface ; xhair eCÞ 1 po p
i
Therefore, we can define the minority group, and indicate this with the couple {C,O}, where C is the set of clusters (into the given feature space) and O is the set of clients such that
8oeO;
Pðxhair 2 CjoÞ ¼ po 40:5 Pðxhair 2 CjoÞ ¼ p
(7)
Since an evidence on o makes independent the events xhairAC and xface. Accordingly, if oAO, Eq. (6) becomes (condition (2.2))
Finally, the union of minority clusters C is such that X Pðxhair 2 CÞ ¼ Pðx 2 C i Þ ¼ po0:5 (3)
8o 2 O;
¼ Pðxhair 2 CjoÞ
(3.1) (3.2)
No other constraints or hypothesis are made on such clusters. In Eq. (3.1), po indicates, for sake of compactness, the value of P(xhairAC|o). This definition reflects our hypothesis about the possibility of exploiting soft biometrics into a user population: there must be a set of users whose hair colour is highly discriminant (condition (2.2)), since it is rare in the same user population (condition (2.1)). Worth noting, the definition of minority cluster also implies that a small amount of clients can be characterized by the above-mentioned properties; thus the term ‘‘minority’’ can refer both to the rarity of hair colour and to the small amount of users exhibiting such hair colour. Condition (2.2) also imposes that if there is no evidence about the membership of a certain client in the minority group, related hair colour can be considered as common. The next step is to derive a relationship between the evidence of having an xhair belonging to such a minority group joint with the evidence of having xface, in order to evaluate the probability of being a genuine user. With
(8)
Otherwise (condition (2.2)), Pðojxface ; xhair 2 CÞ p 1p ¼1 ¼ 1p p Pðojxface ; xhair eCÞ
(9)
Therefore, if there is evidence that xhairAC, that is, xhair is a member of the minority cluster, we can write Pðojxface ; xhair 2 CÞ ¼ L41 Pðojxface ; xhair eCÞ
(10)
Eq. (5) becomes Pðojxface ; xhair 2 CÞ ¼
L Pðojxface Þ 1 p þ Lp
(11)
Score values related to face can be interpreted as a posterior probability [14]. Therefore, if we assume that s ¼ Pðojxface ; xhair 2 CÞ sface ¼ Pðojxface Þ
(12)
sface is given by the comparison of the claimed identity template and the input feature vector xface, and s is the score integrating face and hair colour information: s¼
L s ¼ asface 1 p þ Lp face
(13)
Worth noting, a41 as can be easily proved under the findings pointed out in Eq. (9). But, if oeO (Eq. (9)), L ¼ 1, thus s ¼ sface
(14)
Therefore, hair colour is useful only if there is evidence about its membership to the minority group. The practical consequence of this framework is that, if a good soft
ARTICLE IN PRESS 104
G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
biometric can be found as characterizing the features space according to the above framework, the obtained final score increases the separation between clients with highly discriminant hair colour and impostors. The main advantage of this framework with respect to the one proposed in [4–8] is that estimation of posterior probabilities, or the computation of soft-biometric score, is not needed, since s can be derived from sface. Moreover, whilst a different set of weights should be used for each client [4–8], here the number of parameters per client is limited by the fact that the method is group-specific; thus a unique parameter, namely L, must be used. Anyway, it is worth remarking that a clear comparison of the proposed framework with existing ones is out of the scope of the present paper. A separate study will be devoted to this aim. On the other hand, the framework requires the estimation of N, po and p. The N-value depends on whether some a priori information on the user population is available (e.g., Latin people prevalently, where blond hair or red hair are rarely expected, thus N ¼ 3). This helps in defining the N-value. In general, N will depend on the feature space related to the soft biometric adopted. With regard to po and p, they can be estimated once the minority clusters are defined, by cross-validation on a training set. In particular p is the frequency of hair colour samples in the smallest cluster and po is the frequency of hair colour samples of clients in the minority clusters according to p and Eqs. (3.1) and (3.2). We followed this cross-validation approach in our experiments. In this paper, we only performed experiments to validate the model under the assumptions that N ¼ 3 (two minority clusters). This has been motivated by the kind of user population in the adopted data set for experiments.
Fig. 2. Position of the hair colour samples, pointed out by squares, in a sample image. The eye positions are also pointed out.
reasonable value for capturing the hair colour appears to be 155 pixels above the eye locations. Two square-sized 10 10 pixels were extracted from those locations. This size takes into account the highness of the hair and the forehead. Fig. 2 shows the position of the square related to the hair colour. For each image, two samples of the hair are extracted, corresponding to the left and the right side of the face image. The CbCr values of each square are finally averaged. Therefore the hair colour feature vector is made up of four components, related to left and right sides of the face image. This is motivated by the necessity of considering that each face side can be differently illuminated. 3.2. Minority group identification and exploitation
3. Framework validation 1: experiments with face and hair colour In this section, and in Section 4, we show two sets of experiments aimed to validate the proposed framework. In the first set of experiments, adopted soft biometric is the hair colour, whilst in the second one we used the ethnicity. In both cases, the main biometric is the face. 3.1. The data set We carried out experiments on the AR data set [13]. This data set contains frontal view faces with different facial expressions, illumination conditions, and occlusions (sun glasses and scarf). Each person participated in two acquisition sessions, separated by 2 week’s time. Each session is made up of seven images per person. We selected 100 subjects (50 males and 50 females), manually cropped their face images and, after histogram stretching and equalization, resized them at 80 80 pixels. For computing the PCA space, we considered only the grey-level normalized values of the pixels, whilst we considered the CbCr components for the hair colour. Since the inter-ocular distance has been fixed to 115 pixels, a
Minority group can be identified manually or automatically. In this paper, we adopted the second approach, by partitioning feature vectors of hair colour with a 3means clustering algorithm [15]. In this paper N ¼ 3, since user population is prevalently Latin and it is expected that at most three groups (hair colour different from black hair) should be identified. The clusters in which samples follow the definition in Eqs. (1) and (2) are called ‘‘minority clusters’’: firstly, they must be the smallest; secondly, the frequency of hair colour samples falling in them must be very high for certain clients. The centroids of these clusters define, on the overall, the set C, whilst the correspondent set of clients defines O, thus allowing to obtain the minority group {O,C}. Two experiments were performed for N ¼ 31: (1) the first AR session is used for extracting a set of training samples from each client. A clustering algorithm, namely 3-means, is applied to this set. We identified the N1 1 Worth noting, we tried experiments with N ¼ 2 and N43. In the first case, the minority cluster was characterized by a number of samples so small that predicted value of pM and p was not reliable (pw especially). On the other hand, when N43, no clusters were found and 4-means stopped with error.
ARTICLE IN PRESS G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
105
Face Verification System Normalized face image
Comparison with template client
sface
s
α Eye position extraction
Face image
Claimed identity
Is claimed identity in the minority group ? no
yes Hair colour extraction
α
Eq. (8) yes
Is hair colour in the minority group ? no α=1
Hair Colour Handler Fig. 3. Basis processing flow of the proposed face verification system using hair colour.
cluster(s) such that the number of hair samples falling in those clusters was much less than that of samples falling in the remaining cluster, thus estimating p (Eq. (2.1)). Then, we identified in such clusters the clients for which is verified Eq. (3.1). We also used this session to compute the PCA space and the template image for each user (the mean of images has been considered), and estimating the operational points of the ROC curve. The second AR session has been used for testing the system, i.e., in the verification phase. (2) The same as (1), with the first and the second sessions inverted. Reported results are averaged on these two runs. With regard to the verification phase, the system works as shown in Fig. 3. When an user submits his face and claims a certain identity, it is first verified that this identity falls in O, hair colour is extracted and if the centroid of C is nearest to the hair colour feature vector, evidence on the membership to minority group is given in terms of a (Eqs. (8) and (9)). 3.3. Results This experimental section is organized as follows. In Section 3.3.1, we show the minority group of users identified using the 3-means clustering algorithm. In Section 3.3.2, we show that the application of Eqs. (8) and (9) allows to improve the performance for such minority group, whilst it does not exhibit significant variations for other users. 3.3.1. Automatic identification of minority group It is worth noting that the AR data set is made up of Latin people prevalently. As a consequence, it can be
expected that, using hair colour, the subjects with highly discriminant hair colour are characterized by sharp hair. Fig. 4 shows the users the minority group found by the 3-means algorithm (first run). Some errors occurred during the automatic extraction of hair colour: samples of the forehead have been taken instead of hair samples. For example, this happened to the first user in Fig. 4. By sampling correctly, those subjects are not classified in the minority group. Worth noting, the other clients clearly exhibit black hair colour. As shown in Fig. 4, the ‘‘minority group’’ corresponds to the set of users with hair colour significantly different from the majority of users. The size of the minority group is about 15% of the whole number of subjects. It is worth remarking that in this paper we are not proposing a novel approach to hair colour extraction and evaluation, but we are only interested in the impact of using effectively hair colour under the hypothesis of minority clusters into the user population. Population identified strongly fits the constraints in the definition of ‘‘minority clusters’’: a priori hair colour samples rarely exhibit their value, and their samples exhibit the probability of being more than 0.5 in the cluster. 3.3.2. Effect of soft biometrics on verification performance This section shows that the hair colour applied to the minority group allows reducing the related false rejection rate without a significant impact on the false acceptance rate, whilst the performance does not vary for other users. According to Section 2, estimated a was around 2.0 on average: 2.5 in the first run and 1.5 in the second run. The high standard deviation suggests that estimation is dependent heavily on the available data, and some
ARTICLE IN PRESS 106
G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
Fig. 4. Some subjects belonging to the minority users cluster according to the hair colour obtained by using the 3-means algorithm on 30 runs.
Minority against impostors ...using Hair Colour ...using Hair Colour (Ideal Performance)
10-1
10-2
10-3
All users-Face All users-Face + hair colour Minority group-Face Minority group-Face + hair colour 10-2
FRR
FRR
10-1
10-1 FAR
Fig. 5. Average ROC curve showing the improvement reported for all users (straight and dotted line), and for all users (minority users included) when claiming to be into the minority group (dashed and straight line with dots).
outliers may affect it. Therefore, robust estimation methods (better than ours) should be found. Fig. 5 shows the average ROC curve obtained by estimating the values of po and p using the 3-means clustering algorithm on the hair colour samples captured as described in Sections 3.1 and 3.2. These curves clearly show that, whilst ROC curves of the overall user population are substantially unaltered, the minority group’s ROCs exhibit a clear performance improvement. This is due to Eqs. (8), (9) and (11), which refine the estimation of the score. The reported result is remarkable when considering that AT exhibit a small sample size, and a very few of samples per class has been used for a estimation. Another point is the estimation error of membership in the test set, which strongly impacts performance. In fact, in Fig. 5 we did not separate results of the classification, but only checked the results on the users claiming an identity into the minority group. Therefore, we investigated ROC curves related to users into the minority group, claiming themselves, against the other users. Fig. 6 shows two ROC curves that focus on such an information. In the first one, we show the ROC curve of genuine users into the minority group against the other users claiming to be into the minority group without
10-2 10-3
10-2
10-1
100
FAR Fig. 6. Average ROC curve showing a sharp performance improvement in users of minority group verification against impostors not belonging to the minority group (some of them could be wrongly assigned in the verification phase).
(dashed line) and with the help of hair colour information (straight line with dots). It can be noticed that equal error rate (EER) fell from 6% to 4.5%, and 1%FAR from 10% to 7% about. The straight line without dots in Fig. 6 shows that the potentialities of our framework are more than that shown in our experiments. ROC curves can be further improved by removing the misclassification performance relating to impostors, whose hair colour is classified as belonging to the minority clusters. In Fig. 6 the performance of impostors claiming to be users into the minority group is shown. Misclassification errors are due to the simple classification approach adopted here, and, as it can be seen, they impact on the final results. Ideally, the verification performance related to minority group could lead to 3.5% and 4% by considering EER and 1% FAR points. Therefore, the benefits we pointed out allow us to observe that: (1) the subject identity is verified more frequently for the minority group; therefore, the average verification time decreases with a favourable impact on its acceptability. This means that, if we were able to find a soft biometric peculiar to different subsets of the user population (i.e., ‘‘different’’ minority groups, one, or more, for each soft biometric), the whole acceptability and the
ARTICLE IN PRESS G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
100
100
90
90
70
Face Face+Ethnicity
80
Face Face+Ethnicity
FRR on test set
FRR on test set
80
60 50 40 30
70 60 50 40 30
20
20
10
10
0
107
0 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 x% FAR on training set
1
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 x% FAR on training set
Fig. 7. (a) FRR on the overall user population when operation point computed is between zero%FAR and 1%FAR. (b) Zoom of (a) curve showing the strong reduction of FRR especially at zeroFAR point.
performance of the system could be strongly augmented; (2) the increase in the verification score values is not significant for the majority group, that is, those users are correctly classified. This is because the soft-biometric information is applied corresponding to very rare hair colour values, according to our definition of minority cluster. 4. Framework validation 2: experiments with face and ethnicity 4.1. The data set We adopted the Notre Dame data set2 [17], which has also been used for the data collection at face recognition Grand Challenge3. This data set contains several frontal face images, both indoor and outdoor. Each subject was photographed with a high-resolution digital camera (1600 1200 or 2272 1704) under different lighting and expression conditions. Many subjects were photographed every week for 10 weeks in the Spring of 2002, 13 weeks in the fall of 2002, and 15 weeks in the spring of 2003. For the purpose of this paper, we removed all outdoor images, obtaining 50 images per client, and selected a subset of 75 users in order to have a clear subdivision in terms of ethnicity. We obtained seven African people, sixteen Asians, and the rest are Caucasians. Three kinds of users are hence found, for which Asians and Africans represent clearly the ‘‘minority group’’. The data set was subdivided into training and test set, both made up of 25 images per client. The training set was used to estimate PCA–LDA for face verification and also compute the parameters of our model. In this case the minority group was manually identified. The system integrating ethnicity and face biometric are the same as 2 3
http://www.nd.edu/cvrl/UNDBiometricsDatabase.html http://www.frvt.org/FRGC/
described in Fig. 3. The main changes involve the computation of a parameter, which are described in Section 4.2. With regard to biometric face images, we followed the same protocol adopted in Section 3.
4.2. Results We adopted the whole cropped face by concatenating CbCr components, and performed LDA on the related feature vectors. We classified training set data by adopting 1 nn with the leave-one-out method, in order to compute the confusion matrix. pw was computed from this matrix, for each client. Thus, we evaluated a. In this case, computation of a is shown to be quite stable by inverting training and test set: the average value of 2.15 has been found with a standard deviation of nearly zero. In this section, we first of all point out the advantage of combining face biometric and ethnicity when high security levels must be reached. In particular, we investigated a range of thresholds, computed on the training set, and related to the range 0%FAR–1%FAR. It is well-known that these performance parameters are quite crucial [1,2], and reducing them is strongly helpful for clients. Fig. 7(a) and (b) shows the strong reduction of false rejection rate evaluated on the overall user population, on this range of operational points. Figure (b), in particular, shows the region between 0%FAR and 0.2%FAR. It is worth noting that, for the first value, the corresponding false rejection rate fell from about 90% to 70%. Notice also that 0%FAR is on the test but the threshold was estimated on the training set. Therefore, estimation error of threshold affected the real performance of the system. With regard to this problem, using soft biometric allowed to ‘‘correct’’ the operational point at the same time by maintaining the FAR at the same value (about zero). Fig. 8 shows the ROC curves related to the minority group (equivalent to Fig. 6). In this case, we have a very noticeable improvement of performance, as it explains
ARTICLE IN PRESS 108
G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
5. Discussion
10 Face Face + Ethnicity
9 8
% FRR
7 6 5 4 3 2 1 0 0
1
2
3
4
5
6
7
8
9
10
% FAR Fig. 8. ROC curve of Minority group (Africans and Asians) against Majority group.
%HTER on test set (Minority group)
50 45 40
Face Face+Ethnicity
35 30 25 20 15 10 5 0 0
0.05
0.1
0.15
0.2
0.25
x% FAR on training set Fig. 9. %HTER on the test set for minority group. Plot reports values corresponding to high security level operational points evaluated on the training set.
ethnicity well (better than hair colour). The equal error rate fell from 3% to 1.5%. The improvement is good even for other operational points. Worth noting, even in this case we adopted a quite simple approach for ethnicity classification. Being state-of-the-art very active in this topic [1,5,6], it is reasonable to hypothesize that ethnicity can be a promising soft biometric, which can be adopted according to the proposed framework. Finally, HTER (Half Total Error Rate [1]) for high security level operational points is reported in Fig. 9. Plot is related to minority group. Here we have a confirmation of Fig. 9: in particular, HTER at 0%FAR drops to about 12% when combining ethnicity with faces, whilst the HTER estimated is quite high (close to 50%) using face biometric only.
Reported results showed some clear advantages of using soft biometrics, and also showed some limitations due to their intrinsic nature of ‘‘soft’’ features. In this section, we consider these pros and cons, thus trying to fairly evaluate the actual impact of adopting soft biometric in real verification systems. First of all, according to the definition of soft biometrics, they do not exhibit properties of uniqueness, universality, and permanence. Thus they may change over time more quickly than standard biometrics. This implies that their usefulness strongly depends on the specific context. Moreover, some soft features, hair and eye colour for instance, are very easy to counterfeit, most of all if a malicious user is interested in gaining a false acceptance during identity verification. So far, none of the state-ofthe-art works [4–7] discussed this issue satisfactorily. Recently, another paper involving soft biometrics have been proposed [16] without solving this problem. On the other hand, we must also keep in mind that the most widely used biometrics can be faked (voice, face, fingerprint, iris); thus their use is still focused on assisting, and not substituting, human operators. This holds for soft biometrics too. With regard to our system, it is true that verification performance is not improved on the overall, but this can be considered also an advantage, because, as we pointed out in the paper, the false accept rate (FAR) does not consistently change or increase for clients out of the minority group. Even a trial of faking hair colour, for example, can be viewed in the experimental section as a misclassification of the system. As we showed in Sections 3 and 4, this does not lead to a significant increase of false acceptance rate, due to the relative value of a, which needs to be reliably computed. Another case is that the user population is too small for appreciating significant improvements in using soft biometrics. However, if we take into account the point of view of the user, it is significantly better for him to reduce the false reject rate, under the condition that the false acceptance rate does not increase strongly. As mentioned in the introduction, acceptability of the system may increase. Finally, use of soft biometrics can be made more robust by adopting multiple soft biometrics, in order to cross different minority groups and decrease a. This also involves a modification of the present framework in order to take into account the integration of more soft biometrics. This is a matter of on-going research, and we are not able to give a specific, alternative, and theoretical formulation. However, we can suggest two possible solutions, which must be modelled in order to evaluate the one most suitable for the task. These solutions are presented in Fig. 10(a) and (b). In Fig. 10(a), we have a different contribution, namely, a different a, for each biometric, which is made independent of others from the knowledge of the client to be recognized. In Fig. 10(b), the different contributions of each soft biometric are modelled by a unique set of minority clusters, thus leading to an a-value condensing information from all soft
ARTICLE IN PRESS G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
xhair
xhair ∈C
xhair
xethnicity
xethnicity ∈C
xethnicity
xsoft ∈C
xface
ω
109
xface
ω
Fig. 10. Possible extensions of the proposed model to more than one soft biometric: (a) each soft biometric contributes independent of the other, leading to different a-values and (b) the different contributions of each soft biometric are modelled by a unique set of minority clusters, thus leading to an a-value condensing information from all soft biometrics.
biometrics. As we mentioned, currently we are not in the position to prefer one solution to the other, but this will be matter of a future work.
population and adopted soft biometrics. Extension of the proposed framework to more soft traits is still a matter of on-going research.
6. Conclusions
References
In this paper, we claimed that, due to its limited discriminant power, soft biometrics are useful only for a limited set of users. Therefore, we derived a framework to combine information from the face and two soft biometrics, namely hair colour and ethnicity, represented by feature vectors. The novelty of this framework with respect to the state-of-the-art is that it overcomes the usually adopted concept of soft biometrics, because the soft-biometric information is used in a totally different manner, in particular, group-specific. Moreover, the number of parameters it requires is less than that of the existing ones. In particular, the estimation of posterior probabilities for each soft biometric is avoided. Experiments were carried out on the AR and Notre Dame data sets. In particular, AR is made up of Latin people prevalently. Latin people rarely exhibit sharp hair. On the other hand, Notre Dame allows one to manually, and easily, partition user population according to the ethnicity. Therefore, verification performance could be improved for those users, thanks to the information provided by soft biometrics above. The reported results showed the validity of the proposed framework, since it is experimentally confirmed that: (1) the minority group corresponded to users with highly discriminant hair colour; (2) this soft biometric applied to the minority group allowed reducing the related false rejection rate without a significant impact on the false acceptance rate, whilst the performance did not vary for other users significantly. Definitive conclusions cannot be drawn on the basis of this limited set of experiments. In general, we believe that different soft biometrics can identify different groups of users population; thus the proposed framework should be extended accordingly. It is worth remarking that this framework is an alternative to existing ones. Therefore, a future work will compare it with them by varying the user
[1] A.K. Jain, R. Bolle, S. Pankanti (Eds.), Biometrics: Personal Identification in Networked Society, Kluwer Academic Publishers, Dordrecht, 1999. [2] J. Wayman, A.K. Jain, D. Maltoni, D. Maio, Biometric Systems: Technology, Design and Performance Evaluation, Verlag, 2005. [3] A. Ross, K. Nandakumar, A.K. Jain, Handbook of Multibiometrics, Verlag, 2006. [4] U. Park, A.K. Jain, I. Kitahara, K. Kogure,N. Hagita, ViSE: visual search engine using multiple networked cameras, in: Proceedings of the International Conference on Pattern Recognition (ICPR), Hong Kong, August 2006, pp. 20–24. [5] A.K. Jain, S.C. Dass, K. Nandakumar, Soft biometric traits for personal recognition systems, in: Proceedings of the International Conference on Biometric Authentication (ICBA), Lecture Notes in Computer Sciences, vol. 3072, Hong Kong, July 2004, pp. 731–738. [6] A. K. Jain, K. Nandakumar, X. Lu, U. Park, Integrating faces, fingerprints, and soft biometric traits for user recognition, in: Proceedings of the Biometric Authentication Workshop, Lecture Notes in Computer Sciences, vol. 3087, Prague, May 2004, pp. 259–269 [7] A.K. Jain, S.C. Dass, K. Nandakumar, Can soft biometric traits assist user recognition?, in: Proceedings of SPIE, 5404, 2004, pp. 561–572. [8] Y. Yacob, L.S. Davis, Detection and analysis of hair, IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (7) (2006) 1164–1169. [9] S.Z. Li, A.K. Jain (Eds.), Handbook of Face Recognition, Verlag, 2005. [10] M. Turk, A. Pentland, Eigenfaces for face recognition, Journal of Cognitive Neuroscience 3 (1) (1991) 71–86. [11] P.N. Belhumeur, J.P. Hespanha, D.J. Kriegman, Eigenfaces vs. fisherfaces: recognition using class specific linear projection, IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (7) (1997) 711–720. [12] G.L. Marcialis, F. Roli, Fusion of appearance-based face recognition algorithms, Pattern Analysis and Applications 7 (2) (2004) 151–163. [13] A. Martinez, R. Benavente, The AR Face Database, CVC Technical Report #24, June 1998. [14] A.K. Jain, S. Prabhakar, S. Chen, Combining multiple matchers for a high security fingerprint verification system, Pattern Recognition Letters 20 (11–13) (1999) 1371–1379. [15] R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, Wiley, USA, 2001. [16] J.-E. Lee, A.K. Jain, R. Jin, Scars, marks and tattoos (SMT): soft biometric for suspect and victim identification. In: Sixth IEEE Biometric Symposium, 2008, pp. 1–8. [17] P.J. Flynn, K.W. Bowyer, P.J. Phillips, Assessment of time dependency in face recognition: an initial study. Audio and Video-Based Biometric Person Authentication, 2003, pp. 44–51.
Contents lists available at ScienceDirect
Journal of Visual Languages and Computing journal homepage: www.elsevier.com/locate/jvlc
Group-specific face verification using soft biometrics Gian Luca Marcialis , Fabio Roli, Daniele Muntoni Department of Electrical and Electronic Engineering, University of Cagliari, Piazza d’Armi, 09123 Cagliari, Italy
a r t i c l e in fo
Keywords: Biometrics Soft biometrics Face verification
abstract Soft biometrics have been recently proposed for improving the verification performance of biometric recognition systems. Examples of soft biometrics are skin, eyes, hair colour, height, and ethnicity. Some of them are often cheaper than ‘‘hard’’, standard biometrics (e.g., face and fingerprints) to extract. They exhibit a low discriminant power for recognizing persons, but can add some evidences about the personal identity, and can be useful for a particular set of users. In particular, it is possible to argue that users with a certain high discriminant soft biometric can be better recognized. Identifying such users could be useful in exploiting soft biometrics at the best, as deriving an appropriate methodology for embedding soft-biometric information into the score computed by the main biometric. In this paper, we propose a group-specific algorithm to exploit soft-biometric information in a biometric verification system. Our proposal is exemplified using hair colour and ethnicity as soft biometrics and face as biometric. Hair colour and information about ethnicity can be easily extracted from face images, and used only for a small number of users with highly discriminant hair colour or ethnicity. We show by experiments that for those users, hair colour or ethnicity strongly contributes to reduce the false rejection rate without a significant impact on the false acceptance rate, whilst the performance does not change for other users. & 2009 Elsevier Ltd. All rights reserved.
1. Introduction Biometrics received increasing attention in the last decade [1–3]. They are physiological or behavioural characteristics of human, as the fingerprint, the face, the voice, the iris, and the gait. Since biometrics are unique from person to person, cannot be forgotten and are very difficult to steal or reproduce, they have been proposed for personal verification instead of (or coupled with) passwords and PINs. Although performances of biometric verification systems are promising, they have to handle the large variations of the acquisition environment and the physiological and emotional states of the subject to be recognized. In particular, the time required for verifying
Corresponding author.
E-mail addresses: [email protected] (G.L. Marcialis), [email protected] (F. Roli), [email protected] (D. Muntoni). 1045-926X/$ - see front matter & 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.jvlc.2009.01.005
the person identity strictly depends on the system performance. As an example, if the percentage of genuine users’ rejection false reject rate (FRR) is too high, some users could be often wrongly rejected and they could be constrained to further access trials, that is, to further submissions of their biometric. Hence, the time required for accessing to the system could increase, with an impact on the verification time and, in the worst case, on the acceptability of the system. Some recent works suggested that using soft biometrics can allow increasing the verification performance [4–7]. Soft biometrics are physiological or behavioural human characteristics with a low discriminant power. Examples of soft biometrics are gender, eye colour, hair colour, skin colour, ethnicity, height, and weight. They are not unique from person to person; therefore they cannot be individually used for personal verification. However, the extraction of many of them is cheaper than that of standard biometrics (e.g., face and fingerprints), and can
ARTICLE IN PRESS 102
G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
give some evidences about the subject’s identity. In some cases, biometric and soft-biometrics extraction can have almost the same cost (for example, ethnicity and hair colour). On the basis of this concept, a simple framework has been proposed to integrate the ancillary information with the main biometric system in Refs. [4–7], and applied in identification mode: briefly, the posterior probability of each identity given the biometric score is combined with the correspondent posterior probability given the softbiometric ‘‘scores’’ by weighted averaging. As pointed out in [4–7], this approach exhibits some drawbacks: estimation of weights (no method is proposed, and soft biometrics must be weighted much less than the biometric score due to their low discriminant power) and estimation of posterior probabilities of soft biometrics, which are not permanent and reliable, since they need to be updated continuously. Therefore, we propose an alternative framework, which avoids the problem of evaluating the posterior probabilities related to soft biometrics, and exploits some hypotheses that can be drawn around them. Due to their low discriminant power, it can be argued that a certain soft biometric can be useful only for a limited set of users (this is mentioned in [7], but not exploited in the authors’ framework). For example, Latin people rarely exhibit blond hair. Worth noting, hair colour has been recently proposed as biometric in [8] for person identification. Experiments reported on the AR data set showed very low recognition accuracy, and a slight improvement by integrating hair colour and face as biometrics has been shown in [8]. Despite these results, hair colour could have a good impact on improving the verification performance, if considered as ‘‘soft’’ biometric, especially for users exhibiting highly discriminant hair colour, but it cannot bring information to others. Accordingly, hair colour (in general any soft biometrics) should be considered as able to add further evidences only if the claimed identity is in the group of users mentioned above. Moreover, the relevance of the hair colour is high if this group is much smaller than the overall user population. In other words, hair colour does not exhibit intrinsic discriminant properties, but is informative if it is possible to point out, from the particular user population, some peculiarities in the hair colour belonging to a small group of users only. In the following, we will refer to those users as ‘‘minority group’’. This point has not yet been investigated in the literature, and is addressed in the framework proposed in this paper. For its validation, we adopt a face verification system using hair colour and ethnicity as soft biometrics. We selected the face biometric for two reasons. Firstly, it has been investigated and used widely [9]. Secondly, hair colour does not require additional hardware because it can be extracted directly from the true-colour face image. Therefore, it is cheap to automatically extract from face images, although it has limited discriminant power. The face verification system is based on wellknown techniques [10–12]. Experiments are carried out on the AR data set for hair colour and on the Notre Dame data set for ethnicity. The requirements of this data set fit well with our hypothesis. In fact, it is made up of Latin
people prevalently [13]. On the other hand, Notre Dame can be easily partitioned according to the ethnicity. Our experiments confirm the effectiveness of our framework and show the benefits of using hair colour by reducing the false rejection rate without a significant impact on the false acceptance rate, whilst the performance does not vary for other users. This paper is organized as follows. Section 2 describes our proposal. Section 3 reports its validation by describing the system adopted in our experiments and related results on the AR data set. Section 4 concludes the paper.
2. The proposed framework In this section, we derive a simple framework that shows the combination of the information extracted from a generic soft biometric with the one extracted from the main biometric according to a proposed group-specific approach. In this paper, we show the framework only in the case of one soft biometric, but it can be easily generalized to more soft biometrics. The main characteristic of the proposed framework is that it is not based on the ‘‘fusion’’ of the main biometric score with a correspondent ‘‘soft-biometric’’ score as [4–7]. In other words, we do not express the information captured by soft biometrics by a score, but as an additional evidence aimed to increase the probability of the class, given the main biometric. Therefore, we derive the expression of the final score integrating biometric and soft-biometric information, as an expression of the probability of a certain genuine user given the biometric features and the related soft-biometric feature, as a function of the probability of the user with the only evidence of its biometric, which is viewed as the main biometric score. The feature related to the soft biometric is interpreted as further evidence, which increases the above probability with respect to the initial biometric score, under the hypothesis that the genuine user can be inserted into a small group with highly discriminant soft-biometric feature, exemplified by hair colour in this paper. Let o be a generic identity in the user population U ¼ {o1, y, oM}, xface be the information extracted from the given biometric (e.g., the face feature vector), and xhair, be the feature vector related to the hair colour (in general, to soft biometric). Relationships among face and hair colour can be modelled as a Bayesian network as shown in Fig. 1. This figure points out that the client posterior probability can be viewed as a function of the probability of xhair membership to a sort of ‘‘minority group’’ C (defined in the following) and by the current value of feature vector xface. Given the evidence on the client, information coming from face and hair colour is independent, as can be reasonably supposed [4–8]. If the face matching score is interpreted as P(o|xface), the final posterior probability, under the hypothesis behind the minority group definition, can be derived from the relationships among other events. First of all, we define formally the concept of ‘‘minority group’’. If we suppose that the soft-biometric feature
ARTICLE IN PRESS G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
103
regard to this issue, we can consider the relationship
xhair
Pðojxface Þ ¼ Pðojxface ; xhair 2 CÞPðxhair 2 CÞ þ Pðojxface ; xhair eCÞPðxhair eCÞ
xhair ∈C
(4)
Where we hypothesized the independence of hair colour with respect to the face information: P(xhairAC|xface) ¼ P(xhairAC). Eq. (4) can be written as follows
xface
Pðojxface Þ ¼ Pðojxface ; xhair 2 CÞp þ Pðojxface ; xhair eCÞð1 pÞ
ω
(5)
Now we can analyse the following ratio
Fig. 1. The Bayesian network modelling the proposed framework. In this network, two events are emphasized by two nodes: the simple evidence of having an xhair value, and the fact that xhair is localized into the ‘‘minority group’’ C.
Pðojxface ; xhair 2 CÞ Pðxhair 2 Cjxface ; oÞ Pðxhair eCjxface Þ ¼ Pðojxface ; xhair eCÞ Pðxhair eCjxface ; oÞ Pðxhair 2 Cjxface Þ (6) On the basis of Bayesian network in Fig. 1, we obtain
pðxhair Þ ¼
N X
pðxhair jxhair 2 C i ÞPðxhair 2 C i Þ
Pðxhair 2 C; xface joÞ pðxface joÞ Pðxhair 2 CjoÞpðxface joÞ ¼ pðxface joÞ
Pðxhair 2 Cjxface ; oÞ ¼
space can be partitioned into N clusters, with N5|U|: (1)
i¼1
The cluster Ci is defined as ‘‘minority cluster’’ if and only if Pðxhair 2 C i Þ ¼ pi 51 ( 9O U
8o 2 O;
Pðxhair 2 C i joÞ40:5
8oeO;
Pðxhair 2 C i joÞ ¼ pi
(2.1) (2.2)
Pðojxface ; xhair 2 CÞ po 1 p ¼ L41 ¼ Pðojxface ; xhair eCÞ 1 po p
i
Therefore, we can define the minority group, and indicate this with the couple {C,O}, where C is the set of clusters (into the given feature space) and O is the set of clients such that
8oeO;
Pðxhair 2 CjoÞ ¼ po 40:5 Pðxhair 2 CjoÞ ¼ p
(7)
Since an evidence on o makes independent the events xhairAC and xface. Accordingly, if oAO, Eq. (6) becomes (condition (2.2))
Finally, the union of minority clusters C is such that X Pðxhair 2 CÞ ¼ Pðx 2 C i Þ ¼ po0:5 (3)
8o 2 O;
¼ Pðxhair 2 CjoÞ
(3.1) (3.2)
No other constraints or hypothesis are made on such clusters. In Eq. (3.1), po indicates, for sake of compactness, the value of P(xhairAC|o). This definition reflects our hypothesis about the possibility of exploiting soft biometrics into a user population: there must be a set of users whose hair colour is highly discriminant (condition (2.2)), since it is rare in the same user population (condition (2.1)). Worth noting, the definition of minority cluster also implies that a small amount of clients can be characterized by the above-mentioned properties; thus the term ‘‘minority’’ can refer both to the rarity of hair colour and to the small amount of users exhibiting such hair colour. Condition (2.2) also imposes that if there is no evidence about the membership of a certain client in the minority group, related hair colour can be considered as common. The next step is to derive a relationship between the evidence of having an xhair belonging to such a minority group joint with the evidence of having xface, in order to evaluate the probability of being a genuine user. With
(8)
Otherwise (condition (2.2)), Pðojxface ; xhair 2 CÞ p 1p ¼1 ¼ 1p p Pðojxface ; xhair eCÞ
(9)
Therefore, if there is evidence that xhairAC, that is, xhair is a member of the minority cluster, we can write Pðojxface ; xhair 2 CÞ ¼ L41 Pðojxface ; xhair eCÞ
(10)
Eq. (5) becomes Pðojxface ; xhair 2 CÞ ¼
L Pðojxface Þ 1 p þ Lp
(11)
Score values related to face can be interpreted as a posterior probability [14]. Therefore, if we assume that s ¼ Pðojxface ; xhair 2 CÞ sface ¼ Pðojxface Þ
(12)
sface is given by the comparison of the claimed identity template and the input feature vector xface, and s is the score integrating face and hair colour information: s¼
L s ¼ asface 1 p þ Lp face
(13)
Worth noting, a41 as can be easily proved under the findings pointed out in Eq. (9). But, if oeO (Eq. (9)), L ¼ 1, thus s ¼ sface
(14)
Therefore, hair colour is useful only if there is evidence about its membership to the minority group. The practical consequence of this framework is that, if a good soft
ARTICLE IN PRESS 104
G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
biometric can be found as characterizing the features space according to the above framework, the obtained final score increases the separation between clients with highly discriminant hair colour and impostors. The main advantage of this framework with respect to the one proposed in [4–8] is that estimation of posterior probabilities, or the computation of soft-biometric score, is not needed, since s can be derived from sface. Moreover, whilst a different set of weights should be used for each client [4–8], here the number of parameters per client is limited by the fact that the method is group-specific; thus a unique parameter, namely L, must be used. Anyway, it is worth remarking that a clear comparison of the proposed framework with existing ones is out of the scope of the present paper. A separate study will be devoted to this aim. On the other hand, the framework requires the estimation of N, po and p. The N-value depends on whether some a priori information on the user population is available (e.g., Latin people prevalently, where blond hair or red hair are rarely expected, thus N ¼ 3). This helps in defining the N-value. In general, N will depend on the feature space related to the soft biometric adopted. With regard to po and p, they can be estimated once the minority clusters are defined, by cross-validation on a training set. In particular p is the frequency of hair colour samples in the smallest cluster and po is the frequency of hair colour samples of clients in the minority clusters according to p and Eqs. (3.1) and (3.2). We followed this cross-validation approach in our experiments. In this paper, we only performed experiments to validate the model under the assumptions that N ¼ 3 (two minority clusters). This has been motivated by the kind of user population in the adopted data set for experiments.
Fig. 2. Position of the hair colour samples, pointed out by squares, in a sample image. The eye positions are also pointed out.
reasonable value for capturing the hair colour appears to be 155 pixels above the eye locations. Two square-sized 10 10 pixels were extracted from those locations. This size takes into account the highness of the hair and the forehead. Fig. 2 shows the position of the square related to the hair colour. For each image, two samples of the hair are extracted, corresponding to the left and the right side of the face image. The CbCr values of each square are finally averaged. Therefore the hair colour feature vector is made up of four components, related to left and right sides of the face image. This is motivated by the necessity of considering that each face side can be differently illuminated. 3.2. Minority group identification and exploitation
3. Framework validation 1: experiments with face and hair colour In this section, and in Section 4, we show two sets of experiments aimed to validate the proposed framework. In the first set of experiments, adopted soft biometric is the hair colour, whilst in the second one we used the ethnicity. In both cases, the main biometric is the face. 3.1. The data set We carried out experiments on the AR data set [13]. This data set contains frontal view faces with different facial expressions, illumination conditions, and occlusions (sun glasses and scarf). Each person participated in two acquisition sessions, separated by 2 week’s time. Each session is made up of seven images per person. We selected 100 subjects (50 males and 50 females), manually cropped their face images and, after histogram stretching and equalization, resized them at 80 80 pixels. For computing the PCA space, we considered only the grey-level normalized values of the pixels, whilst we considered the CbCr components for the hair colour. Since the inter-ocular distance has been fixed to 115 pixels, a
Minority group can be identified manually or automatically. In this paper, we adopted the second approach, by partitioning feature vectors of hair colour with a 3means clustering algorithm [15]. In this paper N ¼ 3, since user population is prevalently Latin and it is expected that at most three groups (hair colour different from black hair) should be identified. The clusters in which samples follow the definition in Eqs. (1) and (2) are called ‘‘minority clusters’’: firstly, they must be the smallest; secondly, the frequency of hair colour samples falling in them must be very high for certain clients. The centroids of these clusters define, on the overall, the set C, whilst the correspondent set of clients defines O, thus allowing to obtain the minority group {O,C}. Two experiments were performed for N ¼ 31: (1) the first AR session is used for extracting a set of training samples from each client. A clustering algorithm, namely 3-means, is applied to this set. We identified the N1 1 Worth noting, we tried experiments with N ¼ 2 and N43. In the first case, the minority cluster was characterized by a number of samples so small that predicted value of pM and p was not reliable (pw especially). On the other hand, when N43, no clusters were found and 4-means stopped with error.
ARTICLE IN PRESS G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
105
Face Verification System Normalized face image
Comparison with template client
sface
s
α Eye position extraction
Face image
Claimed identity
Is claimed identity in the minority group ? no
yes Hair colour extraction
α
Eq. (8) yes
Is hair colour in the minority group ? no α=1
Hair Colour Handler Fig. 3. Basis processing flow of the proposed face verification system using hair colour.
cluster(s) such that the number of hair samples falling in those clusters was much less than that of samples falling in the remaining cluster, thus estimating p (Eq. (2.1)). Then, we identified in such clusters the clients for which is verified Eq. (3.1). We also used this session to compute the PCA space and the template image for each user (the mean of images has been considered), and estimating the operational points of the ROC curve. The second AR session has been used for testing the system, i.e., in the verification phase. (2) The same as (1), with the first and the second sessions inverted. Reported results are averaged on these two runs. With regard to the verification phase, the system works as shown in Fig. 3. When an user submits his face and claims a certain identity, it is first verified that this identity falls in O, hair colour is extracted and if the centroid of C is nearest to the hair colour feature vector, evidence on the membership to minority group is given in terms of a (Eqs. (8) and (9)). 3.3. Results This experimental section is organized as follows. In Section 3.3.1, we show the minority group of users identified using the 3-means clustering algorithm. In Section 3.3.2, we show that the application of Eqs. (8) and (9) allows to improve the performance for such minority group, whilst it does not exhibit significant variations for other users. 3.3.1. Automatic identification of minority group It is worth noting that the AR data set is made up of Latin people prevalently. As a consequence, it can be
expected that, using hair colour, the subjects with highly discriminant hair colour are characterized by sharp hair. Fig. 4 shows the users the minority group found by the 3-means algorithm (first run). Some errors occurred during the automatic extraction of hair colour: samples of the forehead have been taken instead of hair samples. For example, this happened to the first user in Fig. 4. By sampling correctly, those subjects are not classified in the minority group. Worth noting, the other clients clearly exhibit black hair colour. As shown in Fig. 4, the ‘‘minority group’’ corresponds to the set of users with hair colour significantly different from the majority of users. The size of the minority group is about 15% of the whole number of subjects. It is worth remarking that in this paper we are not proposing a novel approach to hair colour extraction and evaluation, but we are only interested in the impact of using effectively hair colour under the hypothesis of minority clusters into the user population. Population identified strongly fits the constraints in the definition of ‘‘minority clusters’’: a priori hair colour samples rarely exhibit their value, and their samples exhibit the probability of being more than 0.5 in the cluster. 3.3.2. Effect of soft biometrics on verification performance This section shows that the hair colour applied to the minority group allows reducing the related false rejection rate without a significant impact on the false acceptance rate, whilst the performance does not vary for other users. According to Section 2, estimated a was around 2.0 on average: 2.5 in the first run and 1.5 in the second run. The high standard deviation suggests that estimation is dependent heavily on the available data, and some
ARTICLE IN PRESS 106
G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
Fig. 4. Some subjects belonging to the minority users cluster according to the hair colour obtained by using the 3-means algorithm on 30 runs.
Minority against impostors ...using Hair Colour ...using Hair Colour (Ideal Performance)
10-1
10-2
10-3
All users-Face All users-Face + hair colour Minority group-Face Minority group-Face + hair colour 10-2
FRR
FRR
10-1
10-1 FAR
Fig. 5. Average ROC curve showing the improvement reported for all users (straight and dotted line), and for all users (minority users included) when claiming to be into the minority group (dashed and straight line with dots).
outliers may affect it. Therefore, robust estimation methods (better than ours) should be found. Fig. 5 shows the average ROC curve obtained by estimating the values of po and p using the 3-means clustering algorithm on the hair colour samples captured as described in Sections 3.1 and 3.2. These curves clearly show that, whilst ROC curves of the overall user population are substantially unaltered, the minority group’s ROCs exhibit a clear performance improvement. This is due to Eqs. (8), (9) and (11), which refine the estimation of the score. The reported result is remarkable when considering that AT exhibit a small sample size, and a very few of samples per class has been used for a estimation. Another point is the estimation error of membership in the test set, which strongly impacts performance. In fact, in Fig. 5 we did not separate results of the classification, but only checked the results on the users claiming an identity into the minority group. Therefore, we investigated ROC curves related to users into the minority group, claiming themselves, against the other users. Fig. 6 shows two ROC curves that focus on such an information. In the first one, we show the ROC curve of genuine users into the minority group against the other users claiming to be into the minority group without
10-2 10-3
10-2
10-1
100
FAR Fig. 6. Average ROC curve showing a sharp performance improvement in users of minority group verification against impostors not belonging to the minority group (some of them could be wrongly assigned in the verification phase).
(dashed line) and with the help of hair colour information (straight line with dots). It can be noticed that equal error rate (EER) fell from 6% to 4.5%, and 1%FAR from 10% to 7% about. The straight line without dots in Fig. 6 shows that the potentialities of our framework are more than that shown in our experiments. ROC curves can be further improved by removing the misclassification performance relating to impostors, whose hair colour is classified as belonging to the minority clusters. In Fig. 6 the performance of impostors claiming to be users into the minority group is shown. Misclassification errors are due to the simple classification approach adopted here, and, as it can be seen, they impact on the final results. Ideally, the verification performance related to minority group could lead to 3.5% and 4% by considering EER and 1% FAR points. Therefore, the benefits we pointed out allow us to observe that: (1) the subject identity is verified more frequently for the minority group; therefore, the average verification time decreases with a favourable impact on its acceptability. This means that, if we were able to find a soft biometric peculiar to different subsets of the user population (i.e., ‘‘different’’ minority groups, one, or more, for each soft biometric), the whole acceptability and the
ARTICLE IN PRESS G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
100
100
90
90
70
Face Face+Ethnicity
80
Face Face+Ethnicity
FRR on test set
FRR on test set
80
60 50 40 30
70 60 50 40 30
20
20
10
10
0
107
0 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 x% FAR on training set
1
0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 x% FAR on training set
Fig. 7. (a) FRR on the overall user population when operation point computed is between zero%FAR and 1%FAR. (b) Zoom of (a) curve showing the strong reduction of FRR especially at zeroFAR point.
performance of the system could be strongly augmented; (2) the increase in the verification score values is not significant for the majority group, that is, those users are correctly classified. This is because the soft-biometric information is applied corresponding to very rare hair colour values, according to our definition of minority cluster. 4. Framework validation 2: experiments with face and ethnicity 4.1. The data set We adopted the Notre Dame data set2 [17], which has also been used for the data collection at face recognition Grand Challenge3. This data set contains several frontal face images, both indoor and outdoor. Each subject was photographed with a high-resolution digital camera (1600 1200 or 2272 1704) under different lighting and expression conditions. Many subjects were photographed every week for 10 weeks in the Spring of 2002, 13 weeks in the fall of 2002, and 15 weeks in the spring of 2003. For the purpose of this paper, we removed all outdoor images, obtaining 50 images per client, and selected a subset of 75 users in order to have a clear subdivision in terms of ethnicity. We obtained seven African people, sixteen Asians, and the rest are Caucasians. Three kinds of users are hence found, for which Asians and Africans represent clearly the ‘‘minority group’’. The data set was subdivided into training and test set, both made up of 25 images per client. The training set was used to estimate PCA–LDA for face verification and also compute the parameters of our model. In this case the minority group was manually identified. The system integrating ethnicity and face biometric are the same as 2 3
http://www.nd.edu/cvrl/UNDBiometricsDatabase.html http://www.frvt.org/FRGC/
described in Fig. 3. The main changes involve the computation of a parameter, which are described in Section 4.2. With regard to biometric face images, we followed the same protocol adopted in Section 3.
4.2. Results We adopted the whole cropped face by concatenating CbCr components, and performed LDA on the related feature vectors. We classified training set data by adopting 1 nn with the leave-one-out method, in order to compute the confusion matrix. pw was computed from this matrix, for each client. Thus, we evaluated a. In this case, computation of a is shown to be quite stable by inverting training and test set: the average value of 2.15 has been found with a standard deviation of nearly zero. In this section, we first of all point out the advantage of combining face biometric and ethnicity when high security levels must be reached. In particular, we investigated a range of thresholds, computed on the training set, and related to the range 0%FAR–1%FAR. It is well-known that these performance parameters are quite crucial [1,2], and reducing them is strongly helpful for clients. Fig. 7(a) and (b) shows the strong reduction of false rejection rate evaluated on the overall user population, on this range of operational points. Figure (b), in particular, shows the region between 0%FAR and 0.2%FAR. It is worth noting that, for the first value, the corresponding false rejection rate fell from about 90% to 70%. Notice also that 0%FAR is on the test but the threshold was estimated on the training set. Therefore, estimation error of threshold affected the real performance of the system. With regard to this problem, using soft biometric allowed to ‘‘correct’’ the operational point at the same time by maintaining the FAR at the same value (about zero). Fig. 8 shows the ROC curves related to the minority group (equivalent to Fig. 6). In this case, we have a very noticeable improvement of performance, as it explains
ARTICLE IN PRESS 108
G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
5. Discussion
10 Face Face + Ethnicity
9 8
% FRR
7 6 5 4 3 2 1 0 0
1
2
3
4
5
6
7
8
9
10
% FAR Fig. 8. ROC curve of Minority group (Africans and Asians) against Majority group.
%HTER on test set (Minority group)
50 45 40
Face Face+Ethnicity
35 30 25 20 15 10 5 0 0
0.05
0.1
0.15
0.2
0.25
x% FAR on training set Fig. 9. %HTER on the test set for minority group. Plot reports values corresponding to high security level operational points evaluated on the training set.
ethnicity well (better than hair colour). The equal error rate fell from 3% to 1.5%. The improvement is good even for other operational points. Worth noting, even in this case we adopted a quite simple approach for ethnicity classification. Being state-of-the-art very active in this topic [1,5,6], it is reasonable to hypothesize that ethnicity can be a promising soft biometric, which can be adopted according to the proposed framework. Finally, HTER (Half Total Error Rate [1]) for high security level operational points is reported in Fig. 9. Plot is related to minority group. Here we have a confirmation of Fig. 9: in particular, HTER at 0%FAR drops to about 12% when combining ethnicity with faces, whilst the HTER estimated is quite high (close to 50%) using face biometric only.
Reported results showed some clear advantages of using soft biometrics, and also showed some limitations due to their intrinsic nature of ‘‘soft’’ features. In this section, we consider these pros and cons, thus trying to fairly evaluate the actual impact of adopting soft biometric in real verification systems. First of all, according to the definition of soft biometrics, they do not exhibit properties of uniqueness, universality, and permanence. Thus they may change over time more quickly than standard biometrics. This implies that their usefulness strongly depends on the specific context. Moreover, some soft features, hair and eye colour for instance, are very easy to counterfeit, most of all if a malicious user is interested in gaining a false acceptance during identity verification. So far, none of the state-ofthe-art works [4–7] discussed this issue satisfactorily. Recently, another paper involving soft biometrics have been proposed [16] without solving this problem. On the other hand, we must also keep in mind that the most widely used biometrics can be faked (voice, face, fingerprint, iris); thus their use is still focused on assisting, and not substituting, human operators. This holds for soft biometrics too. With regard to our system, it is true that verification performance is not improved on the overall, but this can be considered also an advantage, because, as we pointed out in the paper, the false accept rate (FAR) does not consistently change or increase for clients out of the minority group. Even a trial of faking hair colour, for example, can be viewed in the experimental section as a misclassification of the system. As we showed in Sections 3 and 4, this does not lead to a significant increase of false acceptance rate, due to the relative value of a, which needs to be reliably computed. Another case is that the user population is too small for appreciating significant improvements in using soft biometrics. However, if we take into account the point of view of the user, it is significantly better for him to reduce the false reject rate, under the condition that the false acceptance rate does not increase strongly. As mentioned in the introduction, acceptability of the system may increase. Finally, use of soft biometrics can be made more robust by adopting multiple soft biometrics, in order to cross different minority groups and decrease a. This also involves a modification of the present framework in order to take into account the integration of more soft biometrics. This is a matter of on-going research, and we are not able to give a specific, alternative, and theoretical formulation. However, we can suggest two possible solutions, which must be modelled in order to evaluate the one most suitable for the task. These solutions are presented in Fig. 10(a) and (b). In Fig. 10(a), we have a different contribution, namely, a different a, for each biometric, which is made independent of others from the knowledge of the client to be recognized. In Fig. 10(b), the different contributions of each soft biometric are modelled by a unique set of minority clusters, thus leading to an a-value condensing information from all soft
ARTICLE IN PRESS G.L. Marcialis et al. / Journal of Visual Languages and Computing 20 (2009) 101–109
xhair
xhair ∈C
xhair
xethnicity
xethnicity ∈C
xethnicity
xsoft ∈C
xface
ω
109
xface
ω
Fig. 10. Possible extensions of the proposed model to more than one soft biometric: (a) each soft biometric contributes independent of the other, leading to different a-values and (b) the different contributions of each soft biometric are modelled by a unique set of minority clusters, thus leading to an a-value condensing information from all soft biometrics.
biometrics. As we mentioned, currently we are not in the position to prefer one solution to the other, but this will be matter of a future work.
population and adopted soft biometrics. Extension of the proposed framework to more soft traits is still a matter of on-going research.
6. Conclusions
References
In this paper, we claimed that, due to its limited discriminant power, soft biometrics are useful only for a limited set of users. Therefore, we derived a framework to combine information from the face and two soft biometrics, namely hair colour and ethnicity, represented by feature vectors. The novelty of this framework with respect to the state-of-the-art is that it overcomes the usually adopted concept of soft biometrics, because the soft-biometric information is used in a totally different manner, in particular, group-specific. Moreover, the number of parameters it requires is less than that of the existing ones. In particular, the estimation of posterior probabilities for each soft biometric is avoided. Experiments were carried out on the AR and Notre Dame data sets. In particular, AR is made up of Latin people prevalently. Latin people rarely exhibit sharp hair. On the other hand, Notre Dame allows one to manually, and easily, partition user population according to the ethnicity. Therefore, verification performance could be improved for those users, thanks to the information provided by soft biometrics above. The reported results showed the validity of the proposed framework, since it is experimentally confirmed that: (1) the minority group corresponded to users with highly discriminant hair colour; (2) this soft biometric applied to the minority group allowed reducing the related false rejection rate without a significant impact on the false acceptance rate, whilst the performance did not vary for other users significantly. Definitive conclusions cannot be drawn on the basis of this limited set of experiments. In general, we believe that different soft biometrics can identify different groups of users population; thus the proposed framework should be extended accordingly. It is worth remarking that this framework is an alternative to existing ones. Therefore, a future work will compare it with them by varying the user
[1] A.K. Jain, R. Bolle, S. Pankanti (Eds.), Biometrics: Personal Identification in Networked Society, Kluwer Academic Publishers, Dordrecht, 1999. [2] J. Wayman, A.K. Jain, D. Maltoni, D. Maio, Biometric Systems: Technology, Design and Performance Evaluation, Verlag, 2005. [3] A. Ross, K. Nandakumar, A.K. Jain, Handbook of Multibiometrics, Verlag, 2006. [4] U. Park, A.K. Jain, I. Kitahara, K. Kogure,N. Hagita, ViSE: visual search engine using multiple networked cameras, in: Proceedings of the International Conference on Pattern Recognition (ICPR), Hong Kong, August 2006, pp. 20–24. [5] A.K. Jain, S.C. Dass, K. Nandakumar, Soft biometric traits for personal recognition systems, in: Proceedings of the International Conference on Biometric Authentication (ICBA), Lecture Notes in Computer Sciences, vol. 3072, Hong Kong, July 2004, pp. 731–738. [6] A. K. Jain, K. Nandakumar, X. Lu, U. Park, Integrating faces, fingerprints, and soft biometric traits for user recognition, in: Proceedings of the Biometric Authentication Workshop, Lecture Notes in Computer Sciences, vol. 3087, Prague, May 2004, pp. 259–269 [7] A.K. Jain, S.C. Dass, K. Nandakumar, Can soft biometric traits assist user recognition?, in: Proceedings of SPIE, 5404, 2004, pp. 561–572. [8] Y. Yacob, L.S. Davis, Detection and analysis of hair, IEEE Transactions on Pattern Analysis and Machine Intelligence 28 (7) (2006) 1164–1169. [9] S.Z. Li, A.K. Jain (Eds.), Handbook of Face Recognition, Verlag, 2005. [10] M. Turk, A. Pentland, Eigenfaces for face recognition, Journal of Cognitive Neuroscience 3 (1) (1991) 71–86. [11] P.N. Belhumeur, J.P. Hespanha, D.J. Kriegman, Eigenfaces vs. fisherfaces: recognition using class specific linear projection, IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (7) (1997) 711–720. [12] G.L. Marcialis, F. Roli, Fusion of appearance-based face recognition algorithms, Pattern Analysis and Applications 7 (2) (2004) 151–163. [13] A. Martinez, R. Benavente, The AR Face Database, CVC Technical Report #24, June 1998. [14] A.K. Jain, S. Prabhakar, S. Chen, Combining multiple matchers for a high security fingerprint verification system, Pattern Recognition Letters 20 (11–13) (1999) 1371–1379. [15] R.O. Duda, P.E. Hart, D.G. Stork, Pattern Classification, Wiley, USA, 2001. [16] J.-E. Lee, A.K. Jain, R. Jin, Scars, marks and tattoos (SMT): soft biometric for suspect and victim identification. In: Sixth IEEE Biometric Symposium, 2008, pp. 1–8. [17] P.J. Flynn, K.W. Bowyer, P.J. Phillips, Assessment of time dependency in face recognition: an initial study. Audio and Video-Based Biometric Person Authentication, 2003, pp. 44–51.