Anti-spoofing method for fingerprint recognition using patch based deep learning machine

Engineering Science and Technology, an International Journal xxx (xxxx) xxx

Contents lists available at ScienceDirect

Engineering Science and Technology, an International Journal journal homepage: www.elsevier.com/locate/jestch

Full Length Article

Anti-spoofing method for fingerprint recognition using patch based deep learning machine Diaa M. Uliyan a,⇑, Somayeh Sadeghi b, Hamid A. Jalab b a b

Middle East University, Faculty of Information Technology, Amman 11831, Jordan University of Malaya, Faculty of Computer Science and Information Technology, Kuala Lumpur 50603, Malaysia

a r t i c l e

i n f o

Article history: Received 7 January 2019 Revised 30 April 2019 Accepted 16 June 2019 Available online xxxx Keywords: Biometric systems Deep learning Discriminative Restricted Boltzmann Machines Fingerprint authentication

a b s t r a c t Today’s with increasing identity theft, biometric systems based on fingerprints have a growing importance in protection and access restrictions. Malicious users violate them by presenting fabricated attempts. For example, artificial fingerprints constructed by gelatin, Play-Doh and Silicone molds may be misused for access and identity fraud by forgers to clone fingerprints. This process is called spoofing. To detect such forgeries, some existing methods using handcrafted descriptors have been implemented for assuring user presence. Most of them give low accuracy rates in recognition. The proposed method used Discriminative Restricted Boltzmann Machines to recognize fingerprints accurately against fabricated materials used for spoofing. Ó 2019 Karabuk University. Publishing services by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction Today, one of major challenges confronting biometric systems is the rapid threat of malicious actions. The most of malicious actors use a common type of presentation attack, known as ‘‘spoofing” to defeat biometric systems [1]. The main goal of presentation attack is impersonating target victims that have the desired authorization. It happens when intermediate spoofing forgers intentionally guess that identity of unsuspecting individuals via steal victim’s fingerprints, tampering them with a certain legal material to defraud fingerprint recognition based systems [2]. Failure to prevent fingerprint spoofing forgeries on devices [3] may compromise confidential information in many applications such as video surveillance [4], biometric identification [5] and face indexing in social media [6]. This issue motivates researchers to employ countermeasure techniques and combined them into the biometric based systems to beat such forgery. Fig. 1 shows some examples of fingerprints which gained from real and fake fingers. Visually by the eyes, it cannot distinguish between real and fake ones. The first row in Fig. 1 shows the real fingerprints. The second row shows fake ones were acquired from artificial fingers which made by different fabricate materials. Many spoof fingerprint ⇑ Corresponding author. E-mail address: [email protected] (D.M. Uliyan). Peer review under responsibility of Karabuk University.

detection techniques have been developed [8–10]. Thus, misrepresentation of fingerprints and detection of forged fingerprints is still an open issue [11]. There are many countermeasure methods have been proposed [12] that employed multiple factors in securing the information. Spoof forgeries [13,14] can be defined as the techniques used to deceive biometric based systems by giving a forged identity of the user to have authentication. The authentication system mainly considers individuals of users based on behavioral and biological features such as face recognition, iris features, voice signals, fingerprint and palm veins [15]. In case of fingerprint spoofing forgeries, the intermediate spoof forger uses finger molds to deceive the biometric authentication system. For instance, palm images have been printed and easily defeat the biometric system [16]. Thus, fingerprint spoofing detection methods are classified into two classes depending on whether additional sensors are used or not: 1) Hardware-based (exploring extra finger readers) [17]: the state of art fingerprint readers can scan the input fingerprints with high resolutions which are suitable for matching. The fingerprint reader executes misrepresentation detection on two types of finger images: non processed and raw fingerprint images [15]. While these raw images contain a clear information that provides discriminate information for spoof detection, these raw images have to be processed such as color conversion, normalization or filtering to extract

https://doi.org/10.1016/j.jestch.2019.06.005 2215-0986/Ó 2019 Karabuk University. Publishing services by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

2

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx

to enhance these traces. This process can be done by capturing fingerprint traces and save it into image. The fingerprint image is enhanced and then, molds are created materials like silicone, gelatine, or PlayDoh over the printed image [28]. For instance, researchers at Michigan State University help police in July 2016 to unlocked a fingerprint secure smart phone with homicide case1 based on a 2D printed fingerprint spoof method proposed by [29].

Fig. 1. Fingerprint samples using Cross Match 2013 reader [7].

features for matching. The main objective of hardware based methods is to reveal the features of liveness, such as blood flow [18], skin distortion [19], odor [13]. As pointed in [20], more details about Liveness Detection methods were tested in the LivDet datasets. 2) Software-based methods rely on the image information scanned by intent sensor without requiring more hardware cost, to classify images into: live and spoof fingers [10]. Among these methods, the software based methods has a virtue it can be adapted in a common fingerprint scanners to recognize and examine the fingerprint if it is forged by using the fake materials in fingerprint image. In the paper, the software based method is considered in research to detect fingerprints under spoof materials. Software methods can be categorized into two main classes [21]: feature based [22] and deep learning based. The feature based recognition method is mainly extracts discriminate fingerprint liveness with a single feature point in early stage, and it has not been shown good performance for various fake materials. It can be noticed, that deep learning methods are used to learn fake fingerprints for the various types of spoof materials [23]. 2. Related works With the rapid use of authentication systems using fingerprint recognition for many applications in recent years, the task of observing fake fingerprint becomes a vital due to fingerprint can be easily forged by using different types of instruments, for instance, wood glue, gelatin, silicone or printed fingerprints [1]. The texture of a real finger may have lasting residual effects on surface and then, is transferred into the material to counterfeit any biometric authentication system. Due to the identical texture between real and fake fingerprint, this issue will easily can deceive the authentication fingerprints based systems. Various fingerprint spoofing materials have been explored in [24]. The real scenario is that the forger tries to bypass a fingerprint recognition sensor by replicating a certain fingerprint image. The cloned fingerprint has an artifact which is employed to do presentation attack. The main challenge in research is how to discriminate genuine living fingerprints from fake ones [25] which is inspired by The ISO standard IEC 30107-3 (E). It gives a foundation for presentation attack detection by establishing a common framework that specify presentation attack events and detect them [26]. Two well known methods are used fabricate and replicate a fingerprint: 1) Cooperative methods: the finger of user must be placed into certain elastic material [27]. These materials help to create a mold of a fingerprint using for example, silicone, gelatine, or PlayDoh. 2) Non-cooperative method: it is made when the user accidently, left a trace of fingerprint on a surface and it required

Various materials can reveal such features: gradients, ridge and intensity in forged fingerprints. Therefore, the task of the existing anti spoof fingerprint recognition systems extensively depends on the type of fabricated materials used to create fake fingerprints via training step [30]. For instance [31], mentioned that single low level feature-based methods struggles to carry out various spoofing fingerprint materials. Features in fingerprint images are the core of discriminating forged fingerprints for different kinds of fake fingerprints. Multiple features become a practical way to characterize the properties of forged fingerprints and the real ones when the sole of the authentication system does not know what the types of materials used for forgery. For instance [10], have combined multiple features: 1-gradient features from the input image where local interest points detected by SURF method. 2-pyramid multiscale features of the Histograms of Oriented Gradient. 3- Texture Gabor features. After combining the three features, the dynamic score level fusion approach is employed to decide if the fingerprint is fake or real. Their method is tested on LivDet 2011 dataset with an Average Equal Error Rate of 3.95% and also it decreased the Average Classification Error rate to 2.27%. Rattani and Ross [32] proposed a fingerprint spoof detector using combined multiple features: Grey Level Co-occurrence Matrix (GLCM) with Histogram of Oriented Gradients (HOG) for exposing liveness fingerprints. Xia et al. [33] developed a fingerprint recognition method based on intensity variance features and local binary gradient orientation. These features are combined in a feature vector using cooccurrence probability method which is finally, classified by support vector machine (SVM) classifiers. In fingerprint spoofing recognition systems, two common ways have been identified for detecting the spoof fingerprint: 1) the active liveness features of fingerprint detectors, for instance, by examining pulse, perspiration patterns and blood pressure [34]; 2) Passive pattern analyzer for fabricating materials [35] for instance, the lack of details of spoofed fingerprints according to the real ones, fingerprint pattern differences. The last type of spoofing problem is the scope of our proposed method discovers a threat of using material and sensor. Several studies spoofing methods in fingerprint recognition have been introduced in [5]. Among the most general methods for static extracted features: texture based anti-spoofing techniques involves statistical feature analysis [36], Ridge based features [37], curvelet transform [38], Power Spectrum Fourier based features [39], Local Phase Quantization patterns [36] and Local Binary Patterns [40]. However, recent methods focused on combining multiple features and probably even multiple liveness detectors. For instance [41], suggest a fusion of liveness detector with a single modality for anti-spoofing capabilities. Galbally et al. [42] employed multiple features of spoof fingerprints such as, orientation certainty level and local clarity score to discover its liveness. Recently, some existing methods inspired by its deep learning feature extraction, such as a Convolution Neural Network (CNN)

1 https://statenews.com/article/2016/08/how-msu-researchers-unlocked-a fingerprint-secure-smartphone-to-help-police-with-homicide-case

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

3

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx

which can be trained to distinguish a live finger from a spoof [6,43]. Deep learning has been adopted to generate a robust fingerprint spoofing system. For instance [31], shows that convolution neural network can identify the forged fingerprints with known fabricate materials. Chugh et al. [44] proposed a method for fingerprint recognition by extracting local regions from the image itself. These patches were centered and aligned using minutiae features to provide salient descriptors. These cues are trained via CNN models to improve detection results. In this paper, we proposed a deep learning method for to recognize real fingerprints and detect spoofs. The main contributions of our method are summarized as follows: 1) It attempts to distinguish real fingerprint images from fake ones. 2) It analyzes the image consistency based on scaled and rotated ROIs. Based on these ROIs features, we propose a deep discriminative model for training detection. The rest of the paper is presented as follows. In Section 3, we introduce two main techniques: DRBM and DBM used in the proposed method identify representation of real and forged fingerprints. Experimental results are presented in Section 4. Finally, Section 5 gives a conclusion and future works. 3. Proposed method We propose a novel method to determine fake fingerprints regarding spoof forgeries were used in authentication based systems, we adapt multiple features called deep features which extracted from images such as, Deep Boltzmann Machine (DBM) [45]. The main benefit of DMB is that its layered architecture helps to investigate a complex relationships between features and enables deep learning of high detailed features of data. Extracting features from the input image based on Deep Neural Networks aids to understand data in depth. DBMs are probabilistic deep learning that copes with complex patterns impressively by extracting highly detailed features from the image. It is suitable for tasks in which the patterns might not be easily detected or forged. Results on the Cross Match [7] dataset show that our technique achieves good results with comparison to handcrafted based methods, the results show high detection rate regarding spoofing attack. The motivation of using deep learning features includes: 1) Restricted Boltzmann Machines (RBMs) adapted in many applications such as image classification and medical image investigation and pattern analysis to solve various learning problems. 2) Restricted Boltzmann Machines typically developed to extract features and build a self-contained framework for generating competitive non-linear classifiers. 3) We introduce RBM algorithm that introduces a discriminant factor to RBM training. Thus, the notion of two main deep learning features: Restricted Boltzmann Machines (RBM) and Deep Boltzmann Machines (DBM) are introduced below regarding spoofing forgeries. 3.1. Restricted Boltzmann Machines (RBM) The (RBM) as introduced in [46] is a kind probabilistic graphical model based stochastic neural networks which are used to describe the dependency among a set of random data variables using a two layer architecture. The architecture of an RBM as

shown in Fig. 2 arrange two types of layers: a visible layer v with stochastic m neurons are direct relations with a hidden layer h with n neurons. However, there are no connections between neurons in the same layer, and this restriction gives the RBM its name. A weight matrix is constructed asWmXn . It has the weights between the connected, visible and hidden neurons, where wij represents symmetric weights between the ith visible neuron vi connected with the jth hidden neuron hj . Each unit in a visible layer V is connected with all the units in the hidden layer h. Assume layers, V and h, with binary stochastic variables, i.e., V 2 ½0; 1m and h 2 ½0; 1n . Where m and n are the numbers of units of the visible and hidden layers. The energy function for connecting configuration between V and h layer from Bernoulli distribution with success probability in RBM method [48] is defined as

EðV; hÞ ¼ 

m X

ai vi 

i¼1

n X

bj hj 

j¼1

m X n X i¼1

vi hj wij

ð1Þ

j¼1

where ai and bj are the biases of ith visible layer and jth hidden layer, respectively. The model parameters ai is composed of vector a ¼ ½a1 ; a2 ;    :; aV  and bj is composed of vector b ¼ ½b1 ; b2 ;    :; bh . The marginal probability of a joint distribution function over visible layer is defined as

PðvÞ ¼

1X expEðv;hÞ Z h

ð2Þ

where Z is the partition function defined as follows:

Z¼

XX v

expEðv;hÞ

ð3Þ

h

Z is the sum of possible pairs of (v, h). Let V be a m dimensional vector and let h be a n dimensional binary vector. while visible units are binary, we have total 2mþn pairs of ðv; hÞ. Since the RBM is a probabilistic binary structure may includes different layers, each higher layer takes the correlation between actions of upper hidden features from lower hidden layer. The lower layer generates visible units connected with the higher layer based on joint distribution with mutual independent conditional distribution probabilities PðhjVÞ and PðVjhÞ are defined as follows

Pðvi ¼ 1jhÞ ¼ /ð

n X

wij hj þ ai Þ

ð4Þ

wij vi þ bj Þ

ð5Þ

j¼1

and

Pðhj ¼ 1jvÞ ¼ /ð

m X i¼1

where / () stands for the sigmoid function, /ðxÞ ¼ 1=ð1 þ expðxÞÞ. For an nhidden-layer RBM, its model parameters consist of value

T ¼ fa1 ; b1 ; w1 ; a1 ; b1 ; w1 ;       :; am1 ; bn1 ; wmn1 ; am ; bn ; wmn g ð6Þ Given the training input data T of the visible neuron, it is not easy to approximate the parameters of vector T by using maximum likelihood criterion directly because of various hidden layers found in the RBM model. To solve this issue, we have implanted a greedy based learning algorithm to train the stacked RBM model [49] in the input image. A greedy learning algorithm tries to find out the   parameters of the first layer RBM a1 ; b1 ; w1 to model the visible training data. Then, it saved the parameters of the first layer into   vector v ¼ a1 ; w1 and produce a Gibbs sampling from the first layer in RBM as Pðhj ¼ 1jvÞ to train the next layer of RBM   a2 ; b2 ; w2 , Let h = (a,b,W) be the set of parameters of RBM. The

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

4

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx

Fig. 2. The graphical model for two layers: v and h in a) first layer of RBM and b) a three hidden layer of RBM as introduced in [47].

goal of training step is to maximize the probabilities of occurrence of all the available training samples from input vector T which is defined as follows

l ¼ argmax

Y

PðvÞ

ð7Þ

v2l

When the fingerprint is scanned by reader, then it is converted to grayscale image that has real value pixels. A discriminative RBM [48] is employed, which generate Gibbs samples from multinomial distribution with probability to make vector T has elements of Gaussian units. Therefore, Equation (1) can be redefined as follows

EðV; hÞ ¼ 

m n m X n X 1X ðv i  ai Þ2 X  b j hj  2 2 i¼1 ri j¼1 i¼1 j¼1

vi

ri

hj wij

ð8Þ

The conditional probability of visible layer based on Eq. (3) is redefined as follows

Pðv i ¼ 1jhÞ ¼ N

v ij

n X j¼1

where

wij hj þ

ai

!

r2i

ð9Þ

r is the variance of the normalGaussian distribution of N. 2

3.2. Deep boltzmann machines (DBM’s) Deep boltzmann machines are adopted for several reasons: 1) DBM’s is a talented method which can learn a complex internal features from shapes and objects in image such as fingerprint. 2)

To improve the recognition performance against noise and missing data. 3) It has the ability to extract deep features obtained from a large supply of unlabeled sensory inputs 4) It deals with vague inputs robustly due to the the approximate inference step works in a bottom up initialization and also can incorporate with topdown view. A deep bottom up Boltzmann machine as shown in Fig. 3, left panel, in which each layer captures complicated, higher-order correlations between the activities of hidden features in the layer below. After a bottom up initialization of the DBM to train layer by layer manner. The learning procedure implemented using called Mean Field (MF) [50] method to enhance its performance. First, a stack of RBMs have been employed in and estimates the parame  ters a1 ; b1 ; w1 of the first layer in RBM to model the visible training data. Then, it saved the parameters of the first layer and produce Gibbs sampling from the first layesto train the second layer of RBM. Mean Field inference method minimizes the total energy of the RBM based on the parameters estimated via partial inferences which converges much faster than with random initialization. Then, it computes an approximation of Q MF ðhjv; lÞ that represents a fully factorized distribution to approximate the true distribution of the hidden layers PðhjVÞ. This approximation is computed as follows

Q

MF

  F L Y   Y  l k  ðhjv; lÞ ¼  q hl    l¼1 k¼1

ð10Þ

where L represents the number of hidden layers, F stands for the   k number of nodes in the hidden layer. q hl ¼ l1 , where l ¼ 1. As a result of DBM, the parameters of the mean field inference are constructed as follows

l ¼ fl1 ; l2 ; :::; lL g

ð11Þ

The choice of mean-field was measured due to two main reasons: First, it is a common type of approximate inference algorithm that calculates the convergence criteria which considerably assists the structure of deep learning machine. Second, when biometric systems have the tasks of image analysis, we assume that subsequent layers over hidden layers of a given image that has a single extracted feature. Then, mean-field should be suitable. Hence, the proposed method aimed to employ presentation attack detection (PAD) and the determination of a presentation attack starts by recording of a latent fingerprint from LiveDet

Fig. 3. a) three-layer Deep Boltzmann Machine. b): Pre training consists of learning a stack of modified RBM’s, which are then composed to create c) a deep Boltzmann machine.

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx

Fig. 4. Input image and selected ROIs with their 10 constructed patches for each ROI.

Dataset and apply post processing to remove the noise from the image due to scanning errors and Close of ridge lines by scaling operation to do the following steps: 1. Inputs and preprocessing: The original input fingerprint images are entered as training set of grayscale images into the RBM model to extract features from regions of interest (ROI). Fig. 4 presents, two modalities of the images were used: Grayscale of the input images and patches of theirs normalized ROIs includes 10 patches of scaled and rotated ROIs. This step might help to increase the size of the training dataset and it’s mandatory to prevent missing of data when approximating the DBM’s parameters. The first step is mandatory to observe the reflection of fingerprint by examining three features: Shape, consistency of the fingerprint from different rotation angles. It can be noticed, that Attack presentation detection rate based on these features differs from artefacts to bona fide fingers. It is implemented as shown in Fig. 4. 2. Deep features extraction based on Discriminative RBM: the next step is to pre-training patches of grayscale images through DRBM. The process is applied in a greedy bottom-up way in order to approximate the real-valued patches to posterior probabilities of activation. 3. DBM training: Mean-Field algorithm was applied in this step to update the weights and the biases of the RBMs stacked in the DBM in a more accurate way. Finally, Multilayer Perceptron network was constructed using the same architecture as the DBM to perform backpropagation for training. The weights of each RBM stack were computed through local parameters: a, b and W of each individually trained RBM as shown in Fig. 5.

5

4. Feature vector generation: for each fingerprint image, we have employed three stacks of RBMs for three Selected regions of input image, which are: original scanned fingerprint image 40  40 pixel size, five patches detected under various scaling factors Sx ¼ 0:3; 0:6; 0:9; 1:2; 1:5 and five rotated cropped images under rotation angles h ¼ 30; 60; 90; 120; 18. While the input layer of these RBMs includes the size of the 40  40  3 visible neurons, the hidden layer has 1000 neurons. These features are saved in the feature vector of size 40  40  3 for every patch image. 5. Linear Discriminant Analysis (LDA): we have considered the 11 (1 ROI + 5 scaled + 5 rotated) patch versions of the original fingerprint image of size 725  800 implies a feature space of 40  40  11 = 17600 dimensions. It is computationally challenging to handle big matrices. The direct LDA algorithm in [51] for high-dimensional data is employed. It optimizes Fisher’s criterion, which maximizes between class scatter and minimizes within class scatter. It is suitable for two class classification tasks like biometric authentication systems to make a decision if the fingerprint is valid or spoofed. 6. K -Nearest Neighbor (KNN): given a fingerprint image, its 10 patches have been constructed and classified using K-Nearest Neighbor algorithm [52]. KNN is a nearest-neighbor classification model in which classifies new cases based on a similarity measure (e.g., distance functions). Because KNN classifier stores training data, the model helps to compute substitution predictions. In particular, KNN classifier aimed to apply binary classification in the context of spoofed fingerprint detection. Only live fingerprint images are used for training the proposed method as shown in Fig. 6.

Fig. 6. A spoof fingerprint detection is trained by KNN classifier to distinguish samples of bona fide fingerprints from samples of known spoof ones.

Fig. 5. The framework of the proposed method.

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

6

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx

4. Experiment results

the influence of modifying learning rate and weight decay parameters in the proposed method.

4.1. Datasets 4.4. Self comparison In this section, we used the state of art LivDet 2datasets: LivDet 2013 [53], and LivDet 2015 for fingerprints spoofing benchmark [20,25] which are considered in the proposed method. The LivDet 2013 Contains two type of images: genuine images and spoofs with total of 16.000 images and equally distributed as 50% of the images were used as a training set and the remaining 50% as the test set for classification. Hence, the real images captured by four fingerprint scanners: Biometrika, Crossmatch, ItalData and Swipe, Spoofs are created using a gelatin materials like latex, play-doh and wood glue. LivDet 2013 utilized for use of the non-cooperative method without user interference for creating spoof images. We used only Biometrika and ItalData due to high resolution of their images. The second dataset is livDet 2015 which contains a set of images were captured by four optical scanners; these scanners are Digital Persona, Green Bit, Biometrika and Crossmatch. It noticed that the testing set has spoof images which forged using unknown materials not saved in the training set. Only gelatin material in the Crossmatch were considered in our experiments as shown in Table 1. Artificial fingerprints in Biometrika and Italdata have been created without user assistance, while fake ones in CrossMatch were created by user cooperation. Several materials for creating the synthetic fingerprints were applied, such as: gelatin, wood glue, silicone and latex. More concisely, experimental evaluation of the proposed method is applied on a collection of a large and challenging LivDet 2013 and 2015 dataset. In LivDet 2013, over 1000 live attempts were collected as well as, 1000 spoof fingers from various materials such as Play-Doh, Gelatin and Ecoflex. In LiveDet 2015, it is about 2011 training attempts have conducted on 1010 live fingers with a resolution of 1000 dpi from 51 subjects depending on the sensor, 2 fingerprints each of all 10 fingers and 1001 spoofed fingerprints across 5 spoof materials to produce nearly 200 fingerprint images per spoof material. 500 spoofed fingers made from each of 5 fingerprints of 20 subjects for each of the five spoof materials. Only two attempts applied on each spoof.

4.2. Experimental setup To achieve a perception of these three fingerprint benchmarks, various parameters in our method have been adjusted to give a good results as shown in Table 2. The implementation of the proposed method was applied on Intel (R) Xeon (R) CPU E5-2690 v2 (3.00 GHz processor) with 20 GB RAM and NVIDIA GPU uses Matlab 2016a.

4.3. Parameters evaluation To do training and testing samples of fake fingerprints in the dataset, we have tried various values of specific parameters: patch numbers, learning rate and weight decay respectively on our experiments. Appropriate values for these parameters are chosen. As shown in Fig. 7, Using the 10 ROIs from the original input fingerprint in the training DBM model, significantly makes the proposed method robust against missed or damaged ROIs of the input images. Furthermore, it gives a high accuracy detection rate when the size of the trained data is suitable in the dataset. The 10 patches of input image show that, the size of trained data is increased to achieve about 0.95 accuracy rate. Figs. 8 and 9 show 2

LivDet datasets are available to researchers at http://livdet.org/registration.php.

DRBM + DBM method is trained on three data sets for fingerprint detection. We use two groups of patches generated from the input image, Scaled and rotated ROIs. To evaluate progress of our method, three fingerprint scanners for spoof images were regarded in our experiments. We have used the following metrics as shown in Table 3: Accuracy rate (ACC), Half total error rate (HTER), Fake Fingerprint Accuracy (FFA) and True Fingerprint Accuracy (TFA), respectively and defined as follows

ACC ¼

NT  100% NT þ NF

ð12Þ

where NT represents the number of the correct fingerprint recognized in the testing dataset and NF is the number of fake fingerprints in testing dataset.

HTER ¼

FAR þ FRR 2

ð13Þ

where FAR is a False acceptance rate and FRR is a false rejection rate. In order to assess the performance of the proposed method based on various materials for spoofing forgeries, fake fingerprint accuracy (FFA) and true fingerprint accuracy (TFA) are defined as follows:

FFA ¼

N1  100% NF

ð14Þ

TFA ¼

N2  100% NT

ð15Þ

where N1 represents the number of fake fingerprint images which are recognized falsely in NF. N2 represents the number of true fingerprint images that, are identified as true ones in NT. As pointed in Table 3, the computation cost of the proposed method for a single tested image is 45 s when we test images on Italdata dataset. Each step is given individually. Fingerprint preprocessing step for one image required 0.2 s and fingerprint deep features extraction required 10 s. The DBM training consumed 30 s and KNN classifier required 4.8 s. Similarly, the total computation time required to test one image on CrossMatch is 85 s, which is slower than Italdata and Biometrica due the large blank areas. We have tested three types of spoof materials: Wood Glue, gelatin, Play-Doh and 2D-printed ones on the proposed method. To examine the time complexity of the proposed method, two experiments are conducted. The experiments considered the number of training fingerprints as well as number of input features used in the tested fingerprint image. The computation times of the proposed DRBM + DBM model are gained for different numbers of training images. The LivDet dataset is employed in the first experiment. The number of training images is ranged from 50 to 1000. The average processing time of the proposed method is calculated against the number of training fingerprint images as shown in Fig. 10. It shows that the time complexity of the proposed method in regard to the number of training samples is O (n) based on big O (.) notation. In the second experiment for examining the time complexity of the proposed method, different number of image features is considered to calculate the computation time of the proposed as shown in Fig. 11. The proposed method used a feature vector with size 128 features due to dimensionality reduction. The running time based on 128 features is 45 s. The size of input image is 750  800. We selected ROI with size 40  40 to produce 1600 features. For the ROI, 5 rotated patches and 5 scaled patches are

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx

7

Table 1 Information of LivDet datasets 2013 and 2015 used in the proposed method. Dataset

LivDet 2013

LivDet 2015

Fingerprint scanner Scanner model Image size Resolution in dpi Training/testing for live image Training/testing for spoof images Cooperative Subject Materials used for spoofing

Biometrika FX2000 315  372 569 1000/1000 1000/1000 No Ecoflex, Gelatine, Latex, Modasil, Wood Glue

Table 2 Experimental setup. Parameter

Value

Size of input image Patch size Patch numbers GB- RBM Maximum epochs Meanfieldalgorithm Learningrate Momentum Weight decay Pooling size Size of feature vector

750  800 40  40 10 patches 864 visible, 1000 hidden neurons 500 30 rounds 0.01 0.5–0.92 10–4 13 128

ItalData ET10 640  480 500 1000/1000 1000/1000 No

CrossMatch L Scan Guardian 640  480 500 1510/1500 1473/1473 Yes Body Double, Ecoflex, Play-Doh, OOMOO, Gelatin

extracted to produce 1600  10 = 16000 features. The running time for 16,000 fatures is increased to 50 s which is slightly high which is still the future direction to reduce the time computation. As shown in Table 4, non-cooperative test was used to enhance a latent fingerprint left on a surface and printing the negative impression on a 2D sheet. This 2D printed image can then be made into a mold. Our method detects about TFA = 89% and FFA = 88%. The main difference between TFA and FFA is that TFA represents the number of fake fingerprint images which are recognized falsely. FFA represents the number of true fingerprint images that, are identified as true ones. Further, the performance of the our method gives a value of posterior probability of the bona fide fingerprints normalized in

Fig. 7. Accuracy rate of the proposed method on Cross Match dataset.

Fig. 8. Accuracy rate versus Learning rate.

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

8

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx

Fig. 9. Accuracy rate versus Weight decay.

Table 3 Performance evaluation of the proposed method on LivDet, 2013 (Biometrica and Italdata) and LivDet, 2015 (CrossMatch). Method DRBM + DBM model

ACC

HTER

96.00% 95.00% 94.50%

3.50% 6.44% 2.80%

Processing time per image in seconds (s) 66 s 85 s 45 s

Dataset Biometrica CrossMatch Italdata

the range 0–1 (0 is the degree of artifact fingerprints and 1 is bona fide ones). The threshold parameter to make decision is set to 0.3. To estimate the classification error rate in the proposed method, two type of classification error rate are used. Attack Presentation Classification Error Rate (APCER) and Bonafide Presentation Classification Error Rate (BPCER). APCER defines the rate of spoof fingerprints called live). BPCER defines the rate of live fingerprints called spoofs. The APCER and BPCER rates of our algorithms are presented in Table 5. The results from the three datasets: Biometrica, CrossMatch and ItalData, DRBM + DBM model performed the best in CrossMatch with APCER = 9.03% and BPCER = 6.4% due to high resolution of images produced by L Scan Guardian reader.

Fig. 11. Computation time of the proposed method based on different number of features per image.

4.5. State of art comparisons Some experimental results are mentioned in Table 6 to show the robustness of our method compared with others: [35,54–56]. The Average Classification Error (ACE) is used as a performance evaluation metric for robustness. It is defined as follows:

Fig. 10. Computation time of the proposed method when using different number of training fingerprints.

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

9

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx Table 4 The performance of the proposed method using LivDet 2013 with regarding four types of fabricate fingerprint materials with memory space 61.5 MB. Method

Detection of artefact presentation with Spoof Materials

DRBM + DBM model

Real TFA 0.95

Wood glue FFAfor spoof materials 0.88

Gelatin

Play-Doh

2D printed

0.90

0.90

0.88

Table 5 Classification error rates in percentage (%) over three datasets with known spoofs. Biometrica APCER 5.80%

CrossMatch BPCER 9.7%

APCER 3.8%

Table 6 Comparison table shows performance evaluation of existing methods compared with our method on LivDet 2013. Reference

Techniques used

ACE

Nogueira et al. [56] Jiang and Xin [35] Gottschlich et al. [55] Zhang et al. [54] The proposed method

Convolutional neural networks Co-occurrence matrix Histograms of gradients. Wavelet features and local binary pattern DRBM + DBM

3.9% 11.00% 6.7% 2.1% 3.6%

ACE ¼

FerrLive þ FerrFake 2

ItalData BPCER 6.4%

ð16Þ

where Ferrlive is the percentage of misclassified real fingerprints and Ferrfake is the percentage of misclassified fake fingerprints. 5. Conclusion We proposed a novel deep learning model for examining fingerprints based on DRBM and Deep Boltzmann Machine, which deals with complex texture patterns in a robust way due to its probabilistic multilayer architecture. In the proposed method, after training a DBM, such structure has employed to extract deep features of the grayscale fingerprints. KNN classifier is applied with the feature vectors of the ROIs extracted by the DBM to examine spoof forgeries. The experiment results demonstrate that the Deep learning model is robust against different kinds of spoof forgeries such as wood glue, Gelatin and Play-Doh. Deep features are extracted from the real image of grayscale and Depth visual structure such as scaled and rotated patch ROIs. The performance evaluation of the DRBM + DBM method achieved state-of-the-art results in three public fingerprint recognition benchmarks. However, our method still struggling to recognize fake fingerprints with unknown materials. For future work, we need to extend the method behavior to deal with spatio-temporal features of fingerprint images to explore the liveness properties. Furthermore, we need to explore the capability of the proposed method in reducing the time complexity of deep learning machine. Acknowledgments The authors are grateful to the Middle East University, Amman, Jordan for the financial support granted to cover the publication process of this research article. References [1] E. Marasco, A. Ross, A survey on antispoofing schemes for fingerprint recognition systems, ACM Comput. Surv. (CSUR) 47 (2) (2015) 28. [2] G. Fumera, Multimodal anti-spoofing in biometric recognition systems, in: S. Marcel, M.S. Nixon, S.Z. Li (Eds.), Handbook of Biometric Anti-Spoofing: Trusted Biometrics under Spoofing Attacks, Springer, London, 2014, pp. 165– 184.

APCER 11.6%

BPCER 9.1%

[3] C. Roberts, Biometric attack vectors and defences, Comput. Sec. 26 (1) (2007) 14–25. [4] K.W. Bowyer, Face recognition technology: security versus privacy, IEEE Technol. Soc. Mag. 23 (1) (2004) 9–19. [5] D. Menotti et al., Deep representations for iris, face, and fingerprint spoofing detection, IEEE Trans. Inform. Forensics Sec. 10 (4) (2015) 864–879. [6] M. Sajjad et al., CNN-based anti-spoofing two-tier multi-factor authentication system, Pattern Recogn. Lett. (2018). [7] L. Ghiani et al., Livdet 2013 fingerprint liveness detection competition 2013, Biometrics (ICB), 2013 International Conference on, IEEE, 2013. [8] Q. Huang et al., An evaluation of fake fingerprint databases utilizing SVM classification, Pattern Recogn. Lett. 60 (2015) 1–7. [9] T. Chugh, K. Cao, A.K. Jain, Fingerprint spoof detection using minutiae-based local patches, Biometrics (IJCB), 2017 IEEE International Joint Conference on, IEEE, 2017. [10] R.K. Dubey, J. Goh, V.L. Thing, Fingerprint liveness detection from single image using low-level features and shape analysis, IEEE Trans. Inf. Forensics Secur. 11 (7) (2016) 1461–1475. [11] C. Wang et al., A DCNN based fingerprint liveness detection algorithm with voting strategy, Chinese Conference on Biometric Recognition, Springer, 2015. [12] Y. Gao et al., Intermediate spoofing strategies and countermeasures, Tsinghua Sci. Technol. 18 (6) (2013) 599–605. [13] D. Baldisserra et al., Fake fingerprint detection by odor analysis, International Conference on Biometrics, Springer, 2006. [14] Z. Akhtar et al., Evaluation of serial and parallel multibiometric systems under spoofing attacks, IEEE Fifth International Conference: Biometrics: Theory, Applications and Systems (BTAS), IEEE, 2012. [15] J.J. Engelsma, K. Cao, A.K. Jain, Raspireader: open source fingerprint reader, IEEE Trans. Pattern Anal. Mach. Intell. (2018). [16] J.-G. Wang et al., Person recognition by fusing palmprint and palm vein images based on ‘‘Laplacianpalm” representation, Pattern Recogn. 41 (5) (2008) 1514– 1527. [17] M. Espinoza, C. Champod, P. Margot, Vulnerabilities of fingerprint reader to fake fingerprints attacks, Forensic Sci. Int. 204 (1–3) (2011) 41–49. [18] M. Drahansky´, W. Funk, R. Nötzel, Liveness detection based on fine movements of the fingertip surface, 2006, IEEE Information Assurance Workshop, 2006. [19] C.J. Lennard et al., Fingerprints and Other Ridge Skin Impressions, CRC Press, 2016. [20] L. Ghiani et al., Review of the fingerprint liveness detection (LivDet) competition series: 2009 to 2015, Image Vis. Comput. 58 (2017) 110–128. [21] D. Yambay, Review of fingerprint presentation attack detection competitions, Springer International Publishing, Cham, 2019, pp. 109–131. [22] Y. Park, U. Jang, E.C. Lee, Statistical anti-spoofing method for fingerprint recognition, Soft. Comput. 22 (13) (2018) 4175–4184. [23] J.B. Kho et al., An incremental learning method for spoof fingerprint detection, Expert Syst. Appl. 116 (2019) 52–64. [24] C. Sousedik, C. Busch, Presentation attack detection methods for fingerprint recognition systems: a survey, IET Biom. 3 (4) (2014) 219–233. [25] V. Mura et al., LivDet 2017 fingerprint liveness detection competition 2017, International Conference on Biometrics (ICB), IEEE, 2018. [26] IEC, I.S., ‘‘ISO/IEC 30107-3:2016, InformationTechnology-Biometric Presentation Attack Detection-Part 1: Framework”, in ISO/IEC 30107-1:2016 2016. [27] H.-U. Jang et al., Fingerprint spoof detection using contrast enhancement and convolutional neural networks, International Conference on Information Science and Applications, Springer, 2017. [28] I. Goicoechea-Telleria et al., Presentation attack detection evaluation on mobile devices: simplest approach for capturing and lifting a latent fingerprint, International Carnahan Conference on Security Technology (ICCST), IEEE, 2018. [29] K. Cao, A.K. Jain, Hacking mobile phones using 2D printed fingerprints. Technical Report. MSU-CSE-16-2, 2016. [30] A. Rattani, W.J. Scheirer, A. Ross, Open set fingerprint spoof detection across novel fabrication materials, IEEE Trans. Inf. Forensics Secur. 10 (11) (2015) 2447–2460.

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005

10

D.M. Uliyan et al. / Engineering Science and Technology, an International Journal xxx (xxxx) xxx

[31] Z. Xu, S. Li, W. Deng, Learning temporal features using LSTM-CNN architecture for face anti-spoofing, Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition (ACPR), IEEE, 2015. [32] Rattani, Ross, Automatic adaptation of fingerprint liveness detector to new spoof materials, IEEE International Joint Conference on Biometrics, IEEE, 2014. [33] Xia, Z., et al., A novel weber local binary descriptor for fingerprint liveness detection, In IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2018. [34] P.D. Lapsley, et al., Anti-fraud biometric scanner that accurately detects blood flow. 1998, Google Patents. [35] Y. Jiang, X. Liu, Spoof fingerprint detection based on co-occurrence matrix, Int. J. Signal Process. Image Process. Pattern Recogn. 8 (8) (2015) 373–384. [36] L. Ghiani, G.L. Marcialis, F. Roli, Fingerprint liveness detection by local phase quantization, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), IEEE, 2012. [37] A. Abhyankar, S. Schuckers, Fingerprint liveness detection using local ridge frequencies and multiresolution texture analysis techniques, International Conference on Image Processing, IEEE, 2006. [38] S.B. Nikam, S. Agarwal, Fingerprint liveness detection using curvelet energy and co-occurrence signatures, Proceedings of the fifth international conference on computer graphics, imaging and visualisation, IEEE, 2008. [39] P. Coli, G.L. Marcialis, F. Roli. Power spectrum-based fingerprint vitality detection, in 2007 IEEE Workshop on Automatic Identification Advanced Technologies, 2007. IEEE. [40] T. de Freitas Pereira et al., LBP TOP based countermeasure against face spoofing attacks, Asian Conference on Computer Vision, Springer, 2012. [41] Z. Akhtar et al., Evaluation of multimodal biometric score fusion rules under spoof attacks, Proceedings of the 5th IAPR International Conference on Biometrics (ICB), IEEE, 2012. [42] J. Galbally et al., A high performance fingerprint liveness detection method based on quality related features, Future Gener. Comput. Syst. 28 (1) (2012) 311–321.

[43] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems. 2012. [44] T. Chugh, K. Cao, A.K. Jain, Fingerprint spoof buster: use of minutiae-centered patches, IEEE Trans. Inf. Forensics Secur. 13 (9) (2018) 2190–2202. [45] D.H. Ackley, G.E. Hinton, T.J. Sejnowski, A learning algorithm for Boltzmann machines, Cognit. Sci. 9 (1) (1985) 147–169. [46] G.E. Hinton, R.R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313 (5786) (2006) 504–507. [47] P. Smolensky, Information processing in dynamical systems: Foundations of harmony theory, 1986, Univ at boulder dept of computer science colorado. [48] V. Nair, G.E. Hinton. Implicit mixtures of restricted Boltzmann machines, in Advances in Neural Information Processing Systems, 2009. [49] G.E. Hinton, S. Osindero, Y.-W. Teh, A fast learning algorithm for deep belief nets, Neural Comput. 18 (7) (2006) 1527–1554. [50] M. Welling, G.E. Hinton, A new learning algorithm for mean field Boltzmann machines, International Conference on Artificial Neural Networks, Springer, 2002. [51] H. Yu, J. Yang, A direct LDA algorithm for high-dimensional data-with application to face recognition, Pattern Recogn. 34 (10) (2001) 2067–2070. [52] M. Muja, D.G. Lowe, Scalable nearest neighbor algorithms for high dimensional data, IEEE Trans. Pattern Anal. Mach. Intell. 11 (2014) 2227–2240. [53] L. Ghiani et al., Livdet 2013 fingerprint liveness detection competition 2013, International Conference on Biometrics (ICB), IEEE, 2013. [54] Y. Zhang et al., Fake fingerprint detection based on wavelet analysis and local binary pattern, Chinese Conference on Biometric Recognition, Springer, 2014. [55] C. Gottschlich et al., Fingerprint liveness detection based on histograms of invariant gradients, Biometrics (IJCB), 2014 IEEE International Joint Conference, Citeseer, 2014. [56] R.F. Nogueira, R. de Alencar Lotufo, R.C. Machado, Evaluating software-based fingerprint liveness detection using convolutional networks and local binary patterns, Biometric Measurements and Systems for Security and Medical Applications (BIOMS) Proceedings, 2014 IEEE Workshop, IEEE, 2014.

Please cite this article as: D. M. Uliyan, S. Sadeghi and H. A. Jalab, Anti-spoofing method for fingerprint recognition using patch based deep learning machine, Engineering Science and Technology, an International Journal, https://doi.org/10.1016/j.jestch.2019.06.005