- Email: [email protected]
Area bound dynamic time warping based fast and accurate person authentication using a biometric pen
Digital Signal Processing 23 (2013) 259–267
Contents lists available at SciVerse ScienceDirect
Digital Signal Processing www.elsevier.com/locate/dsp
Area bound dynamic time warping based fast and accurate person authentication using a biometric pen Muzaffar Bashir a,b,∗ , Jürgen Kempf a a b
University of Applied Sciences Regensburg, Germany University of the Punjab, Lahore, Pakistan
a r t i c l e
i n f o
Article history: Available online 30 August 2012 Keywords: Biometric person identification Biometric signature verification Handwritten PIN recognition Area bound DTW Time series
a b s t r a c t The paper presents a modified dynamic time warping (DTW) technique for person authentication based on time series matching obtained from handwriting. The online data has been acquired by a biometric smart pen device. The proposed method allows fast and accurate classification of human individuals based on handwritten PIN words or signature samples. Although classic DTW provides robust distance measurements essential for accurate classification of sequences, it is computationally expensive. To speed up computations we introduce area bound dynamic time warping (AB_DTW) that divides time series into several areas bounded by segments of consecutive zero crossings including local peaks and valleys. Unlike classic DTW which compares whole signals, the proposed AB_DTW warps areas bounded by the local regions. Two kinds of data abstraction formats of area bound—1 dimensional and 2 dimensional— are evaluated. Experimental results show that because of a higher-level data abstraction, the proposed approach is several times faster than classic DTW. Moreover, AB_DTW does not offer substantial loss of accuracy which is required for authentication performance using handwritten PIN words and signatures sampled by biometric pen device. © 2012 Elsevier Inc. All rights reserved.
1. Introduction One of the widely researched behavior-biometrics is person authentication using handwritten signatures. Online person authentication using a handwritten signature is promising because of long history of use and wide acceptance of traditional signature verification in public domain [1–3]. A biometric pen device is generally used to acquire the dynamics of handwriting for online data. There exist two approaches to look at similarity or dissimilarity of sequences obtained from handwriting. They are called functional and parametric approaches [1,2]. In the functional approach, the feature set is essentially obtained from the complete signal in terms of time series, thus retaining more information on the signing process. On the other hand, in the parametric approach, the features or parameters are selected as a higher-level abstraction of original data. Therefore, the latter approach though produces result much faster but poses difficulty when it comes to the selection of features that can contribute to the classification effectively [3]. The functional approach of complete signal comparisons gives better results [1,2]. Dynamic time warping (DTW) technique has successfully been applied in this regard in recent years [1,2,4,5]. DTW is used to determine the distance of two sequences by minimizing
*
Corresponding author at: University of the Punjab, Lahore, Pakistan. E-mail addresses: [email protected] (M. Bashir), [email protected] (J. Kempf). 1051-2004/$ – see front matter © 2012 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.dsp.2012.08.013
the intra-individual variations. In DTW, the natural nonlinear time variations of sequences that subsist even for genuine signatures are reduced before the actual distance is calculated. But classic DTW is computationally expensive. To speed up computations, there are some modified techniques which operate on a higher-level abstraction of the data as piecewise aggregate approximation (PAA) of time series or data down-sampling [6–8]. In some other methods, the numbers of data point comparisons are reduced as in extreme points warping or segment to segment matching and lower bounding techniques [1,2,9,10]. Moreover, commonly used constraints like Sakoe–Chiba bands or Itakura parallelograms are applied because they limit the allowed warp search path in the cost matrix resulting in a speed up of DTW [9]. This study presents a local data abstraction based DTW approach named Area Bound Dynamic Time Warping (AB_DTW). It warps only the areas bounded by the local regions of sequences obtained from handwriting. The speed up of computations is achieved by taking a time series as a vector of several areas bounded by the segments of consecutive zero crossings (ZCs) which may include positive peak(s) or negative peak(s). Using AB_DTW, biometric person authentication based on time series generated by handwritten personal identification number (PIN) words and signatures is presented. The acquisition of the online data is carried out with the help of a biometrics smart pen (BiSP) device [11]. The work presented here is mainly inspired by the work of [1,2,4,7].
260
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Fig. 1. (a) An example of pressure signal obtained from handwritten PIN “4A5B7EM” and (b) smoothed and normalized pressure signal is shown. The segmentation points are marked with ‘’.
The paper is organized as follows. In Section 2, the BiSP device used for data acquisition and pre-processing of data is briefly described. Section 3 describes DTW based classifier and related work. Section 4 outlines the concept of speeding up of DTW using area bound approximation of the time series. Then in Section 5 results of the experimental work for biometric person authentication are discussed. Section 6 finally summarizes the major findings and outlook. 2. BiSP device for data acquisition and data preprocessing The biometric smart pen (BiSP) used in the study is a ballpoint pen that allows the record and analysis of handwriting, drawing and gesture movements on a paper pad or free in air [11]. For a comprehensive assessment of pen movement and fine motor features of hand and fingers, the device is equipped with a diversity of sensors measuring the acceleration and tilt angle of the pen, the grip forces of the fingers holding the pen, and the forces and vibrations generated in the refill during writing or drawing on a pad. The captured handwriting movements can be represented by the time series of five sensor channels as (i) across horizontal pressure, (ii) grip finger-grip pressure, (iii) z vertical pressure, (iv) α longitudinal and (v) β vertical angles of the pen. All of them are measured as a function of time. The across sensor measures the change of forces resulting from movements of the pen tip along the x– y dimension and generates a single time series signal of the pressure along the x– y dimension due to handwriting. The across sensor is implemented with the help of a piezoelectric polymer film placed close to the front part of the pen (pen tip). The time series obtained from the across sensor of the pen is used in this work. The description of the device is necessarily shortened, for more information the reader is referred to [8,12]. In order to eliminate the potential sensor noise, the data is pre-processed after acquisition. The essential pre-processing steps are smoothing, segmentation, normalization and down-sampling of
data without loss of valuable information. A typical example of a time series (across pressure signal) recorded with BiSP during handwriting a PIN word on a paper pad is shown in Fig. 1(a). While Fig. 1(b) represents the smoothed and normalized signal. A straight-line fit linear-trend is removed using “detrend()” function of MATLAB [13]. Smoothing of data based on local regression is done in order to minimize sensor noise [14]. Each signature or PIN data sample was captured separately during acquisition. Therefore, segmented data was captured in [14,15]. In order to be able to have accurate ZC points at both ends of the signal, additional segmentation is performed here. The baselines at both ends of the signal are removed using segmentation points (SPs) as shown in Fig. 1(b). The handwriting signal part between two ZC index points (or SPs) is extracted from the original time series using peak and zerocrossing determination algorithm. The two SPs are determined on the original time series by searching first ZC points and closest first peaks while the time series is scanned at both ends respectively. The flow of steps in the segmentation algorithm is shown in Table 1. In order to compensate large variations in amplitude values, the data is normalized to [−1, 1]. Further, in order to reduce complexity of classic DTW based classifier when applied to original sequences, the data is re-sampled to a lower sampling rate (data down-sampling). Data pre-processing steps can be viewed in detail in [8,14]. 3. DTW based classifier and related work For person authentication, dynamic time warping (DTW) algorithm was applied to the time series obtained from handwriting sequence data. DTW based classifier is used to determine the distance of two sequences {( Q = q1 , q2 , . . . , qn ) & (C = c 1 , c 2 , . . . , cm )} of even different lengths. In DTW, nonlinear time variations of sequences even for genuine signatures are minimized before the actual distance is calculated. For the alignment of two sequences,
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
261
Table 1 Segmentation using peaks and zero-crossings. Step Step Step Step Step Step Step Step Step Step Step Step
1 2 3 4 5 6 7 8 9 10 11 12
Divide original signal into two signals. signal1 represents the first half and signal2 is the reversal of the second half of original Find threshold1 using 1/4th of STD of signal1 Find values and index points of peaks (local maxima and minima) threshold1 Find index value of first peak PKID Find threshold2 for zero-crossing using MEAN of signal1 Find zero-crossing index values ZC1 using threshold2 Find zero-crossing index values ZC2 < PKID If ZC2 is zero then take negative of the signal1 , i.e., signal1 = signal1 ∗ (−1) and go to step 2, else go to next step Find last index of ZC2 as a first point of segmentation seg1 Repeat step 2 to step 9 for signal2 Find last point of segmentation seg2 using seg1 of signal2 by using expression: seg2 = length(signal) − seg1 Find segmented signal using index points seg1 of signal1 and seg2
Case I: 1D Time index= (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14) R = (0, −1, −2, −1, −1, 0, 1, 1, 2, 3, 2, 1, 1, 0) n = | R | = 14 zc = (1, 6, 14) S 1 = (0, −1, −2, −1, −1, 0), S 2 = (0, 1, 1, 2, 3, 2, 1, 1, 0) R¯ =
6 1
s1 dt ,
6
1
Case II:2D R¯ =
= (−0.833, 1.222) N = | R¯ | = 2
s2 dt
9
1
14 6
3
10
−0.833 1.222
Case II:2DN 0.300 1 R¯ = −0.833 1.222
N = | R¯ | = 2 N = | R¯ | = 2
Fig. 2. Time series R of 14 data points and three zero-crossing points (zc) marks at time indices (1, 6 and 14). The time series is divided into two segments due to zc. The area under the curve is calculated. The area values represent the area bound representation due to three cases as described below the curves.
the squared Euclidean distance d(q i , c j ) = (qi − c j )2 is calculated under global and local constraints. A local constraint or cumulative distance function D (, ) is determined which minimizes the current cell and neighboring cell elements distance. If d(q i , c j ) is the distance found in the current cell, then two common cumulative distance cost functions are as shown in Eq. (1) and Eq. (2).
D (i , j ) = d(qi , c j )
+ min D (i − 1, j − 1), D (i − 1, j ), D (i , j − 1) ,
(1)
D (i , j ) = d(qi , c j ) + min D (i − 1, j − 1), D (i − 1, j − 2),
D ( i − 2, j − 1) .
(2)
DTW with the cumulative cost function as described in Eq. (2) is used in this study because it has shown better performance of sequence matching in preliminary results. DTW can be viewed in detail in [7,9,16]. In DTW based classifier, for a query sequence, the distances to all reference sequences are calculated. A list of nearest reference sequences is determined and the minimum distance of the best match decides for classification (see [15]). Classic DTW has computing time and memory space problems, especially in online handwriting or person recognition. The remedy has somehow been provided by some modified techniques, which operate on a higher-level abstraction of the data as piecewise aggregate approximation PAA of time series or down-sampling of data [6–8]. In some other methods, the numbers of data point comparisons
262
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Fig. 3. Data points are shown in area bound representation as “−0.833” & “1.222” in 1D case, “(3, −0.833)” & “(10, 1.22)” in 2D case and “(0.3, −0.833)” & “(1, 1.222)” in 2DN case.
are reduced as in extreme points warping or segment-to-segment matching and lower bounding techniques [1,2,9,10]. Additionally, to speed up DTW, commonly used constraints like Sakoe–Chiba bands or Itakura parallelograms, which limit the allowed warp search path in the cost matrix, are used to reduce computational time [9]. 4. Area bound warping method This study presents area bound dynamic time warping (AB_DTW) method, which warps only the areas bounded by the local regions of the time series. In [7], a time series is abstracted into piecewise constant approximations using mean values of data in equi-sized frames. This kind of data reduction strongly depends on the size and number of frames on which data is subjected. In our approach, we have chosen automatic dynamic frame sizes using consecutive ZCs and the curves (segments) in the frames are abstracted by corresponding areas as shown in Fig. 2. The objective of our work is data reduction, which has been achieved successfully through an approach based on local data reduction. The curve in Fig. 2(a) represents a time series R of fourteen (n) points and three zero-crossing (ZC) points marked at the indices 1, 6 and 14 respectively. In Fig. 2(b), the time series is divided into two (N = zc − 1) unequal frames. A vector of normalized area values calculated for the segments in the frames becomes the area bound approximation (ABA). Equations below the curves demonstrate the calculation performed for the segments S 1 and S 2 . The area is calculated by trapezoidal rule. The speed up of computations is achieved by higher level reduced representation of time series. A time series is converted into a vector of several areas bounded by the segments of consecutive zero crossings. A segment may include a ZC, the positive peak(s) and a ZC or like wise a ZC, the negative peak(s) and a ZC. Two cases of area bound representations—(I) one dimensional and (II) two dimensional—are presented.
One dimensional case (1D) In this case, a time series consisting of 14 points is projected into 1 dimension. The time series is divided into two frames (segments) due to three ZC points. The area under the curve in each frame is calculated. Therefore, a time series consisting of 14 points is represented by two data points (two areas) in a vector (array) known as reduced time series in 1D case. As shown in Fig. 2, three ZC occur at three time index values (1, 6 and 14). The areas are calculated for segments from time index 1 to 6 and 6 to 14, respectively. This therefore gives two data points (area values “−0.833” and “1.222”) stored in a vector (array) in 1D case. Two dimensional case (2D) Similarly in 2D case, the time series consisting of 14 points is projected into 2 dimensions. Now, a single area is represented by two data points: (i) calculated area value and (ii) time index value. Therefore, the time series is converted into a 2 dimensional vector (matrix). Hence area values “−0.833” and “1.222” are stored in the first column of the matrix (first dimension) while mean time index values (“3.5 ≈ 3” and “10”) are stored in the second column of the matrix (second dimension) respectively. The mean of time index value is rounded to integer. Note in the normalized 2D case denoted by “2DN”, the obtained time index values are normalized to [0, 1] as illustrated in Fig. 2 and Fig. 3. Note: more precisely, a vector in 1D case corresponds to n points (areas) in one dimensional space represented by a vector (array): a(k − 1). While 2 dimensional vector (matrix) represents area and time values with coordinates (a, t ) where “a” measures the area under the curve within a frame (segment) bounded by (t i , t j ) and t is given by(t i + t j )/2. The term k denotes the number of zero crossings in the sequence. For demonstration, data points in area bound representation are shown in Fig. 3. It is observed that area values remain constant though the values of abscissa change with respect to 2D and 2DN cases, as evident from Fig. 3.
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
263
Table 2 Area bound calculation using peaks and zero-crossings. Step Step Step Step Step Step Step Step Step Step Step Step Step Step
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Define threshold1 for peaks detection Define threshold2 for zero crossing determination Find values and index points of peaks (local maxima and minima) threshold1 Find an array ZC of zero-crossing index values using threshold2 Initialize first zero-crossing index point out of ZC array Initialize an array index ind Begin loop (until end of ZC array) Select next zero-crossing index point out of ZC array Find numbers of peaks PK between first and next zero-crossing index If PK is zero go to step 8, else go to next step Find segment of signal from first to next zero-crossing index points Calculate area enclosed by the segment and find normalized area value (divide area by the length of the segment) Put area value into an array A using index ind If two dimensional case then put the mean of first and next into an array of time T using index ind, else go to next step
Fig. 4. Two time series of genuine signatures are shown. The A ± are shown in different shading above and below zero-crossing. The representation of time series in bounded area reduced form is shown with marker ‘’ and zero crossings are shown with marker ‘◦’.
The AB_DTW warping process involves: determination of positive peaks (PK+) and negative peaks (PK−), zero-crossing (ZC) detection, bounded areas calculation and bounded area matching. The extreme points of peaks and valleys (local maxima and minima) are treated as PK+ and PK− respectively. The ZC points are defined as the points where the signal crosses zero line (or a predefined value) while signal values are changing with respect to time: (a) From baseline towards zero. (b) From peak (PK+) towards zero. (c) From peak (PK−) towards zero. 4.1. Bounded areas calculation A time series R of length n is given by
R = r 1 , r 2 , r 3 , . . . , rn .
(3)
If the points zc 1 , zc 2 , zc 3 , . . . , zck are zero crossing points such that zc = k, then the number of segments N in the original sequence R will be zc − 1 (i.e., N = k − 1). Now the time series R¯ in ABA is given by:
R¯ =
zc2
1
|1 + zc 2 − zc 1 |
s1 dt , zc 1
1
|1 + zck − zck−1 |
zck
1
|1 + zc 3 − zc 2 |
sk−1 dt .
zc3 s2 dt , . . . , zc 2
(4)
zck−1
Each calculated area is divided by the number of data points in the segment. Note: there should be at least one segment and two zero crossings in a sequence, i.e. | zc | 2 or N 1. The area bounded by the local curve of the signal is treated as A + or A −. A + is area bounded by all points of sub-sequence (above ZC) between consecutive ZC points including peaks (PK +). Similarly, the area A − includes peaks (PK −). There could be one ZC point contributing to two consecutive areas. An area can contain one or more peaks and is determined for a segment with all points enclosed by the two ZC points. Very small ripples are not considered as peaks (PK ±). Therefore, the area ( A ±) below a certain threshold value is considered to be a result of ripples or noise called “RN area”. The latter can be adjusted with the adjacent areas. Moreover, the area contribution of a segment with two consecutive ZC is also adjusted when it does not contain any
264
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Fig. 5. Good examples of reproducible signals and area bound representations. The alignment between two sequences as discovered by AB_DTW and area match for comparison of A − (dotted lines) and A + (solid lines) are shown.
peak (PK + or PK −) lie within a threshold. Further, an area though very small may not be neglected if the area contributions from the previous and subsequent area are greater than a threshold value. The areas ( A ±) are calculated by trapezoidal function provided by MATLAB [13]. The flow of steps for area bound calculation based on peaks and zero-crossing detection algorithm is shown in Table 2. For illustration, two time series of genuine signatures are shown in Fig. 4 and the A ± are shown in different shades, above and below zero crossing line. The corresponding vectors of normalized areas of consecutive ZC are shown as area sequences marked with ‘’ and dotted lines. The ZC points are marked with ‘◦’. A time series of about 2200 data points (Fig. 4 top) is reduced to a vector of 18 area bound data points, i.e., | zc | = 19. 4.2. Bounded area matching Fig. 5 shows an example of two genuine signatures. Equal number of bounded areas can be observed in the both sequences. It is of course due to equal number of ZCs found in the both sequences. AB_DTW match for sequence comparison thus reduces to oneto-one area match or simple Euclidean distance calculation. For illustration, the match of A + to A + and of A − to A − is shown with solid lines and dotted lines respectively. Moreover, the signal shown at the bottom (Fig. 5) is shifted to a different baseline to increase visibility. There subsist some variations in the sequences even for two similar signatures. Therefore, the Euclidean distance or one-to-one bounded area match is not always a suitable choice. This could be due to missing or extra areas resulting from extra or missing peaks and ZC points or also due to sensor noises (see results). Therefore, for reduced time series of bounded areas, DTW algorithm is used to perform the similarity match of two sequences. Fig. 6 shows an example of possible AB_DTW match for two sequences obtained from two genuine signatures. Due to unequal numbers of zero crossings and/or extra or missing peaks, there are different numbers of bounded areas calculated for the two sequences. The possible area bound warping of the two sequences is shown. The fifth calculated area of the first signal (top) essentially contains missing peaks and can be warped to three areas of the second sig-
nal (bottom) for matching. Similarly, the 12th and 13th areas of first signal may come from extra peaks and zero crossings, which do not occur in the bottom signal. 5. Experiments and results The main objective of the experiments was the classification of handwriting signals obtained from two datasets of handwritten sequences essential for person authentication. Fifty writers contributed to the two databases consisting of 500 signatures and 500 PIN words samples. Each writer donated 10 genuine signatures and PIN samples in a single session. PIN word is a sequence of different seven single characters written by the owner of the PIN (e.g. “4A5B7EM”). Each writer is assigned a unique PIN. The acquisition of online data is carried out with the help of biometric smart pen device. Leave-one-out cross validation (LOOCV) is used for classification of each dataset independently. The focus is to investigate the effect of data abstraction and to compare accuracy as well as runtime of classic DTW and proposed AB_DTW methods. In order to reduce the complexity of classic DTW based classifier, proposed AB_DTW method is applied which operates on reduced representation of time series in 1D and 2D forms. The minimum DTW distance determines the accuracy of match while the number of data points of the two sequences to be compared indirectly determines the computational complexity. Several experiments have been performed in order to evaluate the performance of the proposed method. (1) Experiment 1: Classic DTW method is applied to classify two signatures or PIN sequences in such a way that a longer sequence is normalized to the shorter one in time domain. (2) Experiment 2: The proposed AB_DTW method is applied to classify PIN and signature time series datasets separately. (3) Experiment 3: In order to find the effect of DTW on area bound representation of time series, simple Euclidean distance is also measured to test the accuracy of classification. In other words, the proposed area bound representation of time series data is classified using direct point-to-point data comparisons. This method is denoted by AB_Euclidean.
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
265
Fig. 6. The example of two genuine signature signals and area bound representations carrying different number of areas is shown. Possible AB_DTW for sequence comparison is shown.
(4) Experiment 4: The proposed AB_DTW method is applied to 2D area bound representation of the time series denoted by “AB_DTW2D”. The area bound representation of reference and sample sequences can be expressed as two dimensional data q(ai , t i ) and c (ai , t i ) respectively, where ai is the value of calculated area and t i is the averaged time index. For simplicity, we denote an element of 2D vector (matrix) by two subscripts (i.e. qi ,k ). So, the local distance d( , ) [17] in 2D case is given by
d(qi ,k , c j ,k )2 =
2 (qi ,k − c j ,k )2 .
(5)
k =1
(5) Experiment 5: The proposed AB_DTW method is applied to 2D area bound representation as we did in experiment 4 such that the time index values are normalized to [0, 1]. This method is denoted by “AB_DTW2DN”. Note: Normalized area values are very small while time index values can be very large (see Fig. 4). Further, there could be variations in time index values of two similar sequences obtained from the same writer. Therefore, in order to minimize the effect of such variations, the time index values are normalized to [0, 1] (see Section 3). In order to compare the performance of classification of PIN and signature sequences quantitatively, the accuracy in terms of parameters such as score of recognition SR [15], the runtime and the area under the curve of receiver operating characteristic AUC–ROC [15] is determined. As no skilled forgeries are used, false acceptance rate (FAR) represents the random PIN word or signature acceptance in ROC curve. The performance parameters based on the experiments are shown in Table 3 and Table 4. The values shown are for person authentication using handwritten PIN words and signatures respectively. The SR values of classic DTW and AB_DTW are similar (difference is ≈ 0.02). Similarly the AUC–ROC values in case of classic
Table 3 Performance parameters for person authentication using handwritten PIN. The values of SR, AUC–ROC and runtime are averaged over all writers. Experiment
Score SR
AUC–ROC
Mean time (s)
DTWp AB_DTWp AB_Euclideanp AB_DTW2Dp AB_DTW2DNp
99.995 99.971 91.126 90.971 99.975
0.9996 0.9979 0.8634 0.8456 0.9983
12.5200 0.3100 0.0243 0.3738 0.3505
Table 4 Performance parameters for person authentication using handwritten signature. The values of SR, AUC–ROC and runtime are averaged over all writers. Experiment
Score SR
AUC–ROC
Mean time (s)
DTWs AB_DTWs AB_Euclideans AB_DTW2Ds AB_DTW2DNs
99.800 99.122 86.440 96.738 99.383
0.9961 0.9861 0.8425 0.9155 0.9810
6.1800 0.2720 0.0244 0.3349 0.3040
DTW and AB_DTW are approximately same (difference is ≈ 0.002 see Table 3) as especially evident for handwritten PIN. The SR and AUC–ROC values of DTW and AB_DTW show essentially same high accuracy but AB_DTWp is faster by a factor of ≈ 42 and AB_DTWs is faster by a factor of ≈ 23 for handwritten PIN and signature data respectively, as shown in Table 3 and Table 4. The performance parameters for AB_DTW in two different forms of the 2D cases are also shown in Table 3 and Table 4. The SR and AUC–ROC values are relatively low in experiment 4 (i.e., AB_DTW2D). This is because of large variations in time index values which may exist even in two similar sequences. However, with normalized time index values the increase in accuracy is ≈ 9% in AB_DTW2DNp and ≈ 2.6% in AB_DTW2DNs . Further, the accuracy of AB_DTW and AB_DTW2DN is similar (difference is very small see Table 3 and Table 4). Although AB_Euclidean can quickly classify sequences, its performance is relatively very poor in comparison to proposed
266
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Table 5 The Mean, Maximum and Minimum numbers of data points used in different experiments for handwritten PIN. Number of data points
Original Segment DTWp AB_DTWp
Mean
Maximum
Minimum
4212.50 3646.80 364.00 34.35
6550 6086 608 42
2510 2152 215 26
Table 6 The Mean, Maximum and Minimum numbers of data points used in different experiments for handwritten signatures. Number of data points
Original Segment DTWs AB_DTWs
Mean
Maximum
Minimum
2831.20 2234.30 223.43 28.50
4299.60 3857.00 385.70 55.00
1679.4 1047.0 104.7 12.5
AB_DTW that essentially shows the advantage of DTW algorithm on proposed area bound representation of time series. The time complexity of classic DTW is O (mn) [7] with m = n, where m and n are the length of two sequences. Similarly, the time complexity of proposed AB_DTW is O ( M N ) with M = N, where M and N are the length of two sequences in proposed area bound approximation (ABA). The runtime values shown in Table 3 and Table 4 also depend on running environment, therefore we choose to show computation reduction in terms of reduction in the number of data points in regard of different experiments. In order to reduce the computational time of classic DTW for original sequences, segmentation of data based on peak and zerocrossings detection (see Section 3) was performed. Further, data is down-sampled by a factor of 10. The Mean, Maximum and Minimum numbers of data points (averaged over 50 writers) are shown for different experiments in Table 5 and Table 6. For PIN word data, the Mean number of data point is reduced from 4212.5 to 364 in DTW (i.e., 8.6% of original data). Similarly, the Mean number of data point in AB_DTW is 34.35 (0.82% of original data). For signature data (Table 6), the original Mean number of data points is reduced from 2831.2 to 223.4 in DTW (8.0% of original data). While, the Mean number of data points in AB_DTW is 28.50 (1.0% of original data). Similar effects of data point reduction in case of Maximum and Minimum for both datasets are observed for classic DTW and proposed AB_DTW as shown in Table 5 and Table 6. The averaged Minimum number of data points in signature dataset is less than that of PIN dataset indicates that some writers have very small signature while the handwritten PIN words are restricted to be 7 single characters. Finally, the proposed area bound approximation of time series allows massive data reduction. 6. Conclusions In this paper, we proposed area bound dynamic time warping (AB_DTW) for fast and accurate person identification using two datasets of handwritten PIN and signatures acquired by digital pen. Higher level of data abstraction is achieved by representation of a time series into a vector of several areas bounded by segments of consecutive zero crossings including peaks and valleys. We named this representation as the area bound approximation ABA. 1D and 2D area bound representation of time series are proposed where each calculated area is represented by a single data point of area value or by two data points of area and time index values respectively. Different experiments were performed in order to evaluate the performance of DTW and proposed AB_DTW methods in terms of accuracy and computational complexity. It is
found that the DTW and AB_DTW techniques applied to classify human individuals using handwritten PIN words have similar high score values (better SR > 99.97% & AUC–ROC > 0.997%). However, with AB_DTW the computational time is reduced by a factor of 42 over DTW and by a factor of 23 over DTW for handwritten PIN and signature datasets respectively. The most important contribution of AB_DTW method is highly reduced representation of the time series (determines indirectly computation complexity). Therefore, AB_DTW allows fast classification of sequences by performing massive data reduction without a substantial degradation of accuracy values. The focus of present work was a comparison of DTW and AB_DTW for classification of handwritten PIN and signature samples essential for person authentication. Further speed up of computation can be achieved by involving the state of the art fast DTW methods. Future study work would be area bounding of dynamic time warping (AB_DTW) for multivariate time series. Acknowledgments The support given by G. Scharfenberg and G. Schickhuber from the University of Applied Sciences Regensburg is highly acknowledged. We want to thank Christos Saragiotis and Eli Billauer, for making available online, zero crossing and peak detection codes respectively. We also thank the unknown reviewers for their useful comments. References [1] F. Hao, Chan C. Wah, Online signature verification using a new extreme points warping technique, Pattern Recogn. Lett. 24 (2003). [2] Jian Zhang, S. Kamata, Online signature verification using segment-to-segment matching, in: Int. Conf. on Frontiers in Handwriting Recognition, ICFHR, 2008. [3] C. Gruber, C. Hook, J. Kempf, G. Scharfenberg, B. Sick, A flexible architecture for online signature verification based on a novel biometric pen, in: Proceedings of IEEE Mountain Workshop on Adaptive and Learning Systems (SMCals/06), Logan, 2006, pp. 110–115. [4] Hong-Wei Ji, Zhong-Hua Quan, Signature verification using wavelet transform and support vector machine, in: ICIC 2005, Part I, in: LNCS, vol. 3644, SpringerVerlag, Berlin, Heidelberg, 2005, pp. 671–678. [5] D. Fenton, M. Bouchard, T.H. Yeap, Evaluation of features and normalization techniques for signature verification using dynamic time warping, in: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2006. [6] S. Salvador, P. Chan, FastDTW: Toward accurate dynamic time warping in linear time and space, in: Proc. KDD Workshop on Mining Temporal and Sequential Data, 2004. [7] Eamonn J. Keogh, Michael J. Pazzani, Scaling up dynamic time warping for data mining applications, in: Proc. 6th Int. Conf. on Knowledge Discovery and Data Mining, KDD, 2000. [8] Muzaffar Bashir, Jürgen Kempf, Reduced dynamic time warping for handwriting recognition based on multi-dimensional time series of a novel pen device, Int. J. Electr. Comput. Eng. 3 (8) (2008) 494–500. [9] Eamonn Keogh, Chotirat A. Ratanamahatana, Exact indexing of dynamic time warping, in: Knowledge and Information Systems, Springer-Verlag, London, 2004. [10] Daniel Lemire, Faster retrieval with a two-pass dynamic-time-warping lower bound, Pattern Recogn. 42 (2009) 2169–2180. [11] http://www.bisp-regensburg.de. [12] Muzaffar Bashir, A novel multisensoric system recording and analyzing human biometric features for biometric and biomedical applications, University of Regensburg, Germany, 2012, urn:nbn:de:bvb:355-epub-196730. [13] http://www.mathworks.com. [14] Muzaffar Bashir, Jürgen Kempf, DTW based classification of diverse preprocessed time series obtained from handwritten PIN words and signatures, J. Signal Process. Syst. 64 (2011) 401–411, http://dx.doi.org/10.1007/s11265010-0501-x. [15] Muzaffar Bashir, Jürgen Kempf, Person authentication with RDTW using handwritten PIN and signature with a novel biometric smart pen device, in: IEEE Workshop on CIB, USA, 2009. [16] H. Sakoe, S. Chiba, Dynamic programming algorithm optimization for spoken word recognition, IEEE Trans. Acoust. Speech Signal Process. 26 (1) (1978) 43– 49. [17] T.M. Rath, R. Manmatha, Lower-bounding of dynamic time warping distances for multivariate time series, CIIR Technical Report MM-40, 2003.
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Dr. Muzaffar Bashir received his PhD (January 2011) from the University of Regensburg and the University of Applied Sciences, Regensburg, Germany. He received his MSc (Physics) from the University of the Punjab and MSc (Computer Science) from UET Lahore, Pakistan. He was employed as a Lecturer at the University of the Punjab, Lahore, Pakistan, in 2002. Now he is working as an Assistant Professor at the same university, since March 2011. His research interests are medical data and image processing, time series analysis, pattern
267
recognition, biometrics, bio-sensing, biophysics and computational methods. Prof. Dr. Jürgen Kempf received his PhD in Physics from the University of Heidelberg, Germany. He was a Scientist in IBM research laboratories for many years. Now he is a Professor at the University of Applied Sciences, Regensburg, Germany. His research interests are biometrics, information security, pattern recognition and sensorics.
Contents lists available at SciVerse ScienceDirect
Digital Signal Processing www.elsevier.com/locate/dsp
Area bound dynamic time warping based fast and accurate person authentication using a biometric pen Muzaffar Bashir a,b,∗ , Jürgen Kempf a a b
University of Applied Sciences Regensburg, Germany University of the Punjab, Lahore, Pakistan
a r t i c l e
i n f o
Article history: Available online 30 August 2012 Keywords: Biometric person identification Biometric signature verification Handwritten PIN recognition Area bound DTW Time series
a b s t r a c t The paper presents a modified dynamic time warping (DTW) technique for person authentication based on time series matching obtained from handwriting. The online data has been acquired by a biometric smart pen device. The proposed method allows fast and accurate classification of human individuals based on handwritten PIN words or signature samples. Although classic DTW provides robust distance measurements essential for accurate classification of sequences, it is computationally expensive. To speed up computations we introduce area bound dynamic time warping (AB_DTW) that divides time series into several areas bounded by segments of consecutive zero crossings including local peaks and valleys. Unlike classic DTW which compares whole signals, the proposed AB_DTW warps areas bounded by the local regions. Two kinds of data abstraction formats of area bound—1 dimensional and 2 dimensional— are evaluated. Experimental results show that because of a higher-level data abstraction, the proposed approach is several times faster than classic DTW. Moreover, AB_DTW does not offer substantial loss of accuracy which is required for authentication performance using handwritten PIN words and signatures sampled by biometric pen device. © 2012 Elsevier Inc. All rights reserved.
1. Introduction One of the widely researched behavior-biometrics is person authentication using handwritten signatures. Online person authentication using a handwritten signature is promising because of long history of use and wide acceptance of traditional signature verification in public domain [1–3]. A biometric pen device is generally used to acquire the dynamics of handwriting for online data. There exist two approaches to look at similarity or dissimilarity of sequences obtained from handwriting. They are called functional and parametric approaches [1,2]. In the functional approach, the feature set is essentially obtained from the complete signal in terms of time series, thus retaining more information on the signing process. On the other hand, in the parametric approach, the features or parameters are selected as a higher-level abstraction of original data. Therefore, the latter approach though produces result much faster but poses difficulty when it comes to the selection of features that can contribute to the classification effectively [3]. The functional approach of complete signal comparisons gives better results [1,2]. Dynamic time warping (DTW) technique has successfully been applied in this regard in recent years [1,2,4,5]. DTW is used to determine the distance of two sequences by minimizing
*
Corresponding author at: University of the Punjab, Lahore, Pakistan. E-mail addresses: [email protected] (M. Bashir), [email protected] (J. Kempf). 1051-2004/$ – see front matter © 2012 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.dsp.2012.08.013
the intra-individual variations. In DTW, the natural nonlinear time variations of sequences that subsist even for genuine signatures are reduced before the actual distance is calculated. But classic DTW is computationally expensive. To speed up computations, there are some modified techniques which operate on a higher-level abstraction of the data as piecewise aggregate approximation (PAA) of time series or data down-sampling [6–8]. In some other methods, the numbers of data point comparisons are reduced as in extreme points warping or segment to segment matching and lower bounding techniques [1,2,9,10]. Moreover, commonly used constraints like Sakoe–Chiba bands or Itakura parallelograms are applied because they limit the allowed warp search path in the cost matrix resulting in a speed up of DTW [9]. This study presents a local data abstraction based DTW approach named Area Bound Dynamic Time Warping (AB_DTW). It warps only the areas bounded by the local regions of sequences obtained from handwriting. The speed up of computations is achieved by taking a time series as a vector of several areas bounded by the segments of consecutive zero crossings (ZCs) which may include positive peak(s) or negative peak(s). Using AB_DTW, biometric person authentication based on time series generated by handwritten personal identification number (PIN) words and signatures is presented. The acquisition of the online data is carried out with the help of a biometrics smart pen (BiSP) device [11]. The work presented here is mainly inspired by the work of [1,2,4,7].
260
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Fig. 1. (a) An example of pressure signal obtained from handwritten PIN “4A5B7EM” and (b) smoothed and normalized pressure signal is shown. The segmentation points are marked with ‘’.
The paper is organized as follows. In Section 2, the BiSP device used for data acquisition and pre-processing of data is briefly described. Section 3 describes DTW based classifier and related work. Section 4 outlines the concept of speeding up of DTW using area bound approximation of the time series. Then in Section 5 results of the experimental work for biometric person authentication are discussed. Section 6 finally summarizes the major findings and outlook. 2. BiSP device for data acquisition and data preprocessing The biometric smart pen (BiSP) used in the study is a ballpoint pen that allows the record and analysis of handwriting, drawing and gesture movements on a paper pad or free in air [11]. For a comprehensive assessment of pen movement and fine motor features of hand and fingers, the device is equipped with a diversity of sensors measuring the acceleration and tilt angle of the pen, the grip forces of the fingers holding the pen, and the forces and vibrations generated in the refill during writing or drawing on a pad. The captured handwriting movements can be represented by the time series of five sensor channels as (i) across horizontal pressure, (ii) grip finger-grip pressure, (iii) z vertical pressure, (iv) α longitudinal and (v) β vertical angles of the pen. All of them are measured as a function of time. The across sensor measures the change of forces resulting from movements of the pen tip along the x– y dimension and generates a single time series signal of the pressure along the x– y dimension due to handwriting. The across sensor is implemented with the help of a piezoelectric polymer film placed close to the front part of the pen (pen tip). The time series obtained from the across sensor of the pen is used in this work. The description of the device is necessarily shortened, for more information the reader is referred to [8,12]. In order to eliminate the potential sensor noise, the data is pre-processed after acquisition. The essential pre-processing steps are smoothing, segmentation, normalization and down-sampling of
data without loss of valuable information. A typical example of a time series (across pressure signal) recorded with BiSP during handwriting a PIN word on a paper pad is shown in Fig. 1(a). While Fig. 1(b) represents the smoothed and normalized signal. A straight-line fit linear-trend is removed using “detrend()” function of MATLAB [13]. Smoothing of data based on local regression is done in order to minimize sensor noise [14]. Each signature or PIN data sample was captured separately during acquisition. Therefore, segmented data was captured in [14,15]. In order to be able to have accurate ZC points at both ends of the signal, additional segmentation is performed here. The baselines at both ends of the signal are removed using segmentation points (SPs) as shown in Fig. 1(b). The handwriting signal part between two ZC index points (or SPs) is extracted from the original time series using peak and zerocrossing determination algorithm. The two SPs are determined on the original time series by searching first ZC points and closest first peaks while the time series is scanned at both ends respectively. The flow of steps in the segmentation algorithm is shown in Table 1. In order to compensate large variations in amplitude values, the data is normalized to [−1, 1]. Further, in order to reduce complexity of classic DTW based classifier when applied to original sequences, the data is re-sampled to a lower sampling rate (data down-sampling). Data pre-processing steps can be viewed in detail in [8,14]. 3. DTW based classifier and related work For person authentication, dynamic time warping (DTW) algorithm was applied to the time series obtained from handwriting sequence data. DTW based classifier is used to determine the distance of two sequences {( Q = q1 , q2 , . . . , qn ) & (C = c 1 , c 2 , . . . , cm )} of even different lengths. In DTW, nonlinear time variations of sequences even for genuine signatures are minimized before the actual distance is calculated. For the alignment of two sequences,
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
261
Table 1 Segmentation using peaks and zero-crossings. Step Step Step Step Step Step Step Step Step Step Step Step
1 2 3 4 5 6 7 8 9 10 11 12
Divide original signal into two signals. signal1 represents the first half and signal2 is the reversal of the second half of original Find threshold1 using 1/4th of STD of signal1 Find values and index points of peaks (local maxima and minima) threshold1 Find index value of first peak PKID Find threshold2 for zero-crossing using MEAN of signal1 Find zero-crossing index values ZC1 using threshold2 Find zero-crossing index values ZC2 < PKID If ZC2 is zero then take negative of the signal1 , i.e., signal1 = signal1 ∗ (−1) and go to step 2, else go to next step Find last index of ZC2 as a first point of segmentation seg1 Repeat step 2 to step 9 for signal2 Find last point of segmentation seg2 using seg1 of signal2 by using expression: seg2 = length(signal) − seg1 Find segmented signal using index points seg1 of signal1 and seg2
Case I: 1D Time index= (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14) R = (0, −1, −2, −1, −1, 0, 1, 1, 2, 3, 2, 1, 1, 0) n = | R | = 14 zc = (1, 6, 14) S 1 = (0, −1, −2, −1, −1, 0), S 2 = (0, 1, 1, 2, 3, 2, 1, 1, 0) R¯ =
6 1
s1 dt ,
6
1
Case II:2D R¯ =
= (−0.833, 1.222) N = | R¯ | = 2
s2 dt
9
1
14 6
3
10
−0.833 1.222
Case II:2DN 0.300 1 R¯ = −0.833 1.222
N = | R¯ | = 2 N = | R¯ | = 2
Fig. 2. Time series R of 14 data points and three zero-crossing points (zc) marks at time indices (1, 6 and 14). The time series is divided into two segments due to zc. The area under the curve is calculated. The area values represent the area bound representation due to three cases as described below the curves.
the squared Euclidean distance d(q i , c j ) = (qi − c j )2 is calculated under global and local constraints. A local constraint or cumulative distance function D (, ) is determined which minimizes the current cell and neighboring cell elements distance. If d(q i , c j ) is the distance found in the current cell, then two common cumulative distance cost functions are as shown in Eq. (1) and Eq. (2).
D (i , j ) = d(qi , c j )
+ min D (i − 1, j − 1), D (i − 1, j ), D (i , j − 1) ,
(1)
D (i , j ) = d(qi , c j ) + min D (i − 1, j − 1), D (i − 1, j − 2),
D ( i − 2, j − 1) .
(2)
DTW with the cumulative cost function as described in Eq. (2) is used in this study because it has shown better performance of sequence matching in preliminary results. DTW can be viewed in detail in [7,9,16]. In DTW based classifier, for a query sequence, the distances to all reference sequences are calculated. A list of nearest reference sequences is determined and the minimum distance of the best match decides for classification (see [15]). Classic DTW has computing time and memory space problems, especially in online handwriting or person recognition. The remedy has somehow been provided by some modified techniques, which operate on a higher-level abstraction of the data as piecewise aggregate approximation PAA of time series or down-sampling of data [6–8]. In some other methods, the numbers of data point comparisons
262
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Fig. 3. Data points are shown in area bound representation as “−0.833” & “1.222” in 1D case, “(3, −0.833)” & “(10, 1.22)” in 2D case and “(0.3, −0.833)” & “(1, 1.222)” in 2DN case.
are reduced as in extreme points warping or segment-to-segment matching and lower bounding techniques [1,2,9,10]. Additionally, to speed up DTW, commonly used constraints like Sakoe–Chiba bands or Itakura parallelograms, which limit the allowed warp search path in the cost matrix, are used to reduce computational time [9]. 4. Area bound warping method This study presents area bound dynamic time warping (AB_DTW) method, which warps only the areas bounded by the local regions of the time series. In [7], a time series is abstracted into piecewise constant approximations using mean values of data in equi-sized frames. This kind of data reduction strongly depends on the size and number of frames on which data is subjected. In our approach, we have chosen automatic dynamic frame sizes using consecutive ZCs and the curves (segments) in the frames are abstracted by corresponding areas as shown in Fig. 2. The objective of our work is data reduction, which has been achieved successfully through an approach based on local data reduction. The curve in Fig. 2(a) represents a time series R of fourteen (n) points and three zero-crossing (ZC) points marked at the indices 1, 6 and 14 respectively. In Fig. 2(b), the time series is divided into two (N = zc − 1) unequal frames. A vector of normalized area values calculated for the segments in the frames becomes the area bound approximation (ABA). Equations below the curves demonstrate the calculation performed for the segments S 1 and S 2 . The area is calculated by trapezoidal rule. The speed up of computations is achieved by higher level reduced representation of time series. A time series is converted into a vector of several areas bounded by the segments of consecutive zero crossings. A segment may include a ZC, the positive peak(s) and a ZC or like wise a ZC, the negative peak(s) and a ZC. Two cases of area bound representations—(I) one dimensional and (II) two dimensional—are presented.
One dimensional case (1D) In this case, a time series consisting of 14 points is projected into 1 dimension. The time series is divided into two frames (segments) due to three ZC points. The area under the curve in each frame is calculated. Therefore, a time series consisting of 14 points is represented by two data points (two areas) in a vector (array) known as reduced time series in 1D case. As shown in Fig. 2, three ZC occur at three time index values (1, 6 and 14). The areas are calculated for segments from time index 1 to 6 and 6 to 14, respectively. This therefore gives two data points (area values “−0.833” and “1.222”) stored in a vector (array) in 1D case. Two dimensional case (2D) Similarly in 2D case, the time series consisting of 14 points is projected into 2 dimensions. Now, a single area is represented by two data points: (i) calculated area value and (ii) time index value. Therefore, the time series is converted into a 2 dimensional vector (matrix). Hence area values “−0.833” and “1.222” are stored in the first column of the matrix (first dimension) while mean time index values (“3.5 ≈ 3” and “10”) are stored in the second column of the matrix (second dimension) respectively. The mean of time index value is rounded to integer. Note in the normalized 2D case denoted by “2DN”, the obtained time index values are normalized to [0, 1] as illustrated in Fig. 2 and Fig. 3. Note: more precisely, a vector in 1D case corresponds to n points (areas) in one dimensional space represented by a vector (array): a(k − 1). While 2 dimensional vector (matrix) represents area and time values with coordinates (a, t ) where “a” measures the area under the curve within a frame (segment) bounded by (t i , t j ) and t is given by(t i + t j )/2. The term k denotes the number of zero crossings in the sequence. For demonstration, data points in area bound representation are shown in Fig. 3. It is observed that area values remain constant though the values of abscissa change with respect to 2D and 2DN cases, as evident from Fig. 3.
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
263
Table 2 Area bound calculation using peaks and zero-crossings. Step Step Step Step Step Step Step Step Step Step Step Step Step Step
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Define threshold1 for peaks detection Define threshold2 for zero crossing determination Find values and index points of peaks (local maxima and minima) threshold1 Find an array ZC of zero-crossing index values using threshold2 Initialize first zero-crossing index point out of ZC array Initialize an array index ind Begin loop (until end of ZC array) Select next zero-crossing index point out of ZC array Find numbers of peaks PK between first and next zero-crossing index If PK is zero go to step 8, else go to next step Find segment of signal from first to next zero-crossing index points Calculate area enclosed by the segment and find normalized area value (divide area by the length of the segment) Put area value into an array A using index ind If two dimensional case then put the mean of first and next into an array of time T using index ind, else go to next step
Fig. 4. Two time series of genuine signatures are shown. The A ± are shown in different shading above and below zero-crossing. The representation of time series in bounded area reduced form is shown with marker ‘’ and zero crossings are shown with marker ‘◦’.
The AB_DTW warping process involves: determination of positive peaks (PK+) and negative peaks (PK−), zero-crossing (ZC) detection, bounded areas calculation and bounded area matching. The extreme points of peaks and valleys (local maxima and minima) are treated as PK+ and PK− respectively. The ZC points are defined as the points where the signal crosses zero line (or a predefined value) while signal values are changing with respect to time: (a) From baseline towards zero. (b) From peak (PK+) towards zero. (c) From peak (PK−) towards zero. 4.1. Bounded areas calculation A time series R of length n is given by
R = r 1 , r 2 , r 3 , . . . , rn .
(3)
If the points zc 1 , zc 2 , zc 3 , . . . , zck are zero crossing points such that zc = k, then the number of segments N in the original sequence R will be zc − 1 (i.e., N = k − 1). Now the time series R¯ in ABA is given by:
R¯ =
zc2
1
|1 + zc 2 − zc 1 |
s1 dt , zc 1
1
|1 + zck − zck−1 |
zck
1
|1 + zc 3 − zc 2 |
sk−1 dt .
zc3 s2 dt , . . . , zc 2
(4)
zck−1
Each calculated area is divided by the number of data points in the segment. Note: there should be at least one segment and two zero crossings in a sequence, i.e. | zc | 2 or N 1. The area bounded by the local curve of the signal is treated as A + or A −. A + is area bounded by all points of sub-sequence (above ZC) between consecutive ZC points including peaks (PK +). Similarly, the area A − includes peaks (PK −). There could be one ZC point contributing to two consecutive areas. An area can contain one or more peaks and is determined for a segment with all points enclosed by the two ZC points. Very small ripples are not considered as peaks (PK ±). Therefore, the area ( A ±) below a certain threshold value is considered to be a result of ripples or noise called “RN area”. The latter can be adjusted with the adjacent areas. Moreover, the area contribution of a segment with two consecutive ZC is also adjusted when it does not contain any
264
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Fig. 5. Good examples of reproducible signals and area bound representations. The alignment between two sequences as discovered by AB_DTW and area match for comparison of A − (dotted lines) and A + (solid lines) are shown.
peak (PK + or PK −) lie within a threshold. Further, an area though very small may not be neglected if the area contributions from the previous and subsequent area are greater than a threshold value. The areas ( A ±) are calculated by trapezoidal function provided by MATLAB [13]. The flow of steps for area bound calculation based on peaks and zero-crossing detection algorithm is shown in Table 2. For illustration, two time series of genuine signatures are shown in Fig. 4 and the A ± are shown in different shades, above and below zero crossing line. The corresponding vectors of normalized areas of consecutive ZC are shown as area sequences marked with ‘’ and dotted lines. The ZC points are marked with ‘◦’. A time series of about 2200 data points (Fig. 4 top) is reduced to a vector of 18 area bound data points, i.e., | zc | = 19. 4.2. Bounded area matching Fig. 5 shows an example of two genuine signatures. Equal number of bounded areas can be observed in the both sequences. It is of course due to equal number of ZCs found in the both sequences. AB_DTW match for sequence comparison thus reduces to oneto-one area match or simple Euclidean distance calculation. For illustration, the match of A + to A + and of A − to A − is shown with solid lines and dotted lines respectively. Moreover, the signal shown at the bottom (Fig. 5) is shifted to a different baseline to increase visibility. There subsist some variations in the sequences even for two similar signatures. Therefore, the Euclidean distance or one-to-one bounded area match is not always a suitable choice. This could be due to missing or extra areas resulting from extra or missing peaks and ZC points or also due to sensor noises (see results). Therefore, for reduced time series of bounded areas, DTW algorithm is used to perform the similarity match of two sequences. Fig. 6 shows an example of possible AB_DTW match for two sequences obtained from two genuine signatures. Due to unequal numbers of zero crossings and/or extra or missing peaks, there are different numbers of bounded areas calculated for the two sequences. The possible area bound warping of the two sequences is shown. The fifth calculated area of the first signal (top) essentially contains missing peaks and can be warped to three areas of the second sig-
nal (bottom) for matching. Similarly, the 12th and 13th areas of first signal may come from extra peaks and zero crossings, which do not occur in the bottom signal. 5. Experiments and results The main objective of the experiments was the classification of handwriting signals obtained from two datasets of handwritten sequences essential for person authentication. Fifty writers contributed to the two databases consisting of 500 signatures and 500 PIN words samples. Each writer donated 10 genuine signatures and PIN samples in a single session. PIN word is a sequence of different seven single characters written by the owner of the PIN (e.g. “4A5B7EM”). Each writer is assigned a unique PIN. The acquisition of online data is carried out with the help of biometric smart pen device. Leave-one-out cross validation (LOOCV) is used for classification of each dataset independently. The focus is to investigate the effect of data abstraction and to compare accuracy as well as runtime of classic DTW and proposed AB_DTW methods. In order to reduce the complexity of classic DTW based classifier, proposed AB_DTW method is applied which operates on reduced representation of time series in 1D and 2D forms. The minimum DTW distance determines the accuracy of match while the number of data points of the two sequences to be compared indirectly determines the computational complexity. Several experiments have been performed in order to evaluate the performance of the proposed method. (1) Experiment 1: Classic DTW method is applied to classify two signatures or PIN sequences in such a way that a longer sequence is normalized to the shorter one in time domain. (2) Experiment 2: The proposed AB_DTW method is applied to classify PIN and signature time series datasets separately. (3) Experiment 3: In order to find the effect of DTW on area bound representation of time series, simple Euclidean distance is also measured to test the accuracy of classification. In other words, the proposed area bound representation of time series data is classified using direct point-to-point data comparisons. This method is denoted by AB_Euclidean.
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
265
Fig. 6. The example of two genuine signature signals and area bound representations carrying different number of areas is shown. Possible AB_DTW for sequence comparison is shown.
(4) Experiment 4: The proposed AB_DTW method is applied to 2D area bound representation of the time series denoted by “AB_DTW2D”. The area bound representation of reference and sample sequences can be expressed as two dimensional data q(ai , t i ) and c (ai , t i ) respectively, where ai is the value of calculated area and t i is the averaged time index. For simplicity, we denote an element of 2D vector (matrix) by two subscripts (i.e. qi ,k ). So, the local distance d( , ) [17] in 2D case is given by
d(qi ,k , c j ,k )2 =
2 (qi ,k − c j ,k )2 .
(5)
k =1
(5) Experiment 5: The proposed AB_DTW method is applied to 2D area bound representation as we did in experiment 4 such that the time index values are normalized to [0, 1]. This method is denoted by “AB_DTW2DN”. Note: Normalized area values are very small while time index values can be very large (see Fig. 4). Further, there could be variations in time index values of two similar sequences obtained from the same writer. Therefore, in order to minimize the effect of such variations, the time index values are normalized to [0, 1] (see Section 3). In order to compare the performance of classification of PIN and signature sequences quantitatively, the accuracy in terms of parameters such as score of recognition SR [15], the runtime and the area under the curve of receiver operating characteristic AUC–ROC [15] is determined. As no skilled forgeries are used, false acceptance rate (FAR) represents the random PIN word or signature acceptance in ROC curve. The performance parameters based on the experiments are shown in Table 3 and Table 4. The values shown are for person authentication using handwritten PIN words and signatures respectively. The SR values of classic DTW and AB_DTW are similar (difference is ≈ 0.02). Similarly the AUC–ROC values in case of classic
Table 3 Performance parameters for person authentication using handwritten PIN. The values of SR, AUC–ROC and runtime are averaged over all writers. Experiment
Score SR
AUC–ROC
Mean time (s)
DTWp AB_DTWp AB_Euclideanp AB_DTW2Dp AB_DTW2DNp
99.995 99.971 91.126 90.971 99.975
0.9996 0.9979 0.8634 0.8456 0.9983
12.5200 0.3100 0.0243 0.3738 0.3505
Table 4 Performance parameters for person authentication using handwritten signature. The values of SR, AUC–ROC and runtime are averaged over all writers. Experiment
Score SR
AUC–ROC
Mean time (s)
DTWs AB_DTWs AB_Euclideans AB_DTW2Ds AB_DTW2DNs
99.800 99.122 86.440 96.738 99.383
0.9961 0.9861 0.8425 0.9155 0.9810
6.1800 0.2720 0.0244 0.3349 0.3040
DTW and AB_DTW are approximately same (difference is ≈ 0.002 see Table 3) as especially evident for handwritten PIN. The SR and AUC–ROC values of DTW and AB_DTW show essentially same high accuracy but AB_DTWp is faster by a factor of ≈ 42 and AB_DTWs is faster by a factor of ≈ 23 for handwritten PIN and signature data respectively, as shown in Table 3 and Table 4. The performance parameters for AB_DTW in two different forms of the 2D cases are also shown in Table 3 and Table 4. The SR and AUC–ROC values are relatively low in experiment 4 (i.e., AB_DTW2D). This is because of large variations in time index values which may exist even in two similar sequences. However, with normalized time index values the increase in accuracy is ≈ 9% in AB_DTW2DNp and ≈ 2.6% in AB_DTW2DNs . Further, the accuracy of AB_DTW and AB_DTW2DN is similar (difference is very small see Table 3 and Table 4). Although AB_Euclidean can quickly classify sequences, its performance is relatively very poor in comparison to proposed
266
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Table 5 The Mean, Maximum and Minimum numbers of data points used in different experiments for handwritten PIN. Number of data points
Original Segment DTWp AB_DTWp
Mean
Maximum
Minimum
4212.50 3646.80 364.00 34.35
6550 6086 608 42
2510 2152 215 26
Table 6 The Mean, Maximum and Minimum numbers of data points used in different experiments for handwritten signatures. Number of data points
Original Segment DTWs AB_DTWs
Mean
Maximum
Minimum
2831.20 2234.30 223.43 28.50
4299.60 3857.00 385.70 55.00
1679.4 1047.0 104.7 12.5
AB_DTW that essentially shows the advantage of DTW algorithm on proposed area bound representation of time series. The time complexity of classic DTW is O (mn) [7] with m = n, where m and n are the length of two sequences. Similarly, the time complexity of proposed AB_DTW is O ( M N ) with M = N, where M and N are the length of two sequences in proposed area bound approximation (ABA). The runtime values shown in Table 3 and Table 4 also depend on running environment, therefore we choose to show computation reduction in terms of reduction in the number of data points in regard of different experiments. In order to reduce the computational time of classic DTW for original sequences, segmentation of data based on peak and zerocrossings detection (see Section 3) was performed. Further, data is down-sampled by a factor of 10. The Mean, Maximum and Minimum numbers of data points (averaged over 50 writers) are shown for different experiments in Table 5 and Table 6. For PIN word data, the Mean number of data point is reduced from 4212.5 to 364 in DTW (i.e., 8.6% of original data). Similarly, the Mean number of data point in AB_DTW is 34.35 (0.82% of original data). For signature data (Table 6), the original Mean number of data points is reduced from 2831.2 to 223.4 in DTW (8.0% of original data). While, the Mean number of data points in AB_DTW is 28.50 (1.0% of original data). Similar effects of data point reduction in case of Maximum and Minimum for both datasets are observed for classic DTW and proposed AB_DTW as shown in Table 5 and Table 6. The averaged Minimum number of data points in signature dataset is less than that of PIN dataset indicates that some writers have very small signature while the handwritten PIN words are restricted to be 7 single characters. Finally, the proposed area bound approximation of time series allows massive data reduction. 6. Conclusions In this paper, we proposed area bound dynamic time warping (AB_DTW) for fast and accurate person identification using two datasets of handwritten PIN and signatures acquired by digital pen. Higher level of data abstraction is achieved by representation of a time series into a vector of several areas bounded by segments of consecutive zero crossings including peaks and valleys. We named this representation as the area bound approximation ABA. 1D and 2D area bound representation of time series are proposed where each calculated area is represented by a single data point of area value or by two data points of area and time index values respectively. Different experiments were performed in order to evaluate the performance of DTW and proposed AB_DTW methods in terms of accuracy and computational complexity. It is
found that the DTW and AB_DTW techniques applied to classify human individuals using handwritten PIN words have similar high score values (better SR > 99.97% & AUC–ROC > 0.997%). However, with AB_DTW the computational time is reduced by a factor of 42 over DTW and by a factor of 23 over DTW for handwritten PIN and signature datasets respectively. The most important contribution of AB_DTW method is highly reduced representation of the time series (determines indirectly computation complexity). Therefore, AB_DTW allows fast classification of sequences by performing massive data reduction without a substantial degradation of accuracy values. The focus of present work was a comparison of DTW and AB_DTW for classification of handwritten PIN and signature samples essential for person authentication. Further speed up of computation can be achieved by involving the state of the art fast DTW methods. Future study work would be area bounding of dynamic time warping (AB_DTW) for multivariate time series. Acknowledgments The support given by G. Scharfenberg and G. Schickhuber from the University of Applied Sciences Regensburg is highly acknowledged. We want to thank Christos Saragiotis and Eli Billauer, for making available online, zero crossing and peak detection codes respectively. We also thank the unknown reviewers for their useful comments. References [1] F. Hao, Chan C. Wah, Online signature verification using a new extreme points warping technique, Pattern Recogn. Lett. 24 (2003). [2] Jian Zhang, S. Kamata, Online signature verification using segment-to-segment matching, in: Int. Conf. on Frontiers in Handwriting Recognition, ICFHR, 2008. [3] C. Gruber, C. Hook, J. Kempf, G. Scharfenberg, B. Sick, A flexible architecture for online signature verification based on a novel biometric pen, in: Proceedings of IEEE Mountain Workshop on Adaptive and Learning Systems (SMCals/06), Logan, 2006, pp. 110–115. [4] Hong-Wei Ji, Zhong-Hua Quan, Signature verification using wavelet transform and support vector machine, in: ICIC 2005, Part I, in: LNCS, vol. 3644, SpringerVerlag, Berlin, Heidelberg, 2005, pp. 671–678. [5] D. Fenton, M. Bouchard, T.H. Yeap, Evaluation of features and normalization techniques for signature verification using dynamic time warping, in: IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2006. [6] S. Salvador, P. Chan, FastDTW: Toward accurate dynamic time warping in linear time and space, in: Proc. KDD Workshop on Mining Temporal and Sequential Data, 2004. [7] Eamonn J. Keogh, Michael J. Pazzani, Scaling up dynamic time warping for data mining applications, in: Proc. 6th Int. Conf. on Knowledge Discovery and Data Mining, KDD, 2000. [8] Muzaffar Bashir, Jürgen Kempf, Reduced dynamic time warping for handwriting recognition based on multi-dimensional time series of a novel pen device, Int. J. Electr. Comput. Eng. 3 (8) (2008) 494–500. [9] Eamonn Keogh, Chotirat A. Ratanamahatana, Exact indexing of dynamic time warping, in: Knowledge and Information Systems, Springer-Verlag, London, 2004. [10] Daniel Lemire, Faster retrieval with a two-pass dynamic-time-warping lower bound, Pattern Recogn. 42 (2009) 2169–2180. [11] http://www.bisp-regensburg.de. [12] Muzaffar Bashir, A novel multisensoric system recording and analyzing human biometric features for biometric and biomedical applications, University of Regensburg, Germany, 2012, urn:nbn:de:bvb:355-epub-196730. [13] http://www.mathworks.com. [14] Muzaffar Bashir, Jürgen Kempf, DTW based classification of diverse preprocessed time series obtained from handwritten PIN words and signatures, J. Signal Process. Syst. 64 (2011) 401–411, http://dx.doi.org/10.1007/s11265010-0501-x. [15] Muzaffar Bashir, Jürgen Kempf, Person authentication with RDTW using handwritten PIN and signature with a novel biometric smart pen device, in: IEEE Workshop on CIB, USA, 2009. [16] H. Sakoe, S. Chiba, Dynamic programming algorithm optimization for spoken word recognition, IEEE Trans. Acoust. Speech Signal Process. 26 (1) (1978) 43– 49. [17] T.M. Rath, R. Manmatha, Lower-bounding of dynamic time warping distances for multivariate time series, CIIR Technical Report MM-40, 2003.
M. Bashir, J. Kempf / Digital Signal Processing 23 (2013) 259–267
Dr. Muzaffar Bashir received his PhD (January 2011) from the University of Regensburg and the University of Applied Sciences, Regensburg, Germany. He received his MSc (Physics) from the University of the Punjab and MSc (Computer Science) from UET Lahore, Pakistan. He was employed as a Lecturer at the University of the Punjab, Lahore, Pakistan, in 2002. Now he is working as an Assistant Professor at the same university, since March 2011. His research interests are medical data and image processing, time series analysis, pattern
267
recognition, biometrics, bio-sensing, biophysics and computational methods. Prof. Dr. Jürgen Kempf received his PhD in Physics from the University of Heidelberg, Germany. He was a Scientist in IBM research laboratories for many years. Now he is a Professor at the University of Applied Sciences, Regensburg, Germany. His research interests are biometrics, information security, pattern recognition and sensorics.