- Email: [email protected]
Off-line signature verification using structural feature correspondence
Pattern Recognition 35 (2002) 2467 – 2477
www.elsevier.com/locate/patcog
O -line signature veri"cation using structural feature correspondence Kai Huang ∗ , Hong Yan School of Electrical and Information Engineering, University of Sydney, NSW 2006, Australia Received 27 March 2001; accepted 13 September 2001
Abstract This paper presents an o -line signature veri"cation method using a model-based approach. In this method, statistical models are constructed for both pixel distribution and structural layout description. In addition to simple geometric handwriting features, it is proposed to use the directional frontier feature as a structural descriptor of the signature. The statistical veri"cation algorithm based on the geometric handwriting feature is used to accept signatures which closely resemble the reference samples, and to reject random and less skilled forgeries. For the questionable signatures for which the pixel feature judgement is inconclusive, the structural feature veri"cation algorithm is invoked. This algorithm compares the detailed structural correlation between the input and reference signatures in an attempt to detect skilled forgeries. The e ectiveness of the approach is evaluated on an experimental signature database. ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Signature veri"cation; Structural model; Directional frontier; Relaxation matching
1. Introduction A handwritten signature, being a behaviour biometrics, is well accepted socially and legally as a convenient mean of authorisation and identi"cation. Many documents, e.g. forms, necessitate the signing of a signature. As a consequence, o -line signature veri"cation becomes an essential component in automating the processing of o -line documents with embedded signatures [1]. Although on-line transactions are common-place, there are still large amounts of present and historical o -line data which need to be processed. It is bene"cial, for the ease of manipulation and information sharing, to convert them into electronic form. Automated machine recognition=veri"cation systems hold the promise of performing these repetitive tasks consistently and at high speed. With this motivation, continuous research e orts are being invested in handwriting recognition and veri"cation [2– 4]. ∗ Corresponding author. Tel.: +61-2-981-866-33; +612-981-042-33. E-mail address: [email protected] (K. Huang).
fax:
In general, o -line signature veri"cation is a challenging problem. Unlike for the on-line signature, where dynamic aspects of the signing action are captured directly as a handwriting trajectory [5,6], the dynamic information contained in o -line signature is highly degraded. Handwriting features, such as the handwriting order, writing-speed variation, and skillfulness, are to be recovered from the greylevel pixels by utilising intelligent machine algorithms. Consequently, an o -line veri"cation system has to cope with a signi"cant amount of errors and uncertainties in the recovered data. These diEculties are not present in the on-line case. Signatures are special symbols, which are usually cursively written and graphical in appearance. It is often diEcult, even for humans at instances, to segment the signature images into letters or basic lexical constructs, and to recover the underlying handwriting sequence. The common approaches to o -line signature veri"cation have been to exploit the static features of the handwriting, treating the complete signature as a single entity. These techniques involve the analysis and comparison of image projections [7], gradient features [8,9], geometric features
0031-3203/02/$22.00 ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 0 1 ) 0 0 2 2 2 - 9
2468
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
[10 –13], shadow-code descriptors [14,15], transform features [16,17], and moment features [18]. Recently, neural networks [19,20] and multi-expert veri"cation techniques [21,22] have appeared as viable and robust methodologies in system implementation. Pseudo dynamic features have been incorporated to an extent in o -line signature veri"cation, e.g. pressure features [23]. A technique of o -line signature model construction and matching utilising hand-traced pseudo-dynamic feature has also been proposed [24]. In recent years, numerous attempts have been made in recovering temporal information from static handwritings and drawings [25]. These techniques work either from the thinning output of a binarised image [26 –29] or from the gradient and edge information of a grey-level image [30,31]. The recovery process is based on assumptions of local and global smoothness and continuity criteria. Handwriting parameters such as stroke direction, length, width, and curvature variation are estimated with these techniques. It can be expected that practical diEculties in utilising the recovered temporal data will soon be overcome, that new o -line signature veri"cation systems will bene"t from them. This paper presents a new method for o -line signature veri"cation using a combination of static image pixel features and pseudo-dynamic structural features. Instead of solving the problem of recovering the actual handwriting strokes in generating the signature, the structural feature named directional frontier is chosen as the basis for the veri"er. Directional frontiers are derived from image contours, which can be readily extracted using established techniques [32]. They are closely related to each of the handwriting strokes locally, and to the shape of the signature in a global sense. The structural matching and veri"cation technique is general enough to allow other types of structural descriptors to be used with little modi"cation, e.g. the curvature-segmented contours or the recovered handwriting strokes. In this technique, a signature is considered as an image entity as well as a group of structural entities. A procedure of generating statistical and structural models from a set of reference samples is formulated. The pixel feature model of the veri"er is based on neural networks and is used in the "rst stage veri"cation to eliminate random and less skilled forgeries. The "ltering process rejects dissimilar signatures and accepts well formed signatures. For the questionable signatures for which the pixel feature judgement is inconclusive, the structural feature veri"er is invoked. The structural veri"er compares the detailed structural correlation between input and reference signatures in order to detect skilled forgeries. The paper is organised as follows. First the directional frontier feature of a signature image is described. Then the signature modelling technique based on the static and pseudo-dynamic handwriting features are discussed. Finally the veri"cation strategies and experimental results are presented.
2. Directional frontier feature Directional frontier (DF) is a directional grouping of the contour pixels. It is the set of black pixels of the handwriting image whose left (right, up, down, etc.) neighbors are white (Fig. 1). The information in the DF bitmap is organised by pixel labelling and curve tracing, to generate a set of structural descriptors of the signature. The foreground pixels are labelled using the following scheme: an ‘I’ for an isolated pixel; an ‘E’ for an end pixel; and a ‘C’ for a connecting pixel (Fig. 2a). The bitmap is converted into a set of DF curves by applying a tracing algorithm (Fig. 2b). From visual inspection, the DF curves relate well to the local handwriting strokes, and to the global shape of the signature. E ectively, the signature is transformed from pixels into a set of stroke-like DF curves. 2.1. DF curve tracing In the pixel labelling process, ‘E’ labelled pixels are stored in a list. Tracing starts at the "rst ‘E’ labelled pixel in the list. The algorithm "rst checks the four-connected neighbours for the next tracing pixel before checking the 8-connected ones [32].
Algorithm of tracing DF curves. place pixels labelled ‘E’ into a ‘E’ list; i = 0; while ‘E’ list contains not “visited” elements { mark the "rst ‘E’ pixel “visited” on the ‘E’ list; i = i + 1; create new DF curve T[i]; trace until another ‘E’ pixel is encountered { add current pixel to T[i]; check the 4-connected neighbours for next pixel; check the 8-connected neighbours if not found. } add encountered ‘E’ pixel to T[i], and mark it “visited” on the ‘E’ list. }
2.2. DF curve characterisation A DF curve Ui of n points is approximated by a cubic spline Bi , Ui (xi ; yi ) = Bi (xi ; yi ) + Wi (xi ; yi );
i = i0 ; i1 ; : : : ; in−1 ;
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
2469
Fig. 1. Example DF maps, showing the contours and the directional frontiers of a signature image.
Fig. 2. (a) DF curve elements after pixel labelling. (b) DF curves extracted from pixel map, showing both the jagged and B-spline smoothed curves.
Bi (xi ; yi ) =
3
Bk[3] ( i )Pk ;
For convenience in curve matching, these DF parameters are stored or pre-calculated (Fig. 3):
i = i0 ; i1 ; : : : ; in−1 ;
k=0
Bk[3] ( i ) =
3 k
(1 − i )3−k ki ;
i =
i − i0 ; in−1 − i0
(1)
where Wi ’s are noise terms. Pk are the control points of the cubic spline [33]. The position of control points are estimated using the least-squares method. The analytic spline approximation is used in the curve shape matching calculation later in the paper.
• the direction of the map it is from, i.e. left, right, up, or down; • the control points Pk for reconstructing the curve; • the coordinates of the centre point Bc , the end points B0 , Bn−1 , and two via points, i.e. a reconstruction in "ve points; • the tangents of reconstructed curve at these points above; • the length after smoothing l = n−1 i=1 Bi − Bi−1 .
2470
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
S (E)
S (E)
T E
S (E)
T
T
E
E
E(C)
E(C)
C C
E(C)
C
C
C
C
Fig. 4. Small-gap reconnection templates, where ‘T’ denotes the blank pixel under test, ‘I’ for isolated pixels, and ‘E’ for end pixels. Note that the reNected, mirrored and rotated versions of the templates are also applied.
S2
S2 T2
T2
Tj
Tj
Fig. 3. Features used for characterising a DF curve segment. S1
2.3. Methods for reducing noise A noisy image boundary would introduce undesirable segmentation along a normally contiguous DF curve. Morphological operation, a closing followed by an opening with a square structural element, is used in preprocessing to smooth the foreground object boundaries. Further actions include small gap and large gap reconnections. 2.3.1. Small gap reconnection Small gap reconnection operates on the DF pixel maps. There are segments as well as the ‘I’ labelled pixels that are separated by one pixel. A 3 × 3 template is moved around the labelled pixel-map, to rectify this problem (Fig. 4). 2.3.2. Large gap reconnection The large gap reconnection process operates on the extracted set of analytic DF curves. The width of the gap is relatively large, which corresponds to the disturbance where one handwriting stroke is crossed-over by another stroke or by a background pattern. The reconnection process tries to identify these disturbances and then reconnect the pen-strokes accordingly. The large gap reconnection process produces a set of possible segment combinations with each combination attributed by a probability value. It involves checking the
T1 (a)
S1
T1 (b)
Fig. 5. Large-gap reconnections. In case (a), the reconnection is not possible because the link between the two segments is not in parallel with either the tangent directions at the joining ends of the segments. In case (b), all three directions are parallel, thus the join is permitted.
three characteristic directions between a pair of DF curves (Fig. 5): • end-point tangent direction T1 of segment 1; • end-point tangent direction T2 of segment 2; • tangent direction Tj of line segment joining the two end-points. If these directions are parallel within a pre-de"ned thresh◦ old p , the pair is marked as joinable. p is set to be 45 in the experiments. For example, in Fig. 2(b), DF curves 2 and 4, 3 and 6 are joinable pairs. The threshold of joinable pair identi"cation is obtained by trials. The probability of the join is to be estimated in the model building process. 3. Signature models Since signatures vary enormously, the aim of building signature models is to signify the individuality of each
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
2471
The DF curve segments are grouped by the four directions they are extracted from, i.e. up, down, left, right. Let R be a set of reference signature patterns belonging to a signature class. R = {g1 ; g2 ; : : : ; gNr }, where Nr is the number of reference samples. Each reference pattern gv (v ∈ [1 : : : Nr ]) consists of r v [1 : : : 4] curve segments: v v v v v v gv = {s11 ; s12 ; : : : ; s1r v [1] ; : : : ; s41 ; s42 ; : : : ; s4r v [4] }. Let t denote an input test pattern, which may also be one of the reference patterns. It consists of t[1 : : : 4] curve segments: t = {s11 ; s12 ; : : : ; s1t[1] ; : : : ; s41 ; s42 ; : : : ; s4t[4] }. The signature models Rm = {gm1 ; gm2 ; : : : ; gmNr } are designed to contain the DF curve correspondence relations, the means and variances of their locations, lengths, orientations, the variations of curve distances derived from their functional properties, and stability information. During model building and signature veri"cation, the pair-wise matched curve segments are identi"ed by a structural matching process, and the matching distance is calculated based on the interpolated points from "tted B-spline curves. Fig. 6. The neural network based signature model takes static handwriting features as input. The feature networks are three-layer MLPs, and the decision network is a two-layer MLP. The size of the feature grid indicates the feature extraction resolution. Three resolutions are illustrated, coarse (3 × 10), medium (4 × 15), and "ne (6 × 20).
signature class, and to di erentiate the stable and variable components and component groups. The signature veri"er is then able to emphasize the veri"cation of the relatively stable components and the structure of the stable component groups, while being tolerant towards the less critical variations. In this way, the veri"cation is made adaptive and robust to varying inputs. The signature model includes information about the geometric properties and the layout of the signature as an entity, as well as the stability and the signi"cance of the structural sub-components. 3.1. Pixel feature based signature model The pixel level signature model is described in Refs. [11,12], where the model is stored in a set of trained neural networks representing the signature shape at multiple resolutions (Fig. 6). The input features are simple geometric features, e.g. outline, thinning output. Each of the feature networks is constructed independently. The decision network, which receives input from all feature networks, is constructed after the all the feature networks have been trained. Supervised training method is used. 3.2. Structural-feature based signature model The structural model building process extracts the structural correlations within the reference signatures, and calculates the model parameters accordingly. It is centred on the DF features.
3.2.1. DF curve-set matching using relaxation labelling Structural matching of DF curves is based on a relaxation labelling method. A locally-optimal correspondence search with unary features is susceptible to noise and lacks a global metric. Clustering algorithms are mostly in this category, where a ‘hard-decision’ is made about the membership. Relaxation labelling algorithm is an optimising searching algorithm which utilises high-order feature correspondence and has a global metric. It permits inexactness in the number of clusters and in the actual membership of each cluster. The algorithm yields a set of ‘soft-decisions’, which can be viewed as matching con"dence values. Relaxation matching starts with an initial matrix of compatibility coeEcients denoting the matching likelihoods between the related two curve sets. The relaxation process then iteratively adjusts the compatibility coeEcients by reducing a global cost measurement. The global cost is formulated in terms of the unary and binary structural relations between the curve sets. The line-matching technique developed by Li [34] is adapted to solve the curve-matching problem. The input to the relaxation matching algorithm is two DF curve sets from the same direction. For direction k, the DF v v curve sets are A = {sk1 ; : : : ; skr v [k] } from the reference input, and B = {0; sk1 ; : : : ; skt[k] } from the test input. The element 0 in B is a special symbol used for null matches. For simplicity let i and p denote curves from A, let j and q denote curves from B. The correspondence between the two curve sets is represented as a mapping, : A → B. Let f(i; j) denote the probability of the mapping from i to j. It is a constraint that f(i; j) = 1; ∀i ∈ A: (2) j∈B
The mapping is expressed as a compatibility matrix C, which is of size r v [k] by t[k]. C consists of the compatibility coeEcients cij = f(i; j). The row and column numbers of
2472
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
C denote the reference and test curve components, respectively. The initial correspondence between the two curve sets is established on the following unary feature distances: • the distance ratios Ud[1–5] of the distances between the "ve corresponding points, i.e. the centre Bc , the two end-points, B0 ; Bn−1 , two via-points, and the maximum distance between the corresponding points, 0 ¡ Ud[1–5] 6 1; • the ratios U#[1–5] of the absolute di erences in the tangent directions at the above points and the maximum di erence between corresponding points, 0 ¡ Ud[1–5] 6 1; • the DF curve length ratio, Ul , 0 ¡ Ul 6 1. The noise-free 0=0 case is assigned the value 1. Since the curves are directional while calculating these feature distances, the two curve directions are considered as two separate matching instances, and the column size of matrix C is doubled. In the initial matching state, the compatibility coeEcients cij are set to be the normalised product of the unary feature distances Ud[1–5] (i; j) U#[1–5] (i; j) Ul (i; j), such that 0 ¡ cij 6 1. Binary relational features of the curve set A or B are obtained when the above distance ratios are evaluated between all the pairs of DF curves within their respective sets. For example, if the mapping from i to j and p to q is to be considered, the unary feature similarity $1 (i; j) is: a[1–5] = U [1–5] (i; j); #
1
d1[1–5]
= Ud[1–5] (i; j);
l1 = Ul (i; j); $1 (i; j; p; q) = exp(−log(a1[1–5] ) − log(d1[1–5] ) − log(l1 )) (3) and the binary relational feature similarity $2 (i; j; p; q) between curves i; p ∈ A and j; q ∈ B is expressed with: a[1–5] = U [1–5] (i; p) − U [1–5] (j; q); #
2
d2[1–5]
=
#
Ud[1–5] (i; p)
− Ud[1–5] (j; q);
l2 = Ul (i; p) − Ul (j; q); $2 (i; j; p; q) = exp(−log(a2[1–5] ) − log(d2[1–5] ) − log(l2 )): (4) The global cost function E(f) containing terms upto the second order is $1 (i; j)f(i; j) E(f) = '1 +'2
$2 (i; j; p; q)f(i; j)f(p; q) + (;
(5)
where 'n are the weights of the contributions from the nth order relational similarities, and ( is used to denote the contributions from higher-order terms.
The compatibility matrix C is re"ned iteratively by the relaxation matching algorithm, where a gradient vector g(i; j) is generated for minimising the cost function, 9E(f) 'n gn (i; j) = 9f(i; j) n=1 2
g(i; j) =
(6)
in which gn (i; j) =
9En (f|$n ) : 9f(i; j)
(7)
The "rst-order gradient g1 (i; j) = $1 (i; j)
(8)
and the second-order gradient g2 (i; j) = [$2 (i; j; p; q) + $2 (p; q; i; j)]f(p; q):
(9)
It is used to update the f(i; j) terms in matrix C via a gradient projection operation, one row at a time, until the algorithm converges, i.e. C is not updated for two successive iterations. An upper limit is imposed on the iterative matrix update, so that if the algorithm does not converge after a "nite number of iterations, a null match is assigned. For further details of the algorithm, readers are referred to Ref. [34] and references there in. The output of the relaxation matching algorithm is the matching hypotheses. The non-matches correspond to the null entries in the matrix. The matches identi"ed by the algorithm include both correct matches and possibly erroneous matches. Further validation may be required because mis-identi"ed matches will a ect the accuracy of the statistical parameters in the signature model. 3.2.2. Large-gap reconnection veri7cation On the issue of large gap "lling or treatment of broken segments, the large-gap reconnection procedure produces a list of joinable stroke pairs within each DF curve set. These are considered as virtual segments, which are physically disjoint on the extracted frontier maps, but they are conceptually jointed in the reference models as extra DF curve segments. These segments are matched simultaneously with the physical DF curve segments under the relaxation process as alternative choices. The test pattern serves as an evaluator for their inclusion, such that the con"guration with the least overall distance is chosen. 3.2.3. Statistical model parameter extraction The model parameters are established with the following procedure. One signature is arbitrarily selected from R as a template, and all its DF curves are assigned unique labels. The other signatures in R are matched against the template using the relaxation algorithm. Labels of the template DF curves are transferred to the matched DF curves. DF curves with the same labels are grouped into clusters. Within these clusters, statistical model parameters such as the means and variances are extracted. The procedure is repeated, so that each reference signature in turn serves as a template.
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
sample. The weighting factor of DF segment skiv from the template signature pv is calculated using:
60 "relx.dat"
1 2 3 4
50
5
6
40
119
13
30
7 8 12
10
2473
length(skiv ) × SI (skiv ) weight(skiv ) = r v [k] v v l=1 length(skl ) × SI (skl )
14
20
(11)
and the weighting factor of skj from the test signature t in the kth DF curve set is calculated using:
10 0 1 2 3 4
-10 -20
119
13
-30
7 8 12
10
length(skj ) weight(skj ) = t[k] : l=1 length(skl )
5
6
(12)
14
4. Signature veri cation against model
-40 -50 0
20
40
60
80
100
120
140
160
180
(a) 60 50
2
"relx.dat"
1
3 3 5
40 9
14
12
30
14
20 10 0 1 2 3
-10
5
4.1. Structural distance metrics
-20 9
-30
12
14
-40 -50 0
20
40
60
80
(b)
100
120
140
160
Fig. 7. Pen–stroke correspondence between two signatures. (a) The stroke extraction and labelling of the template signature. (b) A second signature (upper) with identi"ed correspondence relation to the template (lower).
Fig. 7 shows an example of the identi"ed correspondence relations between a pair of similar signatures. One signature is the template, with all the extracted strokes labelled by unique numbers (Fig. 7(a)). The upper part of Fig. 7(b) shows the strokes from the second signature whose corresponding strokes can be found in the template in the lower part. The stability index (SI) of a DF curve in the reference signature is obtained by dividing the sum of compatibility coeEcients of its matching relations by the number of reference signatures Nr . For the DF segment skiv , in the kth DF map of reference pattern pv ∈ R, SI (skiv ) =
1 Nr
r l [k] Nr
The system diagram of the signature veri"cation algorithm is shown in Fig. 8. The DF feature of the input test signature is correlated to the reference models under relaxation matching. The acceptance or rejection of each individual DF curve in the test signature is evaluated by comparing the curve’s feature distances with the pre-stored thresholds corresponding to its matched model DF curves. After all local decisions are made, an integrated global distance value is obtained by combining the local matching distances.
The correspondence establishment by relaxation matching can be viewed as a coarse structural veri"er, where the presence and absence of the DF curves in each direction are examined. The functional distances between related strokes are computed in the detailed veri"cation with the aid of B-spline curves. The distance metrics formulated for online signature curves are adopted here [35]. If Sp is a model DF curve consisting of n sample points, and Sq is the corresponding DF curve in the test pattern, Sq is reconstructed from its B-spline representation to be of the same dimension as Sp . The shape change between these curves is d(Sp ; Sq ), which includes three components; the accumulated square of the Euclidean distance between the sample points, the internal strain energy term, and the internal bending energy term. It is a measure of the elastic deformation between the two loci Sp and Sq . If the deformation from Sp to Sq is expressed by s = Sp − Sq , where s [l] = Sp [l] − Sq [l]; ∀l ∈ [1 : : : n], then the Euclidean distance term Ed is given by Ed [l] =
n
Sp [l] − Sq [l] 2 ;
(13)
l=1
cij :
(10)
l=1 j=1
For the input test signature t, the stability index for each DF curve has the default value one, since there is only one
the strain energy term Se is given by {ds =ds}2 ds: 2 dSp [l] dSq [l] Se [l] = (14) − ds ds
2474
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
DF feature Reference extraction Sample 1
DF curves (4 sets)
Relaxation Matching
Reference Model Construction
Correspondence Relations Reference Sample n
DF curves (4 sets)
Parameters Structural model Extraction Parameters and Distance Estimation Thresholds
Relaxation Matching Test Sample
DF feature extraction
DF curves (4 sets)
Distance Estimation
Correspondence Relations
Decision
Fig. 8. Diagram of the oPine signature veri"cation system based on the DF curve structural matching algorithm.
and the bending energy term Be is given by {d 2 s =ds2 }2 ds: 2 2 d Sp [l] d 2 Sq [l] : − (15) Be [l] = ds2 ds2 The "rst and second-order derivatives in the above equations are obtained analytically from the cubic B-Spline versions of the DF curves. Let d[l] be the point-wise distance between corresponding samples on Sp and Sq . (16) d[l] = #Ed [l] + .Se [l] + /Be [l]; ∀l ∈ [1 : : : n]; where #; . and / are weighting constants denoting the contributions of the distance components, with # + . + / = 1. Finally the distance d(Sp ; Sq ) is expressed as d(Sp ; Sq ) =
n
![l]d[l];
dm [t] ; dm [l]
t=1
=
r [k]
v weight(skl );
v skl → null;
weight(skl );
skl → null:
l=0 t[k]
Dk (null) =
(19)
l=0
The other match criteria are based on functional distances. The overall structural matching distance of the kth DF curve set is v ; skq ) d(skp v v dtotal [k] = → skq : (20) ; ∀skp v 1 − Dk (null) The threshold of the structural matching of the overall DF curve set is determined as the maximum distance value when all the reference signatures within the same class are tested against the template. 5. System implementation and result
where ! is a weighting function related to the stability of the signature. The stability weighting function is the normalised point-wise inverse of the maximum deviations found within the reference signatures. For a DF curve sirv v [k] from a reference signature gv , spatially uniform-resampled into nvi points, let the maximum deformation value for each resampled point l be dm [l] within the reference signature set R. !iv is de"ned to be nvi
v
Dkv (null)
(17)
l=1
!iv [l] =
direction k,
∀l ∈ [1 : : : nvi ]:
(18)
In the relaxation matching output, there are three kinds of matches which include: the null matches, where the compatibility coeEcients are zero; the perfect matches, where the compatibility coeEcients are one; and the in-between matches. The accumulated weightings of null matches in the overall matches are an indicator of how similar the overall structures are. It is one of the match criteria. The accumulated null matching rates are obtained within each DF
5.1. Signature database The signature database consists of 8904 signature images. They are scanned at a resolution of 100 dpi, 8-bit grey-scale, and are organised into 53 sets, where each set corresponds to one signature enrollment. There are 24 genuine and 144 forgery signatures in a set. Each volunteer is asked to sign a page of their own signatures and to imitate up to three pages of other people’s signatures, given photocopies of the originals. The forgeries are either freehand or traced, encompassing varying skill levels (Fig. 9). The data samples are divided into training sets and testing sets. For each new signature enrollment, eight reference signatures out of the 24 samples are used to generate the training data for this signature. The rest of the genuine samples and all the forgery samples are used as testing data. To overcome the shortage of training data, arti"cially generated samples are employed in the training process. The training set consist of a subset of the available genuine signature specimen, and computer generated true and false samples. The
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
2475
Fig. 9. Examples of genuine and forged signatures. Genuine samples are on the left, and forged samples are on the right. The forged signatures are either simple freehand, skilled freehand or traced. (a) and (b) show common cursive type signature samples and forgeries, (c) shows graphical type signatures, and (d) shows oriental type signatures.
generated samples are obtained by applying aEne transformations, e.g. size, slant, and perspective distortions, to the genuine samples [12]. 5.2. Experiment result In the combined signature veri"cation method, the neural-network-based oPine signature veri"er [12] is used as a "rst stage classi"er which assigns labels of ‘pass’, ‘fail’, and ‘questionable’ to the input test signatures. The test signatures labelled as ‘questionable’ are handed to the structural matching algorithm for further examination. Approximately 20% of the total input signatures are examined by the structural matching algorithm (Table 1).
Table 1 Experimental result on signature database for the targeted forgery test of the "rst-stage classi"cation Total Genuine 848 Forgery 7632
Accept
Reject
Undecided
720 (84.9%) 273 (3.6%)
19 (2.2%) 5843 (76.6%)
109 (12.9%) 1516 (19.8%)
The structural matching method has been successful on many test signatures (Table 2). The higher order relational similarity, in this case the second-order one, is given more emphasis in the matching. It is achieved by setting the parameters 'n for the weighting of the nth order relational
2476
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
Table 2 Experimental result on signature database for the targeted forgery test of the second-stage classi"cation
Genuine Forgery
Accept
Reject
75 (68.8%) 352 (23.2%)
34 (31.2%) 1164 (76.8%)
Table 3 Experimental result on signature database for the targeted forgery test of the combined classi"er
Genuine Forgery
Accept
Reject
795 (93.7%) 625 (8.2%)
53 (6.3%) 7007 (91.8%)
similarity, i.e., '1 =0:3, and '2 =0:7. The DF feature appears problematic in some signature samples, and the structural matching algorithm recommends rejections on genuine inputs. The structure matching method is sensitive to changes in the structural appearance. The parameters for calculating the distance between curves are set to be # = 0:3; . = 0:4; / = 0:3 in the experiment. The "nal test result on targeted forgeries is shown in Table 3. 6. Conclusion A structural feature correlation and registration technique for o -line signature veri"cation has been presented. The DF feature is used as a structural descriptor of the signature. It is readily obtainable using established contour following techniques and it is closedly related to the handwriting strokes. DF curves are extracted from the binarised signature pixel map and converted into analytic B-spline curves and structurally matched. The DF feature is not essential in the structural matching algorithm. Other structural constructs such as skeletons or segmented contours can be matched similarly when they are reliably extracted. Relaxation matching is used to establish correspondence between DF curve sets from the reference signatures. A structural distance metric is formulated to classify whether an input signature image belongs to the genuine samples. Coupled with the neural net classi"ers, the overall performance of the o -line signature veri"cation system is improved. Further improvement can be expected when the structural matching classi"er is combined with other established methods in a multi-expert con"guration. References [1] G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, Automatic bankcheck processing: a new engineered system, Int. J. Pattern Recognition Artif. Intell. 11 (4) (1997) 467–504.
[2] R. Plamondon, G. Lorette, Automatic signature veri"cation and writer identi"cation: the state of the art, Pattern Recognition 22 (2) (1989) 107–131. [3] R. Plamondon (Ed.), Special issue on automatic signature veri"cation, Int. J. Pattern Recognition Artif. Intell. 8 (3) (1994). [4] R. Plamondon, S.N. Srihari, On-line and o -line handwriting recognition: a comprehensive survey, IEEE Trans. Pattern Anal. Mac. Intell. 14 (1) (2000) 3–19. [5] Y. Sato, K. Kogure, Online signature veri"cation based on shape, motion, and writing pressure, Proceedings of the Sixth International Conference on Pattern Recognition, Munich, 1982, pp. 823–826. [6] K. Huang, H. Yan, On-line signature veri"cation based on dynamic segmentation and global and local matching, Opt. Eng. 34 (12) (1995) 3480–3487. [7] M. Ammar, Progress in veri"cation of skillfully simulated handwritten signature images, Int. J. Pattern Recogn. Artif. Intell. 5 (1991) 337–351. [8] R. Sabourin, J.P. Drouhard, O -line signature veri"cation using directional PDF and neural networks, Proceedings of the 11th IAPR International Conference on Pattern Recognition, 1992, pp. 321–325. [9] G. Krishnan D. Jones, Machine veri"cation of traced signatures, Proceedings of the SPIE International Society for Optical Engineering, 1991, pp. 563–572. [10] F. Nouboud, Handwritten signature veri"cation: a global approach, in: S. Impedovo (Ed.), Fundamentals in Handwriting Recognition, Springer, Berlin, Heidelberg, 1994, pp. 455–459. [11] K. Huang, H. Yan, Identifying and verifying handwritten signature images utilizing neural networks, Proceedings of the ICONIP’96, Hong Kong, September 1996, pp. 1400 –1404. [12] K. Huang, H. Yan, O -line signature veri"cation based on geometric feature extraction and neural network classi"cation, Pattern Recognition 30 (1) (1997) 9–17. [13] K. Huang, J. Wu, H. Yan, O -line writer veri"cation utilizing multiple neural networks, Opt. Eng. 36 (11) (1997) 3127–3133. [14] R. Sabourin, M. Cheriet, G. Genest, An extended-shadowcode based approach for o -line signature veri"cation, Proceedings of the Second ICDAR 93: International Conference on Document Analysis and Recognition, Tsukuba Science City, Japan, 1993, pp. 1– 6. [15] R. Sabourin, G. Genest, An extended-shadow-code based approach for o -line signature veri"cation: Part-IIevaluation of several multi-classi"er combination strategies, Proceedings of the Third ICDAR 95: International Conference on Document Analysis and Recognition, Montreal, Canada, 1993, pp. 1–5. [16] Yingyong Qi, Bobby R. Hunt, A multiresolution approach to computer veri"cation of handwritten signatures, IEEE Trans. Image Process. 4 (6) (1995) 870–874. [17] R. Sabourin, G. Genest, F. Preteux, OPine signature veri"cation by local granulometric size distributions, IEEE Trans. PAMI 19 (8) (1997) 976–988. [18] C. Liu, R. Dai, Y. Liu, Extracting individual features from moments for chinese writer identi"cation, Proceedings of the Third IEEE International Conference on Document Analysis and Recognition, Montreal, Canada, August 1995, pp. 438– 441.
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477 [19] H. Cardot, Marinette Revenu, A static signature veri"cation system based on a cooperative neural networks architecture, Int. J. Pattern Recognition Artif. Intell. 8 (3) (1994) 679–692. [20] N.A. Murshed, R. Sabourin, F. Bortolozzi, A cognitive approach to o -line signature veri"cation, Int. J. Pattern Recognition Artif. Intell. 11 (5) (1997) 801–825. [21] G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, A multi-expert signature veri"cation system for bankcheck processing, Int. J. Pattern Recognition Artif. Intell. 11 (5) (1997) 827–844. [22] C.P. Codella, P. Foggia, C. Sansone, M. Vento, Document validation by signatures: a serial multi-expert approach, Proceedings of the International Conference on Document Analysis and Recognition (ICDAR’99), Bangalore, India, September 1999, pp. 601– 604. [23] M. Ammar, Y. Yoshida, T. Fukumura, A new e ective approach for o -line veri"cation of signatures by using pressure features, Proceedings of the Eight International Conference on Pattern Recognition, Paris, 1986, pp. 566 –569. [24] Jinhong K. Guo, David Doermann, Azriel Rosenfeld, Local correspondence for detecting random forgeries, International Conference on Document Analysis and Recognition, Ulm, Germany, 1997, pp. 319 –323. [25] Yshiharu Kato, Makoto Yasuhara, Recovery of drawing order from single-stroke handwriting images, IEEE Trans. PAMI 22 (9) (2000) 938–949. [26] G. Boccignone, A. Chianese, L.P. Cordella, A. Marcelli, Recovering dynamic information from static handwriting, Pattern Recognition 26 (3) (1993) 409–418. [27] S. Lee, J.C. Pan, OPine tracing and representation of signatures, IEEE Trans. Systems Man Cybernet. 22 (4) (1992) 755–771.
2477
[28] H. Bunke, R. Kaufmann, T.M. Ha, M. Schenkel, R. Seiler, F. Eggimann, Recovery of temporal information of cursively handwritten words for on-line recognition, International Conference on Document Analysis and Recognition, Ulm, Germany, 1997, pp. 931–935. [29] P.M. Lallican, C. Viard-Gaudin, O -line handwriting modelling as a trajectory problem, in: Advances in Handwriting Recognition, Series in Machine Perception and Arti"cial Intelligence, Vol. 34, World Scienti"c, Singapore, 1999, pp. 449 – 458. [30] D.S. Doermann, A. Rosenfeld, Recovery of temporal information from static images of handwriting, Int. J. Comput. Vision 15 (1–2) (1995) 143–164. [31] R. Plamondon, C.M. Privitera, The segmentation of cursive handwriting: an approach based on o -line recovery of the motor-temporal information, IEEE Trans. Image Process. 8 (1) (1999) 80–91. [32] R.C. Gonzalez, R.E. Woods, Digital Image Processing, Addison-Wesley, Reading, MA, 1992. [33] Leslie Piegl, Wayne Tiller, Curve and surface constructions using rational b-splines, Comput. Aided Des. 19 (9) (1987) 485–498. [34] S.Z. Li, Matching: invariant to translations, rotations and scale changes, Pattern Recognition 25 (6) (1992) 583–594. [35] K. Huang, H. Yan, Signature veri"cation using fractal transformation, Proceedings of the 15th IAPR International Conference on Pattern Recognition, Barcelona, September 2000, pp. 855 –858.
About the Author—KAI HUANG received his B.Sc. and B.E. degree in 1991 and 1993, respectively, from the University of Sydney. He is a Ph.D. candidate in the School of Electrical and Information Engineering at the same university. His research interests include pattern recognition, signal and image processing, and automatic handwritten signature veri"cation. About the Author—HONG YAN received a B.E. degree from Nanking Institute of Posts and Telecommunications in 1982, an M.S.E. degree from the University of Michigan in 1984, and a Ph.D. degree from Yale University in 1989, all in Electrical Engineering. In 1982 and 1983 he worked on signal detection and estimation as a graduate student and research assistant at Tsinghua University. From 1986 to 1989 he was a research scientist at General Network Corporation, New Haven, CT, USA, where he worked on design and optimization of computer and telecommunications networks. He joined the University of Sydney in 1989 and became Professor of Imaging Science in the School of Electrical and Information Engineering in 1997. He is currently Professor of Computer Engineering at the City University of Hong Kong. His research interests include bioinformatics, computer animation, signal and image processing, pattern recognition, neural and fuzzy algorithms. He is author or co-author of one book and over 200 refereed technical papers in these areas. Professor Yan is a fellow of the International Association for Pattern Recognition (IAPR), a fellow of the Institution of Engineers, Australia (IEAust), a senior member of the Institute of Electrical and Electronic Engineers (IEEE), and a member of the International Society for Optical Engineers (SPIE), the International Neural Network Society (INNS), and the International Society for Magnetic Resonance in Medicine (ISMRM).
www.elsevier.com/locate/patcog
O -line signature veri"cation using structural feature correspondence Kai Huang ∗ , Hong Yan School of Electrical and Information Engineering, University of Sydney, NSW 2006, Australia Received 27 March 2001; accepted 13 September 2001
Abstract This paper presents an o -line signature veri"cation method using a model-based approach. In this method, statistical models are constructed for both pixel distribution and structural layout description. In addition to simple geometric handwriting features, it is proposed to use the directional frontier feature as a structural descriptor of the signature. The statistical veri"cation algorithm based on the geometric handwriting feature is used to accept signatures which closely resemble the reference samples, and to reject random and less skilled forgeries. For the questionable signatures for which the pixel feature judgement is inconclusive, the structural feature veri"cation algorithm is invoked. This algorithm compares the detailed structural correlation between the input and reference signatures in an attempt to detect skilled forgeries. The e ectiveness of the approach is evaluated on an experimental signature database. ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. Keywords: Signature veri"cation; Structural model; Directional frontier; Relaxation matching
1. Introduction A handwritten signature, being a behaviour biometrics, is well accepted socially and legally as a convenient mean of authorisation and identi"cation. Many documents, e.g. forms, necessitate the signing of a signature. As a consequence, o -line signature veri"cation becomes an essential component in automating the processing of o -line documents with embedded signatures [1]. Although on-line transactions are common-place, there are still large amounts of present and historical o -line data which need to be processed. It is bene"cial, for the ease of manipulation and information sharing, to convert them into electronic form. Automated machine recognition=veri"cation systems hold the promise of performing these repetitive tasks consistently and at high speed. With this motivation, continuous research e orts are being invested in handwriting recognition and veri"cation [2– 4]. ∗ Corresponding author. Tel.: +61-2-981-866-33; +612-981-042-33. E-mail address: [email protected] (K. Huang).
fax:
In general, o -line signature veri"cation is a challenging problem. Unlike for the on-line signature, where dynamic aspects of the signing action are captured directly as a handwriting trajectory [5,6], the dynamic information contained in o -line signature is highly degraded. Handwriting features, such as the handwriting order, writing-speed variation, and skillfulness, are to be recovered from the greylevel pixels by utilising intelligent machine algorithms. Consequently, an o -line veri"cation system has to cope with a signi"cant amount of errors and uncertainties in the recovered data. These diEculties are not present in the on-line case. Signatures are special symbols, which are usually cursively written and graphical in appearance. It is often diEcult, even for humans at instances, to segment the signature images into letters or basic lexical constructs, and to recover the underlying handwriting sequence. The common approaches to o -line signature veri"cation have been to exploit the static features of the handwriting, treating the complete signature as a single entity. These techniques involve the analysis and comparison of image projections [7], gradient features [8,9], geometric features
0031-3203/02/$22.00 ? 2002 Pattern Recognition Society. Published by Elsevier Science Ltd. All rights reserved. PII: S 0 0 3 1 - 3 2 0 3 ( 0 1 ) 0 0 2 2 2 - 9
2468
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
[10 –13], shadow-code descriptors [14,15], transform features [16,17], and moment features [18]. Recently, neural networks [19,20] and multi-expert veri"cation techniques [21,22] have appeared as viable and robust methodologies in system implementation. Pseudo dynamic features have been incorporated to an extent in o -line signature veri"cation, e.g. pressure features [23]. A technique of o -line signature model construction and matching utilising hand-traced pseudo-dynamic feature has also been proposed [24]. In recent years, numerous attempts have been made in recovering temporal information from static handwritings and drawings [25]. These techniques work either from the thinning output of a binarised image [26 –29] or from the gradient and edge information of a grey-level image [30,31]. The recovery process is based on assumptions of local and global smoothness and continuity criteria. Handwriting parameters such as stroke direction, length, width, and curvature variation are estimated with these techniques. It can be expected that practical diEculties in utilising the recovered temporal data will soon be overcome, that new o -line signature veri"cation systems will bene"t from them. This paper presents a new method for o -line signature veri"cation using a combination of static image pixel features and pseudo-dynamic structural features. Instead of solving the problem of recovering the actual handwriting strokes in generating the signature, the structural feature named directional frontier is chosen as the basis for the veri"er. Directional frontiers are derived from image contours, which can be readily extracted using established techniques [32]. They are closely related to each of the handwriting strokes locally, and to the shape of the signature in a global sense. The structural matching and veri"cation technique is general enough to allow other types of structural descriptors to be used with little modi"cation, e.g. the curvature-segmented contours or the recovered handwriting strokes. In this technique, a signature is considered as an image entity as well as a group of structural entities. A procedure of generating statistical and structural models from a set of reference samples is formulated. The pixel feature model of the veri"er is based on neural networks and is used in the "rst stage veri"cation to eliminate random and less skilled forgeries. The "ltering process rejects dissimilar signatures and accepts well formed signatures. For the questionable signatures for which the pixel feature judgement is inconclusive, the structural feature veri"er is invoked. The structural veri"er compares the detailed structural correlation between input and reference signatures in order to detect skilled forgeries. The paper is organised as follows. First the directional frontier feature of a signature image is described. Then the signature modelling technique based on the static and pseudo-dynamic handwriting features are discussed. Finally the veri"cation strategies and experimental results are presented.
2. Directional frontier feature Directional frontier (DF) is a directional grouping of the contour pixels. It is the set of black pixels of the handwriting image whose left (right, up, down, etc.) neighbors are white (Fig. 1). The information in the DF bitmap is organised by pixel labelling and curve tracing, to generate a set of structural descriptors of the signature. The foreground pixels are labelled using the following scheme: an ‘I’ for an isolated pixel; an ‘E’ for an end pixel; and a ‘C’ for a connecting pixel (Fig. 2a). The bitmap is converted into a set of DF curves by applying a tracing algorithm (Fig. 2b). From visual inspection, the DF curves relate well to the local handwriting strokes, and to the global shape of the signature. E ectively, the signature is transformed from pixels into a set of stroke-like DF curves. 2.1. DF curve tracing In the pixel labelling process, ‘E’ labelled pixels are stored in a list. Tracing starts at the "rst ‘E’ labelled pixel in the list. The algorithm "rst checks the four-connected neighbours for the next tracing pixel before checking the 8-connected ones [32].
Algorithm of tracing DF curves. place pixels labelled ‘E’ into a ‘E’ list; i = 0; while ‘E’ list contains not “visited” elements { mark the "rst ‘E’ pixel “visited” on the ‘E’ list; i = i + 1; create new DF curve T[i]; trace until another ‘E’ pixel is encountered { add current pixel to T[i]; check the 4-connected neighbours for next pixel; check the 8-connected neighbours if not found. } add encountered ‘E’ pixel to T[i], and mark it “visited” on the ‘E’ list. }
2.2. DF curve characterisation A DF curve Ui of n points is approximated by a cubic spline Bi , Ui (xi ; yi ) = Bi (xi ; yi ) + Wi (xi ; yi );
i = i0 ; i1 ; : : : ; in−1 ;
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
2469
Fig. 1. Example DF maps, showing the contours and the directional frontiers of a signature image.
Fig. 2. (a) DF curve elements after pixel labelling. (b) DF curves extracted from pixel map, showing both the jagged and B-spline smoothed curves.
Bi (xi ; yi ) =
3
Bk[3] ( i )Pk ;
For convenience in curve matching, these DF parameters are stored or pre-calculated (Fig. 3):
i = i0 ; i1 ; : : : ; in−1 ;
k=0
Bk[3] ( i ) =
3 k
(1 − i )3−k ki ;
i =
i − i0 ; in−1 − i0
(1)
where Wi ’s are noise terms. Pk are the control points of the cubic spline [33]. The position of control points are estimated using the least-squares method. The analytic spline approximation is used in the curve shape matching calculation later in the paper.
• the direction of the map it is from, i.e. left, right, up, or down; • the control points Pk for reconstructing the curve; • the coordinates of the centre point Bc , the end points B0 , Bn−1 , and two via points, i.e. a reconstruction in "ve points; • the tangents of reconstructed curve at these points above; • the length after smoothing l = n−1 i=1 Bi − Bi−1 .
2470
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
S (E)
S (E)
T E
S (E)
T
T
E
E
E(C)
E(C)
C C
E(C)
C
C
C
C
Fig. 4. Small-gap reconnection templates, where ‘T’ denotes the blank pixel under test, ‘I’ for isolated pixels, and ‘E’ for end pixels. Note that the reNected, mirrored and rotated versions of the templates are also applied.
S2
S2 T2
T2
Tj
Tj
Fig. 3. Features used for characterising a DF curve segment. S1
2.3. Methods for reducing noise A noisy image boundary would introduce undesirable segmentation along a normally contiguous DF curve. Morphological operation, a closing followed by an opening with a square structural element, is used in preprocessing to smooth the foreground object boundaries. Further actions include small gap and large gap reconnections. 2.3.1. Small gap reconnection Small gap reconnection operates on the DF pixel maps. There are segments as well as the ‘I’ labelled pixels that are separated by one pixel. A 3 × 3 template is moved around the labelled pixel-map, to rectify this problem (Fig. 4). 2.3.2. Large gap reconnection The large gap reconnection process operates on the extracted set of analytic DF curves. The width of the gap is relatively large, which corresponds to the disturbance where one handwriting stroke is crossed-over by another stroke or by a background pattern. The reconnection process tries to identify these disturbances and then reconnect the pen-strokes accordingly. The large gap reconnection process produces a set of possible segment combinations with each combination attributed by a probability value. It involves checking the
T1 (a)
S1
T1 (b)
Fig. 5. Large-gap reconnections. In case (a), the reconnection is not possible because the link between the two segments is not in parallel with either the tangent directions at the joining ends of the segments. In case (b), all three directions are parallel, thus the join is permitted.
three characteristic directions between a pair of DF curves (Fig. 5): • end-point tangent direction T1 of segment 1; • end-point tangent direction T2 of segment 2; • tangent direction Tj of line segment joining the two end-points. If these directions are parallel within a pre-de"ned thresh◦ old p , the pair is marked as joinable. p is set to be 45 in the experiments. For example, in Fig. 2(b), DF curves 2 and 4, 3 and 6 are joinable pairs. The threshold of joinable pair identi"cation is obtained by trials. The probability of the join is to be estimated in the model building process. 3. Signature models Since signatures vary enormously, the aim of building signature models is to signify the individuality of each
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
2471
The DF curve segments are grouped by the four directions they are extracted from, i.e. up, down, left, right. Let R be a set of reference signature patterns belonging to a signature class. R = {g1 ; g2 ; : : : ; gNr }, where Nr is the number of reference samples. Each reference pattern gv (v ∈ [1 : : : Nr ]) consists of r v [1 : : : 4] curve segments: v v v v v v gv = {s11 ; s12 ; : : : ; s1r v [1] ; : : : ; s41 ; s42 ; : : : ; s4r v [4] }. Let t denote an input test pattern, which may also be one of the reference patterns. It consists of t[1 : : : 4] curve segments: t = {s11 ; s12 ; : : : ; s1t[1] ; : : : ; s41 ; s42 ; : : : ; s4t[4] }. The signature models Rm = {gm1 ; gm2 ; : : : ; gmNr } are designed to contain the DF curve correspondence relations, the means and variances of their locations, lengths, orientations, the variations of curve distances derived from their functional properties, and stability information. During model building and signature veri"cation, the pair-wise matched curve segments are identi"ed by a structural matching process, and the matching distance is calculated based on the interpolated points from "tted B-spline curves. Fig. 6. The neural network based signature model takes static handwriting features as input. The feature networks are three-layer MLPs, and the decision network is a two-layer MLP. The size of the feature grid indicates the feature extraction resolution. Three resolutions are illustrated, coarse (3 × 10), medium (4 × 15), and "ne (6 × 20).
signature class, and to di erentiate the stable and variable components and component groups. The signature veri"er is then able to emphasize the veri"cation of the relatively stable components and the structure of the stable component groups, while being tolerant towards the less critical variations. In this way, the veri"cation is made adaptive and robust to varying inputs. The signature model includes information about the geometric properties and the layout of the signature as an entity, as well as the stability and the signi"cance of the structural sub-components. 3.1. Pixel feature based signature model The pixel level signature model is described in Refs. [11,12], where the model is stored in a set of trained neural networks representing the signature shape at multiple resolutions (Fig. 6). The input features are simple geometric features, e.g. outline, thinning output. Each of the feature networks is constructed independently. The decision network, which receives input from all feature networks, is constructed after the all the feature networks have been trained. Supervised training method is used. 3.2. Structural-feature based signature model The structural model building process extracts the structural correlations within the reference signatures, and calculates the model parameters accordingly. It is centred on the DF features.
3.2.1. DF curve-set matching using relaxation labelling Structural matching of DF curves is based on a relaxation labelling method. A locally-optimal correspondence search with unary features is susceptible to noise and lacks a global metric. Clustering algorithms are mostly in this category, where a ‘hard-decision’ is made about the membership. Relaxation labelling algorithm is an optimising searching algorithm which utilises high-order feature correspondence and has a global metric. It permits inexactness in the number of clusters and in the actual membership of each cluster. The algorithm yields a set of ‘soft-decisions’, which can be viewed as matching con"dence values. Relaxation matching starts with an initial matrix of compatibility coeEcients denoting the matching likelihoods between the related two curve sets. The relaxation process then iteratively adjusts the compatibility coeEcients by reducing a global cost measurement. The global cost is formulated in terms of the unary and binary structural relations between the curve sets. The line-matching technique developed by Li [34] is adapted to solve the curve-matching problem. The input to the relaxation matching algorithm is two DF curve sets from the same direction. For direction k, the DF v v curve sets are A = {sk1 ; : : : ; skr v [k] } from the reference input, and B = {0; sk1 ; : : : ; skt[k] } from the test input. The element 0 in B is a special symbol used for null matches. For simplicity let i and p denote curves from A, let j and q denote curves from B. The correspondence between the two curve sets is represented as a mapping, : A → B. Let f(i; j) denote the probability of the mapping from i to j. It is a constraint that f(i; j) = 1; ∀i ∈ A: (2) j∈B
The mapping is expressed as a compatibility matrix C, which is of size r v [k] by t[k]. C consists of the compatibility coeEcients cij = f(i; j). The row and column numbers of
2472
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
C denote the reference and test curve components, respectively. The initial correspondence between the two curve sets is established on the following unary feature distances: • the distance ratios Ud[1–5] of the distances between the "ve corresponding points, i.e. the centre Bc , the two end-points, B0 ; Bn−1 , two via-points, and the maximum distance between the corresponding points, 0 ¡ Ud[1–5] 6 1; • the ratios U#[1–5] of the absolute di erences in the tangent directions at the above points and the maximum di erence between corresponding points, 0 ¡ Ud[1–5] 6 1; • the DF curve length ratio, Ul , 0 ¡ Ul 6 1. The noise-free 0=0 case is assigned the value 1. Since the curves are directional while calculating these feature distances, the two curve directions are considered as two separate matching instances, and the column size of matrix C is doubled. In the initial matching state, the compatibility coeEcients cij are set to be the normalised product of the unary feature distances Ud[1–5] (i; j) U#[1–5] (i; j) Ul (i; j), such that 0 ¡ cij 6 1. Binary relational features of the curve set A or B are obtained when the above distance ratios are evaluated between all the pairs of DF curves within their respective sets. For example, if the mapping from i to j and p to q is to be considered, the unary feature similarity $1 (i; j) is: a[1–5] = U [1–5] (i; j); #
1
d1[1–5]
= Ud[1–5] (i; j);
l1 = Ul (i; j); $1 (i; j; p; q) = exp(−log(a1[1–5] ) − log(d1[1–5] ) − log(l1 )) (3) and the binary relational feature similarity $2 (i; j; p; q) between curves i; p ∈ A and j; q ∈ B is expressed with: a[1–5] = U [1–5] (i; p) − U [1–5] (j; q); #
2
d2[1–5]
=
#
Ud[1–5] (i; p)
− Ud[1–5] (j; q);
l2 = Ul (i; p) − Ul (j; q); $2 (i; j; p; q) = exp(−log(a2[1–5] ) − log(d2[1–5] ) − log(l2 )): (4) The global cost function E(f) containing terms upto the second order is $1 (i; j)f(i; j) E(f) = '1 +'2
$2 (i; j; p; q)f(i; j)f(p; q) + (;
(5)
where 'n are the weights of the contributions from the nth order relational similarities, and ( is used to denote the contributions from higher-order terms.
The compatibility matrix C is re"ned iteratively by the relaxation matching algorithm, where a gradient vector g(i; j) is generated for minimising the cost function, 9E(f) 'n gn (i; j) = 9f(i; j) n=1 2
g(i; j) =
(6)
in which gn (i; j) =
9En (f|$n ) : 9f(i; j)
(7)
The "rst-order gradient g1 (i; j) = $1 (i; j)
(8)
and the second-order gradient g2 (i; j) = [$2 (i; j; p; q) + $2 (p; q; i; j)]f(p; q):
(9)
It is used to update the f(i; j) terms in matrix C via a gradient projection operation, one row at a time, until the algorithm converges, i.e. C is not updated for two successive iterations. An upper limit is imposed on the iterative matrix update, so that if the algorithm does not converge after a "nite number of iterations, a null match is assigned. For further details of the algorithm, readers are referred to Ref. [34] and references there in. The output of the relaxation matching algorithm is the matching hypotheses. The non-matches correspond to the null entries in the matrix. The matches identi"ed by the algorithm include both correct matches and possibly erroneous matches. Further validation may be required because mis-identi"ed matches will a ect the accuracy of the statistical parameters in the signature model. 3.2.2. Large-gap reconnection veri7cation On the issue of large gap "lling or treatment of broken segments, the large-gap reconnection procedure produces a list of joinable stroke pairs within each DF curve set. These are considered as virtual segments, which are physically disjoint on the extracted frontier maps, but they are conceptually jointed in the reference models as extra DF curve segments. These segments are matched simultaneously with the physical DF curve segments under the relaxation process as alternative choices. The test pattern serves as an evaluator for their inclusion, such that the con"guration with the least overall distance is chosen. 3.2.3. Statistical model parameter extraction The model parameters are established with the following procedure. One signature is arbitrarily selected from R as a template, and all its DF curves are assigned unique labels. The other signatures in R are matched against the template using the relaxation algorithm. Labels of the template DF curves are transferred to the matched DF curves. DF curves with the same labels are grouped into clusters. Within these clusters, statistical model parameters such as the means and variances are extracted. The procedure is repeated, so that each reference signature in turn serves as a template.
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
sample. The weighting factor of DF segment skiv from the template signature pv is calculated using:
60 "relx.dat"
1 2 3 4
50
5
6
40
119
13
30
7 8 12
10
2473
length(skiv ) × SI (skiv ) weight(skiv ) = r v [k] v v l=1 length(skl ) × SI (skl )
14
20
(11)
and the weighting factor of skj from the test signature t in the kth DF curve set is calculated using:
10 0 1 2 3 4
-10 -20
119
13
-30
7 8 12
10
length(skj ) weight(skj ) = t[k] : l=1 length(skl )
5
6
(12)
14
4. Signature veri cation against model
-40 -50 0
20
40
60
80
100
120
140
160
180
(a) 60 50
2
"relx.dat"
1
3 3 5
40 9
14
12
30
14
20 10 0 1 2 3
-10
5
4.1. Structural distance metrics
-20 9
-30
12
14
-40 -50 0
20
40
60
80
(b)
100
120
140
160
Fig. 7. Pen–stroke correspondence between two signatures. (a) The stroke extraction and labelling of the template signature. (b) A second signature (upper) with identi"ed correspondence relation to the template (lower).
Fig. 7 shows an example of the identi"ed correspondence relations between a pair of similar signatures. One signature is the template, with all the extracted strokes labelled by unique numbers (Fig. 7(a)). The upper part of Fig. 7(b) shows the strokes from the second signature whose corresponding strokes can be found in the template in the lower part. The stability index (SI) of a DF curve in the reference signature is obtained by dividing the sum of compatibility coeEcients of its matching relations by the number of reference signatures Nr . For the DF segment skiv , in the kth DF map of reference pattern pv ∈ R, SI (skiv ) =
1 Nr
r l [k] Nr
The system diagram of the signature veri"cation algorithm is shown in Fig. 8. The DF feature of the input test signature is correlated to the reference models under relaxation matching. The acceptance or rejection of each individual DF curve in the test signature is evaluated by comparing the curve’s feature distances with the pre-stored thresholds corresponding to its matched model DF curves. After all local decisions are made, an integrated global distance value is obtained by combining the local matching distances.
The correspondence establishment by relaxation matching can be viewed as a coarse structural veri"er, where the presence and absence of the DF curves in each direction are examined. The functional distances between related strokes are computed in the detailed veri"cation with the aid of B-spline curves. The distance metrics formulated for online signature curves are adopted here [35]. If Sp is a model DF curve consisting of n sample points, and Sq is the corresponding DF curve in the test pattern, Sq is reconstructed from its B-spline representation to be of the same dimension as Sp . The shape change between these curves is d(Sp ; Sq ), which includes three components; the accumulated square of the Euclidean distance between the sample points, the internal strain energy term, and the internal bending energy term. It is a measure of the elastic deformation between the two loci Sp and Sq . If the deformation from Sp to Sq is expressed by s = Sp − Sq , where s [l] = Sp [l] − Sq [l]; ∀l ∈ [1 : : : n], then the Euclidean distance term Ed is given by Ed [l] =
n
Sp [l] − Sq [l] 2 ;
(13)
l=1
cij :
(10)
l=1 j=1
For the input test signature t, the stability index for each DF curve has the default value one, since there is only one
the strain energy term Se is given by {ds =ds}2 ds: 2 dSp [l] dSq [l] Se [l] = (14) − ds ds
2474
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
DF feature Reference extraction Sample 1
DF curves (4 sets)
Relaxation Matching
Reference Model Construction
Correspondence Relations Reference Sample n
DF curves (4 sets)
Parameters Structural model Extraction Parameters and Distance Estimation Thresholds
Relaxation Matching Test Sample
DF feature extraction
DF curves (4 sets)
Distance Estimation
Correspondence Relations
Decision
Fig. 8. Diagram of the oPine signature veri"cation system based on the DF curve structural matching algorithm.
and the bending energy term Be is given by {d 2 s =ds2 }2 ds: 2 2 d Sp [l] d 2 Sq [l] : − (15) Be [l] = ds2 ds2 The "rst and second-order derivatives in the above equations are obtained analytically from the cubic B-Spline versions of the DF curves. Let d[l] be the point-wise distance between corresponding samples on Sp and Sq . (16) d[l] = #Ed [l] + .Se [l] + /Be [l]; ∀l ∈ [1 : : : n]; where #; . and / are weighting constants denoting the contributions of the distance components, with # + . + / = 1. Finally the distance d(Sp ; Sq ) is expressed as d(Sp ; Sq ) =
n
![l]d[l];
dm [t] ; dm [l]
t=1
=
r [k]
v weight(skl );
v skl → null;
weight(skl );
skl → null:
l=0 t[k]
Dk (null) =
(19)
l=0
The other match criteria are based on functional distances. The overall structural matching distance of the kth DF curve set is v ; skq ) d(skp v v dtotal [k] = → skq : (20) ; ∀skp v 1 − Dk (null) The threshold of the structural matching of the overall DF curve set is determined as the maximum distance value when all the reference signatures within the same class are tested against the template. 5. System implementation and result
where ! is a weighting function related to the stability of the signature. The stability weighting function is the normalised point-wise inverse of the maximum deviations found within the reference signatures. For a DF curve sirv v [k] from a reference signature gv , spatially uniform-resampled into nvi points, let the maximum deformation value for each resampled point l be dm [l] within the reference signature set R. !iv is de"ned to be nvi
v
Dkv (null)
(17)
l=1
!iv [l] =
direction k,
∀l ∈ [1 : : : nvi ]:
(18)
In the relaxation matching output, there are three kinds of matches which include: the null matches, where the compatibility coeEcients are zero; the perfect matches, where the compatibility coeEcients are one; and the in-between matches. The accumulated weightings of null matches in the overall matches are an indicator of how similar the overall structures are. It is one of the match criteria. The accumulated null matching rates are obtained within each DF
5.1. Signature database The signature database consists of 8904 signature images. They are scanned at a resolution of 100 dpi, 8-bit grey-scale, and are organised into 53 sets, where each set corresponds to one signature enrollment. There are 24 genuine and 144 forgery signatures in a set. Each volunteer is asked to sign a page of their own signatures and to imitate up to three pages of other people’s signatures, given photocopies of the originals. The forgeries are either freehand or traced, encompassing varying skill levels (Fig. 9). The data samples are divided into training sets and testing sets. For each new signature enrollment, eight reference signatures out of the 24 samples are used to generate the training data for this signature. The rest of the genuine samples and all the forgery samples are used as testing data. To overcome the shortage of training data, arti"cially generated samples are employed in the training process. The training set consist of a subset of the available genuine signature specimen, and computer generated true and false samples. The
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
2475
Fig. 9. Examples of genuine and forged signatures. Genuine samples are on the left, and forged samples are on the right. The forged signatures are either simple freehand, skilled freehand or traced. (a) and (b) show common cursive type signature samples and forgeries, (c) shows graphical type signatures, and (d) shows oriental type signatures.
generated samples are obtained by applying aEne transformations, e.g. size, slant, and perspective distortions, to the genuine samples [12]. 5.2. Experiment result In the combined signature veri"cation method, the neural-network-based oPine signature veri"er [12] is used as a "rst stage classi"er which assigns labels of ‘pass’, ‘fail’, and ‘questionable’ to the input test signatures. The test signatures labelled as ‘questionable’ are handed to the structural matching algorithm for further examination. Approximately 20% of the total input signatures are examined by the structural matching algorithm (Table 1).
Table 1 Experimental result on signature database for the targeted forgery test of the "rst-stage classi"cation Total Genuine 848 Forgery 7632
Accept
Reject
Undecided
720 (84.9%) 273 (3.6%)
19 (2.2%) 5843 (76.6%)
109 (12.9%) 1516 (19.8%)
The structural matching method has been successful on many test signatures (Table 2). The higher order relational similarity, in this case the second-order one, is given more emphasis in the matching. It is achieved by setting the parameters 'n for the weighting of the nth order relational
2476
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477
Table 2 Experimental result on signature database for the targeted forgery test of the second-stage classi"cation
Genuine Forgery
Accept
Reject
75 (68.8%) 352 (23.2%)
34 (31.2%) 1164 (76.8%)
Table 3 Experimental result on signature database for the targeted forgery test of the combined classi"er
Genuine Forgery
Accept
Reject
795 (93.7%) 625 (8.2%)
53 (6.3%) 7007 (91.8%)
similarity, i.e., '1 =0:3, and '2 =0:7. The DF feature appears problematic in some signature samples, and the structural matching algorithm recommends rejections on genuine inputs. The structure matching method is sensitive to changes in the structural appearance. The parameters for calculating the distance between curves are set to be # = 0:3; . = 0:4; / = 0:3 in the experiment. The "nal test result on targeted forgeries is shown in Table 3. 6. Conclusion A structural feature correlation and registration technique for o -line signature veri"cation has been presented. The DF feature is used as a structural descriptor of the signature. It is readily obtainable using established contour following techniques and it is closedly related to the handwriting strokes. DF curves are extracted from the binarised signature pixel map and converted into analytic B-spline curves and structurally matched. The DF feature is not essential in the structural matching algorithm. Other structural constructs such as skeletons or segmented contours can be matched similarly when they are reliably extracted. Relaxation matching is used to establish correspondence between DF curve sets from the reference signatures. A structural distance metric is formulated to classify whether an input signature image belongs to the genuine samples. Coupled with the neural net classi"ers, the overall performance of the o -line signature veri"cation system is improved. Further improvement can be expected when the structural matching classi"er is combined with other established methods in a multi-expert con"guration. References [1] G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, Automatic bankcheck processing: a new engineered system, Int. J. Pattern Recognition Artif. Intell. 11 (4) (1997) 467–504.
[2] R. Plamondon, G. Lorette, Automatic signature veri"cation and writer identi"cation: the state of the art, Pattern Recognition 22 (2) (1989) 107–131. [3] R. Plamondon (Ed.), Special issue on automatic signature veri"cation, Int. J. Pattern Recognition Artif. Intell. 8 (3) (1994). [4] R. Plamondon, S.N. Srihari, On-line and o -line handwriting recognition: a comprehensive survey, IEEE Trans. Pattern Anal. Mac. Intell. 14 (1) (2000) 3–19. [5] Y. Sato, K. Kogure, Online signature veri"cation based on shape, motion, and writing pressure, Proceedings of the Sixth International Conference on Pattern Recognition, Munich, 1982, pp. 823–826. [6] K. Huang, H. Yan, On-line signature veri"cation based on dynamic segmentation and global and local matching, Opt. Eng. 34 (12) (1995) 3480–3487. [7] M. Ammar, Progress in veri"cation of skillfully simulated handwritten signature images, Int. J. Pattern Recogn. Artif. Intell. 5 (1991) 337–351. [8] R. Sabourin, J.P. Drouhard, O -line signature veri"cation using directional PDF and neural networks, Proceedings of the 11th IAPR International Conference on Pattern Recognition, 1992, pp. 321–325. [9] G. Krishnan D. Jones, Machine veri"cation of traced signatures, Proceedings of the SPIE International Society for Optical Engineering, 1991, pp. 563–572. [10] F. Nouboud, Handwritten signature veri"cation: a global approach, in: S. Impedovo (Ed.), Fundamentals in Handwriting Recognition, Springer, Berlin, Heidelberg, 1994, pp. 455–459. [11] K. Huang, H. Yan, Identifying and verifying handwritten signature images utilizing neural networks, Proceedings of the ICONIP’96, Hong Kong, September 1996, pp. 1400 –1404. [12] K. Huang, H. Yan, O -line signature veri"cation based on geometric feature extraction and neural network classi"cation, Pattern Recognition 30 (1) (1997) 9–17. [13] K. Huang, J. Wu, H. Yan, O -line writer veri"cation utilizing multiple neural networks, Opt. Eng. 36 (11) (1997) 3127–3133. [14] R. Sabourin, M. Cheriet, G. Genest, An extended-shadowcode based approach for o -line signature veri"cation, Proceedings of the Second ICDAR 93: International Conference on Document Analysis and Recognition, Tsukuba Science City, Japan, 1993, pp. 1– 6. [15] R. Sabourin, G. Genest, An extended-shadow-code based approach for o -line signature veri"cation: Part-IIevaluation of several multi-classi"er combination strategies, Proceedings of the Third ICDAR 95: International Conference on Document Analysis and Recognition, Montreal, Canada, 1993, pp. 1–5. [16] Yingyong Qi, Bobby R. Hunt, A multiresolution approach to computer veri"cation of handwritten signatures, IEEE Trans. Image Process. 4 (6) (1995) 870–874. [17] R. Sabourin, G. Genest, F. Preteux, OPine signature veri"cation by local granulometric size distributions, IEEE Trans. PAMI 19 (8) (1997) 976–988. [18] C. Liu, R. Dai, Y. Liu, Extracting individual features from moments for chinese writer identi"cation, Proceedings of the Third IEEE International Conference on Document Analysis and Recognition, Montreal, Canada, August 1995, pp. 438– 441.
K. Huang, H. Yan / Pattern Recognition 35 (2002) 2467 – 2477 [19] H. Cardot, Marinette Revenu, A static signature veri"cation system based on a cooperative neural networks architecture, Int. J. Pattern Recognition Artif. Intell. 8 (3) (1994) 679–692. [20] N.A. Murshed, R. Sabourin, F. Bortolozzi, A cognitive approach to o -line signature veri"cation, Int. J. Pattern Recognition Artif. Intell. 11 (5) (1997) 801–825. [21] G. Dimauro, S. Impedovo, G. Pirlo, A. Salzo, A multi-expert signature veri"cation system for bankcheck processing, Int. J. Pattern Recognition Artif. Intell. 11 (5) (1997) 827–844. [22] C.P. Codella, P. Foggia, C. Sansone, M. Vento, Document validation by signatures: a serial multi-expert approach, Proceedings of the International Conference on Document Analysis and Recognition (ICDAR’99), Bangalore, India, September 1999, pp. 601– 604. [23] M. Ammar, Y. Yoshida, T. Fukumura, A new e ective approach for o -line veri"cation of signatures by using pressure features, Proceedings of the Eight International Conference on Pattern Recognition, Paris, 1986, pp. 566 –569. [24] Jinhong K. Guo, David Doermann, Azriel Rosenfeld, Local correspondence for detecting random forgeries, International Conference on Document Analysis and Recognition, Ulm, Germany, 1997, pp. 319 –323. [25] Yshiharu Kato, Makoto Yasuhara, Recovery of drawing order from single-stroke handwriting images, IEEE Trans. PAMI 22 (9) (2000) 938–949. [26] G. Boccignone, A. Chianese, L.P. Cordella, A. Marcelli, Recovering dynamic information from static handwriting, Pattern Recognition 26 (3) (1993) 409–418. [27] S. Lee, J.C. Pan, OPine tracing and representation of signatures, IEEE Trans. Systems Man Cybernet. 22 (4) (1992) 755–771.
2477
[28] H. Bunke, R. Kaufmann, T.M. Ha, M. Schenkel, R. Seiler, F. Eggimann, Recovery of temporal information of cursively handwritten words for on-line recognition, International Conference on Document Analysis and Recognition, Ulm, Germany, 1997, pp. 931–935. [29] P.M. Lallican, C. Viard-Gaudin, O -line handwriting modelling as a trajectory problem, in: Advances in Handwriting Recognition, Series in Machine Perception and Arti"cial Intelligence, Vol. 34, World Scienti"c, Singapore, 1999, pp. 449 – 458. [30] D.S. Doermann, A. Rosenfeld, Recovery of temporal information from static images of handwriting, Int. J. Comput. Vision 15 (1–2) (1995) 143–164. [31] R. Plamondon, C.M. Privitera, The segmentation of cursive handwriting: an approach based on o -line recovery of the motor-temporal information, IEEE Trans. Image Process. 8 (1) (1999) 80–91. [32] R.C. Gonzalez, R.E. Woods, Digital Image Processing, Addison-Wesley, Reading, MA, 1992. [33] Leslie Piegl, Wayne Tiller, Curve and surface constructions using rational b-splines, Comput. Aided Des. 19 (9) (1987) 485–498. [34] S.Z. Li, Matching: invariant to translations, rotations and scale changes, Pattern Recognition 25 (6) (1992) 583–594. [35] K. Huang, H. Yan, Signature veri"cation using fractal transformation, Proceedings of the 15th IAPR International Conference on Pattern Recognition, Barcelona, September 2000, pp. 855 –858.
About the Author—KAI HUANG received his B.Sc. and B.E. degree in 1991 and 1993, respectively, from the University of Sydney. He is a Ph.D. candidate in the School of Electrical and Information Engineering at the same university. His research interests include pattern recognition, signal and image processing, and automatic handwritten signature veri"cation. About the Author—HONG YAN received a B.E. degree from Nanking Institute of Posts and Telecommunications in 1982, an M.S.E. degree from the University of Michigan in 1984, and a Ph.D. degree from Yale University in 1989, all in Electrical Engineering. In 1982 and 1983 he worked on signal detection and estimation as a graduate student and research assistant at Tsinghua University. From 1986 to 1989 he was a research scientist at General Network Corporation, New Haven, CT, USA, where he worked on design and optimization of computer and telecommunications networks. He joined the University of Sydney in 1989 and became Professor of Imaging Science in the School of Electrical and Information Engineering in 1997. He is currently Professor of Computer Engineering at the City University of Hong Kong. His research interests include bioinformatics, computer animation, signal and image processing, pattern recognition, neural and fuzzy algorithms. He is author or co-author of one book and over 200 refereed technical papers in these areas. Professor Yan is a fellow of the International Association for Pattern Recognition (IAPR), a fellow of the Institution of Engineers, Australia (IEAust), a senior member of the Institute of Electrical and Electronic Engineers (IEEE), and a member of the International Society for Optical Engineers (SPIE), the International Neural Network Society (INNS), and the International Society for Magnetic Resonance in Medicine (ISMRM).