Attributed stroke graph matching for seal imprint verification

Pattern Recognition Letters 9 (1989) 137 145 North-Holhmd

February 1989

Attributed stroke graph matchingfor seal imprint

verification Seongwhan LEE and Jin H. KIM* .4rtifwial lntelh'~ence Laboratory, Department 01" Computer Science, Korea Advanced lnslitute 01" Science and I~'chmdo~z)', P.O. Bo \ 150, ('heongryan~, Seoul 130-650, Korea P,eceived 16 March 1988 Revised 26 September 1988

.4hstract. In this paper, a new scheme** for automatic seal imprint verilication using attributed stroke graph matching is proposed. Our scheme can handle seal imprints under much less constraints. The shape of seals can be a square, an ellipse, a circle, or any close loop. The boundary of the seal imprints is allowed to be imperfect. Furthermore, it is insensitive to the distance between the TV-camera and seal imprints. Experimental results confirm that the proposed scheme might be feasible for practical applications. Key words'." Seal imprint verification, attributed stroke graph matching, fast minimum square error transl'orm.

1. Introduction For the purpose of entity identification, seals, also called chops, signets, or stamps, are extensively used in Korea, China, Japan and other oriental countries (Ueda and N a k a m u r a , 1984; Fan and Tsai, 1984: Chen and Tsai, 1986; Lee and Kim, 1988a). The entity may be a person, a social group, a department of the government, or even the country itself. Especially, seals are used instead of signatures on many types of articles such as paintings, documents, diplomas, checks, receipts, money withdrawing lists, etc. Therefore, it is highly desirable that the huge volume of such articles be verified automatically, fast and reliably. An annoying problem encountered in seal imprint verification is that the seal imprints may have a lot of variations, even if they are all produced from a single seal. This renders automatic seal ira* To whom the correspondence should be addressed. ** The basic idea of the new scheme using attributed stroke graph matching for seal imprint verification was first reported at the International Conference on Computer Processing of Chinese and Oriental Languages, Toronto, Canada, August 1988.

print verification very hard. Fortunately, as pointed out by Fan and Tsai (1984), the skeleton of each stroke is presumably topologically stable with a relatively fixed position in the seal imprint, and two hand-engraved seals will never be identical even if they are engraved by the same person. Thus, it is reasonable to assume that the distributions of the skeleton structures of two distinct seals will never be the same, and that the partial distribution of the skeleton structures in any seal imprint impressed with a single will never change (Fan and Tsai, 1984: Chen and Tsai, 1986). In contrast to other similarity measures (Ueda and N a k a m u r a , 1984; Fan and Tsai, 1984; Chen and Tsai, 1986) which are based on statistical characteristics only, a distance measure used in our scheme accounts for both the structural and the statistical aspects of the seal imprints under verification. We represent a seal imprint by a set of vertices and strokes. A vertex is an end or a joining point. and a stroke is a straight line or a curved line segment between vertices in a single-pixel-width linerepresentation of seal imprints. To both the vertices and the strokes, various attributes may be attached.

0167-8~55'89:S3.50 @ 1989, Elsevier Science Publishers B.V. (North-Holland)

137

Volume 9, Number 2

PATTERN RECOGNITION LETTERS

This representation is called ASG (attributed stroke graph) which is essentially a variation of ARG (attributed relational graph) (Bunke and Allermann, 1983; Groen et al., 1985). Thus, seal imprint verification is equivalent to ASG matching since the seal imprint can be represented by ASG. In other words, the problem of verifying an observed seal imprint can be formulated as the problem of matching an observed ASG (ASGo) to the model ASG (ASGM) and classifying the ASGo as one of the three types: genuine, forgery and ambiguous. Our scheme begins with an assumption that a particular set of vertices, what we call control vertices, are found reliably in the observation. The use of control vertices provides two advantages: First, the amount of data to be processed is reduced by considering only control vertices. Second, errors which may result from dealing with unstable vertices are minimized. In our current implementation, control vertices are selected manually from the ASG~ representation during the model learning phase. In doing so, some considerations should be made: First, control vertices must be stable. This implies that they must have a high probability of occurrence and, therefore, they are almost always present in ASGo's. Second, control vertices must contain reliable information about the pose of the ASGo. They should be in their general position in 2-D space. In other words, they must be neither too collinear nor too close to each other. The ASG matching proceeds as follows. First, an ASGo is constructed from a single-pixel-width linerepresentation of an observed seal imprint. Second, the pose of the ASGo is estimated in terms of translation, rotation and scale with respect to the ASGM by using the fast minimum square error transform (lee and Kim, 1988b). The search space is pruned by considering only control vertices and applying geometrical constraints in the early stage of matching. Third, a correspondence between the primitives of the normalized observed ASG (ASGNo)and those of the ASGM is found for the given pose. Note that ASG~ is the geometrically transformed ASG according to the minimum square error transform parameters with respect to the ASGM. Fourth, a distance measure between the ASG~ and the ASG~ is calculated based upon the correspondences. Finally, according to the distance measure, the ASG~ is 138

February 1989

classified into one of the types: genuine, forgery and ambiguous. Our scheme can handle seal imprints under much less constraints. Constraints found in Fan and Tsai (1984) and Ueda and Nakamura (1984) have mostly been removed. The shape of seals can be a square, an ellipse, a circle, or any close loop. The boundary of the seal imprints is allowed to be imperfect. Furthermore, the problem of variable focusing which was suggested as further research by Chen and Tsai (1986) can be solved easily by using the fast minimum square error transform. The rest of this paper is organized as follows. Section 2 describes an ASG representation for observed and model seal imprints. A method for verifying observed seal imprints using ASG matching is presented in Section 3. In Section 4, the learning phase in which the ASGM'sare learned is described. Experimental results are given in Section 5, and, finally, concluding remarks are discussed in Section 6.

2. Definition of ASG representation Attributes of the ASG are typically numerical measurements of the local structure formed by the primitives such as strokes and vertices in a singlepixel-width line-representation of a seal imprint. The length, the curvature, and two vertices it connects are used as attributes for the stroke. The position, the type (end, corner or junction), the list of strokes attached to this vertex, and the list of angles of strokes attached to this vertex are used as attributes for the vertex. Quantitative information is attached to these attributes and used to improve the efficiency of matching by constraining possible structural correspondences, as well as to improve the matching accuracy and reliability. These attributes also make it possible to calculate the degree of the match. So they give a probabilistic basis to how good a vertex or a stroke matches. The ASG~ is constructed during the model learning phase and contains statistical attributes of the vertices and strokes. For the stroke, these are the number of occurrences, mean length, the variance in the length, mean curvature, and variance in curvature. For the vertex, these are

Volume 9. N u m b e r 2

P A T T E R N R E C O G N I T I O N LETTERS

the number of occurences, the mean position, the variance in position, the histogram of the number of strokes. In the model learning phase, the statistical attributes of the ASGm are updated and a new part may be added to the existing ASGm.

3. Verification of observed ASG As we mentioned earlier, the normalized positions of the vertices in the ASGo are determined by the fast minimum square error transform. This transform is described by a parameter vector, (ro, So, O, c) where r 0, So is the translation, 0 is the rotation, c is the scale, such that the image (x~, y~) of an arbitrary point (r~, s~) of the ASGM is given by the following set of equations: xi = e cos 0 ( r i - ro) + c sin 0 ( s i - So), (1) Yi = - e sin 0 ( r i ro) + c cos 0 (si - So). (2) Given an ASGo, a hypothesis (i.e., a prediction of the position of the ASGo with respect to the ASG m) is generated by matching a set of control vertices in the ASGm to a set of compatible vertices in the ASGo.

February 1989

the control vertex vector is 4-tuple) in order to make the problem much more tractable. Therefore, our pose estimation strategy involves generating 4tuple correspondences between control vertices of the ASG Mand vertices of the ASGo. To incorporate geometric features in the matching of the control vertices, the set of control vertices is split into subsets of pairs. If there are n vertices in the ASGo, this results in ,C2 = n(n - 1)/2 candidate control vertex pairs to be evaluated. A control vertex pair has 4 degrees of freedom: position (2), orientation (1), and the distance between the vertices (1). Among them, only the 2n most likely candidate control vertex pairs are selected for further analysis. These 2n candidate control vertex pairs are combined to form candidate 4-tuple control vertex vectors. A candidate 4-tuple control vertex vector has 8 degrees of freedom: position and orientation (3), the distances of the two pairs (2), the angle between the candidate control vertex pairs (1), and the shift along the candidate control vertex pairs (2). The angle between the candidate control vertex pairs together with the pair-distances is used to calculate the degree of the match between the compatible 4-tuple control vertex vector in the ASGo and the 4-tuple control vertex vector in the

ASGM. 3.1. Pose estimation In order to find the pose of the ASGo, the degree of the match between a vertex in the ASG o and a control vertex in the ASGm is calculated based upon pose-independent attribute of a vertex. The pose-independent attributes of each vertex can be the types of the vertex and angles between strokes of the vertex. When the attributes are reliable and the ASGm is stable, there should exist only a few correspondences between a control vertex in the ASGm and a few vertices in the ASG o. When individual vertices do not carry sufficient information and, therefore, do not strongly constrain the number of possible correspondences, the geometry of comb;nation of vertices can help. We construct an m-tuple control vertex vector by m control vertices, and an m-tuple correspondence as a match between m-tuple control vertex vectors in the ASGm and the ASG o. in our current implementation, we used quadruple correspondence (i.e.,

If there are 2n candidate control vertex pairs in the ASGo, then there are as many as 2,C2 = n ( 2 n - 1) candidate 4-tuple control vertex vectors to be evaluated. Among them, only the 4n most likely candidate 4-tuple control vertex vectors are selected for further analysis. Next, the fast minimum square error transform, which results in the transform parameters (translation, rotation and scale) and transform errors for the most likely 4n correspondences, is calculated. This transform has to be determined so as to minimize the transform error between the control vertex vector in the ASGm and the compatible control vertex vector in the ASGo. Then, a correspondence with minimum transform error is chosen as the best transform parameters.

3.2. Correspondenee jinding and co~[tict resolution Finding the pose of the ASGo with respect to the ASG M means the normalized positions of the verti139

Volume 9, N u m b e r 2

P A T T E R N R E C O G N I T I O N LETTERS

ces in the ASGo are available. Therefore, a list of possible corresponding vertices in the ASG~ is made for all the vertices in the ASG M. This list is based on the distance between the model vertices and the observed vertices in a standardized coordinate system, all vertices within a threshold dmatc h a r e considered as possible matches. When there are multiple matches, structural information is used to rank the matches. The vertex type is compared and the angle of the corner is also taken into account if the vertex type is a corner. When there exists still an ambiguity, the closest Euclidean distance vertex is chosen as the best match in our current implementation. The matched strokes are found from the matched vertices. For the two vertices of each stroke of the ASG~, the matched vertices of the ASGM are known, if any, and the matched model stroke must be present between these two matched model vertices.

February 1989

and statistical knowledge of the distortion processes was assessed in terms of transition probabilities (or densities) acting on the attribute values. These conditional probabilities were approximated by modeling the probability of some attributes of a primitive by a Gaussian distribution, ps(Si) and p~(Vj) are computed as the number of occurrences of each S~ and Vj divided by the total number of learning samples, and pn(Vj) is computed as the number of strokes attached to Vj divided by the histogram of the same number of strokes in ASGM. The final result of matching is a correspondence list and a numerical measure of the distance of the model and the observation. The distance of the ASGg and the ASGM is measured as follows:

DIST(ASG N, ASGM) = - ~(log p,(S,) + log pc(Si) + log ps(Si)) i

- ~(log

pd(Vj) + log p°(Vj) + log p~(Vj)). (3)

J

3.3. Distance measure 3.4. Decision making After finding the correspondence between the primitives in the ASGg and those of the ASGM, the distance measure is calculated for all the strokes and vertices of the ASG o. The distance measure is computed by the contributions of each stroke and vertex of the ASG~. The distance of a stroke match consists of three parts based upon the chance of occurrence of the length of the stroke (p~(S~)), the chance of occurrence of the curvature of the stroke (pc(Si)), and the stability of the stroke (p,(S~)). When a stroke is not matched, a minimum likelihood is assigned. The distance of a vertex match consists of three parts based upon the chance of occurrence of the distance between model and observed vertex (po(Vj)), the chance of occurrence of the number of strokes attached to this vertex (p,(Vj)), and the stability of the vertex (ps(Vj)). pl(Si), pc(S~) and pd(Vj) are computed in terms of a set of likelihoods indicating the conditional probability that a primitive (vertex or stroke) will have some particular value for some attribute in the ASGo, given that its corresponding attribute in the ASGM has some particular probability distribution. In other words, the likelihood of a given mapping function from the attributed primitive descriptions 140

After the distance measure DIST(ASG~, ASGM) is computed, three types of decision can be made accordingly: (i) If DIST(ASG~, ASGM)<~ DIST~ow, the observed seal imprint is classified as a genuine. (ii) If DIST(ASG~, ASGM)>1 DISThigh, the observed seal imprint is classified as a forgery. (iii) Otherwise, the input seal imprint is classified as 'ambiguous' and is displayed for visual inspection by the operator to reduce incorrect verification possibility, where DIST~ow and DISThigh (0 ~< DIST~ow ~< DISThigh) are two experiment-dependent constants.

4. Learning model ASG's

The ASGM'S are constructed during the model learning phase. From an ensemble of observations, which is called the 'learning set', and, in our case, is a collection of imprints from a seal, the structural components and their attributes are extracted to construct model. Our scheme can handle even the case where there is a single seal imprint for the mo-

Volume 9, Number 2

P A T T E R N R E C O G N I T I O N LETTERS

~

-------~~ - - ~ O b s e r v e d

seal imprint image

February 1989

)

Shading correction and thresholding

Skeletonization

Construction of ASG

Pose estimation

Normalization and correspondence finding

,L Distance calculation

Decision making

Figure I. Overview of verification steps,

del learning. In some real application environments such as money withdrawal in a bank, the bank clerk can acquire a set of learning seal imprint images for model learning from a single customer seal. Learning is performed on the learning set, that is, the images contain only an example of the pattern class to be learned. This helps to assure that only the 'significant' set of control vertices are found in the ASGM. The learning phase begins with an empty ASG M. For the first seal imprint image in the learning set, control vertices are selected by the human supervisor and repeats the following procedures for each sample seal imprint image in the learning set: First, a pose (translation, rotation, scale) is obtained which places the ASGM at the position, orientation, and the scale of the pattern in the sample image.

This pose is hypothesized by the system and confirmed by the human supervisor. Second, for each control vertex in the ASGu, the correspondence to the most likely control vertex in the ASG o is determined and displayed. Third, as each correspondence is found, the mean and standard deviation of the probability distributions for the attributes are incrementally calculated, and the probability of occurrence of the control vertices is updated. Fourth, the most likely correspondences are then found for the rest of ASGo, and their probability distributions and probability of appearances are also updated. Finally, if a part of the ASGo being learned is not present in the ASGM, the ASG M may be updated and learned by adding the unmatched part to the ASGM. This addition starts with the matched vcrtices of the unmatched strokes. 141

Volume 9, Number 2

PATTERN RECOGNITION LETTERS

February 1989

snoopy : sulee 1

Natn Menu

N.~$!:

Enter Image File : lusr/AILAB/~leeltmages/

Crn'n

Window Ind| cator

Iteration [512] B O-P Sttplsne I-r~l F-~2 ff'r'~3 F-ri4 F-~5 F-r~6 ~ 7 EaSe v=~ue ~ e ~] i

UA OB ¢]¢ ~3D

AI Laborttor~

KAIST

~

Firet ASG for model construction Second ASG for model construction ~Normalized second ASG with respect to f i r s t ASG I Overlapped ASG's for model construction A Composite Model Observed ASG to be verified Normalized observed ASG Over|apped ASG's Original image display Inverted image display Skeleton image display ASG image display

512 Fie

F~ B|tol~ne Threshold [12B]

F~

lit

I-'r12 I-rl3 f-~4 I-~5 ~ 6 ~ 7

g

I-~8

1255

F~n3

Bitplane I~l I-d2 f-all3 F-MI4I"~5 ~ 6 ~ 7 ~ 8 Minimum Segment length [2] Spit line threshold Circular threshold Recovery distance Recovery angle [Write ASG Structure]

[3] [3] [4] [20]

11 I I i | B~ g

I 512 [512 1512 1512

I gg

Crn~ Figure 2. Advanced interfacing features including menu systems and multiple window capabilities,

5. Experiments The seal imprints were scanned by a TV-camera and processed by a Sun-3 workstation, simulating the procedure of money withdrawal in a banking system. Figure 1 shows the overview of verification steps. Each seal imprint image is an array of 256 by 256 pixels, with 8-bit gray levels. The implementation of the verification system employs advanced interfacing features including menu systems and multiple window capabilities. An example of the toplevel menu and multiple windows is shown in Figure 2. Three distinct seals were used for verification in our experiments. As all of them were engraved by a single person, they looked so similar in appear142

ance that it is difficult to distinguish them simply by visual inspection without the help of other means. The 6 model seal imprints (2 imprints for each model) are scanned and processed during the model learning phase. The test imprints are scanned and verified in the verification phase. By matching each of the 24 test imprints with the 3 model imprints, totally 72 similarity measures are computed. The experimental results of the distance measure for all 72 tested data are shown in Table 1, where the ' x ' denotes a classification error; the "?' denotes a rejection; and the remaining ones are cases with correct classification. From Table 1, we see that the rate of correct verification is r 1 = 1 - 7/72 ~ 90.3%,

Volume 9, Number 2

P A T T E R N R E C O G N I T I O N LETTERS

February 1989

Table 1 The results of matching 24 test images with 3 models

Model1

Model2

Model3

la

327

2370

2384

lb lc

322 335

2254 2176

2337 2428

ld le

2366 1371

2309 2344

2736 2278

If

390

2589

2456

lg

2645

2521

2058

lh

335

2348

2361

2a

2503

367

2543

x ? x

2b

2482

424

2821

2c 2d

2964 2392

2379 x 393

2625 2186

2e 2f

1580

324 380

2630 2037

2g 2h

2758 2452

323 417

2490 2186

3a

2376

2420

311

3b 3c

2393 2226

2374 2110

327 2793

3d 3e

2840 2360

2317 2754

2178 352

2318

?

3f

2732

2387

426

3g

2174

2710

329

3h

2075

2346

380

x

x

when both two ambiguous cases are assumed to be verified incorrectly, or is

a

b

¢

d

Figure 3. Thresholded images of seal imprints. (a) Image of a model imprint. (b) Image of an imprint to bc verified, produced by the same seal as that of (a). (c) and (d) hnages of two other imprints to be verified, produced by two seals ditlizrent from that of(a) and (b).

,"

.__36 k

- '7 .....

r 2= 1-5/72~93.1%,

otherwise. This shows that the proposed scheme might be feasible for practical applications. In Figure 3, a thresholded image for a model seal imprint and thresholded images for three seal imprints to be verified are shown. One of three observed seal imprints is from the model seal. The skeletonized images from Figure 3 are given in Figure 4. Figure 5 illustrates the constructed ASG after piecewise approximation and an error recovery process from Figure 4. (The error recovery proceeds as follows: First, all vertices apart within some threshold are joined. Second, end points near to a straight or a curved stroke are attached to it, resulting in a junction and the splitting of the straight or curved stroke. Third, corners, between straight strokes with an angle close to 180 '~ are deleted and

-I-L

_

I

"-.

z .........

z"

.

_.__ ~'"

)

"

i.

'

I' ',

<

J" , .

'

L

i

....

>

'-.

I I

ii

Figure 4 Results of skeletonizing the inlages in Figure 3.

143

Volume 9, Number 2

PATTERN RECOGNITION LETTERS

February 1989

1 a

b

./

~

a

b

e

d

"--,=~ 'x...

/

e

d

Figure 5. Constructed ASG's after piecewise approximation and an error recovery process from Figure 4.

Figure 6. ASG matching steps. (a) A typical model ASG. (b) An observed ASG to be matched. (c) The normalized observed ASG. (d) ASG's showing the correspondence finding.

the strokes are joined.) A typical ASGMis shown in Figure 6(a) and an ASGo to be matched is given in Figure 6(b). Figure 6(c) shows the ASG~). The resulting ASG's showing the correspondence finding are given in Figure 6(d).

In order to reduce the computational overhead, our scheme began with an assumption that a particular set of vertices called control vertices are found reliably in the observation. This may not be true in the situations where seal imprint images contain heavy noise due to photometric effects. However, the experimental results confirmed that the assumption was acceptable for most seal imprint images, and the proposed scheme might be feasible for practical applications. Although the proposed scheme has been found to perform well in most cases, there were some situations in which it fails. Most of these situations resulted from the incorrect pose estimation and poor skeletonization due to heavy noise.

6. Concluding remarks

In this paper, a new scheme for automatic seal imprint verification using attributed stroke graph matching was proposed. A major advantage of our scheme is that our scheme can handle seal imprints under much less constraints. Constraints found in Ueda and Nakamura (1984) and Fan and Tsai (1984) were mostly removed. The shape of seals can be a square, an ellipse, a circle, or any close loop. The boundary seal imprints can also be partial. Furthermore, the problem of variable focusing which was suggested for further research by Chen and Tsai (1986) can be solved easily by using the fast minimum square error transform. Our method is insensitive to the distance between the TV-camera and seal imprints. 144

Acknowledgements

The authors are grateful to Prof. Dr. H.S. Yang for his valuable suggestions which improved considerably the readability of this paper. They are also thankful to the anonymous referee for his constructive comments which have been included in the final version.

Volume 9, Number 2

PATTERN RECOGNITION LETTERS

References Bunke, H. and G. Allermann (1983). Inexact graph matching for structural pattern recognition. Pattern Recognition Letters 1, 245 253. Chen, C.M. and W.H. Tsai (1986). Automatic identification of Chinese seal imprints with arbitrary or partial shapes. Computer Processing,, qf Chinese and Oriental Languages 2, 216 233. Fan, T.J. and W.H. Tsai (1984). Automatic Chinese seal identification. Computer Vision, Graphics, and hna~e Proeessin'~ 25, 311 330.

February 1989

Groen, F.C.A., A.C. Sanderson and J.F. Schlag (I 985). Symbol recognition in electrical diagrams using probabilistic graph matching. Pattern Recognition Letters 3,343 35t). Lee, S. and J.H. Kim (1988a). Seal imprint verilication using attributed stroke graph matching. Technical Report TR-88-32, Dept. of Computer Science, Korea Advanced Institute of Science and Technology, Seoul, Korea. Lee, S. and J.H. Kim (1988b). A fast computational method for minimum square error transform. Pattern Recognition Letters 8, 143 146. Ueda, K. and Y. Nakamura (1984). Automatic verilication of seal-impression patterns. Proc. 7th Intern. ('on[i Pattern Reco~nition, lOI9 1021.

145