Handwriting based writer recognition using implicit shape codebook

Forensic Science International 301 (2019) 91–100

Contents lists available at ScienceDirect

Forensic Science International journal homepage: www.elsevier.com/locate/forsciint

Handwriting based writer recognition using implicit shape codebook Akram Bennoura,* , Chawki Djeddia , Abdeljalil Gattala , Imran Siddiqib , Tahar Mekhazniaa a b

Larbi Tebessi University, Tebessa, Algeria Bahria University, Islamabad, Pakistan

A R T I C L E I N F O

A B S T R A C T

Article history: Received 31 October 2018 Received in revised form 2 May 2019 Accepted 7 May 2019 Available online 22 May 2019

Writer characterization from images of handwriting has remained an important research problem in the handwriting recognition community that finds applications in forensics, paleography and neuropsychology. This paper presents a study to evaluate the effectiveness of an implicit shape codebook technique to recognize writer from digitized images of handwriting. The technique relies on identifying the key points in handwriting and clustering the patches around these key points to generate an implicit shape codebook. A writer is then characterized by the probability distribution of producing the codebook patterns. Experiments are carried out in text-dependent as well text-independent mode using the standard BFL and CVL databases of handwriting images. Promising identification and verification performance is reported in a number of interesting experimental scenarios. © 2019 Elsevier B.V. All rights reserved.

Keywords: Writer identification and verification Implicit shape codebook Harris key-point detector Agglomerative clustering SVM

1. Introduction Computerized analysis of handwriting and handwritten documents has remained an interesting research area for many decades that has attracted significant research attention of document examiners, forensic experts, psychologists, neurologists and paleographers. Contrary to machine printed text, handwritten text and hand-drawn shapes carry rich information about the individual producing these documents. The unique writing characteristics of an individual (depicted through writing style) make it possible to employ handwriting as a behavioral biometric modality hence allowing identification and verification of writers from handwritten documents. Formally, writer identification task involves finding the writer of a query handwritten document comparing it with a set of writing samples with known writers. Writer verification, on the other hand, includes deciding whether two writing samples have been produced by the same individual or not. The effectiveness of handwriting as a behavioral biometric modality has been discussed by many researchers [1,29–31]. Jain et al. [28] have identified a number of factors determining the appropriateness of a biometric modality. While in terms of “distinctiveness”, handwriting appears to be relatively less effective

* Corresponding author. E-mail addresses: [email protected] (A. Bennour), [email protected] (C. Djeddi), [email protected] (A. Gattal), [email protected] (I. Siddiqi), [email protected] (T. Mekhaznia). https://doi.org/10.1016/j.forsciint.2019.05.014 0379-0738/© 2019 Elsevier B.V. All rights reserved.

once compared to DNA or finger prints, in terms of “measurability”, acquiring handwriting samples is fairly simple that does not require specialized hardware devices. Analysis of handwriting is an established area in the forensic sciences [51,54] and, in addition to other applications, analysis of handwriting for forensic applications has continued to be an attractive problem for document examiners. The individuality of handwriting has been studied in a number of works [1,44,48] and evidences provided by handwritten documents have been employed in courts for many years [33]. Traditionally, the task of forensic writer identification (that involves identifying the writer of a questioned document) is carried out by forensic document experts using standard methodologies as outlined in Refs. [34,35]. In the past, there has been a hesitancy from the domain experts to employ computerized solutions for this problem [36]. The advances in different areas of image analysis and pattern classification however, have led to acceptability of computerized solutions by forensic experts. It is important to mention that such solutions are aimed at facilitating and not replacing the human experts. Considering the scale of forensic databases (order of 104 samples), automatic systems can be employed to reduce the search space so that human experts can focus on a manageable sized list of potential candidates. Schomaker et al. [37] establish that the target performance of computerized writer identification systems is near to 100% identification rate in hit list of 100 writers from writing collections matching the size of real world forensic databases. A number of projects have been carried out to integrate the knowledge from forensic experts into interactive computerized solutions where a subset of features employed by the experts are algorithmically computed by computer programs. Popular of these

92

A. Bennour et al. / Forensic Science International 301 (2019) 91–100

projects include Trigraph [38], Wanda [39], Biografo [53], the CEDAR-FOX Forensic Document Examination system [40], GraphJ [41] and the well-known Forensic Information System for Handwriting (FISH) framework maintained by the U.S. Secret Service that allows searches using computerized solutions. With the recent advancements in different areas of image analysis and machine learning, a number of techniques have been reported in the literature realizing high identification/verification performance [1–16,45–47]. Comprehensive reviews on the latest developments on this problem can be found in Refs. [42,43]. Such systems aim to capture the visual differences in handwritings of individuals typically including allographic variations, inter and intra word spacing, line spacing, cursiveness, legibility, slope of lines and slant of characters etc. Writing samples from two different writers reflecting these differences are illustrated in Fig. 1. It is important to mention that the current state-of-the-art in computerized analysis of handwriting to predict writer’s identity reports near to 100% recognition rates (when a hit-list of most likely ‘K’ writer is retrieved, ‘K’ is typically set to small values up to 10) on datasets of 1000 writers on the average. As discussed earlier, such computerized systems can be employed to reduce the search space and the hit-list returned by such systems can be subjected to examination by human experts to come to a conclusion. A number of recent studies on writer identification and verification rely on extracting the redundant writing patterns (generally known as the "codebook") to characterize the writer [7–12,46]. In our study, we aim to integrate Harris key-point detector and agglomerative clustering technique into a common probabilistic framework to build an implicit shape codebook. The codebook is then employed to classify writer using Support Vector Machine (SVM) and nearest neighbor classifiers. An overview of the proposed method is presented in Fig. 2. The key theme of this study is not to segment the text into semantically meaningful fragments (like characters). We rather extract patches of interest around key-points as we aim to characterize the writing style based on ‘how’ the text is written rather than ‘what’ is written. The proposed technique is validated on two publicly available datasets and high identification rates are reported in a number of experimental scenarios. Such computerized solutions can be helpful for the forensic experts in narrowing down the suspect pool to facilitate the document examination process. Presented with a query handwriting sample, the system can be employed to return a list of potential writers for subsequent analysis by the experts. This paper is organized as follows. In the next section, we discuss the relevant literature on this problem. We introduce the datasets employed in our study in Section 3 and present the implicit shape codebook methodology in Section 4. Section 5 presents the classification scheme while Section 6 details the experimental results and the accompanying analysis. Finally, the

Fig. 2. Steps of the proposed method.

last section concludes the paper with some useful insights for further research on this problem. 2. Related work Writer identification techniques reported in the literature are traditionally categorized into text -dependent and textindependent approaches. Text-dependent methods for writer identification are inspired by matching techniques employed by forensic experts where same characters and character combinations are compared [4,8,10]. Text-independent methods, on the other hand, aim to capture the writing style of a specific writer independent of the semantic content of the writing samples. Among one of the premier contributions to identification of writers from handwriting, Srihari et al. [1] demonstrated the validity of handwriting as a biometric modality through a comprehensive study involving 1500 individuals. This was a text-dependent study and all writers contributed the same text. Bensefia et al. [2] segmented handwriting into graphemes and exploited the redundancy of morphologically similar graphemes to characterize the writer. Later, Siddiqi and Vincent [3] extended the

Fig. 1. Samples of two different writers.

A. Bennour et al. / Forensic Science International 301 (2019) 91–100

same idea and employed small fragments of writing to extract writer-specific frequent patterns. Distribution of these patterns was combined with orientation and curvature features extracted from writing contours. In another study [4], textural measures including Local Binary Patterns (LBP) and Local Phase Quantization (LPQ) were extracted from normalized blocks of handwriting to capture the writing style. Experimental study on two different datasets, the Brazilian forensic letter database and the IAM database, realized high identification rates. Hannad et al. [6] divide handwriting into small windows and consider each window as a unique textural pattern. Textural descriptors including Local Binary Patterns (LBP), Local Ternary Patterns (LTP) and Local Phase Quantization (LPQ) are then computed from these fragments. The technique was evaluated on the complete set of samples in IAM and IFN/ENIT databases and among the employed descriptors, LPQ reported the highest identification rates on both the databases. In another recent work, Christlein et al. [7] employ RootSIFT descriptors computed densely at contour edges. After a normalization step, these local descriptors are used as input to an Exemplar-SVM which ranks the documents as a function of similarity with the query document. Recently, deep learning based automatic feature extraction techniques have also been investigated for identification of writers. In Ref. [26] for instance, authors employ convolutional neural networks to automatically extract local features from writing samples. Experiments on multiple datasets demonstrated the effectiveness of machine learned features over hand-crafted features. Among various techniques to characterize writer from handwriting, codebook based methods have been a popular choice of researchers. These techniques are similar to the bag of words approaches and rely on producing a visual vocabulary of patterns which serves as a codebook. In most cases, given a writing sample, the writer is characterized by the probability distribution of producing the codebook patterns. Among well-known codebook based writer characterization techniques, Bensefia et al. [8], extending the ideas in Ref. [2], produced a codebook of graphemes, image correlation being the similarity measure to compare two graphemes. In another study, Schomaker et al. [9] propose to employ a codebook of fragmented connected-component contours (FCO3) extracted from character fragments (fraglets). Codebook is generated by clustering these fragments using Self Organizing Maps (SOMs) and writers are characterized by the probability distribution of producing the codebook fragments. The study specifically targets writer identification for forensic and historical

93

applications. Brink et al. [10] extended the work to combine codebook features with other statistical measures and studied the impact of slant correction on writer identification performance. Abdi et al. [5] adapted beta-elliptic model to generate a synthetic codebook of graphemes rather than extracting them from handwriting. The proposed technique reported an identification rate of 90% on the 411 writers in the IFN/ENIT database. In another notable work [11], cursive handwriting is divided into fragments of connected components to produce the codebook. The occurrence histogram of patterns in the codebook is then employed to generate the feature vector of each writing sample under study. The technique evaluated on two English and three Farsi handwriting databases realized high identification rates. In a study by He et al. [12], authors propose a junction detection method using crossings of strokes in the handwriting. The detected junctions are then employed to generate a codebook of ‘Junclets’ to characterize the writer. Khalifa et al. [13] extend the idea of codebook to an ensemble of codebooks (of graphemes). To ensure a manageable feature vector size, kernel discriminant analysis using spectral regression (SR-KDA) is used for dimensionality reduction. Multiple codebooks are combined together and an enhancement in identification rates over single codebook methods is demonstrated. In another recent work, Khan et al. [14] employ the bagged discrete cosine transform (BDCT) descriptors for writer identification. Multiple models are generated using universal codebooks and majority voting is used to arrive at the final decision. In another contribution specifically targeting the forensic applications, Al-Maadeed et al. [49] skeletonize the handwritten text and segment it at junction pixels to produce small fragments. The fragments are clustered into a codebook and similar to other studies, given a writing sample, the probability of producing the codebook patterns is exploited to characterize its writer. An identification rate of near to 91% is reported on the ICDAR 2011 dataset [50]. Likewise, Fernandez-de-Sevilla [52], demonstrate the effectiveness of a codebook of (manually segmented) characters to identify writers from a set of real forensic documents. A summary of well-known contributions to writer identification (and verification) reported in the literature is presented in Table 1. IAM and CVL databases have been mostly employed for evaluation while for Arabic writer identification, the IFN/ENIT database of 411 writers has been considered in most of the studies. It is important to mention that many of the databases (for example IAM, IFN/ENIT etc.) were primarily developed for evaluation of handwriting (segmentation and) recognition tasks and not from the perspective of forensic applications. However, since the

Table 1 Performance comparison of well-known writer identification systems. Study

Database

Language

Number of writers

Identification rates (%)

Verification rates (%)

Bulacu and Schomaker [16]

IAM IFN/ENIT IAM IAM CVL IAM BFL IFN/ENIT IAM IAM Firemaker CERUG-MIXED IFN/ENIT IAM CVL IAM CVL IFN/ENIT IAM CVL

English Arabic English English English English Portuguese Arabic English English Dutch Chinese and English Arabic English English English English Arabic English English

650 350 650 650 309 650 315 411 650 650 250 105 411 657 310 650 310 411 650 310

89.00 88.00 91.00 93.70 – 96.70 99.20 90.00 92.00 91.10 89.80 96.20 94.89 89.54 96.20 97.20 99.60 76.00 99.20 98.80

97.20 – 97.70 – 97.80 99.60 99.40 – – – – – – – – – – – – –

Siddiqi and Vincent [3] Ghiasi and Safabakhsh [11] Fiel and Sablatnig [15] Bertolini et al. [4] Abdi and Khemakhem [5] Khalifa et al. [13] He et al. [12]

Hannad et al. [6]

Khan et al. [14]

Christlein et al. [7]

94

A. Bennour et al. / Forensic Science International 301 (2019) 91–100

identity of the contributor was also stored during data collection, the databases were employed in writer identification tasks as well. Examples of writing samples that were specifically collected from the perspective of evaluating the individuality of handwriting include the CEDAR letter [1], the BFL database [22] and ICDAR 2011 writer identification dataset [50]. An analysis of the writer identification techniques reported in the literature reveals that despite the recent sophisticated machine learning based feature extraction techniques, codebooks have continued to remain a popular choice of researchers. Inspired by the intuitive and effective representation the codebooks offer, in the present study, we investigate implicit shape codebooks for identification of writers. Prior to presenting the details of the proposed technique, we first introduce the datasets employed in our study in the next section. 3. Datasets We have employed two standard datasets in our study, the Brazilian Forensic Letter (BFL) database [22] and the CVL database [23]. Writing samples from the two databases are illustrated in Fig. 3. The BFL database contains writing samples of 315 writers (three samples per writer) with a total of 945 images. The textual content on each letter comprises 131 words in Portuguese making the dataset appropriate for evaluation of text-dependent writer identification systems. The BFL database has been specifically developed for forensic applications. The content of the letter comprises of words that are carefully chosen to contain all letters and numerals as well as certain character combinations (‘nh’, ‘lh’, ‘qu’ and ‘00’) which are of interest to the forensic experts. The letter also contains punctuations, special symbols and diacritics. Writers used their own pens to produce the writing samples and were free to choose the type of pen as well. For our experiments, we carry out three fold cross validation. In each of the experiments, 600 samples of 300 writers are used as training set while the remaining 300

samples of these writers constitute the test set. The 45 writing samples of 15 writers are used to generate the codebook. The second database employed in our experiments is the CVL database where 311 writers contributed 7 different handwritten texts each. For each writer there is one text in German and six in English, the latter being a part of our experimental set. Two series of experiments are carried out on the CVL database. In the first series of experiments, for codebook generation, we employ 33 samples of 11 writers while 900 samples of 300 writers (3 samples per writer) are used for experiments. Similar to the BFL database, we divide the images into three subsets for 3-fold cross validation i.e. two subsets for training and one for testing. In the second set of experiments, we consider 5 samples per writer with 55 samples of 11 writers in the codebook generation set, 900 samples of 300 writers in the training and 600 samples of these writers in the test set. It is also important to mention that the writers used for generating the codebooks are not considered in the training and the test set. The codebooks can be generated from the writers in the training set as well. However, using an independent set of small writers for codebook generation ensures that the system remains scalable to add new writers without the need to regenerate the codebook once new writers are added. Other studies exploiting codebooks for writer characterization [3,16] also advocate the idea of using an independent set of writers for generating the codebook. The impact of codebook patterns on the overall performance has been studied in detail in Ref. [32]. 4. Implicit shape codebook methodology This section presents the details of the implicit shape codebook technique to characterize writer from handwriting. The implicit shape model (ISM) framework is inspired by the work of Ref. [17] for object segmentation and classification in real world scenes. The implicit shape model is first trained in an offline learning step and later applied for classification [17,18].

Fig. 3. Sample images from the BFL (left) and CVL (right) databases.

A. Bennour et al. / Forensic Science International 301 (2019) 91–100

95

For adaptation of ISM to handwriting images, we build a codebook of local appearances that are characteristic for a certain viewpoint of handwriting by sampling local features occurring frequently on a set of training images. Features that are similar are grouped together in a clustering step. The following sections present the details on codebook generation and the feature extraction steps.

error (SSE) of the cluster defined in the following. X distðci ;  xÞ2 SSEi ¼

4.1. Codebook generation

where K represents the total number of clusters. The whole idea of using small patches for codebook generation is to exclude any script dependent knowledge (as opposed to characters or graphemes). The similarity between different patches of the same codebook is a function of similarity threshold. Changing the threshold can control the number of clusters and hence it implicitly controls how similar the different element of codebook are. For different codebooks, similarities can be observed between the patches of the two codebooks as the patch size is sufficiently small.

We start by applying the Harris key-point detector [19] for extracting the junctions and corners to obtain a set of informative regions for each handwriting image. By extracting features only from these regions (called patches) of size nxn, the amount of data to be processed is reduced while the interest point detector assures that similar patches are sampled on different objects. These patches are later clustered and are used to determinate a common invariant features space over the entire data set. Fig. 4 shows a sample handwriting image and the extracted regions of interest. The process is repeated for all images under consideration, the extracted patches are then grouped using a clustering step. For grouping similar features (patches) into clusters, we employ agglomerative clustering that automatically determines the number of clusters by sequentially merging features until a cutoff threshold t on the cluster compactness is achieved [17,20]. The process starts by considering each element as a separate cluster. Subsequently, at each step, two most similar clusters C1 and C2 are merged provided the average similarity (SIM) between their constituent patches (cluster compactness) stays higher than a certain threshold t. The value of the threshold t for the patch similarity was empirically chosen (t = 10) and the same value was used for all the databases employed in our study. P p2C 1 ;q2C 2 NGC ðp; qÞ >t SIM ðC 1 ;  C 2 Þ ¼ ð1Þ jC 1 j  jC 2 j While NGC is Normalized Gray scale Correlation and p,q represent the patches in clusters C1 and C2 respectively. NGC ðp; qÞ ¼

X qi Þ ffi ðpi  pi Þðqi  qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi P P i ðqi  qi Þ2 ðpi  pi Þ2

ð2Þ

As a result of this process, visually similar patches are grouped into clusters hence producing the codebook by seeking each cluster centroid. Fig. 5 shows samples of the extracted codebooks from the CVL [23] and BFL [22] databases (the centroid of each cluster is illustrated in the codebook) while Fig. 6 illustrates the patches within a single cluster of the BFL database. The centroid ci of cluster C i is the data point (patch) that minimizes the sum of the squared

x2C i

where dist represents the Euclidean distance in our case. The codebook comprises the set of centroids of each of the clusters. CB ¼ fci gKi¼1

4.2. Feature extraction Once the codebook is generated, we proceed to representation of writings under study using the codebook i.e. creating the implicit shape models for these writings. For a given writing sample, patches around the key points are extracted and each patch is compared with the codebook patterns (using the same correlation measure that is employed in the clustering step). Instead of finding the best match, we activate all codebook entries with which the similarity of the current patch is higher than the threshold t (cut-off threshold used during the clustering). For a given document D with a set of M patches, we increment the indices corresponding to the similar (with respect to threshold t) codebook entries.   D D hb  hb þ 1 ;  8b where dist pj ;  cb <  t

f or j ¼ 1    M Hence, for a given handwritten sample, for each codebook entry, we store the number of times it was activated. The histogram D

h is normalized to distribution PD by dividing the value of each codebook entry by the sum of values of all codebook entries and is employed to characterize the writer. For N writers in the reference base, the distributions are stored in a N  K data matrix, where K is the codebook size as specified previously. The characterization of writers using these distributions is visually illustrated in Fig. 7 (where the x-axes represent the codebook entry while the y-axes represent the probability of

Fig. 4. Local information used in the codebook generation process: a sample handwriting image (left) and the regions extracted around the points of interest (right).

96

A. Bennour et al. / Forensic Science International 301 (2019) 91–100

Fig. 5. Generated codebooks: BFL (left) and CVL datasets (right).

Fig. 6. Patches in a single cluster from the BFL database.

occurrence of the respective codebook patch in a given writing). Fig. 7a shows the distribution of codebook on two samples of the same writer clearly indicating the high within-writer similarity. Fig. 7b, on the other hand, illustrates the same distributions for texts from two different writers highlighting the between-writer variation. 5. Classification Once the implicit shape codebook features are extracted from writing images, we proceed to classification for which we have chosen to employ the k-nearest neighbour (kNN) and the Support Vector Machine (SVM) classifier. With KNN, we compute the distance between the feature vector (distribution) of a query handwritten text Q and those of all the documents in the reference base R and pick the writer of the document that reports the minimum distance. WriterðQ Þ ¼ arg min ðDISðQ; Di ÞÞ Di 2R

where DISðQ; Di Þ is the distance between the distributions PQ and PDi of documents Q and Di respectively. The multi-class SVM is implemented using the one-against-all technique [21]. Three important parameters are required for training the SVM which include the bound on the Lagrangian multipliers ‘C’ (fixed to104), the conditioning parameter for QP method (l) (chosen to be 104) and the Gaussian function kernel parameter (s) (fixed to 2). The classifier is fed with the ðN  KÞ data matrix along with the vector of writer identities to learn the discriminator function. 6. Results and discussion As discussed in Section 3, the experimental study of the system is carried out on two standard datasets, the Brazilian Forensic Letter (BFL) database [22] and the CVL database [23]. We first

present the writer identification rates realized on the BFL database in Table 2 as a function of the patch size (around each key point). The identification rate represents the percentage of writing samples whose writers are correctly identified. The reported identification rates represent the average of three runs (3-fold cross validation). The Top-k identification rates refer to retrieving the list of ‘k’ writers whose handwritings show the most similarities with the handwriting of the questioned document. If the true writer is found within the top-list of k writers, the document is considered to be correctly classified. The value of ‘k’ can be chosen so as to ensure that the system reports near to 100% identification rate. In other words, the idea is to reduce the search space (for expert analysis) in such a way that we do not miss the writer of questioned document. It can be seen from Table 2 that the identification rates are more or less consistent for different patch sizes. The highest reported (average) identification rate is 98.33% with the SVM classifier. Similarly, the (average) identification rates on the CVL database using the two experimental protocols (900 samples and 1500 samples) are summarized in Table 3. It can be observed that, in general, the identification rates on the CVL database are relatively lower as compared to those realized on the BFL database. The observation is very much natural as BFL is a text-dependent dataset while the writing samples in the CVL database represent textindependent evaluation where writing samples in the training and test sets contain different textual content. We also carry out evaluations using a leave-one-out protocol where one writing sample is used as query and the complete set of remaining samples serves as the reference base. The system ranks the writers in the reference base as a function of similarity with the queried document. The performance of these experiments is quantified using the Top-1 identification rate as well as the mean average precision (mAP) defined in Ref. [27]. The results of these experiments for the two databases are summarized in Table 4. Similar to previous experiments, the identification rates are not very sensitive to the patch size and the performance in the

A. Bennour et al. / Forensic Science International 301 (2019) 91–100

97

Fig. 7. Distributions of codebook patterns in (a): two handwritten texts from the same writer and (b): handwritten texts from two different writers.

text-dependent mode (BFL database) is better than that in the text-independent mode (CVL database). For writer verification task, we compute the Manhattan distance between two given documents and consider them to be written by the same person if the distance falls within a predefined decision threshold. Beyond the threshold value, we

consider the samples to be written by different writers. By varying the decision threshold, the ROC curves are computed and the verification performance is quantified by the Equal-Error-Rate (EER) that corresponds to the point where FRR equals FAR. It is important to mention that as a function of application, one of the two errors (FAR or FAR) may be more critical. Consequently, the

98

A. Bennour et al. / Forensic Science International 301 (2019) 91–100

Table 2 Writer identification rates on BFL database. Descriptor parameters

Classifier

Patch size

Dimension

19  19

1194

21  21

1234

23  23

1345

25  25

1360

27  27

1428

29  29

1469

SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN

Average identification rates (%) Top 1

Top 2

Top 5

Top 10

97.33 96.11 97.55 96.67 97.33 96.00 98.11 97.11 98.33 97.55 97.66 97.00

98.89 – 98.78 – 98.83 – 99.11 – 99.11 – 98.89 –

99.89 – 99.22 – 99.67 – 99.33 – 99.44 – 99.11 –

99.78 – 99.78 – 99.78 – 99.56 – 99.67 – 99.45 –

Table 3 Writer identification rates on the CVL database. Samples Descriptor parameters

Classifier Average identification rates (%)

Patch size Dimension 900

1500

19  19

459

21  21

521

23  23

558

25  25

585

27  27

593

29  29

619

19  19

638

21  21

680

23  23

736

25  25

784

27  27

823

29  29

823

SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN SVM KNN

Top1

Top2

Top5

Top10

93.56 87.11 92.67 88.44 93.55 88.55 94.00 88.89 93.00 88.55 93.33 87.55 92.95 86.25 93.68 87.37 93.95 87.35 93.73 88.65 94.32 87.83 93.52 86.80

96.44 – 95.55 – 96.33 – 96.45 – 95.78 – 95.55 – 96.03 – 96.42 – 96.82 – 96.95 – 96.52 – 96.33 –

98.33 – 98.00 – 98.45 – 98.22 – 97.89 – 97.22 – 98.12 – 98.42 – 98.43 – 98.73 – 98.50 – 98.33 –

98.89 – 98.89 – 99.00 – 99.22 – 98.89 – 98.22 – 98.83 – 99.22 – 99.19 – 99.25 – 99.23 – 99.02 –

Table 4 Writer identification performance using a leave-one-out protocol (KNN Classifier). Database

Descriptor parameters

Identification rates (%)

Patch Size Dimension Top 1 19  19 21  21 23  23 25  25 27  27 29  29 CVL database (900 samples) 19  19 21  21 23  23 25  25 27  27 29  29 CVL database (1500 samples) 19  19 21  21 23  23 25  25 27  27 29  29 BFL database

1194 1234 1345 1360 1428 1469 459 521 558 585 593 619 638 680 736 784 823 823

95.67 96.11 96.00 97.33 97.22 96.67 83.78 86.22 85.56 86.67 86.22 85.33 85.73 87.47 87.13 88.53 88.33 87.00

decision threshold can be adjusted according the significance of these errors as changing the threshold would result in increasing one of the errors while decreasing the other. Nevertheless, for quantization of system performance, it is common to report these errors on a single threshold where both FAR and FRR are (approximately) equal. The results of these experiments are presented in Table 5 where the trend is more or less similar to what is observed on the identification performance. EER of as low as 3.22% is reported on the BFL database while those realized on the CVL database read 5.55% and 7.23% on 900 and 1500 samples respectively. In an attempt to study how the proposed technique performs on other scripts, we apply it to characterize writers from Arabic handwriting samples. The well-known KHATT database [24,25] is employed in our experimental study. The database comprises writing samples contributed by 1000 individuals with four samples per writer. The database contains images with same (Pages 1 & 2) as well as different textual content (Pages 3 & 4). For experiments, we carry out 5 fold cross validation using 2000 samples in training and 2000 in the test set for each run. The average identification rates of 5 runs (using SVM classifier) are summarized in Table 6. It can be observed that the identification rates on the KHATT database are much lower once compared to those realized on the BFL and CVL databases. Similar trend can be seen from the performance using leave-one-out protocol (Table 7) as well as the verification errors on the KHATT database (Table 8). These findings call for a deeper investigation and further experimentation for adaptation of implicit shape codebook model to characterize writers from Arabic handwritings. A critical parameter in the presented method is the number of samples (writers) used to generate the codebook and it would be interesting to study the impact of this parameter on the identification rates. In the last series of experiments, we vary the number of writers used to generate the codebook and study the evolution of performance. These experiments are carried out using the BFL, CVL and KHATT databases. The realized identification rates are summarized in Table 9 where it can be seen that in many cases, the identification rates do not seem to be too sensitive to the number of writers employed to generate the codebook. The identification rates on KHATT database seem to be relatively more sensitive to the number of writing samples used to produce the codebook. The previous experiments on KHATT database also reported lower performance as compared to CVL and BFL databases and as mentioned earlier, these observations call for

Table 5 Writer verification performance. Database

Verification errors (%)

Patch size Dimension FRR

mAP 92.01 92.88 92.85 93.50 93.71 93.54 78.62 79.21 79.86 80.92 80.21 79.28 72.82 73.93 73.88 74.10 74.59 72.67

Descriptor parameters

19  19 21  21 23  23 25  25 27  27 29  29 CVL database (900 samples) 19  19 21  21 23  23 25  25 27  27 29  29 CVL database (1500 samples) 19  19 21  21 23  23 25  25 27  27 29  29 BFL database

1194 1234 1345 1360 1428 1469 459 521 558 585 593 619 638 680 736 784 823 823

4.27 3.22 3.83 3.61 3.55 3.55 5.94 5.66 5.55 5.55 5.55 6.33 7.73 7.68 7.23 7.38 7.58 8.06

FAR

EER

4.28 3.22 3.83 3.59 3.55 3.55 5.94 5.65 5.55 5.55 5.55 6.33 7.73 7.68 7.23 7.38 7.58 8.06

4.27 3.22 3.83 3.61 3.55 3.55 5.94 5.66 5.55 5.55 5.55 6.33 7.73 7.68 7.23 7.38 7.58 8.06

A. Bennour et al. / Forensic Science International 301 (2019) 91–100 Table 6 Identification rates on the KHATT database. Descriptor parameters

Identification rates (%)

Patch size

Dimension

Top 1

Top 2

Top 5

Top 10

19  19 21  21 23  23 25  25 27  27 29  29

459 521 558 585 593 619

57.22 59.92 61.3 62.44 61.26 62.81

66.18 68.15 69.48 70.51 77.576 70.52

75.49 77.58 78.56 79.75 84.95 79.25

82.18 83.58 84.51 85.46 89.16 85.12

Table 7 Identification rates on the KHATT database using leave-one-out evaluation protocol. Descriptor parameters

Identification rates (%)

Patch size

Dimension

Top 1

MAP

19  19 21  21 23  23 25  25 27  27 29  29

459 521 558 585 593 619

38.60 40.83 42.25 44.60 45.17 43.32

29.45 31.43 32.41 33.15 34.30 32.20

Table 8 Writer verification performance on the KHATT database. Descriptor parameters

Verification errors (%)

Patch size

Dimension

FRR

FAR

EER

19  19 21  21 23  23 25  25 27  27 29  29

459 521 558 585 593 619

15.10 14.63 15.16 15.73 15.77 16.57

15.10 14.63 15.15 15.73 15.77 16.57

15.10 14.63 15.16 15.73 15.77 16.57

Table 9 Identification rates as a function of the number of samples (writers) used to generate the codebook (classifier: SVM). Database

BFL database

CVL database

KHATT database

Descriptor parameters

Identification rates (%)

Number of writers

Dimension Top1

Top2

Top5

Top10

3 6 9 12 15 20 3 6 9 11 12 3 6 9 10 12

424 762 984 1167 1428 1811 356 612 781 823 954 255 437 574 619 682

98.55 98.78 99.00 98.44 99.11 98.44 95.55 96.00 96.45 96.45 96.35 62.50 64.25 70.00 70.52 70.25

99.22 99.22 99.33 99.33 99.44 99.11 98.00 98.22 98.33 98.22 98. 22 74.30 74.15 78.15 79.25 79.15

99.33 99.77 99.44 99.55 99.67 99.33 98.89 98.89 98.89 99.22 99.22 81.85 81.05 84.05 85.12 85.05

97.77 97.66 98.11 98.00 98.33 98.00 92.67 93.00 93.56 94.00 94.00 53.55 54.45 61.00 62.81 61.45

more research endeavours to adapt this technique for Arabic handwriting. The high identification and low equal error rates reported in the comprehensive series of experiments validate the effectiveness of implicit shape codebooks in characterizing the writer from handwriting images. The Top-10 identification rates are close to 100% advocating the appropriateness of the proposed technique

99

for computerized forensic handwriting systems. The proposed codebook based writer characterization can be effectively employed to develop semi-automatic retrieval systems facilitating the forensic document examiners. Such systems can be employed to retrieve a probable list of potential writers against a questioned document and the reduced set of writings can be subjected to further examination by human experts. Another advantage of implicit codebooks over other codebook based techniques is the patch extraction phase where no specific segmentation scheme is required. The algorithmic details of the technique can be intuitively explained (rather than a black box) which is likely to enhance its acceptability by the forensic experts. From the view point of computational complexity, the most (computationally) expensive part in the proposed technique is the generation of codebook that is based on agglomerative clustering. Once the codebook is generated, the feature extraction step simply involves computing the distance between patches in a writing and the patterns in the codebook. Nevertheless, codebook generation is a one time job that does not need to be repeated and the features of writers in the reference base are also computed offline. During the evaluation phase (identifying authorship of a questioned document), the distribution of the query handwriting needs to be computed and identification involves computation of distance between the query feature vector and those in the reference base (KNN classifier) or simply feeding the query feature vector to the trained model (SVM in our case). This makes it possible to employ such a system for practical applications where a large a number of query documents are to be processed. 7. Conclusion This paper investigated the effectiveness of implicit shape codebook model to characterize writer from handwriting. Interest points in handwriting are identified and small patches around the key-points are used to produce a codebook. The writer of a given sample is characterized by the probability distribution of producing the codebook patterns. Experiments on two publicly available datasets report high identification rates in a number of experimental scenarios. The realized performance validates the effectiveness of codebook distributions in characterizing writers from handwriting images. Not only the system reports high identification rates, the technique is supported by intuitive justification of the using the proposed features, enhancing its acceptability by the forensic experts. Such tools can be effectively employed to automate parts of the examination process narrowing down the search in large repositories. It is expected that the ideas put forward in this study would be useful both for computer scientists and forensic experts. In our further study on this subject, we intend to investigate and compare the performance of different codebooks with the objective to identify the most appropriate scale of observation to characterize the writer. Another important aspect is the identification of writers from writing samples in multiple scripts. It would be interesting to study whether writers share some common characteristics across multiple scripts and are these characteristics discriminative enough to allow identification of writers. We also plan to employ the codebook based techniques for characterization of demographic attributes of writers in addition to the identity. Furthermore, a study on the performance of synthetically generated codebooks will also make the subject of our future research. References [1] S.N. Srihari, S.-H. Cha, H. Arora, S. Lee, Individuality of handwriting, J. Forensic Sci. 47 (4) (2002) 856–872. [2] A. Bensefia, A. Nosary, T. Paquet, L. Heutte, Writer identification by writer’s invariants, Proceedings of the Eighth International Workshop on Frontiers in Handwriting Recognition, (2002) , pp. 274–279.

100

A. Bennour et al. / Forensic Science International 301 (2019) 91–100

[3] I. Siddiqi, N. Vincent, Text independent writer recognition using redundant writing patterns with contour-based orientation and curvature features, Pattern Recognit. 43 (11) (2010) 3853–3865. [4] D. Bertolini, L.S. Oliveira, E. Justino, R. Sabourin, Texture-based descriptors for writer identification and verification, Expert Syst. Appl. 40 (6) (2013) 2069–2080. [5] M.N. Abdi, M. Khemakhem, A model-based approach to offline textindependent Arabic writer identification and verification, Pattern Recognit. 48 (5) (2015) 1890–1903. [6] Y. Hannad, I. Siddiqi, M. Kettani, Writer identification using texture descriptors of handwritten fragments, Expert Syst. Appl. 47 (2016) 14–22. [7] V. Christlein, D. Bernecker, F. Hnig, A. Maier, E. Angelopoulou, Writer identification using GMM supervectors and Exemplar-SVMs, Pattern Recognit. 63 (2017) 258–267. [8] A. Bensefia, T. Paquet, L. Heutte, A writer identification and verification system, Pattern Recognit. Lett. 26 (13) (2005) 2080–2092. [9] L. Schomaker, K. Franke, M. Bulacu, Using codebooks of fragmented connected-component contours in forensic and historic writer identification, Pattern Recognit. Lett. 28 (6) (2007) 719–727. [10] A.A. Brink, R.M.J. Niels, R.A. van Batenburg, C.E. van den Heuvel, L.R.B. Schomaker, Towards robust writer verification by correcting unnatural slant, Pattern Recognit. Lett. 32 (3) (2011) 449–457. [11] G. Ghiasi, R. Safabakhsh, Offline text-independent writer identification using codebook and efficient code extraction methods, Image Vis. Comput. 31 (5) (2013) 379–391. [12] S. He, M. Wiering, L. Schomaker, Junction detection in handwritten documents and its application to writer identification, Pattern Recognit. 48 (12) (2015) 4036–4048. [13] E. Khalifa, S. Al-maadeed, M. Tahir, A. Bouridane, A. Jamshed, Off-line writer identification using an ensemble of grapheme codebook features, Pattern Recognit. Lett. 59 (2015) 18–25. [14] F.A. Khan, M.A. Tahir, F. Khelifi, A. Bouridane, R. Almotaeryi, Robust off-line text independent writer identification using bagged discrete cosine transform features, Expert Syst. Appl. 71 (2017) 404–415. [15] S. Fiel, R. Sablatnig, Writer identification and writer retrieval using the Fisher vector on visual vocabularies, 12th International Conference on Document Analysis and Recognition (ICDAR) (2013) 545–549. [16] M. Bulacu, L. Schomaker, Text-independent writer identification and verification using textural and allographic features, IEEE Trans. Pattern Anal. Mach. Intell. 29 (4) (2007) 701–717. [17] B. Leibe, A. Leonardis, B. Schiele, Combined object categorization and segmentation with an implicit shape model, ECCV workshop on statistical learning in computer vision, (2004) , pp. 17–32. [18] B. Leibe, A. Leonardis, B. Schiele, Robust object detection with interleaved categorization and segmentation, Int. J. Comput. Vis. 77 (1–3) (2008) 259–289. [19] C. Harris, M. Stephens, A combined corner and edge detector, Proceedings of the 4th Alvey Vision Conference, (1988) , pp. 147–151. [20] S. Agarwal, A. Atwan, D. Roth, Learning to detect objects in images via a sparse, part-based representation, IEEE Trans. Pattern Anal. Mach. Intell. 26 (11) (2004) 1475–1490. [21] V.N. Vapnik, The Nature of Statistical Learning Theory, Springer-Verlag, London, UK, 1995. [22] C. Freitas, L.S. Oliveira, R. Sabourin, F. Bortolozzi, Brazilian forensic letter database, 11th International Workshop on Frontiers on Handwriting Recognition (2008) 19–21. [23] M. Diem, S. Fiel, F. Kleber, R. Sablatnig, CVL database: an off-line database for writer retrieval, writer identification and word spotting, 12th International Conference on Document Analysis and Recognition, (2013) , pp. 560–564. [24] S.A. Mahmoud, I. Ahmad, M. Alshayeb, W.G. Al-Khatib, M.T. Parvez, G.A. Fink, V. Margner, H.E.L. Abed, KHATT: Arabic offline handwritten text database, 13th International Conference on Frontiers in Handwriting Recognition (ICFHR), (2012) , pp. 447–452. [25] S.A. Mahmoud, I. Ahmad, W.G. Al-Khatib, M. Alshayeb, M.T. Parvez, V. Märgner, G.A. Fink, KHATT: an open Arabic offline handwritten text database, Pattern Recognit. 47 (3) (2014) 1096–1112. [26] H.T. Nguyen, C.T. Nguyen, T. Ino, B. Indurkhya, M. Nakagawa, Text-independent writer identification using convolutional neural network, Pattern Recognit. Lett. 121 (2019) 104–112. [27] S.M. Beitzel, E.C. Jensen, O. Frieder, MAP, in: L. Liu, M.T. Özsu (Eds.), Encyclopedia of Database Systems., Springer, Boston, MA, 2009. [28] A.K. Jain, A. Ross, P.J. Flynn (Eds.), Handbook of Biometrics, Springer Science & Business Media, 2007.

[29] L. Ballard, D. Lopresti, F. Monrose, Evaluating the security of handwriting biometrics, 10th International Workshop on Frontiers in Handwriting Recognition (2006). [30] S.N. Srihari, L. Meng, L. Hanson, Development of individuality in children’s handwriting, J. Forensic Sci. 61 (5) (2016) 1292–1300. [31] C.P. Saunders, L.J. Davis, J. Buscaglia, Using automated comparisons to quantify handwriting individuality, J. Forensic Sci. 56 (3) (2011) 683–689. [32] C. Djeddi, I. Siddiqi, L. Souici-Meslati, A. Ennaji, Codebook for writer characterization: a vocabulary of patterns or a mere representation space? Proceeding of the 12th International Conference on Document Analysis and Recognition (2013) 423–427. [33] S. Srihari, C. Huang, H. Srinivasan, V. Shah, Biometric and forensic aspects of digital document processing, Digital Document Processing, Springer, Heidelberg, 2007, pp. 379–406 (chapter 17). [34] R.A. Huber, A.M. Headrick, Handwriting Identification Facts and Fundamentals, CRC Press, Boca Raton, LA, 1999. [35] R.N. Morris, Forensic Handwriting Identification Fundamental Concepts and Principles, Academic Press, 2000. [36] R. Niels, L. Vuurpijl, L. Schomaker, Automatic allograph matching in forensic writer identification, Int. J. Pattern Recognit. Artif. Intell. 21 (01) (2007) 61–81. [37] L. Schomaker, F. Katrin, B. Marius, Using codebooks of fragmented connectedcomponent contours in forensic and historic writer identification, Pattern Recognit. Lett. 28 (6) (2007) 719–727. [38] R. Niels, L. Vuurpijl, L. Schomaker, Introducing trigraph-trimodal writer identification, Proceedings of European Network of Forensic Handwriting Experts, (2005) . [39] K. Franke, L. Schomaker, C. Veenhuis, C. Taubenheim, I. Guyon, L. Vuurpijl, M. van Erp, G. Zwarts, WANDA: A Generic Framework Applied in Forensic Handwriting Analysis and Writer Identification, HIS 105 (2003) 927–938. [40] S.N. Srihari, H. Srinivasan, K. Desai, Questioned document examination using, J. Forensic Doc. Examination 18 (2007) 1–20. [41] L. Guarnera, G.M. Farinella, A. Furnari, A. Salici, C. Ciampini, V. Matranga, S. Battiato, Forensic analysis of handwritten documents with GRAPHJ, J. Electron. Imaging 27 (5) (2018)051230. [42] G.J. Tan, G. Sulong, M.S.M. Rahim, Writer identification: a comparative study across three world major languages, Forensic Sci. Int. 279 (2017) 41–52. [43] A. Rehman, S. Naz, M.I. Razzak, Writer identification using machine learning approaches: a comprehensive review, Multimedia Tools and Applications, (2018) , pp. 1–43. [44] J. Moszczynski, The multi-individuality of handwriting, Forensic Sci. Int. 294 (2019) e4–e10. [45] C. Djeddi, I. Siddiqi, L. Souici-Meslati, A. Ennaji, Text-independent writer recognition using multi-script handwritten texts, Pattern Recognit. Lett. 34 (10) (2013) 1196–1202. [46] M. Jehanzeb, G.B. Sulong, I. Siddiqi, Improving codebook-based writer recognition, Int. J. Pattern Recognit. Artif. Intell. 27 (06) (2013)1353003. [47] Y. Hannad, I. Siddiqi, C. Djeddi, M.E.Y. El-Kettani, Improving Arabic writer identification using score-level fusion of textural descriptors, IET Biom. 8 (3) (2019) 221–229. [48] B. Zhang, S.N. Srihari, S. Lee, Individuality of handwritten characters, Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR (2003) 1086–1090. [49] S. Al-Maadeed, A. Hassaine, A. Bouridan, Using codebooks generated from text skeletonization for forensic writer identification, 2014 IEEE/ACS 11th International Conference on Computer Systems and Applications (AICCSA), (2014) , pp. 729–733. [50] G. Louloudis, N. Stamatopoulos, B. Gatos, ICDAR 2011 writer identification contest, International Conference on Document Analysis and Recognition ( 2011) 1475–1479. [51] K. Franke, M. Köppen, A computer-based system to support forensic studies on handwritten documents, Int. J. Doc. Anal. Recognit. 3 (4) (2001) 218–231. [52] R. Fernandez-de-Sevilla, F. Alonso-Fernandez, J. Fierrez, J. Ortega-Garcia, Forensic writer identification using allographic features, 2010 12th International Conference on Frontiers in Handwriting Recognition, (2010) , pp. 308– 313. [53] J. Galbally-Herrero, S. Gonzalez-Dominguez, J. Fiérrez, J. Ortega-Garcia, Biografo: an integrated tool for forensic writer identification, Proceeding International Workshop on Computational Forensics, (2015) , pp. 200–211. [54] S. Srihari, G. Leedham, A survey of computer methods in forensic handwritten document examination, Proceedings of the 11th Conference of the International Graphonomics Society(IGS), 2003 (2003) 278–281.