On-line recognition of handprinted characters: Survey and beta tests

Pattern Recognition. Vol. 23. No. 9. pp. 1031-1044. 1990

0031-3203/9() $3.00 + .00 PergamonPressplc ~) 1990Pattern RecognitionSociety

Printed in Great Britain

ON-LINE RECOGNITION OF HANDPRINTED CHARACTERS: SURVEY AND BETA TESTS FATHALLAH NOUBOUD and R,~,JEAN PLAMONDON* Laboratoire Scribens, Department of Electrical Engineering, Ecole Polytechnique de Montr6al, P.O. Box 6079, Station A, Montreal QC, Canada H3C 3A7

(Received 3 July 1989; in revised form 26 October 1989; received for publication 7 December 1989) Abstract--In the first part of this paper, we present a survey of the state of the art in on-line handprinted character recognition technology. Data preprocessing and classification, and character recognition results are examined. A number of character recognition systems are compared. The second part of this paper describes beta tests carried out on a commercial system. The results are analysed with particular attention to the effects of handwriting variability and constraints imposed, and to the human factors involved.

Handprinted characters

On-line recognition

Alpha and beta tests

INTRODUCTION

Man-machine communication has to date been based on the use of the keyboard. This interface is not well-suited to users who have not mastered the keyboard and if, in addition, the number of characters is very large (as in the Chinese and Japanese alphabets, for example), this mode of communication becomes cumbersome and inefficient. For this reason, automatic handwriting recognition systems have been developed. A number of surveys have been made of the state of the art in this field.(~-3) Automatic handwriting recognition systems are classified into two categories according to the mode of data acquisition used. On-line systems require the use of a digitizing tablet, while off-line systems use optical digitizing devices. ~4.51The mode of data acquisition has a significant effect on the other modules of the system, such as description and comparison, etc. In the first part of this paper, we review the latest on-line recognition methods and examine the various modules which make up a recognition system (Fig. 1). A description of the preprocessing techniques (filtering, smoothing and normalizing) applied to the digitized data is followed by a discussion of the various approaches available for describing a character and the appropriate comparison techniques * To whom correspondence should be addressed.

(dynamic programming, Euclidian distance . . .). Finally, the results obtained by different systems are provided, in addition to comparative tables. Among the problems inherent in handprinted character recognition, and which generate recognition errors, are the variations which may occur in the writing of characters and the resemblance which exists between some of the characters. To minimize these errors, a data base containing a large number of specimens provided by many writers must be studied. The system developed by Ward et al. t6"s) is one of those rare alphanumeric character recognition systems where experimental analysis is supported by just such a large data base. We have, therefore, chosen to carry out Beta tests on this system to evaluate its performance and also to analyse the problems associated with handwritten character recognition. The second part of this paper describes these tests and documents the recognition results, in addition to providing an analysis of the human factors involved.

1. THE STATE OF THE ART

1.1. Variations in handwriting

There may be significant differences in the way a character is written depending on the style of the writer. These variations may be in the shape of the character or in the number or the order of its

t Fig. 1. Diagrammatic representation of a recognition system. 1031

1032

FATHALLAHNOUBOUDand R~JEAN PLAMONDON

7 North American

European

Fig. 2. Writing specimens. components. A component is a stroke between two consecutive pen lifts. (9) Ward and Kuklinski ~8~ have reported that the shape of the characters may vary with the origin of the writer, for example, Europe and North America (Fig. 2) or with a tendency to write cursive handprinted characters (Fig. 3).

Handprinted characters

Cursive handprinted characters

Fig. 3. Writing specimens. The number of components may vary a great deal for one character (Fig. 4). The order of the components in a character may vary as well. Writers have a tendency to draw the vertical components before the horizontal ones, and the left components before the right. ~8~

Fig. 4. Example of the character "E".

1.2. Digitizing characters Digitizing tablets may play the role of a mouse (to indicate and select a point on the screen, in a menu, for example). However, they are also useful for digitizing writing, signatures and drawings. The advent of digitizing tablets has meant a considerable increase in the number of on-line writing recognition tasks performed. I~°) The technologies used are varied, but they belong to two main families. Most handwriting recognition systems use electromagnetic/electrostatic tablets which send the coordinates of the pen tip to the host computer at regular intervals. Others H11 use pressure-sensitive tablets. The advantage of these devices is that an ordinary pen may be used. However, a movable surface above the tablet is essential to accommodate hand pressure and to avoid altering acquisition data. The resolution of the digitizing tablets is usually over 200 points per inch and the transmission speed is more than 100 points per second. Kim and Tappert tL') have studied the evolution of the recognition rate and calculation time as functions of these parameters.

There are other data acquisition devices in addition to digitizing tablets. Special pens are used (13"14~ to extract the dynamic parameters of the writing and to characterize the handwritten character. 1.3. Preprocessing Raw digitized data must be subjected to a number of preliminary processing steps to make it usable in the descriptive phase of character analysis. The main objectives of this processing are a reduction in the amount of information to be retained, the elimination of certain imperfections and the normalization of the data.

I slcl lIle'l l lsl

I

Fig. 5. Segmentation into boxes.

To separate the digitized characters, an external segmentation procedure must be carried out. The most widely used method for doing this is to use a box (Fig. 5) to contain each character, t~'~5-18~ Segmentation may be spatiaF ~9) or temporal t'-°'2n (for example, a pause of 300ms may mean that the character is complete and that subsequent data correspond to the next character). Burr t22~ imposes the constraint that the characters (lower case letters) consist of only one stroke. The technique of hierarchic segmentation 05"23~ involves surrounding the character with a rectangle. The object of the smoothing operation is to eliminate imperfections due to the hardware and the tablet, and to trembles in the writing, hesitations, etc. This is done by replacing a point with the average over its neighbours, oL 16.21.24-277 The data are then filtered to reduce the amount of information to be retained and to eliminate wild points. 111"16-18'26"28~ Testing on the points of pronounced curvatures make it possible to keep characteristic points. In this way, for example, a " U " may be distinguished from a "V". ~9-zn Tang et al. 129) carry out a thresholding on the extrema in x and y in order to reduce the amount of information to be retained. Tappert °9~ reduces the data representing a dot at only one pair of coordinates. Some hooks may appear at the beginning or end of a component as a result of a writer's tendency to connect the components. Their presence may be detected by sudden changes in direction and by their short length. Eliminating these strokes makes it possible to reduce the amount of information to be retained and avoid recognition errors3 ~5'~9'2n There are other preprocessing tasks which are designed to correct breaks in the strokes. H7'-'41 The data representing the character may be normalized in terms of size, O1"15"j6'22"26) orientation ~22~

On-line recognition of handprinted characters and position/H) Doster and Oed (23) normalize the direction codes and then reconstruct the character before proceeding to the comparison phase. The parameters involved in the preprocessing phase (smoothing, filtering and normalizing) may be optimized after several simulation experiments m) have been carried out. 1.4. Recognition

1.4.1. Description. A character is classified by means of a description which is made up of parameters or codes extracted from the various parts of the character. These parts or segments are obtained when the internal segmentation process is completed. There are, however, s o m e c a s e s (30-33) where the character is coded globally, without segmentation. The box containing the character is divided into rectangular zones, and coding consists of recording the sequence of zones touched by the pen. To enter the character " A " , for example, the pen must touch the zone sequence: 6, 4, 1, 6, 4, 3, 4 (Fig. 6).

F irm: o--, L-,.J

- -

{

'

r-~, t

]

Ill I ^

5/16"

r-~

_ .

t l °till u

t~

5

3

8 , 9

Recognition is based on the principle that variations between different characters are more important than variations between specimens of the same character. With certain pairs of characters, however, there may be confusion, for example, G and 6, I and 1, U and V. This must be taken into account in the description of the characters and in the comparison phase in order to avoid recognition errors.

II

1033

llll

z Ittl

^ It

;:s It

__~'-~JtZ__~--~J ¢_L....,_I

Fig. 6. Digitizing zones (according to reference 31). Reprinted with permission (© 1986 Academic Press Inc.).

D,,..---'U

11

13

12

Fig. 7. Direction coding. Hs) Reprinted with permission (© 1982 IEEE).

There are three ways to segment a character. The first is to segment it into elements: the vectors joining successive points. (25'28"~)The elements are generally described by their length or height, and their angles with respect to the horizontal. The angles are generally quantified using quadrants (Fig. 7). The second way to segment a character is to identify the curvature maxima or the local extrema in x and y.(20.21.29) The segments thus obtained may be described by their position and slopes. 12H Ito and Chui (2°) classify segments in straight-lines or curvilinear strokes prior to coding them by direction. The third way to segment a character is to separate it into components where pen lifts occur. (8"11"1°-19"23"24"26"29"35"36) The components may be described by a chain coding the extrema in x and y(8.29) (Fig. 8), or by the direction codes of the elements which make up the component (tT-tg'23`2~ (Fig. 9) and its position in the character, ttS'20) Hidai et al. 13s) use the position and the direction of the component globally. There are other approaches to describing characters. Fourier coefficients may be used for characters consisting mainly of curves (37) or concatenated straight strokes. HI) In the case of Chinese characters. which are formed primarily by straight strokes, approximation by a small number of points has proved effective. 06,3s.39) Parameters specific to a given type of character, for example, Arabic characters, may be defined. (24)

oI L

L

U

~DD~D UU D

Fig. 8. Coding of extrema (according to reference 8). Reprinted with permission (© 1988 IEEE).

1034

FATHALLAHNOUBOUDand REJEAN PLAMONDON 360

:/ I

315

25

z

18o

,c

135

\/

\

270 i

I |

I i I I

I I l II ! I

9O

t

45 ORIGINAL

0

I

. . . . . . . . . . . . . .

2

4

6

8

10 12

|,,

14 16

18

20

NORMALIZEDDIRECTION/ PATH-LENGTH DIAGRAM Fig. 9. Coding of directions (according to reference 23). Reprinted with permission (© 1984 IEEE).

1.4.2. Comparison. Comparison may be carried out in two phases. The most distant prototypes of the character are discarded in the first phase with dynamic programming (28) or linear matching.(27) The second phase compares the candidates which have been selected to complete the classification of the character. This procedure makes it possible to reduce calculation time. Comparison techniques are closely associated with the neture of the description retained. Most of these are based on dynamic programming. (15"18"19"22"26"28"29"34"'1°)Thus, the vector of the variable representing the character is compared to a group of reference vectors and is assigned to the class corresponding to the minimal distance. This represents considerable calculation time, possibly as much as several seconds for each character. I]3) A solution to this problem is the use of a processor

(ne)

g; i I I I

! ! i

(re)

dedicated to dynamic programming (28) for determining the best candidates. Another solution, proposed by Ikeda et al., (26) is to restrict the use of dynamic programming to certain characters, thereby reducing the size of the dictionary and making use of information other than the shape of the components for classification purposes (number and order of components, etc.). To improve recognition results, Yoshida and Sakoe (t8) artificially prepare sections common to some different characters to limit the number of variations (Fig. 10), and use local information to distinguish characters that resemble each other. In this system, by connecting ,:~>mponents, the constraint on their number is avoided (Fig. 11). Euclidian distance is used for the comparison in some systems. (23)Yet another comparison technique, the Bayesian Rule has been found suitable when

(kel)

(shl)

i" °1' ,US',--/

'US:--r' ~

(kl)

'

Ill

(hy(~)

i.....j

:-;,I

,

L

(wa)

I

(kun)

,-, (sh~)

Fig. 10. Sections common to different characters. ""~ Reprinted with permission (© 1982 IEEE).

On-line recognition of handprinted characters

Fig. 11. Connecting the components (according to reference 18). Reprinted with permission (© 1982 IEEE).

using Fourier coefficients to describe characters. Im In another study Odaka et al. f16) carry out a statistical classification using the point coordinates representing the character strokes. Character classification may be accomplished by means of component combinations. Is) Each component of the character is identified by its coding chain. Classification consists in looking for the combination of components in the dictionary corresponding to the character. Using the allowed combinations of elements (Fig. 12a), Huh and Beus c251 have constructed the allowed components (Fig. 12b). The character is thus identified by combinations of these components. Syntactic-statistic classification is another method of classifying characters3 t7''-°'2~'29~ The syntactic phase of the classification is formed using a grammar, 12°)a decision-making tree 1~7~or finite-state non-deterministic machines/29) The statistic phase uses the stroke description parameters to complete character classification. 1.5. R e s u l t s Experiments to test character recognition systems are based on a sizeable group of characters and writers. The writers are asked to write characters * Average writing speed is 1.5-2.5 characters per second for the Roman alphabet, and 0.2-2.5 for Chinese characters. t3)

\ '°'

-J/

1035

within a framework of constraints as to the number and order of the components, and the size of the characters, etc. These constraints are very stringent in some systems. 125291 Also, calculation time is itself a constraint since even in a so-called "'real-time'" system, the recognition of a character requires about one second of processing time at least.* In short, the way data are acquired is quite unnatural. From today's perspective, the use of these systems requires several improvements. Recent studies have been focussed, therefore, on reducing these writing constraints, especially with regard to the order of the componentsjn.17.35) their number ~t8) and the adaptation of the system to the writer 1~'2u by means of updating. The specifications of the various systems are summarized in the following Table 1 with their recognition results. It should be noted, however, that it is very difficult to compare the results obtained by different systems. This is because the equipment, procedures and other factors may vary considerably from one system to another. The systems in this table process different types of characters, including alphanumeric, Chinese, Korean and Arabic characters. Recognition methods vary greatly, but dynamic programming is common to 40% of these systems. This is an indication, once again, of how powerful this comparison tool is. In fact, dynamic programming is found in a number of domains, including shorthand symbol recognition 14~ word recognition 142~and handwritten signature verification, ~43-45) among others. In this table, the recognition rates are nearly all above 92%. However, such tests are not always based on a large enough group of characters and writers. In order to be able to make generalizations about a system's performance, the data base must contain thousands of characters supplied by dozens of participants. The only alphanumeric character recognition system which fits these criteria is the one developed by Ward et al. (~-~) We shall present, therefore, an experimental study (beta tests) that we have carried out on the system marketed by PENCEPT. The second part of this paper gives a description of the P E N P A D 310 system and of our experiments to test its recognition algorithms.

/

I NO

171,

Fig. 12. (a) allowed elements (b) allowed components. 12~ Reprinted with permission (© 1982 Pergamon Press plc).

Alphanumeric

All symbols

Loy, Landau tzn

Lu B o r d e r s e n '-'x~

Ito, Chuit~

et alJ ~)

lkeda

Huh Beus ¢25~

Kanji, Hiragana Katakana and alphanumeric Alphabet

Kanji, Hirangana Katakana and alphanumeric Korean

Hidai

et al. o5~

All symbols

Letters and Arabic numerials Kanji, Hiragana Katakana and alphanumeric Alphanumeric and others + * (), etc. Lower case letters

Types of characters

Doster Oed~Z3~

B u r r 122)

Baron Plamondon o~)

Arakawa" ~)

el al. (24)

Amin

Authors

---Comp. number --Comp. order

Elastic matching Euclidian distance

--Directions - - L e n g t h of elements --Directions --Position of elements --Directions ------Component position ---Comp. number ---Comp. order --Slow writing --Signal end of character ---Comp. number ---Comp. order --Comp. number ---Comp. number ---Comp. order ---Comp. number ---Comp. order

Combination of components Elastic matching Syntacticstatistic Syntacticstatistics Elastic matching

Combination of elements

--Directions --Position oL segments --Directions of segments --Directions --Position of segments --Directions ~urvatures --Velocity

----Comp. number

--Boxes --Zone sequence One component per letter

Identification of code chains

Zone codes

---Comp. order --Comp. number

---Comp. order ----Comp. number

Constraints

Bayesian Rule

Structural

Classification

Fourier coefficients

Special parameters

Feature description

10

3

5O

1

35

Number of writers 219

Number of characters

(b) iO0

1 specimen per symbol

1000

2340

17463

3OO

25OO

96

98.8

98.3

92 to 98

92

99.5

(a) 92

96O

(b)

99.2 to 99.7

95.4

Recognition results (%)

(a) 2880

35 specimens per char.

Table i. Comparison of handwritten character recognition systems

Real time

Real time

Real time

Real time

Real time

Deferred time

Real time

Real time

Calculation time

> o z o z

x

r,

c

o

7 o c

>

>

,-I

,a-i >

ox

Mandler

--Directions --lengths of elements --Directions --Lengths of elements

Numerals

95 characters ASCII

Chinese

Telmosse Plasmondon ~4)

Yoshida Sakoe .a)

el al. (~7)

Ye

et al. ~

Alphabet, Kana and others: + * (), etc.

Chain code

Alphanumeric and others + * (), etc.

Tappert ~27~

Ward

--Directions --Heights - - x y offs. from cent. of gravity --Directions - - x y offs. from cent. of gravity Accelerometer signals

--Directions of elements --Position of strokes --height/ width of character Fixed number of points per stroke Chain code

Upper case, lower case and numbers

Kanji, Hiragana Katakana and alphanumeric Numerals

All symbols

Tappert ql~

el ill. 12,~b

Tang

et al. "6~

Odaka

et al. c~)

Elastic matching

Syntacticstatistic

Combination of components

Mahanalobis' distance

Elastic matching

Elastic matching

Syntacticstatistic

Statistic ( from point coordinates)

Elastic matching

number

order number

order number

--Boxes ---Comp. order

--Boxes --Comp. --Comp. --Boxes ----Comp. --Comp. --Boxes ----Comp.

--Comp. order ~ o m p . number - - N o significant trembles

--Boxes ~ C o m p . number

--Boxes ---Comp. order ~ o m p . number

3

500

30

30

40

12

500

85000

15000

4284

1950

300

9135

9360

99.5

92 to 99

93.4

71.9-94.5

97.3

94. !

89

99.5

96.2

Real time

Real time

Real time

Real time

Real time

"-4

e-~

g

g

O

O

1038

FATHALLAH NOUBOOD and RE.lEAN PLAMONDON

2. PENPAD 310 TESTING

The two main objectives of the work described here were to test the algorithms of the tablet statistically and measure their performance, and to document and analyse the comments and observation of the participants in the experiments. In this part of the paper, we first give a brief description of the recognition system, followed by a detailed description of the experiments, the results obtained and their analysis.

2.1. Description of the tablet The P E N P A D 310 system has a handwritten character or symbol recognition processor. Data acquisition is electromagnetic. When the pen is in contact with the writing surface, the coordinates of the point are sent to the recognition module at regular intervals. This module is linked to the host computer by a Serial RS-232 interface. The characteristics of the tablet are as follows: --active surface area: l l " x l l " (27.94cm by 27.94 cm) --thickness: 0.5" (1.27 cm) --possible resolution: 1000 points per inch (394 points/cm) ----data transmission rate: 100 points per second at 9600 baud. The PENCEPT processor is capable of recognizing the 59 characters shown in Fig. 13. The work surface may be divided into zones. Each of these zones may be programmed in a different mode: character recognition, mouse emulation, graphics digitizing, etc. It is possible to further divide a zone into boxes. When the pen touches a box, the tablet executes the related command or sends it to the host computer. The pen has a three-position switch which may be programmed to send commands to the tablet or to the computer. A driver program interprets the data transmitted by the recognition module. After its installation with a list of commands programming the tablet, it remains resident in the memory with DOS.

A CDEP No?

The host computer is a C O M P A Q 386 microcomputer (16MHz). Further information about the tablet may be found in the PENCEPT documentation. (~)

2.2. Description of the experiments The tests that we have carried out are of the beta type. Pressman (4T) defines alpha and beta tests. Alpha tests are carried out in the system designer's own environment and are controlled by him, whereas beta tests are carried out by the user-client in a real system application environment not controlled by the designer. The goal is to complete alpha testing of the system. Beta testing is important in that it may reveal the existence of certain problems that did not become evident during the alpha testing stage. The designer may then study these problems and improve his product. Our experiments involved a population of 28 writers which included students, researchers and Ecole Polytechnique personnel. Both sexes were represented (14.3% female and 85.7% male). Most of the participants were French-Canadian (78.5%), and the balance were of other nationalities (14.3% European and 7.2% North-African) and their mother tongue was not French (7.2% Arabic and 7.2% Italian). The experiments took place over a period of 3 weeks. Each of the 28 participants spent approximately one hour in the acquisition session and were paid for their time. During a session, the writer was seated and could position the tablet to suit himself. In order to familiarize the writer with the equipment, he was asked to fill in an identification sheet using the P E N P A D 310 pen. Thus, at the same time, we obtained pertinent information such as the writer's name, age, sex, profession, nationality, mother tongue, etc. (4s'49) The writer then wrote nine series of characters. A series contained 59 characters that were recognizable by the P E N P A D 310. The order of the characters was random and different in every series, The participant wrote down the character appearing on the screen. If the recognition result did not match the character

NIO-K STuvwx 7';7L /\

7GEl Fig. 13. Characters recognized by PENPAD 310.

On-line recognition of handprinted characters on the screen, the programme asked for a validation. Thus, if the writer had written the wrong character, had not respected the imposed constraints (see Type B and Type C below), or had written a character the shape of which did not match the character on the screen (for example, an uncrossed 0, an I uncrossed top and bottom, etc.), data acquisition was restarted. At the end of the session, the comments and observations of the participant were documented. The tablet may be configured to recognize characters either in boxes or without boxes. In addition, some characters ( \ / ( ) , etc.) are not easily recognized unless a few constraints are respected. An experimental study cannot be exhaustive unless these facts are taken into account. We have, therefore, defined 3 types of acquisition: Type A: no box and without constraints. Type B: no box and with constraints. Type C: with boxes and constraints. Every writer wrote three series of characters in each of these three types of acquisition. 2.3. R e s u l t s a n d a n a l y s i s 2.3.1. E r r o r rates. The data base contains 14,868 characters:

leads to higher incorrect recognition rates than rejection rates (to a factor of 6). We did not notice any evidence of fatigue among the participants. The error rates did not increase during the final acquisition series. The error rate associated with Type C acquisition is 10.2%. This is higher than the 6.6% rate obtained by Ward et al. I"-~l A higher rate was, however, predictable since it is expected that performance will be inferior in beta tests than in alpha tests. This is because alpha tests are carried out by the designer whose understanding of the algorithm specifications means that the testing conditions are optimal. Most of the characters with high error rates are the punctuation characters (, ! () ; / \ ) . We have. therefore, calculated the global error rate only for the alphanumeric characters: Error rate for alphanumerics:

1 recognized 4 recognized 9 recognized G recognized J and Y recognized P recognized

There are two types of recognition errors. The first is the rejection of a character when the software cannot identify it. The second type is incorrect recognition, where the character is assigned to the wrong class. Table 2 shows the results obtained. This table shows the definite improvement in the Type B and C results over the Type A results. This improvement is due to the important role of the constraints imposed in Types B and C. The use of boxes in Type C does not lead to any significant improvement, the small reduction in error rates being probably due rather to the increased skill of the writer after having written six data acquisition series. Types B and C may, therefore, be regrouped as follows: 21.25%

Error rate with constraints

10.76%

The software has a tendency to assign the character to the "nearest" class, and seldom rejects it. This

8.53%

Table 3 shows detailed recognition results for each character: (in one column) the number of times the character is assigned to each class and the number of rejects. The alphanumeric characters with a high error rate are:

28 writers x 9 series × 59 characters = 14,868.

Error rate without constraints

1039

G

as % as Y as, as 6 as '" as T

6

G

Table 2. Recognition results

Type A Type B Type C Global

Reject rate

Incorrect recognition rate

Global error rate

2.98% 1.57% 2.46% 2.34%

18.26% 9.92f~ 7.56% 11.92%

21.25% 11.50% 10.02% 14.26%

: y P

P

Y -p

Fig. 14. Characters incorrectly identified. PR 2 3 : 9 - H

1040

FATHALLAH NOUBOUD and RF..JEAN PLAMONDON

Table 3. Recognition results I a n c

D

IC F G

~11

J

I(LHll

0

POlt

S

T

U

rid

X

Y

Z

J2~7 1

1

1

2

2

3

*

5

1

I,

~

~

"

t

$

1

(

)

*

*

,

-

.

/

:

~ ~-

~

1

i

I

5

5

$7

1

1

I

2

i

3

2

1

1

1

i 1

3

i

i

I

1

i Z~*

2

1

1

i

1

1

1o

leS 2~5

1

3

3

2

39

1

~S

I

2

1

1

1

1

2

1

)

2

1 S

1

1

1

1

2

1

5

23e 5

1

1 i

1

2

2 23a

II

2

3

5 23s

1

1

1

*

2

1 6

!

1 11

i 218 I

It

1

5

3

2

:137

3

1

I

1

1

i 1112

2

I

1

I1

1 237

1

\

17 2

2

1,

~s

II[

1

1

1

1

2

1

?

2

1

2

a ;t~,3

e

1 1

Zl,7

7

1

Z

225

6

7

1

2'*

0

~

2

1

1

1

11

1

0

1

10 Z41

1'

250

1

1

1

2

1 Z12

1

*b

I

2 1~

i 221

1 1

I

o

1

2

=3

2 1

15

i

3

1

1

2

1 2

71

2

*

1

1

2

~ ~0

6

1

39

1

1

1

B ~1 3

1

1 1

124

12 1

3*

1 1

s

1

~o3

1

73

1

1

1

1

z

5

8

2

3

o 1

17. e

8

1

I

193

2

lO 2

3

1

1

1 65512

27 71 1

243

2 k

102331

1

181 5 2

1

3

1

0

1 2

l

2~3

2

1

* 2

3 242

1 1

'

1 1

111~ 1

:)

9 218

2

2

2

z~z,13

27 2~3

1

1

,,

1

2

3

113

e

l

1 23S

3

1

202

3 161

2

21

1

1

11

1

11"1

1

1

1

Z

1

2

1

S

1

18

1 2~0

t

17

15 21~ i

1 /

3

1

2

~

1

I

2

1

S 1

1

1 11

1

X

238

1

1

2

7

167

B

6

33 240 1

7

2

10

20g

7

2 . 0 11t

*

29

i 113

X

1

223 1 1

117

1

227

?

21o

O

1

177 1117

f \

lla

)

Im~et

3

nJtOR

P~Tll

00

6O

40

80

10

10

[ ............

0 0

I

6

6

! ....

k L!

B 1011111SI41|I6171111111011111UlIMIII?Uill

Fig. 15. Individual error rates (Type A).

3

a

6

2

10

20 e

13 1S

On-line recognition of handprinted characters

60

1041

IRROJt IbL'll

60 40 80

~0 10 0

,Jilld!!!

.... !! .....

1 2 8 4 5 6 ? 8 0 10111|I814t61017181OMRIURSN4UMSTMH

Fig. 16. Individual error rates (Type B).

Figure 14 shows samples of the characters which were incorrectly identified (Type C acquisition). While it is easy to see that the specimen of character G really does look like character 6, there should have been no confusion about P and 4, for example. These recognition errors could perhaps have been avoided if the system had taken account of the characteristics which were evident in the images of the characters. The participants were comfortable using the system to differing degrees and the causes of the errors varied from one participant to another. Figures 15, 16 and 17 show the error rates in relation to individual writers for each type of acquisition. In Fig. 17, it should be noted that in the case of writers who adapted well to using the system, the error rate might be very low, for example, 2.82% for participants 24 and 28. However, for those who were not at ease with the system, the error rates

were quite high, for example, 23.16% for participant 14. The average calculation time for recognizing a character was 1.88seconds/character using boxes (Type C) and 2.36 seconds/characters without boxes.

2.3.2. The human factor. Every participant was asked to fill in an evaluation form at the end of the testing. These forms contained six questions and a few lines for general comments. These were the results:

Question 1: Did you find the equipment limiting? Answer: Very: 3.57% Somewhat: 39.28% Not very: 57.14%

Question 2: Did you find the procedure limiting? Answer: Very: 7.14% Somewhat: 32.14% Not very: 60.71%

IRROR RATT

eo I I

4O

3O

°t

I

0 1 2 3 4. 5 (5 '7 0 9 1 0 1 1 1 | I $ 1 4 t 6 1 6 1 ? I B I 0 2 0 8 1 1 J R U 2 4 U 8 8 2 " T M R

WRITERS

Fig. 17. Individual error rates (Type C).

1042

FATHALLAH NOUBOUD and Re:JEAN PLAMONDON

Question 3: Did you find Type A acquisition natural? Answer: Yes: 46.42% Somewhat: 35.71% Not very: 17.85%

Question 4: Did you find Type B acquisition natural? Answer: Yes: 3.57% Somewhat: 39.28% Not very: 57.14%

Question 5: Did you find Type C acquisition natural? Answer: Yes: 46.42% Somewhat: 39.28% Not very: 14.28%

Question 6: Would you prefer to use a keyboard? Answer: Yes: 89.29% No: 10.71% The pen used was thicker than an ordinary pen and was attached by a cord to the tablet. In addition, the writer had to wait until the character had been recognized before writing the next one. Although this might have been an inconvenience, most of participants did not find either the equipment or the software a hindrance (Questions 1 and 2). In the Type A testing, the characters were written naturally. However, only 46% of the participants found this type of acquisition natural (Question 3). This may be explained by the constraint imposed on the size of the characters. In fact, if the tablet is configured in the no-box recognition mode, the characters must be at least 6 cm high to be processed properly (Fig. 18). In Type B acquisition, constraints were introduced on some of the characters so that they would be recognized by the software. For example, the parenthesis had to be written from low to high so that it would not be confused with the C. The participants found that this type of constraint, added to the necessity of writing very large characters, made the procedure seem quite unnatural (Question 4). The tablet for Type C was configured in such a way that the characters could be written normal size. The writers found this type of acquisition to be natural in spite of the constraints imposed on some of the characters (Question 5). The great majority of the participants prefer the keyboard because of the slow speed of recognition and the constraints (Question 6).

Several points were brought up in the writers' comments. The most frequent comment was on the slow recognition speed. Eleven participants hoped that the procedure would be improved in this respect. The second most frequent comment was on the number of errors. Eight participants thought that there were too many recognition errors. Seven of the participants mentioned the problem of constraints. They pointed out that it was not at all natural, for example, to draw parentheses from bottom to top. The efficiency of the equipment was mentioned by six of the participants who found that the pen was too thick and that the cord attaching it to the tablet had a tendency to become tangled which bothered them a great deal during the testing process. Two participants found the procedure tiring (compared to using a keyboard) and one could see no point to the system at all. The comments were not all negative, however. Six participants felt that the software performed well (few errors) and one found that the procedure was not too limiting. In designing a handwritten character recognition system, such observations must be taken into account in order to make the procedure acceptable to the user. The primary objective must be to lower the percentage of those who would prefer to use a keyboard (nearly 90% in this study). It does seem to be very difficult, however, to find a compromise solution between writing freedom on the one hand and accurate recognition and processing speed on the other, especially when the equipment's capability is limited (a 16 MHz microcomputer). Even the P E N C E P T system, which is considered to be one of the best systems available commercially at the present time, is a compromise solution in these respects. The results obtained during this experimental study confirm the observations made in our laboratory by Jean Berthiaume35°~ His work was concentrated on perfecting a handwritten interface using software programming support designed by Beauregard and Plamondon. (m He confirmed the existence of problems associated with writing constraints, slow calculation speed and recognition errors. The work carried out in our study enabled us to quantify his observations and to delineate these problems with greater precision. 3. CONCLUSION

Ilcm Fig. 18. Character entered without a box.

In this paper, we have described the various online handprinted character recognition systems currently in existence. We have summarized the preprocessing, internal and external segmentation techniques, and the various approaches to character description and the related comparison methods. We have also carried out a series of tests on the P E N P A D 310 system. We have presented and analysed our results. The

On-line recognition of handprinted characters

c o m m e n t s of the participants and their observations on this application have also been analysed in depth. Some research studies have focussed on the reduction of writing constraints in existing recognition systems in o r d e r to make the acquisition procedure m o r e natural and m o r e efficient. For practical purposes, however, these systems are only applicable to languages with very large alphabets (Chinese, Japanese, etc.) or for use by those with little keyboard skill and only wishing to enter characters occasionally.

Acknowledgements--This research has been made possible through grants from the CRSNG of Canada, the FCAR Foundation of Quebec and with the support of the Ecole Polytechnique de Montr6al. Fathallah Nouboud wishes to thank the Ecole Polytechnique for its post-doctoral fellowship, and Pierre Yergeau (research assistant, Scribens Laboratory) for his valuable assistance.

REFERENCES

1. S. Moil, K. Yamamoto and M. Yasuda, Research on machine recognition of handprinted characters, IEEE Trans. PAMI-6, 386--405 (1984). 2. C. Y. Suen, M. Berthod and S. Mori, Automatic recognition of handprinted characters: The state of the art, Proc. IEEE 68, 469-487 3. C. C. Tappert, C. Y. Suen and T. Wakahara, Online handwriting recognitionaA survey. 9th Int. Conf. Pattern Recognition 2, 1123-1132 (1988). 4. D. Gokana and D. Juvin, Classification structurelle higrarchique en reconnaissance de caract/~res manuscrits, AFCET Conference, Identification et Reconnaissance des Formes, Tome 2, pp. 835-851 (1986). 5. O. Horn, J. Ciccotelli, P. Castel and R. Husson, Une m6thode de reconnaissance automatique de chiffres manuscrits, AFCET Conference, Identification et Reconnaissance des Formes, Tome 2. pp. 853-864 (1986). 6. J. R. Ward and B. Blesser, Interactive recognition of handprinted characters for computer input, IEEE Computer Graphics and Application 5, 24--37 (1985). 7. J. R. Ward, Issues in the validity of testing protocols and criteria for on-line recognition of handwritten text, PENCEPT report (1987). 8. J. R. Ward and T. Kuklinski. A model for variability effects in handprinting with implications for the design of handwriting character recognition systems, IEEE Trans. Syst. Man Cyber. 18,438--451 (1988). 9. R. Plamondon, A handwriting model based on differential geometry, Computer and Human Applications of Handwriting, R. Plamondon. C. Y. Suen, M. Simner, eds. World Scientific Publishing. Singapore (1989). 10. G. Lorette and R. Plamondon. Electronic handwriting: Trade-offs and new perspectives for education, 4th IGS Conference, Trondheim, Norway (1989). 11. H. Arakawa. On-line recognition of handwritten characters--Alphanumerics, Hiragana, Katakana, Kanji, Pattern Recognition 16, 9-21 (1983). 12. J. Kim and C. C. Tappert, Handwriting recognition accuracy versus tablet resolution and sampling rate. IEEE 7th Int. Conf. Pattern Recognition, Montreal, pp. 917-918 (1984). 13. H. D. Crane and R. E. Savoie, An on-line data entry system for handprinted characters, Computer 10, 4350 (1977). 14. P. Telmosse and R. Plamondon, Caract6risation et classification de hombres manuscrits par m6thode star-

1043

istique, Proc. RA1/IPAR 86, Identification and Pattern Recognition, Toulouse, pp. 865~--877 (1986). 15. E. Mandler, Advanced preprocessing technique for online script recognition of non-connected symbols. 3rd Int. Symp. Handwriting and Computer Applications. pp. 64--66 (1987). 16. K. Odaka, T. Wakahara and I. Masuda. Stroke-orderindependent on-line character recognition algorithm and its application, Reo. Elec. Com. Lab. 34. 79-85 (1986). 17. P. J. Ye, H. Hugli and F. Pellandini, Techniques for online Chinese character recognition with reduced writing constraints, 7th Int. Conf. Pattern Recognition, pp. 1043--t045 (1984). 18. K. Yoshida and H. Sakoe. On-line handwritten character recognition for a personal computer system, IEEE Trans. Consumer Electronics 28, 202-209 (1982). 19. C. C. Tappert, Adaptive on-line handwriting recognition, 7th ICPR, pp. 1004-1007 (1984). 20. M. R. Ito and T. L. Chui, On-line computer recognition of proposed standard ANSI (USASI) handprinted characters, Pattern Recognition 10, 341-349 (1978). 21. W. W. Loy and I. D. Landau. An on-line procedure for recognition of handwritten alpha-numeric characters, 5th ICPR, pp. 712-714 (1980). 22. D. J, Burr, Designing a handwriting reader, IEEE Trans. Pattern Anal. Mach. lntell, PAMI-5, 554--559 (1983). 23. W. Doster and R. Oed, Word processing with one-line script recognition, IEEE Micro 4, 36--43 (1984), 24. A. Amin, A. Kaced, J. P. Haton and R. Mohr. Handwritten Arabic character recognition by the IRAC system, 7th ICPR, pp. 729-731 (1980). 25. Y. H. Huh and H. L. Beus. On-line recognition of handprinted Korean characters, Pattern Recognition 15, 445-453 (1982). 26. K. lkeda, T. Yamamura, Y. Mitamura, S. Fujiwara, Y. Tominaga and T. Kiyono, On-line recognition of handwritten characters utilizing positional and stroke vector sequences, Pattern Recognition 13. I91-2(16 (1981). 27. C. C. Tappert, Speed. accuracy, flexibility trade-offs in on-line character recognition. IBM Research Report RC13228 (1987). 28. P. Y. Lu, W. Brodersen, Real-time on-line symbol recognition using a DTW processor, 7th ICPR, Vol. 2, pp. 1281-1283 (1984). 29. G. Y. Tang, P. S. Tzeng and C. C. Hsu, A microcomputer system to recognize handwritten numerals using a syntactic-statistic approach, 7th ICPR, Vol. 2, pp. 1061-1064 (1984). 30. R. M, Brown, On-line computer recognition of handprinted characters, IEEE Trans. Elec. Comput. 13. 750--752 (1964). 31. R. Plamondon and R. Baron, A dedicated microcomputer for handwritten interaction with a software tool: System prototyping, J. Microcomput. Appl. 9, 5160 (1986). 32. R. Plamondon and R. Baron. On-line recognition of handprint schematic pseudocodc for automatic FORTRAN code generator. 8th ICPR, Paris, pp. 741-745 (1986). 33. R. Baron and R. Plamondon, Logiciel dc developpement de programmes par interaction manuscrite, Technical report, Scribens Lab., Ecole Polytechnique de Montr6al. 34. D. J. Burr. Elastic matching of line drawings. 1EEE Trans. Pattern Anal. Mach. lntell. PAMI-3, 708-713 (1981). 35. Y. Hidai, K. Ooi and Y. Nakamura, Stroke re-ordering algorithm for on-line handwritten character recognition, 8th ICPR, Paris, pp. 934--936 (1986). 36. E. Mandler, R. Oed and W. Doster, Experiments in on-

1044

37.

38.

39.

40.

41.

42.

43.

FATHALLAHNOUBOUDand REJEAN PLAMONDON

line script recognition, 4th Scandinavian Conference on Image Analysis, pp. 75-86 (1985). S. Impedovo, B. Marangelli and A. M. FaneUi, A Fourier descriptor set for recognizing nonstylized numerals, IEEE Trans. Syst. Man. Cybern. SMC-8, 640-645 (1987). K. Odaka, H. Arakawa and I. Masuda, On-line recognition of handwritten characters by approximating each stroke with several points, Trans. IECE Japan E63, 168-169 (1980). K. Odaka, H. Arakawa and I. Masuda, On-line recognition of handwritten characters by approximating each stroke with several points, IEEE Trans. Syst. Man. Cybern SMC-12,898--903 (1982). R. Oed and W. Doster, On-line script recognition--A userfriendly man machine interface, COMPINT Conf. Computer-Aided Technologies, IEEE Cat., pp. 610614 (1985). C. G. Leedham and A. C. Downton, Automatic recognition and transcription of PITMAN's handwritten shorthand--An approach to shortforms, Pattern Recognition 20, 341-348 (1987). H. Sakoe and S. Chiba, Dynamic programming algorithm optimization for spoken word recognition, IEEE Trans. Acoustics Speech and Signal Processing ASSP26, 43--49 (1978). F. Nouboud, F. Cuozzo, R. Collot and M. Achemlal,

44.

45. 46. 47. 48. 49.

50. 51.

Authentification de signatures manuscrites par programmation dynamique, PIXIM Conference, Paris, pp. 345-360 (1988). F. Nouboud, Contribution ~ l'~tude et ~ la mise au point d'un syst~me d'authentification de signatures manuscrites, Thesis of Doctorat de I'Universit~ de Caen, France (1988). R. Plamondon and G. Lorette, Automatic signature verification and writer identification--The state of the art, Pattern Recognition 22, 107-131 (1989). Pencept Inc, Software toolkit for the PENPAD 310 and the PENPAD 320, PENCEPT Documentation (1986). R. S. Pressman, Software Engineering--A Practionner's Approach. McGraw-Hill, New York (1987). F. Nouboud and R. Plamondon, Experiences en reconnaissance en-ligne de lettres moul~es, Scribens Technical Report, Ecole Polytechnique de Montreal (1989). F. Nouboud and R. Plamondon, Reconnaissance enligne de caract~,res moul~s: techniques et r~sultats comparatifs, Canadian Conference on Electrical and Computer Engineering, Montreal, pp. 761-764 (1989). J. Berthiaume, Interface manuscrite pour ~diteur de programmation sch~matique, Projet de fin d'~tudes report, Ecole Polytechnique de Montreal (1988). D. Beauregard and R. Plamondon, Schematic coding on an IBM-PC, J. Microcomput. AppL 10, 91-100 (1987).

About the Author--R~EAN PLAMONDONreceived a B.Sc. degree in physics, and a M.Sc.A. and a Ph.D.

degrees in electrical engineering from Universit~ Laval, Quebec, QC, Canada in 1973, 1975, and 1978 respectively. In 1978, he joined the Ecole Polyteehnique, Universit~ de Montr¢~al, Montreal, QC, Canada, where he is currently an Associate Professor. In 1985--1986, he was involved in several research projects while a guest of the Computer Science Department, Concordia University, Montreal, Canada. the Motor Behavior Laboratory, University of Madison, Wisconsin, USA, the Department of Experimental Psychology, University of Nijmegen, The Netherlands and the Laboratoire de G~nie Electrique de Cr6teil, Universit~ de Paris Val-de Marne, France. His research interests are the computer applications of handwriting: biomechanical models, neural and motor aspects, character recognition, signature verification, signal analysis and processing, computer-aided design via handwriting, forensic sciences, software engineering and artificial intelligence. He is the founder and director of Laboratoire Scribens at the Ecole Polytechnique de Montreal, a research group devoted exclusively to the study of these topics. An active member of several professional societies and a senior member of IEEE ('85), Dr Plamondon is also a member of the board of the International Graphonomics Society and president of the IAPR Technical Committee on text processing applications. He is the author of numerous publications and technical reports. He is an avid swimmer and enjoys writing poetry, children's book and novels.

About the AUIhor--FATHALLAH NOUBOUD received a Ph.D. degree in pattern recognition from Universit~ de Caen, France in 1988. His Doctorate project consisted in the design of a handwritten signature verification system. At present, he is cooperating, as guest researcher, with Scribens Laboratory (Ecole Polytechnique de Montreal). His research interests are the computer applications of handwriting as character recognition and signature verification.