- Email: [email protected]
Application of autoregressive models to the study of the temporal structure of a handwritten text
Pattern Recognition Vol. 12, pp. 89-96. Pergamon Press Ltd. 1980. Printed in Great Britain. © Pattern Recognition Society.
0031-3203/80/0401-0089 $02.00/0
APPLICATION OF AUTOREGRESSIVE MODELS TO THE STUDY OF THE TEMPORAL STRUCTURE OF A HANDWRITTEN TEXT J. DUVERNOY
Information Systems Laboratory, Stanford University, Stanford, CA 94305, U.S.A. on leave from the Laboratoire de Physique G~6rale et Optique, Universit6 de Franche Corot6, 25030 Besan¢on Codex, France
(Received 14 May 1979; receivedfor publication 9 August 1979) Abstract - Variations of the slope of handwriting in several hundreds of lines are predicted by using autoregressive models. Most of the unpredictable lines correspond to. features such as the lack or excess of ink, the beginning of pages or paragraphs. The temporal succession of models is statistically analyzed in order to define descriptors, such as the transition matrix, that account for the building of a further state-space model for handwriting. Handwriting
Autoregressive models
Classification
INTRODUCTION:
TIME DEPENDENCE OF HANDWRITING Handwriting has been described as the analog of an imaging process. Questions of writer recognition and dating of manuscripts have been examined in the case of space invariant handwriting. ~1 Results of datings performed by a principal components analysis of writers' modulation transfer functions, extracted from pages covering a 35-year period, have shown that handwriting presents at least two kinds of temporal behaviour. A long term evolution clusters pages in successive periods (i.e. temporal classes). Short term variations - - within the extent of a page - - play the role of noise with respect to the former type of process, As a consequence it was possible to date a page by assignment to its most probable temporal class, but a finer dating inside the class was not possible. This uncertainty can be overcome by means of a more detailed study of the short-term processes. Their second order properties have been investigated ~2~and their temporal structure has been found to consist of successive stationarity domains isolated from each other by periods of non-stationarity. The size of these domains ranges from a few lines to a few pages. At this point the problem of characterizing a writer has no direct answer insofar as useful descriptors oftbese time sequences are not available. In fact, the questions of dating or identifying a sample of handwriting must be delayed as long as the underlying temporal process has not been investigated, The imaging system equivalent to the writer was first analyzed as a pure optical system. Its possible properties of space invariance were deduced from the study of the amount of distortion c31 that handwritten letters exhibit across a page. Parameters taken into
Stationarity
account were the coefficients of a space-variant transform that synthesizes a given handprinted letter from a model. When generating successive letters of a text, there is no deterministic relation between the successive sets of coefficients~21 of the successive spacevariant transforms. In some cases, they have a certain degree of stability (i.e. the ratio of their average value to their variance is high). Therefore the optical system equivalent to the writer is nearly time-invariant. This approximation holds for ancient writings (up to the 19th Century)but current modern handwriting turns out to be highly time-variant. Then pure optical parameters are no longer sufficient for describing the process, which has to be considered as the output of a time-variant and space-variant imaging system. In this paper we propose an approach to the modelling of short term processes ofbandwriting. The choice of the model one has to deal with results from a trade-off between the accuracyand tbecomputational cost of the description of the system involved. The input variables are not well-defined: several physical factors, such as the position of the hand, the fatigue of the writer, or the flow of ink, can contribute to the variations of handwriting. Other sources are difficult to assess, for instance the relationships between the meaning of what is being written and how it is written. Therefore we will assume that the system is driven by noise. One of the most powerful models currently in use is that where the output of a system is predictable from linear combinations o f past inputs and outputs. ~*~ In our case, it is specified as an autoregressive (AR) model. In this first approach we restricted ourselves to the prediction of a single parameter (viz. the average slope of handwriting with respect to the axis oftbe line. This slope is controlled by one of the parameters of the 89
PR 12 2--1~
Text segmentation
90
J. DUVERNOY
space-variant transform. (2~) A further study will deal with the prediction of sets of parameters ~5~(i.e. vectors) such as those mentioned above. Data were recorded fr°m several hundreds °flines written in a few days bY the same writer. The slope of handwriting was optically measured for each. The measurements are considered as a time series to be studied. This paper deals with the fitting of AR models to the considered time series, and with their use for depicting rhythms of writing. The first section of the paper recalls previous results on using a single AR model; the second section presents the linear prediction and filtering of the nonstationary time series by a collection of AR models, and the last section is devoted to a further statistical analysis of the succession of AR models along the text. As a conclusion, possibilities of performing a real time filtering of the text by analog representations of AR models in coherent light are suggested, as well as the definition of a state space model for a further Kalman filtering.
~ 0
~
// ~ ~ ~
X
~ ,z, L ..... [V "~ ~ ~ ' " ' ~
____, [
, n j
.
V
\ \ A
II I
\
/
U
1. USING A SINGLE AR MODEL
In this section we present successively the data to be processed and the corresponding time series, then the AR model that fits the whole series, and finally the filtering of the series by the model. This shows the existence of different domains of stationarity, which introduce the need for several AR models as described in the next section, 1.1 Data to be processed The laboratory of French literature of the Besanqon University provided us with the negatives of 17 pages from the notebook of the French writer, P. Claudel. They represent 454 lines written in a few days. The slope of handwriting, 0n, was measured for the 454 lines (n = 1..... 454) by using the optical system shown in Fig. 1. The negative of a page being illuminated with coherent light, the line No. n, is selected by a slit. The Fourier spectrum of the line is displayed in the back focal plane of the lens. It consists of two elongated distributions of light with an angular separation 0n. The vertical distribution is the Fourier transform of the slit along its principal axis. It plays the role of an axis of reference. The other distribution is the superposition of the Fourier spectra of the individual pupils supported by the line, i.e. the elements of the letters, which present an average slope, 0~, with respect to the axis ofthe line. Infactthisdistributionconsistsofafan of elementary spectra associated with each pupil, Therefore its angular width depends on the spread of the slope around the mean value 0,. An angular sampling of this fan will give a multivariate description of the regularity of handwriting in a further study, The Fourier spectrum of each line was scanned by a slit rotating around the focus of the lens. The variations of the luminous intensity, measured by a photodiode, were recorded in order to determine the angular separation of the two maxima of energy, the
//
[
Fig.1. The negative of a text is placed in the front focal plane (X, Y) of the convergent lens L. When illuminated with coherent light, its Fourier spectrum is displayed in the back focal plane. A line being selectedby a narrow slit parallel to X, the spectrum shows two axes of light, their angle 0 is that of handwriting with respect to X.
angle 0n. Such a measurement yields the average slope of handwriting, provided the number of letters per line is high enough (in order to avoid spurious maxima due to the spectrum of a sparse collection of pupils). A collection {On}, n = 1.... ,454 is then available, and represents a time series to be investigated. Because the temporal interval between any two lines is certainly not constant, this time series can be expected to be split into several different sets. 1.2 Fittin 0 an AR model The system that determines the time variations of the slope of handwriting is assumed to be linear, its output, 0F, is supposed to be predicted by a linear combination, 0'~, or past outputs~*~: p 0', = ~ aiOn-i. (1) i=' The set of coefficients {a~} and the order p are to be determined. Practically the order p can be defined by looking for the value that minimizes the squared prediction error,~10; - 0 n [2. According to Akaike ~8~ one can begin with a maximum value Pm~ = 3 N ~/2, where N denotes the number of data, i.e. the length of the time series (here N = 454 and Pm~xis close to 60). In
Temporal structure of a handwritten text our case, handwriting on a page did not seem to depend strongly on handwriting two pages before, i.e. approximately 40 lines. Therefore we chose p _ . = 40. Eventually the optimal order was p --- 5. In any case coefficients {a+} are computed by the A t a l - H a n a u e r method, (6) which consists of expressing the fact that the average squared prediction error is m i n i m u m for the {a+} ; in other words, its partial derivatives with respect to the {a+} are simultaneously equal to zero. Reference 9 presents the details of the model we found,
91
threshold is low. Particular AR models have to be fitted to each of these blocks. Therefore the time series one has to deal with appears to be a locally stationary process where, in terms of handwriting, particular "rhythms" can be detected. A first explanation has been proposed for the 120 unpredictable lines : more than 60 % are distinguished by features such as the lack (or excess) of ink, the presence of crossings out, or by the structure of the text itself (top of pages, beginning of new paragraphs or of n o n - c o n t e m p o r a r y units in a page). Figure 2 shows some of these characteristics, associated with unpredictable lines. But, as the time series was obviously nonstationary, we could not restrict ourselves to this single A R model and propose premature explanations to handwriting features. A thorough coverage of the time series by as many models as necessary was needed before any such attempt.
1.3 Filtering the time series An A R model allows the prediction with a m i n i m u m average error of successive samples of a time process• Besides being efficient by the small n u m b e r of coefficients it requires, the A R model is also useful in locating samples that do not have the average statistical properties of the time series the AR model is fitted to. Filtering the time series consists of rejecting samples O. which differ from the predicted value 0', by an a m o u n t greater than a given fraction e, of the actual value 0, % 10" _ 0,1 -- = - > +:. (2) 0, 0,
z. usiNt; SEVERAL Ar MODELS
With a threshold e. equal to 10% (e, = 0.1) half of the 454 lines c a n n o t be predicted• With +: = 20 % only 120 lines remain unpredictable• This fact shows that one deals only with an approximate stationarity when processing the series as a whole. More interesting is the segmentation of the series into homogeneous domains (or blocks) which c a n n o t be predicted when the
When filtering the time series by the previous model, it turned out that some blocks needed particular A R models. In fact, three different kinds of models were necessary to cover the complete time series with a prediction error less than 5 %. Figure 3 shows the filtering of three different blocks by the AR models which were defined on them. Models of the first kind are of order 10, models of the second kind deal with a past of 20 lines, and models of the third kind are of order 5 (one of them is the previous model defined in section 1.2.). Two models ofeach kind were necessary
@
L.,,L.
•
L ,..~.....> .....
......... /
/ ]
I
/
,
~
,,~.,"
+.Z.
'
,
"~ . . . . . . , . . . , ~ ............ "
+ •. . . . . . . . . . . . . . .
•
l,
.
'.~.'.
q
r'.,.,.l ..... ........
........... .
.... t- .+ +4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . e,,.,.A.,.. ..... , ,',1. - J+,P+,. , . , t .
i
'~
~..p
;..+
+-
7.-,~/'...,..~,~,~....../. ....;,.....~..; r'~l
+
I V ....
', i ,'
'
o
40+
P" . . . . . . .
I , * I
.
I
.'
[ *.,
;'1 /
t i X I~
T
.~/~
~ ....
q,
,
- ....
I ........
.+ . . . .
:.,- . . . .
' -+.
' ..
++ .+' . . . . . . . . . •
6+-."
+ J(", +''+'-'", ,-?'~ ......... ~ +,
/~
t.
/
.".<(.
-
]
"
/7 ....
,,,t ,'-', .,'. z ,.t ,, ,.,
"
/+~.:-. . . . . , " ' " ......... '+ ~"~ ". . . . ' " +'
"p
r
,
=.
. ....
+
131
-<
I
.,,.,,
:. . . .
"' ,"'" . . . . . . . .
.~.-<',,,.
I
I ........
b."" ","': ":~ .~,o,+, t~ . + . , , . . .
]
,7,J',",
......
z,./ ";
....
, ,:
.."
+.++.+
/ . , ¢, / , ..... . . . . . . .
,r;,"
"
",'.
....
l I
151
lime
Fig. 2. Filtering of a page by an AR model. Samples (i.e. lines), their predicted slope differing from their real slope by more than ~%, are rejected. The same page is filtered with two different thresholds of rejection. Features that turn out to be unpredictable from the slope of past lines are (arrows): the beginning of the page, an excess of ink and the beginning of a non-contemporary unit of text.
J. DUWRNOy
92 17!10 -]
to cover the time series. Figure 4 presents the filtering of two sections of the time series by using the collection of models, The threshold of rejection is fixed at r, = 5 %. Each sample ofthe series is predicted by the model that gives the least error, eventually with an accuracy better than 5 %. Only a few lines remain unpredictable. They are explained by the same reasons as those previously described. Now, the trouble is that the succession of samples is described by an equivalent succession of models that predict them well, but there is no homogeneity in this equivalent series. A better view of the process is obtained by applying a simple decision rule which yields a segmentation in homogeneous domains. Each sample is assigned the model which is dominant in a given fraction of its past (a past of 20 samples was chosen, where the most frequent model was selected). This rule leads to the segmentation depicted in Fig. 5 : periods denoted MI, MII and MIII are those where the first, second and third models are dominant, respectively. Periods denoted U are not determined, i.e. at least two kinds of models have the same frequency of occurrence. This first debugging proposes an approach to the overall time structure of the text. This structure does not fit the separation in successive pages, but does agree with the a priori information on the genesis of the text: the first seven pages are likely to have been written in one block, possibly the same day, as the following pages are chopped into smaller paragraphs written in successive days. Moreover each kind of model corresponds (Fig. 4) to a different and characteristic temporal behaviour ; for example, type II is very irregular, exhibiting many peaks, as type III is smooth. Therefore possibilities of further literary interpretations are opened to the specialists of Claudel manuscripts.
/~
"
.
Sl'
T i
J~ II ii ''
, '
line
74'. ~
|~.
T
' ,
line
~
I
- Y['~I~\VAVY"I
~^ V
SO', T
"
line
Fig. 3. Examples of the three kinds of AR models needed to cover the complete text. Their order is, respectively, 10,20and 5 for models I, II and III. The domains where these models were defined and their filtering by their own models (with a rejection threshold of 5 %) are shown here.
O
,,,
51"
~'J
T,
' iF
,I' I I I I
i
- -
I I
77"16~56! 0~~~ :~~i]~~~ £.5% 9'9line 21s
i
251line
Fig. 4. Filteringof two pieces of text by using the three kinds of AR models. The rejection threshold is 5 ~. For each sample, one chooses the model that predicts its value with the lowest error.
Temporal structure of a handwritten text
93
page +
page 6
. . . . "..... , .... :~:'.,;-~:';~:'T,~.':.:.~;'-:.:: [ ~: ': + ;:+ #+, ,:: ~+:+~+,=-/++£~++%-/:++~.-.~, :: ::., ;.:; /
[
~.~",.++'++,~m+ "" "
+ ,-,-. " ' ~
+.,,
:+;'++'" . . . . ++. .~' . ":
"-~
~,
+,+ + +'-+
, " : '::,,:: t:~;-" "~,':';. ,':.' " " ' ,:::, t: ++::+:+,m,. %!: ....
:'"~+~'+", . . . . ."'+;'. . . . . . . . .. ......
M III
,,.:,.
+." a . , + ' , , ~ + .
"
=.:~++:.~:,-': . . .+ . . '
m
:
:.
MN
Mine
page 7
page 12 ;:
,: : :.:-'.+,~::+.,:
_
_U
162
":! ~ ! . i : "
~ , "'
. , " ~,,'+ ,,+.',.~+~+. . . . .
'
:,,+ ,+',m+,.,:+,..,'
:
=
172
'
M I
.
i~;:"
,..++-~:,r:,
.
218
~!:!!:;"
.
O
"~ ~'" ~ .J'.,N =
," +- "- : ~! + ;'":
'~;.+~-
,+
'
_l,l.
ui_
327 336
....
lu
245
Mill
363 370 M I I
'i~
~ _
391
u
_
406
,:.,~,+.+;.
. MIll
/i
. . + .
!q!q
29~ ~00 page
"
'""
P,~+;III~
page 13
+':
.,
3t2
n. line
17
~ '++"~ .... " ~ " ~ " _
MI
436
n.line
O
Fig. 5. Structure of the complete text after segmentation by application of a majority decision rule. MI, MII
and MIII denote the period where modelsof the 1st, 2nd and 3rd kind are dominant. Undeterminedperiods are denoted U. The beginningof the text, whichis supposedto have been written the same day, is coveredby two models only. The apparent complexity of the description of the text by several kinds of models can be overcome without any major loss of information by a convenient statistical analysis, which is presented in the next section. 3. STATISTICAL ANALYSIS OF THE SUCCESSION OF An MODELS
We have to use three kinds of AR models that correspond to different temporal behaviours (i.e. to different domains of stationarity associated with different but constant parameters such as average values, variances, etc.). Any set of lines extracted from the text can be described by the frequency of occurrence of the three kinds of models, which therefore play the role of basis functions in a 3-dimensional space. Figure 6 presents a straightforward application: the question w a s to decide whether pages could be globally described and compared. Here each page is characterized by the histogram of the frequency of occurrence of the three models. A direct qualitative inspection shows that pages 12, 13 and 14 have the same profiles as the average (Av.) over the 17 pages; other pairs of pages are similar, for example pages 11 and 15, or pages 3 and 6. More interesting from the point of view of the dynamics of handwriting is the manner in which AR models succeed each other, and how they compete in fitting each sample. The next paragraph investigates
these points, taking into account the complete series. The subsequent paragraph will make use of this description for the classification of pages. 3.1 Choice matrix and transition matrix Figure 7 shows (top) a first assessment of the stability of each kind of model by comparing their
2
8 ~"-~
14 []--]_~
~_ q
3~_
n
BET_]__7 1 5 ~ _ ]
4~ 10[--~ 16r~_]_, [ I I I / 1 1 l I I I I 5 r - ~I l { 11[--~--1 171 I [ L_J_£..I I I I I I i f i ~ 12~._7 7~7
1 3 ~
FREQUENCY
~_~v.
vs
PAGES
Fig. 6. Representation of the pages by the frequency of occurrenceof the three kinds of models. Av. denotes the histogram computed on the complete text.
94
J. DUVERNOY
MODELSFREQUENCY %
49 -
.
AFTERSEGMENTATION second kind.
~]
18
1352 6
1
20[. I
II
SECONDMODEL II
According to Fig. 3, the corresponding temporal behaviour is rather irregular. This fact agrees with the result of the segmentation (top of Fig. 7) which frequent, shows that models of the second kind are more
I
33]
~I QNN~I Rf I
.
1%
|U
III
i I
II
Ill
! [1
TRANSITION --7
[[U~l ~ - - ~ - -I- - [ I
-----~l [11 l1
The transition matrix (right) describes what happens to the sample (n + 1) of the time series when a given kind of model is assigned to the sample (n). The structure of this matrix differs from that of the previous matrix: all kinds of models have increased their tendency to succeed to themselves. Nevertheless the
25 46 29 ~-~--] 160 9 31
I~,~ _ I ~ 1~_5_4_~32 14._1
til 34 6 6 0 i 11111 1-4--[--P]--~-'~ (ptn :O THEN:R)
~-~ L~144---J-!-8]
3.2 Class(fication of pages by principal components
(ptn:Q, ptn.l:R)
analysis
1,Ill
Fig. 7. Statistical description of the succession of models: frequency of occurrence, before and after segmentation of the text (top); choice matrix (left) which indicates what would be the second-best model for a given sample; transition matrix (right) that gives the probabilities for the sample n + 1 to be fitted by the three kinds of models, provided the model matches the sample n.
frequency of occurrence before and after the segmentat|on (by application of a majority rule, described at the end of the first section). It appears that models of the first kind are relatively unstable, their frequency being cut by a factor of two, to the benefit of undetermined samples (noted U). With an order p equal to 10, they represent an intermediate step between models of the second kind ( p = 20) and models of the third kind (p = 5). The reason why they are wiped out must be that they are not well grouped in homogeneous blocks, like the other two kinds of models. They occur in a transition period, or compete with the other models for some particular samples (as Akaike pointed out, (7) choosing an AR model with an order either too low or too high results in the same increase of prediction error: models of the first kind may occasionally overcome the others), The two matrices presented in Fig. 7 deal with the time series before segmentation. The choice matrix (left) describes how a model is assigned to a given sample of the series. Rows of the matrix are associated with the successive kinds of models (for example, the first row corresponds to the assignment of a model of the first kind to the current sample). Each row gives the probabilities of each kind of model to be the second best choice under the criterion of the magnitude of the quadratic prediction error. The first characteristic to be noted is that the diagonal terms of the matrix are less than the off-diagonal terms: if a model of a given kind is the best, the model ranking just after it is seldom of the same kind. Models of the second kind are the second best choice for the other kinds of models, Therefore, taking into account these results on the complete time series, it can be concluded that the general trend in the text is that depicted by models of
same c h conclusions hold. o i C ase those m adrawn t rfrom i the x
Using the three kinds of AR models, pages can be described from a static or from a dynamic point of view. The first approach consists of computing the frequency of occurrence of the models for each of the 17 pages one has to deal with, whatever the manner in which models succeed to each other. Then a page is processed as a vector whose three components are the frequencies of the three kinds of model. Obviously, these vectors have only two independent components, because the sum of the probabilities of the models is equal to unity. Then the collection of pages can be represented in a reduced space with only two dimensions ; but there is no reason for one of these two axes to correspond to the direction along which the data exhibit the highest variance. A principal component analysis is still necessary ; the resuit of this analysis is a rotationoftheaxessothatthefirstvectorofbasis(say, KL t) is parallel to the direction of highest variance. Let ~;, n 2, and ~3 be the frequency of occurrence of the three kinds of models, with, obviously ct3 = 1 - ~q - ~t2. (3) The collection of vectors (data vectors) associated with the pages can be represented in the (~q, ~q) plane. A new basis, (KL~, KLD, is defined by the two eigenvectOTSof the covariance matrix of the above data vectors. The principal component analysis consists here of projecting the data vectors on the (KLt, KL2) plane. The first new axis, KLt, carries most of the variance. Nevertheless, for sake of legibility of the figure, the components of the data vector on KLt and KL z have been normalized by dividing them by the amount of the variance the two basis vectors carry. Adynamic point ofview is introduced by computing for each page the transition matrix defined in Section 3.1. Because of the difficulty in handling matrices in classification procedures, each transition matrix was transformed into its diagonal form. Then each page was again characterized by only three components, here the diagonal terms of the diagonalized transition matrix. In the same way as previously, a principal component analysis was performed. But in this case the pages were described by three independent variables instead of two as in the previous case. Con-
Temporal structure of a handwritten text K[~
95
KL2 7
dividing them by the variances along the two axes. Figure 8 suggests two representations of the set of s pages. As a reference, the point denoted A stands for 7 17 2 A 15 the text taken as a whole (in this case the frequencies of 1614 models and the transition matrix are those given at the 9 10 13 top of Fig. 7). This reference can play the role of an 16 average, especially in the case of Fig. 8(a), where it is A 3 6 1711 1312 5 6 3 well centered in the cluster of points. This is no longer 4 the case for Fig. 8(b). The reason for this discrepancy is 8 12 2 10 14 that Fig. 8(a) takes into account the composition of the ]]: 9 page, which, more or less, resembles that of the frequency KL1 transition KL1 complete text; Fig. 8(a) deals with the dynamics in the feG~) page, which may vary from page to page according to the meaning of the text, and may differ for a particular Fig. 8. Principal component analysis of the 17 pages, taking page from the text as a whole. Further exegesis is into account (left) the frequency of occurrence of the models reserved for experts on the writer. or (right) their transition matrix. 5
4. CONCLUSIONS
sequently, the principal component analysis makes sense, and it turns out that 86 9o of the total variance in the collection of data is accounted for by the first two eigenvectors of the covariance matrix (again denoted KLI and KL2). The projections of the threecomponent vectors that represent the pages on the (KL1, KL2) plane are shown in Fig. 8(b). As for Fig. 8(a), components on KLI and KL2 are normalized by
Autoregressive models have proved useful in the prediction of the variations of a parameter of handwriting. More precise results are expected from the use - - at some more computational cost - - of multivariate models "°) instead of univariate ones as here. The first possibility is in predicting not only the average value of the slope of handwriting, but also its
AR MODEL ns
coef.s:
1 2
4-. . . . . .
5 I
~
L~-~.~.~.\\\\~\"~.. i e3
,,p
text .........
.~
;]
(~n X ei(~n.i i,1 ,- . . . . . . . . .
-~
I
i
I
i
! I
! i
,
I -L ....
'
'
Fig. 9. Optical system that performs the prediction of the slope of handwriting - - as determined at left - - by filtering the previous lines with a set of transparencies representing the values of the coefficientsof the AR model (right).
96
J. DUVERNOY
dispersion. Two other candidates will be examined : a sampling of the transfer function of the writer ~1~ and the coefficients of the space variant transform that generates a current hardwritten letter from its model/2~ An analog optical system can perform the filtering of the time series in real time, once the coefficients of the A R model are computed. The current line to be investigated is selected by a slit from the negative of the text (Fig. 9). When illuminated in coherent light, its Fourier spectrum is available in the back focal plane of a convergent lens. As in Fig. I, the sample of the time series corresponding to this line is determined by measuring the angle 0 (left). Coefficients of the model weight the previous lines: they can be represented by suitable optical transparencies. Therefore an estimate 0' of 0 is obtained by looking at the Fourier spectrum of the previous lines weighted by the optical filter. The prediction error is computed by subtracting the two successive measurements. The scheme applies without any problem in the case of an A R model with positive coefficients. When negative coefficients are involved, a multichannel optical processor is required. ~1" The A R models, namely the vectors whose components are the coefficients {a~}, can play the role of states of handwriting. Some noise is introduced when observing this state for a given sample. Moreover, the transition matrix is known. Then a Kalman filter could give an answer to the problems of estimating the true state of handwriting along the time series. SUMMARY The slope of handwriting in a series of successive lines is measured by using a coherent optical system. The measurements make up the time series to be investigated. In a first step an autoregressive (AR) model is fitted to the complete series. Several domains
(blocks) that cannot be predicted are detected by filtering the series by this A R model. In a second step, as many A R models as there are particular domains are computed and used to filter the time series.Temporal homogeneous periods are found by segmenting the text according to a majority decision rule, so that they are described by only one kind of model. Interpretation in terms o f s o - c a l l e d r h y t h m s o f writing can be proposed, associated to the behaviour of the models. Then a statistical analysis gives some descriptors of the temporal succession of AR models in the text (i.e. the choice matrix, for the prediction of one sample of the time series, and the transition matrix for the succession of models). Pages of the text are classified by a principal component analysis of these descriptors. As a conclusion, further applications of multivariate A R mddels are suggested, as well as the introduction of Kaiman filtering. Acknowledgement - - I am grateful to Professor H. Arsenault from Universit6 Laval for his help in editing this paper.
REFERENCES 1. J. Duvernoy, Applied Optics 15(6), 1584-1590 (1976). 2. J. Duvernoy and D. Charraut, "Stability and Stationarity of Cursive Handwriting", to appear in Pattern Recognition.
3. 4. 5. 6.
A. W. Lohmann, J. opt. Soc. Am. 55, 1007 (1965). J. Makhoul, Proc. IEEE 63(4), 561-580 (1975). P. Whittle, Biometrika 50(1), 129-134 (1963). B. S. Atal and S. L. Hanauer, J. acoust. Soc. Am. 50, 2 (1971). 7. H. Akaike, Ann. inst. Statist. Math. 21, 243-247 (1969). 8. Reference 4, p. 575. 9. J. Duvernoy and D. Charraut, Opt. Comms. 27(1), 27-32 (1978). 10. R.L. KashyapandA. R. Rao, Dynamic Stochastic Models from Empirical Data, Academic Press, New York (1976). 11. J. W. Goodman, Proc. IEEE 65, 29-38 (1977).
About the Author - J. DUVERNOYreceived his M.S. degree in Physics (1967), his M.A. degree in Philosophy
(1972), and his State Doctorate in Physics (1973), from the University of Besan~on (France). He received his M.S. degree in Psychology from the University of Paris Sorbonne (1971). From 1968 to 1971 he worked on optical processing of seismic data as a contractual researcher on industrial grants. Since 1971 he has been with the Centre National de la Recherche Scientifique (NRS). He proposed a modelling of handwriting by space variant imaging and developed a hybrid optical/digital processing method, including principal component analysis and linear prediction that has also been applied to aerial photographs and thermographies. His work on handwriting was awarded the Bronze Medal of the CNRS in 1978. He is' a member of the Optical Society of America and the Pattern Recognition Society.
0031-3203/80/0401-0089 $02.00/0
APPLICATION OF AUTOREGRESSIVE MODELS TO THE STUDY OF THE TEMPORAL STRUCTURE OF A HANDWRITTEN TEXT J. DUVERNOY
Information Systems Laboratory, Stanford University, Stanford, CA 94305, U.S.A. on leave from the Laboratoire de Physique G~6rale et Optique, Universit6 de Franche Corot6, 25030 Besan¢on Codex, France
(Received 14 May 1979; receivedfor publication 9 August 1979) Abstract - Variations of the slope of handwriting in several hundreds of lines are predicted by using autoregressive models. Most of the unpredictable lines correspond to. features such as the lack or excess of ink, the beginning of pages or paragraphs. The temporal succession of models is statistically analyzed in order to define descriptors, such as the transition matrix, that account for the building of a further state-space model for handwriting. Handwriting
Autoregressive models
Classification
INTRODUCTION:
TIME DEPENDENCE OF HANDWRITING Handwriting has been described as the analog of an imaging process. Questions of writer recognition and dating of manuscripts have been examined in the case of space invariant handwriting. ~1 Results of datings performed by a principal components analysis of writers' modulation transfer functions, extracted from pages covering a 35-year period, have shown that handwriting presents at least two kinds of temporal behaviour. A long term evolution clusters pages in successive periods (i.e. temporal classes). Short term variations - - within the extent of a page - - play the role of noise with respect to the former type of process, As a consequence it was possible to date a page by assignment to its most probable temporal class, but a finer dating inside the class was not possible. This uncertainty can be overcome by means of a more detailed study of the short-term processes. Their second order properties have been investigated ~2~and their temporal structure has been found to consist of successive stationarity domains isolated from each other by periods of non-stationarity. The size of these domains ranges from a few lines to a few pages. At this point the problem of characterizing a writer has no direct answer insofar as useful descriptors oftbese time sequences are not available. In fact, the questions of dating or identifying a sample of handwriting must be delayed as long as the underlying temporal process has not been investigated, The imaging system equivalent to the writer was first analyzed as a pure optical system. Its possible properties of space invariance were deduced from the study of the amount of distortion c31 that handwritten letters exhibit across a page. Parameters taken into
Stationarity
account were the coefficients of a space-variant transform that synthesizes a given handprinted letter from a model. When generating successive letters of a text, there is no deterministic relation between the successive sets of coefficients~21 of the successive spacevariant transforms. In some cases, they have a certain degree of stability (i.e. the ratio of their average value to their variance is high). Therefore the optical system equivalent to the writer is nearly time-invariant. This approximation holds for ancient writings (up to the 19th Century)but current modern handwriting turns out to be highly time-variant. Then pure optical parameters are no longer sufficient for describing the process, which has to be considered as the output of a time-variant and space-variant imaging system. In this paper we propose an approach to the modelling of short term processes ofbandwriting. The choice of the model one has to deal with results from a trade-off between the accuracyand tbecomputational cost of the description of the system involved. The input variables are not well-defined: several physical factors, such as the position of the hand, the fatigue of the writer, or the flow of ink, can contribute to the variations of handwriting. Other sources are difficult to assess, for instance the relationships between the meaning of what is being written and how it is written. Therefore we will assume that the system is driven by noise. One of the most powerful models currently in use is that where the output of a system is predictable from linear combinations o f past inputs and outputs. ~*~ In our case, it is specified as an autoregressive (AR) model. In this first approach we restricted ourselves to the prediction of a single parameter (viz. the average slope of handwriting with respect to the axis oftbe line. This slope is controlled by one of the parameters of the 89
PR 12 2--1~
Text segmentation
90
J. DUVERNOY
space-variant transform. (2~) A further study will deal with the prediction of sets of parameters ~5~(i.e. vectors) such as those mentioned above. Data were recorded fr°m several hundreds °flines written in a few days bY the same writer. The slope of handwriting was optically measured for each. The measurements are considered as a time series to be studied. This paper deals with the fitting of AR models to the considered time series, and with their use for depicting rhythms of writing. The first section of the paper recalls previous results on using a single AR model; the second section presents the linear prediction and filtering of the nonstationary time series by a collection of AR models, and the last section is devoted to a further statistical analysis of the succession of AR models along the text. As a conclusion, possibilities of performing a real time filtering of the text by analog representations of AR models in coherent light are suggested, as well as the definition of a state space model for a further Kalman filtering.
~ 0
~
// ~ ~ ~
X
~ ,z, L ..... [V "~ ~ ~ ' " ' ~
____, [
, n j
.
V
\ \ A
II I
\
/
U
1. USING A SINGLE AR MODEL
In this section we present successively the data to be processed and the corresponding time series, then the AR model that fits the whole series, and finally the filtering of the series by the model. This shows the existence of different domains of stationarity, which introduce the need for several AR models as described in the next section, 1.1 Data to be processed The laboratory of French literature of the Besanqon University provided us with the negatives of 17 pages from the notebook of the French writer, P. Claudel. They represent 454 lines written in a few days. The slope of handwriting, 0n, was measured for the 454 lines (n = 1..... 454) by using the optical system shown in Fig. 1. The negative of a page being illuminated with coherent light, the line No. n, is selected by a slit. The Fourier spectrum of the line is displayed in the back focal plane of the lens. It consists of two elongated distributions of light with an angular separation 0n. The vertical distribution is the Fourier transform of the slit along its principal axis. It plays the role of an axis of reference. The other distribution is the superposition of the Fourier spectra of the individual pupils supported by the line, i.e. the elements of the letters, which present an average slope, 0~, with respect to the axis ofthe line. Infactthisdistributionconsistsofafan of elementary spectra associated with each pupil, Therefore its angular width depends on the spread of the slope around the mean value 0,. An angular sampling of this fan will give a multivariate description of the regularity of handwriting in a further study, The Fourier spectrum of each line was scanned by a slit rotating around the focus of the lens. The variations of the luminous intensity, measured by a photodiode, were recorded in order to determine the angular separation of the two maxima of energy, the
//
[
Fig.1. The negative of a text is placed in the front focal plane (X, Y) of the convergent lens L. When illuminated with coherent light, its Fourier spectrum is displayed in the back focal plane. A line being selectedby a narrow slit parallel to X, the spectrum shows two axes of light, their angle 0 is that of handwriting with respect to X.
angle 0n. Such a measurement yields the average slope of handwriting, provided the number of letters per line is high enough (in order to avoid spurious maxima due to the spectrum of a sparse collection of pupils). A collection {On}, n = 1.... ,454 is then available, and represents a time series to be investigated. Because the temporal interval between any two lines is certainly not constant, this time series can be expected to be split into several different sets. 1.2 Fittin 0 an AR model The system that determines the time variations of the slope of handwriting is assumed to be linear, its output, 0F, is supposed to be predicted by a linear combination, 0'~, or past outputs~*~: p 0', = ~ aiOn-i. (1) i=' The set of coefficients {a~} and the order p are to be determined. Practically the order p can be defined by looking for the value that minimizes the squared prediction error,~10; - 0 n [2. According to Akaike ~8~ one can begin with a maximum value Pm~ = 3 N ~/2, where N denotes the number of data, i.e. the length of the time series (here N = 454 and Pm~xis close to 60). In
Temporal structure of a handwritten text our case, handwriting on a page did not seem to depend strongly on handwriting two pages before, i.e. approximately 40 lines. Therefore we chose p _ . = 40. Eventually the optimal order was p --- 5. In any case coefficients {a+} are computed by the A t a l - H a n a u e r method, (6) which consists of expressing the fact that the average squared prediction error is m i n i m u m for the {a+} ; in other words, its partial derivatives with respect to the {a+} are simultaneously equal to zero. Reference 9 presents the details of the model we found,
91
threshold is low. Particular AR models have to be fitted to each of these blocks. Therefore the time series one has to deal with appears to be a locally stationary process where, in terms of handwriting, particular "rhythms" can be detected. A first explanation has been proposed for the 120 unpredictable lines : more than 60 % are distinguished by features such as the lack (or excess) of ink, the presence of crossings out, or by the structure of the text itself (top of pages, beginning of new paragraphs or of n o n - c o n t e m p o r a r y units in a page). Figure 2 shows some of these characteristics, associated with unpredictable lines. But, as the time series was obviously nonstationary, we could not restrict ourselves to this single A R model and propose premature explanations to handwriting features. A thorough coverage of the time series by as many models as necessary was needed before any such attempt.
1.3 Filtering the time series An A R model allows the prediction with a m i n i m u m average error of successive samples of a time process• Besides being efficient by the small n u m b e r of coefficients it requires, the A R model is also useful in locating samples that do not have the average statistical properties of the time series the AR model is fitted to. Filtering the time series consists of rejecting samples O. which differ from the predicted value 0', by an a m o u n t greater than a given fraction e, of the actual value 0, % 10" _ 0,1 -- = - > +:. (2) 0, 0,
z. usiNt; SEVERAL Ar MODELS
With a threshold e. equal to 10% (e, = 0.1) half of the 454 lines c a n n o t be predicted• With +: = 20 % only 120 lines remain unpredictable• This fact shows that one deals only with an approximate stationarity when processing the series as a whole. More interesting is the segmentation of the series into homogeneous domains (or blocks) which c a n n o t be predicted when the
When filtering the time series by the previous model, it turned out that some blocks needed particular A R models. In fact, three different kinds of models were necessary to cover the complete time series with a prediction error less than 5 %. Figure 3 shows the filtering of three different blocks by the AR models which were defined on them. Models of the first kind are of order 10, models of the second kind deal with a past of 20 lines, and models of the third kind are of order 5 (one of them is the previous model defined in section 1.2.). Two models ofeach kind were necessary
@
L.,,L.
•
L ,..~.....> .....
......... /
/ ]
I
/
,
~
,,~.,"
+.Z.
'
,
"~ . . . . . . , . . . , ~ ............ "
+ •. . . . . . . . . . . . . . .
•
l,
.
'.~.'.
q
r'.,.,.l ..... ........
........... .
.... t- .+ +4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . e,,.,.A.,.. ..... , ,',1. - J+,P+,. , . , t .
i
'~
~..p
;..+
+-
7.-,~/'...,..~,~,~....../. ....;,.....~..; r'~l
+
I V ....
', i ,'
'
o
40+
P" . . . . . . .
I , * I
.
I
.'
[ *.,
;'1 /
t i X I~
T
.~/~
~ ....
q,
,
- ....
I ........
.+ . . . .
:.,- . . . .
' -+.
' ..
++ .+' . . . . . . . . . •
6+-."
+ J(", +''+'-'", ,-?'~ ......... ~ +,
/~
t.
/
.".<(.
-
]
"
/7 ....
,,,t ,'-', .,'. z ,.t ,, ,.,
"
/+~.:-. . . . . , " ' " ......... '+ ~"~ ". . . . ' " +'
"p
r
,
=.
. ....
+
131
-<
I
.,,.,,
:. . . .
"' ,"'" . . . . . . . .
.~.-<',,,.
I
I ........
b."" ","': ":~ .~,o,+, t~ . + . , , . . .
]
,7,J',",
......
z,./ ";
....
, ,:
.."
+.++.+
/ . , ¢, / , ..... . . . . . . .
,r;,"
"
",'.
....
l I
151
lime
Fig. 2. Filtering of a page by an AR model. Samples (i.e. lines), their predicted slope differing from their real slope by more than ~%, are rejected. The same page is filtered with two different thresholds of rejection. Features that turn out to be unpredictable from the slope of past lines are (arrows): the beginning of the page, an excess of ink and the beginning of a non-contemporary unit of text.
J. DUWRNOy
92 17!10 -]
to cover the time series. Figure 4 presents the filtering of two sections of the time series by using the collection of models, The threshold of rejection is fixed at r, = 5 %. Each sample ofthe series is predicted by the model that gives the least error, eventually with an accuracy better than 5 %. Only a few lines remain unpredictable. They are explained by the same reasons as those previously described. Now, the trouble is that the succession of samples is described by an equivalent succession of models that predict them well, but there is no homogeneity in this equivalent series. A better view of the process is obtained by applying a simple decision rule which yields a segmentation in homogeneous domains. Each sample is assigned the model which is dominant in a given fraction of its past (a past of 20 samples was chosen, where the most frequent model was selected). This rule leads to the segmentation depicted in Fig. 5 : periods denoted MI, MII and MIII are those where the first, second and third models are dominant, respectively. Periods denoted U are not determined, i.e. at least two kinds of models have the same frequency of occurrence. This first debugging proposes an approach to the overall time structure of the text. This structure does not fit the separation in successive pages, but does agree with the a priori information on the genesis of the text: the first seven pages are likely to have been written in one block, possibly the same day, as the following pages are chopped into smaller paragraphs written in successive days. Moreover each kind of model corresponds (Fig. 4) to a different and characteristic temporal behaviour ; for example, type II is very irregular, exhibiting many peaks, as type III is smooth. Therefore possibilities of further literary interpretations are opened to the specialists of Claudel manuscripts.
/~
"
.
Sl'
T i
J~ II ii ''
, '
line
74'. ~
|~.
T
' ,
line
~
I
- Y['~I~\VAVY"I
~^ V
SO', T
"
line
Fig. 3. Examples of the three kinds of AR models needed to cover the complete text. Their order is, respectively, 10,20and 5 for models I, II and III. The domains where these models were defined and their filtering by their own models (with a rejection threshold of 5 %) are shown here.
O
,,,
51"
~'J
T,
' iF
,I' I I I I
i
- -
I I
77"16~56! 0~~~ :~~i]~~~ £.5% 9'9line 21s
i
251line
Fig. 4. Filteringof two pieces of text by using the three kinds of AR models. The rejection threshold is 5 ~. For each sample, one chooses the model that predicts its value with the lowest error.
Temporal structure of a handwritten text
93
page +
page 6
. . . . "..... , .... :~:'.,;-~:';~:'T,~.':.:.~;'-:.:: [ ~: ': + ;:+ #+, ,:: ~+:+~+,=-/++£~++%-/:++~.-.~, :: ::., ;.:; /
[
~.~",.++'++,~m+ "" "
+ ,-,-. " ' ~
+.,,
:+;'++'" . . . . ++. .~' . ":
"-~
~,
+,+ + +'-+
, " : '::,,:: t:~;-" "~,':';. ,':.' " " ' ,:::, t: ++::+:+,m,. %!: ....
:'"~+~'+", . . . . ."'+;'. . . . . . . . .. ......
M III
,,.:,.
+." a . , + ' , , ~ + .
"
=.:~++:.~:,-': . . .+ . . '
m
:
:.
MN
Mine
page 7
page 12 ;:
,: : :.:-'.+,~::+.,:
_
_U
162
":! ~ ! . i : "
~ , "'
. , " ~,,'+ ,,+.',.~+~+. . . . .
'
:,,+ ,+',m+,.,:+,..,'
:
=
172
'
M I
.
i~;:"
,..++-~:,r:,
.
218
~!:!!:;"
.
O
"~ ~'" ~ .J'.,N =
," +- "- : ~! + ;'":
'~;.+~-
,+
'
_l,l.
ui_
327 336
....
lu
245
Mill
363 370 M I I
'i~
~ _
391
u
_
406
,:.,~,+.+;.
. MIll
/i
. . + .
!q!q
29~ ~00 page
"
'""
P,~+;III~
page 13
+':
.,
3t2
n. line
17
~ '++"~ .... " ~ " ~ " _
MI
436
n.line
O
Fig. 5. Structure of the complete text after segmentation by application of a majority decision rule. MI, MII
and MIII denote the period where modelsof the 1st, 2nd and 3rd kind are dominant. Undeterminedperiods are denoted U. The beginningof the text, whichis supposedto have been written the same day, is coveredby two models only. The apparent complexity of the description of the text by several kinds of models can be overcome without any major loss of information by a convenient statistical analysis, which is presented in the next section. 3. STATISTICAL ANALYSIS OF THE SUCCESSION OF An MODELS
We have to use three kinds of AR models that correspond to different temporal behaviours (i.e. to different domains of stationarity associated with different but constant parameters such as average values, variances, etc.). Any set of lines extracted from the text can be described by the frequency of occurrence of the three kinds of models, which therefore play the role of basis functions in a 3-dimensional space. Figure 6 presents a straightforward application: the question w a s to decide whether pages could be globally described and compared. Here each page is characterized by the histogram of the frequency of occurrence of the three models. A direct qualitative inspection shows that pages 12, 13 and 14 have the same profiles as the average (Av.) over the 17 pages; other pairs of pages are similar, for example pages 11 and 15, or pages 3 and 6. More interesting from the point of view of the dynamics of handwriting is the manner in which AR models succeed each other, and how they compete in fitting each sample. The next paragraph investigates
these points, taking into account the complete series. The subsequent paragraph will make use of this description for the classification of pages. 3.1 Choice matrix and transition matrix Figure 7 shows (top) a first assessment of the stability of each kind of model by comparing their
2
8 ~"-~
14 []--]_~
~_ q
3~_
n
BET_]__7 1 5 ~ _ ]
4~ 10[--~ 16r~_]_, [ I I I / 1 1 l I I I I 5 r - ~I l { 11[--~--1 171 I [ L_J_£..I I I I I I i f i ~ 12~._7 7~7
1 3 ~
FREQUENCY
~_~v.
vs
PAGES
Fig. 6. Representation of the pages by the frequency of occurrenceof the three kinds of models. Av. denotes the histogram computed on the complete text.
94
J. DUVERNOY
MODELSFREQUENCY %
49 -
.
AFTERSEGMENTATION second kind.
~]
18
1352 6
1
20[. I
II
SECONDMODEL II
According to Fig. 3, the corresponding temporal behaviour is rather irregular. This fact agrees with the result of the segmentation (top of Fig. 7) which frequent, shows that models of the second kind are more
I
33]
~I QNN~I Rf I
.
1%
|U
III
i I
II
Ill
! [1
TRANSITION --7
[[U~l ~ - - ~ - -I- - [ I
-----~l [11 l1
The transition matrix (right) describes what happens to the sample (n + 1) of the time series when a given kind of model is assigned to the sample (n). The structure of this matrix differs from that of the previous matrix: all kinds of models have increased their tendency to succeed to themselves. Nevertheless the
25 46 29 ~-~--] 160 9 31
I~,~ _ I ~ 1~_5_4_~32 14._1
til 34 6 6 0 i 11111 1-4--[--P]--~-'~ (ptn :O THEN:R)
~-~ L~144---J-!-8]
3.2 Class(fication of pages by principal components
(ptn:Q, ptn.l:R)
analysis
1,Ill
Fig. 7. Statistical description of the succession of models: frequency of occurrence, before and after segmentation of the text (top); choice matrix (left) which indicates what would be the second-best model for a given sample; transition matrix (right) that gives the probabilities for the sample n + 1 to be fitted by the three kinds of models, provided the model matches the sample n.
frequency of occurrence before and after the segmentat|on (by application of a majority rule, described at the end of the first section). It appears that models of the first kind are relatively unstable, their frequency being cut by a factor of two, to the benefit of undetermined samples (noted U). With an order p equal to 10, they represent an intermediate step between models of the second kind ( p = 20) and models of the third kind (p = 5). The reason why they are wiped out must be that they are not well grouped in homogeneous blocks, like the other two kinds of models. They occur in a transition period, or compete with the other models for some particular samples (as Akaike pointed out, (7) choosing an AR model with an order either too low or too high results in the same increase of prediction error: models of the first kind may occasionally overcome the others), The two matrices presented in Fig. 7 deal with the time series before segmentation. The choice matrix (left) describes how a model is assigned to a given sample of the series. Rows of the matrix are associated with the successive kinds of models (for example, the first row corresponds to the assignment of a model of the first kind to the current sample). Each row gives the probabilities of each kind of model to be the second best choice under the criterion of the magnitude of the quadratic prediction error. The first characteristic to be noted is that the diagonal terms of the matrix are less than the off-diagonal terms: if a model of a given kind is the best, the model ranking just after it is seldom of the same kind. Models of the second kind are the second best choice for the other kinds of models, Therefore, taking into account these results on the complete time series, it can be concluded that the general trend in the text is that depicted by models of
same c h conclusions hold. o i C ase those m adrawn t rfrom i the x
Using the three kinds of AR models, pages can be described from a static or from a dynamic point of view. The first approach consists of computing the frequency of occurrence of the models for each of the 17 pages one has to deal with, whatever the manner in which models succeed to each other. Then a page is processed as a vector whose three components are the frequencies of the three kinds of model. Obviously, these vectors have only two independent components, because the sum of the probabilities of the models is equal to unity. Then the collection of pages can be represented in a reduced space with only two dimensions ; but there is no reason for one of these two axes to correspond to the direction along which the data exhibit the highest variance. A principal component analysis is still necessary ; the resuit of this analysis is a rotationoftheaxessothatthefirstvectorofbasis(say, KL t) is parallel to the direction of highest variance. Let ~;, n 2, and ~3 be the frequency of occurrence of the three kinds of models, with, obviously ct3 = 1 - ~q - ~t2. (3) The collection of vectors (data vectors) associated with the pages can be represented in the (~q, ~q) plane. A new basis, (KL~, KLD, is defined by the two eigenvectOTSof the covariance matrix of the above data vectors. The principal component analysis consists here of projecting the data vectors on the (KLt, KL2) plane. The first new axis, KLt, carries most of the variance. Nevertheless, for sake of legibility of the figure, the components of the data vector on KLt and KL z have been normalized by dividing them by the amount of the variance the two basis vectors carry. Adynamic point ofview is introduced by computing for each page the transition matrix defined in Section 3.1. Because of the difficulty in handling matrices in classification procedures, each transition matrix was transformed into its diagonal form. Then each page was again characterized by only three components, here the diagonal terms of the diagonalized transition matrix. In the same way as previously, a principal component analysis was performed. But in this case the pages were described by three independent variables instead of two as in the previous case. Con-
Temporal structure of a handwritten text K[~
95
KL2 7
dividing them by the variances along the two axes. Figure 8 suggests two representations of the set of s pages. As a reference, the point denoted A stands for 7 17 2 A 15 the text taken as a whole (in this case the frequencies of 1614 models and the transition matrix are those given at the 9 10 13 top of Fig. 7). This reference can play the role of an 16 average, especially in the case of Fig. 8(a), where it is A 3 6 1711 1312 5 6 3 well centered in the cluster of points. This is no longer 4 the case for Fig. 8(b). The reason for this discrepancy is 8 12 2 10 14 that Fig. 8(a) takes into account the composition of the ]]: 9 page, which, more or less, resembles that of the frequency KL1 transition KL1 complete text; Fig. 8(a) deals with the dynamics in the feG~) page, which may vary from page to page according to the meaning of the text, and may differ for a particular Fig. 8. Principal component analysis of the 17 pages, taking page from the text as a whole. Further exegesis is into account (left) the frequency of occurrence of the models reserved for experts on the writer. or (right) their transition matrix. 5
4. CONCLUSIONS
sequently, the principal component analysis makes sense, and it turns out that 86 9o of the total variance in the collection of data is accounted for by the first two eigenvectors of the covariance matrix (again denoted KLI and KL2). The projections of the threecomponent vectors that represent the pages on the (KL1, KL2) plane are shown in Fig. 8(b). As for Fig. 8(a), components on KLI and KL2 are normalized by
Autoregressive models have proved useful in the prediction of the variations of a parameter of handwriting. More precise results are expected from the use - - at some more computational cost - - of multivariate models "°) instead of univariate ones as here. The first possibility is in predicting not only the average value of the slope of handwriting, but also its
AR MODEL ns
coef.s:
1 2
4-. . . . . .
5 I
~
L~-~.~.~.\\\\~\"~.. i e3
,,p
text .........
.~
;]
(~n X ei(~n.i i,1 ,- . . . . . . . . .
-~
I
i
I
i
! I
! i
,
I -L ....
'
'
Fig. 9. Optical system that performs the prediction of the slope of handwriting - - as determined at left - - by filtering the previous lines with a set of transparencies representing the values of the coefficientsof the AR model (right).
96
J. DUVERNOY
dispersion. Two other candidates will be examined : a sampling of the transfer function of the writer ~1~ and the coefficients of the space variant transform that generates a current hardwritten letter from its model/2~ An analog optical system can perform the filtering of the time series in real time, once the coefficients of the A R model are computed. The current line to be investigated is selected by a slit from the negative of the text (Fig. 9). When illuminated in coherent light, its Fourier spectrum is available in the back focal plane of a convergent lens. As in Fig. I, the sample of the time series corresponding to this line is determined by measuring the angle 0 (left). Coefficients of the model weight the previous lines: they can be represented by suitable optical transparencies. Therefore an estimate 0' of 0 is obtained by looking at the Fourier spectrum of the previous lines weighted by the optical filter. The prediction error is computed by subtracting the two successive measurements. The scheme applies without any problem in the case of an A R model with positive coefficients. When negative coefficients are involved, a multichannel optical processor is required. ~1" The A R models, namely the vectors whose components are the coefficients {a~}, can play the role of states of handwriting. Some noise is introduced when observing this state for a given sample. Moreover, the transition matrix is known. Then a Kalman filter could give an answer to the problems of estimating the true state of handwriting along the time series. SUMMARY The slope of handwriting in a series of successive lines is measured by using a coherent optical system. The measurements make up the time series to be investigated. In a first step an autoregressive (AR) model is fitted to the complete series. Several domains
(blocks) that cannot be predicted are detected by filtering the series by this A R model. In a second step, as many A R models as there are particular domains are computed and used to filter the time series.Temporal homogeneous periods are found by segmenting the text according to a majority decision rule, so that they are described by only one kind of model. Interpretation in terms o f s o - c a l l e d r h y t h m s o f writing can be proposed, associated to the behaviour of the models. Then a statistical analysis gives some descriptors of the temporal succession of AR models in the text (i.e. the choice matrix, for the prediction of one sample of the time series, and the transition matrix for the succession of models). Pages of the text are classified by a principal component analysis of these descriptors. As a conclusion, further applications of multivariate A R mddels are suggested, as well as the introduction of Kaiman filtering. Acknowledgement - - I am grateful to Professor H. Arsenault from Universit6 Laval for his help in editing this paper.
REFERENCES 1. J. Duvernoy, Applied Optics 15(6), 1584-1590 (1976). 2. J. Duvernoy and D. Charraut, "Stability and Stationarity of Cursive Handwriting", to appear in Pattern Recognition.
3. 4. 5. 6.
A. W. Lohmann, J. opt. Soc. Am. 55, 1007 (1965). J. Makhoul, Proc. IEEE 63(4), 561-580 (1975). P. Whittle, Biometrika 50(1), 129-134 (1963). B. S. Atal and S. L. Hanauer, J. acoust. Soc. Am. 50, 2 (1971). 7. H. Akaike, Ann. inst. Statist. Math. 21, 243-247 (1969). 8. Reference 4, p. 575. 9. J. Duvernoy and D. Charraut, Opt. Comms. 27(1), 27-32 (1978). 10. R.L. KashyapandA. R. Rao, Dynamic Stochastic Models from Empirical Data, Academic Press, New York (1976). 11. J. W. Goodman, Proc. IEEE 65, 29-38 (1977).
About the Author - J. DUVERNOYreceived his M.S. degree in Physics (1967), his M.A. degree in Philosophy
(1972), and his State Doctorate in Physics (1973), from the University of Besan~on (France). He received his M.S. degree in Psychology from the University of Paris Sorbonne (1971). From 1968 to 1971 he worked on optical processing of seismic data as a contractual researcher on industrial grants. Since 1971 he has been with the Centre National de la Recherche Scientifique (NRS). He proposed a modelling of handwriting by space variant imaging and developed a hybrid optical/digital processing method, including principal component analysis and linear prediction that has also been applied to aerial photographs and thermographies. His work on handwriting was awarded the Bronze Medal of the CNRS in 1978. He is' a member of the Optical Society of America and the Pattern Recognition Society.