- Email: [email protected]
On-line handwritten segmentation in linear drawings
Pattern Recognition LeRers ELSEVIER
Pattern Recognition Letters 18 (1997) 193-202
On-line handwritten segmentation in linear drawings D. de Brucq a, M. Amara a,*, V. Ruiz b a PSI-La3i, UFR des Sciences, Universit[ de Rouen, 76821 Mont-Saint-Aignan Cedex, France b GREYC, CNRS URA 1526, ISMRA, 6 Bd Mar£chal Juin, 14050 Caen Cedex, France
Received 1 November 1995; revised 22 October 1996
Abstract This paper addresses the problem of tracking line segments corresponding to on-line handwritten obtained through a digitizer tablet. The approach is based on Kalman filtering to model linear portions of on-line handwritten, particularly, handwritten numerals, and to detect abrupt changes in handwritten direction underlying a model change. This approach uses a Kalman filter framework constrained by a normalized line equation, where quadratic terms are linearized through a first-order Taylor expansion. The modeling is then carried out under the assumption that the state is deterministic and time-invariant, while the detection relies on double thresholding mechanism which tests for a violation of this assumption. The first threshold is based on an approach of layout kinetics. The second one takes into account the jump in angle between the past observed direction of layout and its current direction. The method proposed enables real-time processing. To illustrate the methodology proposed, some results obtained from handwritten numerals are presented. © 1997 Elsevier Science B.V. Keywords: Extended Kalman filtering; Identification; Primitive; Model changes; Abrupt changes of model; Handwritten numerals; On-line
processing 1. Introduction Tracking a primitive (point, segment, curve, etc.) in the course of the time through a handwritten form is a fundamental problem in computer vision. Indeed, these lists of primitives matched are symbolic representations that are going to allow considering the handwritten segmentation in view of the recognition of the handwriting as well as the construction of traced handwriting. Our study deals with on-line modeling of handwritten through three types of geometric primitives: line segment, arc of circle and arc of ellipse. Due to the variability o f handwriting,
* Correponding author. E-mail: [email protected].
these primitives have to be robust to noise and describe layouts as precisely as possible. Today, we have encouraged numerical results for each of these primitives. The aim of this paper is to present a new approach to line segmentation by an extended Kalman filter. Firstly, the approach has to estimate the parameters of a line as precisely as possible and secondly it has to extract segments from handwritten layouts which are the constituents of complex shapes in an automatic way. W e present an on-line process to express a symbolic representation o f handwritten layouts through a succession of linear segments. W e have restricted this paper to linear segmentation because particulars of the extended Kalman filter equations for line, circle and ellipse are quite different.
0167-8655/97/$17.00 Copyright © 1997 Elsevier Science B.V. All rights reserved. PII S0167-8655(96)00126-2
194
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
Many different techniques have been implemented to extract line segments on outlines. Polygonal approximation on images is still the method commonly used in handwriten recognition systems. Most of the works try to minimize the distance between a part of the contour and a line segment (e.g. Ramer, 1972, or Pavlidis and Horowitz, 1974). The Hough Transform has been introduced to detect complex shapes of points in binary images. The interest of this technique is to transfer the problem of parameter extraction from the image space to a problem of local spike extraction from the representative space of parameters as presented in (Ostentad, 1988). The precision of the detection depends on the choice of discrete partition of the parameter space. A complete processing of the Hough Transform can be found in (Maitre, 1985). Other approximation techniques have been experienced such as discrete chord calculation (e.g. Martin, 1992). Some fuzzy technique-based methods arise from these works to alleviate the connection problem as in (Dave, 1989). With the fuzzy techniques, a point cloud can be approached through a line by considering points assigned to a given class with a belonging coefficient. In particular, (Deleu, 1991) presents a very detailed description of various existing methods for segment extraction from an outline in which analytical, morphological or connection methods can be found. These methods are limited as compared to our approach. They do not allow an on-line estimation of parameters from data. Thus, we can anticipate an on-line segmentation. Our objective is to estimate line segment parameters in a recursive manner that enables on-line data processing from the set of points constituting handwritten layouts and on-line segmentation of complex forms. The recursive least squares methods use line segment models that are commonly described through Cartesian equations. This approach suffers from a major drawback. It prevents from finding a reliable solution for segments with an infinite direction coefficient which can yield numerical problems in terms of computation. Many researchers have used the Kalman filter in computer vision for complex forms, notably GiaiCheca et al. (1993) and Ayache and Faugeras (1986). The last two authors describe silhouettes of objects
with a polygonal approximation of their outlines. The technique of recognition uses the Kalman filter to match a segment of the model with a segment of the scene observed under hypotheses as well as the position of models in the scene. Poulain d'Andecy et al. (1994) propose a recursive approach for line segment detection in score or stave images based on the Kalman filtering. But, the model proposed is tight with the time-invariant speed assumption which prevents the processing of some particular types of line segments. In our approach, the recursive form arises from the layout representation through particular state and measure (observation) equations that enable the use of extended Kalman filtering techniques. The measure equations are constituted of two constraints. A first constraint is the normal equation of line, ax + b y - c = 0, where x and y are time-varying point coordinates. A second constraint is the condition, a 2 + b 2 = 1, over line parameters in order to ensure a unique representation. We are here considering that the values 0 and 1 are relative to the first and second constraint and constitute the (time-invariant) observations of the state modeling. Linear approximation implies time-invariant parameters a and b. The state equations express this property. The method proposed is extended to detect model changes (in layout direction). The notion of abrupt changes in the model has been introduced in (Wald, 1974) and this notion has particularly been paid attention to in the last twenty years. A detailed state of art concerning the different types of detectors applied to slowly-varying signals can be found in (Traitement du Signal, 1992). To free oneself from the non-stationary problem, methods of detection based on a distance test between models have been introduced in (Appel and Brand, 1983) and were then extended in (Basseville and Benveniste, 1983). Finally, Nikivorov (1983) solved the problem of change detection in signal spectral characteristics with a local approach. Our application has to detect and localize changes of direction in the characteristics of layouts. According to (Sauter, 1991) the statistical tests conceived to detect this kind of abrupt changes cannot always be applied and a new approach should therefore be proposed. We are thus introducing a double threshold detec-
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
tor. The first threshold results from an approach related to handwriting kinetics and the second threshold takes into account the angle variation between the layout direction resulting from the extended Kalman filter and its current measure. We will first present the line model. We will then define the Kalman filter equations after a linearizing stage. Covariance matrices of errors and extended Kalman filter-initial conditions will then be explained. Finally, we will present the results obtained from the application to digitized numerals once the theoretical detector approach has been implemented.
195
Y
-~3.'~Y__~)x,y)
(D) Fig. 1. Normal line equation.
The hypothesis of the non-variation of the single line leads to the following system of state equations, holding for all k:
2. State-space model
a(k+ll=a(k) The parametrization of a line by ( y = ax + b) is not adequate because the (a,b) space is inhomogeneous (Duda and Hart, 1972). Observations x and y are taken to act in a symmetric way, thus, the line equation in its form can be defined as follows:
a x + f l y - y=O,
(I)
where a , 13 and 3' are real-valued constants. With variation in time, the model taken to describe various observations
x(1) y(1)
...
x(K)]
.-"
y(K)]
k) +by(k) - c = 0 ,
(2)
where c represents ~ e algebraic distance of the vector ~ = ( ~ , n ) n . Here, we define the normal vector, ~, relative to the line (D) (see Fig. 1), by
/3 sin0e ~7+/32
(3) where X(k + l) represents the state equation characterizing the X(k)-state constant evolution, and X tr= [a,b,c]. We thus assume a time-invariant line slope. Globally, the line is invariant over the point cloud under consideration. We are searching for a single line verifying the unicity of the parameters (normal vector) due to the constraint a2(k) + b2(k) : 1.
is the canonical representation of the straight line defined as follows:
g=
b(k + l ) = b ( k ) thatmeans X(k + l ) = X ( k ) , c(k+ 1) = c(k)
-
and the direction vector d of the line (D) is ( - b a) r. With this equation, c and 0 are always finite.
(4)
Eqs. (2) and (4) define a system of observation where x(k) and y(k), k = 1,2 . . . . . K, are the points issued from the digitizer tablet. The line (D) is covered in the points acquisition direction. Eqs. (2) and (4) define two constraints on the line. The observation is defined by these constraints, in an original way. The novelty here consists in defining the measure equations through the two constraints instead of using only the measures, x(k) and y(k), from the digitizer tablet. Indeed, these are the values 0 and 1 from Eqs. (2) and (4) that are being observed. Due to the accuracy of numerical computations and measurements of x(k) and y(k), we will have to define for each constraint constituting the measure equations (2) and (4), a measurement noise denoted Vd and V,., respectively. However, the second constraint in Eq. (4) has to be linearized with respect to
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
196
the state parameters a and b, for the purpose of using the extended Kalman technique. Eq. (4) is linearized by a first-order Taylor expansion around the estimated values a ( k / k - 1 ) and 12(k/k-1) taking into account the known observations towards ( k - 1). Let us assume:
a(k) ~=~ ( k / k - 1) + 6 a ( k - 1),
b(k) ~ ~,(k/k-l) + 8 b(k- 1).
(5)
The linearization can be accomplished by substituting Eq. (5) into Eq. (4), and we can show that this is expressed by
r~(k) ~ 1 + ~ 2 ( k / k - 1) + b 2 ( k / k - 1) :
2~(k/k-1)a(k) + ~b(k/k-l)b(k) + Vc, (6)
measurement Y(k) and is identified by Eqs. (6) and (7). It is written as follows: zx I x ( k )
h,(k)=L2~(k/k-l)
y(k)
-1]
2~,(k/k-1)
0
For this linearized model we used the classical Kalman filter equations (Kalman and Bucy, 1961; Labarrere et al., 1978; Jacobs, 1993). We define ~'(k/k- 1)~ R 3 to be our a priori state estimate at step k given knowledge of the process prior to step k, and X(k/k) ~ E3 to be our a posteriori state estimate at step k given measurement Y(k). We can define a priori and a posteriori estimation errors as
2?(k/k- l) =x(k) -,¢(k/k- 1)
where V, is the measurement noise on the constraint including numerical errors due to the canceling of the quadratic differences. Eq. (2) can be written in the form,
The a priori estimation error covariance is then
Ya( k) & a(k) x(k) + b(k) y(k) - c(k) + V,,,
P;(k/k-,) = E[ )(( k / k - 1))~( k / k - 1) tr]
(7) where Vd is the measurement noise on the line including numerical errors due to uncertainties in the position of x(k) and y(k). Eqs. (6) and (7) constitute the system of observations.
3. Kaiman filtering equations We are introducing below the Kalman filter equations. The Kalman filter addresses the general problem of trying to estimate the state X ~ I ~ 3 of a first-order, discrete-time controlled process that is governed by the linear difference Eq. (4), and with a measurement Y ~ ~2 that is r'(k) = H ( k ) X ( k )
+ V(k),
(8)
where
r(k~ [ r~'(k~ ] vc(k~) is the measurement vector (observation). The random variable V(k) represents the measurement noise with zero mean and covariance R, and the matrix H(k) in the measurement equation (8) relates the state to the
" (9)
and
.,~( k/k ) = X(k) - .~( k/k ).
and the a posteriori estimation error covariance is
Px~k/k~=e[~(k/klY(k/k-l)'~].
(101
In deriving the equations for the Kalman filter, we begin with the goal of finding an equation that computes an a posteriori state estimate X(k/k) and a weighted difference between an actual measurement Y(k) and a measurement prediction H(k))((k/k- 1) as follows:
2?(k/k) = 2?( 1,/k - l) + x(k~[r(k)-I-l(k~?(~/~-1~]. (11) The difference [Y(k) - H ( k ) X ( k / k - 1)] in Eq. (11) is called the measurement innovation, or the residual. The residual reflects the discrepancy between the predicted measurement H ( k ) X ( k / k - 1) and the actual measurement Y(k). A residual of zero means that both are in complete agreement. The matrix K in Eq. (11) is chosen as being the gain or blending factor that minimizes the a posteriori error covariance of Eq. (10). The minimization can be accomplished by first substituting Eq. (11) into the above definition for ,~(k/k), substituting
197
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
that into Eq. (10), performing the indicated expectations, taking the derivative of the trace of the result with respect to K, setting that result equal to zero, and then solving for K. For more details see (De Brucq and Folliot, 1988; Najim, 1988; Brown and Hwang, 1992; Jacobs, 1993). One form of the resulting K that minimizes Eq. (10) is given by K(k) =
P.f,,~/k_,,H(k)tr(H(k)P~(wk_,)H(k) tr + R ) - ' . (12) Finally, the updated covariance matrix of the state estimation error satisfies the equation
estimation of variances in errors over state parameters (a,b,c):
V
]
We assume that variation in the line angle between two consecutive instants of observation cannot take on a value beyond a fixed threshold. Considering two points M and M' with respective coordinates (x,y) and (x',y') distant from d. The interval of error around M or M' is more or less one point. The change in angle A 0 between these points is then about l/d. Estimation of error over a is defined by Aa = (cos 0) A0,
P.f(k+ l/k) = P.~tk/k~ = [I -- K(k) H(
k)]
P&Wk-l)" (13)
Looking at Eq. (12) we see that as the measurement error covariance R(k) approaches zero, the actual measurement Y(k) is trusted more and more while the predicted measurement H ( k ) X ( k / k - 1) is trusted less and less. In contrast, as the a priori estimation error covariance PY(k/k-l) approaches zero, the actual measurement Y(k) is trusted less and less, while the predicted measurement H(k),~(k/k - 1) is trusted more and more. Blurs in layout generate errors on the points. Hence, a pre-processing procedure is required to give access to the first significant points. We only need for our process an on-line initial procedure searching for the first non-stationary points in order to start properly with the extended Kalman filter algorithm. It is reminded that stationary points (blurs) have identical coordinates. The initial procedure considers two consecutive points. If the distance between these points is greater than one, proper starting of the layout is detected and we can then process with the extended Kalman filter, otherwise, we will start again the initial procedure with an ensuing point. In any practical application of the Kalman filter, the initializing stage is essential. Errors or divergence on the Kalman filter can result from this stage (e.g. Najim, 1988 or Labarrere et al., 1978). We estimate the initial covariance matrix P(0) __a P~m/o) of errors over parameter estimates through an
so that the variance of a is approximated as follows: 1 O'a2 ~ ( a 0 ) 2 = d--7 .
The same approximation is made for parameter b. The smaller d is the larger the variance of the error is.
As far as the c-parameter is concerned, the error admitted is at the most Ac= (xcos 0)A0-- (ysin 0)A0, and majoring cos 0 and sin 0 with the value 1 yields ~,2 ~< (Ix[ + l y [ ) E ( A 0 ) 2 = (Ixl + lyl) 2
d2 where c represents the perpendicular distance from the system's origin to the main axis of inertia. This distance is commonly less than the distance from the center of the digitizer tablet to the diagonal end of its active surface or equal to it. The matrix P translates the confidence that we can have in the model adopted. Experimentally, the handwritten lines have a size superior to 1 centimeter, d is therefore very large compared to 1. Consequently we can fix the matrix P to an important value, which translates our weak confidence in the model, thus the Kalman gain K will be large and the contribution of the term of correction I7 pondered by the gain will be stronger. This remains true for first iterations. Then, the error of estimation .~ will be updated in the course of the time. The model will
198
D. de Brucq et a l . / Pattern Recognition Letters 18 (1997) 193-202
A M(k+l) .ll
thus be able to adapt to the line under way. Nevertheless, the period of filter transition is subdued to good initial values of parameters, which implies that the term of correction will be all the weaker as the initial values are the best. The initial covariance matrix of estimation error is defined by the values o-. = 1, ~rb = 1 and ~ = 8273. From a similar consideration, the covariance R of measurement noise V is taken to be constant:
.°
~. ~
M(k+l)
M(k)
.=[160o o] 0
~-.4/"
Fig. 2. The ~-angle deviation between model and observation.
0,0016 " where /~ is the normal vector to the plane surface formed by the vectors (d, dM)Using vector coordinates, the relation gives
Thus, the initial values for ~(0), b(0) and ~(0) are in the solution neighborhood in order to ensure a fast convergence of the filter.
-D( k) A y - a( k) Ax sin ~o=
v/(Ax) + (Ay)
4. Detection of direction changes Let the ~-angle between average direction and velocity be the parameter for the detection (see Fig. 2). At each instant k, we consider the line point M(k). The extended Kalman filter provides us with an estimation of the line parameters ~(k), b(k) and ~(k) modeling. The normal K to the line with direction vector ( a ( k ) b ( k ) ) ~, has an angle of + ½~- with segment [M(k), M(k + 1)] having direction vector dM=(Ax Ay) y. Defining d & (-b(k) a ( k ) ) T as the vector perpendicular to such that (~/,K) is direct, we are freed from the + ½w-deviation between the normal vector ~" and the line. Thus, the 9-angle satisfies the relation
The above relation is defined if and only if the ()+M-vector length is non-zero. The first decision rule is: detection of a possible model change as soon as the distance between points is less than a minimal distance threshold, denoted Dmi n. This threshold is defined from tool uncertainty. Indeed, errors over point measurements are more or less one unit of point onto point coordinates. The value estimated for Dm~n is 2 units of point. The second test is based on the use of information extracted from the 9-angle between the line modeled and the line observed. When the model is valid, one can approximate sin ~ by ~o. The thresholding value for ~p is G = 15°. The detector is put into effect after filter initialization in order to avoid false alarms. In the same way
d A d~ = IId~lIldM II(sin ,p) K,
N Area of Model Change
Area of almost Stationa D"Points
~
D~. Minimal Disiance to put Alarm
+¢
/
-¢,
-
rl12
~ '
Area of Valid Model
I
Fig. 3. The ~graph relative to distance between points. Detection area of model change.
D. de Brucq et al. / Panern Recognition Letters 18 (1997) 193-202
after a model change detection, the detector is frozen meanwhile re-initializing the extended Kalman filter. We decide to freeze the detector during the duration required to process these two points observed just after the change, or if necessary, the first two nonstationary points following the model change. To illustrate the double thresholding detector, Fig. 3 shows the mangle in terms of the distance between consecutive points. This figure reveals areas corresponding to a change of model.
5. Application to handwritten numerals To test the performance of the line segmentation method, several handwritten numerals are written through a digitizer tablet. Note, that in the examples presented, handwritten numerals are sampled to one hundred Hertz. Each of the numerals has been written by a different writer. The experimental results of line segment identification for the real handwritten Numeral
(a)
199
(b)
Fig. 5. Segmentation of numeral " 1 " . (a) Sampled handwritten numeral composed of 43 points. (b) Result: 4 segments.
patterns are shown in Figs. 5-9. To each example of numeral corresponds a segmentation in terms of line segments. A darker point appears at each end of a line, this is to tell each pen lift of the writer, thus indicating the direction of handwriting. Figs. 5 - 7 present results on handwritten numerals presenting linear parts. The method has been tested on other amounts as indicated in Figs• 8 and 9. A straight line is defined by two points and two consecutive straight lines by three points. So, we introduce the next relationship that means the rate of data reduction:
1 R~educ
Ks KN
First segment
'*
~.~" 4 Secondsegment Third segment
•
• ""
I segment
detected
"
l~Imrth
4 segments
(a)
$eglllellt
detected
(b)
Numeral 7 First segment
....
.•.... .. " '~~
/,'SecondSegment 2 segmentdetected
(c)
First segment
Secondsegment Z
-,,,~-.
....
i
Fourth segment
~
where K s represents the number of points retained by the segmentation method and K N the number of points on the line. This rate allows us to measure the performance of the segmentation. In the examples of Figs. 5-7, Rr~d,c is in the order 1/8. When the line curve consists of more complex forms, see Figs. 8 and 9, this rate is in the order of 1/5. The threshold ~, which decides whether a line is broken or not, is mostly set to be 15 °. Thus, if the actual line segment and the dM-vector measurement are further apart than 15° , they are considered to be abrupt changes and a new line segment is beginning.
..... .• ". , ~ : ~ . ~
'••• ~"'x
Fifth segment
~" Third segment 5 segmentsdetected
~°. °*°~
(d)
Fig. 4. Illustration of detected segments• (a) One segment detected for numeral " 1 " . (b) Four segments detected for numeral " 1 " . (c) Two segments detected for numeral " 7 " . (d) Five segments detected for numeral " 7 " .
(a)
(b)
Fig. 6. Segmentation of numeral "4". (a) Sampled handwritten numeral composed of 47 points.(b) Result: 5 segments.
200
D. de Brucq et a l . / Pattern Recognition Letters 18 (1997) 193-202 t t *
.It
OO • O,/tO
i (a)
(b)
Fig. 7. Segmentation of numeral " 7 " . (a) Sampled handwritten numeral composed of 55 points. (b) Result: 6 segments.
The threshold Dmi., which triggers the detector alarm, is mostly set to be two units of points. To show the sensitivity of the segmentation method, for numeral " 1 " , we can detect from one segment in the situation as given in Fig. 4(a), to four segments in the cases as shown in Fig. 4(b). In the same way with the numeral " 7 " , we can detect from two segments, the case in Fig. 4(c), to five segments, the cases as given in Fig. 4(d). These examples show that the algorithm has detected and modeled all linear parts of the patterns. It is observed that the last segment detected in Fig. 4(b) and Fig. 4(d) is a hook onto these numerals. This hook is due to, on one hand, the pen lift and on the other hand, the digitizer tablet. The matter here is to know if this hook is informative. For that purpose, we propose a supplementary test over the length of the detected segment with respect to the numeral size. For example, if the segment is less than 1% of the size, the hook is rejected. Experimental results do show that the algorithm should be amended to take care of hooks (cf. the third segment in Fig. 4(b) and the fourth segment in Fig. 4(d), for example), as recommended in (Tappert et al., 1988). These hooks are not useful (in fact, may be deleterious) for recognition and should there-
(a)
(b)
Fig. 8. Segmentation of numeral " 2 " . (a) Sampled handwritten numeral composed of 46 points. (b) Result: 8 segments.
~k "41'O O O O (a)
(b)
Fig. 9. Segmentation of numeral "5". (a) Sampled handwritten numeral composed of 62 points. (b) Result: I l segments.
fore be eliminated as part of any performance modeling stage. To estimate the proposed segmentation method in a quantitative manner we introduce an appropriate criterion Q of line segmentation. In principle, the criterion might take the following general form: Q = [the degree of information reduction] + A[the closeness of the solution of the input data], where the factor )t controls the compromise between the two terms. In this application, the first term is the rate of data reduction R~ed~c; the second term is the squared error caused by line segment approximation. To illustrate this criterion on our data, we just used the first term which is the degree of information reduction. On the other hand, the second term is not used in this criterion but it is taken in account during the processing (cf. Section 4). This article presents an algorithm of segmentation by line segments, and we have shown with two examples the necessity to have other primitives including loops, an arc of a circle or an arc of an ellipse for example. Our research work is under way to include an other primitive than the arc of circle by using the same approaches as for the line segment. As a result we can say that segmentation is correct and provides a fine description of rectilinear lines. Some difficulties for the extended Kalman filter to converge appear in cases where layouts present an important degree of curvature. Nevertheless, for very noisy lines, the filter converges assuming a proper initial value for the covariance matrix of measurement noise.
D. de Brucq et a l . / Pattern Recognition Letters 18 (1997) 193-202
6. Conclusion and prospect We have implemented a segmentation method using an extended Kalman filter enabling the detection of line segments in manuscript layouts. The filter used integrates constraints as observation (output) within the state-space model proposed. This approach enables a recursive form easy to implement so that the fast processing resulting enables on-line data analysis. Thus, real-time processing is permitted thanks to linear handwritten layouts or linearizable ones. Layout description through the mere line segment parameters provides robust and reliable primitives. This method is expected to be applied to detection and tracking problems for linear lines within manuscript numerals. More generally, it could contribute to solve some segmentation problems in the field of pattern recognition. A segmentation problem is a test hypothesis problem with false alarm and non-detection error. The optimal threshold value should be presented in this context. Some statistic estimation should be done previously. The extended Kalman filter-implementation requires a measurement noise estimation. This estimation has shown that measurement noise due to the digitizer tablet and measurement noise due to the writing process should be taken into account. A hesitant and nervous writer would have some difficulties to make a perfect rectilinear line. The approach also enables the processing of layouts which are more or less curved. We have pointed out that the layout can be tracked in real-time and that abrupt changes in the model can be detected. The methodology proposed is serious and meets all our requirements: it detects changes of models, it estimates the instant of abrupt changes and it identifies model parameters after a change. The detector proposed turns out to be efficient, enabling on-line data processing adapted to different situations encountered. Thanks to the algorithm introduced, the segmentation of manuscript layouts through line segments proves to be efficient for the examples of handwritten numerals presented. In some cases, the convergence of the extended Kalman filter is perturbed when layouts present an important degree of curvature. Nevertheless, for very noisy layouts, the filter
201
converges assuming a proper initial value for the covariance matrix of output noise. Further prospects consider the curved layout modeling through an arc of a circle, enabling a better description of handwritten numerals.
References Appel, U. and A.V. Brand (1983). Adaptive sequential segmentation of picewise stationary time series. Inform. Sci. 29. Ayache, N. and O.D. Faugeras (1986). Hyper: a new approach for the recognition and positioning of two-dimensional objects. IEEE Trans. Pattern Anal. Machine lntell. 8 (1). Basseville, M. and A. Benveniste (1983). Design and comparative study of some sequential jump detection algorithms for digital signals. IEEE Trans. Acoust. Speech Signal Process. 31 (31). Brown, R.G. and P.Y.C. Hwang (1992). Introduction to Random Signals and Applied Kalman Filtering, 2nd edition. Wiley, New York. Dave, R.N. (1989). Use of adaptive fuzzy clustering algorithm to detect lines in digital images. Intelligent Robots and Computer Vision VIII: Algorithms and Techniques, SPIE 1192, 600-611. De Brucq, D. and G. Folliot (1988). Mod~lisation Statistique, Automatique et Traitement. Masson, Paris. Deleu, J. (1991). Le projet PASTIS, Reconnaissance approch6e de dessins au trait. Th~se de l'Universit6 des Sciences et Techniques de Lille Flandres Artois. Duda, R.O. and P.E. Hart (1972). Use of the Hough transformation to detect lines and curves in pictures. Commun. ACM 15
(1), 11-15. Giai-Checa, B., R. Deriche, T. Vi6ville and O. Faugeras (1993). Suivi de segments dans une S&luence d'images monoculaire. Rapport de Recherche INRIA, No. 2113. Jacobs, O.L.R. (1993). Introduction to Control Theory, 2nd edition. Oxford University Press, Oxford. Kalman, R.E. and R.S. Bucy (1961). New results in linear filtering and prediction theory. Trans. ASME J. Basic Engrg., Serie D, 83, 95-108. Labarrere, M., J.P. Rief and B. Gimonet (1978). Le Filtrage et Ses Applications. Cepadues-Editions, Paris. Maitre, H. (1985). Un panorama sur la Transformation de Hough. Traitement du Signal 2 (4), 305-317. Martin, P. (1992). R6seaux de neurones artificiels: application h la reconnaissance optique de partitions musicales. Th~se de l'Universit6 Joseph Fourier, Grenoble I. Najim, M. (1988). Mod~lisation et Identification en Traitement du Signal. Masson, Paris. Nikivorov, I.V. (1983). Sequential Detection of Changes in Time Series Properties. Nauka, Moscow. Ostentad, B. (1988). D6composition des objets d'une partition musicale num6ris6e en entit6s classables, lnstitut d'Informatique de l'Universit~ d'Oslo, Report in Norvegian, No. 31. Pavlidis, T. and S.L. Horowitz (1974). Segmentation of plane curves. IEEE Tram'. Comput. 23 (8), 860-870.
202
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
Poulain d'Andecy, V., J. Camillerap and 1. Lepplumey (1994). D&ecteur robuste de segments, Application h l'analyse de partitions musicales. Reconnaissance des Farmes et Intelligence Artificielle, 1994, Paris. Ramer, U. (1972). An iterative procedure for the polygonal approximation of plane curves. Computer Graphics and Image Processing 1,244-256. Sauter, D. (1991). Contribution ~ l'Etude des mEthodes de dEtection de rupture de module: Application ~ la detection d'EvEne-
ment darts les signaux et au diagnostic de dEfauts de syst~mes. Thbse de I'UniversitE de Nancy I. Tappert, C.C., C.Y. Suen and T. Wakahara (1988). On-line Handwritten Recognition 2. November 1988, Rome. Traitement du Signal (1992). Collectif GRECO-TDSI, GDR 134. Signanx non stationnaires, analyse temps-fr~quence et segmentation. Traitement du Signal 9 (1). Wald, A. (1974). Sequential Analysis. Wiley, New York.
Pattern Recognition Letters 18 (1997) 193-202
On-line handwritten segmentation in linear drawings D. de Brucq a, M. Amara a,*, V. Ruiz b a PSI-La3i, UFR des Sciences, Universit[ de Rouen, 76821 Mont-Saint-Aignan Cedex, France b GREYC, CNRS URA 1526, ISMRA, 6 Bd Mar£chal Juin, 14050 Caen Cedex, France
Received 1 November 1995; revised 22 October 1996
Abstract This paper addresses the problem of tracking line segments corresponding to on-line handwritten obtained through a digitizer tablet. The approach is based on Kalman filtering to model linear portions of on-line handwritten, particularly, handwritten numerals, and to detect abrupt changes in handwritten direction underlying a model change. This approach uses a Kalman filter framework constrained by a normalized line equation, where quadratic terms are linearized through a first-order Taylor expansion. The modeling is then carried out under the assumption that the state is deterministic and time-invariant, while the detection relies on double thresholding mechanism which tests for a violation of this assumption. The first threshold is based on an approach of layout kinetics. The second one takes into account the jump in angle between the past observed direction of layout and its current direction. The method proposed enables real-time processing. To illustrate the methodology proposed, some results obtained from handwritten numerals are presented. © 1997 Elsevier Science B.V. Keywords: Extended Kalman filtering; Identification; Primitive; Model changes; Abrupt changes of model; Handwritten numerals; On-line
processing 1. Introduction Tracking a primitive (point, segment, curve, etc.) in the course of the time through a handwritten form is a fundamental problem in computer vision. Indeed, these lists of primitives matched are symbolic representations that are going to allow considering the handwritten segmentation in view of the recognition of the handwriting as well as the construction of traced handwriting. Our study deals with on-line modeling of handwritten through three types of geometric primitives: line segment, arc of circle and arc of ellipse. Due to the variability o f handwriting,
* Correponding author. E-mail: [email protected].
these primitives have to be robust to noise and describe layouts as precisely as possible. Today, we have encouraged numerical results for each of these primitives. The aim of this paper is to present a new approach to line segmentation by an extended Kalman filter. Firstly, the approach has to estimate the parameters of a line as precisely as possible and secondly it has to extract segments from handwritten layouts which are the constituents of complex shapes in an automatic way. W e present an on-line process to express a symbolic representation o f handwritten layouts through a succession of linear segments. W e have restricted this paper to linear segmentation because particulars of the extended Kalman filter equations for line, circle and ellipse are quite different.
0167-8655/97/$17.00 Copyright © 1997 Elsevier Science B.V. All rights reserved. PII S0167-8655(96)00126-2
194
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
Many different techniques have been implemented to extract line segments on outlines. Polygonal approximation on images is still the method commonly used in handwriten recognition systems. Most of the works try to minimize the distance between a part of the contour and a line segment (e.g. Ramer, 1972, or Pavlidis and Horowitz, 1974). The Hough Transform has been introduced to detect complex shapes of points in binary images. The interest of this technique is to transfer the problem of parameter extraction from the image space to a problem of local spike extraction from the representative space of parameters as presented in (Ostentad, 1988). The precision of the detection depends on the choice of discrete partition of the parameter space. A complete processing of the Hough Transform can be found in (Maitre, 1985). Other approximation techniques have been experienced such as discrete chord calculation (e.g. Martin, 1992). Some fuzzy technique-based methods arise from these works to alleviate the connection problem as in (Dave, 1989). With the fuzzy techniques, a point cloud can be approached through a line by considering points assigned to a given class with a belonging coefficient. In particular, (Deleu, 1991) presents a very detailed description of various existing methods for segment extraction from an outline in which analytical, morphological or connection methods can be found. These methods are limited as compared to our approach. They do not allow an on-line estimation of parameters from data. Thus, we can anticipate an on-line segmentation. Our objective is to estimate line segment parameters in a recursive manner that enables on-line data processing from the set of points constituting handwritten layouts and on-line segmentation of complex forms. The recursive least squares methods use line segment models that are commonly described through Cartesian equations. This approach suffers from a major drawback. It prevents from finding a reliable solution for segments with an infinite direction coefficient which can yield numerical problems in terms of computation. Many researchers have used the Kalman filter in computer vision for complex forms, notably GiaiCheca et al. (1993) and Ayache and Faugeras (1986). The last two authors describe silhouettes of objects
with a polygonal approximation of their outlines. The technique of recognition uses the Kalman filter to match a segment of the model with a segment of the scene observed under hypotheses as well as the position of models in the scene. Poulain d'Andecy et al. (1994) propose a recursive approach for line segment detection in score or stave images based on the Kalman filtering. But, the model proposed is tight with the time-invariant speed assumption which prevents the processing of some particular types of line segments. In our approach, the recursive form arises from the layout representation through particular state and measure (observation) equations that enable the use of extended Kalman filtering techniques. The measure equations are constituted of two constraints. A first constraint is the normal equation of line, ax + b y - c = 0, where x and y are time-varying point coordinates. A second constraint is the condition, a 2 + b 2 = 1, over line parameters in order to ensure a unique representation. We are here considering that the values 0 and 1 are relative to the first and second constraint and constitute the (time-invariant) observations of the state modeling. Linear approximation implies time-invariant parameters a and b. The state equations express this property. The method proposed is extended to detect model changes (in layout direction). The notion of abrupt changes in the model has been introduced in (Wald, 1974) and this notion has particularly been paid attention to in the last twenty years. A detailed state of art concerning the different types of detectors applied to slowly-varying signals can be found in (Traitement du Signal, 1992). To free oneself from the non-stationary problem, methods of detection based on a distance test between models have been introduced in (Appel and Brand, 1983) and were then extended in (Basseville and Benveniste, 1983). Finally, Nikivorov (1983) solved the problem of change detection in signal spectral characteristics with a local approach. Our application has to detect and localize changes of direction in the characteristics of layouts. According to (Sauter, 1991) the statistical tests conceived to detect this kind of abrupt changes cannot always be applied and a new approach should therefore be proposed. We are thus introducing a double threshold detec-
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
tor. The first threshold results from an approach related to handwriting kinetics and the second threshold takes into account the angle variation between the layout direction resulting from the extended Kalman filter and its current measure. We will first present the line model. We will then define the Kalman filter equations after a linearizing stage. Covariance matrices of errors and extended Kalman filter-initial conditions will then be explained. Finally, we will present the results obtained from the application to digitized numerals once the theoretical detector approach has been implemented.
195
Y
-~3.'~Y__~)x,y)
(D) Fig. 1. Normal line equation.
The hypothesis of the non-variation of the single line leads to the following system of state equations, holding for all k:
2. State-space model
a(k+ll=a(k) The parametrization of a line by ( y = ax + b) is not adequate because the (a,b) space is inhomogeneous (Duda and Hart, 1972). Observations x and y are taken to act in a symmetric way, thus, the line equation in its form can be defined as follows:
a x + f l y - y=O,
(I)
where a , 13 and 3' are real-valued constants. With variation in time, the model taken to describe various observations
x(1) y(1)
...
x(K)]
.-"
y(K)]
k) +by(k) - c = 0 ,
(2)
where c represents ~ e algebraic distance of the vector ~ = ( ~ , n ) n . Here, we define the normal vector, ~, relative to the line (D) (see Fig. 1), by
/3 sin0e ~7+/32
(3) where X(k + l) represents the state equation characterizing the X(k)-state constant evolution, and X tr= [a,b,c]. We thus assume a time-invariant line slope. Globally, the line is invariant over the point cloud under consideration. We are searching for a single line verifying the unicity of the parameters (normal vector) due to the constraint a2(k) + b2(k) : 1.
is the canonical representation of the straight line defined as follows:
g=
b(k + l ) = b ( k ) thatmeans X(k + l ) = X ( k ) , c(k+ 1) = c(k)
-
and the direction vector d of the line (D) is ( - b a) r. With this equation, c and 0 are always finite.
(4)
Eqs. (2) and (4) define a system of observation where x(k) and y(k), k = 1,2 . . . . . K, are the points issued from the digitizer tablet. The line (D) is covered in the points acquisition direction. Eqs. (2) and (4) define two constraints on the line. The observation is defined by these constraints, in an original way. The novelty here consists in defining the measure equations through the two constraints instead of using only the measures, x(k) and y(k), from the digitizer tablet. Indeed, these are the values 0 and 1 from Eqs. (2) and (4) that are being observed. Due to the accuracy of numerical computations and measurements of x(k) and y(k), we will have to define for each constraint constituting the measure equations (2) and (4), a measurement noise denoted Vd and V,., respectively. However, the second constraint in Eq. (4) has to be linearized with respect to
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
196
the state parameters a and b, for the purpose of using the extended Kalman technique. Eq. (4) is linearized by a first-order Taylor expansion around the estimated values a ( k / k - 1 ) and 12(k/k-1) taking into account the known observations towards ( k - 1). Let us assume:
a(k) ~=~ ( k / k - 1) + 6 a ( k - 1),
b(k) ~ ~,(k/k-l) + 8 b(k- 1).
(5)
The linearization can be accomplished by substituting Eq. (5) into Eq. (4), and we can show that this is expressed by
r~(k) ~ 1 + ~ 2 ( k / k - 1) + b 2 ( k / k - 1) :
2~(k/k-1)a(k) + ~b(k/k-l)b(k) + Vc, (6)
measurement Y(k) and is identified by Eqs. (6) and (7). It is written as follows: zx I x ( k )
h,(k)=L2~(k/k-l)
y(k)
-1]
2~,(k/k-1)
0
For this linearized model we used the classical Kalman filter equations (Kalman and Bucy, 1961; Labarrere et al., 1978; Jacobs, 1993). We define ~'(k/k- 1)~ R 3 to be our a priori state estimate at step k given knowledge of the process prior to step k, and X(k/k) ~ E3 to be our a posteriori state estimate at step k given measurement Y(k). We can define a priori and a posteriori estimation errors as
2?(k/k- l) =x(k) -,¢(k/k- 1)
where V, is the measurement noise on the constraint including numerical errors due to the canceling of the quadratic differences. Eq. (2) can be written in the form,
The a priori estimation error covariance is then
Ya( k) & a(k) x(k) + b(k) y(k) - c(k) + V,,,
P;(k/k-,) = E[ )(( k / k - 1))~( k / k - 1) tr]
(7) where Vd is the measurement noise on the line including numerical errors due to uncertainties in the position of x(k) and y(k). Eqs. (6) and (7) constitute the system of observations.
3. Kaiman filtering equations We are introducing below the Kalman filter equations. The Kalman filter addresses the general problem of trying to estimate the state X ~ I ~ 3 of a first-order, discrete-time controlled process that is governed by the linear difference Eq. (4), and with a measurement Y ~ ~2 that is r'(k) = H ( k ) X ( k )
+ V(k),
(8)
where
r(k~ [ r~'(k~ ] vc(k~) is the measurement vector (observation). The random variable V(k) represents the measurement noise with zero mean and covariance R, and the matrix H(k) in the measurement equation (8) relates the state to the
" (9)
and
.,~( k/k ) = X(k) - .~( k/k ).
and the a posteriori estimation error covariance is
Px~k/k~=e[~(k/klY(k/k-l)'~].
(101
In deriving the equations for the Kalman filter, we begin with the goal of finding an equation that computes an a posteriori state estimate X(k/k) and a weighted difference between an actual measurement Y(k) and a measurement prediction H(k))((k/k- 1) as follows:
2?(k/k) = 2?( 1,/k - l) + x(k~[r(k)-I-l(k~?(~/~-1~]. (11) The difference [Y(k) - H ( k ) X ( k / k - 1)] in Eq. (11) is called the measurement innovation, or the residual. The residual reflects the discrepancy between the predicted measurement H ( k ) X ( k / k - 1) and the actual measurement Y(k). A residual of zero means that both are in complete agreement. The matrix K in Eq. (11) is chosen as being the gain or blending factor that minimizes the a posteriori error covariance of Eq. (10). The minimization can be accomplished by first substituting Eq. (11) into the above definition for ,~(k/k), substituting
197
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
that into Eq. (10), performing the indicated expectations, taking the derivative of the trace of the result with respect to K, setting that result equal to zero, and then solving for K. For more details see (De Brucq and Folliot, 1988; Najim, 1988; Brown and Hwang, 1992; Jacobs, 1993). One form of the resulting K that minimizes Eq. (10) is given by K(k) =
P.f,,~/k_,,H(k)tr(H(k)P~(wk_,)H(k) tr + R ) - ' . (12) Finally, the updated covariance matrix of the state estimation error satisfies the equation
estimation of variances in errors over state parameters (a,b,c):
V
]
We assume that variation in the line angle between two consecutive instants of observation cannot take on a value beyond a fixed threshold. Considering two points M and M' with respective coordinates (x,y) and (x',y') distant from d. The interval of error around M or M' is more or less one point. The change in angle A 0 between these points is then about l/d. Estimation of error over a is defined by Aa = (cos 0) A0,
P.f(k+ l/k) = P.~tk/k~ = [I -- K(k) H(
k)]
P&Wk-l)" (13)
Looking at Eq. (12) we see that as the measurement error covariance R(k) approaches zero, the actual measurement Y(k) is trusted more and more while the predicted measurement H ( k ) X ( k / k - 1) is trusted less and less. In contrast, as the a priori estimation error covariance PY(k/k-l) approaches zero, the actual measurement Y(k) is trusted less and less, while the predicted measurement H(k),~(k/k - 1) is trusted more and more. Blurs in layout generate errors on the points. Hence, a pre-processing procedure is required to give access to the first significant points. We only need for our process an on-line initial procedure searching for the first non-stationary points in order to start properly with the extended Kalman filter algorithm. It is reminded that stationary points (blurs) have identical coordinates. The initial procedure considers two consecutive points. If the distance between these points is greater than one, proper starting of the layout is detected and we can then process with the extended Kalman filter, otherwise, we will start again the initial procedure with an ensuing point. In any practical application of the Kalman filter, the initializing stage is essential. Errors or divergence on the Kalman filter can result from this stage (e.g. Najim, 1988 or Labarrere et al., 1978). We estimate the initial covariance matrix P(0) __a P~m/o) of errors over parameter estimates through an
so that the variance of a is approximated as follows: 1 O'a2 ~ ( a 0 ) 2 = d--7 .
The same approximation is made for parameter b. The smaller d is the larger the variance of the error is.
As far as the c-parameter is concerned, the error admitted is at the most Ac= (xcos 0)A0-- (ysin 0)A0, and majoring cos 0 and sin 0 with the value 1 yields ~,2 ~< (Ix[ + l y [ ) E ( A 0 ) 2 = (Ixl + lyl) 2
d2 where c represents the perpendicular distance from the system's origin to the main axis of inertia. This distance is commonly less than the distance from the center of the digitizer tablet to the diagonal end of its active surface or equal to it. The matrix P translates the confidence that we can have in the model adopted. Experimentally, the handwritten lines have a size superior to 1 centimeter, d is therefore very large compared to 1. Consequently we can fix the matrix P to an important value, which translates our weak confidence in the model, thus the Kalman gain K will be large and the contribution of the term of correction I7 pondered by the gain will be stronger. This remains true for first iterations. Then, the error of estimation .~ will be updated in the course of the time. The model will
198
D. de Brucq et a l . / Pattern Recognition Letters 18 (1997) 193-202
A M(k+l) .ll
thus be able to adapt to the line under way. Nevertheless, the period of filter transition is subdued to good initial values of parameters, which implies that the term of correction will be all the weaker as the initial values are the best. The initial covariance matrix of estimation error is defined by the values o-. = 1, ~rb = 1 and ~ = 8273. From a similar consideration, the covariance R of measurement noise V is taken to be constant:
.°
~. ~
M(k+l)
M(k)
.=[160o o] 0
~-.4/"
Fig. 2. The ~-angle deviation between model and observation.
0,0016 " where /~ is the normal vector to the plane surface formed by the vectors (d, dM)Using vector coordinates, the relation gives
Thus, the initial values for ~(0), b(0) and ~(0) are in the solution neighborhood in order to ensure a fast convergence of the filter.
-D( k) A y - a( k) Ax sin ~o=
v/(Ax) + (Ay)
4. Detection of direction changes Let the ~-angle between average direction and velocity be the parameter for the detection (see Fig. 2). At each instant k, we consider the line point M(k). The extended Kalman filter provides us with an estimation of the line parameters ~(k), b(k) and ~(k) modeling. The normal K to the line with direction vector ( a ( k ) b ( k ) ) ~, has an angle of + ½~- with segment [M(k), M(k + 1)] having direction vector dM=(Ax Ay) y. Defining d & (-b(k) a ( k ) ) T as the vector perpendicular to such that (~/,K) is direct, we are freed from the + ½w-deviation between the normal vector ~" and the line. Thus, the 9-angle satisfies the relation
The above relation is defined if and only if the ()+M-vector length is non-zero. The first decision rule is: detection of a possible model change as soon as the distance between points is less than a minimal distance threshold, denoted Dmi n. This threshold is defined from tool uncertainty. Indeed, errors over point measurements are more or less one unit of point onto point coordinates. The value estimated for Dm~n is 2 units of point. The second test is based on the use of information extracted from the 9-angle between the line modeled and the line observed. When the model is valid, one can approximate sin ~ by ~o. The thresholding value for ~p is G = 15°. The detector is put into effect after filter initialization in order to avoid false alarms. In the same way
d A d~ = IId~lIldM II(sin ,p) K,
N Area of Model Change
Area of almost Stationa D"Points
~
D~. Minimal Disiance to put Alarm
+¢
/
-¢,
-
rl12
~ '
Area of Valid Model
I
Fig. 3. The ~graph relative to distance between points. Detection area of model change.
D. de Brucq et al. / Panern Recognition Letters 18 (1997) 193-202
after a model change detection, the detector is frozen meanwhile re-initializing the extended Kalman filter. We decide to freeze the detector during the duration required to process these two points observed just after the change, or if necessary, the first two nonstationary points following the model change. To illustrate the double thresholding detector, Fig. 3 shows the mangle in terms of the distance between consecutive points. This figure reveals areas corresponding to a change of model.
5. Application to handwritten numerals To test the performance of the line segmentation method, several handwritten numerals are written through a digitizer tablet. Note, that in the examples presented, handwritten numerals are sampled to one hundred Hertz. Each of the numerals has been written by a different writer. The experimental results of line segment identification for the real handwritten Numeral
(a)
199
(b)
Fig. 5. Segmentation of numeral " 1 " . (a) Sampled handwritten numeral composed of 43 points. (b) Result: 4 segments.
patterns are shown in Figs. 5-9. To each example of numeral corresponds a segmentation in terms of line segments. A darker point appears at each end of a line, this is to tell each pen lift of the writer, thus indicating the direction of handwriting. Figs. 5 - 7 present results on handwritten numerals presenting linear parts. The method has been tested on other amounts as indicated in Figs• 8 and 9. A straight line is defined by two points and two consecutive straight lines by three points. So, we introduce the next relationship that means the rate of data reduction:
1 R~educ
Ks KN
First segment
'*
~.~" 4 Secondsegment Third segment
•
• ""
I segment
detected
"
l~Imrth
4 segments
(a)
$eglllellt
detected
(b)
Numeral 7 First segment
....
.•.... .. " '~~
/,'SecondSegment 2 segmentdetected
(c)
First segment
Secondsegment Z
-,,,~-.
....
i
Fourth segment
~
where K s represents the number of points retained by the segmentation method and K N the number of points on the line. This rate allows us to measure the performance of the segmentation. In the examples of Figs. 5-7, Rr~d,c is in the order 1/8. When the line curve consists of more complex forms, see Figs. 8 and 9, this rate is in the order of 1/5. The threshold ~, which decides whether a line is broken or not, is mostly set to be 15 °. Thus, if the actual line segment and the dM-vector measurement are further apart than 15° , they are considered to be abrupt changes and a new line segment is beginning.
..... .• ". , ~ : ~ . ~
'••• ~"'x
Fifth segment
~" Third segment 5 segmentsdetected
~°. °*°~
(d)
Fig. 4. Illustration of detected segments• (a) One segment detected for numeral " 1 " . (b) Four segments detected for numeral " 1 " . (c) Two segments detected for numeral " 7 " . (d) Five segments detected for numeral " 7 " .
(a)
(b)
Fig. 6. Segmentation of numeral "4". (a) Sampled handwritten numeral composed of 47 points.(b) Result: 5 segments.
200
D. de Brucq et a l . / Pattern Recognition Letters 18 (1997) 193-202 t t *
.It
OO • O,/tO
i (a)
(b)
Fig. 7. Segmentation of numeral " 7 " . (a) Sampled handwritten numeral composed of 55 points. (b) Result: 6 segments.
The threshold Dmi., which triggers the detector alarm, is mostly set to be two units of points. To show the sensitivity of the segmentation method, for numeral " 1 " , we can detect from one segment in the situation as given in Fig. 4(a), to four segments in the cases as shown in Fig. 4(b). In the same way with the numeral " 7 " , we can detect from two segments, the case in Fig. 4(c), to five segments, the cases as given in Fig. 4(d). These examples show that the algorithm has detected and modeled all linear parts of the patterns. It is observed that the last segment detected in Fig. 4(b) and Fig. 4(d) is a hook onto these numerals. This hook is due to, on one hand, the pen lift and on the other hand, the digitizer tablet. The matter here is to know if this hook is informative. For that purpose, we propose a supplementary test over the length of the detected segment with respect to the numeral size. For example, if the segment is less than 1% of the size, the hook is rejected. Experimental results do show that the algorithm should be amended to take care of hooks (cf. the third segment in Fig. 4(b) and the fourth segment in Fig. 4(d), for example), as recommended in (Tappert et al., 1988). These hooks are not useful (in fact, may be deleterious) for recognition and should there-
(a)
(b)
Fig. 8. Segmentation of numeral " 2 " . (a) Sampled handwritten numeral composed of 46 points. (b) Result: 8 segments.
~k "41'O O O O (a)
(b)
Fig. 9. Segmentation of numeral "5". (a) Sampled handwritten numeral composed of 62 points. (b) Result: I l segments.
fore be eliminated as part of any performance modeling stage. To estimate the proposed segmentation method in a quantitative manner we introduce an appropriate criterion Q of line segmentation. In principle, the criterion might take the following general form: Q = [the degree of information reduction] + A[the closeness of the solution of the input data], where the factor )t controls the compromise between the two terms. In this application, the first term is the rate of data reduction R~ed~c; the second term is the squared error caused by line segment approximation. To illustrate this criterion on our data, we just used the first term which is the degree of information reduction. On the other hand, the second term is not used in this criterion but it is taken in account during the processing (cf. Section 4). This article presents an algorithm of segmentation by line segments, and we have shown with two examples the necessity to have other primitives including loops, an arc of a circle or an arc of an ellipse for example. Our research work is under way to include an other primitive than the arc of circle by using the same approaches as for the line segment. As a result we can say that segmentation is correct and provides a fine description of rectilinear lines. Some difficulties for the extended Kalman filter to converge appear in cases where layouts present an important degree of curvature. Nevertheless, for very noisy lines, the filter converges assuming a proper initial value for the covariance matrix of measurement noise.
D. de Brucq et a l . / Pattern Recognition Letters 18 (1997) 193-202
6. Conclusion and prospect We have implemented a segmentation method using an extended Kalman filter enabling the detection of line segments in manuscript layouts. The filter used integrates constraints as observation (output) within the state-space model proposed. This approach enables a recursive form easy to implement so that the fast processing resulting enables on-line data analysis. Thus, real-time processing is permitted thanks to linear handwritten layouts or linearizable ones. Layout description through the mere line segment parameters provides robust and reliable primitives. This method is expected to be applied to detection and tracking problems for linear lines within manuscript numerals. More generally, it could contribute to solve some segmentation problems in the field of pattern recognition. A segmentation problem is a test hypothesis problem with false alarm and non-detection error. The optimal threshold value should be presented in this context. Some statistic estimation should be done previously. The extended Kalman filter-implementation requires a measurement noise estimation. This estimation has shown that measurement noise due to the digitizer tablet and measurement noise due to the writing process should be taken into account. A hesitant and nervous writer would have some difficulties to make a perfect rectilinear line. The approach also enables the processing of layouts which are more or less curved. We have pointed out that the layout can be tracked in real-time and that abrupt changes in the model can be detected. The methodology proposed is serious and meets all our requirements: it detects changes of models, it estimates the instant of abrupt changes and it identifies model parameters after a change. The detector proposed turns out to be efficient, enabling on-line data processing adapted to different situations encountered. Thanks to the algorithm introduced, the segmentation of manuscript layouts through line segments proves to be efficient for the examples of handwritten numerals presented. In some cases, the convergence of the extended Kalman filter is perturbed when layouts present an important degree of curvature. Nevertheless, for very noisy layouts, the filter
201
converges assuming a proper initial value for the covariance matrix of output noise. Further prospects consider the curved layout modeling through an arc of a circle, enabling a better description of handwritten numerals.
References Appel, U. and A.V. Brand (1983). Adaptive sequential segmentation of picewise stationary time series. Inform. Sci. 29. Ayache, N. and O.D. Faugeras (1986). Hyper: a new approach for the recognition and positioning of two-dimensional objects. IEEE Trans. Pattern Anal. Machine lntell. 8 (1). Basseville, M. and A. Benveniste (1983). Design and comparative study of some sequential jump detection algorithms for digital signals. IEEE Trans. Acoust. Speech Signal Process. 31 (31). Brown, R.G. and P.Y.C. Hwang (1992). Introduction to Random Signals and Applied Kalman Filtering, 2nd edition. Wiley, New York. Dave, R.N. (1989). Use of adaptive fuzzy clustering algorithm to detect lines in digital images. Intelligent Robots and Computer Vision VIII: Algorithms and Techniques, SPIE 1192, 600-611. De Brucq, D. and G. Folliot (1988). Mod~lisation Statistique, Automatique et Traitement. Masson, Paris. Deleu, J. (1991). Le projet PASTIS, Reconnaissance approch6e de dessins au trait. Th~se de l'Universit6 des Sciences et Techniques de Lille Flandres Artois. Duda, R.O. and P.E. Hart (1972). Use of the Hough transformation to detect lines and curves in pictures. Commun. ACM 15
(1), 11-15. Giai-Checa, B., R. Deriche, T. Vi6ville and O. Faugeras (1993). Suivi de segments dans une S&luence d'images monoculaire. Rapport de Recherche INRIA, No. 2113. Jacobs, O.L.R. (1993). Introduction to Control Theory, 2nd edition. Oxford University Press, Oxford. Kalman, R.E. and R.S. Bucy (1961). New results in linear filtering and prediction theory. Trans. ASME J. Basic Engrg., Serie D, 83, 95-108. Labarrere, M., J.P. Rief and B. Gimonet (1978). Le Filtrage et Ses Applications. Cepadues-Editions, Paris. Maitre, H. (1985). Un panorama sur la Transformation de Hough. Traitement du Signal 2 (4), 305-317. Martin, P. (1992). R6seaux de neurones artificiels: application h la reconnaissance optique de partitions musicales. Th~se de l'Universit6 Joseph Fourier, Grenoble I. Najim, M. (1988). Mod~lisation et Identification en Traitement du Signal. Masson, Paris. Nikivorov, I.V. (1983). Sequential Detection of Changes in Time Series Properties. Nauka, Moscow. Ostentad, B. (1988). D6composition des objets d'une partition musicale num6ris6e en entit6s classables, lnstitut d'Informatique de l'Universit~ d'Oslo, Report in Norvegian, No. 31. Pavlidis, T. and S.L. Horowitz (1974). Segmentation of plane curves. IEEE Tram'. Comput. 23 (8), 860-870.
202
D. de Brucq et al. / Pattern Recognition Letters 18 (1997) 193-202
Poulain d'Andecy, V., J. Camillerap and 1. Lepplumey (1994). D&ecteur robuste de segments, Application h l'analyse de partitions musicales. Reconnaissance des Farmes et Intelligence Artificielle, 1994, Paris. Ramer, U. (1972). An iterative procedure for the polygonal approximation of plane curves. Computer Graphics and Image Processing 1,244-256. Sauter, D. (1991). Contribution ~ l'Etude des mEthodes de dEtection de rupture de module: Application ~ la detection d'EvEne-
ment darts les signaux et au diagnostic de dEfauts de syst~mes. Thbse de I'UniversitE de Nancy I. Tappert, C.C., C.Y. Suen and T. Wakahara (1988). On-line Handwritten Recognition 2. November 1988, Rome. Traitement du Signal (1992). Collectif GRECO-TDSI, GDR 134. Signanx non stationnaires, analyse temps-fr~quence et segmentation. Traitement du Signal 9 (1). Wald, A. (1974). Sequential Analysis. Wiley, New York.