Scene labeling: an optimization approach

00 1- 20 /80/1001 0

Pattern Recognition Vol. 12 . pp . 9- 7 . Pergamon Press Ltd . 1980. Printed in Great Britain . O Pattern Recognition Society .

9 50200/0

SCENE LABELING : AN OPTIMIZATION APPROACH 0 .FAUGERASw and M . BERTHOD Iria-laboria, Domaine de Voluceau-Rocquencourt, 78150 Le Chesnay, 78150 France (Received 10 January

1980)

Abstract-This paper deals with an optimization technique applied to the problem of stochastic labeling. We propose a definition of a global criterion on a set of objects to be labeled that combines both ambiguity and consistency with adjustable weights . A projected gradient algorithm is developed to minimize the criterion . Results are shown on a toy example and on the edge detection problem . Comparisons are made with relaxation labeling techniques . Relaxation labeling Stochastic labeling

Steepest descent

l . INTRODUCTION

Scene analysis attempts to automatically produce a description or a model of a scene . The first step is to identify different subparts in the given scene and to assign a name or a class to them . This is a pattern recognition problem . As the output of any pattern recognition algorithm is not error-free, it is natural (and commonplace) to bring in some contextual information to improve its performance . This contextual information may be thought of as an a priori knowledge of the likelihood of the occurence of any combination of subparts . To cast this into a more precise setting, let us suppse that we have a set of N objects a 1 , . . .,a,k which fall into Q classes Furthermore we are capable of assigning to each object a, a set of probabilities p 1 (i.,) for i = 1, . . . , N and .,) is the probability that object a, k = 1, . . .,Q- pi() belongs to class L, . In what follows we will refer to this set of probabilities as a stochastic labeling of objects a„ . . . , a,r . We suppose further that for every object a, we can define a set of neighbouring objects V, which are in relation to object a,. This means that for every object a; belonging to the set V, we have a conditional probability po(),,J),,) which gives the probability that object a, belongs to class )„ given that object a 2 belongs to class L t . The p,J(;.,JL,)s represent our contextual information . The problem of determining them is of course a crucial one . They can be obtained through some a priori modeling or some statistical estimation on 'ground truth' data, that is to say, hand-labeled data .

We will not be concerned with that problem in this paper and consider in what follows that the p ;r(i.,~L,)s are given to us. It must be pointed out that this is a somewhat limited form of contextual information . A more complete contextual information would be given by the conditional probability that object a, belongs to class L, for a given labeling of objects in V, . This information would be of course very difficult to gather and to store . In most applications the probabilities p i().,) are obtained through noisy measurements and thus suffer from two drawbacks . (i) They are not consistent, that is do not verify PI(Lk)

Q = 2: PI>(Lkk,)P JO.1)

(1)

1=1

for j in V;. Consistency of stochastic labeling indicates how measured data are compatible with respect to the contextual information represented by the p, (1,, ),). (ii) They yield an ambiguous labeling of the objects a a, This means that we are uncertain as to which class object a, belongs to or equivalently that the probability vector p; _ [pi( ) . . .p,(;.Q)]r is different from a unit vector Lo, . . ., 0, 1, 0' . . .o]r. If the entropy of object a, under a given stochastic labeling is defined to be ~Q ;-A)[1 - Pt(i H, = C, PI( .k)] = 1 A-1

Q 2:

Pr(i, t) 2 .

( 2)

k=1

it is well known that H, is a positive quantity equal to zero if, and only if the probability vector p i is a unit vector of the form [0, . . .,0, 1, 0, . . .0]T and maximum when the p i().,) are all equal to 1/Q . An ambiguous labeling corresponds to a high value of H, for every object a, . Our purpose here is to define an algorithm that uses contextual information and the notion of ambiguity to modify the probability weights for every object in such a way that consistency is increased and ambiguity is decreased . Rosenfeld et al ."t proposed two such algor-

Present address : University of Southern California, Imaging Processing Institute, Powell Hall, Los Angeles, CA 90007, U .S.A . t 2979 IEEE. Reprinted with permission from IEEE Computer Society Conference on Pattern Recognition and Image Processing . 6-8 August 1979, Chicago, Illinois . PS 12/5 . E

Edge detection

9



0

0 . FAUGERAs

ithms . The linear one is shown to converge to the same limit irrespective of any initial bias on the label probabilities . This algorithm cannot be very useful since its limit does not depend on the initial probabilities, that is on the measurements . The nonlinear algorithm is shown to behave in a more interesting way on several examples ." .21 But since no explicit measure of ambiguity is used in the design of the algorithm, convergence proofs and characterization of fixed points have proved to be difficult . This is what prompted us to use a different approach based upon the minimization of a criterion . 2. DEFINITION OF THE CRITERION

The criterion is designed to measure two quantities : (i) The consistency of the stochastic labeling . Following,"' we define for every object a, the `consistency' probability vector q, by : 1(Ak)=

S

au JEVI

k=I Q

PijOkIkjPJ(2,)

()

1=l

The a, js are positive weights which verify : Y, ap=1

for i=1, . . .,N .

and

M . BERTNOD

the ambiguity criterion as Q = .!P,Hz 1 N 11 CQ-I [1 C2

and the total criterion as C = ac, + (1 - a)C2 ,

. MINIMIZATION OF THE CRITERION

The problem at hand is to minimize the criterion C(p1, . . .,pN) subject to the linear constraints :

E ;-k) pd =1

~~P, - q,II2 (ii) The ambiguity of a stochastic labeling . It can very simply be measured by :

Y

H .=N -

I1p~Ui

where the entropies H, are defined by equation (2). it p, - q, ~J z has a minimum value of 0 and a maximum value equal to 2 when p, and q, are different unit vectors . H,, on the other hand, has a minimum value of 0 and a maximum value equal to (Q - 1)/Q when P,(J.k ) = 1/Q fork = I, . . .,Q. We can now define the consistency criterion over the whole set of objects as I N ( S) C,=2 A, 5 11 Pl-9i~~z,

i=1, . . .,N

(8)

k=1

Pd) k)>t0

i=1 k=1, . . .,Q .

(9)

Because of our choice of the Euclidean norm, C is a quadratic criterion. Let us denote by x the NO = Mdimensional column vector obtained by concatenating the N Q-dimensional column vectors p pv . We then have : C(P,, . .,

They measure the importance we attribute to the consistency between objects a, and aj. Condition ( ) guarantees that the vectors q, are probability vectors. They are what we would expect to find for object a, given our limited contextual knowledge and the label probability vectors p j for objects aj in V 1. The ideal consistency defined by equation (1) is too stringent . A more realistic definition of a consistent stochastic labeling is to impose p, = q, for every object a, . We can then measure consistency for object a ; by the vector norm lip, - q,11 For computational tractability (although it is by no means a necessity) we have chosen the Euclidean vector norm II . 11 a . Consistency over the whole set of objects is thus simply measured by

( 7)

where a, between 0 and 1, weights the relative importance we assess to consistency versus ambiguity . Criterion C varies between 0 and 1 .

()

jay'

(6)

PV) = C(x) = XT Ax + c,

(10)

where A is an M x M symmetric matrix (A T = A), and c a constant . From equations (5), (6) and (7), it is easy to derive c = (I - x)

Q (11) Q-1

One of the most widely used techniques to minimize a nonlinear functional C(x) is the steepest descent procedure. It is an iterative procedure that can be described as follows . Given an initial point x the iterations are constructed according to an equation of the form : x,,, . l = x~ + p^

(12)

where u„ is the negative of the criterion's gradient computed at point x,' and p„ is a positive number which is determined as explained in Section . When the problem is linearly constrained this method can also be used . It then bears the name of the gradient projection method ."' The linear constraints of equations (8) and (9) define a bounded convex region R in an M-dimensional Euclidean space E,M . The gradient projection method is the steepest descent method where the vector u . in equation (12) is the result of applying a linear projection operator to the negative of the gradient (this operator is described in Appendix). We now proceed to describe the gradient projection method with some more detail and study its convergence properties . . CONVERGENCE PROPERTIES

Let us denote by P„ the linear projection operator at



Scene labeling : an optimization approach



point x, and by v, the negative of the gradient . W e then have : V,=P,v,+x,=a,+z,

(1 )

and : UT .

= 0.

Z,

(1 )

If we evaluate the criterion at point equation (12) we obtain : C(x,+ 1) = C(x.) -

x,+,

defined by

t - x .) + (x . .

I

- x,)TA(x,+t - x.).

(15)

This is simply the Taylor series expansion of C(x, + , ) taking into account the fact that C is quadratic . Using equations (12), (1 ) and (1 ), equation (15) can be rewritten as : C(x. .t) = C(x.) -

p.11u.11i + p:(u .Au.) .

1

points in R such that C(x, + ,) < C(x,). Since C(x) >_ 0 for all points x in R this sequence is convergent . In practice we are of course interested in converging on a finite number of interations . This happens when C(xN+ ,) = C(xn ) for some N . We have then reached a constrained minimum (or possibly a saddle point) for which the projected gradient q v is zero . The limit point x . depends essentially on the first point x, that is on the initial conditions . Indeed, the hypersurface of equation z = C(x) in the M + I dimensional space has many local minima because matrix A of equation (10) is not definitely positive in region R . In fact criterion C z has N minima, one at each vertex of region R . The projected gradient method will converge toward the local minimum closest to x 1 .

(16)

p. is a positive number chosen so that it minimizes C(x, + ,) with the condition that point x,+t lies inside the bounded convex region R mentioned in Section . Let p,,,, denote the largest positive value of p, such that x.,t lies inside R . Since we know from our definition of criterion C that it remains positive in R we have five possible cases for p,,,, as shown in Fig . 1 . If p,,, is the optimum value of p, we then have the following rule :

5 . COMPUTATIONAL ASPECTS

As seen previously we need to compute the gradient of criterion C(p p,) defined by equations (5), (6) and (7). It is readily shown that : aC _ o N

cPt(%)

~ e V, ( •t I J.k)[P,vt) -

u,r Au, :- 0 p .,, = p ., .x

(1 - a) N

u, Au. > 0 p,,, = Min { p,,,,,

I

ay

[PI(J•k)-91(J.k)] -

Pi; °,

q,G•, )]}

2Q Q - 1 P`(J.k)

I1U.112

2u, Aa,

i=1 N

Thus, starting from a point x, in region R, this algorithm will provide us with a sequence x, (n >- 1) of

k=1, . . .,Q,

where q,().k ) is given by equation ( ) . Let us rewrite equation ( ) in vector form :

C fi . .J

P„

P

PMA

(b)

P

P„ (c)

Fig.

I.

(d)

Five possible positions of p_, (a) and (b) correspond to, A, > 0 .

to uT,Au,

< 0, (c), (d) and (e) correspond

(17)



2

0 .FAUGERAS

qt=

and M.

BERTHOD

(18)

Atjpj

where A ;j is a Q x Q matrix whose (k, 1) element is equal to Equation (17) can now be rewritten as :

ac

__ ar

r P. - 9 . - F Ajs(Pi - qi) NL jell

apt

_ 2Q(1-a) N(Q - 1)

P1

(19)

We have split our M-dimensional gradient vector (M = NQ) into N Q-dimensional `local' gradient vectors which we can attach to each object at . Let v, = -DC/5p, ; in terms of our stochastic labeling equation (1 ) rewrites as :

W O

=

pI'°

+ P"P,"'A"

,

i = I__, N,

(20)

+t, are the probability vectors for where p ;" , and J" object at at steps n and n + 1 of the steepest descent algorithm, respectively . P;"' is the projection operator defined in the Appendix at point p;"' of the closed convex region R t defined by : Q Pt(,, = I

0 Pf(J.k) >_

k = 1, . . .,Q.

k=1

The vector v" of Section is simply the column concatenation of the vectors v( "' for i = I, . . ., N. This formulation will allow us to study the amount of parallelism that can be introduced in solving our Mdimensional problem . Indeed, we need to compute v, for every object a,. This is done by first computing q„ i = 1, . . ., N, which can be done in parallel . As shown in equation (19) we

Fig . 2, Picture to be labeled . Possible interpretations are : triangle floating above background ; triangular hole in background ; triangular flaps folded upward, triangular flaps folded downward .

6 . EXAMPLES

6 .1 . Toy triangle example Let us apply our algorithm to the toy triangle example.tt .r ' In this example, each side of a triangle is to be labeled : J., = occluding edge, forward object above 1. Z = occluding edge, forward object below 2 = convex fold 2 = concave fold . The eight possible cases are shown in Fig. 2 . As pointed out in,"' there are at least two ways of .,) de.& choosing the conditional probabilities p,,() pending upon whether we assume that pairs are equally likely (p,i(i,,,).,) = 1/6 for all k,l) or whether each semantically possible triangle is equally likely . In the first case we obtain"' :

A,

=

EPii 1 (AkP1)] =

also need to compute

0

1

0-

0

i

0

1

?

0

0

0

1

0

0-

0 w, _ Y A (P, - ql) E V,

for i = 1 N . To do this we have to wait until all vectors q, have been computed and then we can compute the vectors w ; in parallel . We thus require two sequential passes over the set of objects to compute the gradient . But within each pass, operations can be performed in parallel . Once v ;"' has been computed for every object a 1, the projection operators P;"' have to be determined . This again, can be done in parallel as well as determining the maximum values p i .., for which p 1 ` 1 still belongs to R, . We then have :

In order to compute p", as shown in equation (16), we need to compute 11 u" 112' and u, Au" . From equation (10) we derive : u„r Au" = C(u") - c . This means that we have one criterion to compute at every iteration in order to choose p" in an optimal way .

and in the second case : 1

0

0

1

0

0

0

i

0

0

0 0 Az

=

[AU, 1O.k d,)] = 0

Results are shown in Figs

and

. In Fig.

we used

matrix A, to compute the vectors q t of equation ( ) whereas in Fig . we used matrix A 2 . In both cases, there are eight sets of initial probabilities, the same as in ."' We tried different values of the weighting coefficient a in equation (7) . We made it vary from 0 (pure entropy criterion) to 1 (pure consistency criterion) by steps of 0 .2. Every block in Figs and corresponds to a given value of a and initial probabilities . Inside the block are shown the initial value of the criterion, the resulting probabilities when the algorithm has converged, the number of iterations as well as the final value of the criterion . For a = 1 we notice that we converge toward two different distributions of probabilities which are exactly consistent, that is verify p;=q, for i=1, 2, For



Scene labeling : an optimization approach mtm1 conaltlons .

a=p .2

n=0 1

0.811

0 .25 0.25 0 .25 025 0 .25 0.25 0.25025 0 .5 05 0 .250 .250 .250250250 .250 .250.2 5 0,5 05 0 250 .250 .250250 .250 .250 .250 .2 5 0.5 O .5 0

1

0 .5 05 0 .5

0 05 0 0 0 .5 0 0 0 .5 0

0 .5 0 0.5 0 0 .5 0 0

0 .5

0 0 .5 0 0 0 .6 0 0 0 .5 0

0 .5 0 0 .5

0 0 0

1

0 .5 0 0 .5 0 0 .5 0 0667

0 0 0 0 0 O 0 0 0 2 .22 . 10-2

1

0 0 0

0.

2

0 0 .5 0 0 0 7 0 0 05 0

05 0 0 0 0 .5 0

1

0 .5 0.

0 0 0

1 0 I 2

0 .595 0 0 07 0 0 . 0 07 0 0 .5 0 0 .5 0

0 .2 0. 0 .5

0. 0. 0

0 0 .8 0 0 .7 0 0 .5

0 0 0

0 0 0 5

0 0. 0 .5 2

0 .20 . 0 .2 0 .5 0 .2 0 . 0.2 0 .5 0 .20 . 0 .2 0 .5 1

I 1

. 16'

2

0 0 0 0 0 0 0 0 0 5 .10 2

0 0 0 6 .67

I 0 I

0 0 0

75

2

0 .5 0

0 .551 0 1 0 0 1 0 0 0 .5 0 0 .222

0 0 1

0 .505 1 0 0 1 0 0 0 0 0 9

0 1 1

0 .987 0 0 .5 0 0 0 .5 0 0 0 .5 0 0667

1 1 1

0 .801 0 0 0 0 0 0 0 0 0 2

10 1 0 1 0

9

I I I

0 0 0 0 0 0 0 0 0

2

5 .10 2 0

2

5 .55 . 10 2

0 .17

0 0 0 0 0 0 .10 -2

0 .9 0 0 .9 0 0 .9 0 1

0 .1 0 0 .1 0 0 .1 0

a .10 2

1 .2

075 0 0 .75 0 0 .75 0 1

0 .2 7 0 .9 0 0 .9 0 0 .9 0

0 0 0 0 0 0 0 0 0 6 .67x10 2

2

0. 6

0 .27

0 0 0 0 .9 0 0 0 0 .9 0 . 0 .9 0 0 6 2

0 0 .1 0 0 0 .1 0 0 0 .1 0

0 .599

0.

0 .75 0 .75 0 .75

0 0 0

0 .25 0 0 .25 0 0 .25 0 9 . x10

1

0 267

1

6

2

ax10' 2

0 0 0

0 .75 0 0 .75 0 0 .75 0

0 0 .1 0 0 0 .1 0 0 0 .1 0 8x10 2

0 .25 0 0 .25 0 0 25 0 9 x10 1

0 . 66

0 . 29

0 25 0 0 .25 0 0 .25 0 1 9 . x16

0 .75 0 0 .75 0 0 .75 0

6 . 10,2

2

0 .9 0 .9 0 .9

0 .25 0 0 .25 0 0 .25 0 9 . e1o 1

0 18

0 .1 0 .1 0.1

0 0 0 0 0 0 0 0 0 6

.16"

0 .1

0 .1 0 O.1 0 0 .1 0 8 .10 2

0 .9 0 0 .9 0 0 .9 0

2

I 0 .111

0 0 0 0 0 0 0 0 0

0 .569

0 0 .75 0 .75 0 .75

0 0 0

19 0 .25 0 025 0 0 25 0 9 .10 1

5 .78 . 10-2

0 .2

0 0 0 0 0 0 5 .10 2

0 0 0 0 0 0 0 0 0 6 .67x10 2

0 .9 0 0 .1 0 0 .9 0 0 . 1 0 0 .9 0 0 .1 0 8,10 2

0 . 75 0 . 75 0 125 0,25 0 . 75 0. 75 0 .125 0125 0 750. 750 .1250.125 2 1 .2x10 12

1

0 0 0 0 0 0 0 0 0 0 0

0 .260 0 .9 0 0 1 0 0 .9 0 0.1 0 0 9 0 0. 1 0

0. 75 0 . 75 0 .1250125 0 750 750.1250 .125 0 75 0 . 75 0 .125 0 .125

2

8

6 .67210

1 0

2

9

7 .5x10

1

0.2 1

0 .222

0 12

0 .576 0 0 0 0 0 0 0 1 0

x10

2

7

0 I 0 0 0 0 0 0 0 '7 .5x10 2

0 .802 ' 0 0 0 0 0 0 0 I 0 .

1

0. 8

0 I 0 0 0 0 0 0 0 9 .26 .10 2

0 .555

0. 0 1 0 0 0 . 1 62

n=1

1

0 . 50

0 0 0 0 0 0 0 0 0 5 .10 2

0 1 1 2

I 1 0

1 1

0. 0

0 0 0 0 1 0 0 0 0 . .10

0 . 00 0.

0 . 27 1 1 I

0 0 0 0 1 0 0 0 0 .

I

0. 17

0 0 C 1 0 0 0 .5 0 0 .222

0 .982 0. 0 .2 0 . 0 .2 C .5 0 0 . 5 0 0.250 .250 .250 .250 .250 .250 .25025 0 . 2 0 2 0 . 0 .2 0 0 1 0 1

I 1

0 .5 2

0 .5

0.

a=09

0.505 0 0 0 5 0 5 0 .05 0.W 0 . 75 0 . 75 0 .1250 125 0.505 0 0 0. 50 50 .050.0`0 . 750. 750 .1250 .125 0 .505 0 0 0 50 5 0050MO 750 . 75 0 .1250.125

0 . 92

0 .555

05 0 1 0 0 .5

0 .6 1 0 .5 0. 0 .5

I

0 .5555

0 .656 0.5 0 0 .5

0 .575 0 0 0 .5 0 .5 0 0 0 0 0 .505 0 0 0 0 0 .5 0 .5 0 0

1

0 .667

a=0 .6

=OAS

2

9

8x10 2

7 .92 x 10 _2

7

1 .2 .10 12

Fig. . Results for the toy triangle example using matrix A, .

matrix A, we find and [, 0 10]' for every object. For matrix A,, we find [}, }, }, }]T and ['j 0 n 0]T for every object . In both cases the first vector is the label a priori probabilities, whereas the second vector is labels l., and 1. a priori probabilities given the fact that labels %, and 2 do not occur. When a decreases we begin to take ambiguity into account. We notice a shift of the minima toward distributions with less entropy . For a = 0.6 for matrix A, and a = 0. 5 for matrix A, the behaviour of the algorithm is the same as described in"' except for the initial condition [0 .25 0.25 0 .25 0 .25]T where it converges toward [0 .5 0 .5 00]T which is precisely the `NoInformation' fixed point referred to in . (51 Moreover the

speed of convergence is much higher than in ( " and higher than in . 161 6.2 . Edge detection A more realistic application is the detection of edges in a greyscale image ." 8 .9) One fairly standard method for doing this consists in colvolving an input image I(k,I) with n masks M,(k, 1) . . .M (k, 1) . The masks M ilk, 1), i = 1, . . .,n are differential operators which respond strongly to edges in n particular oriented directions . The results of the n convolutions are n images I I .. At every point in the image we now have an n-dimensional edge vector . The norms (Euclidean, maximum, etc . . . .) of these vectors are then



0 .FAUGERAS and M . BERTHOD

_

I .M .1 cm tt ni,

°

° C 1

0.2

0758

0 .612

0 25 025 0.25025 025 025 0.25025 05 0,5 0 0.250 .25025 :25 .250 .250.250.250 .50 .5 0 0 .25 0.25 025025 025 0 .25 0 .25025 0 .5 0 .5 0 0

0 0.50 0 0 .50 0 0 .50

0 .5 0 .5 05 0

0 5 0 0 0 0 .5 0

0 5 0 0 6 0 0 5 0

0 5 0 0 .5 1

0 .5 0 0 .5

1 1

0 .667

I

1

1 0 1

0.

2

0 0 1

0 0 0.5 1

0 .6 0

0

0 0 0 .5 0

0 7 0 0 .5 0

0 0.5 2

1

0 1

C I 0 0 0,5 0

1 0 1

,12 . 10'

2

0 0 1

2 9 . 7 210 2

2

0 78 0 0 1

0 .222

0

0

1 0 0 0

C 1

0 0. 5 0 0 0 .5 0 0 0. 5 0

10 1 0 1 0

0 .667

0 .555

0 0 9

1 i

0 0 0 0 1 0

5 .16'

2

0

.12x10 2

0 0 0 1 0 0

0 0 0

1 0 1

5210 2

2

0 0 0 0

1 1 1

0 .15

0 0 0

0 1 0

0 09 1

0

0

I 0

0 0

1 1

0 0 0 0

0 0 0

0 0 0

0 .09 1

0 0 0 0

0 .59 0.59 1

0 .09

0 0

1 1

9 12 : 10 -2 0 0 0

06667 0 06667 0 06667 0

0 091 0

0 0 0

0 0 0

0 0 0

0 .517 0 0 .91 0 0 0 0

5 .15 16 1°

06667 0 0 .6667 0 06667 0

0 0

5 .15,10'°

0 .6667 0 06667 0 06667 0

0

0 1

9

0

7 .5x10 2

0

0 0 5 .15x101

0 . 0 O166701667 0, 0, 0.16670 1667 0. 0. OK67 01667 55210 15 5 .21 210 2

1

0 0

0 0

0 0

0.09

0

0 .91

0

12

0 0 0

0 .9 25 06667 0 0. 0 .6667 0 0 06667 0 0.

0 .268

0

0 0. 0.

.25 210 2 0 0 0 0 .112

1

0 1. 0

5 .15 2 10 1

0

0 .26 2 0.26 2 0 .26 2

0 .150

52102

0. 0. 0

9 .9210 2

0 0 0

C 0 0

0 207

0 1 0. 1 0

2

0 0 0 07 68 0 0 0 07 68 0 0 0 0 .7 68

0 0 0 0 0 0

0 0. 0

0 .127

0 .22

1 1

0 1

0 0 0

0 0 0

5 .15 10= °

0 .109

0 .12 0 0

1

9 .9 10 2

0

7 .5210 2

10 2

9 .9x10 2

0. 1

1

.55,10 "5

0 .228

0150

1

1

1

0 .255

0 0 0 0 0 0

0 76 v 1

0 0 0

0 .20 0 0 0 0 .91 0 0

75 :10 2

2

0 610

9

1

01667 01667 016670IM? 01667 01667

06667 0 0 . 06667 0 0. 06667 0 0.

0 112

0. 6

0 1 1 0 0 0

0.26 2 0 0 .26 2 0 0 .26 2 0

7.5210 2

0100

0

0 0 0

0 0 0

6 .25

0 0 0

1

0 . 29

C 0 0 0 0 0

1 1

0 .189 0 .15

0 .150

015

0 0 0 0 0 0

1

0 0 0 07 68 0 0 0 07 68 0 0 0 0 .7 68

0 .605

0 .750 1 0 1 0

1

a= 125210 2

0 01 K 01 16 0 0 6" 0 6M 01 1601 16 OA5 0 68 0 68 0 .1 1$ 01 16 0.

0 . 17

0 0 0 1 1 0

0 0 0 0

0 0 0

0. 1

1 0 1

6 .25x10 2

0 .982

1

0 0 0

9 . 7x10 2 0 .7 8

0 .225

0. 0 0 0 0

0 1 0 0 0 1 0 0

0.

1 1 0

°=C8

0 . 19

0 .100

0.

0 0 0 C 1 0 0 C 0

0 0 1 0

0 . 0.2 0 0 .2 0 .5 0 0 .5 0 0 .2502502502502`-0.25025025 0 .2 0 .2 0 0 2 0 0 1 0 1

2

0 0 0

-0987 0, 0 . 2 0 . 0 . 2 0 .5 0 . 0,2 0 0 .5 0 . 0 . 2 0. C .2 0 .5

.122102

L 0 0 0 0

0 .222

0 .2 0

1 0 1

0 . 96

0 1 0 0 1 0 0 F5 0

0 .551 0 1

1

a=0 .6

0 .50 .5 0 050.5 0 0505 0

25

0 0 0 0 0 C

0 .516 0 0 C 1 0 0 0 0 .•

2

0 .595 0 .7 0 0 .7 0 0 .5 0

0 1 1 I

0 0 0 0 0 0 0 0 0

0 505

0 0 .5 o 0 1 0 0 0 .5 0

0 0 0

0 . 50

6.-5u0 2 )

0 .656 C 0 .5 0 0 0 0 0 5 0

0.

0. 0 0. 0 0 .5 0

0 .5 0 .5 0 0505 0 0 .5 0 .5 0

0 516

0 0 5 0 0 0 .5 C 0 0 .5 0

0 .6 1 0 .5 0 0 .5 0 0 0 0 7 0 0 .5 0 0 5 0

0 0 0

0 .5 1 0 667

0 5 0 .5 0=_

1,

9 .9210 2

0. 0, 0.

0.

016670 .1667

0

01667 01667

10

.55 10 1$

Fig. . Results for the toy triangle example using matrix A 2 .

computed and form an indication of the likelihood of finding an edge at a given point . These values are then

Mt=

L1

-0

-I

0

-1

compared to a threshold (constant or adaptive) and

M2= L1 0 -1)

the decision edge/no edge is taken . This is summarized in Fig . 5 . The well-known disadvantage of this class of tech-

to detect horizontal and vertical edges . If we orient

nique is the difficulty of eliminating spurious edges in

edges in such a manner that dark is on the right when

busy areas while keeping the main contour edges

moving in the direction of the edge, M t and M2 will

connected or even present!

respond positively for edges pointing to the East and

The labeling algorithm can be used very successfully to alleviate this problem . In the examples to be

North, respectively and negatively for edges pointing to the West and South, respectively .

presented we used n = 2 namely we considered four

From the results 1, and 1 2 of the convolution of

oriented possible directions at every pixel, North, East,

image I with the masks M t and M 2 we can construct

South and West . We used two simple masks,

four images J t , J2 , J

and J

such that :



5

Scene labeling : an optimization approach Western edge Southern edge i5

Point

Each object a, has four neighbours in V ;, namely the four nearest neighbours of the pixel, numbered from 1 to :

G(k,0

(norm)

No edge

SOn

I

We then have the following conditional probabilities (matrix A; 1 ) :

T(kd)

.I Fig . 5 . Classical edge detection technic .

J,(k,1)=J11(k,1) if 1,(k,l)>0 if 1,(k,1) <_ 0 l0 Jz(k,l)

J (k,l)

1,(k,1)

_

f =

J,(k t) -

0 7 z (k,1)

10 -Iz (k,!) 0

if 1,(k,1)<0 if 1,(k,1)>-0 if I2 (k,l)>0

if Iz ( k,l) <_ 0 if 12(k, 1) < 0 if

2 (k, 1) >- 0

We are now ready to jump into the stochastic labeling formalism . Objects are pixels. There are five possible labels for every object :

--*1 0.2 25

I I

0

-*1 0 .2 25

i-'

11

1-

11

0.

0 . 025

0.

0

0.

~---k 0

-1

-F

0 .2 25

0.

-1 0 .2 25

0

1-'

11

1

0

1, 0.

11

0.

0 . 025

0.

0.

ti 0

Fig .6 . Iterations ofthelabeling algorithm onthestar picture . Upper left hand : original . Upper right hand : initial probabilities. Lower left hand : after 5 iterations . Lower right hand : after 10 iterations .

iC

of

t 0 .2 25

0

0.2 25

0.

Matrices A t,, A 1 and A„ are rotated versions of A ; 1 . The initial probabilities are given by : Jm

p

A 1 -* Eastern edge i. z 1 Northern edge

0

i <_ m 5

T

PPS)

= D

Fig . 7 . Iterations of the labeling algorithm on the wheels picture. Upper left hand : original . Upper right hand : initial probabilities . Lower left hand : after 5 iterations . Lower right hand : after 10 iterations .



6

O . FAUGERAs and M . BERTHOD

where D = J, + Jz + J + J ` + T, and T the threshold at pixel i. Figures 6 and 7 show the results on two different images . 'The star' is of size 18 x 16, 'the wheels' is of size 6 x 6 . In the first case it can be seen how nicely the shape is reconstructed by the labeling process in a few iterations. In the second case the structure in the upper part of the picture is also nicely reconstructed while very few spurious edges are added in the lower part of the picture which is a textured grass area. 7. CONCLUSIONS A practically useful stochastic labeling has certainly to be unambiguous. It also has to be consistent with the a priori knowledge that we may have about the set of possible labelings . Unfortunately, because of noisy measurements and imperfect feature extraction algorithms we have to deal with stochastic labelings which are ambiguous and inconsistent . Thus the need to design methods that will provide ways of decreasing ambiguity and increasing consistency of an initial or 'raw' stochastic labeling . The linear algorithm defined in"' only attempts to increase consistency . As shown by Rosenfeld et al. and in this paper it converges toward consistent labelings which are ambiguous. This illustrates what in our opinion is the key point of the stochastic labeling problem namely that consistency and unambiguity are in most cases somewhat contradictory . Our definition of a quantitative criterion based upon explicit measures of both ambiguity and consistency allows us to solve that problem in a reasonable way . We can trade ambiguity for consistency and vice versa through the parameter r of equation (7) . Moreover, the introduction of a quantitative criterion permits us to define in a natural way an iterative algorithm that guarantees that this criterion decreases from one iteration to the next . In terms of computational complexity, the algorithm is more complex than the non-linear one proposed by Rosenfeld et al . since it requires two sequential passes over the set of objects instead of one, each pass being of a similar complexity and completely parallel . This should be compensated for by the fact that our approach guarantees convergence toward a stochastic labeling corresponding to a smaller value of the criterion .

REFERENCES

I . A . Rosenfeld. R . A . Hummel and S. W . Zucker, Scene labeling by relaxation operations . IEEE Traits . Systems . Man . Cybernet. SMC-6, 20(1976). 2 . S. W. Zucker, R . A . Hummel and A. Rosenfeld, An application of relaxation labeling to line and curve enhancement. IEEE Trans. Comput . C-26, 9 - 02 (1977) .

. D . G, Luenberger, Optimization by Vector Space methods, John Wiley, New York (1969). . J . B . Rosen, The gradient projection method for nonlinear programming . Part I . Linear constraints, J. Soc. ind . app! . Math . 8, 181-217 (1960). 5. S . W. Zucker and J . L . Mohamed, Analysis of probabilistic relaxation labeling processes . Proc . 1978 IEEE Computer Soc . Conf. on Pattern Recognition and Image Processing, pp. 07- 12 (1978) . 6. S . W . Zucker, F . V. Krishnamuthy and R . L . Haar, Relaxation processes for scene labeling : convergence, speed and stability, IEEE Trans . Systems. . fan, Cybernet . SMC-8, 1- 8 (1978) . 7. S . W . Zucker, R . A . Hummel and A. Rosenfeld, An application of relaxation labeling to line and curve enhancement, IEEE Trans. Comput . C-26, 9 - 02 (1977). 8 . B. Schachter, A . Lev, S . W. Zucker and A. Rosenfeld, An application of relaxation methods to edge reinforcement . Computer Science Technical Report No . 76, MCS-720 610 . University of Maryland (1976). 9 . A . R . Hanson, E . M . Riseman, Segmentation of natural scenes . In Computer Vision Systems, pp. 129-16 , Academic Press, New York (1978) . APPENDIX

We describe how the projection operator p,"' is obtained at point pl"' . We will drop the index i and the upper index n which are irrelevant to what follows . If no coordinate of p is zero, P is simply the projection S on the hyperplane of equation :

-1

that is : Sx=x=1v where : x = [x,, . . .,xQ1 r V

= [1, . . .,!] r

X

= -

1 v X,, .

If some components of p are equal to zero, the situation is slightly more complex . Let K be the set of indexes for which pod,) = 0 . We compute the projection w of the negative of the gradient v on the intersection of hyperplanes : Q

y p(A,)=1 p(1.,)=0

for keK.

(21)

=1

Let SK be the corresponding projection operator . If w is not zero we take P = S, that is we do not attempt to desaturate any of the constraints p(i.,) _ 0. If w is zero there is no direction of descent in the intersection of hyperplanes defined by (21) and we look for the index I e K for which the projection w of v on the intersection of hyper planes : a p(i,„)=I p(1,)=0 for keK-{1}=K' r=' has the largest positive value for w, . Let Sxx be the corresponding projection operator . If there is no index I in K such that w, is positive there is no direction of descent at point p, we have reached a constrained minimum . If there is such an index I we take P = S, . . At the next iteration we will desaturate the constraint p(1,) = 0.



Scene labeling : an optimization approach

About the Author - OuvIER FAUGERAS was bom on December 22 19 9 in Paris, France . He graduated from the Ecole Polytechnique in 1970, the Ecole Nationale Superieure des Telecommunications in 1972 and got a PhD . i n Computer Science from the University of Utah in 1976 . From 1976 to 1979 he was the head of the Image Processing Group at IRIA, France . He is presently an Assistant Professor in Electrical Engineering at the University of Southern California and conducting research at the Image Processing Institute . His research interests are in Signal Processing, Computer Graphics, Pattern Recognition, Image Processing and Understanding . Dr. Faugeras is a member of EURASIP (European Association for Signal Processing), IEEE and ACM. He is also on the Editorial Board of Signal Processing . About the Author-MARC BERTNOD graduated from Ecole Polytechnique (Paris) in 1969 . After two years as a consultant in French "Compagnie Generale d'Informatique", he has been devoted to research since 1972 in IRIA (Institut de Recherche en Informatique et Automatique) . He passed a rd cycle thesis on On-line Character Recognition in 1975. His main interests are on Character Recognition, non-statistical methods in Pattern Recognition, and context in the field of Pattern Recognition and Scene Analysis .

7