NOTE Shape
Completion
This report describes some simple tcchniqrlc,s for slnoothly filling ill gaps in object contours. The first techniqrlr considered was rccent1.v proposed by Ullman; it constrnct,s the completion of the contorlr using two arks of ckclcs that arc tangent to the gap ends :UKI to each other, and that have minimum total c~luvatu,r. An analysis of this trchniqrux is presented, and examples of its ,lsc arc givc,n. A second t,(~chniqllc IMPS cubic polynomial conlpletions; when suitably constraitlcd, this tcll.hniqllcs yicllds very rc~nsonabk~ cc)lilpl&ons. 1. I?u’T1:ol)I:(:‘lYoS
This report describes some simple techniques for filling in gaps in obj(sct contours. Such gaps occur in many classes of images as a result of occlusion, shadows, etc. Many levels of knowledge may 1)~brought to bear upon the problrm of filling in a missing scgmcnt of an object boundary. The techniques considtxd here utilize only local information. Specifically, WV are givc>n t,wo points through which t,hc contour must pass and th(h directions in which the contour is going at those points.
The lirst tcchniquc WC consider was proposed by ‘I’llman [I]. He suggests that. in order for a shape completion to have propertics of isotropy, smoothness, and minimum curv:aturc, it should be composed of the arcs of two circles, one tangent. t,o one edge, the other tangent, to the second edg(l, and such that the two arcs share a common tangent at their mc>cting point. Out of all such pairs of arcs, WC pick the pair which minimizes t’hc total curvaturcl, defined as the integral J’(dc~,‘dZ)~ over the curve, where LYis the slope of the curve. IJllm:tn goes on to describe a locally conncctcld n&work which computcls tlhcscs contours. In the abscncc of such a network, a conventional approach must 1~ employed to compute the contour. Kc now d(bscribt: such an approach. In Fig.
Ia w: hnvc
a. circul:lr
arc c~tnnnntin S!)
g from
point,
P in dirwtion
I-(, 2nd
WALLACE
S. liUTKOWSK1
,’
13
‘c: ni~:~su~d fr01ti th(b initial tcmiiuat,ing at point (2 in dirc&on po. ‘I’hft angl(b CY1. tliroction vector to a vector pointing from the initial point of th(l arc to thcb I cmninal point. Sotc that the total angular changch isP,fi? r/B) ia ?a. l’his holds true for a E C-r, +K]. WI, run into difficrlltp if we’ IIS(’ :I difY(mmt r:tngc~ for a. For in Fig. lb WV ham: 01 = --7 2, alid th(b arc goc7 through an :mgulxr cmTlpl~~, change, of L)(Y= -g. I1ow~~v~~r,if w(’ instcxtl tii(~:1surod (2 3s +:Ir 2, and constructod a circular arc with iot:d :mgul:~r ch:ingc, Ztr, w(’ obtain tlic, situation shown in Fig. Ic. 7’ho vc>ctor from th(s initial to thcb tc~rrninal point of the> arc now point,< in n dircxtion oppositct to thus tlircsc+ion tl(Lfincd IJ~ th(l :mgl(x :jx ‘1. Of (‘ours(‘, ii is obviously foolish to cori.~truc~t conipl~~tions u>ing circul:Lr arcs with IV(’ can rwtrict Lb 10 angular changchs cmc4ing L)7r in nl:lgnitu(l(~. ‘L’hc~rc~forc~ (,-2~. +Zrj, or LY E (-5~. $7~). for tlich :Ir( 2s usc~l in our contours. It is :i mattclr of ctlcmmt:iry g(~omc~try to +Iir~w tht for I;ig. 13
\\‘v can now d(bfinc our probl(~ni with ill(a ait of Fig. 5. ‘I’ho contour WC’wish to compute consist of two circular arcs which join at point .I. For angle ~1, 8,, and a1, associ:ztcld with the point I’,, l(bt us corlsidm countclrclockwisc rotation as posit’ivo, while for F2, 6?, and eq, WC consid(~r clockwiscx rotation positivcl. Although this may SWIM confusing at first, it lands to symrnc~try in the quution< dcvc~lopcd h&w. ‘I’hus in Fig. 21, d1. ql, Oz. +Q would 1~ posit iv<‘! and a1 and a2 would b(l nclgativc>. All mglcs arc in C-K, +x]. T,c+ us now consider tho total :mgul:lr chung~~ in l’ollowing thcl contour front I’, to I’,. In going from PI to point .I, th(l mgular chnngcl (considering countcbrclockwise> as positivcl) is %I~, as discu,+cd cL:trlicsrin connmtion with Fig. 1. .1s WC go from J to I’,, the angul:lr chmgc~ [:tg:l,in considering conntclrclockwis(~ as positive) is ~cx:~..It first this may s(~‘m puzzling, sincc~ a2 is rncxsurcd considering clockwisck rotation as positivcb. Intl~~cd. th(a countc~rclockwisc~ rotation would b(l - 2a2 if WV wm’ procmading from I’, to J ; how(~v(~r, sina, w(’ arc moving froril .I to I’,, the> countc,rclock\vi~(, rotxtion is ~)a,. ‘l’hw the tot:11 nngulnr chnqq. travclling from I’, to I’?, is ?a1 + ?a?, whcbr(s tlic> nngulnr chnngc is nicasurcd according to the, convcbntions usicd :tt I’,. 1’h(~ total angular chmgcl from I’, to I-‘, is also givcm by - qI - (-? (again tno:muml with rcxspcbctto the convcmtions at PI). For txtmplc, in Fig. 2b t hcbtotal avcbr:lgcLchange from /jl to I’, is -8. ‘I’hus wr h:tvcb 20~ + ?a2 = - ( c1 + +). ( )f course’. w(s cm ndd or subtract multipl(~s of ‘IT to t.hct total angular ch:tngcL r(qllir(s(l to go from 1’1 to I’,. so that in gmcml
92
WALLACE
m
AVG
S. RUTKOWSKI
= n/2
a ‘AVG
= n/4
b ‘AVG
=’ /
---e1 = ?T,
Pl
e1 + e2 = mAVG
=
’
----_ P2
e1 = e2 = 0
e1 = 0, e2 = fil
e2 = 0 c FIGURE
3
entire technique breaks down, as there is obviously no pair of circular arcs which will join PI and P2 as required. Hence, +AvG E (-r, +?r) for any situation in which it is possible to construct the contour. Now if we wish to form a triangle PI J Pz as in Fig. la, t,hen 1e1 + & 1 < ?r. Thus: 12 can equal $1 only if @~vo 5 0, n can equal - 1 only if @Eve 2 0, [n 1 can never exceed 1. To summarize, 01 and & obviously d&ermine the point J at’ which the circular arcs join, and we have now derived a constraint on e1 and &. It is a constraint which has a simple geometric interpretation. The locus of points J for which e1 + & = @~vo + n?r, with n restricted as given above, is a circle. For example, in Fig. 3 we show cases where @~vo = 7r/2, ?r/4, and 0. The last of these cases is unusual in that we can only form degenerate straight line triangles. Although
is possible lo compute: corltour~ using the c*onslr:tillt, 0, + II2 = +A\,(: + )(TT wit,h nonzero )I, we can get into serious difticulti(bs when doing so. Kotc that WC’ art: attempting to find a contour which fits thts above constraint and has minimull curvaiurc. ?rJow consider the case shown in Fig. 4:~ If WC look at the family of completions defined by 19~+ OS= @A\.(: = 0, wo SW that they look as shown in Fig. Ah, and the minimum curvature contour obeying th(l constraint O1+ O2 = 0 is as shown in Fig. 4c, as we would cq~ct from symmetry. Sow, howc~~sr. consider t,hc family of contours obeying thr constraint O1+ B2 = +a. Th(Bscs contours look as shown in Fig. 4d. ‘l’h(b rurvaturck of :I circlrhtr arc is il.
WALLACE
S. RUTKOWSKI
a '1
+ '2
= 'A"G
t
FIGURE
5
where 2a! is the angular change along the arc, and R is the radius of curvature. Note that in the family of cont’ours illustrated in Fig. 4d, a1 and (Y:!remain fixed at 3r/2, while t’he radii can increase without bound. Thus there is no finite solut’ion in the case @AVG = 0, or cp1 = - cpz.This special symmetric case with cpl = cpz is not the only problem. Consider the “almost symmetric case” in Fig. 5a. The locus of points at which the arcs can join is illustrated in Fig. 5b. If we examine the family of contours obeying 81 + 81 = @AvG, we get a minimum solution which is “almost symmetric” (Fig. 5~). However, there exist solutions obeying 01 + 02 = @AVG- s which have lower curvature than the curve above (assuming @AVGclose enough to 0), as in Fig. 5d. Although in this case @A~G# 0, so we can find a finite minimum curvature contour obeying e1 + & = +AvG = -r, we find that the shape of the contour is not “almost symmetric” as we would intuitively like it to be. In fact, the closer WC approach the symmetric case +q = - ‘pz, t’he larger tho minimum contour obeying 81 + es = @AVG- 7r grows.
,~ll.\I’I~: (‘o\I1’l,I~:‘l’Ios
! ).i
- cpd sill (01 - 0) + if%--(4-- -._____ -~-sin e2 (-aDIF
-
02)
sin
(-aDIF
=
(-@DIP
-
82)(Sin +
=
(-@DIF
-
-
62)
+
(&IF
(-a
(%IF
-
sin
(@DIF
DIF ) cos el)
(sin
(-02)
+
cos (-81)
$
81)
e1
(-aDIF)
COS
(@DIF>
-
sin
(-e2))/sh
COS
02
Sin(-L%))/Sin
(@Dw)
81
sin @DIF cot 82 - cos @DE?)
e2) (-
!bIF[(+DIF
81)
sin
+
= sin
-
e2
sin =
(~2)sin (0, - ~3 -sin e1
+
82)
cot
82 +
(aDIF
(@DIF
-
01)
cot
-
01) (sin el]
+
@DIF
cot
81 -
@.AVG
cos
@DIF.
cos
@DIF)
The total curvature can thus be written as (4 sin @AvG/D) times the above expression. If, for the moment, we ignore the special cases in which QjAvo and/or @DIF are zero, we see that this function goes to infinity as either e1or & approach zero. Since the minimum curvature obviously does not occur at either end of the range in which 01 and e2 must lie, we must look for a local minimum. The derivative of the above expression is 4 sin a.4~~ sin &IIF
d -&
D
[&IF
+
02)
cot
82 +
(aIF
-
01)
cot
811.
1
Since we are assuming that +Avo and @J DIF are nonzero, 4 sin +~vo sin +DIF/D is nonzero, and we must have fJ
[(%IF
+
e2)
cot
e2 +
(+DIF
1
-
e,) cot, e,] = 0.
Since 01 + 02 = @.AVG,cB2/dOl = -1, and we must obtain @DIF
+
siriV2 Multiplying
82
-@DIF +
through by sin
sin e1 (%IF+ e2)sin +
+
e1
sin2B1
e1sin
e2 gives sin
(-@DIF
+
- cote1- cote2= 0.
01)
e2
sin
2
-
cos e1sin
e2-
cos e2sin e1 = 0,
1
&IF
+
82)
sin
e1
-sin
e2
+
sin (-@DIF
+
01)
__
sin
e2 e1 -
sin @~vo = 0.
With the derivative in this form, we can see that as e1--P 0, the derivative goes to-infinity with the same sign as -aDIF, and as e2---) 0, the derivative goes to infinity with the same sign as +D~F. (Remember that e1 and e2 must have the same
sign, so sin 8,,‘sin 02 and sin &‘sin 01 > 0). Sinccl w know that 61 and & must 1~ between 0 and @A~G,and WC know the behavior of the dcrivat,ivc at thcsc points, it, is easy to solve the above equation numerically by bisection. We still have two special CRI;W to take cart’ of. If @nlp = 0, then Q = Q. Jn this cast, the contour is just a singhI circular arc. This can, of course, bc vicwcd as two circular arcs of equal radius joined at any point we choose along t,hc contJour. Kote that the derivative is zero, as expectc>d, since the contour and curvature remain the same regardltbss of whcrc we choose to join the arcs. If +fiDIr is nonzcro, WC must still have @.AVC; = 0. In this cast p1 = - p2. Sincfl BI and 19~must lie between 0 and (PAvC:!$1 = & = 0, so the arcs must join at. somcb point on a lint between PI and P, (Fig. 6). The curvature> is ~1 sin CFI - q1 sin -- (=1 -.___ +----~--.-- ~._ = qlsi*l vl(i Dl IL!
+-.!l),
wh(lrc D, + Dz = D, and D1 and 17, must be bctwcen 0 and 11. It is not difficult, to show that the minimum value occurs at II1 = I).’ = ILL?!. (Soto that since @DIP # 0, ~1 sin ~1 # 0.) WC thus obtain t’hc result we would expect due to symm&ry. Wc could also a’rrive at the last result by examining the behavior of the expressions we derived for t’he curvature and its dcrivntivc as p1 -+ - q2, although this approach is mor(h complicated.
We can take another approach, using polynomials. Suppose wr havcl two points P1 and P,, and two vectors VI and v?! as in the diagram below.
We can define a curve (CC(~),y(t)) which passes t,hrough point PI with vector derivative ~1, xnd passes through point P2 with vclctor derivative vq in the following way: r.ot
WATLACJC
S. EUTKOWSKI
m
+,=r/2 b
c
&=”
100
WALLACE
S. RUTKO
WSKI
We can satisfy thr above conditions by defining 3-(t) = ‘4 JR + n,t2 + c,t + I),, y(t) = A,t” + But2 + C,t + D,, where A, = WI, - 2Pzx + VIZ + vzy, B, = 3P2, - 3P1, - 2Vl, - vpz, c,
=
Viz,
D,
=
PI.,
A,
=
2P1,
-
=a,
+
vlu
+
vzu,
B, = 3P,, - 3P, - 20, - vzzI, c,
=
%I,
D, = PI,. Of course, since WC are really only interested in matching the directions of VI and v2 at PI and Px, the magnitudes of v1 and v2 are free parameters. Note that the solution curves are invariant with respect to rotation and translation. This follows from the following two properties of the solution curves generated by Ad3 + B,t2 + C,t + D, (20) Y(t)
= ( A,t3 + B,t2 + Cut + D, )* *
(1) This class of curves is obviously closed under rotation and translation. (2) Any pair of points and vectors, PI, P2, vl, v2 generates a unique solution curve. In other words, suppose we have points PI, P2 and vectors IQ, v2, and the corresponding solution curve is (z(t), y(t)) . Th en if we apply a rotation and translation to PI, P2, ~1, v2 to get P’l, P’z, ~‘1, V’Z,clearly the curve (x(t), y(t)) taken through the same rotation and translation will yield a solution curve to the problem with ~‘1, v12. But since P’l, P’2, ~‘1, vt2 determine a unique solution curve P’l, P’z, W(t), y’(t)>, th is curve must be a rotated and translated version of (x(t), y(t)). To implement this method, it was of course necessary to devise a means of determining the two free parameters (the magnitudes of the vectors vl and v2). The method used to determine these parameters is based on the following observation. Consider the figure below : I
L is the perpendicular biscct,or of line segment PrP2, and C is a circular arc. 11 is easily shown that a polynomial based completion, gcncrated by the technique clescribcd previously, will pa,ss through point .r (the 1 wamcbpoint, through which a circular cornplct~ion past) when IVli /=I
;vq I --
e..-L”‘I + cos
e’
‘l’hc rcbsulting polynomial completion clost~ly approsimates a circular completion until p becomes much greater than ~j.‘. Therefore, to facilitate a comparison botwocn the two m&hods of generating completions, it was dccidcd to determine the magnitudes of v1 and v2 in such a way that in the symmetrical cast diagrammcd above /VI/ = iv?: = Ul,‘(l + c0s ~1. There are obviously many ways to do thi.5. The one used in the program is 211 1 = v1 I
/v2/
=
----._
1 + ((CM i’l + cos C&/L))
(see the figure: below)
i1 sot of examples comparing the two methods is shown in Figs. 7 and 8. In all of these examples, both ~‘1and cp2are multiples of r/4. It appears that, in many cases, the cubic polynomial completions have a more pleasing appearance than the two-circle completions.
The support of the Drcctorate of Mathematical and Information Sciences, 1:. S. Air Force Oflicc of Scientific Research, under Grant AFOSR-77-3271, is gratefully acknowledged, as is t)hc help of JIrs. Shclly Rowe in preparing this paper.