A scene description method using three-dimensional information

0031-3203/79/02014~09

Pattern Recognition, Vol. 11, pp. 9-17. Pergamon Press Ltd, 1979. Printed in Great Britain. © Pattern Recognition Society.

$02.00~)

A SCENE DESCRIPTION METHOD USING THREE-DIMENSIONAL INFORMATION M. OSHIMA and Y. SHIRAI Electrotechnical Laboratory, 2-6, Nagatacho, Chiyodaku, Tokyo, 100, Japan (Received 3 January 1978; in revised form 29 June 1978)

Abstract - This paper presents a method for describing sceneswith polyhedra and curved objects from threedimensional data obtained by a range finder. A scene is divided into many surface elements consisting of several data points. The surface elements are merged together into regions. The regions are classified into three classes : plane, curved and undefined. The program extends the curved regions by merging adjacent curved and undefined regions. Thus the scene is described by plane regions and smoothly curved regions, which might be usefulfor the recognition of the objects. From the results obtained so far the program seems to achieve the desired goals. Pattern recognition Object recognition Scene analysis Artificial intelligence Computer vision Image processing Feature extraction Scenedescription Rangefinding information Region method Leastsquares fit

I. I N T R O D U C T I O N

One of the central problems in the field of threedimensional object recognition is the description of objects. The description method is intimately related to the feature extraction process. A useful feature of polyhedra, for example, is a set of planes or edges. It is, however, difficult to extract proper features for curved objects in general. There have been some methods which use light intensity information for the scene including curved objects. Horn m dealt with smooth objects, on the assumption that the position of the light-source and the surface photometry were known and reconstructed the geometry of the surface. Barrow and Popplestond 2~ partitioned an image into connected regions and gave properties of regions and relations between regions. Tsuji and Fujiware~31segmented a scene into regions by a syntactic method. Shirai ~4~ proposed a recognition system including an edge finding process. There are, however, some difficulties with these studies. Without assumptions on illumination and optical characteristics of the surfaces, one cannot relate an image to the geometry. No one can guarantee that brightness always changes at an edge. A useful alternative to light intensity information is range or three-dimensional information. The program using range information can directly treat geometrical characteristics and easily interpret scenes with occlusion. In order to measure the three-dimensional position directly, two typical range finders have been studied. One is based on matching the stereopair of views. This method requires a great deal of calculation. On the

Robot Depth

other hand, the method proposed by Shirai 151employs a vertical slit projector and a TV camera to pick up its reflected light. By rotating a projector from the left to right, the distance of many points in a field of view is quickly obtained. By using this range finder, Shirai Is~ found lines from points, found planes from lines and finally recognized polyhedra, Agin~m gave a method for describing curved objects using a range finder similar to that used by Shirai/5) Complex objects were represented as structures of joining parts called generalized cylinders. Although the method is advantageous in describing bodies which consist of cylinder-like parts, there may arise some difficulties for other kind of objects. In the work by Shirai ~s~and Agin,~6~the programs at first approximated the images of projected slit beam by straight lines or curves. The first process is not always reliable because it depends only on the data points along a one-dimensional slit image. To overcome this difficulty, Kyura and Shirai ~7~ proposed the use of a surface element, which consists of a set of data points in a small region, as a unit for representing the surface. Merging the surface elements with similar properties, their program described scenes by plane surfaces. They showed that this approach was effective, even though the scene included a complicated polyhedron such as a dodecahedron or an icosahedron. Our method presented here is an extension of Kyura and Shirai's one. While only polyhedra are treated in their method, in our method objects are also extended to simple smoothly curved ones. Our intent is to make a program which describes a scene including plane surfaced and curved objects in a consistent way.

l0

M. OSHIMAand Y. SHIRAI

/~'IFAIj rl EI r #

/

•

(o)

(b)

C] C

'.

,4

(c]

i

Fig. 3. Image of projected slits.

(d|

(el

(f)

Fig. 1. Conceptual scheme of the process. (a) Original scene; (b) 3D co-ordinates; (c) surface elements; (dJ elementary regions; (e) classified regions; (f) global regions.

2. OUTLINE OF THE PROGRAM The raw data consists of positions of projected slit images which come from a TV camera. Threedimensional co-ordinates of the points on the slit are calculated by triangulation (Fig. lb). The processing of the scene proceeds as follows: (1) G r o u p the points into small surface elements and assuming each element to be a plane, get the equations of the surface elements (Fig. lc). (2) Merge the surface elements together into approximately plane regions (elementary regions, Fig. ld). (3) Classify the elementary regions into three classes: plane, curved and undefined (Fig. le). (4) Try to extend the curved regions by merging adjacent curved or undefined regions to produce larger regions (the global curved regions) and fit the quadratic surface to them (Fig. If). (5) Finally, describe the scene in terms of properties of regions and relations between regions.

!ii

3. CONSTRUCTIONOF SURFACEELEMENTS 3.1 Three-dimensional co-ordinates A vertical slit light is projected on the object through a rotating mirror. The signal from the TV camera is digitized by a real time A/D converter and is sent to a special piece of hardware (a SLITTER). ~s) The SLITTER extracts the center point of a slit image for every scanning line (Fig. 2) in real time - (the time required to get a set of data for a slit is 16 ms) and sends it to the computer. Each time the data for one slit line are obtained, a stepping motor rotates the mirror to change the direction of the slit beam. The raw data are represented by a 400 x 240 array of the slit positions. The (i, j) element corresponds to the horizontal position of the point on the jth scanning line in the TV image of the ith slit. An example of the data is shown in Fig. 3 (the scene consists of a sphere and an icosahedron). The calculated three-dimensional coordinates are stored in the same way. 3.2 Surface element The points on the surface of the objects are grouped into surface elements. Each element consists of adjoining 8 x 8 points in the array as shown in Fig. 4 - they are overlapping. Assuming each element to be a plane, the equation of the plane is found by the least square

skit number 234 1

2 3

S

•~

4

.5

I

c

Xl

X2

Vt= Vp{ 1-E¢1)-C2 x s = (x~

+

2(

c

o o o o

o e o o

o o o o

o o o o

in scene i

....

o o o o

o o o o

o : o o o o o o o o

qJooo

•

o o o o o U o o o • o o o o o o o o o o o o o o DOOO 9 o O0 I I gJ • O OOOO0 DOO0 O0 O • • • • • •

qJoo qloo quoo

• • •

e oo ql o o ooo tm •

• • • •

olo

•

(~,C2~-0 )

x2)/2

Fig. 2. Video signal corresponding to a slit light.

Fig. 4. Allocation of surface elements.

Scene description method

kerne

11

4.1 Kernel finding The candidates of a kernel are those elements which do not belong to any region already found. In finding a plane surface, it is desirable that a kernel is a part of a smooth surface. For this purpose, the evaluation function m is calculated for every candidate from the number n of the other candidates in its eight neighbors and their average standard deviation s:

m=w/n+g, ]

" elements already

]

:

elements

under

(2)

where w is a constant. If m is small it means that the element has many candidates in its neighbor and constitutes a smooth surface. The element with the minimum m is chosen as the kernel.

merged examination

Fig. 5. Region growing process. 4.2 Region merging method from its member points. A surface element is denoted by the equation: f ( x , y, z) = O, f(x,y,z)

(1)

= 2 x + # y + v z - p,

where x, y and z are co-ordinates of the point on the fitting plane. Thus the scene is represented by 80 x 60 surface elements. Some of the surface elements, for example at an edge, are meaningless. Therefore, the elements with too large a standard deviation s of the fitting are ignored in the further processing. 4, MERGING SURFACE ELEMENTS The surface elements are merged into approximately plane regions (elementary regions). The process is divided into two stages: (1) search for a kernel of an elementary region in the scene and (2) find a region around the kernel by merging surface elements. These stages are repeated iteratively.

The program extends a region around the kernel by merging its neighboring elements. The candidate element to be merged must currently be adjacent to the region. If the plane equation of a candidate is similar to that of the region, it is merged into the region. The conditions are : d2 =

(,]~k - - /+r) 2 Jr ( , ~ k - - / L r )

d~,

(3)

(4)

where ;:,/~, v and p correspond to those in equation (1), k denotes the element, r denotes the region and denotes thresholds. In Fig. 5, the dotted elements are already connected to the kernel and a set of marginal elements (hatched in Fig. 5) are under examination. Each time a set of marginal elements is fixed, plane equation of the region is updated. The program repeats this process until no extension is possible, An example of the region map is illustrated in Fig. 6 (which corresponds to Fig. 3). In Fig. 6, a letter represents a surface element and the character of it means its region label.

~R~MRMMM~ S SS~MR~MRM~ S55555R~MRRmMMMNNNNN

111

111111 1111111111 1111111111 3333

6 66

~'II

BBBBgSA~AAAnnADDDOI II I WUHHHSBSH A A A n A A A X X X X X X I |1fill ~HHHHHHHAAAAAAAAOXXXXXX lI ]L UHHHHHHHHHHPAAAOA000656E EIIILL UUUHHHHH~HHPPPA000000EE[E EE L L L UUUHHHHHHHPPPP000C000EEEEEEEULLL UFRFF PPPPPPKOOCOO~EEEEEEEULLL FFFKFF~PPPPP~KKOKOOOEEEEEEEVLLL F~FFFFFFPPPKRKKKKOOOE EEEVVUUL ~FFFFFFP KKKKKK[K JJJVUUV0 eFFFFgg9999KKKKCCCC£CJJJJJJOGQ FFF~g9999999KRCCCCCCCCJJJJJ0~O Fggg999999gKRCCCCCCCCJJJJJQOQ ~99999999???7?CCCCCCJJjjjQQO

666

WMW

~

2 <

IP~-P,I < P,,

$555555GGGGGGRN~NNMNNN 88885GGGGGGGDDDNNNNNM 88888868GGGGGDODDONMNNNN ~S88~8888GGGGDDDDDDD~NZlM UW 888 8 8 8 8 A A A A D D D D D D D ~ IIII

~.~.66,6,,~ ..... °°°°°°i

2 Jr ( Y k - - Y r )

L

LL LL LL LL

11111

33333333 11 33333333~333] 3333333 333 3 3 3

~666 333~33333333~3444 6 6666333 33333333 4 6666666 33333333333 4444 666666663333333333 44444 666666666 33333333 444444

~666666666

66666666663 666666666666 66666666666666 66666666666666

3~33333 4 , 4 4 4 4 ,

3333 44444444 333 444444444 3 4444444444 4444444444 20 44444444444 22222 ~6S556~66565555 444444444444 Oooee~ 555665656S565 4~4444414 eOOO 99999777777777CCCCCJJjQOQQ 55SS55555S55 ~222 44444 eerie 999777777777781BI|2R~QQ 55565565665 22~222 44444 llOIlt Y997777??7777B~JB~BRRQ 5555555655 22222~22~ 444 O~OOOO0 Y~Y7777777777| |||RRR 22 @ 555555555 2222~222~22~ 44 202~220~# VV77777777||JJ|BR ~ o e OO 55555555 2~22222222222 OOlOIO@0220 YVYYYT?7|II|I|| 2222 2eoe 5 556555222222222222222 eoleeoteoeee J|JJ eeeeoe e2oeo 55555 22222222222 22 eoeooeoeeooeo eeeeeeee 042022 5~55 22222222 OOOOOOOO00$2OOO 2eOOOOOOOOOOO e e e e e e o 2 65222222 eeoooeeooeeeoooooeeeoeooeeeooeeeooeoe eoeeooeee % 222 oooee Ooooooeeoeeeooeeooeeooee2oeoeeeeooo oooooooooo oooooooo OOOOO0tOOO0000000eOOO00000001000OO000000OOOOOO gO0000000000 OOOOO00OO000000$OOO$OO0000OO00$O000000000000000010000OO00000000 eoeooeeoeoooeeeeoeeeeeooe$oeooeoooeeooooeoooooeoooooooooooooe eeeeeeoeeooeeeeeeeeeeeeeeeoeeoeeoeooooooeeooooooooooooooooooo ooeeoeeeooeo$oeooooee$ooeoeoeooeoeoeeooeoooeoioooooooooooooo oeoeooeeoooeeeoeeoeeeeooee$oooeeoeooeeoooooooooooeooooooooo

Fig. 6. Elementary regions.

T

r T

,,

7T

~T TT

T

T T

?

4

12

M. OSHIMAand Y. SHIRAI 5 4

are classified based on two parameters. The first parameter d2 is the mean variance of the angle of the region which is given by:

2

where d~ is defined by equation (3) and n is the number of elements in the region. The second parameter is the effective diameter of the region map, which is defined as follows (Fig. 7):

///~, ,/,,/ ~m~1¢2,,

~b, = min(max(~blt . . . . . q~l,,) . . . . .

-,~..,)

max(q~,l . . . . , q~4,,)). (6) A curved surface with the larger curvature has the smaller 4~e.The regions with too small #~e mean small or slender ones. They are classified into undefined ones. The rest are classified by the classification criteria given by the following equations:

Fig. 7. Effective diameter.

5. C L A S S I F I C A T I O N O F E L E M E N T A R Y R E G I O N

Each elementary region corresponds to a plane surface or a part of a curved surface. In this stage the elementary regions are discriminated based on their parameters. It is difficult to draw a strict distinction between plane and curved regions on the basis of local criteria. A smooth plane, for example, does not always yield a region with uniformly distributed points due to noise and digitization error. At this stage, therefore, regions are classified into plane, curved and undefined ones. The undefined ones are considered later with the assistance of global information. First we should confirm the reliability of each region. In Section 3, surface elements with too large a standard deviation s are rejected, mainly to discard meaningless elements around plane edges. Some elements which are left are not reliable, especially at the position where slit light is broad and weak. The elementary region which consists of these kinds of elements should be rejected before the classification. Therefore, by using the average of standard deviation g of the member elements, a region with large g is classified as an undefined region. The rest of the elementary regions

15

l

o

o x~<~,V z~\\\\~j

o o

~" /

actual surface a plane

/x\\\\\'~Y

o

aoo

Iooo ,~;~iof~,,~ aooo

o o

o

OOooo: °°o°oo°~°

0

curved region

oo

0.01

0.02

o

o

0.03

0.04

plane region, curved region,

Lotherwise

undefined region,

0.05

~ 2 ( r a d 2)

Fig. 8. Classification of regions and example of data.

(8)

where ct is a constant, and gt(>0) is the threshold. Data obtained from several scenes and classification criteria are shown in Fig. 8. An example of classification (which corresponds to Fig. 6) is shown in Fig. 9. In our experiments for various scenes including plane surfaced and simple smoothly curved objects, the results so far have been satisfactory. Most of the undefined regions consist of a small number of elements. Misclassification rarely happens in processing important regions. 6. C U R V E D R E G I O N S

The program attempts to merge curved and undefined regions into curved global regions. The process is divided into two stages: (1) search for a kernel region of a global region in the scene; (2) find the global curved region around it, merging curved or undefined regions. These stages are repeated iteratively. After the global curved regions are found, the program fits quadratic surfaces to them and classifies the surfaces.

The candidates of a kernel region are curved elementary regions which do not belong to any global region already found. As the larger region is the more reliable for a kernel, we calculate the evaluation function I for every candidate region : 1 = w~'q~e + n

°2°° °o . . . . .

variance of angle

~<-:: g

(7)

6.1 Kernel region finding

curved

undefined

.o_

O = a. 2 z - ~be,

(9)

where n is the number of elements in the region and wo is a constant. A region with large I means that it has a large effective diameter and has many surface elements. The candidate with the maximum I is chosen as the kernel region.

Scene descriplion method

13

111 Illlll l,ltllllll 1111111111 3333 lllll G 33333333 11 66 3 3 3 3 3 3 3 3 3 3 3 3 3 6G6 333333333333333 6666 33333333333333 4 6666663333333333333 444 6666666 33333333333 4444 66666G663333333333 44444 666666666 33333333 444444 6666666666 3333333 4444444 66666666666333333 44444444 666666666666 333 444444444 66666666666666 3 4444444444 66£66666666666 4444444444 44444444444 eo SSSSSSSSSSSSSSS 44444444444~ 6ooeo 5SSSSSSSSSSS5 44444444 660o66 SSSSSSSSSSS~ ~ 4444444 ololo 6SSSSSSSSS~ ~Z~2~ 44444 60161 ~SSSSSSSSS ~ E ~ I 444 11o001 1o61111 lee SSSSSSSSS aaz~z~ez~ 44 61o1166140 11111 ssssssss aazaaaazaaaaa 4 oolelooooo4 eoAeloee SSSSSSS2224222~2E2222E 6101o1101601 00110016116 SSSSS 2 a 2 a 2 2 z 2 2 2 4 22 1066100644040 61101001011000 SSSS 2 2 4 2 2 2 4 2 eoooeooooeeeeeo eooeoeeoloeeelooooeeo sss222222 0004111006011111114010000014006111A10044011114 S 222 OOlOl 111111114111111111111111111111111111111111111 11111111 1111111111111111111111111111111011111111111111 111114111111 111111111111114111111111111111111111111111111111111111111111111 lllllllllllllllllllllllllOllllllllllllllllllllllllllllleOllll llllllllllllOlllllllllllilllOllllOlllllllllllllllllllllllelll IIIlIIIIIlIIIlIIIIIIIIIIIIIIlIIIIIIIIIIIIIlIIIIOIIIIlIIIIlll lltllelllllllllllllllllllllllllllllllllllllllOOllllllllllOl

(a) RRNRR~R~R S SSHRMHMRRRM SSSSSSRRRRRHRRRHHNNN SSSSSSSGGGGGGMNNNNNHNN S|aISGGGGGGGDDDNNHflNN 18||8|SIGGGGGDDDDONHNNNN 8|1811|11GGGGDDDDDDDNNIIH UWIllIlII81+*ARDDDODDDDIIII! WlIlIlISIlIM*++~ODDDeDIIllI| UUWlI$IISI4AA*IRAADODDIIIIIIII WUHHHIIIH RRRRR~RXXXXXX~III|I|LL WUHHHHHHHRRARRRRROXXXXXXX|I; ILLL UHHHNHMHHHHPAAROROOOEE[EEEIIILLLL UUUHHHHHHHHPPPROOOOOOEEEEEEE LLLLL UUUNHHHHHHPPPPOOOOOOOEEEEEEEULLLLL UUFFFF PPPPPPK000000EEEEEEEULLL~ UFFFFFFFPPPPP~KKOKOOOEEEEEEEULLL FFFFFFFFpPPK[KKKKOOOE EEEUVUUL FFFFFFFP KKKKKKKK JJJUUUUQ FFFFFeegllgKKKKCCCCCCJJJJJJQQQ FFFFeggggeggKKCCCCCCCCJJJJJggg FiIigggglegKKCCCCCCCCJJJJJGiO 999gegB99????TCCCCCCJJJJJGOQ 19gEgg???7?????CCCCCJJJIOGI !iI????????77|||||RIlIQ Vlg777777777?ll|lllnlG VYV7777777777|l|lllll VVT?77???71|lD|IR VYVVY777||l|||| ||||

T T T T TT TT TT TT T T T T

(b)

(C)

Fig. 9. Classified elementary regions. (a) Plane; (b) curved: (c) undefined.

6.2 Region merging The program extends a global curved region around the kernel region by merging its neighboring elementary regions. The candidates to be merged are the curved and undefined elementary regions which are currently adjacent to the global region. The program proceeds step by step. At first candidates are only those that are just adjacent to the kernel region. In Fig. 10 a global region growing is shown. The program checks whether or not each candidate region can be merged to the global region using the elementary regions called "touch stones". The touch stones are those elementary regions which are included in the global region and adjacent to the candidate. Suppose region 11 is checked as shown in Fig. 10, the regions 4 and 5 are the touch stones. If the candidate is determined to be smoothly connected with one of the touch stones, it is merged. The test is made as follows : Suppose in Fig. 11, CA and CB are the three-dimensional points corresponding to the centroid of the candidate and that of the touch stone respectively. The planes PA and PB are set to be perpendicular to the line segment CACB. Two hatched

areas in Fig. 11 are approximated by the plane equations. The two regions are determined to be smoothly connected if the angle 0 between the normals of the planes satisfies the following condition: 0 < w 0 .d,,

(10)

where d, is the same as in equation (3) and w0 is a constant. The smoothly curved body (for example, a sphere} may yield different elementary regions by

kerneJ ~q

region

]

:etemen?ary regions already merged

]

:eand[clo?eselementary

regions to

be merged

Fig. 10. Extension of curved global region.

14

M. OSHIMAand Y. SHIRAI where h(x,y,z) = a l l X2 + a22Y2 + a3av,2 + 2ax2xy + 2a23yz + 2aalzX + 2al,x + 2a24y + 2a3,z + a44, d 2 = ~ h2(xl, yi,zi).

(11) (12)

i=1

touch

stone

clreos used for fhe decision

Condidote

Fig. l 1. Candidate region and touch stone.

equation (3) and the expected angle between two neighboring regions is proportional to d t. The threshold in equation (10) is related to d t so that the same body yields the same global region for various dt's. After curved global regions are found, the unassigned surface elements are reconsidered. If the angle between an element and its neighboring elementary region is small, the element is merged into the same global region ; [this time the threshold is greater than that used in equation (3)]. Figure 12 shows an example of the results (corresponding to Fig. 9). 6.3 Fittin 9 quadratic surface to the curved reoion We have made no constraints on curved surfaces in the preceding processings. Now we try to approximate curved surfaces by quadratic ones. The quadratic surface is expressed in equation (11), and the evaluation function d 2 to be minimized is given by equation (12). t~(x, v z) = O,

In equation (12), (x i, yi, zi) are the co-ordinates of the point in the curved global region. Ifd2/(n - 3) is greater than a threshold, the surface is not approximated by the quadratic surface. The quadratic curved surface is classified using coefficients in equation (11). The classes are a parabolic cylinder, an elliptic cylinder, a hyperbolic cylinder, a paraboloid, a cone, an ellipsoid, a hyperboloid and undefined. The classification is done in the conventional mathematical way with the exception of using some thresholds. Though exhaustive experiment for every class has not been made until now, there is no difficulty for typical objects. Figure 13 shows an example of a fitting quadratic surface (corresponding to Fig. 12), where the quadratic surface is represented by the contour of equal z and it is classified to an ellipsoid.

7. M A K I N G D E S C R I P T I O N S

Having found the map of the global regions, the program describes the scene in terms of properties of regions and relations between regions. 7.1 Edges of regions In finding edges of the global regions, two kinds of processing are applied corresponding to the following classes of edges ; they are applied to the whole scene sequentially : (1) The edges between neighboring regions of plane surfaces or quadratic surfaces. (2) The others.

7777777777 ?7?77777777?7777 7777?777??7?77777777 77777?77777777???77777 7?777777777??777?7777777 777777777?77??77?7777??77?

7??777???777?7???77777777777 7??7777?77777?77777777777?7777 7??77?77777777??77???7777?7777~ 777??7?7?777?7??777777???777?77

7?777???7???777?7777???77?77?773 ??7?7777?77?7?77?77????7?777?77 77?7777?77777?77777777??7777?777?7

?

77?7??????777?7?7?77???7?777?7777? 77??77777??7?7?777777??777777777?7 77777777?77????7777??????77777?7~? ???7??777?7?777?7?77???7??7777?7 777777777?7777777777??????777~777 ???????7?????7??????77??7??7??77 7777777?777777?777?777777777?77 ??77?7????77???????7??777?????7 7???7?7???????7???????77???7?7 7777777777?777777777777777777 7777????7777??77?????7777??7 7???777?7?777????777?7?7? ???77?7??7?77?????7??7? ?7??77??7?7???7??????? 777777777777777777 777777777777777 7777777777

Fig. 12. Merged global curved region.

Fig. 13. Fitted quadratic surface to the curved region.

Scene description method

15

For those in class 1, the intersecting line is at first determined by the equations of the regions. This method is less sensitive to noise than the application of local edge operator, because all the points in the regions concerned contribute to the determination. If more than two plane regions adjoin one another, the corresponding line segments should meet at one vertex. In this; case, the co-ordinates (x,.,y,, z,) of the vertex are determined to minimize d2, the total sum of the square of the distance to each plane. Fig. 14. Described scene. d2 = ~ .f~(x,.,y,.,z,.) i

(13)

1

where 11is the same in equation (1), n is the number of the related planes and i corresponds to each plane. The edge in class 2 is described by a series of the edge points which are extracted from region map by using 3 x 3 local operator. 7.2 Properties of regions and relations between them For each region the property of the type, curved or plane, has been given in the preceding process. For each pair of the regions, the adjacency between them has been checked. If a pair of regions is adjacent from the border points those points which are in the vicinity of the border between two regions - of the regions, the centroid of them is calculated and the two groups of border points corresponding to the regions are approximated by planes. If the centroid is not in the origin side (the origin is at the center of the rotating mirror) of both planes, the common edge of the regions is regarded as convex.

(a)

1 11 1111 11111 I

tl I1|11111 III11111111111 1111111111111111 11111111111111111 I|1111111111111111 |1111111111|!11111I 1 1 1 1 1 1 1 1$11111 1I11|11111111|11|111 111111 11111111111111111111111 111111111111111|11|1111| 111111151111111111111111 11111111|11111111151111 1111111111111111111111 1111|11511|t111J11111 11111111111111111111

The scene could be expressed as a graph whose nodes are the regions and arcs between nodes that correspond to connecting relations between them. These properties and relations could be useful in describing the scene. Figure 14 shows an example of the description of the scene (corresponding to Fig. 6). In Fig. 14, the marks [] and © mean that the regions are plane and curved respectively; the dotted line with the mark + means that the two regions are adjacent and the relation that the common edge is convex holds between them. Only the two-dimensional picture (though the threedimensional one exists in the computer) is shown here.

8. S O M E

RESULTS

Figure 15 shows another example of the results (the scene consists of a block, a sphere and a cylinder). The

444444444444 44444444444444444 44444444444444444 444444444444444 PP 44444 PPP fF~???? 00 PPPFFF?????SI$ISO00 PPP~FF???T?SISIi000 PPP F Y F ? ? ? ? ? S S S I S O 0 @ PPP~F~F????iSSIS0@@ PPPFFFf????SISSIOO@ PPPPFFF?????SSIIi000 PPP FFFT??T?SIIIIOO PPPFFFF????$$ISIOO0 PPPFFSF???,$ISSi0@@ PPPFFff???'IISSSOO@ PPP FFF7??II,$$Ii@O PPP FF????'S,SI$OOO PP FF????IISSIIO0O PPP FFF??7?,ISIIOOO PPPFFfF????SSIS,OOO FPPFFFF????IIISIOOO PP F F F ???ISIS000@ IIIII $SOOO 1111| 00@ Ill IISI$$GGGG 0 It I $IIIEGGGGGGGGNNN ! III[[([EGGGGGINNNNN K[IIIII([IIGGGGGGNNNNNNN [[$[[IIIIIIIGIIIIINNNNNNHN [[[[(IlI[[EItlglIIIINmN~N*N [[[II88111[[IIII0iIII iNNN~H [(KI,IIIII III,IIIIIIIIINNHH [[81,IIIIIIIlIIIIIIIIIIIINNN

•

::lllllll~l~l~lllllllllllIIl ::::IIIIIII~i~LRLIlIIIIIIII | | I ,3 333 1,3,3.3 3,3,] 3, 3 *::::,,II'~II~LtRIIIIIIIIII RII6IIIIIiIIIILLLLLIIIIIIINIi IlJIlIIlIIIllIIIIII *I:llllllllllllIIllllllllIl ° IIIIIIIIIIIIIIIIIIIIII%11111 IIIIIUlUlUlIIIIIIIII' c* o* @ llllllllllllllllllllll * *@* c* o ,,,,,*,*o,,*.,,-,,. , IIIIIIIIlIIIllllIIIIl OOO,O 'IIlJIII.llllll' III, tllJlJJltllJlllJJJJJt Ill@ ° . . . IIIIIIIIIIIIIIIIIIIII' lll*,lllllh,. 'ii""'"",..... ,.It@Ill, "'It@It@It@It@Ill It" .IIIIIIIUIIh,. .,,IIIIIIIIUII, , ,,,,,,,,, • ,.,....,..,....,....o.,.,,.....,...,..,.,..,... ~)333 )I}}3333

11111111111111111 Iltltllllllll

llh,,

,I

OO OIO 1111 loll OOOl O00OO

IlJtililI

"'II

"

,IIIIllllIllllllllllllllIllllllllllIlllllllllllllh

IIIIIIIh,, ' ,IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIh .....ooooooo. ,,..ooooo..ooo.....-...,.- o..,. Fig. 15. Another example of the results. (a) Elementary regions (Figs. b e overleaf): (b) plane regions; (c) merged global curved regions; (d) fitted quadratic surfaces; (e} described scene.

:: .

::::. ..* . . . . . . .

..a. . . . . ..*. .a.. . . . . .

l *a . . . . . . . . . . . . . .

:.

z

r

v1

5

6 a

Scene description method

17

ed into small surface elements; each surface element is assumed to be a plane and its equation is found. The surface elements are merged into approximately plane regions. Then the regions are classified into three classes: plane, curved and undefined. The program extends the curved regions into global regions by merging adjacent curved and undefined regions. The curved global region is approximated by a quadratic surface. Thus, the scene is described by plane regions and curved regions, which might be useful for the recognition of the scene. We would like to thank N. Kyura and I. Miyamoto for providing data handling programs. We are grateful to the members of the Bionics section at the Electrotechnical Laboratory for their helpful discussions. Acknowledgements

REFERENCES Fig. 15. ( e )

regions are classified into three classes : plane, curved and undefined. The curved and undefined regions are merged into global curved regions. Finally descriptions of the scene are made in terms of properties of regions and relations between regions. The results obtained so far have been satisfactory. The method will be useful for the recognition of various scenes by a computer. tO. SUMMARY An approach to describing scenes by a computer using three-dimensional information is reported. Three-dimensional information is taken from a range finder which uses a vertical slit projector and a TV camera. The co-ordinates of the points on the objects are calculated by triangulation. The points are group-

1. B. Horn, Shape from shading: A method for obtaining the shape of a smooth opaque object from one view, MAC TR-79, MIT 0970), 2. H. Barrow and R. Popplestone, Relational descriptions in picture processing, Machine Intelligence Vol. 6, p. 377. Edinburgh University Press (1971). 3. S, Tsuji and R. Fujiwara, Linguistic segmentation of scenes into regions, 2nd Int. Joint Conf. on Pattern Recognition, p. 104 (1974). 4. Y. Shirai, Edge finding, segmentation of edges and recognition of complex objects, 4th Int. Joint Conj. on Artificial Intelligence. p. 674 (1975). 5. Y. Shirai, Recognition of polyhedrons with a range finder, Pattern Recognition 4, 243 (1972). 6. G. Agin, Representation and description of curved objects, AIM-173, Stanford University (1972). 7. N. Kyrua and Y. Shirai, Recognition of objects using three-dimensional region method, Bull. electrotechnical Lab. 37, 996 (1973) (in Japanese with English abstract). 8. M. Oshima and Y. Takano, Special hardware for the recognition system of three-dimensional objects, Bull. eleetrotechnical Lab. 37, 493 (1973) (in Japanese with English abstract).

About the Author MASAK10SH1MAwas born in Nagoya, Japan on 14 December 1944. He received his B.E. and M.E. degrees in Electrical Engineering from the University of Electro-communications. He has been working for the Electrotechnical Laboratory since 1970. He has been working on the recognition of three-dimensional objects. He is a member of the Institute of Electronics and Communication Engineers of Japan and the Information Processing Society of Japan. About the Author - YOSHIAKISHIRAIwas born in Toyota, Japan on 3 August 1941. He received his B.E. degree

from Nagoya University in 1966 and his Ph.D. in Mechanical Engineering from Tokyo University in 1969. Dr. Shirai then joined the staff of the Department of Information Science at the Electrotechnical Laboratory. From 1971 to 1972 he was a visiting researcher in the Artificial Intelligence Laboratory at the Massachusetts Institute of Technology working on computer vision. He continues his research in recognition of objects in the PIPS project. He is a member of the Institute of Electronics and Communication Engineers of Japan and the Information Processing Society of Japan.