- Email: [email protected]
A sequential approach to the extraction of shape features
COMPUTER GR.~PHICS AND IMAGE PROCESSING 5,
538-557 (1977)
A Sequential Approach to the Extraction of Shape Features* .A_SHOK~[(. AGRAWALAAND ASHOK V. I(ULKARNI~
Department of Compuler Science, University of Maryland, CollegePark, ~Iaryland ~07~2 Received July 26, 1976 Most shape feature extraction techniques require enormous computing resources. This paper presents a framework for the sequential processing of aa image which cat1 be used for extracting a variety of shape features, making only one pass over the image. In addition to the extraction of simple features~ such as perimeter and area, the paper also presents algorithms which eat~be used to extract complexfeatures based on "necks" and segmentation of the b~)undary. 1. INTRODUCTION A variety of object recognition problems require features to be measured on the object from a two-dimensional view ['1~. Such object recognition problems arise in biomedicine, metallurgy, cartography, character recognition, snd other fields. T h e shape of obieets plays an important role in industrial automation problems involving the assembly of complex objects. As is evident from surveys (Rosenfeld ~2-4~), various aspects of the problem of shape have received significant attention in the literature on picture processing. A n u mb er of books have also discussed this problem [-7-10~ 121. The major emphasis, in most of the litera.ture, has been on constructing new shape features r ath er t h a n on techniques for the quantitative extraction of such features. A common characteristic of shape feature extraction schemes presented in the literature is t h a t t h e y require enormous computing resources. The entire image is often assumed to be stored in computer memory. The feature extraction techniques m a y examine the image many times before extracting the values for the shape features. F o r extraction of shape features from coded line drawings, Freeman has described a number of techniques E5~. A complete survey of techniques applicable to line drawings is contained in ['111. A quick examination of these techniques indicates t h a t the computational advantage is gained mainly as a result of the coding used for storing the line drawing. Except for very simple features, the lines have to be examined many times, and converting a sequentially scanned image to th e chain code form requires additional computing resources. * This work was partially supported by L. N. K. Corporation, Inc., Silver Spring, MD. 1 Dr. Kulkarni is with Coulter Biomedical Research, Inc., Waltham, Massachusetts. 538 Copyrigh~ ~ 1977 b y Academic Press, Inc. All rightB of reproduction in ~ny form reaerved.
I~SN 0146-664X
SEQUENTIAL
SHAPE
FEATURE
EXTRACTION
539
As a result of the complexity of extraction techniques, only very simple features such as P2/A have been used extensively [-13]. Most of the other features have only been used in research studies. By now, research studies have established the desirability of shape features for some object recognition tasks. However, for practical use, they need to be computed efficiently and quickly without requiring the use of a large computer. Algorithms which extract such features by scanning an image sequentially are better suited in this regard. Ideas about sequential processing of an image for shape features such as area, perimeter, and moments have existed for some time [-10, pp. 19, 20, 138"]. In this work, the efficiency of the algorithm was rarely considered, and therefore the description of the algorithm was kept to a bare minimum. In 1970, a patent on such an algorithm for counting the number of blobs was issued [21"]. The first part of this paper (Sections 2 to 6) is devoted to a detailed description of a sequential tracking algorithm. While there are no significant new concepts in this portion, it is included here for its pedagogical value, as well as its importance in providing a framework within which algorithms for extracting other features, such as neck-related ones, can be presented. The problem is defined in Section 2 and the basic approach is described in Section 3. The boundary continuation algorithm is presented in Section 4 and an example is given in Section 5. In Section 6, we discuss the use of the boundary continuation algorithm in extracting some simple shape features, such as the number of objects, perimeter, area, moments, etc. The measurement of the neck of an object is significant for many problems. In Section 7, we present a sequential technique for making such measurements. The segmentation of the boundary in a sequential one-pass approach is described in Section 8. An example using a computer implementation of the algorithm is presented in Section 9. The algorithms described here are rather well suited for implementation on a minicomputer and for use in a real time operating enviroment. All the techniques described here have been implemented on a small minicomputer for use in a biomedical application. 2. PROBLEM
DEFINITION
For the discussion here, let us assume that the field of view of a scanner is rectangular with M X N points in it (see Fig. 1), and contains a number of objects, bodies and holes. I t is assumed that the grey level of a body is distinct from that of the background} A body may have holes having the grey level of the background. These holes may contain other bodies. The boundary of any object is assumed to be simply connected. No other assumptions are made about the object o1' its shape. The scanner scans this field of view in a raster scan from top to bottom, scanning each row from left to right. Thus a total of N scan lines are generated. As shown in Fig. 2, the output of the scanner goes into an edge detector which detects the location of points on the scan line which are on the edges of any object. A point In practice this may not be so ; it is assumed here in order to avoid a lengthy discussion of the edge detectionproblem.
AGI~AWALA AND KULKAKN][
540
N
F~Gur%~I in the body region whose adjacent left point is in the background will be called a left-edge point and denoted by x~, while xa will refer to a right-edge point which is in the background, with its adjacent left point in the body region. It should be noted that even simple techniques of gradient detection [3] can be used to detect the left- and right-edge points as described here. Detection of such points due to noise can be reduced significantly by appropriate design of the gradient detection filters. For the discussion here, we assume that the boundary of the field of view is always in the background. If an object goes outside the field of view, its visible portion can be handled by giving it an edge point on the boundary of the field of view. On any scan line, therefore, the left- and right-edge points will always occur in pairs, and many such pairs may occur on a scan line. On scan line k,, the ith pair will be denoted by x~.~(k), xg~(lc), where 1 < k < N and 1 _< x u _< M. The output of this front-end processor is a sequence of such x~, xR pairs for each scan line, which is an edge-point coded version of the digitized image in the field of view. The idea of using sorted edge points has been discussed in [6, 18]. In both of these works, it was assumed that enough storage is available to store all edge points in the image, ~nd the features are not to be extracted in real time. Shelman [6_-] has described his techniques for fingerprint identification ~pplications. Postproeessing is required in his technique to determine the connectivity of the boundary. No such postprocessing or backtracking is required in the work presented here. Merrill [18] uses a sorted edge-point list to encode line drawings for cartographic information retrieval. While he describes some techniques which m a y be useful for extracting shape features, the main thrust of his work is toward rapid retrieval of line drawing information.
~, [ SCANNER~ FzGUR~ 2
SEQUENTIAL
SHAPE
FEATURE
Ed~e Pt
xu(k-l)
Object #
OLl(k-l)
Old
New
Edge Pt
XLl(k)
Object #
OLl(k)
xRl(k-1) .,.
ORl(k-l) xRl(k) "Oi1(k )
-
541
EXTRACTION
, . .
.-. ,..
Scan Line Array. Object
#
Type
Parent
Status
Feature I
2
1 2
3
Object Array.
Fmum~ 3
Next let us consider the prob}em of extracting various shape features by sequentially examining an image given in edge-point encoded form. 3. BASIC APPROACH The basic approach taken in the algorithms described here is to process the information sequentially as it becomes available from the front-end processor, i.e., one scan line at a time and from left to right on the scan line. The procedures require storage of edge-point information about the current scan line and the previous scan line, but not the edge points of the whole image} In addition to this, only some information about objects encountered so far need be kept. The information about the shape features of these objects is also computed sequentially. The algorithm keeps track of the connections or continuations of all the boundaries seen so far in the image. As these boundaries can only belong to objects, each boundary is uniquely assigned to an object, body or hole. As the scanning of the image continues, a few distinct boundaries may merge. The boundary assignments are adjusted to such merges and the scanning continues until the complete image has been examined. The shape feature computations are arranged in a sequential manner so that the values for shape features can be updated for each scan line. At the end of scanning, except for a possible normalization, most of the shape features are available directly. As mentioned above, the algorithm requires storing information about two consecutive scan lines. This information is stored in the scan line array (see Fig. 2The ideas here can be e~ily extended to more complexoperations by storing a few previous lines.
542
AGRAWALA AND KULKAI~NI
3) which contains four rows and as many columns as are needed to store all the edge points likely to be found on any single scan line. The i n f o r m a t i o n about the previous line is retained in the two rows labeled "old." The first of these rows contains the actual edge points ExL~(k - - 1), xr~j(k -- 1)], for t h e line (k -- 1), when the current scan line is k. The second row contains the object n u m b e r OL~(k ~ 1 ) . . - for each edge point, which is the number of the object to which the edge is known to belong. For the current line, 1~, t h e set of edge points {xL~( k ) x R ~ ( k ) . . . } is read into the third row and the contents of the fourth row are initialized. In sequential processing, the status of various objects e n c o u n t e r e d changes. For example, if we have already seen an object completely, it will have no edge points on a scan line. A code for the status of the objects is stored in the third column of the array. Additional information is stored in the object array to facilitate f e a t u r e extraction or to speed up computations. For example, backpointers f r o m t h e object to the scan tine array may be stored in the object array, pointing to) t h e edge points of this object on lines (k -- 1) and k, as stored in the scan line array. 4. THE BOUNDARY CONTINUATION ALGORITI-IIVI The algorithm is organized to process all the information a b o u t each scan line before going to the next. For each scan line, the algorithm proceeds to determine the object to which each of the edge points on this scan line belongs. This determination is done sequentially and shape features are updated after assigning each edge point to an object. Therefore, the basic step of this algorithm is to read in { x ~ . ~ ( k ) x ~ ( k ) } into the scan line array, determine the object n u m b e r s for these edges, enter them into the scan line array, and update the f e a t u r e values. The process of boundary continuation is not different from t h e process of tracking El2]. A specific criterion, which is well suited for the p r o b l e m at hand, is described here. It is interesting to note t h a t the ideas of t r a c k i n g can be used sequentially in extracting shape features without requiring any b a c k t r a c k i n g or reseanning of the image. Before considering the feature extractor, let us consider the process of assigning the edges to the objects. This assignment is done by noting the e o n t i n n a t i o n of the boundaries of the objects on the previous line (k -- 1) and finding, for line k, if any new edge points have appeared, or boundaries on the previous line have disappeared. The actions to be taken under these conditions are examined next. 4.1.
Continuation
For a left boundary at x~j(k -- 1) to continue to xL~(k), it m u s t satisfy the conditions (Fig. 4) XR,_,(k) < x L ¢ ( k - - 1) < x r , ( k ) , (1) x~i_l(k -
1) < xL~(k) < xxj(k -- 1).
(2)
Similarly, for the right boundaries, xL~(k) < Xr~¢(k -- 1) < XL,+I(k)
(3)
SEQUENTIAL SHAPE FEATURE EXTRACTION k-~.
. xRj_ 1
XL;l
.
/
xRi_ 1
XL i
Fzo. 4. Continuation. Left boundary: x ~ _ l ( k )
xt~,I
543
fLl+l
\ XR i
XLi+ I
< x L j ( k - - 1) < ~ R i ( k ) , x r c j - l ( k -- 1) < xI,~ (Iv)
< xrq(k). Right boundary: xT.~(k) < xu~(k -- 1) < XLi_I(k), ZLj(k -- 1) < mite(k) < XLi+l(k --1).
and
x~(k.
t) < ~ ( k )
< x~i~,(k - 1).
(4)
Note t h a t in this continuation a left boundary has to continue as a left b o u n d a r y and a right boundary has to continue as a right boundary. Further, i a n d j do not have to be the same because there m a y be some new boundaries on ~ or discontinued boundaries on ( k - 1). The conditions for continuation are a direct result of our assumption t h a t the boundaries of objects are simply connected, i.e., a left b o u n d a r y is n o t allowed to cross a right boundary or vice versa. Similar continuation conditions have been used by Shelman [6]. Whenever a b o u n d a r y (left or right) at x(lc - 1) continues to x(Ic), b o t h the edge points belong to the same obiect, and hence x ( k ) can be assigned the object number O (k - 1) and entered in the scan line array for x (]~). At this point, all the features dependent on the single boundary of an object, e.g., perimeter, can be updated. Features depending on both boundaries of an object can only be u p d a t e d if the boundary at x ( k ) was the second boundary for the object, i.e., t h e first boundary was to the ]eft of x ( k ) and has already been associated. If any backpointers are retained in the object array, t h e y can also be updated easily. The process of checking for the continuation starts fl'om the left and goes to the right, examining one boundary fl'om line (]c - 1) a n d one fi'om ]~ at a time. Boundary continuation conditions (1)-(4) m a y be violated whenever boundaries of line (k - 1) disappear on line 7~ or new boundaries appear on line k. These conditions can be detected due to the violation of the continuation rules when attempting the continuation of boundaries of line (/~ -- 1) to line k, proceeding left to right. Note t h a t since left boundal:ies continue as left boundaries only if xLj.(k- 1) does not continue at x~(/c) but continues at xL,.+, a connection between xL~(k) and x~(/c) is implied. In general, whenever a contim~ation condition is violated, the next two boundaries have a horizontal connection. L e t us consider these cases next. 4.2 Creation
When a bound~ry at xLj(k -- 1) does not continue at XL~(k), or x~j(k -- 1) does not continue at xr~(]~), where i and j are the current pointers, it m a y be becanse on line k the boundary of a new object is encountered. This condition is detected when zL,+~(]c) < x~.(]c -
indicating the starting of a hole (Fig. 5a).
1),
(5)
544
AGltAWALA AND KULKAR,NI
XRj
XLj+l z/ ////
~
xR'I
XRi+l
/'~ i +2
Fro. 5(a) Creation of a hole. zi, i.H(k)
When an object terminates, the new scan line will h a v e no b o u n d a r y points for it. T h e conditions that distinguish between hole and b o d y t e r m i n a t i o n are shown in Figs. 6a and b. The detection of these conditions indicates ~ horizontal connection between the two edge points on line (k - 1). If b o t h of these edge points belong to t h e same object, t h a t object has been observed completely b y the time we reach line It. Hence we proceed to terminate the c o m p u t a t i o n of features for t h a t obiect. We may, however, find t h a t the object numbers, for t h e two boundaries of line (k -- 1) which are not continuing, are not t h e same. T h e horizontal connection on line (k - 1) indicates t h a t the two objects are merging into one. A sequential scan cannot detect such merging until it reaches line k. Thus, different processing is required to handle this case.
k-I k kk-'[
~ /'~XRj-|
XRj.t
XL~Rj
xL.
XRj
x?.
XRi_| Fla. 6 (a) Termination of a hole. xz4 < rare. (b) Termination of ~ body. xg~- < x~,~,
SEQUENTIAL SHAPE F E A T U R E E X T R A C T I O N
xLi'lxLi ~
545
xRi ×Ri
XLi ~
XR]+1
b"
.., p///t'.
Fro. 7. (~) Merging of the storm type o[ objects both bodies. (b) Merging of different types of objects. Left objee.~ is a p~rent of right. Right ob]ec~ is ~ pm'ent of left,
4.4. Merging The objects are assigned unique numbers on their creation. When two of them merge, one of the object numbers continues after the merge. This continuation may occur in three different ways. The geometries leading to such merges ~re shown in Fig. 7. Note that for detecting a merge, the conditions on the edge point are the same as those for detecting a termination. The merge is indicated by the additional condition that the object numbers for the two boundaries, which are to be discontinued, are not the same. The steps required in merge processing are: selecting the object which is continuing, correcting the boundary assignments made so far, and updating the features. In correcting the boundary assignments made so ~ar, we have to use the parent relationship of the objects. Using the technique discussed above, we can proceed to process any image sequentially and keep the correct assignments of the object mlmbers to various boundsries. To start the algorithm, we simply initialize the scan line ~nd object arrays and then start processing the first scan line. The algorithm ha.s been described here only for a single body type. Ho]es a.re
2
4. 5 6 7
S
6 5 5
1 36 56
li
B
9 10
24
F~Gu~ 8
546
AGRAWALA AND I4ULKARNI
o
,# co
oO
~
~
%
#
o
~
o
o
0
O o
:4
¢:
o
o
o
#
%° Io
,o
o
a
rl
e4
0
0
0
SEQUENTIAL SHAPE FEATURE EXTRACTION
547
At the end of Soan Line I0 19 24 A
A
Hole
Term
0
0
Hole
Merged
Hole
Merged
i0
Fro. 9. Arrays during the processing of objects on Fig. 8. empty areas in a body. This idea can be extended easily to take into account different types of objects. The only condition will be that the edge points be clearly marked as to which two types of bodies they refer to, and no occlusions be allowed. In the actual implementation of the algorithm various speedup and storagesaving mechanisms can be used easily. For example, data movement in the scan line array for each new line may be avoided by simple double buffering. 5. AN EXAMPLE OF THE BOUNDARY CONTINUATION ALGORITHM As an example of this algorithm, let us consider the objects in Fig. 8. The figure shows a complex shaped body containing a hole. The body spans scan lines 1 through 9. The coordinates of edge points on each scan line are marked. The algorithm proceeds to examine the edge points on each scan line and uses the scan line and obiect array in the process. The contents of these arrays at various intermediate levels of processing are shown in Fig. 9. Note that the algorithm starts b y considering objects B, C, and D as holes inside the body A. It so happens that C and D are not actually holes. This fact is revealed when the merges take place during the processing of lines 7 and 9. At the end, from the object array, we note that the figure contained one body which had a hole in it. 6. SHAPE FEATURE EXTRACTION The algorithm described in the preceding section shows how we m a y sequentially process the scanned information. The main task, however, is to extract shape features. The general idea of extracting features using this algorithm is to organize the computations in a sequential manner and re±ain the most up-todate information about the feature in all instances. A variety of shape features lend themselves to such a sequential computation. The extraction of a number of standard features is described first, while in the next section we discuss neck features, which do not directly lend themselves to sequential processing. 1. Number of objects. The number of objects of both types, i.e., bodies and holes, is directly available from the obiect array by counting the obiects that were terminated. The object array may contain many more entries corresponding to objects which got merged with other objects. Note that the relationships between objects, e.g., containment, merge, etc., are readily available in the status field of the object array. 2. Perimeter. On continuing any boundary, the incremental perimeter corresponding to it can be computed easily by noting that x(k -- 1) and x(k) give the
548
AGRAWALA AND KULKARNI
rnal neck
FIG, ]0. Neck feature.
x coordinates, and the y coordinates are only 1 apart. Hence, the incremental perimeter is {1 + [x(k) - x(7 (6) Squaring and square rooting may be avoided by making suitable approximations to this formula. The perimeter so obtained, however, is good only to the degree of digitization2 On creating or terminating an object, the length of the horizontal segment of the boundary has to be added to the perimeter. On merges, the perimeter of the two objects merged is added and assigned to the object continuing. 8. Area. The incremental area for any object for a sca,n line is the distance between its two edge points on the line. Therefore the area can easily be accumulated in sequential processing. No special steps are required on creation of an object. On termination the area of a hole may be subtracted from the body if so desired. Similar]y the area can be adjusted on merges. 4. Moments. The moments of the object shapes can be determined in sequential processing by accumulating the appropriate v~lues. For example, the first moments in the x and y directions are obtained by accumulating the values for Mx and My where = ½ E
My
= E/c['xR(/c)
-
z
2(k)-l,
- xT,(k)-].
(7)
(8)
T h e final moments are obtained by dividing Mx and My by the area. Similarly, higher-order moments can be computed by accumulating the appropriate terms. Based on primary features such as perimeter, area, etc., we can easily compute derived features such as P~/A [13]. The feature P2/A is one of the most commonly used featnres for shape and has often been used to measure the "circularity" of a shape. 7. NECK FEATURES A shape feature of interest in a variety of studies is the neck of an object. A few examples of necks are shown in Fig. 10. Most of the neck detection proFor further properties of perimeters of digitM objects, see [14].
SEQUENTIAL SHAPE FEATURE EXTRACTION
549
z2 ~-~-~-~z 6
z3f FmUR~ 11 eedures reported in the literature require making a pairwise distance measurement for all points on the boundary [5]. Note that for many biomedical applications we are only interested in "external necks" (see Fig. 10). This algorithm is geared to finding internal necks efficiently. Before proceeding further, let us define a local neck. Let d(z~, zi) be the distance between two points (z~, zj). Consider two portions of the boundary of an object on which z~, z,, z, are three adjacent points and z4, zh, z, are three other adjacen~ points (see Fig. 11). Then we say that a local neck occurs between z2, z~ if
d(z~, z~) < d(z,, z~), d(z~, zO < d(z~, z~), d(z~, zO < d(z~, z~),
(9)
and
d(z2, z~) < d(z~, z~). In other words, d(z2, z~) is the minimum of the five distances marked. To explain the working of the algorithm let us consider a single body. When we have reached scan line k, we have observed all the scan lines from 1 to (k - 1), and hence have seen all the edge points of this body on those lines. For x~(k) we define dE(k) as the minimum distance from XL(lC) to any right boundary of this body occurring on lines 1 to k. Then a local neck occurs from the left boundary at line k if dL(~ -- 1) > dL(k) and dL(k q- 1) > dL(k). We note that the other two conditions, of (9) are implicitly met as dE(k) is the minimum distance. A corresponding definition can be made for the right neck. The global neck on the body will be the smallest local neck. By always measuring distances to the next right boundalT from a left boundary and to the previous left boundary from a right boundary, we assure the measurement of internal necks only. In addition to the neck measurements made from the left and right boundaries, measurements are also made from a bottom boundary which is discovered as a result of a termination of merge. In arranging these computations sequentially without requiring huge amounts of computations and storage, we make use of some simple geometric relations. As shown in Fig. 12, if we have a right triangle, ABD, then the distance AC will always be less than AD. Thus, if on the boundary of an object seen so far, two edge points occur at the same x value, we only need to measure the distance to the most recent point. We also note that the distance from A to any point to the
AGI:tAWALA AND KULKAI~NI
550
Y
FmURF~ 12
right of the line B D will be greater than A B . Therefor~ we need not consider any right-edge points occurring beyond xR (k). In other words, we only need to consider the distance ~rom ~(I~) to the right-edge points in the range x~(lc) to x~(l~) visible when looking straight up from that line. A few of these situations are shown in Fig. 13a. Such a veiw of the image can be retained easily in an array o:f size (M X 2). As shown in Fig. 14, in this TOPVIEW array we make an entry for every one of the M horizontal locations and fill in the line number, i.e., y coordinate :for the boundary occurring most recently at that x v~luo. The type of the boundary is also retained by m~rking it as left, right, top, or bottom. Therefore, at scan line k, this array contains the view of the image looking up. The calculation of dL (l~) now reqnires calculating distances to the right boundary entries in the T O P V I E W array between x locations mL(/~) and x~ (l~). Further heuristics can be used to reduce the number of distance computations. a
El=Areaof.seorchfromle,ftboundory dmTn
XL.(k)
Fza. 13(b). The portion of the right boundary to which the disl~nces have Lobe measured from xT.(k) are shown in the dark.
SEQUENTIAL SHAPE FEATURE EXTRACTION Line
H
I:l 1
Y2
-
551
.......
- --
,
Type 2
M
Fro. 1.4. TOPVIEW
array.
For example, we may keep a value dm~ for this computation and not consider those right boundary points which cannot have a value leading to a distance less than dm~,. For this dm~,, we may use the minimum neck measured so far, or use ( x R - xn). A way of limiting the computations is to not consider those line numbeIs, y2s, for which (7~ - y~) > dm,~. (10) This limits the area of search to be a rectangle (see Fig. 13b). If a distance less than drain iS found, the value of d,,~. ms,y be changed to reduce the area of search further. Ideally we would like to limit the area of search to be bounded by a circular arc with radius d,.~ and center x~(lc). The cutoff y values for a circular cutoff can be computed iteratively without requiring any multiplication. To consider the technique for doing this let us examine Fig. 15. From the geometry of the figure, we note that d ~ = Xl 2 -F y i ~ = x2 a -~- y2 2 -- x~ 2 -J- ya2.
(n)
Hence y~2 = d ~ _ x 2 2
(i2)
= yl 2 - 2xl - 1 and y~=d
2-x3
~=d
~-
= y2 ~ -
(xi-F2)~ 2 ( x ~ ) - - 1.
(i3)
~rom Eqs. (12) and (13), we note that the values of y2 2 and y~ require subtracting a number, allowing for a simple iterative computation. Experimentation with this approach indicated that for a variety of shapes the distance computations required may be reduced to the order of 2P for an object with P edge points on the boundary. In a straightforward approach, the nnmber of distance computations is of order P:. A simple explanation for this improvement is as follows. Consider an object which has distance d~ between its left and right edges on line i. If no cutoff heuristics are used, a total of d~ distance computations will be required for that line. Hence, the total number of distance computations N
g d
h
D
E
F G
Fm. 15. A E = x l ; A F = x ~ = xI+ I ; A G = x 3 = x l + 2 i B E =
y~;CF =y2iDG=y~.
552
AGRAWALA AND KULKARNI
would be h r = ~ d~ = Area of the object p2.
As a result of cutoff heuristics the number of points go which the distance is computed is reduced. If we assume that for each distance computation the reduction is b y a factor of r, then the number of computations N 1 is given as N 1 = ~ l o g , d~, which for normal ranges of d~'s is likely to increase in proportion to P. This heuristic argument tends to confirm our experimental observations. We found that this technique of determining necks is very robust and is not significantly affected by the orientation of the object, presence of multiple lobes, points of inflexions, or horizontal edges. 8. SEGMENTATION OF THE BOUNDARY
Many approaches to segmentation of object boundaries have been reported in the literature [17]. Our work differs from these approaches in its emphasis on sequential processing. A class of shape features can be extracted using syntactic analysis techniques for the boundary of an object. Such a syntactic analysis requires that the boundary be broken into primitive segments first where the types of primitives used depend upon the application. The algorithms here allow us to segment the boundary of an object into primitive segments of various types and maintain the connectivity segments of various types and maintain the connectivity of these segments by creating a list structure. The primitive segments considered here are straight lines, convexities, concavities, etc. In breaking a boundary into these three primitive types of segments, we have to examine only one boundary of the object at a time because the shape at one boundary does not a.ffeet the segmentation decisions at the other boundary. Therefore, the segmentation can proceed as soon as the continuation of a boundary has been determined. However, following the general philosophy of completing the computations in a single puss, we may develop the segments for all the boundaries at each scan line. These segments have to be linked properly when merging or termination is encountered. To maintain the proper connectivity, we may
-XL(k-l) .,.
XL ( k )
// 7/xL (k+1) FmUR~ 16
SEQUENTIAL SHAPE FEATURE EXTRACTION XL= 73 l 2 3
553
XR=I7C
5 6 7 lo Iz ~3 ~5 16 I7 ~s 19 2o 2L ~2 23 2~ 25 26 27 2B z? 3o 31 32 33
d
g
•
• Q
0
g
o
Q
e #
35 ~6 37 3a 3g ~o
0
# I
D
~2
|
# ~5
•
•
e•
•
•
m
~7
O 5o 51 5~ 53 5a 55 5o 5~ 5o
O
| w.
qP #
#
•
. •g I O
#
62 b
#
e
~5
~9 71 7~ 7~ 75 76 77 78 79
FmURE17 treat each segment as a directed line and follow the convention of traversing the boundary of an obiect in a counterclockwise direction. Then the segment on a left boundary will always be going down, that on a right boundary will be going up, that on the top will be going from right to left, and that at the bottom will be going from left to right.
554
&OI%AWALA A N D KULKAI%NI ~L= 73 ~R=178 l 3 g 5
6 7 9 ~0 13 15 25 17 Ca le ZO 2L
A*
25 26 27 d8 29 30 3L 32 33 3~ 35 36 37 39 4L
A
A
;%
**
* *A
/
A
]
^At~a
B
"
h
#
/
0
~
*
"
A
# A
A A
^
C
~
6
a ^a /
~
A
A A
C
A A A
C c
c c c
C c C C C
A ~,
A
1
c
c
4S
~7
A A
3
/
2~
Aa A A
.p9 50 bl Ua b3 M* 55 5b 57 5a .59 60 6L b2 b5
A
2/
8 /
¢ C
7
cc
cc
A A A A
5 ~ A
c k
A
A*
~A^ A
9
*
AA
A A
o5 ~6 67
A
A A
b9 70 ?Z 72 "/3 7~ ~5 76 77 7a 79 h0
k
.9 ~'IGURN
18
In the edge-point encoding we know the positions of connected edge points of a bounda,ry. _is shown in Fig. ]6~ the three consecutive left-boundary points are at [oeations x~@ -- i), xL(fi), and xl.(lc + 1). F r o m these three values, we can detel~nine the angle 0. If we were to break the b o u n d a r y into straight line segments, then a nonzero value for 0 would indicate a termination of a segment.
555
SEQUENTIAL SHAPE :FEATURE EXTRACTION TABLE 1 Primary Features~ Obiect No. 1 3
Type
Contained in
Area
Body Hole
-1
1595 111
Perimeter F-/A 170.78 44.00
1.45 1.39
Object 2 merged with obiect 1 on line 28. We m a y m a r k t h e straight line segments by noting their lengths and the angles 0 at their ends. T h e r e is no difference in t h e segmentation decision for straight line segments on the right b o u n d a r y except t h a t the boundary has to be considered as going up. To carry out t h e segmentation in the framework of this algorithm, for each edge point we consider the segmentation decision, and either let the current segment continue, or t e r m i n a t e the current segment and start a new one. Different segmentation decision rules will give different primitive segments. For example, when we allow a continuation only for 0 = 0, straight line segments result. If we allow t h e continuation as long as the sign of 0 remains the same, the boundary is broken u p in convex and conc~Lve segments. By considering 0 = 0 as an additional condition, we segment the boundary into convex, concave, and straight line segments. F o r a convex or a conc'ave section, in addition to the length, we m a y also retain the t o t a l angular change as a descriptor. T h e procedures considered here will yield a description of the boundary of an obiect as a sequence of connected segments. Due to digitization noise~ the segments are likely to be too small. For example, when a straight line is not at the proper angle, its digitized version is likely to consist of a large number of small straight line segments. A simple syntactic procedure can be used to remove this and other similar types of noise. We m a y then describe the boundary in terms of larger segments which m a y be of more interest, and m a y be used in coming up with such shape descriptors as the m l m b e r of lobes. TABLE 2 Neck:Features Neck No. 1 2 3 4 5 6 7 8 9 10
Neck length
Left object
Right object
Left line
14.1 14.1 16.8 18.0 14.0 17.7 8.6 ].7.1 11.7 14.9
1 1 3 1 3 1 3 1 1 3
1 3 1 3 1 3 1 3 3 1
31 33 43 45 49 50 49 55 64 54
Right line 21 43 27 45 49 45 54 53 54 65
556
AGI~AWALA AND I(ULKARNI TABLE 3 Segments
Segment No.
1 2 3 4 5 6 7 8 9 10 11 12 1.3 14 15
Obiec~ No.
1 1 1 1 1 1 2 2 1 1 1 3 3 1 1
Perimeter
4.19 3.69 3.13 5.03 70.48 10.94 7.83 6.89 2.25 ].4.00 11.75 19.85 24.85 11.75 20.11
Number of points
2 2 1 1 47 6 3 3 1 12 9 13 ]3 7 12
Start coordinates
End coordinates
X
Y
X
Y
132 132 129 135 126 140 155 155 1.61 ]63' 1.67 138 138 ]62 156
19 19 21 21 22 22 25 25 28 29 41. 42 42 50 57
129 135 126 140 143 148 148 161 163 167 162 ]39 139 156 143
21 21 22 22 69 28 28 28 29 ~1 50 55 55 57 69
9. AN EXAMPLE The algorithms described here were i m p l e m e n t e d on a c o m p u t e r a n d used for the extraction of shape features. As an example, we processed the s h a p e (~ lake with an island) shown in Fig. 17. T h e o u t p u t of processing this figure is shown in Fig. 18, where the b o u n d a r y is m a r k e d b y A, for object n u m b e r 1 ; B, for object n u m b e r 2; and C, for object n u m b e r 3. T h e asterisk indicates the points of the boundary where it was segmented. The local necks picked up b y the algorithm are also m a r k e d on the figure. A_ n u m b e r of features were c o m p u t e d for this figure. These features are shown in Tables 1, 2, and 3. Table 1 shows t h a t t h e algorithm found two objects. Object 1 is a lake and object 2 is an island inside the lake. The values of their are.% perimeter, and P2/A are also shown. T h e local necks picked up b y t h e algorithm are shown in T a b l e 2. T h e length of the neck, t h e object numbers, and the line n u m b e r s for the two ends of the local neck are also shown, T a b l e 3 shows the segments of the boundaries and some of their properties. 10. CONCLUSIONS T h e sequential algorithms described here are well suited for i m p l e m e n t a t i o n in software on a small machine or in hardware. T h e y require v e r y little m e m o r y , and because of the strictly sequential n a t u r e of the operations, are easily implementable in hardware. Using these algorithms, we m a y extract shape features which would otherwise need m u c h more t i m e on a large computer. T h e approach described as the b o u n d a r y continuati'on algorithm is a basic sequential approach which allows the extraction of various shape features. T h e
SEQUENTIAL SHAPE FEATURE EXTRACTION
557
basic idea is to drastically reduce the information to be stored at every stage, and to keep a correct accumulated value for features where possible. Some examples of this approach for a few shape features are also presented. The algorithm has been described here in its simplest form. Extensions to it are possible in many ways. For example, a certain amount of smoothing of the contour may be performed b y retaining more than one previous line of the image. It is interesting to note that the sequential approach can be used for measuring such complex features as the internal necks of an object. When combined with segmentation features, a variety of powerful shape featul'es can be derived. ACKNOWLEDGMENT
The authors thank Dr. Laveen N. Kanal for many useful discussions and for continued encouragement and support during the course of this work. REFEItENCES
1. L. N. Kanal, Patterns in pattern recognition, IEEE Trans. Information Theory 20, 1974, 697-722. 2, A. Rosenfeld, Picture processing by computer, Computing Surveys 1, 1969, 147-176. 3. A. Rosenfeld, Picture processing: 1969-71, Computing Surveys 5, 1973, 81-I08. 4. A. Rosenfeld, Picture processing: 1972, Computer Graphics and Image Processing 1, 1972, 394-416. 5. H. Freeman, "Boundary Encoding and Processing," in Picture Processing and Psychopietorics (B. S. Lipkin and A. Rosenfeld, Eds.), pp. 241-266. Academic Press, New York, 1970. 6. C. B. Shelman, The application of list processing techniques to picture processing, Pattern Recognition 4, 1972, 201-210. 7. lt. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis, pp. 341-371. Wiley, New York, 1973. 8. A. Rosenfeld, "Picture processing: 1973," Compuler Graphics and Image Processing 3, 1974, 178. 9. B. S. Lipkin and A. Rosenfeld (51.ds.), Picture Processing and Psychopietories, Academic Press, New York, 1970. 10. A. Rosenfeld, Picture Processing by Computer, Academic Press, New York, 1969. 11. It. Freeman, Computer processing of line-drawing images, Computing Surveys 6, 1974, 57-97. 12. A. Rosenfeld and A. C. Kak, Digital Picture Processing, Academic Press, New York, 1976. 13. F. Attneave, Some informational aspects of visual perception," Psychol. Rev. 61, 1954, 183-193. 14. A. Rosenfeld, A Note on Perimeter and Diameter in Digital Pictures, University of Maryland Computer Science Center, Technical Report TR-252, July 1973. 15. H. Freeman, On the classification of line-drawing data, in Models for the Perception of Speech and Visual Form (W. Wuthen-Dunn Ed.), pp. 408-412. MIT Press, Cambridge, Mass., 1967. 16. H. A. Glucksman, A parapropagation pattern clasifier, IEEE Trans. Electronic Computers E(]-14, 1965, 434-443. 17. H. F. Feng and T. Pavlidis, Decomposition of polygons into simpler components: Feature generation for syntactic pattern recognition, IEEE Trans. Computers C-24, 1975, 636-650. 18. R. D. Merrill, Representation of contoum and regions for efficient computer search, Comm. ACM 16, 1973, 69-82. 19. A. Rosenfeld and J. L. Pfaltz, Sequential operations in digital picture processing, J. Assoc. Comput. Mach. 13, 1966, pp. 478-479. 20. D. Rutovitz, Data structures for operations on digital images, in Pictorial Pattern Recognition (G. C. Cheng et al., eds.), pp. 115-117. Thompson Book Co., Washington, D.C., 1968. 21. L. B. Scott ~nd K. Preston, Apparatus for counting irregularly shaped objects," U.S. Patent No. 3,408,485, 1968.
538-557 (1977)
A Sequential Approach to the Extraction of Shape Features* .A_SHOK~[(. AGRAWALAAND ASHOK V. I(ULKARNI~
Department of Compuler Science, University of Maryland, CollegePark, ~Iaryland ~07~2 Received July 26, 1976 Most shape feature extraction techniques require enormous computing resources. This paper presents a framework for the sequential processing of aa image which cat1 be used for extracting a variety of shape features, making only one pass over the image. In addition to the extraction of simple features~ such as perimeter and area, the paper also presents algorithms which eat~be used to extract complexfeatures based on "necks" and segmentation of the b~)undary. 1. INTRODUCTION A variety of object recognition problems require features to be measured on the object from a two-dimensional view ['1~. Such object recognition problems arise in biomedicine, metallurgy, cartography, character recognition, snd other fields. T h e shape of obieets plays an important role in industrial automation problems involving the assembly of complex objects. As is evident from surveys (Rosenfeld ~2-4~), various aspects of the problem of shape have received significant attention in the literature on picture processing. A n u mb er of books have also discussed this problem [-7-10~ 121. The major emphasis, in most of the litera.ture, has been on constructing new shape features r ath er t h a n on techniques for the quantitative extraction of such features. A common characteristic of shape feature extraction schemes presented in the literature is t h a t t h e y require enormous computing resources. The entire image is often assumed to be stored in computer memory. The feature extraction techniques m a y examine the image many times before extracting the values for the shape features. F o r extraction of shape features from coded line drawings, Freeman has described a number of techniques E5~. A complete survey of techniques applicable to line drawings is contained in ['111. A quick examination of these techniques indicates t h a t the computational advantage is gained mainly as a result of the coding used for storing the line drawing. Except for very simple features, the lines have to be examined many times, and converting a sequentially scanned image to th e chain code form requires additional computing resources. * This work was partially supported by L. N. K. Corporation, Inc., Silver Spring, MD. 1 Dr. Kulkarni is with Coulter Biomedical Research, Inc., Waltham, Massachusetts. 538 Copyrigh~ ~ 1977 b y Academic Press, Inc. All rightB of reproduction in ~ny form reaerved.
I~SN 0146-664X
SEQUENTIAL
SHAPE
FEATURE
EXTRACTION
539
As a result of the complexity of extraction techniques, only very simple features such as P2/A have been used extensively [-13]. Most of the other features have only been used in research studies. By now, research studies have established the desirability of shape features for some object recognition tasks. However, for practical use, they need to be computed efficiently and quickly without requiring the use of a large computer. Algorithms which extract such features by scanning an image sequentially are better suited in this regard. Ideas about sequential processing of an image for shape features such as area, perimeter, and moments have existed for some time [-10, pp. 19, 20, 138"]. In this work, the efficiency of the algorithm was rarely considered, and therefore the description of the algorithm was kept to a bare minimum. In 1970, a patent on such an algorithm for counting the number of blobs was issued [21"]. The first part of this paper (Sections 2 to 6) is devoted to a detailed description of a sequential tracking algorithm. While there are no significant new concepts in this portion, it is included here for its pedagogical value, as well as its importance in providing a framework within which algorithms for extracting other features, such as neck-related ones, can be presented. The problem is defined in Section 2 and the basic approach is described in Section 3. The boundary continuation algorithm is presented in Section 4 and an example is given in Section 5. In Section 6, we discuss the use of the boundary continuation algorithm in extracting some simple shape features, such as the number of objects, perimeter, area, moments, etc. The measurement of the neck of an object is significant for many problems. In Section 7, we present a sequential technique for making such measurements. The segmentation of the boundary in a sequential one-pass approach is described in Section 8. An example using a computer implementation of the algorithm is presented in Section 9. The algorithms described here are rather well suited for implementation on a minicomputer and for use in a real time operating enviroment. All the techniques described here have been implemented on a small minicomputer for use in a biomedical application. 2. PROBLEM
DEFINITION
For the discussion here, let us assume that the field of view of a scanner is rectangular with M X N points in it (see Fig. 1), and contains a number of objects, bodies and holes. I t is assumed that the grey level of a body is distinct from that of the background} A body may have holes having the grey level of the background. These holes may contain other bodies. The boundary of any object is assumed to be simply connected. No other assumptions are made about the object o1' its shape. The scanner scans this field of view in a raster scan from top to bottom, scanning each row from left to right. Thus a total of N scan lines are generated. As shown in Fig. 2, the output of the scanner goes into an edge detector which detects the location of points on the scan line which are on the edges of any object. A point In practice this may not be so ; it is assumed here in order to avoid a lengthy discussion of the edge detectionproblem.
AGI~AWALA AND KULKAKN][
540
N
F~Gur%~I in the body region whose adjacent left point is in the background will be called a left-edge point and denoted by x~, while xa will refer to a right-edge point which is in the background, with its adjacent left point in the body region. It should be noted that even simple techniques of gradient detection [3] can be used to detect the left- and right-edge points as described here. Detection of such points due to noise can be reduced significantly by appropriate design of the gradient detection filters. For the discussion here, we assume that the boundary of the field of view is always in the background. If an object goes outside the field of view, its visible portion can be handled by giving it an edge point on the boundary of the field of view. On any scan line, therefore, the left- and right-edge points will always occur in pairs, and many such pairs may occur on a scan line. On scan line k,, the ith pair will be denoted by x~.~(k), xg~(lc), where 1 < k < N and 1 _< x u _< M. The output of this front-end processor is a sequence of such x~, xR pairs for each scan line, which is an edge-point coded version of the digitized image in the field of view. The idea of using sorted edge points has been discussed in [6, 18]. In both of these works, it was assumed that enough storage is available to store all edge points in the image, ~nd the features are not to be extracted in real time. Shelman [6_-] has described his techniques for fingerprint identification ~pplications. Postproeessing is required in his technique to determine the connectivity of the boundary. No such postprocessing or backtracking is required in the work presented here. Merrill [18] uses a sorted edge-point list to encode line drawings for cartographic information retrieval. While he describes some techniques which m a y be useful for extracting shape features, the main thrust of his work is toward rapid retrieval of line drawing information.
~, [ SCANNER~ FzGUR~ 2
SEQUENTIAL
SHAPE
FEATURE
Ed~e Pt
xu(k-l)
Object #
OLl(k-l)
Old
New
Edge Pt
XLl(k)
Object #
OLl(k)
xRl(k-1) .,.
ORl(k-l) xRl(k) "Oi1(k )
-
541
EXTRACTION
, . .
.-. ,..
Scan Line Array. Object
#
Type
Parent
Status
Feature I
2
1 2
3
Object Array.
Fmum~ 3
Next let us consider the prob}em of extracting various shape features by sequentially examining an image given in edge-point encoded form. 3. BASIC APPROACH The basic approach taken in the algorithms described here is to process the information sequentially as it becomes available from the front-end processor, i.e., one scan line at a time and from left to right on the scan line. The procedures require storage of edge-point information about the current scan line and the previous scan line, but not the edge points of the whole image} In addition to this, only some information about objects encountered so far need be kept. The information about the shape features of these objects is also computed sequentially. The algorithm keeps track of the connections or continuations of all the boundaries seen so far in the image. As these boundaries can only belong to objects, each boundary is uniquely assigned to an object, body or hole. As the scanning of the image continues, a few distinct boundaries may merge. The boundary assignments are adjusted to such merges and the scanning continues until the complete image has been examined. The shape feature computations are arranged in a sequential manner so that the values for shape features can be updated for each scan line. At the end of scanning, except for a possible normalization, most of the shape features are available directly. As mentioned above, the algorithm requires storing information about two consecutive scan lines. This information is stored in the scan line array (see Fig. 2The ideas here can be e~ily extended to more complexoperations by storing a few previous lines.
542
AGRAWALA AND KULKAI~NI
3) which contains four rows and as many columns as are needed to store all the edge points likely to be found on any single scan line. The i n f o r m a t i o n about the previous line is retained in the two rows labeled "old." The first of these rows contains the actual edge points ExL~(k - - 1), xr~j(k -- 1)], for t h e line (k -- 1), when the current scan line is k. The second row contains the object n u m b e r OL~(k ~ 1 ) . . - for each edge point, which is the number of the object to which the edge is known to belong. For the current line, 1~, t h e set of edge points {xL~( k ) x R ~ ( k ) . . . } is read into the third row and the contents of the fourth row are initialized. In sequential processing, the status of various objects e n c o u n t e r e d changes. For example, if we have already seen an object completely, it will have no edge points on a scan line. A code for the status of the objects is stored in the third column of the array. Additional information is stored in the object array to facilitate f e a t u r e extraction or to speed up computations. For example, backpointers f r o m t h e object to the scan tine array may be stored in the object array, pointing to) t h e edge points of this object on lines (k -- 1) and k, as stored in the scan line array. 4. THE BOUNDARY CONTINUATION ALGORITI-IIVI The algorithm is organized to process all the information a b o u t each scan line before going to the next. For each scan line, the algorithm proceeds to determine the object to which each of the edge points on this scan line belongs. This determination is done sequentially and shape features are updated after assigning each edge point to an object. Therefore, the basic step of this algorithm is to read in { x ~ . ~ ( k ) x ~ ( k ) } into the scan line array, determine the object n u m b e r s for these edges, enter them into the scan line array, and update the f e a t u r e values. The process of boundary continuation is not different from t h e process of tracking El2]. A specific criterion, which is well suited for the p r o b l e m at hand, is described here. It is interesting to note t h a t the ideas of t r a c k i n g can be used sequentially in extracting shape features without requiring any b a c k t r a c k i n g or reseanning of the image. Before considering the feature extractor, let us consider the process of assigning the edges to the objects. This assignment is done by noting the e o n t i n n a t i o n of the boundaries of the objects on the previous line (k -- 1) and finding, for line k, if any new edge points have appeared, or boundaries on the previous line have disappeared. The actions to be taken under these conditions are examined next. 4.1.
Continuation
For a left boundary at x~j(k -- 1) to continue to xL~(k), it m u s t satisfy the conditions (Fig. 4) XR,_,(k) < x L ¢ ( k - - 1) < x r , ( k ) , (1) x~i_l(k -
1) < xL~(k) < xxj(k -- 1).
(2)
Similarly, for the right boundaries, xL~(k) < Xr~¢(k -- 1) < XL,+I(k)
(3)
SEQUENTIAL SHAPE FEATURE EXTRACTION k-~.
. xRj_ 1
XL;l
.
/
xRi_ 1
XL i
Fzo. 4. Continuation. Left boundary: x ~ _ l ( k )
xt~,I
543
fLl+l
\ XR i
XLi+ I
< x L j ( k - - 1) < ~ R i ( k ) , x r c j - l ( k -- 1) < xI,~ (Iv)
< xrq(k). Right boundary: xT.~(k) < xu~(k -- 1) < XLi_I(k), ZLj(k -- 1) < mite(k) < XLi+l(k --1).
and
x~(k.
t) < ~ ( k )
< x~i~,(k - 1).
(4)
Note t h a t in this continuation a left boundary has to continue as a left b o u n d a r y and a right boundary has to continue as a right boundary. Further, i a n d j do not have to be the same because there m a y be some new boundaries on ~ or discontinued boundaries on ( k - 1). The conditions for continuation are a direct result of our assumption t h a t the boundaries of objects are simply connected, i.e., a left b o u n d a r y is n o t allowed to cross a right boundary or vice versa. Similar continuation conditions have been used by Shelman [6]. Whenever a b o u n d a r y (left or right) at x(lc - 1) continues to x(Ic), b o t h the edge points belong to the same obiect, and hence x ( k ) can be assigned the object number O (k - 1) and entered in the scan line array for x (]~). At this point, all the features dependent on the single boundary of an object, e.g., perimeter, can be updated. Features depending on both boundaries of an object can only be u p d a t e d if the boundary at x ( k ) was the second boundary for the object, i.e., t h e first boundary was to the ]eft of x ( k ) and has already been associated. If any backpointers are retained in the object array, t h e y can also be updated easily. The process of checking for the continuation starts fl'om the left and goes to the right, examining one boundary fl'om line (]c - 1) a n d one fi'om ]~ at a time. Boundary continuation conditions (1)-(4) m a y be violated whenever boundaries of line (k - 1) disappear on line 7~ or new boundaries appear on line k. These conditions can be detected due to the violation of the continuation rules when attempting the continuation of boundaries of line (/~ -- 1) to line k, proceeding left to right. Note t h a t since left boundal:ies continue as left boundaries only if xLj.(k- 1) does not continue at x~(/c) but continues at xL,.+, a connection between xL~(k) and x~(/c) is implied. In general, whenever a contim~ation condition is violated, the next two boundaries have a horizontal connection. L e t us consider these cases next. 4.2 Creation
When a bound~ry at xLj(k -- 1) does not continue at XL~(k), or x~j(k -- 1) does not continue at xr~(]~), where i and j are the current pointers, it m a y be becanse on line k the boundary of a new object is encountered. This condition is detected when zL,+~(]c) < x~.(]c -
indicating the starting of a hole (Fig. 5a).
1),
(5)
544
AGltAWALA AND KULKAR,NI
XRj
XLj+l z/ ////
~
xR'I
XRi+l
/'~ i +2
Fro. 5(a) Creation of a hole. zi, i.H(k)
When an object terminates, the new scan line will h a v e no b o u n d a r y points for it. T h e conditions that distinguish between hole and b o d y t e r m i n a t i o n are shown in Figs. 6a and b. The detection of these conditions indicates ~ horizontal connection between the two edge points on line (k - 1). If b o t h of these edge points belong to t h e same object, t h a t object has been observed completely b y the time we reach line It. Hence we proceed to terminate the c o m p u t a t i o n of features for t h a t obiect. We may, however, find t h a t the object numbers, for t h e two boundaries of line (k -- 1) which are not continuing, are not t h e same. T h e horizontal connection on line (k - 1) indicates t h a t the two objects are merging into one. A sequential scan cannot detect such merging until it reaches line k. Thus, different processing is required to handle this case.
k-I k kk-'[
~ /'~XRj-|
XRj.t
XL~Rj
xL.
XRj
x?.
XRi_| Fla. 6 (a) Termination of a hole. xz4 < rare. (b) Termination of ~ body. xg~- < x~,~,
SEQUENTIAL SHAPE F E A T U R E E X T R A C T I O N
xLi'lxLi ~
545
xRi ×Ri
XLi ~
XR]+1
b"
.., p///t'.
Fro. 7. (~) Merging of the storm type o[ objects both bodies. (b) Merging of different types of objects. Left objee.~ is a p~rent of right. Right ob]ec~ is ~ pm'ent of left,
4.4. Merging The objects are assigned unique numbers on their creation. When two of them merge, one of the object numbers continues after the merge. This continuation may occur in three different ways. The geometries leading to such merges ~re shown in Fig. 7. Note that for detecting a merge, the conditions on the edge point are the same as those for detecting a termination. The merge is indicated by the additional condition that the object numbers for the two boundaries, which are to be discontinued, are not the same. The steps required in merge processing are: selecting the object which is continuing, correcting the boundary assignments made so far, and updating the features. In correcting the boundary assignments made so ~ar, we have to use the parent relationship of the objects. Using the technique discussed above, we can proceed to process any image sequentially and keep the correct assignments of the object mlmbers to various boundsries. To start the algorithm, we simply initialize the scan line ~nd object arrays and then start processing the first scan line. The algorithm ha.s been described here only for a single body type. Ho]es a.re
2
4. 5 6 7
S
6 5 5
1 36 56
li
B
9 10
24
F~Gu~ 8
546
AGRAWALA AND I4ULKARNI
o
,# co
oO
~
~
%
#
o
~
o
o
0
O o
:4
¢:
o
o
o
#
%° Io
,o
o
a
rl
e4
0
0
0
SEQUENTIAL SHAPE FEATURE EXTRACTION
547
At the end of Soan Line I0 19 24 A
A
Hole
Term
0
0
Hole
Merged
Hole
Merged
i0
Fro. 9. Arrays during the processing of objects on Fig. 8. empty areas in a body. This idea can be extended easily to take into account different types of objects. The only condition will be that the edge points be clearly marked as to which two types of bodies they refer to, and no occlusions be allowed. In the actual implementation of the algorithm various speedup and storagesaving mechanisms can be used easily. For example, data movement in the scan line array for each new line may be avoided by simple double buffering. 5. AN EXAMPLE OF THE BOUNDARY CONTINUATION ALGORITHM As an example of this algorithm, let us consider the objects in Fig. 8. The figure shows a complex shaped body containing a hole. The body spans scan lines 1 through 9. The coordinates of edge points on each scan line are marked. The algorithm proceeds to examine the edge points on each scan line and uses the scan line and obiect array in the process. The contents of these arrays at various intermediate levels of processing are shown in Fig. 9. Note that the algorithm starts b y considering objects B, C, and D as holes inside the body A. It so happens that C and D are not actually holes. This fact is revealed when the merges take place during the processing of lines 7 and 9. At the end, from the object array, we note that the figure contained one body which had a hole in it. 6. SHAPE FEATURE EXTRACTION The algorithm described in the preceding section shows how we m a y sequentially process the scanned information. The main task, however, is to extract shape features. The general idea of extracting features using this algorithm is to organize the computations in a sequential manner and re±ain the most up-todate information about the feature in all instances. A variety of shape features lend themselves to such a sequential computation. The extraction of a number of standard features is described first, while in the next section we discuss neck features, which do not directly lend themselves to sequential processing. 1. Number of objects. The number of objects of both types, i.e., bodies and holes, is directly available from the obiect array by counting the obiects that were terminated. The object array may contain many more entries corresponding to objects which got merged with other objects. Note that the relationships between objects, e.g., containment, merge, etc., are readily available in the status field of the object array. 2. Perimeter. On continuing any boundary, the incremental perimeter corresponding to it can be computed easily by noting that x(k -- 1) and x(k) give the
548
AGRAWALA AND KULKARNI
rnal neck
FIG, ]0. Neck feature.
x coordinates, and the y coordinates are only 1 apart. Hence, the incremental perimeter is {1 + [x(k) - x(7 (6) Squaring and square rooting may be avoided by making suitable approximations to this formula. The perimeter so obtained, however, is good only to the degree of digitization2 On creating or terminating an object, the length of the horizontal segment of the boundary has to be added to the perimeter. On merges, the perimeter of the two objects merged is added and assigned to the object continuing. 8. Area. The incremental area for any object for a sca,n line is the distance between its two edge points on the line. Therefore the area can easily be accumulated in sequential processing. No special steps are required on creation of an object. On termination the area of a hole may be subtracted from the body if so desired. Similar]y the area can be adjusted on merges. 4. Moments. The moments of the object shapes can be determined in sequential processing by accumulating the appropriate v~lues. For example, the first moments in the x and y directions are obtained by accumulating the values for Mx and My where = ½ E
My
= E/c['xR(/c)
-
z
2(k)-l,
- xT,(k)-].
(7)
(8)
T h e final moments are obtained by dividing Mx and My by the area. Similarly, higher-order moments can be computed by accumulating the appropriate terms. Based on primary features such as perimeter, area, etc., we can easily compute derived features such as P~/A [13]. The feature P2/A is one of the most commonly used featnres for shape and has often been used to measure the "circularity" of a shape. 7. NECK FEATURES A shape feature of interest in a variety of studies is the neck of an object. A few examples of necks are shown in Fig. 10. Most of the neck detection proFor further properties of perimeters of digitM objects, see [14].
SEQUENTIAL SHAPE FEATURE EXTRACTION
549
z2 ~-~-~-~z 6
z3f FmUR~ 11 eedures reported in the literature require making a pairwise distance measurement for all points on the boundary [5]. Note that for many biomedical applications we are only interested in "external necks" (see Fig. 10). This algorithm is geared to finding internal necks efficiently. Before proceeding further, let us define a local neck. Let d(z~, zi) be the distance between two points (z~, zj). Consider two portions of the boundary of an object on which z~, z,, z, are three adjacent points and z4, zh, z, are three other adjacen~ points (see Fig. 11). Then we say that a local neck occurs between z2, z~ if
d(z~, z~) < d(z,, z~), d(z~, zO < d(z~, z~), d(z~, zO < d(z~, z~),
(9)
and
d(z2, z~) < d(z~, z~). In other words, d(z2, z~) is the minimum of the five distances marked. To explain the working of the algorithm let us consider a single body. When we have reached scan line k, we have observed all the scan lines from 1 to (k - 1), and hence have seen all the edge points of this body on those lines. For x~(k) we define dE(k) as the minimum distance from XL(lC) to any right boundary of this body occurring on lines 1 to k. Then a local neck occurs from the left boundary at line k if dL(~ -- 1) > dL(k) and dL(k q- 1) > dL(k). We note that the other two conditions, of (9) are implicitly met as dE(k) is the minimum distance. A corresponding definition can be made for the right neck. The global neck on the body will be the smallest local neck. By always measuring distances to the next right boundalT from a left boundary and to the previous left boundary from a right boundary, we assure the measurement of internal necks only. In addition to the neck measurements made from the left and right boundaries, measurements are also made from a bottom boundary which is discovered as a result of a termination of merge. In arranging these computations sequentially without requiring huge amounts of computations and storage, we make use of some simple geometric relations. As shown in Fig. 12, if we have a right triangle, ABD, then the distance AC will always be less than AD. Thus, if on the boundary of an object seen so far, two edge points occur at the same x value, we only need to measure the distance to the most recent point. We also note that the distance from A to any point to the
AGI:tAWALA AND KULKAI~NI
550
Y
FmURF~ 12
right of the line B D will be greater than A B . Therefor~ we need not consider any right-edge points occurring beyond xR (k). In other words, we only need to consider the distance ~rom ~(I~) to the right-edge points in the range x~(lc) to x~(l~) visible when looking straight up from that line. A few of these situations are shown in Fig. 13a. Such a veiw of the image can be retained easily in an array o:f size (M X 2). As shown in Fig. 14, in this TOPVIEW array we make an entry for every one of the M horizontal locations and fill in the line number, i.e., y coordinate :for the boundary occurring most recently at that x v~luo. The type of the boundary is also retained by m~rking it as left, right, top, or bottom. Therefore, at scan line k, this array contains the view of the image looking up. The calculation of dL (l~) now reqnires calculating distances to the right boundary entries in the T O P V I E W array between x locations mL(/~) and x~ (l~). Further heuristics can be used to reduce the number of distance computations. a
El=Areaof.seorchfromle,ftboundory dmTn
XL.(k)
Fza. 13(b). The portion of the right boundary to which the disl~nces have Lobe measured from xT.(k) are shown in the dark.
SEQUENTIAL SHAPE FEATURE EXTRACTION Line
H
I:l 1
Y2
-
551
.......
- --
,
Type 2
M
Fro. 1.4. TOPVIEW
array.
For example, we may keep a value dm~ for this computation and not consider those right boundary points which cannot have a value leading to a distance less than dm~,. For this dm~,, we may use the minimum neck measured so far, or use ( x R - xn). A way of limiting the computations is to not consider those line numbeIs, y2s, for which (7~ - y~) > dm,~. (10) This limits the area of search to be a rectangle (see Fig. 13b). If a distance less than drain iS found, the value of d,,~. ms,y be changed to reduce the area of search further. Ideally we would like to limit the area of search to be bounded by a circular arc with radius d,.~ and center x~(lc). The cutoff y values for a circular cutoff can be computed iteratively without requiring any multiplication. To consider the technique for doing this let us examine Fig. 15. From the geometry of the figure, we note that d ~ = Xl 2 -F y i ~ = x2 a -~- y2 2 -- x~ 2 -J- ya2.
(n)
Hence y~2 = d ~ _ x 2 2
(i2)
= yl 2 - 2xl - 1 and y~=d
2-x3
~=d
~-
= y2 ~ -
(xi-F2)~ 2 ( x ~ ) - - 1.
(i3)
~rom Eqs. (12) and (13), we note that the values of y2 2 and y~ require subtracting a number, allowing for a simple iterative computation. Experimentation with this approach indicated that for a variety of shapes the distance computations required may be reduced to the order of 2P for an object with P edge points on the boundary. In a straightforward approach, the nnmber of distance computations is of order P:. A simple explanation for this improvement is as follows. Consider an object which has distance d~ between its left and right edges on line i. If no cutoff heuristics are used, a total of d~ distance computations will be required for that line. Hence, the total number of distance computations N
g d
h
D
E
F G
Fm. 15. A E = x l ; A F = x ~ = xI+ I ; A G = x 3 = x l + 2 i B E =
y~;CF =y2iDG=y~.
552
AGRAWALA AND KULKARNI
would be h r = ~ d~ = Area of the object p2.
As a result of cutoff heuristics the number of points go which the distance is computed is reduced. If we assume that for each distance computation the reduction is b y a factor of r, then the number of computations N 1 is given as N 1 = ~ l o g , d~, which for normal ranges of d~'s is likely to increase in proportion to P. This heuristic argument tends to confirm our experimental observations. We found that this technique of determining necks is very robust and is not significantly affected by the orientation of the object, presence of multiple lobes, points of inflexions, or horizontal edges. 8. SEGMENTATION OF THE BOUNDARY
Many approaches to segmentation of object boundaries have been reported in the literature [17]. Our work differs from these approaches in its emphasis on sequential processing. A class of shape features can be extracted using syntactic analysis techniques for the boundary of an object. Such a syntactic analysis requires that the boundary be broken into primitive segments first where the types of primitives used depend upon the application. The algorithms here allow us to segment the boundary of an object into primitive segments of various types and maintain the connectivity segments of various types and maintain the connectivity of these segments by creating a list structure. The primitive segments considered here are straight lines, convexities, concavities, etc. In breaking a boundary into these three primitive types of segments, we have to examine only one boundary of the object at a time because the shape at one boundary does not a.ffeet the segmentation decisions at the other boundary. Therefore, the segmentation can proceed as soon as the continuation of a boundary has been determined. However, following the general philosophy of completing the computations in a single puss, we may develop the segments for all the boundaries at each scan line. These segments have to be linked properly when merging or termination is encountered. To maintain the proper connectivity, we may
-XL(k-l) .,.
XL ( k )
// 7/xL (k+1) FmUR~ 16
SEQUENTIAL SHAPE FEATURE EXTRACTION XL= 73 l 2 3
553
XR=I7C
5 6 7 lo Iz ~3 ~5 16 I7 ~s 19 2o 2L ~2 23 2~ 25 26 27 2B z? 3o 31 32 33
d
g
•
• Q
0
g
o
Q
e #
35 ~6 37 3a 3g ~o
0
# I
D
~2
|
# ~5
•
•
e•
•
•
m
~7
O 5o 51 5~ 53 5a 55 5o 5~ 5o
O
| w.
qP #
#
•
. •g I O
#
62 b
#
e
~5
~9 71 7~ 7~ 75 76 77 78 79
FmURE17 treat each segment as a directed line and follow the convention of traversing the boundary of an obiect in a counterclockwise direction. Then the segment on a left boundary will always be going down, that on a right boundary will be going up, that on the top will be going from right to left, and that at the bottom will be going from left to right.
554
&OI%AWALA A N D KULKAI%NI ~L= 73 ~R=178 l 3 g 5
6 7 9 ~0 13 15 25 17 Ca le ZO 2L
A*
25 26 27 d8 29 30 3L 32 33 3~ 35 36 37 39 4L
A
A
;%
**
* *A
/
A
]
^At~a
B
"
h
#
/
0
~
*
"
A
# A
A A
^
C
~
6
a ^a /
~
A
A A
C
A A A
C c
c c c
C c C C C
A ~,
A
1
c
c
4S
~7
A A
3
/
2~
Aa A A
.p9 50 bl Ua b3 M* 55 5b 57 5a .59 60 6L b2 b5
A
2/
8 /
¢ C
7
cc
cc
A A A A
5 ~ A
c k
A
A*
~A^ A
9
*
AA
A A
o5 ~6 67
A
A A
b9 70 ?Z 72 "/3 7~ ~5 76 77 7a 79 h0
k
.9 ~'IGURN
18
In the edge-point encoding we know the positions of connected edge points of a bounda,ry. _is shown in Fig. ]6~ the three consecutive left-boundary points are at [oeations x~@ -- i), xL(fi), and xl.(lc + 1). F r o m these three values, we can detel~nine the angle 0. If we were to break the b o u n d a r y into straight line segments, then a nonzero value for 0 would indicate a termination of a segment.
555
SEQUENTIAL SHAPE :FEATURE EXTRACTION TABLE 1 Primary Features~ Obiect No. 1 3
Type
Contained in
Area
Body Hole
-1
1595 111
Perimeter F-/A 170.78 44.00
1.45 1.39
Object 2 merged with obiect 1 on line 28. We m a y m a r k t h e straight line segments by noting their lengths and the angles 0 at their ends. T h e r e is no difference in t h e segmentation decision for straight line segments on the right b o u n d a r y except t h a t the boundary has to be considered as going up. To carry out t h e segmentation in the framework of this algorithm, for each edge point we consider the segmentation decision, and either let the current segment continue, or t e r m i n a t e the current segment and start a new one. Different segmentation decision rules will give different primitive segments. For example, when we allow a continuation only for 0 = 0, straight line segments result. If we allow t h e continuation as long as the sign of 0 remains the same, the boundary is broken u p in convex and conc~Lve segments. By considering 0 = 0 as an additional condition, we segment the boundary into convex, concave, and straight line segments. F o r a convex or a conc'ave section, in addition to the length, we m a y also retain the t o t a l angular change as a descriptor. T h e procedures considered here will yield a description of the boundary of an obiect as a sequence of connected segments. Due to digitization noise~ the segments are likely to be too small. For example, when a straight line is not at the proper angle, its digitized version is likely to consist of a large number of small straight line segments. A simple syntactic procedure can be used to remove this and other similar types of noise. We m a y then describe the boundary in terms of larger segments which m a y be of more interest, and m a y be used in coming up with such shape descriptors as the m l m b e r of lobes. TABLE 2 Neck:Features Neck No. 1 2 3 4 5 6 7 8 9 10
Neck length
Left object
Right object
Left line
14.1 14.1 16.8 18.0 14.0 17.7 8.6 ].7.1 11.7 14.9
1 1 3 1 3 1 3 1 1 3
1 3 1 3 1 3 1 3 3 1
31 33 43 45 49 50 49 55 64 54
Right line 21 43 27 45 49 45 54 53 54 65
556
AGI~AWALA AND I(ULKARNI TABLE 3 Segments
Segment No.
1 2 3 4 5 6 7 8 9 10 11 12 1.3 14 15
Obiec~ No.
1 1 1 1 1 1 2 2 1 1 1 3 3 1 1
Perimeter
4.19 3.69 3.13 5.03 70.48 10.94 7.83 6.89 2.25 ].4.00 11.75 19.85 24.85 11.75 20.11
Number of points
2 2 1 1 47 6 3 3 1 12 9 13 ]3 7 12
Start coordinates
End coordinates
X
Y
X
Y
132 132 129 135 126 140 155 155 1.61 ]63' 1.67 138 138 ]62 156
19 19 21 21 22 22 25 25 28 29 41. 42 42 50 57
129 135 126 140 143 148 148 161 163 167 162 ]39 139 156 143
21 21 22 22 69 28 28 28 29 ~1 50 55 55 57 69
9. AN EXAMPLE The algorithms described here were i m p l e m e n t e d on a c o m p u t e r a n d used for the extraction of shape features. As an example, we processed the s h a p e (~ lake with an island) shown in Fig. 17. T h e o u t p u t of processing this figure is shown in Fig. 18, where the b o u n d a r y is m a r k e d b y A, for object n u m b e r 1 ; B, for object n u m b e r 2; and C, for object n u m b e r 3. T h e asterisk indicates the points of the boundary where it was segmented. The local necks picked up b y the algorithm are also m a r k e d on the figure. A_ n u m b e r of features were c o m p u t e d for this figure. These features are shown in Tables 1, 2, and 3. Table 1 shows t h a t t h e algorithm found two objects. Object 1 is a lake and object 2 is an island inside the lake. The values of their are.% perimeter, and P2/A are also shown. T h e local necks picked up b y t h e algorithm are shown in T a b l e 2. T h e length of the neck, t h e object numbers, and the line n u m b e r s for the two ends of the local neck are also shown, T a b l e 3 shows the segments of the boundaries and some of their properties. 10. CONCLUSIONS T h e sequential algorithms described here are well suited for i m p l e m e n t a t i o n in software on a small machine or in hardware. T h e y require v e r y little m e m o r y , and because of the strictly sequential n a t u r e of the operations, are easily implementable in hardware. Using these algorithms, we m a y extract shape features which would otherwise need m u c h more t i m e on a large computer. T h e approach described as the b o u n d a r y continuati'on algorithm is a basic sequential approach which allows the extraction of various shape features. T h e
SEQUENTIAL SHAPE FEATURE EXTRACTION
557
basic idea is to drastically reduce the information to be stored at every stage, and to keep a correct accumulated value for features where possible. Some examples of this approach for a few shape features are also presented. The algorithm has been described here in its simplest form. Extensions to it are possible in many ways. For example, a certain amount of smoothing of the contour may be performed b y retaining more than one previous line of the image. It is interesting to note that the sequential approach can be used for measuring such complex features as the internal necks of an object. When combined with segmentation features, a variety of powerful shape featul'es can be derived. ACKNOWLEDGMENT
The authors thank Dr. Laveen N. Kanal for many useful discussions and for continued encouragement and support during the course of this work. REFEItENCES
1. L. N. Kanal, Patterns in pattern recognition, IEEE Trans. Information Theory 20, 1974, 697-722. 2, A. Rosenfeld, Picture processing by computer, Computing Surveys 1, 1969, 147-176. 3. A. Rosenfeld, Picture processing: 1969-71, Computing Surveys 5, 1973, 81-I08. 4. A. Rosenfeld, Picture processing: 1972, Computer Graphics and Image Processing 1, 1972, 394-416. 5. H. Freeman, "Boundary Encoding and Processing," in Picture Processing and Psychopietorics (B. S. Lipkin and A. Rosenfeld, Eds.), pp. 241-266. Academic Press, New York, 1970. 6. C. B. Shelman, The application of list processing techniques to picture processing, Pattern Recognition 4, 1972, 201-210. 7. lt. O. Duda and P. E. Hart, Pattern Classification and Scene Analysis, pp. 341-371. Wiley, New York, 1973. 8. A. Rosenfeld, "Picture processing: 1973," Compuler Graphics and Image Processing 3, 1974, 178. 9. B. S. Lipkin and A. Rosenfeld (51.ds.), Picture Processing and Psychopietories, Academic Press, New York, 1970. 10. A. Rosenfeld, Picture Processing by Computer, Academic Press, New York, 1969. 11. It. Freeman, Computer processing of line-drawing images, Computing Surveys 6, 1974, 57-97. 12. A. Rosenfeld and A. C. Kak, Digital Picture Processing, Academic Press, New York, 1976. 13. F. Attneave, Some informational aspects of visual perception," Psychol. Rev. 61, 1954, 183-193. 14. A. Rosenfeld, A Note on Perimeter and Diameter in Digital Pictures, University of Maryland Computer Science Center, Technical Report TR-252, July 1973. 15. H. Freeman, On the classification of line-drawing data, in Models for the Perception of Speech and Visual Form (W. Wuthen-Dunn Ed.), pp. 408-412. MIT Press, Cambridge, Mass., 1967. 16. H. A. Glucksman, A parapropagation pattern clasifier, IEEE Trans. Electronic Computers E(]-14, 1965, 434-443. 17. H. F. Feng and T. Pavlidis, Decomposition of polygons into simpler components: Feature generation for syntactic pattern recognition, IEEE Trans. Computers C-24, 1975, 636-650. 18. R. D. Merrill, Representation of contoum and regions for efficient computer search, Comm. ACM 16, 1973, 69-82. 19. A. Rosenfeld and J. L. Pfaltz, Sequential operations in digital picture processing, J. Assoc. Comput. Mach. 13, 1966, pp. 478-479. 20. D. Rutovitz, Data structures for operations on digital images, in Pictorial Pattern Recognition (G. C. Cheng et al., eds.), pp. 115-117. Thompson Book Co., Washington, D.C., 1968. 21. L. B. Scott ~nd K. Preston, Apparatus for counting irregularly shaped objects," U.S. Patent No. 3,408,485, 1968.