- Email: [email protected]
Techniques for segmenting image curves into meaningful descriptions
0031-3203/91 $3.00 + .00 Pergamon Press plc ~) 1991 Pattern Recognition Society
Pattern Recognition, Vol. 24, No. 7, pp. 643-652, 1991 Printed in Great Britain
T E C H N I Q U E S FOR SEGMENTING IMAGE CURVES INTO M E A N I N G F U L DESCRIPTIONS G. A. W. WEST and P. L. ROSIN School of Computing Science, Curtin University of Technology, Bentley, Perth 6102, Western Australia
(Received 25 April 1990; received for publication 14 December 1990) Abstract--This paper describes an approach to the detection of straight lines and circular arcs in images. The algorithm uses a measure based on significance proposed by D. G. Lowe (Three-dimensional object recognition from single two-dimensional images, Artificial Intelligence 31,355-395 (1987)). Edge points a r e processed to produce a description combining lines and arcs. No thresholds are required for either (1) fitting the best lines and arcs to the data or (2) choosing between the best line and arc fits as the most appropriate description. Results are presented demonstrating the performance on complex images. Arc fitting
Line fitting
Tree searching
Feature extraction
1. INTRODUCTION
A key problem area in computer vision is the extraction of meaningful features from images, with the most popular approach based on edges. Edge detectors produce edge points which can be linked together on the basis of image connectivity to form linked lists. To be useful for model matching, these lists of edges must be approximated so as to overcome local noise, and be represented in a more manageable form. The more comprehensive that representation is, the better the performance of the recognition strategy would be. Obtaining straight line descriptions is fairly straightforward, and they have been used by many researchers in computer vision. 0'2) However, less time has been devoted to extracting higher order representations such as arcs, O~) ellipses, curvature primitives ~7"8) and arbitrary curves. In this paper, techniques for extracting arc and line descriptions from edge images will be presented based on the method proposed by Lowe O~ for line detection. Numerous techniques have been proposed for straight line approximations, mainly based on the detection of points at which the pixel strings can be segmented, after which straight lines are fitted to the curves between adjacent points. Most techniques are based on segmenting at points of maximum deviation~9,10) and at curvature extrema/11' 12) These techniques are usually dependent on the parameters chosen which determine the accuracy of fit of the derived representation with the original pixel string, e.g. the length of the "window" over which measurements are made. More recently, a technique has been proposed o3) that detects points based on computing local support for each point, followed by the significance of each point and finally non-maximum suppression. The use of local support overcomes
Image curves
some of the problems of deciding the size of the "window" over which to make measurements of curvature, etc. used in the other techniques. Considering higher order representations, circular arc fitting algorithms have been proposed based on a least squares error criteria. Pridmore et al. 0) classify a pixel string in 3D as a straight line, plane or space curve using orthogonal regression. If the line fit is poor (in terms of accuracy), an arc is fitted. If this fit is also poor the curve is segmented at the extrema of curvature and curvature difference. The algorithm is then applied recursively to the segmented parts of the curve. Again, these procedures require thresholds to control the accuracy and complexity of the representation. Some techniques attempt to mimic human performance. Fischler eta/. (14) describe two methods of determining segmentation points on a curve, the first based on analysing local discontinuities, and the second on analysing process homogeneity. Process in this context is the function used to represent the curve, e.g. a sine wave, so that a discontinuity in processes occurs, for example, at a junction of a sine wave and a straight line. In the first method, points are classified as being in a smooth or noisy part of the line, or as a point of segmentation. In the second method, arc and line fitting are performed by randomly selecting seed points and growing the arc or line process along the curve. Critical or segmentation points are detected at the junctions between the processes. A desirable quality for segmentation algorithms is scale invariance. That is, a pixel string of a particular shape should have the same description whatever its size. This does not occur for methods that use fixed thresholds since the shorter the pixel string, the lower the threshold should be. Hierarchical descriptions of
643
644
G . A . W . WESTand P. L. ROSXN
the strings (e.g. strip trees05) and chain pyramids (16)) are advantageous since lines are represented at multiple scales. However, the large redundant representations can be inconvenient when used for model matching, etc. A more elegant approach has been suggested by Lowe I1) based on replacing fixed thresholds by a measure that accommodates scale invariance. This measure normalises the maximum error by the length of the representation. Lowe's technique has been extended ~17)to deal with circular arcs. An advantage of this method is that a pixel string is segmented into a combination of lines and arcs without requiring any thresholds, producing an approximately scale invariant description. In this paper, the technique is examined in detail for a number of simulated and real images to determine its characteristics and performance. Certain limitations are found, and modifications are suggested to overcome them. Different versions of the algorithm are examined that use least mean square criteria for arc and line fitting. Results are also presented showing the performance of fitting arcs and lines to ellipses. Ellipses are important in computer vision as they can be the representation, on the image plane, of a circle after undergoing a 3D transformation (either perspective or orthographic). Results show that a reasonable representation based on arcs and lines can be obtained. 2. EDGE EXTRACTION
Both the Marr-Hildreth 0s) and the Canny09) edge detectors have been used to extract edges. They have been chosen in preference to other operators as they give long (implicitly) connected sequences of edge points, one pixel wide. The edge points are then linked on the basis of connectivity to form lists. Long connected space curves produce better results, since it is difficult to reliably match higher order representations to small edge sections. 3. LINE DETECTION
In Lowe's work,m a list of edge pixels is hypothesised as being a straight line passing through its end points. The list is segmented into two at its point of maximum deviation from the straight line, and the process is repeated recursively on each of the two lists. Figure 1 shows the position of maximum deviation for a curve. The recursive process is halted when a line segment is smaller than the minimum length representation (three pixels long) or the deviation is less than three pixels. A three pixel length line is the smallest line that can contain non-zero deviations since a two pixel long line will fit a straight line perfectly. These constraints are required to prevent small lines being matched with zero deviation. This would prevent "perfect" small line segments being replaced by longer non-perfect, but intuitively more appropriate, lines. It is unlikely that all the
pixels of a long space curve would lie directly on the approximating straight line segment unless the curve is truly a significant straight line (especially with the effects of noise and quantization). However, edge lists containing only a few pixels often lie exactly on the straight line. The result of the recursive process is a tree in which the curve description at each level is a finer approximation of the level above. As the recursion unwinds, the tree is traversed back up to the root. At each level, a line is retained and passed on as a candidate to the level above if it is more significant than all its children lines. Otherwise, the line is replaced by its children, which are considered as a candidate description by the next level. Once the recursion has completely unwound, a polygonal approximation of the space curve is obtained. The significant rating for lines is based on a pseudopsychological measure of perceptual significance: the longer the line, the greater the maximum deviation tolerated. This favours long lines provided they fit reasonably. Lowe's measure of significance is the ratio of the line segment length divided by the maximum deviation of the space curve from the straight line segment. However, the significance becomes infinite for zero maximum deviations. To prevent divide-by-zero errors we use the reciprocal of the significance. An advantage of this measure is that it is normalised by the line length, and is therefore scale independent. The algorithm produces a high quality, general purpose polygonal approximation. No arbitrary error threshold is required, instead the most appropriate values are chosen dynamically throughout the procedure. For instance, curves of different sizes give very similar approximations (but at different scales). The use of a fixed threshold would approximate the smaller curves crudely, and larger curves with unnecessary detail. For an image of a bicycle, Fig. 2(a) shows pixels detected by the Marr-Hildreth edge detector, and Fig. 2(b) the result of detecting line segments, with the ends of each line indicated by crosses. Note the polygonal approximations of the wheels are a good approximation to the original (Fig. 2(a)). 4. ARC DETECTION
Extending Lowe's work, the same approach can be applied to curve detection. In the Line and Arc Detection Algorithm (LAD),07) lines are found as described in Section 3 by hypothesising sequences of pixels as straight line segments, and recursively refining these descriptions by subdividing at the point of maximum deviation. In a similar manner, these sequences of line segments are then hypothesised as circular arcs, and refined by subdividing at the point of maximum deviation. Arcs are fitted to the vertices of the polygonal approximation rather than to the original pixel values. This speeds up the arc fitting since the vertices provide a great data reduction over
Segmenting image curves
645
Maximum deviation
s t a r t ' ~ ~ - ~
~"
~ ' Line representation
finish
Fig. 1. Breakpoint determination for straight line segmentation.
i
I
t
O0
~
,
I
I
(a)
(b)
Fig. 2. (a) Edge pixels detected using the Marr-Hildreth edge detector for an image of a bicycle (512 x 512); (b) result of Lowe's straight line detection algorithm. The crosses indicate the vertices of the representation.
the edge pixels. The replacement of lines by arcs is simplified since arc breakpoints correspond to line endpoints. Circular arc detection is more complicated than line detection since the specification of an arc requires more parameters because it has more degrees of freedom than a line. To fit a circular arc to a space curve requires the determination of the centre, start and end coordinates, and the radius. Algorithms for general purpose arc fitting have been proposed by Landau (4) and Albano. (s) However, the process can be simplified by applying the condition that arcs are constrained to start and finish at the two endpoints of the space curve of interest. This is the same constraint that Lowe used for line fitting. Given the two endpoints of the arc, three parameters need to be determined, namely the radius and the coordinates of the centre of the circle forming the arc. Minimising the error for the three parameters is a non-trivial task, but the problem can be simplified since the endpoints of the arc are known. The centre point of the circle must be equidistant from the endpoints of the arc, and is therefore constrained to lie on the normal bisecting the line joining the
two endpoints. The algorithm used to determine the parameters is a simple gradient descent procedure.(17) The recursive procedure repeatedly subdivides arcs into subarcs, stopping when the sequence of line segments to be approximated contains fewer than three line segments. This is analogous to the three pixel limit in the polygonal line approximation. Obviously a single line segment should not be replaced by a single arc. Two line segments contain three points and would always be replaced by a perfectly fitting arc, preventing its replacement by larger arcs later. The replacement of lines by arcs is straightforward since arc breakpoints (and therefore subarc endpoints) correspond to line endpoints. That is, an arc always replaces a sequence of complete line segments. In addition, the decision to replace lines by arcs needs no arbitrary threshold. The significance of an arc is calculated in a similar manner to the significance of a line, namely the ratio of the maximum deviation divided by the length of the representation, i.e. the arc. Therefore, since their significances are commensurate, an arc replaces a
646
G . A . W . WESTand P. L. RosIN /i::~A'~r~
,~
~
~
,,~r,
~,~d"~
,e,
~
r ~
w
!
w
~..
X
U,
Fig. 3. Results of the LAD algorithm, with arcs shown as bold lines and crosses delineating the lines and arcs at vertices.
Fig. 4. Result of fitting arcs directly to the pixel data of Fig. 2(a). Crosses show vertices.
sequence of lines and/or arcs if it is more significant than all of them. The arc and line significances are comparable only because the previous procedure results in such good polygonal approximations. If the most appropriate error threshold was not dynamically determined, then small curves would be crudely approximated by line segments, producing arcs with large deviations and thus poor significances. Figure 3 shows line and arc segments (detected by processing the data of Fig. 2(b)) demonstrating how well the algorithm correctly detects circular arcs and straight lines in the same image. Crosses indicate the ends of each line/arc and arcs are shown in bold. Note that for most cases each truly circular feature in the image is represented by a small number of arcs. In fact, for the right hand wheel, the two concentric arcs describing the wheel are each represented by two arcs. Although both Lowe's and the L A D algorithm produce acceptable results, they have several shortcomings, and a number of modifications can be made to attempt to improve their performance. These are now examined.
over another. Secondly, throughout the hierarchical representation lines and arcs are continuous, since the only possible breakpoints within arcs are the vertices of the straight line description. However, it is still worthwhile considering the possibility of fitting arcs directly to the pixel data as this would indicate the best results the L A D algorithm could produce. Comparing arcs fitted to pixel data with the results of the L A D algorithm would allow the performance of the L A D algorithm to be assessed. Figure 4 shows the results of arcs fitted directly to the pixel data. All curves are represented as circular arcs; straight lines are best fitted by arcs with large radii. This can be compared with the results for the LAD algorithm shown in Fig. 3. Treating the arcs with large radii as straight lines for convenience, there are two categories in which arcs are matched to pixel data but not to line data. Either the arc to pixel algorithm is correct, or the L A D algorithm is correct. An example of the first case is the "R" in the bottom left hand corner. Arcs fitted to the pixel data produce a subjectively nice representation, but the line representation from the L A D algorithm is more accurate. Demonstrating the second case, good arc fits are made to the pixel data at the top right hand parts of the left hand wheel. L A D incorrectly classifies these as lines because arc fitting is prevented since the feature is represented by too few line segments to reliably fit an arc to. This also occurs in other parts of the image, e.g. handlebars. Apart from such cases, both methods give similar results. This indicates that arc fitting to the reduced representation of the pixel data (i.e. line segments) does not significantly degrade the quality of output.
$. FITTING ARCS TO PIXEL DATA
The line and arc fitting algorithms described in Sections 3 and 4 work in exactly the same way except that lines are fitted to pixel data and arcs are fitted to line data. Fitting arcs to line data rather than pixel data increases the speed of the algorithm since less data need to be processed; the number of vertices of the polygonal segments is much less than the number of pixels. It is also straightforward to produce a description using both arcs and lines by replacing a number of lines by an arc if the arc is a better representation. This is possible for two reasons. Firstly, lines and arcs have a comparable significance measure that can be used to choose one feature
6. DETERMINING BREAKIN)INTS
The choice of breakpoints for segmentation is an important factor in the appearance and performance
Segmenting image curves a
b
A
c
(a) b
c
647
Although the L A D algorithm breakpoints are suboptimal, in particular for arc fitting, they appear to give results which are a good qualitative approximation of the image. A poor choice of breakpoints is more likely to cause oversegmentation (and an accurate approximation) rather than undersegmentation (and a poor approximation with a loss of information). Pavlidis(22) concluded that even rough curve fitting and goodness of fit measures result in good approximations. We suggest that the same tolerance applies to the selection of breakpoints.
7. L I M I T A T I O N S O F THE I N T E R P R E T A T I O N T R E E
Although both Lowe's and the L A D algorithm appear to give good results on our sample images, certain curves can be generated that result in a poor (b) segmentation. This is caused by the limitations of the a interpretation tree that is generated. As described previously, this is a hierarchical description of the space curve containing a coarse scale representation at the top of the hierarchy and a fine scale representation at the bottom. To produce a single level representation, the tree is traversed; if any line has a greater significance than its parent, then that parent line is replaced by all its children lines. Thus, lines x (c) with a common parent in the tree may be merged and replaced by their parent line. However, situaFig. 5. Situation where correct breakdown is not found: tions arise where it would be desirable to merge lines (a) curve plotted showing significant points; (b) ideal or which are adjacent in the tree but have different required results; (c) result generated by line algorithm. parents. Currently, such merges are not considered. For example, Fig. 5 shows a line between x and y whose first breakpoint is at a. The lines xa and ay are subsequently split at points b and c. This results of subsequent line and arc fitting. This is because the in line segments xb, ba, ac and cy. A better segchoice of the wrong breakpoint cannot be corrected mentation would replace ba and ac by bc, However, later (see Section 7). In the simplest case of line even though bc has a greater significance than both fitting to pixel data, i.e. the straight line between the ba and ac it will never be considered as a replacement curve endpoints, a good breakpoint can easily be since ba and ac are in separate parts of the interfound by choosing the point of maximum deviation pretation tree and therefore bc is not a candidate from the straight line. This is usually a point of line in the tree. high curvature. However, this is not necessarily the To a certain degree, this limitation can be overoptimal point of segmentation, particularly for less come by determining several potential breakpoints constrained line and arc fits. (e.g. using maximum curvature) and then computing More common methods of higher order curve the significances of each resulting representation. segmentation generally use curvature and curvature The breakpoint resulting in the best representation difference to determine breakpoints. Pridmore et would be chosen as the most appropriate for the line al.(3) segment curves into lines and arcs. Breakpoints are located by curvature difference extrema if any at that level. This would result in a better repexist, and otherwise by curvature extrema. Mar- resentation, but at the cost of extra computation. An alternative is the split-and-merge algorithm (9) imont (2°) located breakpoints at extrema and zeros of curvature at a variety of scales. Dynamic pro- which repeatedly considers adjacent pairwise gramming is used to choose the best points, optimis- merges. However, we may require n-wise merges (with n > 2) over two or more branches of the intering the trade-off between the accuracy and conciseness of the representation. Hoffman et al. (7) pretation tree. Ultimately the best method would be to binary split the curve down to the lowest level segment curves at points of curvature minima into a representation as before. An exhaustive search for set of primitives named codons. Lowe (21) segments the best combinations of these lines could then be space curves at points of curvature difference performed by repeatedly combining adjacent lines if extrema into arbitrary curves at various scales.
648
G . A . W . WESTand P. L. ROSIN Maximum deviation
Figl 6. Breakpoint determination for straight line segmentation with least mean square fitting.
b
a
b a
a\
(b)
[
-
C
b
(c) Fig. 7. Situations requiring modification of breakpoint determination: (a) L M S linefitshowing breakpoint detection satisfyingcondition (I);(b) L M S linefitshowing need for condition (2) (lessthan two zero crossings);(c) L M S linefitshowing need for condition (3) (breakpointtoo near one end). the resulting line has a greater significance. C o m putationally, this would be extremely expensive, leading to a combinatorial explosion.
8. LEASTMEANSQUARESLINEANDARC DETECTION The L A D algorithm constrains the chosen representation (line or arc) to go through the endpoints of the data that are being approximated. This ensures that curves are segmented into connected sequences, sets of lines and arcs, and simplifies the line and arc fitting. A unique line joins the endpoints, and the best arc is constrained to be a least mean square fit to the data that pass through the endpoints. However, the least mean squares algorithm (LMS) gives a more accurate curve fit to data. For lines this implies the use of linear regression, and for arcs an LMS algorithm such as that proposed by Landau (4) or Thomas. (6) The use of LMS fitting alters the algorithm by complicating the method of choosing the best breakpoints for subdividing the curves (Fig. 6). For line
fitting, the point of maximum deviation is now not necessarily the best breakpoint as it may occur at one • end of the curve. This problem has been examined by Pavlidis et al. (9) who define breakpoints as lying midway between adjacent points of local maximum deviation. If less than two points of maximum deviation are found, then the breakpoint is defined as the midpoint of the curve. We propose a method that attempts to find the maximum deviation at a point not near the ends of the curve. There are three possible conditions considered in turn, one of which will be satisfied. (1) Determine the positions of the zero crossings nearest the ends of the curve. If these exist, then the breakpoint is detected at the peak or plateau of maximum deviation (with the point of maximum deviation defined as the centre of the plateau) between these points. This is the most likely situation given that an LMS fit line should cut the curve at at least two points. (2) If two zero crossings are not detected then the breakpoint is taken as the point of maximum deviation (using same method as for condition (1)) between the endpoints which is not within a specified distance from an endpoint. In fact, this distance is set to the length of the minimum representation allowed (see Section 3), as this piece of curve will then not be further subdivided. This prevents the detection of breakpoints at ends of the curve. (3) If less than two zero crossings are detected, and if the only breakpoints are near the ends, then the curve is divided at the midpoint. Figure 7 shows situations that justify the above three conditions. Figure 7(a) shows the case where condition (1) is satisfied. Here, the maximum deviation occurs at the extreme end of the curve (point c). Choosing the breakpoint between the zero crossings (point b) gives the best results. Condition (1) is not satisfied for the situation of Fig. 7(b) where only one zero crossing occurs. Although this situation seldom occurs it is necessary to take account of it with condition (2). Finally, condition (3) is used to catch all further unresolved situations such as shown in Fig. 7(c). The measure of error between the straight line and the curve is not now defined as the maximum
Segmenting image curves
649
I
I (a)
(b)
Fig. 8. (a) Representation of circles as straight lines using Lowe's algorithm; (b) representation of circles as straight lines using least mean square fit algorithm.
.
'
.
.
.
)
. . . . .
-
--~30
%a
Fig. 9. Result of using least mean square line fit algorithm on data of Fig. 2(a). Crosses are not used here, for clarity.
deviation partly because, in some cases, the point of maximum deviation is not used as the breakpoint, and partly because in a least mean square fit context there are other quantities that indicate the error in better ways. In this algorithm, the error is defined as the total deviation between the straight line and curve between the endpoints of the straight line. Figure 8 shows the result of using Lowe's and the LMS line fitting technique on a simulated image of a number of concentric circles. The crosses indicate the ends of each line and both algorithms produce similar results. In fact, the LMS line fit results in 33 lines for the largest circle whereas Lowe's algorithm results in 37 lines. However, comparing the accuracy of representation by applying the error measure for LMS line fitting to Lowe's algorithm reveals that the LMS line fit representation is more accurate although
the endpoints of each line do not now connect. For example, for the largest circle the maximum error for Lowe's algorithm was 0.4140, whereas for the LMS fitting algorithm the maximum error was 0.0127. Figure 9 shows the results for a real image. Good results are obtained and are comparable with those of Fig. 2(b) in terms of number of lines. However, as for the circles of Fig. 8, the maximum errors are less. Removing the constraint of lines and arcs passing through endpoints reduces the sensitivity of the algorithm to endpoint position. This is illustrated in Fig. 10, which compares the results, for both algorithms, of fitting lines to a number of sine waves with varying non-integer numbers of wavelengths. For Lowe's algorithm (Fig. 10(a)), in most cases one line is fitted to the data but in two cases two lines are fitted. These occur for the cases where there is almost an integer number of wavelengths meaning that the biggest maximum deviation occurs. Then the significance of one line representing the data is worse than that for the two lines. The LMS fitting algorithm is more stable for the various cases although each waveform is represented by a larger number of lines (Fig. 10(b)). The results of LMS fitting indicate that it is less sensitive to endpoint position. However, experience on many images leads to the conclusion that the variation in the position of endpoints is not of great importance for most applications, and robust results are obtained for both Lowe's algorithm and for LMS fitting. 9. TAIL RECURSION
AND ABSOLUTE
THRESHOLDING
The L A D algorithm relies on tail recursion to
650
G.A.W.
WEST and P. L. ROSIN
V tits _
Ar~ix/_\l_\i,X'l
~j~V V
v
v
~
~ ^ - ^
'\~I-~-I^\-I^\-/~IAW,',VXvr~vAvA'-'I\ I xw \-'/^ '\~I.\~L\-I^\-f^\ I^\ I^\l~,V~V/,,vt,,vr~vAvi\vl f\Vl \~Z \ ~ l _ \ - / _ \ T i ^ \
1^\
In\ J ^ \
l,.,,\ I , , , , \ l ^ t ] t , , \ 7 A V r
~ \ V l ~ l \~/ \~1 \~1 \'1 \-/_\-/.VLA I ^ \ / ^ \ s^\ I ^ \ / . 7~1@-?\~1-\~1\~1 \~1 \~1 \'1 \vl ,~./ t,~,l~r~lr
/\VA\If\Vl\Vi\Vl\Vf\Vl\\q\Vl\Vf\Vl\V/\v/\ \~ k. I\VI\\If\\IAVAVAVA\q\VAVI\\q\vA\~I\VAX' l\Vl\\li\Vl\\li\Vl\\li\\li\Vl~iigil\\li\V/gil9 ,
l\Vl\Vf\Vl\\IAVl\\iAVl\Vl\Vl\Vl\VAVl\V/\ ~' I\VI\\q\vI\VA\q\VA\q\vI\vI\VA\/I\\q\\/A~' I\VA\q\vI\\q\\/I\Vi\VI\VAVA\/i\v/\\q\vI\~ l\\ll\\if\\ll\\if\\il\\i/\\i/\\ll\\il\\li\\ll\\l/\\ll\~
I\VI\\IA\i/\XlAXi/\\IA\i/\v/\\II\XlA\I/\\II\\II.\~ I\\IIt\II\V/t\I/\V/t\I/\V/\VA\
~~-/^\-/AV~,VAVAVAVAVA Y~/~-Z^V~VnVmVAVAVAVAV/~
t\
ix V sx\ / f\ \ / /~ \ l lx \ l /x \ l fx \ l /~ \ l / ~ / / / / ~ / /\ \l l\ \/ tt v / t ~
//\V/
f ,I
v V
(a)
(b)
Fig. 10. (a) Lowe's algorithm for representing sine waves, result superimposed on original data; (b) least mean square algorithm applied to sine waves, original data removed for clarity.
replace lower level combinations of descriptions by one larger more significant description. This is obvious when considering arc fitting, as the only way an arc replaces a collection of lines is through tail recursion. However, the importance of tail recursion for line fitting is not so obvious. Experiments reveal that in the majority of cases tail recursion is seldom used and so the lowest level representation is retained. Without tail recursion the algorithm is effectively one based on the use of absolute thresholding, except that two thresholds are used, one on the maximum error and the other on the minimum length of line (see Section 3).
Fig. 11. LAD algorithm applied to ellipse detection. Arcs in bold.
10. ELLIPSE DETECTION
An important issue in computer vision is the detection of ellipses in images. Ellipses occur in 2D images through the projection (either perspective or orthogonal) of 3D circles onto the image plane. In the absence of 3D data acquisition (stereo, motion, rangefinding), ellipse detection can be used to detect arcs. However, ellipse detection is a more difficult problem than arc detection because of the increased number of parameters or degrees of freedom and the ill-conditioned nature of the problem. Rosin and West (231 recently extended the L A D algorithm to fit a combination of straight lines and elliptic arcs. Still, due to the greater stability of circle fitting it is worth examining the feasibility of fitting circular arcs to elliptic data. Attempting to use the L A D algorithm on simulated ellipses results in a combination of lines and arcs as shown in Fig. 11 (arcs shown in bold); few arcs are produced. Fitting arcs directly to the pixel data produces an approximately scale invariant result with each ellipse represented by several arcs of different radii (see Fig. 12). Intuitively this is as
Fig. 12. Result of arc fitting to pixel data of ellipses. Crosses show vertices between individual arcs.
expected, since a rough description of an ellipse can be made up from arcs of small radius at the ends and arcs of large radius in the middle. However, just fitting arcs to the data is computationally more expensive than the L A D algorithm and will not produce straight line descriptions. Given a priori knowledge of the scene the performance of the L A D algorithm can be improved by weighting the significances of lines or arcs so that there is a bias towards one or the other. In this case we could increase the weighting of arcs causing the ellipses to be represented by more arcs and less lines.
Segmenting image curves The results of just fitting arcs to ellipses suggests that there should be enough structure to enable a higher level process to detect an ellipse from the combination of arcs and possibly lines.
II. CONCLUSIONS
The task of describing edges present in images is an important area of interest and research in the field of computer vision. Many algorithms have been proposed for detecting lines, arcs and higher order curves. In this paper, a method for the detection of lines and arcs, based on a method proposed by Lowe, (I) has been examined. It has the advantage over other methods in that no thresholds are required. Instead, a significance measure is used which is relatively scale invariant and produces perceptually agreeable results. Performance has been shown to depend on a number of issues such as the method of line and arc fitting, dependency on endpoint position, choice of breakpoint location, the structure of the interpretation tree, etc. Some improvements to the basic LAD algorithm have been proposed and demonstrated. These produce more accurate representations and are more robust, but require an increased computational cost. Overall it is concluded that these improvements are minimal, and that for most applications the basic LAD algorithm produces adequate results. 12. SUMMARY
This paper describes an approach to the detection of straight lines and circular arcs in images. The algorithm uses a measure based on significance proposed by Lowe. ~ Edge points are processed to produce a description combining lines and arcs. Initially, the edge data are processed and a line description obtained. Then, the line data are processed and groups of lines replaced if an arc is a better description. No thresholds are required for either (1) fitting the best lines and arcs to the data or (2) choosing between the best line and arc fits as the most appropriate description. Results are presented demonstrating the performance on complex images. The technique is examined in detail for a number of simulated and real images to determine its characteristics and performance. Certain limitations are found, and modifications are suggested to overcome them. Different versions of the algorithm are examined that use least mean square criteria for arc and line fitting. Results are also presented showing the performance of fitting arcs and lines to ellipses. Results show that a reasonable representation based on arcs and lines can be obtained.
651 REFERENCES
1. D. G. Lowe, Three-dimensional object recognition from single two-dimensional images, Artif. Intell. 31, 355-395 (1987). 2. P. L. Rosin, Model driven image understanding: a frame-based approach, Ph.D. thesis, City University, London (1988). 3. A. P. Pridmore, J. Porrill and J. E. W. Mayhew, Segmentation and description of binocularly viewed contours, Image Vision Comput. 5, 132-138 (1987). 4. U. Landau, Estimation of a circular arc centre and its radius, Comput. Vision Graphics Image Process. 38, 317-326 (1987). 5. A. Albano, Representation of digitised contours in terms of conic ares and straight-line segments, Comput. Vision Graphics Image Process. 3, 23-33 (1974). 6. S. M. Thomas and Y. T. Chan, A simple approach for the estimation of circular arc centre and its radius, Comput. Vision Graphics Image Process. 45, 362-370 (1989). 7. D. D. Hoffman and W. A. Richards, Parts of recognition, Cognition 18, 65-96 (1984). 8. H. Asada and M. Brady, The curvature primal sketch, IEEE Tram. PatternAnal. Mach. lntell, g, 2-14 (1986). 9. T. Pavlidis and S. L. Horowitz, Segmentation of plane curves, IEEE Tram. Comput. 23, 860-870 (1974). 10. U. Ramer, An iterative procedure for the polygonal approximation of plane curves, Comput. Graphics Image Process. 1, 244-256 (1972). 11. B. Kruse and C. V. Kameswara Rao, A matched filter technique for corner detection, Proc. 4th Int. Conf. Pattern Recognition (1978). 12. H. Freeman and L. S. Davis, A corner finding algorithm for chain coded curves, 1EEE Trans. Comput. 26,297303 (1977). 13. C-H. Teh and R. T. Chin, A scale-independent point detection algorithm, Proc. IEEE Comp. Soc. Comput. Vision Pattern Recognition, Ann Arbor, MI, U.S.A., pp. 229-234 (1988). 14. M. A. Fischler and R. C. Bolles, Perceptual organisation and curve partitioning, 1EEE Trans. Pattern Anal. Mach. lntell. 8 (1986). 15. D. H. Ballard, Strip trees: a hierarchical representation for curves, Commum ACM 24, 310-321 (1981). 16. P. Meer, E. S. Baugher and A. Rosenfeld, Extraction of trend lines and extrema from multiscale curves, Pattern Recognition 21,217-226 (1988). 17. P. L. Rosin and G. A. W. West, Segmentation of edges into lines and arcs, Image Vision Comput. 7, 109-114 (1989). 18. D. Marr and E. Hildreth, Theory of edge detection, Proc. R. Soc. Lond. B207, 187-217 (1980). 19. J. Canny, A computational approach to edge detection, IEEE Trans. Pattern Anal. Mach. lntell. 8, 679-698 (1986). 20. D. H. Marimont, A representation for image curves, Proc. AAA1-84, Austin, Texas, U.S.A., pp. 237-242 (1984). 21. D. G. Lowe, Organisation of Smooth Image Curves at Multiple Scales, 1EEE 2nd Int. Conf. Comput. Vision, Miami, U.S.A., pp. 558-567 (1988), 22. T. Pavlidis, Curve fitting as a pattern recognition problem, Proc. 6th Int. Conf. Pattern Recognition, pp. 853859 (1982). 23. P. L. Rosin and G. A. W. West, Segmenting curves into elliptic arcs and straight lines, IEEE 3rd Int. Conf. Comput. Vision, Osaka, Japan, pp. 75-78 (1990).
652
G . A . W . WEST and P. L. RosIN
About the Author--GEorF WEST (born 1956) obtained the B.Sc. in Electronic Engineering and Ph.D. in Systems Engineering at the City University, London, in 1978 and 1982, respectively. The subject of his Ph.D. research was bare printed circuitboard inspection. Between 1982 and 1987 he was employed as a post-doctoral Research Fellow at the City University where he researched into various visual inspection problems. In 1987 he was awarded a five year Advanced Fellowship to research into CAD guided inspection of 3D machine components. He is presently employed as a lecturer at the Curtin University of Technology where his research interests are CAD guided inspection, sub-pixel measurements, curved feature extraction and adaptive vision systems. He is a Chartered Engineer, a member of the Instituteof Electrical Engineers and the British Machine Vision Association. He is the author of numerous scientificpublications.
About the Author--PAuL RosIN was born in Glasgow in 1963. He gained the B.Sc. degree in Computer Science and Microprocessor Systems in 1984 at Strathclyde University, Glasgow, and a Ph.D. degree in Information Engineering at City University, London, in 1988. He was a Research Fellow at City University, developing a prototype system for the Home Office to detect and classify intruders in image sequences. While at the Department of Neurology at Guy's Hospital, London, he worked on the Alvey project "Model-Based Interpretation of Radiological Images". He is currently in the Department of Computing Science at Curtin University where his research interests are frame-based vision systems, image representation, curve representation and perceptual grouping.
Pattern Recognition, Vol. 24, No. 7, pp. 643-652, 1991 Printed in Great Britain
T E C H N I Q U E S FOR SEGMENTING IMAGE CURVES INTO M E A N I N G F U L DESCRIPTIONS G. A. W. WEST and P. L. ROSIN School of Computing Science, Curtin University of Technology, Bentley, Perth 6102, Western Australia
(Received 25 April 1990; received for publication 14 December 1990) Abstract--This paper describes an approach to the detection of straight lines and circular arcs in images. The algorithm uses a measure based on significance proposed by D. G. Lowe (Three-dimensional object recognition from single two-dimensional images, Artificial Intelligence 31,355-395 (1987)). Edge points a r e processed to produce a description combining lines and arcs. No thresholds are required for either (1) fitting the best lines and arcs to the data or (2) choosing between the best line and arc fits as the most appropriate description. Results are presented demonstrating the performance on complex images. Arc fitting
Line fitting
Tree searching
Feature extraction
1. INTRODUCTION
A key problem area in computer vision is the extraction of meaningful features from images, with the most popular approach based on edges. Edge detectors produce edge points which can be linked together on the basis of image connectivity to form linked lists. To be useful for model matching, these lists of edges must be approximated so as to overcome local noise, and be represented in a more manageable form. The more comprehensive that representation is, the better the performance of the recognition strategy would be. Obtaining straight line descriptions is fairly straightforward, and they have been used by many researchers in computer vision. 0'2) However, less time has been devoted to extracting higher order representations such as arcs, O~) ellipses, curvature primitives ~7"8) and arbitrary curves. In this paper, techniques for extracting arc and line descriptions from edge images will be presented based on the method proposed by Lowe O~ for line detection. Numerous techniques have been proposed for straight line approximations, mainly based on the detection of points at which the pixel strings can be segmented, after which straight lines are fitted to the curves between adjacent points. Most techniques are based on segmenting at points of maximum deviation~9,10) and at curvature extrema/11' 12) These techniques are usually dependent on the parameters chosen which determine the accuracy of fit of the derived representation with the original pixel string, e.g. the length of the "window" over which measurements are made. More recently, a technique has been proposed o3) that detects points based on computing local support for each point, followed by the significance of each point and finally non-maximum suppression. The use of local support overcomes
Image curves
some of the problems of deciding the size of the "window" over which to make measurements of curvature, etc. used in the other techniques. Considering higher order representations, circular arc fitting algorithms have been proposed based on a least squares error criteria. Pridmore et al. 0) classify a pixel string in 3D as a straight line, plane or space curve using orthogonal regression. If the line fit is poor (in terms of accuracy), an arc is fitted. If this fit is also poor the curve is segmented at the extrema of curvature and curvature difference. The algorithm is then applied recursively to the segmented parts of the curve. Again, these procedures require thresholds to control the accuracy and complexity of the representation. Some techniques attempt to mimic human performance. Fischler eta/. (14) describe two methods of determining segmentation points on a curve, the first based on analysing local discontinuities, and the second on analysing process homogeneity. Process in this context is the function used to represent the curve, e.g. a sine wave, so that a discontinuity in processes occurs, for example, at a junction of a sine wave and a straight line. In the first method, points are classified as being in a smooth or noisy part of the line, or as a point of segmentation. In the second method, arc and line fitting are performed by randomly selecting seed points and growing the arc or line process along the curve. Critical or segmentation points are detected at the junctions between the processes. A desirable quality for segmentation algorithms is scale invariance. That is, a pixel string of a particular shape should have the same description whatever its size. This does not occur for methods that use fixed thresholds since the shorter the pixel string, the lower the threshold should be. Hierarchical descriptions of
643
644
G . A . W . WESTand P. L. ROSXN
the strings (e.g. strip trees05) and chain pyramids (16)) are advantageous since lines are represented at multiple scales. However, the large redundant representations can be inconvenient when used for model matching, etc. A more elegant approach has been suggested by Lowe I1) based on replacing fixed thresholds by a measure that accommodates scale invariance. This measure normalises the maximum error by the length of the representation. Lowe's technique has been extended ~17)to deal with circular arcs. An advantage of this method is that a pixel string is segmented into a combination of lines and arcs without requiring any thresholds, producing an approximately scale invariant description. In this paper, the technique is examined in detail for a number of simulated and real images to determine its characteristics and performance. Certain limitations are found, and modifications are suggested to overcome them. Different versions of the algorithm are examined that use least mean square criteria for arc and line fitting. Results are also presented showing the performance of fitting arcs and lines to ellipses. Ellipses are important in computer vision as they can be the representation, on the image plane, of a circle after undergoing a 3D transformation (either perspective or orthographic). Results show that a reasonable representation based on arcs and lines can be obtained. 2. EDGE EXTRACTION
Both the Marr-Hildreth 0s) and the Canny09) edge detectors have been used to extract edges. They have been chosen in preference to other operators as they give long (implicitly) connected sequences of edge points, one pixel wide. The edge points are then linked on the basis of connectivity to form lists. Long connected space curves produce better results, since it is difficult to reliably match higher order representations to small edge sections. 3. LINE DETECTION
In Lowe's work,m a list of edge pixels is hypothesised as being a straight line passing through its end points. The list is segmented into two at its point of maximum deviation from the straight line, and the process is repeated recursively on each of the two lists. Figure 1 shows the position of maximum deviation for a curve. The recursive process is halted when a line segment is smaller than the minimum length representation (three pixels long) or the deviation is less than three pixels. A three pixel length line is the smallest line that can contain non-zero deviations since a two pixel long line will fit a straight line perfectly. These constraints are required to prevent small lines being matched with zero deviation. This would prevent "perfect" small line segments being replaced by longer non-perfect, but intuitively more appropriate, lines. It is unlikely that all the
pixels of a long space curve would lie directly on the approximating straight line segment unless the curve is truly a significant straight line (especially with the effects of noise and quantization). However, edge lists containing only a few pixels often lie exactly on the straight line. The result of the recursive process is a tree in which the curve description at each level is a finer approximation of the level above. As the recursion unwinds, the tree is traversed back up to the root. At each level, a line is retained and passed on as a candidate to the level above if it is more significant than all its children lines. Otherwise, the line is replaced by its children, which are considered as a candidate description by the next level. Once the recursion has completely unwound, a polygonal approximation of the space curve is obtained. The significant rating for lines is based on a pseudopsychological measure of perceptual significance: the longer the line, the greater the maximum deviation tolerated. This favours long lines provided they fit reasonably. Lowe's measure of significance is the ratio of the line segment length divided by the maximum deviation of the space curve from the straight line segment. However, the significance becomes infinite for zero maximum deviations. To prevent divide-by-zero errors we use the reciprocal of the significance. An advantage of this measure is that it is normalised by the line length, and is therefore scale independent. The algorithm produces a high quality, general purpose polygonal approximation. No arbitrary error threshold is required, instead the most appropriate values are chosen dynamically throughout the procedure. For instance, curves of different sizes give very similar approximations (but at different scales). The use of a fixed threshold would approximate the smaller curves crudely, and larger curves with unnecessary detail. For an image of a bicycle, Fig. 2(a) shows pixels detected by the Marr-Hildreth edge detector, and Fig. 2(b) the result of detecting line segments, with the ends of each line indicated by crosses. Note the polygonal approximations of the wheels are a good approximation to the original (Fig. 2(a)). 4. ARC DETECTION
Extending Lowe's work, the same approach can be applied to curve detection. In the Line and Arc Detection Algorithm (LAD),07) lines are found as described in Section 3 by hypothesising sequences of pixels as straight line segments, and recursively refining these descriptions by subdividing at the point of maximum deviation. In a similar manner, these sequences of line segments are then hypothesised as circular arcs, and refined by subdividing at the point of maximum deviation. Arcs are fitted to the vertices of the polygonal approximation rather than to the original pixel values. This speeds up the arc fitting since the vertices provide a great data reduction over
Segmenting image curves
645
Maximum deviation
s t a r t ' ~ ~ - ~
~"
~ ' Line representation
finish
Fig. 1. Breakpoint determination for straight line segmentation.
i
I
t
O0
~
,
I
I
(a)
(b)
Fig. 2. (a) Edge pixels detected using the Marr-Hildreth edge detector for an image of a bicycle (512 x 512); (b) result of Lowe's straight line detection algorithm. The crosses indicate the vertices of the representation.
the edge pixels. The replacement of lines by arcs is simplified since arc breakpoints correspond to line endpoints. Circular arc detection is more complicated than line detection since the specification of an arc requires more parameters because it has more degrees of freedom than a line. To fit a circular arc to a space curve requires the determination of the centre, start and end coordinates, and the radius. Algorithms for general purpose arc fitting have been proposed by Landau (4) and Albano. (s) However, the process can be simplified by applying the condition that arcs are constrained to start and finish at the two endpoints of the space curve of interest. This is the same constraint that Lowe used for line fitting. Given the two endpoints of the arc, three parameters need to be determined, namely the radius and the coordinates of the centre of the circle forming the arc. Minimising the error for the three parameters is a non-trivial task, but the problem can be simplified since the endpoints of the arc are known. The centre point of the circle must be equidistant from the endpoints of the arc, and is therefore constrained to lie on the normal bisecting the line joining the
two endpoints. The algorithm used to determine the parameters is a simple gradient descent procedure.(17) The recursive procedure repeatedly subdivides arcs into subarcs, stopping when the sequence of line segments to be approximated contains fewer than three line segments. This is analogous to the three pixel limit in the polygonal line approximation. Obviously a single line segment should not be replaced by a single arc. Two line segments contain three points and would always be replaced by a perfectly fitting arc, preventing its replacement by larger arcs later. The replacement of lines by arcs is straightforward since arc breakpoints (and therefore subarc endpoints) correspond to line endpoints. That is, an arc always replaces a sequence of complete line segments. In addition, the decision to replace lines by arcs needs no arbitrary threshold. The significance of an arc is calculated in a similar manner to the significance of a line, namely the ratio of the maximum deviation divided by the length of the representation, i.e. the arc. Therefore, since their significances are commensurate, an arc replaces a
646
G . A . W . WESTand P. L. RosIN /i::~A'~r~
,~
~
~
,,~r,
~,~d"~
,e,
~
r ~
w
!
w
~..
X
U,
Fig. 3. Results of the LAD algorithm, with arcs shown as bold lines and crosses delineating the lines and arcs at vertices.
Fig. 4. Result of fitting arcs directly to the pixel data of Fig. 2(a). Crosses show vertices.
sequence of lines and/or arcs if it is more significant than all of them. The arc and line significances are comparable only because the previous procedure results in such good polygonal approximations. If the most appropriate error threshold was not dynamically determined, then small curves would be crudely approximated by line segments, producing arcs with large deviations and thus poor significances. Figure 3 shows line and arc segments (detected by processing the data of Fig. 2(b)) demonstrating how well the algorithm correctly detects circular arcs and straight lines in the same image. Crosses indicate the ends of each line/arc and arcs are shown in bold. Note that for most cases each truly circular feature in the image is represented by a small number of arcs. In fact, for the right hand wheel, the two concentric arcs describing the wheel are each represented by two arcs. Although both Lowe's and the L A D algorithm produce acceptable results, they have several shortcomings, and a number of modifications can be made to attempt to improve their performance. These are now examined.
over another. Secondly, throughout the hierarchical representation lines and arcs are continuous, since the only possible breakpoints within arcs are the vertices of the straight line description. However, it is still worthwhile considering the possibility of fitting arcs directly to the pixel data as this would indicate the best results the L A D algorithm could produce. Comparing arcs fitted to pixel data with the results of the L A D algorithm would allow the performance of the L A D algorithm to be assessed. Figure 4 shows the results of arcs fitted directly to the pixel data. All curves are represented as circular arcs; straight lines are best fitted by arcs with large radii. This can be compared with the results for the LAD algorithm shown in Fig. 3. Treating the arcs with large radii as straight lines for convenience, there are two categories in which arcs are matched to pixel data but not to line data. Either the arc to pixel algorithm is correct, or the L A D algorithm is correct. An example of the first case is the "R" in the bottom left hand corner. Arcs fitted to the pixel data produce a subjectively nice representation, but the line representation from the L A D algorithm is more accurate. Demonstrating the second case, good arc fits are made to the pixel data at the top right hand parts of the left hand wheel. L A D incorrectly classifies these as lines because arc fitting is prevented since the feature is represented by too few line segments to reliably fit an arc to. This also occurs in other parts of the image, e.g. handlebars. Apart from such cases, both methods give similar results. This indicates that arc fitting to the reduced representation of the pixel data (i.e. line segments) does not significantly degrade the quality of output.
$. FITTING ARCS TO PIXEL DATA
The line and arc fitting algorithms described in Sections 3 and 4 work in exactly the same way except that lines are fitted to pixel data and arcs are fitted to line data. Fitting arcs to line data rather than pixel data increases the speed of the algorithm since less data need to be processed; the number of vertices of the polygonal segments is much less than the number of pixels. It is also straightforward to produce a description using both arcs and lines by replacing a number of lines by an arc if the arc is a better representation. This is possible for two reasons. Firstly, lines and arcs have a comparable significance measure that can be used to choose one feature
6. DETERMINING BREAKIN)INTS
The choice of breakpoints for segmentation is an important factor in the appearance and performance
Segmenting image curves a
b
A
c
(a) b
c
647
Although the L A D algorithm breakpoints are suboptimal, in particular for arc fitting, they appear to give results which are a good qualitative approximation of the image. A poor choice of breakpoints is more likely to cause oversegmentation (and an accurate approximation) rather than undersegmentation (and a poor approximation with a loss of information). Pavlidis(22) concluded that even rough curve fitting and goodness of fit measures result in good approximations. We suggest that the same tolerance applies to the selection of breakpoints.
7. L I M I T A T I O N S O F THE I N T E R P R E T A T I O N T R E E
Although both Lowe's and the L A D algorithm appear to give good results on our sample images, certain curves can be generated that result in a poor (b) segmentation. This is caused by the limitations of the a interpretation tree that is generated. As described previously, this is a hierarchical description of the space curve containing a coarse scale representation at the top of the hierarchy and a fine scale representation at the bottom. To produce a single level representation, the tree is traversed; if any line has a greater significance than its parent, then that parent line is replaced by all its children lines. Thus, lines x (c) with a common parent in the tree may be merged and replaced by their parent line. However, situaFig. 5. Situation where correct breakdown is not found: tions arise where it would be desirable to merge lines (a) curve plotted showing significant points; (b) ideal or which are adjacent in the tree but have different required results; (c) result generated by line algorithm. parents. Currently, such merges are not considered. For example, Fig. 5 shows a line between x and y whose first breakpoint is at a. The lines xa and ay are subsequently split at points b and c. This results of subsequent line and arc fitting. This is because the in line segments xb, ba, ac and cy. A better segchoice of the wrong breakpoint cannot be corrected mentation would replace ba and ac by bc, However, later (see Section 7). In the simplest case of line even though bc has a greater significance than both fitting to pixel data, i.e. the straight line between the ba and ac it will never be considered as a replacement curve endpoints, a good breakpoint can easily be since ba and ac are in separate parts of the interfound by choosing the point of maximum deviation pretation tree and therefore bc is not a candidate from the straight line. This is usually a point of line in the tree. high curvature. However, this is not necessarily the To a certain degree, this limitation can be overoptimal point of segmentation, particularly for less come by determining several potential breakpoints constrained line and arc fits. (e.g. using maximum curvature) and then computing More common methods of higher order curve the significances of each resulting representation. segmentation generally use curvature and curvature The breakpoint resulting in the best representation difference to determine breakpoints. Pridmore et would be chosen as the most appropriate for the line al.(3) segment curves into lines and arcs. Breakpoints are located by curvature difference extrema if any at that level. This would result in a better repexist, and otherwise by curvature extrema. Mar- resentation, but at the cost of extra computation. An alternative is the split-and-merge algorithm (9) imont (2°) located breakpoints at extrema and zeros of curvature at a variety of scales. Dynamic pro- which repeatedly considers adjacent pairwise gramming is used to choose the best points, optimis- merges. However, we may require n-wise merges (with n > 2) over two or more branches of the intering the trade-off between the accuracy and conciseness of the representation. Hoffman et al. (7) pretation tree. Ultimately the best method would be to binary split the curve down to the lowest level segment curves at points of curvature minima into a representation as before. An exhaustive search for set of primitives named codons. Lowe (21) segments the best combinations of these lines could then be space curves at points of curvature difference performed by repeatedly combining adjacent lines if extrema into arbitrary curves at various scales.
648
G . A . W . WESTand P. L. ROSIN Maximum deviation
Figl 6. Breakpoint determination for straight line segmentation with least mean square fitting.
b
a
b a
a\
(b)
[
-
C
b
(c) Fig. 7. Situations requiring modification of breakpoint determination: (a) L M S linefitshowing breakpoint detection satisfyingcondition (I);(b) L M S linefitshowing need for condition (2) (lessthan two zero crossings);(c) L M S linefitshowing need for condition (3) (breakpointtoo near one end). the resulting line has a greater significance. C o m putationally, this would be extremely expensive, leading to a combinatorial explosion.
8. LEASTMEANSQUARESLINEANDARC DETECTION The L A D algorithm constrains the chosen representation (line or arc) to go through the endpoints of the data that are being approximated. This ensures that curves are segmented into connected sequences, sets of lines and arcs, and simplifies the line and arc fitting. A unique line joins the endpoints, and the best arc is constrained to be a least mean square fit to the data that pass through the endpoints. However, the least mean squares algorithm (LMS) gives a more accurate curve fit to data. For lines this implies the use of linear regression, and for arcs an LMS algorithm such as that proposed by Landau (4) or Thomas. (6) The use of LMS fitting alters the algorithm by complicating the method of choosing the best breakpoints for subdividing the curves (Fig. 6). For line
fitting, the point of maximum deviation is now not necessarily the best breakpoint as it may occur at one • end of the curve. This problem has been examined by Pavlidis et al. (9) who define breakpoints as lying midway between adjacent points of local maximum deviation. If less than two points of maximum deviation are found, then the breakpoint is defined as the midpoint of the curve. We propose a method that attempts to find the maximum deviation at a point not near the ends of the curve. There are three possible conditions considered in turn, one of which will be satisfied. (1) Determine the positions of the zero crossings nearest the ends of the curve. If these exist, then the breakpoint is detected at the peak or plateau of maximum deviation (with the point of maximum deviation defined as the centre of the plateau) between these points. This is the most likely situation given that an LMS fit line should cut the curve at at least two points. (2) If two zero crossings are not detected then the breakpoint is taken as the point of maximum deviation (using same method as for condition (1)) between the endpoints which is not within a specified distance from an endpoint. In fact, this distance is set to the length of the minimum representation allowed (see Section 3), as this piece of curve will then not be further subdivided. This prevents the detection of breakpoints at ends of the curve. (3) If less than two zero crossings are detected, and if the only breakpoints are near the ends, then the curve is divided at the midpoint. Figure 7 shows situations that justify the above three conditions. Figure 7(a) shows the case where condition (1) is satisfied. Here, the maximum deviation occurs at the extreme end of the curve (point c). Choosing the breakpoint between the zero crossings (point b) gives the best results. Condition (1) is not satisfied for the situation of Fig. 7(b) where only one zero crossing occurs. Although this situation seldom occurs it is necessary to take account of it with condition (2). Finally, condition (3) is used to catch all further unresolved situations such as shown in Fig. 7(c). The measure of error between the straight line and the curve is not now defined as the maximum
Segmenting image curves
649
I
I (a)
(b)
Fig. 8. (a) Representation of circles as straight lines using Lowe's algorithm; (b) representation of circles as straight lines using least mean square fit algorithm.
.
'
.
.
.
)
. . . . .
-
--~30
%a
Fig. 9. Result of using least mean square line fit algorithm on data of Fig. 2(a). Crosses are not used here, for clarity.
deviation partly because, in some cases, the point of maximum deviation is not used as the breakpoint, and partly because in a least mean square fit context there are other quantities that indicate the error in better ways. In this algorithm, the error is defined as the total deviation between the straight line and curve between the endpoints of the straight line. Figure 8 shows the result of using Lowe's and the LMS line fitting technique on a simulated image of a number of concentric circles. The crosses indicate the ends of each line and both algorithms produce similar results. In fact, the LMS line fit results in 33 lines for the largest circle whereas Lowe's algorithm results in 37 lines. However, comparing the accuracy of representation by applying the error measure for LMS line fitting to Lowe's algorithm reveals that the LMS line fit representation is more accurate although
the endpoints of each line do not now connect. For example, for the largest circle the maximum error for Lowe's algorithm was 0.4140, whereas for the LMS fitting algorithm the maximum error was 0.0127. Figure 9 shows the results for a real image. Good results are obtained and are comparable with those of Fig. 2(b) in terms of number of lines. However, as for the circles of Fig. 8, the maximum errors are less. Removing the constraint of lines and arcs passing through endpoints reduces the sensitivity of the algorithm to endpoint position. This is illustrated in Fig. 10, which compares the results, for both algorithms, of fitting lines to a number of sine waves with varying non-integer numbers of wavelengths. For Lowe's algorithm (Fig. 10(a)), in most cases one line is fitted to the data but in two cases two lines are fitted. These occur for the cases where there is almost an integer number of wavelengths meaning that the biggest maximum deviation occurs. Then the significance of one line representing the data is worse than that for the two lines. The LMS fitting algorithm is more stable for the various cases although each waveform is represented by a larger number of lines (Fig. 10(b)). The results of LMS fitting indicate that it is less sensitive to endpoint position. However, experience on many images leads to the conclusion that the variation in the position of endpoints is not of great importance for most applications, and robust results are obtained for both Lowe's algorithm and for LMS fitting. 9. TAIL RECURSION
AND ABSOLUTE
THRESHOLDING
The L A D algorithm relies on tail recursion to
650
G.A.W.
WEST and P. L. ROSIN
V tits _
Ar~ix/_\l_\i,X'l
~j~V V
v
v
~
~ ^ - ^
'\~I-~-I^\-I^\-/~IAW,',VXvr~vAvA'-'I\ I xw \-'/^ '\~I.\~L\-I^\-f^\ I^\ I^\l~,V~V/,,vt,,vr~vAvi\vl f\Vl \~Z \ ~ l _ \ - / _ \ T i ^ \
1^\
In\ J ^ \
l,.,,\ I , , , , \ l ^ t ] t , , \ 7 A V r
~ \ V l ~ l \~/ \~1 \~1 \'1 \-/_\-/.VLA I ^ \ / ^ \ s^\ I ^ \ / . 7~1@-?\~1-\~1\~1 \~1 \~1 \'1 \vl ,~./ t,~,l~r~lr
/\VA\If\Vl\Vi\Vl\Vf\Vl\\q\Vl\Vf\Vl\V/\v/\ \~ k. I\VI\\If\\IAVAVAVA\q\VAVI\\q\vA\~I\VAX' l\Vl\\li\Vl\\li\Vl\\li\\li\Vl~iigil\\li\V/gil9 ,
l\Vl\Vf\Vl\\IAVl\\iAVl\Vl\Vl\Vl\VAVl\V/\ ~' I\VI\\q\vI\VA\q\VA\q\vI\vI\VA\/I\\q\\/A~' I\VA\q\vI\\q\\/I\Vi\VI\VAVA\/i\v/\\q\vI\~ l\\ll\\if\\ll\\if\\il\\i/\\i/\\ll\\il\\li\\ll\\l/\\ll\~
I\VI\\IA\i/\XlAXi/\\IA\i/\v/\\II\XlA\I/\\II\\II.\~ I\\IIt\II\V/t\I/\V/t\I/\V/\VA\
~~-/^\-/AV~,VAVAVAVAVA Y~/~-Z^V~VnVmVAVAVAVAV/~
t\
ix V sx\ / f\ \ / /~ \ l lx \ l /x \ l fx \ l /~ \ l / ~ / / / / ~ / /\ \l l\ \/ tt v / t ~
//\V/
f ,I
v V
(a)
(b)
Fig. 10. (a) Lowe's algorithm for representing sine waves, result superimposed on original data; (b) least mean square algorithm applied to sine waves, original data removed for clarity.
replace lower level combinations of descriptions by one larger more significant description. This is obvious when considering arc fitting, as the only way an arc replaces a collection of lines is through tail recursion. However, the importance of tail recursion for line fitting is not so obvious. Experiments reveal that in the majority of cases tail recursion is seldom used and so the lowest level representation is retained. Without tail recursion the algorithm is effectively one based on the use of absolute thresholding, except that two thresholds are used, one on the maximum error and the other on the minimum length of line (see Section 3).
Fig. 11. LAD algorithm applied to ellipse detection. Arcs in bold.
10. ELLIPSE DETECTION
An important issue in computer vision is the detection of ellipses in images. Ellipses occur in 2D images through the projection (either perspective or orthogonal) of 3D circles onto the image plane. In the absence of 3D data acquisition (stereo, motion, rangefinding), ellipse detection can be used to detect arcs. However, ellipse detection is a more difficult problem than arc detection because of the increased number of parameters or degrees of freedom and the ill-conditioned nature of the problem. Rosin and West (231 recently extended the L A D algorithm to fit a combination of straight lines and elliptic arcs. Still, due to the greater stability of circle fitting it is worth examining the feasibility of fitting circular arcs to elliptic data. Attempting to use the L A D algorithm on simulated ellipses results in a combination of lines and arcs as shown in Fig. 11 (arcs shown in bold); few arcs are produced. Fitting arcs directly to the pixel data produces an approximately scale invariant result with each ellipse represented by several arcs of different radii (see Fig. 12). Intuitively this is as
Fig. 12. Result of arc fitting to pixel data of ellipses. Crosses show vertices between individual arcs.
expected, since a rough description of an ellipse can be made up from arcs of small radius at the ends and arcs of large radius in the middle. However, just fitting arcs to the data is computationally more expensive than the L A D algorithm and will not produce straight line descriptions. Given a priori knowledge of the scene the performance of the L A D algorithm can be improved by weighting the significances of lines or arcs so that there is a bias towards one or the other. In this case we could increase the weighting of arcs causing the ellipses to be represented by more arcs and less lines.
Segmenting image curves The results of just fitting arcs to ellipses suggests that there should be enough structure to enable a higher level process to detect an ellipse from the combination of arcs and possibly lines.
II. CONCLUSIONS
The task of describing edges present in images is an important area of interest and research in the field of computer vision. Many algorithms have been proposed for detecting lines, arcs and higher order curves. In this paper, a method for the detection of lines and arcs, based on a method proposed by Lowe, (I) has been examined. It has the advantage over other methods in that no thresholds are required. Instead, a significance measure is used which is relatively scale invariant and produces perceptually agreeable results. Performance has been shown to depend on a number of issues such as the method of line and arc fitting, dependency on endpoint position, choice of breakpoint location, the structure of the interpretation tree, etc. Some improvements to the basic LAD algorithm have been proposed and demonstrated. These produce more accurate representations and are more robust, but require an increased computational cost. Overall it is concluded that these improvements are minimal, and that for most applications the basic LAD algorithm produces adequate results. 12. SUMMARY
This paper describes an approach to the detection of straight lines and circular arcs in images. The algorithm uses a measure based on significance proposed by Lowe. ~ Edge points are processed to produce a description combining lines and arcs. Initially, the edge data are processed and a line description obtained. Then, the line data are processed and groups of lines replaced if an arc is a better description. No thresholds are required for either (1) fitting the best lines and arcs to the data or (2) choosing between the best line and arc fits as the most appropriate description. Results are presented demonstrating the performance on complex images. The technique is examined in detail for a number of simulated and real images to determine its characteristics and performance. Certain limitations are found, and modifications are suggested to overcome them. Different versions of the algorithm are examined that use least mean square criteria for arc and line fitting. Results are also presented showing the performance of fitting arcs and lines to ellipses. Results show that a reasonable representation based on arcs and lines can be obtained.
651 REFERENCES
1. D. G. Lowe, Three-dimensional object recognition from single two-dimensional images, Artif. Intell. 31, 355-395 (1987). 2. P. L. Rosin, Model driven image understanding: a frame-based approach, Ph.D. thesis, City University, London (1988). 3. A. P. Pridmore, J. Porrill and J. E. W. Mayhew, Segmentation and description of binocularly viewed contours, Image Vision Comput. 5, 132-138 (1987). 4. U. Landau, Estimation of a circular arc centre and its radius, Comput. Vision Graphics Image Process. 38, 317-326 (1987). 5. A. Albano, Representation of digitised contours in terms of conic ares and straight-line segments, Comput. Vision Graphics Image Process. 3, 23-33 (1974). 6. S. M. Thomas and Y. T. Chan, A simple approach for the estimation of circular arc centre and its radius, Comput. Vision Graphics Image Process. 45, 362-370 (1989). 7. D. D. Hoffman and W. A. Richards, Parts of recognition, Cognition 18, 65-96 (1984). 8. H. Asada and M. Brady, The curvature primal sketch, IEEE Tram. PatternAnal. Mach. lntell, g, 2-14 (1986). 9. T. Pavlidis and S. L. Horowitz, Segmentation of plane curves, IEEE Tram. Comput. 23, 860-870 (1974). 10. U. Ramer, An iterative procedure for the polygonal approximation of plane curves, Comput. Graphics Image Process. 1, 244-256 (1972). 11. B. Kruse and C. V. Kameswara Rao, A matched filter technique for corner detection, Proc. 4th Int. Conf. Pattern Recognition (1978). 12. H. Freeman and L. S. Davis, A corner finding algorithm for chain coded curves, 1EEE Trans. Comput. 26,297303 (1977). 13. C-H. Teh and R. T. Chin, A scale-independent point detection algorithm, Proc. IEEE Comp. Soc. Comput. Vision Pattern Recognition, Ann Arbor, MI, U.S.A., pp. 229-234 (1988). 14. M. A. Fischler and R. C. Bolles, Perceptual organisation and curve partitioning, 1EEE Trans. Pattern Anal. Mach. lntell. 8 (1986). 15. D. H. Ballard, Strip trees: a hierarchical representation for curves, Commum ACM 24, 310-321 (1981). 16. P. Meer, E. S. Baugher and A. Rosenfeld, Extraction of trend lines and extrema from multiscale curves, Pattern Recognition 21,217-226 (1988). 17. P. L. Rosin and G. A. W. West, Segmentation of edges into lines and arcs, Image Vision Comput. 7, 109-114 (1989). 18. D. Marr and E. Hildreth, Theory of edge detection, Proc. R. Soc. Lond. B207, 187-217 (1980). 19. J. Canny, A computational approach to edge detection, IEEE Trans. Pattern Anal. Mach. lntell. 8, 679-698 (1986). 20. D. H. Marimont, A representation for image curves, Proc. AAA1-84, Austin, Texas, U.S.A., pp. 237-242 (1984). 21. D. G. Lowe, Organisation of Smooth Image Curves at Multiple Scales, 1EEE 2nd Int. Conf. Comput. Vision, Miami, U.S.A., pp. 558-567 (1988), 22. T. Pavlidis, Curve fitting as a pattern recognition problem, Proc. 6th Int. Conf. Pattern Recognition, pp. 853859 (1982). 23. P. L. Rosin and G. A. W. West, Segmenting curves into elliptic arcs and straight lines, IEEE 3rd Int. Conf. Comput. Vision, Osaka, Japan, pp. 75-78 (1990).
652
G . A . W . WEST and P. L. RosIN
About the Author--GEorF WEST (born 1956) obtained the B.Sc. in Electronic Engineering and Ph.D. in Systems Engineering at the City University, London, in 1978 and 1982, respectively. The subject of his Ph.D. research was bare printed circuitboard inspection. Between 1982 and 1987 he was employed as a post-doctoral Research Fellow at the City University where he researched into various visual inspection problems. In 1987 he was awarded a five year Advanced Fellowship to research into CAD guided inspection of 3D machine components. He is presently employed as a lecturer at the Curtin University of Technology where his research interests are CAD guided inspection, sub-pixel measurements, curved feature extraction and adaptive vision systems. He is a Chartered Engineer, a member of the Instituteof Electrical Engineers and the British Machine Vision Association. He is the author of numerous scientificpublications.
About the Author--PAuL RosIN was born in Glasgow in 1963. He gained the B.Sc. degree in Computer Science and Microprocessor Systems in 1984 at Strathclyde University, Glasgow, and a Ph.D. degree in Information Engineering at City University, London, in 1988. He was a Research Fellow at City University, developing a prototype system for the Home Office to detect and classify intruders in image sequences. While at the Department of Neurology at Guy's Hospital, London, he worked on the Alvey project "Model-Based Interpretation of Radiological Images". He is currently in the Department of Computing Science at Curtin University where his research interests are frame-based vision systems, image representation, curve representation and perceptual grouping.