- Email: [email protected]
Consensus sets for affine transformation uncertainty polytopes
ARTICLE IN PRESS
Computers & Graphics 30 (2006) 77–85 www.elsevier.com/locate/cag
Consensus sets for affine transformation uncertainty polytopes Peter Veelaert, Kristof Teelen Hogeschool Gent, University of Ghent, Schoonmeersstraat 52, B9000 Ghent, Belgium
Abstract We introduce an uncertainty model for geometric transformations that can be used in computer vision as well as interactive computer graphics. The model is based on polygonal uncertainty regions, transformation polytopes and their consensus sets. The most important result of this paper is that the RANSAC algorithm can be placed in a broader framework of search algorithms that look for good parameter uncertainty polytopes. To guide the search, each polytope has a consensus set whose size is a measure for its quality. r 2005 Elsevier Ltd. All rights reserved. Keywords: Uncertain geometry; Affine transformations; RANSAC; Feature matching
1. Introduction Modeling geometric uncertainty is important in computer vision as well as in interactive computer graphics. Possible applications are: recognition of geometric relations (e.g. similarity, symmetry, parallelism, shape) in hand-drawn line sketches, error analysis in photogrammetry, or maintaining geometric consistency for graphical input. In one possible approach, geometric parameters and relations are derived by statistical inference to take into account geometric uncertainty [1]. One major limitation of statistical inference, however, is that the computations soon become too complicated unless one introduces assumptions about the statistical model that are simplifying, but not always realistic, e.g. the assumption that uncertainty can be truly represented by covariance matrices and uncertainty ellipsoids. Since a purely statistical approach has its shortcomings, other methods have been proposed. The way in Corresponding author. Tel.: +32 9 2643368; fax: +32 9 2643594. E-mail address: [email protected] (P. Veelaert).
which geometric uncertainty has been modeled and handled, however, depends very much on the field of application. Several methods aim at obtaining numerical robustness in geometric modeling: robust geometric computation [2], interval geometry [3,4], rounded geometry [5]. One of the basic problems in this field is the robust intersection of line segments [2]. Also in robotics and mechanical design there is a need to deal with imprecision. The models proposed there include the use of finite precision arithmetic [6], the use of probability density functions [7,8], and the use of tolerance zones for mechanical parts [9]. In geometry when computing the propagation of uncertainty, we must take into account geometric relations and constraints, which are often non-linear. This is another difficulty. Some early ad hoc approaches have been quite successful, however. For example, Lowe as well as Nacken introduced significance measures for geometric relations, e.g. a function that measures how far two line segments are from being collinear [10,11]. A recurring problem is that such a function does not necessarily satisfy the basic rules of geometry, such as the transitivity of collinearity [11]. Recently, developments have become more systematic.
0097-8493/$ - see front matter r 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.cag.2005.10.013
ARTICLE IN PRESS 78
P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
Kanatani proposed a rigorous mathematical framework for uncertain geometric reasoning [12]. Criminisi used several methods to model the uncertainty of homographies and distance measurements, with Monte Carlo simulations to support the approach [13]. Forstner developed new, elegant representations for uncertainty and discussed the propagation of uncertainty in projective geometry using covariance matrices [1]. In our approach we advocate a uniform approach to geometric uncertainty that takes into account geometric relations, which consist in the ideal case of one coherent methodology applied to many geometric problems [14]. Positional and geometric uncertainty will be modeled by polytopes instead of uncertainty ellipsoids, since polytopes have some advantages. First, in applications regarding positional or geometric uncertainty, polytopes provide a better description of the shape of the uncertainty region. Second, when polytopes are used to capture uncertainty, many recognition problems in computer vision can be reformulated as either convex or combinatorial optimization problems for which standard algorithms are known. The proposed method is not seen as a replacement of statistical inference, but rather as an additional tool, in which we make use of the hard bounds often known for geometric parameters, such as minima and maxima for the intrinsic or extrinsic camera parameters, or the maximum velocities of objects. In fact, to get more precise information, the uncertainty ellipsoids can be embedded in the polytopes. Section 2 describes how geometric uncertainty is modeled in this paper. Section 3 introduces several kinds of polytopes used to captivate the uncertainty of transformations. Section 4 describes the properties of intersections of polytopes, and Section 5 gives experimental results.
2. Uncertainty regions and parameter uncertainty polytopes We consider the following general matching problem. Let S be a finite set of source points pi 2 R2 . For each point pi , let Si ¼ fqi1 ; qi2 ; . . .g be a finite set of candidate image points qij 2 R2 . Then we look for the optimal affine transformation T that maps source points upon image points. Hartley and Zisserman give an overview of the cost functions that are used to compute the optimal transformation T [15]. The cost may either be measured in the image plane (geometric distance), in the source plane (reprojection error) or in both planes. For example, once a possible correspondence map qi ¼ f ðpi Þ has been given between source and image points, with qi 2 SP i , we can look for the transformation T that minimizes i ðdðpi ; p^ i Þ2 þ dðqi ; q^ i Þ2 Þ, subject to the condition q^ i ¼ Tðp^ i Þ. A matching algorithm first separates the inliers from the outliers and determines a
correspondence map f by a robust estimation method, such as RANSAC, and next finds an optimal estimation for the transformation T, by minimizing the cost function [15]. In this work, we focus on the first step of the algorithm which separates the inliers from the outliers, determines the correspondences, and gives a first approximation for T. The goal of this paper is to show that this step can be enhanced if we know additional constraints for the transformation parameters. These constraints will be given either in the form of uncertainty regions for the image points or as n-dimensional polytopes for the transformation parameters. Our aim is to model geometric transformations for which the position of the image points is not known precisely. To this end we shall introduce the following notions: an uncertainty region, a transformation polytope, an implied uncertainty region and a consensus set. More specifically, to model the uncertainty of the image ðx0 ; y0 Þ of a point p ¼ ðx; yÞ in R2 , we define the uncertainty region R as a convex polygon bounded by n halfplanes, ri x0 þ si y0 X1;
1pipn
(1)
~ RÞ denote the set of all in the x0 y0 -plane. We let Tðp; transformations T that map p into the uncertainty region R. In this work, to simplify some of the definitions, we restrict ourselves to affine transformations of the form x0 ¼ ax þ e, y0 ¼ dy þ f 0
(2)
0
with ðx ; y Þ as image-point, ðx; yÞ as source-point and the four parameters of the transformation: a; e; d; f . ~ RÞ : p ! R form a convex The transformations Tðp; polyhedron T~ in four dimensions, supported by the hyperplanes that can be found by substituting Eqs. (2) in (1), yielding ri ðax þ eÞ þ si ðdy þ f ÞX1;
1pipn.
(3)
The notion of a transformation polyhedron can be extended to sets of points: let S be a finite set of points pi 2 R2 , U a collection of convex polygons Ri in R2 , one ~ for each point pi , then TðS; UÞ denotes the set of all affine transformations that map each pointTpi into the ~ i ; Ri Þ. ~ polygon Ri . Clearly, we have TðS; UÞ ¼ i Tðp When S contains at least 2 points the polyhedron ~ TðS; UÞ is bounded, and becomes a convex polytope of transformations in four-dimensional space. Note that the restriction to four parameters instead of the usual six of a general affine transformation is not essential from the theoretical viewpoint, but it will enable us to simplify notations. Most of the results in this paper can be extended in a straightforward manner to more general transformations.
ARTICLE IN PRESS P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
1000
p1 p2
R1
R2
800
79
hand, let v be any vector in Q, then Tðpi Þ þ v 2 Ri for all i. It follows that the affine transformation T þ v (that is, ~ T followed by a translation over v) belongs to TðS; UÞ. ~ Hence, Tðpi Þ þ v lies in the implied region of TðS; UÞ. Any point in Ri , however, can be written as Tðpi Þ þ v for ~ some v 2 Q, and it follows that Ri Rðpi ; TðS; UÞÞ. & With each transformation polytope T~ we associate a ~ consensus set CðTÞ.
600 q1
Definition 2. Let S be a finite set of points pi 2 R2 , and
400 p3
for each point pi , let Qi ¼ fqi1 ; qi2 ; . . .g be a finite set of points qij associated with pi . Let T~ be a transformation ~ of the polytope polytope. Then the consensus set CðTÞ S ~ ¼ ðRðpi ; TÞ ~ \ Qi Þ. T~ is defined as CðTÞ i
R3
200
200
400
600
800
1000
~ for a number of points Fig. 1. The uncertainty regions Rðqj ; TÞ qj where the transformation uncertainty polytope has been determined by ðpi ; Ri Þ for i ¼ 1; . . . ; 3. The size of the uncertainty regions varies with the position of the point q.
Once a transformation polytope T~ has been determined, we can use it to find the image of other points by uncertain transformations [16]. To be precise, let p be a point not in S and T~ a given polytope, then we let ~ denote the implied uncertainty region resulting Rðp; TÞ ~ that is, from mapping p by the transformations in T; ~ ¼ fq 2 R2 : q ¼ TðpÞ for some T 2 Tg. ~ One can Rðp; TÞ ~ is the convex hull of the points T k ðpÞ show that Rðp; TÞ where the transformations T k denote the vertices of the polytope T~ [16]. Fig. 1 shows how three points p1 ; p2 ; p3 and their image regions R1 ; R2 ; R3 restrict the set of possible transformationsT to the transformation uncertainty ~ i ; Ri Þ, which consists of all polytope T~ ¼ i¼1;...;3 Tðp transformations that map pi into Ri , for i ¼ 1; . . . ; 3. ~ With T~ we can compute the uncertainty region Rðq; TÞ for any point q in the plane. Some of these regions are shown in Fig. 1. From the definition of the transforma~ it tion polytope T~ and the implied regions Rðpi ; TÞ, ~ Ri , for i ¼ 1; . . . ; 3. follows that Rðpi ; TÞ
Lemma 1. Let S be a finite set of points pi in the plane, and let T be an affine transformation. Let Q be an arbitrary polygon and let Ri denote the polygon Q translated by Tðpi Þ, and let U be the set of polygons Ri . ~ ~ Then TðS; UÞ is non-empty and Rðpi ; TðS; UÞÞ ¼ Ri , that ~ is, the regions implied by the polytope TðS; UÞ are identical to the regions Ri . Proof. From the definition of implied regions it follows ~ that, for any pi 2 S, Rðpi ; TðS; UÞÞ Ri . On the other
~ we must Thus to find the consensus set CðTÞ, ~ and find compute the implied image regions Rðpi ; TÞ, the points qij contained in the implied regions. The matching problem can now be restated. We must find a ~ parameter polytope T~ with a high consensus jCðTÞj. Furthermore, we require that the polytope constrains the transformation sufficiently. To be precise, we require that for the points qij in the consensus set, the implied ~ of the corresponding source points regions Rðpi ; TÞ pi have height at most .
3. Consistency and RANSAC polytopes In this section, we introduce different types of polytopes that will be used to find polytopes with high consensus and sufficiently small implied regions. A special kind of transformation polytope arises from a priori constraints on the model parameters. Suppose we know that the transformed image of each point pi is confined to a region Ri , where, from now on, the Ri are squares of height t centered around each point pi . Then T the transformation must be contained in ~ i ; Ri Þ. Next, to take into account small T~ g ¼ i Tðp displacement errors, e.g. made by a feature detector, for each candidate image point qij , we define a small square Rij of height , with ot. We define the ~ i ; Rij Þ. From the consistency polytope T~ ij ¼ T~ g \ Tðp above definitions it follows that the implied uncertainty regions satisfy Rðpi ; T~ g Þ Ri , as well as Rðpi ; T~ ij Þ Ri . Note that the requirement that the a priori constraints on the transformation are formulated in terms of confinement regions Ri is not so important. We could have been given a constraint polytope T~ g instead, and computed the implied regions Rðpi ; T~ g Þ if necessary. The consistency polytope T~ ij is our basic building block to find consistent transformations with a high consensus. In fact, there is a consistent transformation T in T~ g mapping pk into Rkl and pm into Rmn , provided T~ kl \ T~ mn a;. If the polytopes intersect, we let Bkl;mn ðpi Þ denote the region implied by T~ kl \ T~ mn . Alternatively,
ARTICLE IN PRESS 80
P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
for two consistency polytopes T~ kl ; T~ mn we may also look at the intersections of the implied regions, C kl;mn ðpi Þ ¼ Rðpi ; T~ kl Þ \ Rðpi ; T~ mn Þ, instead of the implied region of the intersection of transformation polytopes. Fig. 2 shows the implied regions Rðpi ; T~ ij Þ of a consistency polytope T~ ij . Fig. 3(a) shows the implied regions Bkl;mn ðpi Þ of the intersection T~ kl \ T~ mn , and
Fig. 2. To determine a consistency polytope, we introduce large squares R2 ; . . . for the points p2 ; . . . ; and a small square R11 centered around a candidate point q11 . The consistency polytope T~ 11 then consists of all transformations that map each point pi into the corresponding region Ri , and p1 into R11 R1 . With T~ 11 we can compute the implied regions Rðpi ; T~ 11 Þ, e.g., the region Rðp2 ; T~ 11 Þ.
Fig. 3(b) shows the intersections of the implied regions C kl;mn ðpi Þ. As can be seen from Fig. 3 the regions C kl;mn ðpi Þ ¼ Rðpi ; T~ kl Þ \ Rðpi ; T~ mn Þ. The regions in Fig. 3(b) are somewhat larger than the regions in Fig. 3(a). In Fig. 3(a) two points pk and pm are mapped into small rectangles, and all the other points are mapped into implied regions whose size grows when they get farther from the line segment pk pm . In Fig. 3(a), the situation is similar, except that the implied regions also get larger between pk and pm . Another kind of polytope arises from the RANSAC algorithm [17]. The RANSAC algorithms selects k random sample points, finds the model determined by these sample points, and counts the number of consensus points that agree with the proposed model up to an allowed deviation. By repeating this process RANSAC tries to find the best model. In our case the model is an affine transformation determined by 2 sample points. Let qkl and qmn be two image points and let W kl;mn be the affine transformation for which qkl ¼ W kl;mn ðpk Þ and qmn ¼ W kl;mn ðpm Þ. In the RANSAC algorithm the allowed deviation comes from uncertainty regions that are squares Rkl;mn ðpi Þ with height and centered at the points W kl;mn ðpi Þ. That is, all implied regions have the same size. In general, for arbitrarily chosen qkl and qmn , it is not true that Rkl;mn ðpi Þ Bkl;mn ðpi Þ. However, we may require that the transformation W kl;mn is consistent. To be precise, we require that Rkl;mn ðpi Þ Ri , for all Ti, and we define the RANSAC polytope V kl;mn ¼ i Tðpi ; Rkl;mn Þ. Then let Akl;mn denote the implied regions of V kl;mn , i.e., Akl;mn ðpi Þ ¼ Rðpi ; V kl;mn Þ. The RANSAC polytope measures the uncertainty that we allow for the transformation W kl;mn . Usually this uncertainty is only measured in
Fig. 3. Image (a) shows the regions Bkl;mn ðpi Þ implied by the polytope T~ 11 \ T~ 41 . Image (b) shows the intersection of the regions C kl;mn ðpi Þ implied separately by the polytopes T~ 11 and T~ 41 . Theorem 3 shows that the regions of (a) are always contained in the regions of (b).
ARTICLE IN PRESS P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
the image plane. Here we translate it also into the transformation space.
Theorem 3. Let the regions Akl;mn ðpi Þ, Bkl;mn ðpi Þ,
C kl;mn ðpi Þ, and Rðpi ; T~ kl Þ be defined as above. Then Akl;mn ðpi Þ Bkl;mn ðpi Þ C kl;mn ðpi Þ, and C kl;mn ðpi Þ Rðpi ; T~ kl Þ as well as C kl;mn ðpi Þ Rðpi ; T~ mn Þ.
Proof. First, since Rkl;mn ðpk Þ ¼ Rkl , Rkl;mn ðpm Þ ¼ Rmn , and Rkl;mn ðpi Þ Ri , for all i, it follows that V kl;mn T~ kl and V kl;mn T~ mn . Hence V kl;mn T~ kl \ T~ mn , and therefore Akl;mn ðpi Þ Bkl;mn ðpi Þ. Second, by definition we have Bkl;mn ðpi Þ ¼ Rðpi ; T~ kl \ T~ mn Þ. Furthermore, Bkl;mn ðpi Þ Rðpi ; T~ kl Þ, and Bkl;mn ðpi Þ Rðpm ; T~ mn Þ, and it follows that Bkl;mn ðpi Þ C kl;mn ðpi Þ. & To summarize, for the different types of implied regions we have a chain of inclusions, Akl;mn ðpi Þ Bkl;mn ðpi Þ C kl;mn ðpi Þ, while C kl;mn ðpi Þ is a subset of the consistency region Rðpi ; T~ kl Þ as well as Rðpi ; T~ mn Þ. Each of these implied regions constrains the point positions in a different way. The sets Rðpi ; T~ kl Þ arise when we constrain a single point to a small square, the regions Bkl;mn ðpi Þ when we constrain two points at the same time, the regions C kl;mn ðpi Þ when we constrain two points one after the other, the regions Akl;mn ðpi Þ when we constrain two points and add the requirement that all the implied regions are small. As can be seen from Figs. 2 and 3, the implied regions are rectangles whose size varies over the image. In fact, one can prove that the size must increase piecewise linearly, according to either the value of the x or y coordinate. It is sufficient to note that the implied region of a point p, is the convex hull of the points T j ðpÞ, where the transformations T j are the vertices of the transformation polytope [16]. Suppose we have a transformation polytope with vertices ðai ; ei ; d i ; f i Þ. Then the implied regions have vertices ðai x þ ei ; d i y þ f i Þ. Hence the size of the region varies along the x-direction as ðai x þ ei Þ ðaj x þ ej Þ. In fact, one can show that the size of the implied regions varies as a function that has the shape of a bow tie, similar to what we have for the collinearity of digital straight lines or the uncertainty cones for point-line coincidences [1,18]. Clearly, when either T~ 1 T~ 2 or Rðpi ; T~ 1 Þ Rðpi ; T~ 2 Þ for all i, then CðT~ 1 Þ CðT~ 2 Þ and jCðT~ 1 ÞjpjCðT~ 2 Þj. It follows, that for the consensus sets we have a chain of inclusions similar as the one stated in Theorem 3. An interesting consequence is that jCðV kl;mn Þjp minðjCðT~ kl Þj; jCðT~ mn ÞjÞ. Thus, the consensus size of the RANSAC polytope is bounded by the consensus sizes of the consistency polytopes. Or in other words, for a given image point qkl the consensus CðT~ kl Þ tells how promising the point is as part of a model in the RANSAC algorithm, without having to evaluate all possible pairs ðqkl ; qij Þ, where qij can be any image point.
81
4. Intersections of N parameter polytopes The consensus size of V kl;mn is bounded by the consensus size of T~ kl \ T~ mn , which in term is bounded by minðjCðT~ kl Þj; jCðT~ mn ÞjÞ. Here we will derive more precise results concerning the consensus of the intersection of N polytopes. The main result is that the intersection of N consistency polytopes T~ kl is strongly related to the consensus found by the RANSAC polytope V kl;mn . We shall prove the following. First, a non-empty intersection of N consistency polytopes tells us that there is a polytope with a consensus size of at least N. Second, if there is a RANSAC polytope with consensus N, then there is also a non-empty intersection of N consistency polytopes.
Theorem 4. Suppose the intersection T~ ij \ \ T~ mn of N consistency polytopes is non-empty. Then there is a consistency polytope T~ defined on N small squares centered around the points Tðpi Þ with consensus size at least N. Proof. Let T be a transformation in T~ ij \ \ T~ mn , and let T~ ij be any polytope over which the intersection is taken. Then we have Tðpi Þ 2 Rij , where Rij is a square of height . Let RðTðpi ÞÞ be the square centered around Tðpi Þ of height . Then the two squares overlap and qij 2 RðTðpi ÞÞ. According to Lemma 1, we have ~ i ; RðTðpi ÞÞÞ. Hence qij lies in the consenRðTðpi ÞÞ ¼ Tðp ~ i ; RðTðpi ÞÞÞ. & sus set of the polytope T~ ¼ Tðp Note that Theorem 4 does not say that there will be a RANSAC polytope V kl;mn with consensus size N. In fact, the image points qij may not provide a good RANSAC transformation W kl;mn , a situation that may well occur for small data sets. In the opposite direction, however, we can prove the following result.
Theorem 5. Suppose we have a consistent RANSAC transformation W kl;mn determined by the points qkl and qmn . Let C ¼ fqij ; . . . ; quv g be the consensus set of W kl;mn . Then the intersection T~ ij \ \ T~ uv is non-empty. Proof. Let V kl;mn be the transformation polytope defined by the squares Rkl;mn ðpi Þ of size centered around the points W kl;mn ðpi Þ. Let qij be any point in the consensus set C. Then we must have qij 2 Rkl;mn ðpi Þ. It follows that W kl;mn ðpi Þ 2 Rij , where Rij is the square of size centered around qij . Furthermore, since W kl;mn is consistent, W kl;mn ðpj Þ 2 Rj for all j. It follows that the transformation W kl;mn 2 T~ ij .Because this must hold for any qij in the consensus set C, it follows that the intersection T~ ij \ \ T~ uv contains at least one transformation and is therefore non-empty. & Theorem 5 gives us a promising result: if there is a RANSAC transformation with consensus size N, then there are N consistency polytopes T~ ij that have a
ARTICLE IN PRESS 82
P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
non-empty intersection. Hence, we should be looking for non-empty intersections of consistency polytopes, and the most straightforward tool to do so is an intersection graph.
Definition 6. Suppose we have points qij , rectangles Rij , ~ i ; Rij Þ defined as above. and polytopes T~ ij ¼ T~ g \ Tðp The intersection graph GðT~ ij Þ represents each polytope T~ ij by a vertex. Two vertices are joined by an edge when the intersection of the corresponding polytopes is nonempty. The idea now is to look for a clique of vertices T kl ; . . . ; T mn in the intersection graph. Since any pair of vertices in a clique is joined by an edge, it follows that T ij \ T uv a; for any pair of polytopes of a clique. Thus there is some reason to assume that we will also have T kl \ \ T mn a;. This assumption is premature, however, as we have the following famous result.
Theorem 7 (Helly [19]). Let F be a family of at least d þ 1 polytopes in Rd . If the intersection of any d þ 1 polytopes in F is non-empty then the intersection of all the polytopes in F is non-empty. Theorem 7 is one version of Helly’s famous theorem for convex sets, here restricted to polytopes [19–21]. It states that only for one-dimensional polytopes (intervals) a clique in an intersection graph corresponds to a non-empty intersection. An example that illustrates why intersection graphs are not sufficient for d-dimensional sets d41 is easy to find. In R2 the three edges of a triangle are convex line-segments. Each pair of edges has a non-empty intersection at a corner, the intersection of the three edges, however, is empty. Likewise, in threedimensional space a tetrahedron has four two-dimensional sides. For each triple of sides there is a non-empty
intersection at a corner point, but the intersection of the four sides is empty. In detection problems, however, not only the correctness of the result but also the computational effort matters. Even though an intersection graph may give wrong information about common intersections of sets, it may still be useful as long as we can verify afterwards whether an intersection is non-empty. A maximum clique in an intersection graph may not correspond to a nonempty intersection. If the non-emptiness is easy to verify, however, and if the probability of false information is low, then the worst thing that can happen is that one must try the second largest clique. In fact, by using new types of intersection graphs we can improve the chance that a clique in an intersection graph corresponds to a non-empty intersection of sets, even in Rd , d41. In fact, the probability that a clique corresponds to a non-empty intersection of polytopes is quite high. According to Helly’s Theorem a non-empty intersection of two dimensional polytopes corresponds to a clique in a 3-uniform intersection hypergraph, where each hyperedge represents a non-empty intersection of three sets. Thus, let H be a 3-uniform intersection hypergraph, with N vertices vi each representing a set S i , and a set of hyperedges fvi ; vj ; vk g such that Si \ Sj \ Sk a;. Suppose, we construct a graph G with NðN 1Þ=2 vertices denoted as vi vj . The edges of G consist of all vertex pairs fvi vj ; vk vl g satisfying Si \ Sj \ S k \ S l a;. The motivation to replace H by G is as follows. Suppose we find a clique in this graph with N vertices vi vj ; vk vl ; . . . ; vm vn . For each edge fvi vj ; vk vl g we have Si \ Sj \ S k \ Sl a; and therefore Si \ Sj \ Sk a;; . . . ; Sj \ Sk \ Sl a; for the four combinations of three sets in a collection of four. Since the clique has NðN 1Þ=2 edges this implies that for the collection C ¼ fS i ; Sj ; . . . ; Sn g there are at least 4NðN 1Þ=2 intersections among the 2Nð2N 1Þ ð2N 2Þ=6 possible intersections of 3 arbitrary sets in C
Fig. 4. Image (a) shows the feature points detected in the reference image, while the feature points detected for the second, transformed test image are depicted in image (b).
ARTICLE IN PRESS P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
that are known to have a non-empty intersection. That is, the ratio of triples that have been verified to the total amount of triples is 6ðN 1Þ=ðð2N 1Þð2N 2ÞÞ. This means that if one finds a clique in G for which N is not too large, there is a good chance that it also corresponds to a clique in the hypergraph H. Furthermore, although finding a maximal clique is in general an NP-hard problem, G is a highly partitioned graph whose partitioning is known, so that the maximum cliques can be found much more efficiently [14].
83
5. Experimental results Consistency polytopes can be used to solve registration problems. Fig. 4 shows an example in which two similar images points have been marked as feature points by a feature detector [22]. We must find a transformation that maps the left image upon the right image as good as possible. Maximal bounds are known for the transformation parameters, i.e. 0:9pa; bp1:1, and 20pe; f p20. We also know that the feature
Fig. 5. In image (a), the uncertainty regions are centered on a subset of feature points (þ) from the reference image projected in the test image with the image feature points (). All candidate feature image points are located within these regions. The intersection graph for situation (a) is shown in (b).
Fig. 6. Image (a) is a test image while (b) is a reference image. Image (a) shows the feature points (diamonds) detected in the reference image, with large squares centered around them, in which we look for feature points (stars) in the test image. The image in (b) was taken from a slightly different position and another zooming factor and also shows the 5 image points that where eventually found in a best match. Although the correct transformation is a homography, in this example the affine transformation found could register the features of the building within a pixel.
ARTICLE IN PRESS 84
P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
detector is not completely accurate, that is even after adequate scaling and translation, a feature point in the second image may still be displaced a few pixels from the corresponding feature point in the first image. ~ i ; Rij Þ as We compute the polytopes T~ ij ¼ T~ g \ Tðp well as the graph G shown in Fig. 5(b) which is the intersection graph of the polytopes T~ ij . Although G is an intersection graph, the existence of an edge in G corresponds to a non-empty intersection ~ i ; Rij Þ\ of three polytopes (not two), that is, T~ g \ Tðp ~ k ; Rkl Þa;. Therefore, as practice shows, a clique in G Tðp corresponds almost always to a non-empty transformation polytope, containing transformations that map all the feature points in the clique from the first to the second image. Fig. 6 shows a similar example with an outdoor scene. It is important to note that in the above matching problems we only made use of consistency polytopes. Thus they provide an alternative to the RANSAC algorithm. In general, we expect however that the best algorithm would use a mixture of different techniques. In fact, the most reasonable thing to do is to use in a matching algorithm all the information that is available, hence also parametric consistency.
[2]
[3]
[4]
[5]
[6]
[7] [8]
[9]
[10] [11]
6. Concluding remarks The study of transformation polytopes and uncertainty regions originated in a much larger framework which studies the properties of digitized objects such as lines and planes, a field which started 30 years ago [23,24]. Part of this work was aimed at extracting geometric relationships, e.g., algorithms that find and group collinear, parallel and concurrent line segments [14,18,25–27]. In these algorithms intersection graphs and interval graphs (i.e., intersection graphs for intervals) play an important role. Recent work in computer vision focused on the detection of affine and scale-invariant feature points [28–30], and on feature descriptors [29]. These methods have shown impressive results for matching multiple viewpoint images of static scenes. However, matching, recognizing and tracking moving objects in dynamic outdoor scenes is still a difficult problem [31]. The work presented in this paper was motivated by the idea that one of the missing tools is a good framework for incorporating a priori knowledge on the transformation parameters of the scene and its objects.
[12]
References
[19]
[1] Fo¨rstner W. Uncertainty and projective geometry. Handbook of computational geometry for pattern recognition,
[13]
[14]
[15]
[16]
[17]
[18]
[20]
computer vision, neurocomputing and robotics. Berlin: Springer; 2004, to appear. Yap CK. Robust geometric computation. In: Goodman, Rourke, editors. Handbook of discrete and computational geometry. Boca Raton, FL: CRC Press; 1997. p. 653–68. Segal MG, Sequin CH. Consistent calculations for solids modeling. In: Proceedings of the first annual ACM symposium on computers and geometry; 1985. p. 29–38. Segal MG. Using tolerances to guarantee valic polyhedral modeling results. In: Computer & Graphics (Proceedings of SIGGRAPH ’90), vol. 24; 1990. p. 105–14. Sugihara K. On finite-precision representations of geometric objects. Journal of Computer and System Sciences 1989;40:2–18. Milenkovic VJ. Verifiable implementations of geometric algorithms using finite precision arithmetic. In: Kapur, Mundy, editors. Geometric reasoning. Cambridge: MIT Press; 1989. p. 377–401. Durrant-Whyte HF. Uncertain geometry in robotics. IEEE Transactions on Robotics Automation 1988; 23–31. Durrant-Whyte HF. Uncertain geometry. In: Kapur, Mundy, editors. Geometric reasoning. Cambridge: MIT Press; 1989. p. 447–81. Fleming A. Geometric relationships between toleranced features. In: Kapur, Mundy, editors. Geometric reasoning. Cambridge: MIT Press; 1989. p. 403–12. Lowe D. 3-d object recognition from single 2-d images. Artificial Intelligence 1987;31:355–95. Nacken P. A metric for line segments. IEEE Transactions on Pattern Analysis and Machine Intelligence 1993;15: 1312–8. Kanatani K. Statistical optimization for geometric computation: theory and practice. Amsterdam: Elsevier Science; 1996. Criminisi A. Accurate visual metrology from single and multiple uncalibrated image. Ph.D. Thesis, London: Springer; 2001. Veelaert P. Uncertain geometry in computer vision. In: Proceedings of DGCI 2005. Lecture notes in computer science, vol. 3429. Poitiers, France: Springer; 2005. p. 359–70. Hartley R, Zisserman A. Multiple view geometry in computer vision. Cambridge: Cambridge University Press; 2003. Teelen K, Veelaert P. Uncertainty of affine transformations in digital images. In: Proceedings of ACIVS 2004 (Advanced Concepts for Intelligent Vision Systems), Brussels; 2004. p. 23–30. Fischler MA, Bolles RC. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the Association of Computers and Machines 1981;24: 381–95. Veelaert P. Collinearity and weak collinearity in the digital plane. Digital and image geometry. Lecture notes in computer science, vol. 2243. Berlin: Springer; 2001. p. 434–47. Helly E. U¨ber Mengen konvexer Ko¨rper mit gemeinschaftligen Punkten. Jahresber. D.M.V. 1923;32:175–6. Rockafellar RT. Convex analysis. Princeton, NJ: Princeton University Press; 1970.
ARTICLE IN PRESS P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85 [21] Stoer J, Witzgall C. Convexity and optimization in finite dimensions I. Berlin: Springer; 1970. [22] Teelen K, Veelaert P. Computing the uncertainty of transformations in digital images. In: Proceedings of vision geometry XIII. San Jose; 2005. p. 1–12. [23] Klette R, Rosenfeld A. Digital geometry: geometric methods for digital picture analysis. San Francisco: Morgan Kaufman; 2004. [24] Rosenfeld A. Digital straight line segments. IEEE Transactions on Computers 1974;23:1264–9. [25] Veelaert P. Geometric constructions in the digital plane. Journal of Mathematical Imaging and Vision 1999;11: 99–118. [26] Veelaert P. Concurrency of line segments in uncertain geometry. Proceedings of DGCI 2002. Lecture notes in computer science, vol. 2301. Bordeaux, France: Springer; 2002. p. 289–300.
85
[27] Veelaert P. Graph-theoretical properties of parallelism in the digital plane. Discrete Applied Mathematics 2003; 125:135–60. [28] Jurie F, Schmid C. Scale-invariant shape features for recognition of object categories. In: Proceedings of CVPR’04, vol. 2; 2004. p. 90–6. [29] Lowe D. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 2004;60:91–110. [30] Mikolajczyk K, Tuytelaars T, Schmid C, Zisserman A, Matas J, Schaffalitzky F, Kadir T, Van Gool L. A comparison of affine region detectors. International Journal of Computer Vision, to appear. [31] Rothganger F, Lazebnik S, Schmid C, Ponce J. Segmenting, modeling, and matching video clips containing multiple moving objects. In: Proceedings of CVPR’04, vol. 2; 2004. p. 914–21.
Computers & Graphics 30 (2006) 77–85 www.elsevier.com/locate/cag
Consensus sets for affine transformation uncertainty polytopes Peter Veelaert, Kristof Teelen Hogeschool Gent, University of Ghent, Schoonmeersstraat 52, B9000 Ghent, Belgium
Abstract We introduce an uncertainty model for geometric transformations that can be used in computer vision as well as interactive computer graphics. The model is based on polygonal uncertainty regions, transformation polytopes and their consensus sets. The most important result of this paper is that the RANSAC algorithm can be placed in a broader framework of search algorithms that look for good parameter uncertainty polytopes. To guide the search, each polytope has a consensus set whose size is a measure for its quality. r 2005 Elsevier Ltd. All rights reserved. Keywords: Uncertain geometry; Affine transformations; RANSAC; Feature matching
1. Introduction Modeling geometric uncertainty is important in computer vision as well as in interactive computer graphics. Possible applications are: recognition of geometric relations (e.g. similarity, symmetry, parallelism, shape) in hand-drawn line sketches, error analysis in photogrammetry, or maintaining geometric consistency for graphical input. In one possible approach, geometric parameters and relations are derived by statistical inference to take into account geometric uncertainty [1]. One major limitation of statistical inference, however, is that the computations soon become too complicated unless one introduces assumptions about the statistical model that are simplifying, but not always realistic, e.g. the assumption that uncertainty can be truly represented by covariance matrices and uncertainty ellipsoids. Since a purely statistical approach has its shortcomings, other methods have been proposed. The way in Corresponding author. Tel.: +32 9 2643368; fax: +32 9 2643594. E-mail address: [email protected] (P. Veelaert).
which geometric uncertainty has been modeled and handled, however, depends very much on the field of application. Several methods aim at obtaining numerical robustness in geometric modeling: robust geometric computation [2], interval geometry [3,4], rounded geometry [5]. One of the basic problems in this field is the robust intersection of line segments [2]. Also in robotics and mechanical design there is a need to deal with imprecision. The models proposed there include the use of finite precision arithmetic [6], the use of probability density functions [7,8], and the use of tolerance zones for mechanical parts [9]. In geometry when computing the propagation of uncertainty, we must take into account geometric relations and constraints, which are often non-linear. This is another difficulty. Some early ad hoc approaches have been quite successful, however. For example, Lowe as well as Nacken introduced significance measures for geometric relations, e.g. a function that measures how far two line segments are from being collinear [10,11]. A recurring problem is that such a function does not necessarily satisfy the basic rules of geometry, such as the transitivity of collinearity [11]. Recently, developments have become more systematic.
0097-8493/$ - see front matter r 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.cag.2005.10.013
ARTICLE IN PRESS 78
P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
Kanatani proposed a rigorous mathematical framework for uncertain geometric reasoning [12]. Criminisi used several methods to model the uncertainty of homographies and distance measurements, with Monte Carlo simulations to support the approach [13]. Forstner developed new, elegant representations for uncertainty and discussed the propagation of uncertainty in projective geometry using covariance matrices [1]. In our approach we advocate a uniform approach to geometric uncertainty that takes into account geometric relations, which consist in the ideal case of one coherent methodology applied to many geometric problems [14]. Positional and geometric uncertainty will be modeled by polytopes instead of uncertainty ellipsoids, since polytopes have some advantages. First, in applications regarding positional or geometric uncertainty, polytopes provide a better description of the shape of the uncertainty region. Second, when polytopes are used to capture uncertainty, many recognition problems in computer vision can be reformulated as either convex or combinatorial optimization problems for which standard algorithms are known. The proposed method is not seen as a replacement of statistical inference, but rather as an additional tool, in which we make use of the hard bounds often known for geometric parameters, such as minima and maxima for the intrinsic or extrinsic camera parameters, or the maximum velocities of objects. In fact, to get more precise information, the uncertainty ellipsoids can be embedded in the polytopes. Section 2 describes how geometric uncertainty is modeled in this paper. Section 3 introduces several kinds of polytopes used to captivate the uncertainty of transformations. Section 4 describes the properties of intersections of polytopes, and Section 5 gives experimental results.
2. Uncertainty regions and parameter uncertainty polytopes We consider the following general matching problem. Let S be a finite set of source points pi 2 R2 . For each point pi , let Si ¼ fqi1 ; qi2 ; . . .g be a finite set of candidate image points qij 2 R2 . Then we look for the optimal affine transformation T that maps source points upon image points. Hartley and Zisserman give an overview of the cost functions that are used to compute the optimal transformation T [15]. The cost may either be measured in the image plane (geometric distance), in the source plane (reprojection error) or in both planes. For example, once a possible correspondence map qi ¼ f ðpi Þ has been given between source and image points, with qi 2 SP i , we can look for the transformation T that minimizes i ðdðpi ; p^ i Þ2 þ dðqi ; q^ i Þ2 Þ, subject to the condition q^ i ¼ Tðp^ i Þ. A matching algorithm first separates the inliers from the outliers and determines a
correspondence map f by a robust estimation method, such as RANSAC, and next finds an optimal estimation for the transformation T, by minimizing the cost function [15]. In this work, we focus on the first step of the algorithm which separates the inliers from the outliers, determines the correspondences, and gives a first approximation for T. The goal of this paper is to show that this step can be enhanced if we know additional constraints for the transformation parameters. These constraints will be given either in the form of uncertainty regions for the image points or as n-dimensional polytopes for the transformation parameters. Our aim is to model geometric transformations for which the position of the image points is not known precisely. To this end we shall introduce the following notions: an uncertainty region, a transformation polytope, an implied uncertainty region and a consensus set. More specifically, to model the uncertainty of the image ðx0 ; y0 Þ of a point p ¼ ðx; yÞ in R2 , we define the uncertainty region R as a convex polygon bounded by n halfplanes, ri x0 þ si y0 X1;
1pipn
(1)
~ RÞ denote the set of all in the x0 y0 -plane. We let Tðp; transformations T that map p into the uncertainty region R. In this work, to simplify some of the definitions, we restrict ourselves to affine transformations of the form x0 ¼ ax þ e, y0 ¼ dy þ f 0
(2)
0
with ðx ; y Þ as image-point, ðx; yÞ as source-point and the four parameters of the transformation: a; e; d; f . ~ RÞ : p ! R form a convex The transformations Tðp; polyhedron T~ in four dimensions, supported by the hyperplanes that can be found by substituting Eqs. (2) in (1), yielding ri ðax þ eÞ þ si ðdy þ f ÞX1;
1pipn.
(3)
The notion of a transformation polyhedron can be extended to sets of points: let S be a finite set of points pi 2 R2 , U a collection of convex polygons Ri in R2 , one ~ for each point pi , then TðS; UÞ denotes the set of all affine transformations that map each pointTpi into the ~ i ; Ri Þ. ~ polygon Ri . Clearly, we have TðS; UÞ ¼ i Tðp When S contains at least 2 points the polyhedron ~ TðS; UÞ is bounded, and becomes a convex polytope of transformations in four-dimensional space. Note that the restriction to four parameters instead of the usual six of a general affine transformation is not essential from the theoretical viewpoint, but it will enable us to simplify notations. Most of the results in this paper can be extended in a straightforward manner to more general transformations.
ARTICLE IN PRESS P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
1000
p1 p2
R1
R2
800
79
hand, let v be any vector in Q, then Tðpi Þ þ v 2 Ri for all i. It follows that the affine transformation T þ v (that is, ~ T followed by a translation over v) belongs to TðS; UÞ. ~ Hence, Tðpi Þ þ v lies in the implied region of TðS; UÞ. Any point in Ri , however, can be written as Tðpi Þ þ v for ~ some v 2 Q, and it follows that Ri Rðpi ; TðS; UÞÞ. & With each transformation polytope T~ we associate a ~ consensus set CðTÞ.
600 q1
Definition 2. Let S be a finite set of points pi 2 R2 , and
400 p3
for each point pi , let Qi ¼ fqi1 ; qi2 ; . . .g be a finite set of points qij associated with pi . Let T~ be a transformation ~ of the polytope polytope. Then the consensus set CðTÞ S ~ ¼ ðRðpi ; TÞ ~ \ Qi Þ. T~ is defined as CðTÞ i
R3
200
200
400
600
800
1000
~ for a number of points Fig. 1. The uncertainty regions Rðqj ; TÞ qj where the transformation uncertainty polytope has been determined by ðpi ; Ri Þ for i ¼ 1; . . . ; 3. The size of the uncertainty regions varies with the position of the point q.
Once a transformation polytope T~ has been determined, we can use it to find the image of other points by uncertain transformations [16]. To be precise, let p be a point not in S and T~ a given polytope, then we let ~ denote the implied uncertainty region resulting Rðp; TÞ ~ that is, from mapping p by the transformations in T; ~ ¼ fq 2 R2 : q ¼ TðpÞ for some T 2 Tg. ~ One can Rðp; TÞ ~ is the convex hull of the points T k ðpÞ show that Rðp; TÞ where the transformations T k denote the vertices of the polytope T~ [16]. Fig. 1 shows how three points p1 ; p2 ; p3 and their image regions R1 ; R2 ; R3 restrict the set of possible transformationsT to the transformation uncertainty ~ i ; Ri Þ, which consists of all polytope T~ ¼ i¼1;...;3 Tðp transformations that map pi into Ri , for i ¼ 1; . . . ; 3. ~ With T~ we can compute the uncertainty region Rðq; TÞ for any point q in the plane. Some of these regions are shown in Fig. 1. From the definition of the transforma~ it tion polytope T~ and the implied regions Rðpi ; TÞ, ~ Ri , for i ¼ 1; . . . ; 3. follows that Rðpi ; TÞ
Lemma 1. Let S be a finite set of points pi in the plane, and let T be an affine transformation. Let Q be an arbitrary polygon and let Ri denote the polygon Q translated by Tðpi Þ, and let U be the set of polygons Ri . ~ ~ Then TðS; UÞ is non-empty and Rðpi ; TðS; UÞÞ ¼ Ri , that ~ is, the regions implied by the polytope TðS; UÞ are identical to the regions Ri . Proof. From the definition of implied regions it follows ~ that, for any pi 2 S, Rðpi ; TðS; UÞÞ Ri . On the other
~ we must Thus to find the consensus set CðTÞ, ~ and find compute the implied image regions Rðpi ; TÞ, the points qij contained in the implied regions. The matching problem can now be restated. We must find a ~ parameter polytope T~ with a high consensus jCðTÞj. Furthermore, we require that the polytope constrains the transformation sufficiently. To be precise, we require that for the points qij in the consensus set, the implied ~ of the corresponding source points regions Rðpi ; TÞ pi have height at most .
3. Consistency and RANSAC polytopes In this section, we introduce different types of polytopes that will be used to find polytopes with high consensus and sufficiently small implied regions. A special kind of transformation polytope arises from a priori constraints on the model parameters. Suppose we know that the transformed image of each point pi is confined to a region Ri , where, from now on, the Ri are squares of height t centered around each point pi . Then T the transformation must be contained in ~ i ; Ri Þ. Next, to take into account small T~ g ¼ i Tðp displacement errors, e.g. made by a feature detector, for each candidate image point qij , we define a small square Rij of height , with ot. We define the ~ i ; Rij Þ. From the consistency polytope T~ ij ¼ T~ g \ Tðp above definitions it follows that the implied uncertainty regions satisfy Rðpi ; T~ g Þ Ri , as well as Rðpi ; T~ ij Þ Ri . Note that the requirement that the a priori constraints on the transformation are formulated in terms of confinement regions Ri is not so important. We could have been given a constraint polytope T~ g instead, and computed the implied regions Rðpi ; T~ g Þ if necessary. The consistency polytope T~ ij is our basic building block to find consistent transformations with a high consensus. In fact, there is a consistent transformation T in T~ g mapping pk into Rkl and pm into Rmn , provided T~ kl \ T~ mn a;. If the polytopes intersect, we let Bkl;mn ðpi Þ denote the region implied by T~ kl \ T~ mn . Alternatively,
ARTICLE IN PRESS 80
P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
for two consistency polytopes T~ kl ; T~ mn we may also look at the intersections of the implied regions, C kl;mn ðpi Þ ¼ Rðpi ; T~ kl Þ \ Rðpi ; T~ mn Þ, instead of the implied region of the intersection of transformation polytopes. Fig. 2 shows the implied regions Rðpi ; T~ ij Þ of a consistency polytope T~ ij . Fig. 3(a) shows the implied regions Bkl;mn ðpi Þ of the intersection T~ kl \ T~ mn , and
Fig. 2. To determine a consistency polytope, we introduce large squares R2 ; . . . for the points p2 ; . . . ; and a small square R11 centered around a candidate point q11 . The consistency polytope T~ 11 then consists of all transformations that map each point pi into the corresponding region Ri , and p1 into R11 R1 . With T~ 11 we can compute the implied regions Rðpi ; T~ 11 Þ, e.g., the region Rðp2 ; T~ 11 Þ.
Fig. 3(b) shows the intersections of the implied regions C kl;mn ðpi Þ. As can be seen from Fig. 3 the regions C kl;mn ðpi Þ ¼ Rðpi ; T~ kl Þ \ Rðpi ; T~ mn Þ. The regions in Fig. 3(b) are somewhat larger than the regions in Fig. 3(a). In Fig. 3(a) two points pk and pm are mapped into small rectangles, and all the other points are mapped into implied regions whose size grows when they get farther from the line segment pk pm . In Fig. 3(a), the situation is similar, except that the implied regions also get larger between pk and pm . Another kind of polytope arises from the RANSAC algorithm [17]. The RANSAC algorithms selects k random sample points, finds the model determined by these sample points, and counts the number of consensus points that agree with the proposed model up to an allowed deviation. By repeating this process RANSAC tries to find the best model. In our case the model is an affine transformation determined by 2 sample points. Let qkl and qmn be two image points and let W kl;mn be the affine transformation for which qkl ¼ W kl;mn ðpk Þ and qmn ¼ W kl;mn ðpm Þ. In the RANSAC algorithm the allowed deviation comes from uncertainty regions that are squares Rkl;mn ðpi Þ with height and centered at the points W kl;mn ðpi Þ. That is, all implied regions have the same size. In general, for arbitrarily chosen qkl and qmn , it is not true that Rkl;mn ðpi Þ Bkl;mn ðpi Þ. However, we may require that the transformation W kl;mn is consistent. To be precise, we require that Rkl;mn ðpi Þ Ri , for all Ti, and we define the RANSAC polytope V kl;mn ¼ i Tðpi ; Rkl;mn Þ. Then let Akl;mn denote the implied regions of V kl;mn , i.e., Akl;mn ðpi Þ ¼ Rðpi ; V kl;mn Þ. The RANSAC polytope measures the uncertainty that we allow for the transformation W kl;mn . Usually this uncertainty is only measured in
Fig. 3. Image (a) shows the regions Bkl;mn ðpi Þ implied by the polytope T~ 11 \ T~ 41 . Image (b) shows the intersection of the regions C kl;mn ðpi Þ implied separately by the polytopes T~ 11 and T~ 41 . Theorem 3 shows that the regions of (a) are always contained in the regions of (b).
ARTICLE IN PRESS P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
the image plane. Here we translate it also into the transformation space.
Theorem 3. Let the regions Akl;mn ðpi Þ, Bkl;mn ðpi Þ,
C kl;mn ðpi Þ, and Rðpi ; T~ kl Þ be defined as above. Then Akl;mn ðpi Þ Bkl;mn ðpi Þ C kl;mn ðpi Þ, and C kl;mn ðpi Þ Rðpi ; T~ kl Þ as well as C kl;mn ðpi Þ Rðpi ; T~ mn Þ.
Proof. First, since Rkl;mn ðpk Þ ¼ Rkl , Rkl;mn ðpm Þ ¼ Rmn , and Rkl;mn ðpi Þ Ri , for all i, it follows that V kl;mn T~ kl and V kl;mn T~ mn . Hence V kl;mn T~ kl \ T~ mn , and therefore Akl;mn ðpi Þ Bkl;mn ðpi Þ. Second, by definition we have Bkl;mn ðpi Þ ¼ Rðpi ; T~ kl \ T~ mn Þ. Furthermore, Bkl;mn ðpi Þ Rðpi ; T~ kl Þ, and Bkl;mn ðpi Þ Rðpm ; T~ mn Þ, and it follows that Bkl;mn ðpi Þ C kl;mn ðpi Þ. & To summarize, for the different types of implied regions we have a chain of inclusions, Akl;mn ðpi Þ Bkl;mn ðpi Þ C kl;mn ðpi Þ, while C kl;mn ðpi Þ is a subset of the consistency region Rðpi ; T~ kl Þ as well as Rðpi ; T~ mn Þ. Each of these implied regions constrains the point positions in a different way. The sets Rðpi ; T~ kl Þ arise when we constrain a single point to a small square, the regions Bkl;mn ðpi Þ when we constrain two points at the same time, the regions C kl;mn ðpi Þ when we constrain two points one after the other, the regions Akl;mn ðpi Þ when we constrain two points and add the requirement that all the implied regions are small. As can be seen from Figs. 2 and 3, the implied regions are rectangles whose size varies over the image. In fact, one can prove that the size must increase piecewise linearly, according to either the value of the x or y coordinate. It is sufficient to note that the implied region of a point p, is the convex hull of the points T j ðpÞ, where the transformations T j are the vertices of the transformation polytope [16]. Suppose we have a transformation polytope with vertices ðai ; ei ; d i ; f i Þ. Then the implied regions have vertices ðai x þ ei ; d i y þ f i Þ. Hence the size of the region varies along the x-direction as ðai x þ ei Þ ðaj x þ ej Þ. In fact, one can show that the size of the implied regions varies as a function that has the shape of a bow tie, similar to what we have for the collinearity of digital straight lines or the uncertainty cones for point-line coincidences [1,18]. Clearly, when either T~ 1 T~ 2 or Rðpi ; T~ 1 Þ Rðpi ; T~ 2 Þ for all i, then CðT~ 1 Þ CðT~ 2 Þ and jCðT~ 1 ÞjpjCðT~ 2 Þj. It follows, that for the consensus sets we have a chain of inclusions similar as the one stated in Theorem 3. An interesting consequence is that jCðV kl;mn Þjp minðjCðT~ kl Þj; jCðT~ mn ÞjÞ. Thus, the consensus size of the RANSAC polytope is bounded by the consensus sizes of the consistency polytopes. Or in other words, for a given image point qkl the consensus CðT~ kl Þ tells how promising the point is as part of a model in the RANSAC algorithm, without having to evaluate all possible pairs ðqkl ; qij Þ, where qij can be any image point.
81
4. Intersections of N parameter polytopes The consensus size of V kl;mn is bounded by the consensus size of T~ kl \ T~ mn , which in term is bounded by minðjCðT~ kl Þj; jCðT~ mn ÞjÞ. Here we will derive more precise results concerning the consensus of the intersection of N polytopes. The main result is that the intersection of N consistency polytopes T~ kl is strongly related to the consensus found by the RANSAC polytope V kl;mn . We shall prove the following. First, a non-empty intersection of N consistency polytopes tells us that there is a polytope with a consensus size of at least N. Second, if there is a RANSAC polytope with consensus N, then there is also a non-empty intersection of N consistency polytopes.
Theorem 4. Suppose the intersection T~ ij \ \ T~ mn of N consistency polytopes is non-empty. Then there is a consistency polytope T~ defined on N small squares centered around the points Tðpi Þ with consensus size at least N. Proof. Let T be a transformation in T~ ij \ \ T~ mn , and let T~ ij be any polytope over which the intersection is taken. Then we have Tðpi Þ 2 Rij , where Rij is a square of height . Let RðTðpi ÞÞ be the square centered around Tðpi Þ of height . Then the two squares overlap and qij 2 RðTðpi ÞÞ. According to Lemma 1, we have ~ i ; RðTðpi ÞÞÞ. Hence qij lies in the consenRðTðpi ÞÞ ¼ Tðp ~ i ; RðTðpi ÞÞÞ. & sus set of the polytope T~ ¼ Tðp Note that Theorem 4 does not say that there will be a RANSAC polytope V kl;mn with consensus size N. In fact, the image points qij may not provide a good RANSAC transformation W kl;mn , a situation that may well occur for small data sets. In the opposite direction, however, we can prove the following result.
Theorem 5. Suppose we have a consistent RANSAC transformation W kl;mn determined by the points qkl and qmn . Let C ¼ fqij ; . . . ; quv g be the consensus set of W kl;mn . Then the intersection T~ ij \ \ T~ uv is non-empty. Proof. Let V kl;mn be the transformation polytope defined by the squares Rkl;mn ðpi Þ of size centered around the points W kl;mn ðpi Þ. Let qij be any point in the consensus set C. Then we must have qij 2 Rkl;mn ðpi Þ. It follows that W kl;mn ðpi Þ 2 Rij , where Rij is the square of size centered around qij . Furthermore, since W kl;mn is consistent, W kl;mn ðpj Þ 2 Rj for all j. It follows that the transformation W kl;mn 2 T~ ij .Because this must hold for any qij in the consensus set C, it follows that the intersection T~ ij \ \ T~ uv contains at least one transformation and is therefore non-empty. & Theorem 5 gives us a promising result: if there is a RANSAC transformation with consensus size N, then there are N consistency polytopes T~ ij that have a
ARTICLE IN PRESS 82
P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
non-empty intersection. Hence, we should be looking for non-empty intersections of consistency polytopes, and the most straightforward tool to do so is an intersection graph.
Definition 6. Suppose we have points qij , rectangles Rij , ~ i ; Rij Þ defined as above. and polytopes T~ ij ¼ T~ g \ Tðp The intersection graph GðT~ ij Þ represents each polytope T~ ij by a vertex. Two vertices are joined by an edge when the intersection of the corresponding polytopes is nonempty. The idea now is to look for a clique of vertices T kl ; . . . ; T mn in the intersection graph. Since any pair of vertices in a clique is joined by an edge, it follows that T ij \ T uv a; for any pair of polytopes of a clique. Thus there is some reason to assume that we will also have T kl \ \ T mn a;. This assumption is premature, however, as we have the following famous result.
Theorem 7 (Helly [19]). Let F be a family of at least d þ 1 polytopes in Rd . If the intersection of any d þ 1 polytopes in F is non-empty then the intersection of all the polytopes in F is non-empty. Theorem 7 is one version of Helly’s famous theorem for convex sets, here restricted to polytopes [19–21]. It states that only for one-dimensional polytopes (intervals) a clique in an intersection graph corresponds to a non-empty intersection. An example that illustrates why intersection graphs are not sufficient for d-dimensional sets d41 is easy to find. In R2 the three edges of a triangle are convex line-segments. Each pair of edges has a non-empty intersection at a corner, the intersection of the three edges, however, is empty. Likewise, in threedimensional space a tetrahedron has four two-dimensional sides. For each triple of sides there is a non-empty
intersection at a corner point, but the intersection of the four sides is empty. In detection problems, however, not only the correctness of the result but also the computational effort matters. Even though an intersection graph may give wrong information about common intersections of sets, it may still be useful as long as we can verify afterwards whether an intersection is non-empty. A maximum clique in an intersection graph may not correspond to a nonempty intersection. If the non-emptiness is easy to verify, however, and if the probability of false information is low, then the worst thing that can happen is that one must try the second largest clique. In fact, by using new types of intersection graphs we can improve the chance that a clique in an intersection graph corresponds to a non-empty intersection of sets, even in Rd , d41. In fact, the probability that a clique corresponds to a non-empty intersection of polytopes is quite high. According to Helly’s Theorem a non-empty intersection of two dimensional polytopes corresponds to a clique in a 3-uniform intersection hypergraph, where each hyperedge represents a non-empty intersection of three sets. Thus, let H be a 3-uniform intersection hypergraph, with N vertices vi each representing a set S i , and a set of hyperedges fvi ; vj ; vk g such that Si \ Sj \ Sk a;. Suppose, we construct a graph G with NðN 1Þ=2 vertices denoted as vi vj . The edges of G consist of all vertex pairs fvi vj ; vk vl g satisfying Si \ Sj \ S k \ S l a;. The motivation to replace H by G is as follows. Suppose we find a clique in this graph with N vertices vi vj ; vk vl ; . . . ; vm vn . For each edge fvi vj ; vk vl g we have Si \ Sj \ S k \ Sl a; and therefore Si \ Sj \ Sk a;; . . . ; Sj \ Sk \ Sl a; for the four combinations of three sets in a collection of four. Since the clique has NðN 1Þ=2 edges this implies that for the collection C ¼ fS i ; Sj ; . . . ; Sn g there are at least 4NðN 1Þ=2 intersections among the 2Nð2N 1Þ ð2N 2Þ=6 possible intersections of 3 arbitrary sets in C
Fig. 4. Image (a) shows the feature points detected in the reference image, while the feature points detected for the second, transformed test image are depicted in image (b).
ARTICLE IN PRESS P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
that are known to have a non-empty intersection. That is, the ratio of triples that have been verified to the total amount of triples is 6ðN 1Þ=ðð2N 1Þð2N 2ÞÞ. This means that if one finds a clique in G for which N is not too large, there is a good chance that it also corresponds to a clique in the hypergraph H. Furthermore, although finding a maximal clique is in general an NP-hard problem, G is a highly partitioned graph whose partitioning is known, so that the maximum cliques can be found much more efficiently [14].
83
5. Experimental results Consistency polytopes can be used to solve registration problems. Fig. 4 shows an example in which two similar images points have been marked as feature points by a feature detector [22]. We must find a transformation that maps the left image upon the right image as good as possible. Maximal bounds are known for the transformation parameters, i.e. 0:9pa; bp1:1, and 20pe; f p20. We also know that the feature
Fig. 5. In image (a), the uncertainty regions are centered on a subset of feature points (þ) from the reference image projected in the test image with the image feature points (). All candidate feature image points are located within these regions. The intersection graph for situation (a) is shown in (b).
Fig. 6. Image (a) is a test image while (b) is a reference image. Image (a) shows the feature points (diamonds) detected in the reference image, with large squares centered around them, in which we look for feature points (stars) in the test image. The image in (b) was taken from a slightly different position and another zooming factor and also shows the 5 image points that where eventually found in a best match. Although the correct transformation is a homography, in this example the affine transformation found could register the features of the building within a pixel.
ARTICLE IN PRESS 84
P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85
detector is not completely accurate, that is even after adequate scaling and translation, a feature point in the second image may still be displaced a few pixels from the corresponding feature point in the first image. ~ i ; Rij Þ as We compute the polytopes T~ ij ¼ T~ g \ Tðp well as the graph G shown in Fig. 5(b) which is the intersection graph of the polytopes T~ ij . Although G is an intersection graph, the existence of an edge in G corresponds to a non-empty intersection ~ i ; Rij Þ\ of three polytopes (not two), that is, T~ g \ Tðp ~ k ; Rkl Þa;. Therefore, as practice shows, a clique in G Tðp corresponds almost always to a non-empty transformation polytope, containing transformations that map all the feature points in the clique from the first to the second image. Fig. 6 shows a similar example with an outdoor scene. It is important to note that in the above matching problems we only made use of consistency polytopes. Thus they provide an alternative to the RANSAC algorithm. In general, we expect however that the best algorithm would use a mixture of different techniques. In fact, the most reasonable thing to do is to use in a matching algorithm all the information that is available, hence also parametric consistency.
[2]
[3]
[4]
[5]
[6]
[7] [8]
[9]
[10] [11]
6. Concluding remarks The study of transformation polytopes and uncertainty regions originated in a much larger framework which studies the properties of digitized objects such as lines and planes, a field which started 30 years ago [23,24]. Part of this work was aimed at extracting geometric relationships, e.g., algorithms that find and group collinear, parallel and concurrent line segments [14,18,25–27]. In these algorithms intersection graphs and interval graphs (i.e., intersection graphs for intervals) play an important role. Recent work in computer vision focused on the detection of affine and scale-invariant feature points [28–30], and on feature descriptors [29]. These methods have shown impressive results for matching multiple viewpoint images of static scenes. However, matching, recognizing and tracking moving objects in dynamic outdoor scenes is still a difficult problem [31]. The work presented in this paper was motivated by the idea that one of the missing tools is a good framework for incorporating a priori knowledge on the transformation parameters of the scene and its objects.
[12]
References
[19]
[1] Fo¨rstner W. Uncertainty and projective geometry. Handbook of computational geometry for pattern recognition,
[13]
[14]
[15]
[16]
[17]
[18]
[20]
computer vision, neurocomputing and robotics. Berlin: Springer; 2004, to appear. Yap CK. Robust geometric computation. In: Goodman, Rourke, editors. Handbook of discrete and computational geometry. Boca Raton, FL: CRC Press; 1997. p. 653–68. Segal MG, Sequin CH. Consistent calculations for solids modeling. In: Proceedings of the first annual ACM symposium on computers and geometry; 1985. p. 29–38. Segal MG. Using tolerances to guarantee valic polyhedral modeling results. In: Computer & Graphics (Proceedings of SIGGRAPH ’90), vol. 24; 1990. p. 105–14. Sugihara K. On finite-precision representations of geometric objects. Journal of Computer and System Sciences 1989;40:2–18. Milenkovic VJ. Verifiable implementations of geometric algorithms using finite precision arithmetic. In: Kapur, Mundy, editors. Geometric reasoning. Cambridge: MIT Press; 1989. p. 377–401. Durrant-Whyte HF. Uncertain geometry in robotics. IEEE Transactions on Robotics Automation 1988; 23–31. Durrant-Whyte HF. Uncertain geometry. In: Kapur, Mundy, editors. Geometric reasoning. Cambridge: MIT Press; 1989. p. 447–81. Fleming A. Geometric relationships between toleranced features. In: Kapur, Mundy, editors. Geometric reasoning. Cambridge: MIT Press; 1989. p. 403–12. Lowe D. 3-d object recognition from single 2-d images. Artificial Intelligence 1987;31:355–95. Nacken P. A metric for line segments. IEEE Transactions on Pattern Analysis and Machine Intelligence 1993;15: 1312–8. Kanatani K. Statistical optimization for geometric computation: theory and practice. Amsterdam: Elsevier Science; 1996. Criminisi A. Accurate visual metrology from single and multiple uncalibrated image. Ph.D. Thesis, London: Springer; 2001. Veelaert P. Uncertain geometry in computer vision. In: Proceedings of DGCI 2005. Lecture notes in computer science, vol. 3429. Poitiers, France: Springer; 2005. p. 359–70. Hartley R, Zisserman A. Multiple view geometry in computer vision. Cambridge: Cambridge University Press; 2003. Teelen K, Veelaert P. Uncertainty of affine transformations in digital images. In: Proceedings of ACIVS 2004 (Advanced Concepts for Intelligent Vision Systems), Brussels; 2004. p. 23–30. Fischler MA, Bolles RC. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the Association of Computers and Machines 1981;24: 381–95. Veelaert P. Collinearity and weak collinearity in the digital plane. Digital and image geometry. Lecture notes in computer science, vol. 2243. Berlin: Springer; 2001. p. 434–47. Helly E. U¨ber Mengen konvexer Ko¨rper mit gemeinschaftligen Punkten. Jahresber. D.M.V. 1923;32:175–6. Rockafellar RT. Convex analysis. Princeton, NJ: Princeton University Press; 1970.
ARTICLE IN PRESS P. Veelaert, K. Teelen / Computers & Graphics 30 (2006) 77–85 [21] Stoer J, Witzgall C. Convexity and optimization in finite dimensions I. Berlin: Springer; 1970. [22] Teelen K, Veelaert P. Computing the uncertainty of transformations in digital images. In: Proceedings of vision geometry XIII. San Jose; 2005. p. 1–12. [23] Klette R, Rosenfeld A. Digital geometry: geometric methods for digital picture analysis. San Francisco: Morgan Kaufman; 2004. [24] Rosenfeld A. Digital straight line segments. IEEE Transactions on Computers 1974;23:1264–9. [25] Veelaert P. Geometric constructions in the digital plane. Journal of Mathematical Imaging and Vision 1999;11: 99–118. [26] Veelaert P. Concurrency of line segments in uncertain geometry. Proceedings of DGCI 2002. Lecture notes in computer science, vol. 2301. Bordeaux, France: Springer; 2002. p. 289–300.
85
[27] Veelaert P. Graph-theoretical properties of parallelism in the digital plane. Discrete Applied Mathematics 2003; 125:135–60. [28] Jurie F, Schmid C. Scale-invariant shape features for recognition of object categories. In: Proceedings of CVPR’04, vol. 2; 2004. p. 90–6. [29] Lowe D. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 2004;60:91–110. [30] Mikolajczyk K, Tuytelaars T, Schmid C, Zisserman A, Matas J, Schaffalitzky F, Kadir T, Van Gool L. A comparison of affine region detectors. International Journal of Computer Vision, to appear. [31] Rothganger F, Lazebnik S, Schmid C, Ponce J. Segmenting, modeling, and matching video clips containing multiple moving objects. In: Proceedings of CVPR’04, vol. 2; 2004. p. 914–21.