- Email: [email protected]
Determining motion and depth from binocular orthographic views
CVGIP:
IMAGE
UNDERSTANDING
Vol. 54, No. 1, July, pp. 47-55, 1991
Determining
Motion and Depth from Binocular Orthographic
Views
HOMER H. CHEN Machine Perception Research Department, AT&T Bell Laboratories, Room 4E-630, Crawfords Corner Road, Holmdel, New Jersey 07733 Communicated by Thomas S. Huang Received February 9, 1990
ferent approach. Instead of focusing on error modeling, The problemaddressedin this paperarisesin motion estimation our ideal is to eliminate completely the contribution of from stereoimagesequences. The imagesequences are taken from depth errors to motion estimates. This is done by discardtwo stereophoto stations which look at an objectfrom different ing the depth coordinate in computing the motion paramangles.To eliminate the effect of depth errors on motion estima- eters. Thus, the input data can be thought of as orthotion, we discardthe noisy z coordinateand keeponly the x and y graphic views of the object. This approach is found to be coordinates.Therefore, the problem becomescomputing motion useful in dealing with noisy data where one coordinate and depth of an objectfrom binocular orthographicviews.A geoerror is more dominant than errors in the other coordimetric approachis presented,and the solutionsfor both exactand nates. The idea has been successfully applied to verify noisy imagedata are described. o 1991 Academic press, inc. 3-D point matches [2]. Here, we apply it to estimate motion parameters. Therefore, the current problem can be formulated as I. INTRODUCTION computing the motion and depth of an object from binocIf a robot is to plan and execute actions intelligently, it ular orthographic views taken from two widely separated viewing angles. As mentioned earlier, the relative orienmust interact with its environment and interpret correctly the sensory data it obtains. In many applications of robot tation and position between the two photo stations are vision, one needs to acquire information about the posi- known. Object features, such as corners of a polyhedron, tion and motion of a robot relative to a target object. For are detected and tracked over time in each photo station. example, in tracking a moving object for teleconferenc- That is, point correspondences between temporal frames are known; however, point correspondences between ing, the orientation of the camera must be continuously adjusted to keep the target object in the field of view of spatial frames are not assumed. the camera. Another example is the retrieval of a satellite For a monocular image sequence, Ullman has shown using a space shuttle. If the satellite is tumbling, it is that two orthographic views are insufficient and that the necessary to match the motion of a robot hand to the motion/structure of an object can be uniquely determined motion of the satellite so that excessive forces and by tracking four points over three orthographic views torques are not produced when the hand grasps the satel- [ 121.This classical motion problem has recently been relite [4]. visited by several researchers, and linear methods (for The work described in this paper is abstracted from an perfect data) have been found [7], [9], [ 131. In an earlier application task similar to the second example described paper [2], we described a least-square algorithm for above in which two robot hands, each mounted with a monocular image data. We first review the monocular pair of stereo cameras, are commanded to grasp a moving case and show that, with two views, the solution for the object. The relative position and orientation between the rotation is a one-parameter family which forms a great two robot hands at any time instant are known. How- circle on the unit quaternion sphere. We then describe ever, it is possible for the two photo stations to observe how to determine the motion parameters from binocular completely different object points. data without stereo point correspondences. Given stereo image sequences, the depth of the object can be computed by the simple triangulation technique. 2. PROBLEMSTATEMENT To improve the accuracy of motion estimates, much recent work has been directed toward the development of Two orthographic views at time instants Tl and T2, more sophisticated methods for modeling the measure- respectively, are taken of the moving rigid object from ment errors (for a brief review, see [lo]). We take a dif- each photo station. By processing these orthographic 47 1049~9&o/91 All
$3.00
Copyright 0 1991 by Academic Press, Inc. rights of reproductionin any form reserved.
48
HOMER H. CHEN
Using matrix notation, Eq. (1) can be written as
Suppose the object is orthographically projected along the z-axis onto the x-y plane and the x- and y-coordinates of an objectpoint are taken as its image coordinates, then we have camera
1
camera 2
FIG. 1. Basic geometry of the problem: Orthographic views of a moving object at time instants Tl and T2 are taken from two cameras with different viewing angles.
views, we want to determine the motion and position of the object. Without loss of generality, we assume that the image planes are stationary and set up two coordinate frames, one on each photo station. The basic geometry of the problem is shown in Figure 1, where R and t represent the rotation and translation of the object measured in the first coordinate frame, G and g denote the motion measured in the second coordinate frame, and H and h denote the relative orientation and position between the two coordinate frames. Suppose there are n points. The positions of the points at time Tl in the first coordinate frame are represented by ui, and the positions of the points in the second coordinate frame are represented by vi. The positions of the points at time T2 are primed. Therefore,
(3) Note that uiz’s are unavailable from the image data. To eliminate (uiz - ulZ), we premultiply both sides of Eq. (3) by the row vector [RZ3- RI31to get Cd - u;x)R23 - (dy - u;y)R13 + (uix - ~1x)R32 - (uiy - ~1ylJ331= 0, i=2,.
. . ,n
(1)
RzRu - klR23 =
R32
V( = Gvi + g
(2)
Rdb3 - Rdb
R31
MONOCULAR
ANALYSIS-A
REVIEW
The first step of our approach computes the motion/ structure information of the object by using the image data obtained from each photo station. For monocular analysis, we adopt the linear method developed by Huang and Lee [7] to show how much motion information can be recovered from two orthographic views.
(4)
where the following two identities
U; = Rui + t
where i = 1, 2, . . . , n. Given H, h, and the image data of the points at two time instants, the problem is to determine the motion and depth of the object. In the following, we will first show that only partial motion information can be recovered from monocular image data. Then we will show how to make use of the known relative geometry between the photo stations to determine the complete motion information. 3.
The term [t, t,lT on the right hand side can be eliminated by subtracting the equation of point uI from the other equations. Thus
=
are used to simplify the expression. These two identities are derived from the orthogonality of a rotation matrix for which the cross product of the first two rows is equal to the third row, following the right-hand rule. Eq. (4) contains four unknowns R13,R23, R31,and R32. (Note: the remaining elements in R can be expressed as functions of these four unknowns [7], [8].) Since Eq. (4) is a set of linear and homogeneous equations in the four unknowns, the rotation cannot be uniquely determined from two views. To state it more clearly: l
l
The only thing we can determine about R is the values of these four unknowns, Ri3, R23, R31, and R32, to within a scale factor. That is, we can only determine the relative values of these four unknowns, not their absolute values. The relative depth (uiz - uiZ) cannot be determined
DETERMINING
MOTION
AND DEPTH
FROM BINOCULAR
if the scale factor of the four unknowns is unsolved (see Eq. (3)), nor can the translation be determined. Note that the operation performed in deriving Eq. (4) is equivalent to decomposing the motion in such a way that ul is brought to coincide with ul by the translation
ORTHOGRAPHIC
49
VIEWS
tion involved is very tedious. Described here is a linear method using the unit quaternion representation of rotation. We first explore some mathematical characteristics of the rotation information obtained from the monocular analysis. 1. The solution to Eq. (4) is a family of rotations which forms a great circle on the unit quaternion sphere.
THEOREM
ru;,
- ulxl
tj=ILliy - I
(5)
UlY
I,! LUlZ
-
Prosf. Let a be the rotation axis and 8 be the rotation angle. Then the rotation matrix is
11 UlZJ
The rotation under this decomposition takes place around the point u;, as opposed to around the origin of the coordinate frame. Therefore, the translation tl is different from t in Eq. (1). The latter is equal to u; - Rul . The rotation matrix is unchanged, however.
B =
a’, + (1 - ai) cos 8
axay[ - a, sin 0
aXay[ + a, sin 8
at + (1 - at) cos 0
[ axaz[ - aY sin 8
a,a,t + a, sin 8 a,a,.$ + aY sin 8
4.
PIECING
TOGETHER
MONOCULAR
MOTION
DATA
In this section, we show how to make use of the known relative geometry between the photo stations to integrate monocular motion data. 4.1
Relationship
between Monocular
Motion
Data
a,a& - a, sin 8 a2 + (1 - a:) cos 8 where 5 = 1 - cos 0. A closer look at the rotation matrix reveals that
The coordinates of a point in the two photo stations are related by vi = Hui + h
(6)
vi = Hu; + h.
(7)
Substituting uf of Eq. (1) into Eq. (7) yields Vf
= HRH-‘vi
R32
+
&I
=
3
= -
Ul
a,
(10)
and R13
+
R31
R32
-
R23
I3
= a, tan- = c2. 2
(11)
+ (h + Ht - HRH-‘h).
(8)
to represent the rotation, where El = a, sin(6/2), r2 = ay sin(0/2), r3 = a, sin(8/2), and 4 = cos(0/2). Then Eqs. (10) and (11) can be rewritten as
(9)
f2 = aIf{
Comparing this equation to Eq. (2), we thus have G = HRH-’ g = h + Ht - HRH-‘h.
These two equations describe the relationship between the motions observed by the two photo stations. The rotation can be uniquely determined by solving Eq. (8); however, the translation can only be determined to within an additive constant if there is no stereo correspondence. 4.2
+
h
Because the common scale factor is canceled out in these two equations, o1 and u2 are constants. Now, use a unit quaternion [l]
= HRUi + (h + Ht).
Using Eq. (6) to substitute vi for ni yields Vi
R23
A Geometric Approach for Solving the Rotation
Although directly solving for the two unknown scale factors of R and G from Eq. (8) is possible, the computa-
113 =
(T2f4
which represent two hyperplanes in the quaternion space. These two hyperplanes are orthogonal to each other and all pass through the origin. The intersection of these two hyperplanes is a plane that also passes through the origin. Since i is a unit vector, the family of rotations defined by Eq. (4) lies at the intersection of the plane with the unit quaternion sphere. A plane passing through the center of a unit n-sphere intersects the sphere at a great
50
HOMER H. CHEN
R. Postmultiplying both sides of the equation GH = HR by a yields GHa = Ha. Since b = Ha, we therefore prove that b is the rotation axis of G. Q.E.D.
FIG. 2. The locus of the family of rotation axes is a great circle lying on a plane that is perpendicular to the x-y plane. The angle between the plane and the x-z plane is tan-‘u, .
circle [ 111.Therefore, the locus of the family of rotations is a great circle on a unit 3-sphere. Q.E.D. The family of rotation axes defined by Eq. (10) also has one free parameter. It should be clear that the locus of the family of rotation axes is a great circle on a unit 2sphere centered at the origin (see Figure 2). The plane of the great circle is perpendicular to the x-y plane and forms an angle tan-l o1 with the y-z plane. Considering a, as a free parameter, we can represent the remaining rotation parameters by
Therefore, according to Corollary 1, we can first construct two planes, one in each coordinate frame, then, according to Lemma 1, rotate one of them to the other coordinate frame and intersect it with the plane constructed in that coordinate frame. The intersection of the two planes is a line, which gives the solution for the rotation axis. After the rotation axis a is determined, we can compute the rotation angle: 0 = 2 tanl(uz/a,). Since similar matrices have the same eigenvalues, any rotation matrices G and R which satisfy Eq. (8) must have the same rotation angle. Therefore 8 should be equal to the angle that is computed in the second photo station. We summarize the procedure in the following steps: STEP R13,
71
R23,
=
G23 G,3
1. Solve Eq. (4) for both photo stations to get R31,
+ +
R32,
G13,
G32 G3, ? and
G23,
Y2 =
G31,
and
G32.
G3
+
G31
G32
-
G23 ’
aY = crlaX a, = *Vl 8 = 2 tan-l
- (1 + u2)a2 1 x i
z
1
.
Clearly, each point on the great circle is associated with a unique 8. COROLLARY 1. The family of rotation axes of the solution to Eq. (4) forms a great circle on a unit 2-sphere.
Note that Theorem 1, which involves 4-D planes, can be used to solve for the rotation. In practice, however, it is more desirable to work in a smaller dimension. Corollary 1 and the following lemma allow us to do so. LEMMA 1. Let the unit vector a be a rotation axis of R. Zf b is a unit vector such that b = Ha, then b is a rotation axis of G. According to Euler’s theorem ([5], p. l-58), a real rotation matrix R must have the eigenvalue + 1. The rotation axis is the vector corresponding to the eigenvector associated with this eigenvalue. Therefore, if a is the rotation axis of R then Ra = a. On the other hand, any nontrivial vector a such that Ra = a is the rotation axis of
.
STEP 4. The rotation axis a lies at the intersection of the following two planes:
UIX - y = 0 d,x + d,y + d,z = 0. Therefore, 1
Proof.
a=- 1 u
(+I --Cd, + dyad/dz,
where c = Jl
+ (r: + (“’ ‘dZdy”‘)2.
DETERMINING
MOTION
AND DEPTH
STEP 5.
Compute 8 = 2 tan-l(uJa,). also equal to 2 tanl(YZIbZ).
FROM BINOCULAR
This angle is
+ (H,,u,, + -
4.2.1 Discussion
Solving for Translation
and Depth
Once the rotation of the object is determined, the relative depth (u,z - u,J of each object point can be computed from Eq. (3), and the translation components t,, and t,, can be computed directly from Eq. (5). To determme tlz, the relationship between u;, and ulZ should be found. To do so, we make use of the known relative geometry between photo stations. Transform an object point, for example, u,, to the second coordinate frame and let
w, = Hu, + h w; = Hu; + h. Then, similar to Eq. (4), the transformed coordinates wl and w; must satisfy the following equation
+ h, -
VI&32
+ H22~1y
+ h2 -
v1JG31.
4.4
A Numerical
vlx)G32
u;, = - l
a1
-
b’ly
-
v,yF31
= 0
(12)
(Q!2Ulz + a3)
where aI = H22G13 - H13G23 a2 = H&32
-
~3 = (HIIu;,
+ Hl2dy
(H21&
Example
A numerical example is given in this section to illustrate the mathematical procedure described above. In this example,
0.9107 -0.2440 H=
0.3333 -0.2440
1[I
0.3333
1
0.9107 -0.2440 , h =
1 .
0.3333
1
0.9107
Eight object points are generated; each photo station observes four points. The image coordinates of these points at the second time instant are generated by rotating these points around the vector (1, 2, 3) by 30” followed by a translation (- 1.O, 0.5, -0.5). This motion is performed in the first coordinate frame. Then, according to Eqs. (3) and (4), four of the points are transformed into the second coordinate frame. The coordinates of these points are
- C&y - v;&
which, in terms of the depth variables u;, and ulZ, reduces to
-
H12u1,
Clearly, only the relative depth can be determined. If the stereo point correspondence is known, the absolute depth (and translation) can be easily determined from Eq. (6). In fact, one point correspondence between the photo stations is enough.
u2= -
+ (Wlx
(H21~1x
51
VIEWS
Finally, t,, is determined by
1) The sign of the direction vector of the intersected line in Step 4 is immaterial. If we reverse the sign of a, the sign of 8 will be reversed also in Step 5. Both signs lead to the same result since rotation around +a by +8 is the same as rotation around -a by -13. By the same reason, the signs of c and d can be assigned arbitrarily. 2) If H is a rotation around the z-axis, i.e., the image planes of the two photo stations are parallel, the second photo station is redundant; it does not provide additional information about the motion of the object. In this case, the two planes in Step 4 overlap completely. Thus the method would be unable to solve the problem. In [3] we prove that the two planes overlap if and only if the image planes of the two photo stations are parallel. 4.3
ORTHOGRAPHIC
H&I + h, -
+ H22u;y
V;x)G23
+ h2 -
v;,)G13
I]+;=
l],u3=
[%].--
[;I>
HOMER
H. CHEN
Therefore, t,, = -5.7967 - 6.2967~1, which agrees with the ground true. 5.
v; =
5.0675
0.8125
3.9323 , v; =
1.1427 .
1[ 1 -5.0659
1.6789
Note that vi’s are not the corresponding points of ui’s, though the same subscripts are used. In Step 1 we find
LEAST-SQUARES
G23 = 1.0,
p =
WRY [1 WRY
,
W&I
R31 = 1.0
G13 = -1.5837,
G32 = - 1.5837, G31 = 1.0
ANALYSIS
The method described above works for ideal image data. In practice, the image data are corrupted with noise (for example, quantization or measurement noise) that should be taken into account. To handle the noisy case, we use least-squares technique for both monocular analysis and binocular analysis. For simplicity, we use matrix notation and define
RZ3= 0.3195, RI3 = -1.2407, RX2= -0.8009,
MONOCULAR
r E=
I
-@l;, - ii;>
Cd, -. SC> -(dx
-
$1
(ll,X - ii,) (u2x
-
G>
1 3
where R3, and G3r are set to 1. In Step 2 we get CT1= 2.0,
u2 = 0.2148
yI = 1.0, y2 = 0.2259 and in Step 3, we find the two planes 2x-y=o 0.5774x - 1.1547y + 0.57742 = 0.
where the bar-vectors represent the centroids of the image points. The problem here is to find a least-squares solution for Ep - e = 0.
Thus the rotation axis is
1 a=7T4
As compared to Eq. (4), the difference here is that the centroids are taken as the origin of the coordinate frames. The centroids are used because if (R*, t*) is the optimal least-squares solution,
rli : 11
ii:R;3 - $Rt
and the rotation angle is 8 = 2 tan-’
0.2148vri;i 3
= 30”
which is equal to 2 tan-‘(0.2259/0.8431) computed in the second coordinate frame. Finally, plugging the corresponding numbers into Eq. (12), we have -0.0531~;~ - 0.2839~~~+ 0.3079 = 0.
+ U,Rt2 - ii,R?~ = t,*R& + ty*R;3.
This condition is derived by using the Lagrange method [2]. Therefore, we can simplify the least-squares problem by first translating the image points {(uix, ui,)} by (AX, Ay) UJ so that at this new position the point =(iik-ii,,+ set will have a centroid that is coincident with the centroid of {(u,‘,, I&)}. A nice result of doing so is that the estimation of rotation and translation is decoupled. Now the least-squares problem can be formally stated as follows: Given the image coordinates of n correspond-
DETERMINING
MOTION
AND DEPTH
FROM BINOCULAR
ing points over two views, find p to minimize the squared error
subject to the constraint p: + p: - p: = 1
ORTHOGRAPHlC
53
VIEWS
the two rotations in the quaternion space. (In this paper we choose 0 5 8,, &, 2 7~to make the quaternion representation of rotations unique [I, 61. Note also that an equivalent expression to E can be derived by using matrix notation of the equation g 0 Ii = fi 0 i;, where 0 denotes quatemion multiplication.) Rewrite E as
(13)
which is imposed because 1 - RT3 - R& = R& = 1 R:i - R:P By using the Lagrange method, the necessary condition for a minimal solution is (ETE - AD)p - ETe = 0
where g and i are the unit quatemion representations of G and R respectively. Since gTg = iTi = 1 and HTH = I,
(14)
where the superscript T denotes matrix transposition, h is the Lagrangian multiplier (a real number), and
E = 2 - 2gT&. Therefore, the problem is equivalent to maximizing E’ = gT&. Since g2 = y&i, g3 = y2g4, f2 = air,, and f3 = q2r4. we can rewrite E’ as
LaJ
I[1 fl
HII + YIHZI + mW12 + y&22) fl2(Hl3 + Y&23) Although Eq. (14) is nonlinear, it can be solved easily. First treat h as a known number and solve for p as a 1 + c2YZH33 ih Y2H3l + fllY2H32 function of A, then substitute the resulting p into Eq. (13) and compute the value for A. This results in finding the zeros of a 6th-order polynomial of A. The zeros could be Then the least-squares problem can be formally stated real or complex. The real zero which gives rise to the as follows: minimum squared-error is the optimal solution. Once A is found, the optimal p is readily available. Maximize As in the ideal case, the four rotation unknowns RZ3, Ri3, R32,and Rji (and similarly G23,G~J,Gj2, and Gji) can only be determined to within a scale factor. Again, the locus of rotation obtained from each photo station is a subject to the following two conditions great circle on a unit 3-sphere.
sa@.
6.
LEAST-SQUARES
BINOCULAR
zz 1. iT1+a:0If~iTbf
ANALYSIS
6.1 Solving the Rotation
0
Ideally, we know that b, the rotation axis associated with G, should be equal to Ha and that the rotation angles, & and 19,, associated with G and R respectively should be the same. However, in the presence of noise, the equation GH - HR = 0 may not necessarily hold. In this case, we define the error to be minimized as 2
E = b sin? - Ha sin 2 !I
cos $’ - cos ?
and want to find a least-squares solution for it. The error E measures the square of the Euclidean distance between
1 + a:
(16)
The two conditions are imposed because g and f are unit 4-vectors. The problem can be solved by introducing two Lagrangian multipliers pI and ~2 and by maximizing E” = fjTf& + ~~(1 - fjT@ + &I
- fTI%).
Differentiating with respect to g and i and setting the results to zero yields: Ai - 2&g
= 0
(17)
ATg - 2p&
= 0
(18)
HOMER H. CHEN
54
Premultiplying obtain
Eq. (17) by gT and Eq. (18) by iT, we gTiii = 2/L, iTBT$j = 2/.,&z.
0.18 -
Therefore, pI = p2 = p. Since e and B are nonsingular, we have from Eqs. (17) and (18)
0.10
This is a typical eigenvalue problem. Therefore i can be determined by solving for the eigenvector associated with the largest positive eigenvalue of the matrix 8-‘fiT&lfr, with the magnitude of i constrained by Eq. (16). Once f,, f4, and hence f2 and f3, are obtained, the rotation matrix R can be easily constructed [81. 6.2
Solving for Translation
0.08 1;
(w;~ - V:)G23 - (wb - V;)Gn + (Wix - VJG32 - (Wiy - Vy)G31= 0 where wf and wi are the positions of u: and ui in the second coordinate frame, V’ is the centroid of {v:}, and V
8 0.14
5
r
-3
-2
-o---o0.05
0.10
--0.15 0.20 Image Noise
I
0.10
.
I
I
0.20 0.15 Image Noise
I.,
1..
0.25
1 I
0.30
is the centroid of {Vi}. The least-squares solution for these unknowns is straightforward. However, as discussed earlier, we can only determine the relative values of these unknowns. 6.3
0.16
..I.
0.05
FIG. 4. Errors of rotation due to image noise of fixed magnitude.
and Depth
The remaining unknowns t,, t,, t,, u~‘s, and uiz’s can be determined by the following equations
0.18
--e---
1 --e
---o-
-He--0.25
0.30
FIG. 3. Errors of rotation due to image noise of random magnitude.
Simulations
In the simulations, we tested the algorithm on random points generated by a SUN 3160. These points were uniformly distributed within a unit box centered at (0.5, 0.5, 0.5). Ten points were generated for each photo station in each run of the simulations. Artificial image noise was added to the (x, y) coordinates of the points. The noise added to each point was within a circle of radius r centered at the point. This was done by generating two random numbers: one for the orientation of the noise vector on the image plane and the other for the magnitude of the noise. We tried six different noise levels, ranging from r = 0.05 to r = 0.30. In these simulations the second photo station was rotated about the x-axis of the first photo station by 30” and then translated by (1, 1, I). The simulated motion of the object was a rotation around the vector (1, 2, 3) by 30” followed by a translation (- 1, 0.5, -0.5). Results of the first simulation are shown in Figure 3, where the scale for the errors of quaternion and axis is shown on the left side, and the scale for the error of angle (in degrees) is shown on the right side. All the errors were obtained by taking the absolute value of the difference between the real data and the computed data. Particularly, the error of angle was defined to be I(tYi2) (8,,,1/2)). The results were obtained by running 100 sets of random points; each was repeated with 30 sets of image noise. In order words, each point shown on the figure
DETERMINING
MOTION AND DEPTH FROM BINOCULAR
ORTHOGRAPHIC
VIEWS
55
motion of the object from two monocular views. The solution for the rotation has been shown to be a oneparameter family which forms a great circle on the unit quaternion sphere. Then, the algorithm determines the rotation from binocular data by finding the intersection of two great circles. The least-square solution for noisy data has also been described. Determining the translation and depth is relatively straightforward, once the rotation has been found. REFERENCES
t;....,....~....,....,....,~o 0.05 0.10
0.15 0.20 Image Noise
0.25
0.30
FIG. 5. Errors of rotation when the angle between cameras is reduced from 30” to IO”.
is a result of 3000 tests. Figure 4 shows the results of the same simulation but with image noise of constant magnitude (instead of randomly varying from 0 to r). As was expected, the errors increased. The last simulation reduced the angle between the photo stations from 30” to lo”. Results shown in Figure 5 clearly indicate that better performance is achieved when the angle between the photo stations is increased. 7.
SUMMARY
We have presented an algorithm for computing the motion and depth of an object from binocular orthographic views of the object. The main emphasis of the work is on the integration of partial pieces of motion information obtained from two photo stations. A geometric approach has been developed. A numerical example on exact data and simulations on noisy data have been described to illustrate the approach. The algorithm first computes the
1. L. Brand, Vector and Tensor Analysis, John Wiley, New York, NY, 1947. 2. H. H. Chen and T. S. Huang, “Using motion from orthographic views to verify 3-D point matches,” to appear in IEEE Trans. Pattern Analysis and Machine Intelligence. 3. H. H. Chen, “Motion and depth from binocular orthographic views, II. General case and analysis,” in preparation. 4. D. B. Gennery, “Stereo vision for the acquisition and tracking of moving three-dimensional objects, ” in Techniques for 3-D Machine Perception, ed., A. Rosenfeld, North-Holland, 1986. 5. H. Goldstein, Classical Mechanics, Addison-Wesley, 2nd Ed., Reading, MA, 1981. 6. B. K. P. Horn, Robot Vision, MIT Press, Cambridge, MA & McGraw-Hill, New York, NY, 1986. 7. T. S. Huang and C. H. Lee, “Motion and structure from orthographic projections,” Proc. Int’l. Conf. Pattern Recognition, Miami Beach, FL, 1988, pp. 885-887. 8. G. Kom and T. Kom, Mathematical Handbook for Scientists and Engineers, McGraw-Hill, Inc., 1961. 9. C. H. Lee, Structure from Motion: An Augmented Problem and a New Algorithm, Technical Report, Dept. of Computer Science, Purdue University, Sept. 1986. 10. L. Matthies and S. Shafer, “Error modeling in stereo navigation,” IEEE J. Robotics and Automation, vol. RA-3, no. 3, 1987, pp. 239248. 11. D. Sommerville, Introduction to the Geometry of N Dimensions, Dover Publications, Inc., New York, NY, 1958. 12. S. Ullman, The Interpretation of Visual Motion, MIT Press, Cambridge, MA, 1979. 13. X. Zhuang, T. S. Huang, and R. M. Harahck, “A simple procedure to solve motion and structure from three orthographic views,” IEEE .I. Robotics and Automation, vol. 4, no. 2, 1988,pp. 236-239.
IMAGE
UNDERSTANDING
Vol. 54, No. 1, July, pp. 47-55, 1991
Determining
Motion and Depth from Binocular Orthographic
Views
HOMER H. CHEN Machine Perception Research Department, AT&T Bell Laboratories, Room 4E-630, Crawfords Corner Road, Holmdel, New Jersey 07733 Communicated by Thomas S. Huang Received February 9, 1990
ferent approach. Instead of focusing on error modeling, The problemaddressedin this paperarisesin motion estimation our ideal is to eliminate completely the contribution of from stereoimagesequences. The imagesequences are taken from depth errors to motion estimates. This is done by discardtwo stereophoto stations which look at an objectfrom different ing the depth coordinate in computing the motion paramangles.To eliminate the effect of depth errors on motion estima- eters. Thus, the input data can be thought of as orthotion, we discardthe noisy z coordinateand keeponly the x and y graphic views of the object. This approach is found to be coordinates.Therefore, the problem becomescomputing motion useful in dealing with noisy data where one coordinate and depth of an objectfrom binocular orthographicviews.A geoerror is more dominant than errors in the other coordimetric approachis presented,and the solutionsfor both exactand nates. The idea has been successfully applied to verify noisy imagedata are described. o 1991 Academic press, inc. 3-D point matches [2]. Here, we apply it to estimate motion parameters. Therefore, the current problem can be formulated as I. INTRODUCTION computing the motion and depth of an object from binocIf a robot is to plan and execute actions intelligently, it ular orthographic views taken from two widely separated viewing angles. As mentioned earlier, the relative orienmust interact with its environment and interpret correctly the sensory data it obtains. In many applications of robot tation and position between the two photo stations are vision, one needs to acquire information about the posi- known. Object features, such as corners of a polyhedron, tion and motion of a robot relative to a target object. For are detected and tracked over time in each photo station. example, in tracking a moving object for teleconferenc- That is, point correspondences between temporal frames are known; however, point correspondences between ing, the orientation of the camera must be continuously adjusted to keep the target object in the field of view of spatial frames are not assumed. the camera. Another example is the retrieval of a satellite For a monocular image sequence, Ullman has shown using a space shuttle. If the satellite is tumbling, it is that two orthographic views are insufficient and that the necessary to match the motion of a robot hand to the motion/structure of an object can be uniquely determined motion of the satellite so that excessive forces and by tracking four points over three orthographic views torques are not produced when the hand grasps the satel- [ 121.This classical motion problem has recently been relite [4]. visited by several researchers, and linear methods (for The work described in this paper is abstracted from an perfect data) have been found [7], [9], [ 131. In an earlier application task similar to the second example described paper [2], we described a least-square algorithm for above in which two robot hands, each mounted with a monocular image data. We first review the monocular pair of stereo cameras, are commanded to grasp a moving case and show that, with two views, the solution for the object. The relative position and orientation between the rotation is a one-parameter family which forms a great two robot hands at any time instant are known. How- circle on the unit quaternion sphere. We then describe ever, it is possible for the two photo stations to observe how to determine the motion parameters from binocular completely different object points. data without stereo point correspondences. Given stereo image sequences, the depth of the object can be computed by the simple triangulation technique. 2. PROBLEMSTATEMENT To improve the accuracy of motion estimates, much recent work has been directed toward the development of Two orthographic views at time instants Tl and T2, more sophisticated methods for modeling the measure- respectively, are taken of the moving rigid object from ment errors (for a brief review, see [lo]). We take a dif- each photo station. By processing these orthographic 47 1049~9&o/91 All
$3.00
Copyright 0 1991 by Academic Press, Inc. rights of reproductionin any form reserved.
48
HOMER H. CHEN
Using matrix notation, Eq. (1) can be written as
Suppose the object is orthographically projected along the z-axis onto the x-y plane and the x- and y-coordinates of an objectpoint are taken as its image coordinates, then we have camera
1
camera 2
FIG. 1. Basic geometry of the problem: Orthographic views of a moving object at time instants Tl and T2 are taken from two cameras with different viewing angles.
views, we want to determine the motion and position of the object. Without loss of generality, we assume that the image planes are stationary and set up two coordinate frames, one on each photo station. The basic geometry of the problem is shown in Figure 1, where R and t represent the rotation and translation of the object measured in the first coordinate frame, G and g denote the motion measured in the second coordinate frame, and H and h denote the relative orientation and position between the two coordinate frames. Suppose there are n points. The positions of the points at time Tl in the first coordinate frame are represented by ui, and the positions of the points in the second coordinate frame are represented by vi. The positions of the points at time T2 are primed. Therefore,
(3) Note that uiz’s are unavailable from the image data. To eliminate (uiz - ulZ), we premultiply both sides of Eq. (3) by the row vector [RZ3- RI31to get Cd - u;x)R23 - (dy - u;y)R13 + (uix - ~1x)R32 - (uiy - ~1ylJ331= 0, i=2,.
. . ,n
(1)
RzRu - klR23 =
R32
V( = Gvi + g
(2)
Rdb3 - Rdb
R31
MONOCULAR
ANALYSIS-A
REVIEW
The first step of our approach computes the motion/ structure information of the object by using the image data obtained from each photo station. For monocular analysis, we adopt the linear method developed by Huang and Lee [7] to show how much motion information can be recovered from two orthographic views.
(4)
where the following two identities
U; = Rui + t
where i = 1, 2, . . . , n. Given H, h, and the image data of the points at two time instants, the problem is to determine the motion and depth of the object. In the following, we will first show that only partial motion information can be recovered from monocular image data. Then we will show how to make use of the known relative geometry between the photo stations to determine the complete motion information. 3.
The term [t, t,lT on the right hand side can be eliminated by subtracting the equation of point uI from the other equations. Thus
=
are used to simplify the expression. These two identities are derived from the orthogonality of a rotation matrix for which the cross product of the first two rows is equal to the third row, following the right-hand rule. Eq. (4) contains four unknowns R13,R23, R31,and R32. (Note: the remaining elements in R can be expressed as functions of these four unknowns [7], [8].) Since Eq. (4) is a set of linear and homogeneous equations in the four unknowns, the rotation cannot be uniquely determined from two views. To state it more clearly: l
l
The only thing we can determine about R is the values of these four unknowns, Ri3, R23, R31, and R32, to within a scale factor. That is, we can only determine the relative values of these four unknowns, not their absolute values. The relative depth (uiz - uiZ) cannot be determined
DETERMINING
MOTION
AND DEPTH
FROM BINOCULAR
if the scale factor of the four unknowns is unsolved (see Eq. (3)), nor can the translation be determined. Note that the operation performed in deriving Eq. (4) is equivalent to decomposing the motion in such a way that ul is brought to coincide with ul by the translation
ORTHOGRAPHIC
49
VIEWS
tion involved is very tedious. Described here is a linear method using the unit quaternion representation of rotation. We first explore some mathematical characteristics of the rotation information obtained from the monocular analysis. 1. The solution to Eq. (4) is a family of rotations which forms a great circle on the unit quaternion sphere.
THEOREM
ru;,
- ulxl
tj=ILliy - I
(5)
UlY
I,! LUlZ
-
Prosf. Let a be the rotation axis and 8 be the rotation angle. Then the rotation matrix is
11 UlZJ
The rotation under this decomposition takes place around the point u;, as opposed to around the origin of the coordinate frame. Therefore, the translation tl is different from t in Eq. (1). The latter is equal to u; - Rul . The rotation matrix is unchanged, however.
B =
a’, + (1 - ai) cos 8
axay[ - a, sin 0
aXay[ + a, sin 8
at + (1 - at) cos 0
[ axaz[ - aY sin 8
a,a,t + a, sin 8 a,a,.$ + aY sin 8
4.
PIECING
TOGETHER
MONOCULAR
MOTION
DATA
In this section, we show how to make use of the known relative geometry between the photo stations to integrate monocular motion data. 4.1
Relationship
between Monocular
Motion
Data
a,a& - a, sin 8 a2 + (1 - a:) cos 8 where 5 = 1 - cos 0. A closer look at the rotation matrix reveals that
The coordinates of a point in the two photo stations are related by vi = Hui + h
(6)
vi = Hu; + h.
(7)
Substituting uf of Eq. (1) into Eq. (7) yields Vf
= HRH-‘vi
R32
+
&I
=
3
= -
Ul
a,
(10)
and R13
+
R31
R32
-
R23
I3
= a, tan- = c2. 2
(11)
+ (h + Ht - HRH-‘h).
(8)
to represent the rotation, where El = a, sin(6/2), r2 = ay sin(0/2), r3 = a, sin(8/2), and 4 = cos(0/2). Then Eqs. (10) and (11) can be rewritten as
(9)
f2 = aIf{
Comparing this equation to Eq. (2), we thus have G = HRH-’ g = h + Ht - HRH-‘h.
These two equations describe the relationship between the motions observed by the two photo stations. The rotation can be uniquely determined by solving Eq. (8); however, the translation can only be determined to within an additive constant if there is no stereo correspondence. 4.2
+
h
Because the common scale factor is canceled out in these two equations, o1 and u2 are constants. Now, use a unit quaternion [l]
= HRUi + (h + Ht).
Using Eq. (6) to substitute vi for ni yields Vi
R23
A Geometric Approach for Solving the Rotation
Although directly solving for the two unknown scale factors of R and G from Eq. (8) is possible, the computa-
113 =
(T2f4
which represent two hyperplanes in the quaternion space. These two hyperplanes are orthogonal to each other and all pass through the origin. The intersection of these two hyperplanes is a plane that also passes through the origin. Since i is a unit vector, the family of rotations defined by Eq. (4) lies at the intersection of the plane with the unit quaternion sphere. A plane passing through the center of a unit n-sphere intersects the sphere at a great
50
HOMER H. CHEN
R. Postmultiplying both sides of the equation GH = HR by a yields GHa = Ha. Since b = Ha, we therefore prove that b is the rotation axis of G. Q.E.D.
FIG. 2. The locus of the family of rotation axes is a great circle lying on a plane that is perpendicular to the x-y plane. The angle between the plane and the x-z plane is tan-‘u, .
circle [ 111.Therefore, the locus of the family of rotations is a great circle on a unit 3-sphere. Q.E.D. The family of rotation axes defined by Eq. (10) also has one free parameter. It should be clear that the locus of the family of rotation axes is a great circle on a unit 2sphere centered at the origin (see Figure 2). The plane of the great circle is perpendicular to the x-y plane and forms an angle tan-l o1 with the y-z plane. Considering a, as a free parameter, we can represent the remaining rotation parameters by
Therefore, according to Corollary 1, we can first construct two planes, one in each coordinate frame, then, according to Lemma 1, rotate one of them to the other coordinate frame and intersect it with the plane constructed in that coordinate frame. The intersection of the two planes is a line, which gives the solution for the rotation axis. After the rotation axis a is determined, we can compute the rotation angle: 0 = 2 tanl(uz/a,). Since similar matrices have the same eigenvalues, any rotation matrices G and R which satisfy Eq. (8) must have the same rotation angle. Therefore 8 should be equal to the angle that is computed in the second photo station. We summarize the procedure in the following steps: STEP R13,
71
R23,
=
G23 G,3
1. Solve Eq. (4) for both photo stations to get R31,
+ +
R32,
G13,
G32 G3, ? and
G23,
Y2 =
G31,
and
G32.
G3
+
G31
G32
-
G23 ’
aY = crlaX a, = *Vl 8 = 2 tan-l
- (1 + u2)a2 1 x i
z
1
.
Clearly, each point on the great circle is associated with a unique 8. COROLLARY 1. The family of rotation axes of the solution to Eq. (4) forms a great circle on a unit 2-sphere.
Note that Theorem 1, which involves 4-D planes, can be used to solve for the rotation. In practice, however, it is more desirable to work in a smaller dimension. Corollary 1 and the following lemma allow us to do so. LEMMA 1. Let the unit vector a be a rotation axis of R. Zf b is a unit vector such that b = Ha, then b is a rotation axis of G. According to Euler’s theorem ([5], p. l-58), a real rotation matrix R must have the eigenvalue + 1. The rotation axis is the vector corresponding to the eigenvector associated with this eigenvalue. Therefore, if a is the rotation axis of R then Ra = a. On the other hand, any nontrivial vector a such that Ra = a is the rotation axis of
.
STEP 4. The rotation axis a lies at the intersection of the following two planes:
UIX - y = 0 d,x + d,y + d,z = 0. Therefore, 1
Proof.
a=- 1 u
(+I --Cd, + dyad/dz,
where c = Jl
+ (r: + (“’ ‘dZdy”‘)2.
DETERMINING
MOTION
AND DEPTH
STEP 5.
Compute 8 = 2 tan-l(uJa,). also equal to 2 tanl(YZIbZ).
FROM BINOCULAR
This angle is
+ (H,,u,, + -
4.2.1 Discussion
Solving for Translation
and Depth
Once the rotation of the object is determined, the relative depth (u,z - u,J of each object point can be computed from Eq. (3), and the translation components t,, and t,, can be computed directly from Eq. (5). To determme tlz, the relationship between u;, and ulZ should be found. To do so, we make use of the known relative geometry between photo stations. Transform an object point, for example, u,, to the second coordinate frame and let
w, = Hu, + h w; = Hu; + h. Then, similar to Eq. (4), the transformed coordinates wl and w; must satisfy the following equation
+ h, -
VI&32
+ H22~1y
+ h2 -
v1JG31.
4.4
A Numerical
vlx)G32
u;, = - l
a1
-
b’ly
-
v,yF31
= 0
(12)
(Q!2Ulz + a3)
where aI = H22G13 - H13G23 a2 = H&32
-
~3 = (HIIu;,
+ Hl2dy
(H21&
Example
A numerical example is given in this section to illustrate the mathematical procedure described above. In this example,
0.9107 -0.2440 H=
0.3333 -0.2440
1[I
0.3333
1
0.9107 -0.2440 , h =
1 .
0.3333
1
0.9107
Eight object points are generated; each photo station observes four points. The image coordinates of these points at the second time instant are generated by rotating these points around the vector (1, 2, 3) by 30” followed by a translation (- 1.O, 0.5, -0.5). This motion is performed in the first coordinate frame. Then, according to Eqs. (3) and (4), four of the points are transformed into the second coordinate frame. The coordinates of these points are
- C&y - v;&
which, in terms of the depth variables u;, and ulZ, reduces to
-
H12u1,
Clearly, only the relative depth can be determined. If the stereo point correspondence is known, the absolute depth (and translation) can be easily determined from Eq. (6). In fact, one point correspondence between the photo stations is enough.
u2= -
+ (Wlx
(H21~1x
51
VIEWS
Finally, t,, is determined by
1) The sign of the direction vector of the intersected line in Step 4 is immaterial. If we reverse the sign of a, the sign of 8 will be reversed also in Step 5. Both signs lead to the same result since rotation around +a by +8 is the same as rotation around -a by -13. By the same reason, the signs of c and d can be assigned arbitrarily. 2) If H is a rotation around the z-axis, i.e., the image planes of the two photo stations are parallel, the second photo station is redundant; it does not provide additional information about the motion of the object. In this case, the two planes in Step 4 overlap completely. Thus the method would be unable to solve the problem. In [3] we prove that the two planes overlap if and only if the image planes of the two photo stations are parallel. 4.3
ORTHOGRAPHIC
H&I + h, -
+ H22u;y
V;x)G23
+ h2 -
v;,)G13
I]+;=
l],u3=
[%].--
[;I>
HOMER
H. CHEN
Therefore, t,, = -5.7967 - 6.2967~1, which agrees with the ground true. 5.
v; =
5.0675
0.8125
3.9323 , v; =
1.1427 .
1[ 1 -5.0659
1.6789
Note that vi’s are not the corresponding points of ui’s, though the same subscripts are used. In Step 1 we find
LEAST-SQUARES
G23 = 1.0,
p =
WRY [1 WRY
,
W&I
R31 = 1.0
G13 = -1.5837,
G32 = - 1.5837, G31 = 1.0
ANALYSIS
The method described above works for ideal image data. In practice, the image data are corrupted with noise (for example, quantization or measurement noise) that should be taken into account. To handle the noisy case, we use least-squares technique for both monocular analysis and binocular analysis. For simplicity, we use matrix notation and define
RZ3= 0.3195, RI3 = -1.2407, RX2= -0.8009,
MONOCULAR
r E=
I
-@l;, - ii;>
Cd, -. SC> -(dx
-
$1
(ll,X - ii,) (u2x
-
G>
1 3
where R3, and G3r are set to 1. In Step 2 we get CT1= 2.0,
u2 = 0.2148
yI = 1.0, y2 = 0.2259 and in Step 3, we find the two planes 2x-y=o 0.5774x - 1.1547y + 0.57742 = 0.
where the bar-vectors represent the centroids of the image points. The problem here is to find a least-squares solution for Ep - e = 0.
Thus the rotation axis is
1 a=7T4
As compared to Eq. (4), the difference here is that the centroids are taken as the origin of the coordinate frames. The centroids are used because if (R*, t*) is the optimal least-squares solution,
rli : 11
ii:R;3 - $Rt
and the rotation angle is 8 = 2 tan-’
0.2148vri;i 3
= 30”
which is equal to 2 tan-‘(0.2259/0.8431) computed in the second coordinate frame. Finally, plugging the corresponding numbers into Eq. (12), we have -0.0531~;~ - 0.2839~~~+ 0.3079 = 0.
+ U,Rt2 - ii,R?~ = t,*R& + ty*R;3.
This condition is derived by using the Lagrange method [2]. Therefore, we can simplify the least-squares problem by first translating the image points {(uix, ui,)} by (AX, Ay) UJ so that at this new position the point =(iik-ii,,+ set will have a centroid that is coincident with the centroid of {(u,‘,, I&)}. A nice result of doing so is that the estimation of rotation and translation is decoupled. Now the least-squares problem can be formally stated as follows: Given the image coordinates of n correspond-
DETERMINING
MOTION
AND DEPTH
FROM BINOCULAR
ing points over two views, find p to minimize the squared error
subject to the constraint p: + p: - p: = 1
ORTHOGRAPHlC
53
VIEWS
the two rotations in the quaternion space. (In this paper we choose 0 5 8,, &, 2 7~to make the quaternion representation of rotations unique [I, 61. Note also that an equivalent expression to E can be derived by using matrix notation of the equation g 0 Ii = fi 0 i;, where 0 denotes quatemion multiplication.) Rewrite E as
(13)
which is imposed because 1 - RT3 - R& = R& = 1 R:i - R:P By using the Lagrange method, the necessary condition for a minimal solution is (ETE - AD)p - ETe = 0
where g and i are the unit quatemion representations of G and R respectively. Since gTg = iTi = 1 and HTH = I,
(14)
where the superscript T denotes matrix transposition, h is the Lagrangian multiplier (a real number), and
E = 2 - 2gT&. Therefore, the problem is equivalent to maximizing E’ = gT&. Since g2 = y&i, g3 = y2g4, f2 = air,, and f3 = q2r4. we can rewrite E’ as
LaJ
I[1 fl
HII + YIHZI + mW12 + y&22) fl2(Hl3 + Y&23) Although Eq. (14) is nonlinear, it can be solved easily. First treat h as a known number and solve for p as a 1 + c2YZH33 ih Y2H3l + fllY2H32 function of A, then substitute the resulting p into Eq. (13) and compute the value for A. This results in finding the zeros of a 6th-order polynomial of A. The zeros could be Then the least-squares problem can be formally stated real or complex. The real zero which gives rise to the as follows: minimum squared-error is the optimal solution. Once A is found, the optimal p is readily available. Maximize As in the ideal case, the four rotation unknowns RZ3, Ri3, R32,and Rji (and similarly G23,G~J,Gj2, and Gji) can only be determined to within a scale factor. Again, the locus of rotation obtained from each photo station is a subject to the following two conditions great circle on a unit 3-sphere.
sa@.
6.
LEAST-SQUARES
BINOCULAR
zz 1. iT1+a:0If~iTbf
ANALYSIS
6.1 Solving the Rotation
0
Ideally, we know that b, the rotation axis associated with G, should be equal to Ha and that the rotation angles, & and 19,, associated with G and R respectively should be the same. However, in the presence of noise, the equation GH - HR = 0 may not necessarily hold. In this case, we define the error to be minimized as 2
E = b sin? - Ha sin 2 !I
cos $’ - cos ?
and want to find a least-squares solution for it. The error E measures the square of the Euclidean distance between
1 + a:
(16)
The two conditions are imposed because g and f are unit 4-vectors. The problem can be solved by introducing two Lagrangian multipliers pI and ~2 and by maximizing E” = fjTf& + ~~(1 - fjT@ + &I
- fTI%).
Differentiating with respect to g and i and setting the results to zero yields: Ai - 2&g
= 0
(17)
ATg - 2p&
= 0
(18)
HOMER H. CHEN
54
Premultiplying obtain
Eq. (17) by gT and Eq. (18) by iT, we gTiii = 2/L, iTBT$j = 2/.,&z.
0.18 -
Therefore, pI = p2 = p. Since e and B are nonsingular, we have from Eqs. (17) and (18)
0.10
This is a typical eigenvalue problem. Therefore i can be determined by solving for the eigenvector associated with the largest positive eigenvalue of the matrix 8-‘fiT&lfr, with the magnitude of i constrained by Eq. (16). Once f,, f4, and hence f2 and f3, are obtained, the rotation matrix R can be easily constructed [81. 6.2
Solving for Translation
0.08 1;
(w;~ - V:)G23 - (wb - V;)Gn + (Wix - VJG32 - (Wiy - Vy)G31= 0 where wf and wi are the positions of u: and ui in the second coordinate frame, V’ is the centroid of {v:}, and V
8 0.14
5
r
-3
-2
-o---o0.05
0.10
--0.15 0.20 Image Noise
I
0.10
.
I
I
0.20 0.15 Image Noise
I.,
1..
0.25
1 I
0.30
is the centroid of {Vi}. The least-squares solution for these unknowns is straightforward. However, as discussed earlier, we can only determine the relative values of these unknowns. 6.3
0.16
..I.
0.05
FIG. 4. Errors of rotation due to image noise of fixed magnitude.
and Depth
The remaining unknowns t,, t,, t,, u~‘s, and uiz’s can be determined by the following equations
0.18
--e---
1 --e
---o-
-He--0.25
0.30
FIG. 3. Errors of rotation due to image noise of random magnitude.
Simulations
In the simulations, we tested the algorithm on random points generated by a SUN 3160. These points were uniformly distributed within a unit box centered at (0.5, 0.5, 0.5). Ten points were generated for each photo station in each run of the simulations. Artificial image noise was added to the (x, y) coordinates of the points. The noise added to each point was within a circle of radius r centered at the point. This was done by generating two random numbers: one for the orientation of the noise vector on the image plane and the other for the magnitude of the noise. We tried six different noise levels, ranging from r = 0.05 to r = 0.30. In these simulations the second photo station was rotated about the x-axis of the first photo station by 30” and then translated by (1, 1, I). The simulated motion of the object was a rotation around the vector (1, 2, 3) by 30” followed by a translation (- 1, 0.5, -0.5). Results of the first simulation are shown in Figure 3, where the scale for the errors of quaternion and axis is shown on the left side, and the scale for the error of angle (in degrees) is shown on the right side. All the errors were obtained by taking the absolute value of the difference between the real data and the computed data. Particularly, the error of angle was defined to be I(tYi2) (8,,,1/2)). The results were obtained by running 100 sets of random points; each was repeated with 30 sets of image noise. In order words, each point shown on the figure
DETERMINING
MOTION AND DEPTH FROM BINOCULAR
ORTHOGRAPHIC
VIEWS
55
motion of the object from two monocular views. The solution for the rotation has been shown to be a oneparameter family which forms a great circle on the unit quaternion sphere. Then, the algorithm determines the rotation from binocular data by finding the intersection of two great circles. The least-square solution for noisy data has also been described. Determining the translation and depth is relatively straightforward, once the rotation has been found. REFERENCES
t;....,....~....,....,....,~o 0.05 0.10
0.15 0.20 Image Noise
0.25
0.30
FIG. 5. Errors of rotation when the angle between cameras is reduced from 30” to IO”.
is a result of 3000 tests. Figure 4 shows the results of the same simulation but with image noise of constant magnitude (instead of randomly varying from 0 to r). As was expected, the errors increased. The last simulation reduced the angle between the photo stations from 30” to lo”. Results shown in Figure 5 clearly indicate that better performance is achieved when the angle between the photo stations is increased. 7.
SUMMARY
We have presented an algorithm for computing the motion and depth of an object from binocular orthographic views of the object. The main emphasis of the work is on the integration of partial pieces of motion information obtained from two photo stations. A geometric approach has been developed. A numerical example on exact data and simulations on noisy data have been described to illustrate the approach. The algorithm first computes the
1. L. Brand, Vector and Tensor Analysis, John Wiley, New York, NY, 1947. 2. H. H. Chen and T. S. Huang, “Using motion from orthographic views to verify 3-D point matches,” to appear in IEEE Trans. Pattern Analysis and Machine Intelligence. 3. H. H. Chen, “Motion and depth from binocular orthographic views, II. General case and analysis,” in preparation. 4. D. B. Gennery, “Stereo vision for the acquisition and tracking of moving three-dimensional objects, ” in Techniques for 3-D Machine Perception, ed., A. Rosenfeld, North-Holland, 1986. 5. H. Goldstein, Classical Mechanics, Addison-Wesley, 2nd Ed., Reading, MA, 1981. 6. B. K. P. Horn, Robot Vision, MIT Press, Cambridge, MA & McGraw-Hill, New York, NY, 1986. 7. T. S. Huang and C. H. Lee, “Motion and structure from orthographic projections,” Proc. Int’l. Conf. Pattern Recognition, Miami Beach, FL, 1988, pp. 885-887. 8. G. Kom and T. Kom, Mathematical Handbook for Scientists and Engineers, McGraw-Hill, Inc., 1961. 9. C. H. Lee, Structure from Motion: An Augmented Problem and a New Algorithm, Technical Report, Dept. of Computer Science, Purdue University, Sept. 1986. 10. L. Matthies and S. Shafer, “Error modeling in stereo navigation,” IEEE J. Robotics and Automation, vol. RA-3, no. 3, 1987, pp. 239248. 11. D. Sommerville, Introduction to the Geometry of N Dimensions, Dover Publications, Inc., New York, NY, 1958. 12. S. Ullman, The Interpretation of Visual Motion, MIT Press, Cambridge, MA, 1979. 13. X. Zhuang, T. S. Huang, and R. M. Harahck, “A simple procedure to solve motion and structure from three orthographic views,” IEEE .I. Robotics and Automation, vol. 4, no. 2, 1988,pp. 236-239.