- Email: [email protected]
Image irradiance equations for a zooming camera
Pattern RecognitionLetters 10 (1989) 189-194 North-Holland
September 1989
Image irradiance equations for a zooming camera Jens A R N S P A N G and Jun MA DIKU, Computer Science Department, University o[Copenhagen, Universitetsparken I, DK 2100 0, Copenhagen, Denmark
Received 19 August 1988 Revised 1 March 1989 Abstract: Image irradianceequations for a camera, zoomingonto a static surface,are discussed. Surfacerange may be determined
directly from image irradiance. The classic motion constraint equation is not valid during zooming; its exact and approximate extensions are discussed. Key words: Vision theory, image irradiance,motion constraint equations, surface depth.
1. Introduction It has been suggested that determination of surface depth in a static scene may be achieved by changing the focal length (Ma & Olsen, 1988); surface depth may be determined if the original focal length and the zoom rate are known, and the resulting image displacement has been determined. In (Ma & Olsen, 1988) two discrete focal lengths with significant separation were used, and image displacement was determined by a matching technique. In this paper local relations are derived for an image sequence, produced from a static surface by a continuous change of focal length, i.e. a zooming operation. It is noted, first that optic flow produced by a continuous zooming operation is not automatically a solution to the classic motion constraint equation of Horn and Schunck (Horn & Schunck, 1981), not even under the same surface illumination conditions as in (Horn & Schunck, 1981). An explanation for this is given and extended motion constraint equations are derived. Secondly, if a zooming operation with constant zooming rate is performed without moving the optic centre, a certain local differential relation among image in-
tensities is always valid, regardless the surface illumination. Thirdly, if the zooming operation involves moving the optic centre, surface depth becomes part of the image irradiance equation, and surface reflectance properties become essential. In the case of a Lambertian surface, a closed form solution for surface depth from local intensities and camera parameters is derived; this relation does not involve optic flow nor any other image displacements as (Ma & Olsen, 1988) does. Throughout this paper the optic system is modelled by one thin lens, capable of changing its focal length continuously. Modern varifocal glasses, and many modern optics for photographic use, may be modelled thus, although in the latter case a model using two principal planes, as explained in (Klein, 1970), might be a better approximation; such refined models are left for further research.
2. The classic image irradiance equation The relation between surface radiance and image irradiance for a thin fixed focus lens has been derived in (Horn, 1986, Ch. 10). The result is resumed here; consider Figure 1.
0167-8655/89/$3.50 O 1989.ElsevierSciencePublishers B.V. (North-Holland)
189
Volume10, Number3
PATTERNRECOGNITIONLETTERS imageplane
E
lens
-
s~
-
September1989 xrface
~
B: opticangle d: aperturediameter f: effectivefocallength L: surfaceradiance E: imageirradiance Figure 1. Imageformationby a thin lens. We have the relation: E = ¼rcd2f - 2 cos4fl L
(1)
where d and f are constants for a fixed focus lens and fl is known at any image position. This classic image irradiance equation (1) is the analytic basis for much work performed in photometry, shape from shading and optic flow, as surveyed in (Horn, 1986).
3. The image irradiance equation during zoom with fixed position optic centre
x(t) = - f(t)
xz-
1,
y(t) = - f(t)
YZ
1.
By taking the time derivative on either side of (2) and afterwards using (2) for substitution, we obtain expressions for the optic flow (u(t), f i t ) ) which is produced: dx(t) u(t)=
190
dr(t) -
dt
- -
XZ-1
dt
= f ( t ) - I d f ( t ) x(t), dt v(t) =
Consider the case where zooming is performed by continuously changing the effective focal lengthf(t) with time t in such a way that only the image plane is moved relatively to a spatial reference system, but not the optic center of the lens and not the surface either. A variant of the classic image irradiance equation (1) shall be derived, and also a variant of the classic motion constraint equation (Horn & Schunck, 1981) shall be derived. Firstly, note an expression for the optic flow produced by the zooming operation; consider Figure 2. From the geometry of the camera, the transformation is obtained between spatial coordinates (X, Y,Z) of a given surface patch and its projected image coordinates ( x ( t ) , y ( t ) ) during the zooming operation with time t:
(2)
dy(t) dt
-
dr(t) -dt
yz -1
= f ( t ) - i ~df(_~,t, y(t) dt
(3)
or denoted shortly withf' meaning df(t)/dt: u = if'If)x,
v = 0c'/f)y.
(4)
It is then noted from (1), that the proportionality between surface radiance L and image irradiance E is not a constant during zooming: aperture diameter d and optic angle are constant but the effective focal lengthfchanges with time; we have: E ( t ) = ¼rrd 2 cos#fl f ( t ) - 2
L.
(5)
Volume 10, Number 3
PATTERN RECOGNITION LETTERS
September 1989
of (5), and afterwards using knowledge of the scene and (5) again for substitution, we then obtain for a given image patch in the image sequence:
surface
dE{x(t), y(t), t} /dt optic axis
dE dx dE dy dE -~?x dt + ~ y &- + d/-t (using chain rule)
lens
~x
Af
f
image plane
= (¼1zd2 cos4fl)
(f-2d--L-Ldt-- 2f 3 dfdtL'~j
zf
(using (5)) Z
spatial Y axis out of paper imagey axis into paper
= (¼ridz cos4fl)
Figure 2. Zooming by moving the optic plane.
- 2f--3 ~ L
= - 2f 1 df E dt
Equation (5) is the image irradiance equation for the assumed zooming operation. A differential form of (5) is now derived; consider Figure 3. An image sequence with coordinate axes x, y and t is shown. The image patch of a given surface patch will follow a linear optic curve {x(t),y(t),t} in the image sequence, and its projection {x(t),y(t)} onto the image plane will be oriented towards or away from the optic centre. Three scalar functions, image irradiance E, effective focal lengthfand optic angle t, have been defined. For a given image patch the function fl will be constant along the optic curve {x(t),y(t),t} for the image patch. Taking the total time differential along this optic curve on either side
(L is static)
(using (5)).
(6)
From (6) the motion constraint equation may be extracted for a fixed position lens during a zoom operation onto a static scene. It is noted briefly without arguments:
Exu + Erv + E, + 20r'/f)E = O.
(7)
Equations (6) and (7) are valid regardless of the surface illumination and the surface reflectance properties because the scene is static and only the image plane is moving. It is clear that the optic flow produced in the assumed zoom operation is not a solution to the classic motion constraint equation, derived originally in (Horn & Schunck, 1981). R
E
{x(t),Y~,y(t
),t }
f
v
X
Figure 3. Image sequences and image data. 191
Volume 10, Number 3
PATTERN RECOGNITION LETTERS
If the expressions for optic flow (4) are substituted into the motion constraint equation (7), a local differential relation between image position (x,y), image irradiance E and its gradient (Ex, Ey, Et), effective focal l e n g t h f a n d zoom r a t e f ' is obtained:
(8)
(Exx + Ery + 2E)f' + EOc = O.
Three different possibilities for camera calibration, using (8) and a sufficiently varying image irradiance are suggested: firstly, the relative zoom rate (f'/f) may be determined; secondly, if the initial effective focal length is known, as might be the case due to a general initial camera calibration, the zoom rate f ' may be determined; thirdly, if the zoom r a t e f ' is known, as might be the case with a given motor zoom, the instantaneously effective focal l e n g t h f m a y be determined. These possibilities will not be pursued further here, but left for further application oriented research. Finally, a local differential relation between image position and image irradiance is derived, which is valid with a constant zoom r a t e f ' , meani n g f ' = dZf(t)/dt z = 0. Taking the total time derivative on either side of (8) along the optic curve {x(t), y(t), t}, and using the chain rule several times, we obtain:
{(ExxU + Exrv + ExOx + E~,u
September 1989
application for securing a constant zoomrate f ' . These and other possible applications of(10) are left for further research.
4. The image irradiance equation during zoom with moving optic centre Now consider the situation where the zooming operation is performed by moving the optic centre, instead of moving the image plane. The same type of expressions and local differential relations are derived as in the former section; it can be seen that they become more complicated, and that the distance to the surface becomes part of the relations. Firstly, an expression for the optic flow (u,v) produced by the zooming operation is derived; consider Figure 4. It is noted that, unlike in Figure 2, the Z coordinate of a surface point is not constant with time: Z changes according to the zoomrate, i.e.,
dZ(t)/dt = dflt)/dt or denoted briefly Z' = f ' .
(11)
As the instantaneous transformation between spatial coordinates of a given surface patch and its projected image coordinates is still given by formu-
+ (Eyxu + Eryv + Eyt)Y + Erv
+ 2(Exu + Erv + Et)}f' (9)
+ {(EtxU + E,yv + Ett)}f+ Ed"' = O.
Using (4) to substitute for optic flow (u, v) in (9) and afterwards using (8) and common algebra to e l i m i n a t e f a n d f ' from (9), the following is obtained after some manipulation:
optic axis
{Exx X2 + 2Exrxy + Eryy 2 -- 6E} E 2
X
+ {Exx + Ery + 2E}{{Exx + Ery + 2E} E,t - 2{Extx + EstY} Et} = 0.
(10)
This equation is valid for any surface illumination and any surface reflectance properties. It may be part of a further analysis of the zooming operation; it may be included as a servo mechanism in an
192
X
image plane
lens 1
Z
f
spatialY axis out of paper imagey axis intopaper
Figure 4. Zooming by moving the optic centre.
Volume 10, Number 3
PATTERN RECOGNITION LETTERS
la (2), an expression for optic flow during the zooming operation is obtained by taking time derivatives on either side of (2) and afterwards using (2) and (11) for substitution, denoted briefly without arguments: u = - f'XZ
= xf'(f
' - fX( - Z
1-
Z
~),
v = --f' YZ- 1 _fy( = yf'O c ' - Z
2)Z'
_ Z -2)Zt
(12)
').
The optic angle fl is at any time and a chosen focal l e n g t h f a function of image position (x,y); fl' is for chosen focal l e n g t h f a n d z o o m r a t e f ' furthermore a function of optic flow (u, v). We may consider fl as a function of (x, y, u, v , f , f ' ) and obtain for fl and fl': tan fi = ( x 2 + y2)1/2f 1, f(x z + y2) - 1/2(X u _]_ yl,') - - f ' ( x 2 + y2)1/2 (13) fl'= (X 2 + y2 +f2)
September 1989
Exu + Erv + Et + 2E(f'/f+
2 tan fl if) = 0.
Equations (15) and (16) are valid regardless of the surface illumination but only for a Lambertian surface; they are not necessarily valid for specular surfaces due to the motion of the optic centre. As in the former section, it may be seen that the optic flow produced in the zoom operation is not in general a solution to the classic motion constraint equation derived originally in (Horn & Schunck, 1981). If the expressions for optic flow (12) and for optic angle (l 3) are now substituted into the motion constraint equation (16), then a local differential relation between image position, image irradiance, effective focal length, zoom rate and surface depth may be obtained after some manipulation: ff'[(E~,x + E r y ) ( x 2 + y2 + f z ) __ 4E(x 2 + y2)] __ Z ( x 2 + y2 + f 2 ) × [f'(E,,x + Eyy + 2E) + f E , ] = O.
It is noted again from (1) that the proportionality between surface radiance L and image irradiance E is not a constant during zooming: aperture diameter d is constant but both effective focal l e n g t h f a n d optic angle fl change with time, i.e., E(t) = ¼ ~ d 2 cos4fl(t) f ( t ) - z L.
(14)
Equation (14) is the image irradiance equation for the assumed zooming operation. A differential form of (14) may be derived, using the technique of the former section (consider Figure 3 and equations (5) and (6) again), but noting that the optic angle fl now changes with time. Thus the following is obtained, denoting briefly:
dt
ff'{a
lr12 - ( v e .
{rE, + f ' ( V E . - Exu + E~.v + E,
(17)
The camera or image sequence dependant vectors are now introduced (remembering that image coordinate axes x and y are reversed with respect to spatial axes X and Y): a = ( - x, - y,f): position vector in camera from optic center to image point; r = (x, y): position vector in image plane from image origin to image point; V E = (Ex, Er): greylevel gradient in the image sequence. Equation (17) may then be rewritten into a closed form solution for surface depth - Z:
- z =
dE
(16)
r)lal z}
(18)
r + 2E)}lal 2
(using chain rule)
3f, COS4fl (using (13) and - - f 24 COS3fl sin fl fl') using L static) = -- E ( 2 f ' / f + 4 tan fi fl') (using (13)). (15) = ¼rcd2L( -- 2 f
From (15) the motion constraint equation may be extracted for a fixed position image plane during a zoom operation onto a static scene, denoted briefly without arguments:
Equation (17) or (18) may be compared with the equations derived in (Ma & Olsen, 1988) for determination of surface depth by zooming using optical flow; the equation in (Ma & Olsen, 1988) does not involve image irradiance but involves optic flow or image diplacement. Equation (17) or (18) may also be compared with the equation derived in (Arnspang, 1988) for determination of surface depth by 193
Volume 10, Number 3
PATTERN RECOGNITION LETTERS
motion along the optic axis using image irradiance; the equation in (Arnspang, 1988) is similar to (17) and (18), but much simpler. The formulae derived above and in (Ma & Olsen, 1988) and in (Arnspang, 1988) seem to represent three alternative but mutual similar techniques for determination of depth: (Arnspang, 1988) using motion and image irradiance; (Ma & Olsen, 1988) using zoom and feature matching; this paper using zoom and image irradiance.
5. Approximate and compact motion constraint equations It is noted, that the motion constraint equation (16) for the case of moving optic centre may often be approximated with the more compact motion constraint equation (7) for the case of fixed optic centre. From (12) and (13) using the vector notation of (18) we obtain: fl'tan fl = -f'l,-12(I,.I 2 -~-f2)--1Z 1, 2(f'/f+ 2 tan fl fl') = 2/,[f-1 _ 21r12(irl 2 + f z ) 1Z-~].
(19) (20)
For the cases where surface depth - Z is much greater than double focal length 2f, the second term on the right-hand side of (20) may be approximated to zero. The motion constraint equation (16) for the case of moving optic centre may then be written as:
Exu + Eyv + Et + 20c'/f)E ~ 0
194
&-0 In E
~3In E
2d In f + ---0, dt 2d In jr
~t--
+
dt
The equations (22) may be named the logarithmic motion constraint equations for a zooming camera.
6. Conclusion Image irradiance equations for a perspective camera zooming onto a static and smooth surface with arbitrary illumination have been derived. In their differential form these equations show that optic flow, produced during the zooming operation is not in general a solution to the classic motion constraint equation, derived by Horn and Schunck. This motion constraint equation has been extended to describe optic flow during the zooming operation. For the case of a fixed optic centre a relation between image irradiance and the relative zoom rate has been derived. For the case of a fixed optic centre and constant zoom rate, a relation among derivatives of image irradiance has been shown to be valid, regardless of surface illumination and surface reflectance properties. For the case of a moving optic centre, a relation between image irradiance and surface depth has been derived; it is valid for static Lambertian surfaces with arbitrary illumination. This relation offers an alternative to a recently suggested method for determination of surface depth using zoom and image displacement.
(21)
which may be seen to be an approximate form of the motion constraint equation (7) for the case of a fixed optic centre. Using logarithmic differentiation, the equations (7) and (21) may be rewritten into the more compact versions: O In E ~ In E - - u + - - v + t3x t3y 0 In E t3 In E - - u + - - v + c3x Oy
September 1989
(22a) 0,
(22b)
References Arnspang, J. (1988). Direct scene determination. DIKU Technical Report 88.3. Horn, B.K.P. and B.G. Schunck (1981). Determining optic flow. Artificial Intelligence 17, 185503. Horn, B.K.P. (1986). Robot Hsion. McGraw-Hill, New York. Klein, M.V. (1970). Optics. Wiley, New York. Ma, Jun and S. Olsen (1988). Depth from zooming. DIKU Technical Report 88.14.
September 1989
Image irradiance equations for a zooming camera Jens A R N S P A N G and Jun MA DIKU, Computer Science Department, University o[Copenhagen, Universitetsparken I, DK 2100 0, Copenhagen, Denmark
Received 19 August 1988 Revised 1 March 1989 Abstract: Image irradianceequations for a camera, zoomingonto a static surface,are discussed. Surfacerange may be determined
directly from image irradiance. The classic motion constraint equation is not valid during zooming; its exact and approximate extensions are discussed. Key words: Vision theory, image irradiance,motion constraint equations, surface depth.
1. Introduction It has been suggested that determination of surface depth in a static scene may be achieved by changing the focal length (Ma & Olsen, 1988); surface depth may be determined if the original focal length and the zoom rate are known, and the resulting image displacement has been determined. In (Ma & Olsen, 1988) two discrete focal lengths with significant separation were used, and image displacement was determined by a matching technique. In this paper local relations are derived for an image sequence, produced from a static surface by a continuous change of focal length, i.e. a zooming operation. It is noted, first that optic flow produced by a continuous zooming operation is not automatically a solution to the classic motion constraint equation of Horn and Schunck (Horn & Schunck, 1981), not even under the same surface illumination conditions as in (Horn & Schunck, 1981). An explanation for this is given and extended motion constraint equations are derived. Secondly, if a zooming operation with constant zooming rate is performed without moving the optic centre, a certain local differential relation among image in-
tensities is always valid, regardless the surface illumination. Thirdly, if the zooming operation involves moving the optic centre, surface depth becomes part of the image irradiance equation, and surface reflectance properties become essential. In the case of a Lambertian surface, a closed form solution for surface depth from local intensities and camera parameters is derived; this relation does not involve optic flow nor any other image displacements as (Ma & Olsen, 1988) does. Throughout this paper the optic system is modelled by one thin lens, capable of changing its focal length continuously. Modern varifocal glasses, and many modern optics for photographic use, may be modelled thus, although in the latter case a model using two principal planes, as explained in (Klein, 1970), might be a better approximation; such refined models are left for further research.
2. The classic image irradiance equation The relation between surface radiance and image irradiance for a thin fixed focus lens has been derived in (Horn, 1986, Ch. 10). The result is resumed here; consider Figure 1.
0167-8655/89/$3.50 O 1989.ElsevierSciencePublishers B.V. (North-Holland)
189
Volume10, Number3
PATTERNRECOGNITIONLETTERS imageplane
E
lens
-
s~
-
September1989 xrface
~
B: opticangle d: aperturediameter f: effectivefocallength L: surfaceradiance E: imageirradiance Figure 1. Imageformationby a thin lens. We have the relation: E = ¼rcd2f - 2 cos4fl L
(1)
where d and f are constants for a fixed focus lens and fl is known at any image position. This classic image irradiance equation (1) is the analytic basis for much work performed in photometry, shape from shading and optic flow, as surveyed in (Horn, 1986).
3. The image irradiance equation during zoom with fixed position optic centre
x(t) = - f(t)
xz-
1,
y(t) = - f(t)
YZ
1.
By taking the time derivative on either side of (2) and afterwards using (2) for substitution, we obtain expressions for the optic flow (u(t), f i t ) ) which is produced: dx(t) u(t)=
190
dr(t) -
dt
- -
XZ-1
dt
= f ( t ) - I d f ( t ) x(t), dt v(t) =
Consider the case where zooming is performed by continuously changing the effective focal lengthf(t) with time t in such a way that only the image plane is moved relatively to a spatial reference system, but not the optic center of the lens and not the surface either. A variant of the classic image irradiance equation (1) shall be derived, and also a variant of the classic motion constraint equation (Horn & Schunck, 1981) shall be derived. Firstly, note an expression for the optic flow produced by the zooming operation; consider Figure 2. From the geometry of the camera, the transformation is obtained between spatial coordinates (X, Y,Z) of a given surface patch and its projected image coordinates ( x ( t ) , y ( t ) ) during the zooming operation with time t:
(2)
dy(t) dt
-
dr(t) -dt
yz -1
= f ( t ) - i ~df(_~,t, y(t) dt
(3)
or denoted shortly withf' meaning df(t)/dt: u = if'If)x,
v = 0c'/f)y.
(4)
It is then noted from (1), that the proportionality between surface radiance L and image irradiance E is not a constant during zooming: aperture diameter d and optic angle are constant but the effective focal lengthfchanges with time; we have: E ( t ) = ¼rrd 2 cos#fl f ( t ) - 2
L.
(5)
Volume 10, Number 3
PATTERN RECOGNITION LETTERS
September 1989
of (5), and afterwards using knowledge of the scene and (5) again for substitution, we then obtain for a given image patch in the image sequence:
surface
dE{x(t), y(t), t} /dt optic axis
dE dx dE dy dE -~?x dt + ~ y &- + d/-t (using chain rule)
lens
~x
Af
f
image plane
= (¼1zd2 cos4fl)
(f-2d--L-Ldt-- 2f 3 dfdtL'~j
zf
(using (5)) Z
spatial Y axis out of paper imagey axis into paper
= (¼ridz cos4fl)
Figure 2. Zooming by moving the optic plane.
- 2f--3 ~ L
= - 2f 1 df E dt
Equation (5) is the image irradiance equation for the assumed zooming operation. A differential form of (5) is now derived; consider Figure 3. An image sequence with coordinate axes x, y and t is shown. The image patch of a given surface patch will follow a linear optic curve {x(t),y(t),t} in the image sequence, and its projection {x(t),y(t)} onto the image plane will be oriented towards or away from the optic centre. Three scalar functions, image irradiance E, effective focal lengthfand optic angle t, have been defined. For a given image patch the function fl will be constant along the optic curve {x(t),y(t),t} for the image patch. Taking the total time differential along this optic curve on either side
(L is static)
(using (5)).
(6)
From (6) the motion constraint equation may be extracted for a fixed position lens during a zoom operation onto a static scene. It is noted briefly without arguments:
Exu + Erv + E, + 20r'/f)E = O.
(7)
Equations (6) and (7) are valid regardless of the surface illumination and the surface reflectance properties because the scene is static and only the image plane is moving. It is clear that the optic flow produced in the assumed zoom operation is not a solution to the classic motion constraint equation, derived originally in (Horn & Schunck, 1981). R
E
{x(t),Y~,y(t
),t }
f
v
X
Figure 3. Image sequences and image data. 191
Volume 10, Number 3
PATTERN RECOGNITION LETTERS
If the expressions for optic flow (4) are substituted into the motion constraint equation (7), a local differential relation between image position (x,y), image irradiance E and its gradient (Ex, Ey, Et), effective focal l e n g t h f a n d zoom r a t e f ' is obtained:
(8)
(Exx + Ery + 2E)f' + EOc = O.
Three different possibilities for camera calibration, using (8) and a sufficiently varying image irradiance are suggested: firstly, the relative zoom rate (f'/f) may be determined; secondly, if the initial effective focal length is known, as might be the case due to a general initial camera calibration, the zoom rate f ' may be determined; thirdly, if the zoom r a t e f ' is known, as might be the case with a given motor zoom, the instantaneously effective focal l e n g t h f m a y be determined. These possibilities will not be pursued further here, but left for further application oriented research. Finally, a local differential relation between image position and image irradiance is derived, which is valid with a constant zoom r a t e f ' , meani n g f ' = dZf(t)/dt z = 0. Taking the total time derivative on either side of (8) along the optic curve {x(t), y(t), t}, and using the chain rule several times, we obtain:
{(ExxU + Exrv + ExOx + E~,u
September 1989
application for securing a constant zoomrate f ' . These and other possible applications of(10) are left for further research.
4. The image irradiance equation during zoom with moving optic centre Now consider the situation where the zooming operation is performed by moving the optic centre, instead of moving the image plane. The same type of expressions and local differential relations are derived as in the former section; it can be seen that they become more complicated, and that the distance to the surface becomes part of the relations. Firstly, an expression for the optic flow (u,v) produced by the zooming operation is derived; consider Figure 4. It is noted that, unlike in Figure 2, the Z coordinate of a surface point is not constant with time: Z changes according to the zoomrate, i.e.,
dZ(t)/dt = dflt)/dt or denoted briefly Z' = f ' .
(11)
As the instantaneous transformation between spatial coordinates of a given surface patch and its projected image coordinates is still given by formu-
+ (Eyxu + Eryv + Eyt)Y + Erv
+ 2(Exu + Erv + Et)}f' (9)
+ {(EtxU + E,yv + Ett)}f+ Ed"' = O.
Using (4) to substitute for optic flow (u, v) in (9) and afterwards using (8) and common algebra to e l i m i n a t e f a n d f ' from (9), the following is obtained after some manipulation:
optic axis
{Exx X2 + 2Exrxy + Eryy 2 -- 6E} E 2
X
+ {Exx + Ery + 2E}{{Exx + Ery + 2E} E,t - 2{Extx + EstY} Et} = 0.
(10)
This equation is valid for any surface illumination and any surface reflectance properties. It may be part of a further analysis of the zooming operation; it may be included as a servo mechanism in an
192
X
image plane
lens 1
Z
f
spatialY axis out of paper imagey axis intopaper
Figure 4. Zooming by moving the optic centre.
Volume 10, Number 3
PATTERN RECOGNITION LETTERS
la (2), an expression for optic flow during the zooming operation is obtained by taking time derivatives on either side of (2) and afterwards using (2) and (11) for substitution, denoted briefly without arguments: u = - f'XZ
= xf'(f
' - fX( - Z
1-
Z
~),
v = --f' YZ- 1 _fy( = yf'O c ' - Z
2)Z'
_ Z -2)Zt
(12)
').
The optic angle fl is at any time and a chosen focal l e n g t h f a function of image position (x,y); fl' is for chosen focal l e n g t h f a n d z o o m r a t e f ' furthermore a function of optic flow (u, v). We may consider fl as a function of (x, y, u, v , f , f ' ) and obtain for fl and fl': tan fi = ( x 2 + y2)1/2f 1, f(x z + y2) - 1/2(X u _]_ yl,') - - f ' ( x 2 + y2)1/2 (13) fl'= (X 2 + y2 +f2)
September 1989
Exu + Erv + Et + 2E(f'/f+
2 tan fl if) = 0.
Equations (15) and (16) are valid regardless of the surface illumination but only for a Lambertian surface; they are not necessarily valid for specular surfaces due to the motion of the optic centre. As in the former section, it may be seen that the optic flow produced in the zoom operation is not in general a solution to the classic motion constraint equation derived originally in (Horn & Schunck, 1981). If the expressions for optic flow (12) and for optic angle (l 3) are now substituted into the motion constraint equation (16), then a local differential relation between image position, image irradiance, effective focal length, zoom rate and surface depth may be obtained after some manipulation: ff'[(E~,x + E r y ) ( x 2 + y2 + f z ) __ 4E(x 2 + y2)] __ Z ( x 2 + y2 + f 2 ) × [f'(E,,x + Eyy + 2E) + f E , ] = O.
It is noted again from (1) that the proportionality between surface radiance L and image irradiance E is not a constant during zooming: aperture diameter d is constant but both effective focal l e n g t h f a n d optic angle fl change with time, i.e., E(t) = ¼ ~ d 2 cos4fl(t) f ( t ) - z L.
(14)
Equation (14) is the image irradiance equation for the assumed zooming operation. A differential form of (14) may be derived, using the technique of the former section (consider Figure 3 and equations (5) and (6) again), but noting that the optic angle fl now changes with time. Thus the following is obtained, denoting briefly:
dt
ff'{a
lr12 - ( v e .
{rE, + f ' ( V E . - Exu + E~.v + E,
(17)
The camera or image sequence dependant vectors are now introduced (remembering that image coordinate axes x and y are reversed with respect to spatial axes X and Y): a = ( - x, - y,f): position vector in camera from optic center to image point; r = (x, y): position vector in image plane from image origin to image point; V E = (Ex, Er): greylevel gradient in the image sequence. Equation (17) may then be rewritten into a closed form solution for surface depth - Z:
- z =
dE
(16)
r)lal z}
(18)
r + 2E)}lal 2
(using chain rule)
3f, COS4fl (using (13) and - - f 24 COS3fl sin fl fl') using L static) = -- E ( 2 f ' / f + 4 tan fi fl') (using (13)). (15) = ¼rcd2L( -- 2 f
From (15) the motion constraint equation may be extracted for a fixed position image plane during a zoom operation onto a static scene, denoted briefly without arguments:
Equation (17) or (18) may be compared with the equations derived in (Ma & Olsen, 1988) for determination of surface depth by zooming using optical flow; the equation in (Ma & Olsen, 1988) does not involve image irradiance but involves optic flow or image diplacement. Equation (17) or (18) may also be compared with the equation derived in (Arnspang, 1988) for determination of surface depth by 193
Volume 10, Number 3
PATTERN RECOGNITION LETTERS
motion along the optic axis using image irradiance; the equation in (Arnspang, 1988) is similar to (17) and (18), but much simpler. The formulae derived above and in (Ma & Olsen, 1988) and in (Arnspang, 1988) seem to represent three alternative but mutual similar techniques for determination of depth: (Arnspang, 1988) using motion and image irradiance; (Ma & Olsen, 1988) using zoom and feature matching; this paper using zoom and image irradiance.
5. Approximate and compact motion constraint equations It is noted, that the motion constraint equation (16) for the case of moving optic centre may often be approximated with the more compact motion constraint equation (7) for the case of fixed optic centre. From (12) and (13) using the vector notation of (18) we obtain: fl'tan fl = -f'l,-12(I,.I 2 -~-f2)--1Z 1, 2(f'/f+ 2 tan fl fl') = 2/,[f-1 _ 21r12(irl 2 + f z ) 1Z-~].
(19) (20)
For the cases where surface depth - Z is much greater than double focal length 2f, the second term on the right-hand side of (20) may be approximated to zero. The motion constraint equation (16) for the case of moving optic centre may then be written as:
Exu + Eyv + Et + 20c'/f)E ~ 0
194
&-0 In E
~3In E
2d In f + ---0, dt 2d In jr
~t--
+
dt
The equations (22) may be named the logarithmic motion constraint equations for a zooming camera.
6. Conclusion Image irradiance equations for a perspective camera zooming onto a static and smooth surface with arbitrary illumination have been derived. In their differential form these equations show that optic flow, produced during the zooming operation is not in general a solution to the classic motion constraint equation, derived by Horn and Schunck. This motion constraint equation has been extended to describe optic flow during the zooming operation. For the case of a fixed optic centre a relation between image irradiance and the relative zoom rate has been derived. For the case of a fixed optic centre and constant zoom rate, a relation among derivatives of image irradiance has been shown to be valid, regardless of surface illumination and surface reflectance properties. For the case of a moving optic centre, a relation between image irradiance and surface depth has been derived; it is valid for static Lambertian surfaces with arbitrary illumination. This relation offers an alternative to a recently suggested method for determination of surface depth using zoom and image displacement.
(21)
which may be seen to be an approximate form of the motion constraint equation (7) for the case of a fixed optic centre. Using logarithmic differentiation, the equations (7) and (21) may be rewritten into the more compact versions: O In E ~ In E - - u + - - v + t3x t3y 0 In E t3 In E - - u + - - v + c3x Oy
September 1989
(22a) 0,
(22b)
References Arnspang, J. (1988). Direct scene determination. DIKU Technical Report 88.3. Horn, B.K.P. and B.G. Schunck (1981). Determining optic flow. Artificial Intelligence 17, 185503. Horn, B.K.P. (1986). Robot Hsion. McGraw-Hill, New York. Klein, M.V. (1970). Optics. Wiley, New York. Ma, Jun and S. Olsen (1988). Depth from zooming. DIKU Technical Report 88.14.