A new approach to photometric stereo1

Pattern Recognition Letters 20 (1999) 535±540

www.elsevier.nl/locate/patrec

A new approach to photometric stereo Jose R.A. Torre~ ao

1

2

Instituto de Computacßa~o, CAA, Universidade Federal Fluminense, 24210-240 Niter oi, Rio de Janeiro, Brazil Received 11 August 1998; received in revised form 2 March 1999

Abstract We introduce a new approach to shape estimation from photometric stereo images. The input images are matched through an optical ¯ow algorithm, with the matching direction iteratively re®ned. The resulting disparity ®eld is then used in a structure-from-motion reconstruction which does not require re¯ectance map information. Ó 1999 Elsevier Science B.V. All rights reserved. Keywords: Photometric stereo; Optical ¯ow; Shape estimation

oI2 oI2 ‡v : ox oy

1. A disparity-based photometric stereo

DI…s†  I1 …s† ÿ I2 …s†  u

Photometric stereo (PS) images are obtained from a single camera under di€erent illuminations, and have traditionally been employed for the estimation of surface gradient through a re¯ectancemap based process (Woodham, 1980; Lee and Kuo, 1992). Recently, it has been suggested that such images can also be matched to yield depth estimates of the imaged surfaces (Fernandes and TorreaÄo, 1998). Extending these last results, we propose a new approach to PS reconstruction, which is independent of re¯ectance-map information. The approach in (Fernandes and TorreaÄo, 1998) is based on matching a pair of PS images, I1 …s† and I2 …s†, of a uniform albedo surface, to obtain a disparity ®eld, …u…s†; v…s††, where s ˆ …x; y†, such that I1 …x; y†  I2 …x ‡ u; y ‡ v†, or, using a linear Taylor-series expansion,

Now, assuming that a linear expansion of the re¯ectance-map function is also applicable (Pentland, 1990), we can write DI ˆ k0 ‡ k1 p ‡ k2 q, where …p…s†; q…s†† denotes the surface gradient, and the ki 's are constants given by ki ˆ ki1 ÿ ki2 , with k0j ˆ k 0j ÿ k1j p0 ÿ k2j q0 …j ˆ 1; 2†, and with k 0j ; k1j and k2j denoting the values of the jth re¯ectance map and its derivatives at the ®xed orientation …p0 ; q0 †. From the above, we get the di€erential equation

1

Electronic Annexes available. See www.elsevier.nl/locate/ patrec 2 E-mail: [email protected]€.br

k1 p ‡ k2 q  u

oI2 oI2 ‡v ÿ k0 ; ox oy

…1†

…2†

whose approximate solution can be found, as in (Fernandes and TorreaÄo, 1998), if the matching is performed along the direction given by v k2 ˆ  ÿc; u k1

…3†

and if the disparity ®eld varies slowly with position. It is then given by

0167-8655/99/$ ± see front matter Ó 1999 Elsevier Science B.V. All rights reserved. PII: S 0 1 6 7 - 8 6 5 5 ( 9 9 ) 0 0 0 2 6 - 4

536

z…x; y† ˆ

J.R.A. Torre~ ao / Pattern Recognition Letters 20 (1999) 535±540

…k1 u ‡ k2 v†I2 ÿ k0 …k1 x ‡ k2 y† : k12 ‡ k22

…4†

2. Structure-from-motion formulation We will now show that, when k0 ˆ 0, which corresponds to taking the linear expansion of the re¯ectance map around an orientation …p0 ; q0 † satisfying k 01 ÿ k 02 ÿ k1 p0 ÿ k2 q0 ˆ 0, Eq. (4) can be obtained through a standard least-squares approach, based on modelling the disparity ®eld according to the equations uI2  k1 z;

vI2  k2 z:

The error integral would then be ZZ 2 2 Fˆ …uI2 ÿ k1 z† ‡ …vI2 ÿ k2 z† dx dy;

…5†

…6†

and minimization of the integrand with respect to z would yield Eq. (4), which, using Eq. (3), reduces in this case to uI2 : z…x; y† ˆ k1

…7†

Now, the relations in Eq. (5) are consistent with a rotation model for the displacement of the irradiance pattern over the scene, due to the change of illumination. Calling R ˆ …x; y; z† the vector position of a point on the imaged surface (given with respect to a coordinate system with the ÿz direction pointing along the optical axis), and assuming orthographic imaging projection, such displacement would be expressed by DR  …u; v; Dz† ˆ H  R;

…8†

where H is the rotation vector and Dz is the unobservable displacement along the optical axis direction. For a rotation H ˆ …ÿk2 ; k1 ; 0†=I2 , Eq. (5) is recovered, also yielding Dz ˆ ÿ…k1 x ‡ k2 y†=I2 : Assuming that the PS image pair has been matched along the direction given by Eq. (3), through the algorithm to be described in the following section, Eq. (7) immediately yields the surface estimate up to an overall multiplicative constant, z0 ˆ k1 z ˆ uI2 , there remaining only the parameter k1 to be found, for the depth map to be

completely recovered. Since the functional in Eq. (6) cannot be used for this purpose, we introduce a new constraining relation, by requiring that the displacement vector be at each point approximately perpendicular to the normal,   psurface i.e., DR
n^  DR
…ÿp; ÿq; 1†= p2 ‡ q2 ‡ 1  0. With the use of Eqs. (8) and (3), this relation becomes up ‡ vq ‡ k1 …x ÿ cy†=I2  0, which can be expressed as a global constraint, through a second error integral, ZZ 2 ‰uI2 …p ÿ cq† ‡ k1 …x ÿ cy†Š dx dy: …9† F0 ˆ Rewriting F0 in terms of the gradient components, p0  oz0 =ox ˆ k1 p and q0  oz0 =oy ˆ k1 q, and minimizing with respect to k1 , we ®nally ®nd RR ‰uI2 …p0 ÿ cq0 †Š2 dx dy 4 k1 ˆ RR : …10† 2 …x ÿ cy† dx dy

3. Matching algorithm Essential to the above approach to photometric stereo is to estimate the disparity ®eld by matching the input images along a ®xed direction, determined by the ratio of the linear expansion components, k1 and k2 , of the re¯ectance map for DI. We will now show that such a matching can be performed, in the absence of re¯ectance map information, through a simple modi®cation of the optical ¯ow algorithm by Horn and Schunck (1981). The approach in (Horn and Schunck, 1981) is based on the minimization of a functional consisting of a smoothness term and the optical ¯ow constraint error, ZZ Eˆ …u2x ‡ u2y † ‡ …v2x ‡ v2y † 2

‡ k…Ix u ‡ Iy v ÿ DI† dx dy;

…11†

where the subscripts denote di€erentiation, k is an empiric parameter, and I here stands for the second image, I2 . Such approach thus essentially seeks a smooth approximate solution to Eq. (2). The disparity-based photometric stereo, on the other hand, also tries to solve the same equation,

J.R.A. Torre~ ao / Pattern Recognition Letters 20 (1999) 535±540

now for the surface function z, assuming that the disparity ®eld varies slowly, and that the linear re¯ectance map approximation holds. When k0 ˆ 0, and when the matching is performed along the di-

537

rection (3), such a solution is found as function (7). Now, if in the error integral E we substitute ÿcu for v, with c unknown, and k1 p ‡ k2 q for DI, with p and q given by the derivatives of Eq. (7), not surpris-

Fig. 1. Top: input image pair. Bottom: estimated depth map

538

J.R.A. Torre~ ao / Pattern Recognition Letters 20 (1999) 535±540

ingly we ®nd that the matching direction which minimizes such integral is given by c ˆ ÿk2 =k1 , as long as the disparity ®eld is smooth enough, so that its derivatives can be neglected. Thus, the solution

of the photometric stereo problem, given by Eq. (7), is consistent with matching the input images along the direction c which minimizes the functional in Eq. (11).

Fig. 2. Top: input image pair. Bottom: estimated depth map.

J.R.A. Torre~ ao / Pattern Recognition Letters 20 (1999) 535±540

From these considerations, we are led to propose, as matching algorithm for the photometric stereo input, a modi®ed version of Horn and Schunck's approach, based on the functional E, but with v re-

539

placed by ÿcu. Minimization with respect to u at each image point, and with respect to c, yield the following iterative schemes, which are run starting from the initial estimates u…s† ˆ 0; 8s and c ˆ 0:

Fig. 3. Top: input image pair. Bottom: estimated depth map.

540

u…n‡1† ˆ

J.R.A. Torre~ ao / Pattern Recognition Letters 20 (1999) 535±540

u…n† …1 ‡ c2n † ÿ kDI…Ix ÿ cn Iy † 1 ‡ c2n ‡ k…Ix ÿ cn Iy †

2

;

…12†

where the overline indicates local average value, and RR k …Ix u…n† ‡ DI†…Iy u…n† † dx dy : …13† cn ˆ RR k…Iy u…n† †2 ‡ …u2x ‡ u2y †…n† dx dy We conclude by remarking that our initial assumption that k0 ˆ 0 can be made independently of the estimation of c, since we can always assume that the linear expansion of the re¯ectance map is taken about an orientation for which q0 ˆ 0. The condition k0 ˆ 0 would thus lead to the equation k1 p0 ˆ k 02 ÿ k 01 , which is independent of k2 , and thus of c. It is easy to show that, in the particular case of lambertian re¯ectance, this would correspond to a cubic equation, for which a real root can always be found.

linear re¯ectance map. It should be remarked that, in the case of the synthetic sphere, the recovered depth range is quite accurate (from 0 to 0:43), and that, in the mannequin experiment, the reconstructed surface is fairly faithful to the original object, as can be seen from the chin angle and from the crease of the ear. The matching algorithm was run, respectively, for 1000 iterations with k ˆ 0:1, 2000 iterations with k ˆ 0:05, and 3000 iterations with k ˆ 0:0075. The c values yielded in the three cases were 0:0; ÿ0:07 and ÿ0:6.

Acknowledgements Partially supported by Faperj E-26/171.081/96 and Finep-Recope 0626/96-SAGE.

References 4. Experiments Figs. 1±3 depict the input images and the respective depth maps, recovered through the new PS approach, for a synthetic lambertian sphere of radius 0:4, an approximately lambertian pottery vase, and a wax mannequin head with a considerable glossy re¯ectance component. In all three cases, the estimated shape is qualitatively correct, though slightly oversmoothed near the edges, due to the unrestricted smoothing imposed by the matching algorithm, and to the assumption of a

Fernandes, J.L., Torre~ ao, J.R.A., 1998. Estimating depth through the fusion of photometric stereo images. In: Proceedings of the ACCV'98, Lect. Notes Comp. Sci. 1351, pp. 64±71. Horn, B.K.P., Schunck, B.G., 1981. Determining optical ¯ow. Artif. Intell. 17, 185±203. Lee, K.M., Kuo, C.-C.J., 1992. Shape reconstruction from photometric stereo. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 479±484. Pentland, A., 1990. Linear shape from shading. Internat. J. Comput. Vision 4, 153±162. Woodham, R.J., 1980. Photometric method for determining surface orientation from multiple images. Opt. Engr. 19 (1), 139±144.