On error analysis for surface normals determined by photometric stereo

Signal Processing 23 (1991) 221-226 Elsevier

221

On error analysis for surface normals determined by photometric stereo X.Y. J i a n g a n d H . B u n k e lnstitut fiir angewandte Mathematik and lnformatik, University of Bern, Liinggassstrasse 51, CH-3012 Bern, Switzerland Received 24 October 1990 Revised 17 January 1991

Abstract. The local surface orientation of a three-dimensional object in a scene can be determined by photometric stereo. Photometric methods are susceptible to a number of errors. Ray, Birk and Kelly have identified and analysed the various error sources. Especially, a sensitivity analysis of local surface normals computed by photometric stereo with respect to major error sources is given. Unfortunately, due to the gradient space representation of reflectance maps, the derivation process is tedious and the formulas are very complicated. This paper gives a new sensitivity analysis technique using reflectance maps represented by unit surface normal vectors and vector ditterentiation. This new technique leads to a much simpler derivation process and substantially compacter sensitivity formulas.

Znsammenfassung. Die lokale Oberflfichenorientierung eines dreidimensionalen Objektes in einer Szene kann mithilfe des photometrischen Stereoverfahrens ermittelt werden. Photometrische Verfahren sind empfindlich gegen eine Reihe yon Fehlerquellen. Ray, Birk and Kelly haben diverse Fehlerquellen identifiziert und analysiert, lnsbesondere wurde eine Analyse der Sensibilit~it der mittels photometrischen Stereoverfahrens berechneten lokalen Oberfl~ichennormalenvektoren gegeniiber den wichtigsten Fehlerquellen gegeben. Wegen Verwendnung des Gradientenraums als Reprfisentation der Reflektanzkarte ist die Analyse jedoch langwierig. Die abgeleiteten Formeln sind sehr kompliziert. In dieser Arbeit stellen wit eine neue Sensibilit~itsanalyse vor, in welcher der Einheitsnormalenvektor zur Repr~isentation der Reflektanzkarte und Vektordifferentialrechnung zur Ableitung der Formeln herangezogen werden. Dies fiihrt zu einer wesentlich einfacheren Ableitung und kompakteren Formeln.

Rrsumr. L'orientation de surface locale d'un objet tri-dimensionnel dans une sc~ne peut &re drterminre par strrro photomrtrique. Les mrthodes photomrtriques sont sujettes h nombre d'erreurs. Ray, Birk et Kelly ont identifirs et analysrs les ditirrentes sources d'erreurs. On y fournit en particulier une analyse de sensibilit6 des normales aux surface locales calculres par strrro photomrtrique par rapport aux principales sources d'erreur. Malheureusement, du fait de la reprrsentation sur l'espace des gradients des diagrammes de rrflectivitr, le processus de drrivation est fastidieux et les formules sont tr~s compliqures. Cet article propose une nouvelle technique d'analyse de la sensibilit6 utilisant les diagrammes de rrflectivit6 reprrsentrs par des vecteurs normaux unitaires et une ditirrentiation vectorielle. Cette technique nouvelle conduit ~ un processus de drrivation beaucoup plus simple et h des formules de sensitivit6 substantiellement plus compactes.

Keywords. Pattern recognition, image processing, shape from shading, photometric stereo, Lambertian reflectance, needle map, sensitivity analysis.

1. Introduction

s t e r e o , first f o r m u l a t e d b y W o o d h a m

[ 1 3 ] , is p e r -

haps the most widely used one in experimental Shape

from

approaches tion from shape

shading

to determining image

from

refers

irradiance.

shading

to

photometric

local surface orienta-

methods

c o m p u t e r v i s i o n s y s t e m s [ 1 - 3 , 6, 7, 10]. Historically,

Horn

introduced

the reflectance

Under

the various

m a p [8]. T h i s m a p r e p r e s e n t s t h e r e l a t i o n s b e t w e e n

the

photometric

local surface orientation and image irradiance in

0165-1684/91/$03.50 © 1991 - Elsevier Science Publishers B.V.

222

X. Y. Jiang, H. Bunke/ Error analysis for photometric stereo

gradient space. Woodham then developed the photometric stereo method based on reflectance maps. In this method three images~ of a scene are taken by a stationary camera under different directions of incident illumination. For each point in the image plane, the intensity in each of the three images constrains the local surface orientation on an iso-intensity contour in the gradient space. The surface orientation can then be computed by intersecting three iso-intensity contours. The photometric stereo method is susceptible to a number of factors, such as errors in the measurement of light source orientations, errors in the measurement of image irradiance, feasibility of the reflectance map, the spatial and spectral properties of the light sources, the imaging geometry etc. In [11] the various error sources have been identified and analysed. Some means to rectify these errors have been suggested. Furthermore, a sensitivity analysis of surface normals computed by the photometric stereo method with respect to major error sources has been derived. The Lambertian surface was assumed there and the corresponding reflectance map was expressed in the gradient space. The surface orientation was obtained by a method of algebraic solution of three nonlinear equations. Finally, an estimate of errors in the surface orientation due to errors in the measurement of image irradiance and the orientation of light sources was derived. It was the gradient representation of reflectance maps that made the derivation process and the resulting formulas very complicated. In this paper we show that another representation of the reflectance map in terms of unit surface normal vectors, in combination with vector differentiation, allows us to derive much compacter formulas. The derivation process also becomes substantially simpler.

2. Surface shape from photometric stereo The kernel of any shape from shading method is the image irradiance equation

I(x,y)=R(p,q), Signal Processing

(1)

where the reflectance map R(p, q) represents the relations between image intensity and surface orientation in gradient space. From the reflectance map, a single intensity restricts the surface gradient (p, q) to a particular iso-intensity contour. The goal of shape from shading is then to determine which (p, q) value on the contour is the correct surface gradient. If a single source of illumination is used, the single intensity value in each point in the image plane maps into many possible surface orientations. To resolve these ambiguities, iterative constraint propagation (relaxation) algorithms have been proposed [5, 12]. An alternative to the single source shape from shading is the photometric stereo. In this method three images of a scene are taken by a stationary camera under different directions of incident illumination. So three reflectance maps can be determined because the reflectance map depends on the direction of the light source. For each point in the image plane, the intensity in each of the three images constrains the local surface orientation on an iso-intensity countour in the gradient space according to the image irradiance equation. The surface orientation, then, can be computed by intersecting three iso-intensity contours. Photometric stereo can be potentially used where the illumination can be controlled, for example in industrial applications. In that case the photometric properties of the surface of the workpieces, i.e. the reflectance map R(p, q), must be known in advance. The reflectance map accounts for the specular and diffuse reflections that occur when a ray of light strikes the surface of a workpiece. These reflectance characteristics depend on a number of factors like the surface material, surface microstructure, geometry and photometry of the incident illumination, etc. In many cases, however, the ideal Lambertian model of reflection is a feasible choice. The Lambertian surfaces scatter the incident light equally in all directions and appear equally bright from all viewing directions. In this case a point source illumination is used. For workpieces with specularly reflecting surfaces,

223

X.Y. Jiang, H. Bunke/ Error analysis for photometric stereo

nevertheless, a non-Lambertian reflectance map and a distributed light source must be used [4]. We consider in this work only the first case. Assuming a Lambertian surface, the image irradiance equation (1) becomes

I(x, y) = p(x, y) cos i,

(2)

where I(x, y) is the image irradiance at the image point (x, y), p(x, y) the surface reflectance factor, and i the incidence angle between the direction of incident illumination and the surface normal vector. Using the gradient representation (Pn, q~), (p~, q~) of the surface orientation and direction of light source, respectively, we obtain

1 +psp.+ qsqn /l+p~+q2s~/l+p~+q2n.

I(x,y)=p(x,y)

1 +plpn+ qlqn I1 = Px/1 +p21+ q~ 41 +p~+ qn2' 1 +p2pn+ q2q. : 2 /1 2 2' +p2+q2 +pn+qn

(4)

1 +p3p.+ q3q. I3 =Px/1

2 2/1 2 2" +p3+q3 +pn+qn

In terms of the zenith angle 0~ (measured between the surface normal and the z-axis) and the azimuth angle ~b; (measured between the projection of the surface normal onto the xy-plane and the x-axis), the light source gradient can be written as p~ = - c o s ~bi tan 0~, qi = - s i n ~bi tan 0. i = 1,2,3.

On=arctanx/p2n+q2n ,

~bn=arctan qn. Pn

(8)

Thus, 0. and gb. are functions of 11,/2,/3, 0~, ~b., 02, ~b2, 03 and ~b3, respectively.

3. Sensitivity analysis

In [11] two formulas, d0. = 0(dll. dI2, dI3, d01, d~bl, d02, d$2, d03, dq)3),

(3)

Given three incident illumination sources positioned in the directions (Pl,qt), (P2, q2) and (P3, q3) and the measured values of image intensities 11, 12 and 13, we get a set of three nonlinear equations

t2=px/1

Furthermore, the surface normal gradient is related to the surface normal angles in the following way

den

p. = f ( I t , 12,/3, 01, ~1, 02, ~2, 03, ~3),

(6)

q. = g(ll, 12, 13, 01, dh, 02, ~b2, 03, Oh3).

(7)

$ ( d l l , dI2, dI3, d0,, dd~, d02, d¢2, d03, d~3 )

(10)

were obtained from (6)-(8) by derivation. Thus, it is possible to estimate the magnitude of errors in 0n and d~n due to small changes in various parameters of the photometric stereo algorithm. Because of the gradient space representation of reflectance maps, however, the expressions Pn and qn in (6) and (7) had many terms. This led to a tedious and error-prone derivation process, and complicated formulas for both d o n and dd~n. In the following we give a new sensitivity analysis technique that makes both the derivation process and the sensitivity formulas much simpler. The geometry of a photometric stereo system is presented in Fig. 1. Instead of gradients we represent surface orientation in terms of unit normal vectors. In what follows we will use n. m to represent the scalar product of two vectors n and m. Given three light sources

(5)

The values of p and q, then, can be determined uniquely by algebraically solving the nonlinear equation system (4) and we get

=

(9)

hi=

Ixq y, ,

In,l=l,

i=1,2,3,

(11)

I_Z,l and the unit surface normal vector n, the image irradiance equations in (4) become

L=p(n,.n),

i = 1,2,3,

(12) Vol. 23, No. 3, June 1991

224

X.Y. Jiang, 14. Bunke/ Error analysis for photometric stereo Thus, the unit surface normal vector can be computed as

n

ni = (xl, Yl, zi)

/

I!--

1

/3[/il n2 n3]

(I1(n2 × n3) +/2(/13 X hi)

+ I3(n, x n2)) 1

y

\

p[nl n2 n3]

\ x Fig. 1. Geometry of a photometric stereo system.

or in matrix form

1'] r'q

12 =

p| nT l .

=

(13)

pNsn,

L.~J

/3

where the superscript T means the transpose of a vector. I f we use I to symbolize the intensity vector, (13) becomes

I=pNsn.

(14)

To solve for the surface reflectance factor p and the unit normal n, (14) can be written as

pn = Ns1L

(15)

The surface reflectance factor p can be found simply by taking the magnitude of the right-hand side of (15). This is because the surface normal n is of unit length. Thus, we get

p=lN~'II.

(16)

Once p is found the unit surface normal n can be computed as

m.

(19)

Naturally, we could differentiate (19) to compute d0, and den in (9) and (10). Because of the fraction in (19), however, the derivation would he tedious. Without loss of generality we assume [nl n2 n3] > 0. Note that both n and m point into the same direction. This means that they have identical zenith angle 0. and azimuth angle ¢ , . Thus, we can use the much simpler expression m to compute d O n and d e , . This observation is in fact the central point of our new sensitivity analysis technique. The surface normal vector m can be written as

m=(Xn,Yn, Z,) =Ii(n2×n3)+I2(n3xnt)+I3(nlxn2),

(20)

x, : f x ( I , , / 2 , 13, 01, ¢1, 02, ¢2, 03, ~3), y. = f r ( I , , 12, I3, 0,, ~b,, 02, ¢2, 03, ~3),

(21)

Zn =L(I1, I2, I3, 01, ¢1, 02, e~2, 03, ¢3). Now if small changes are made in 11, 12, 13, 01, ¢1, 02, ¢2, 03 and ~)3 w e can estimate deviations in m as well. Using differentiation of a vector with respect to a vector, this can be written in a very compact notation: d m = ( d x , , dyn, dz,) 3 0m

1

n

= - N~IL

(17)

p

As Ns is a 3 × 3 matrix, it is easy to show that N [ 1- - -

[nl

1

n2 n3]

(n2 x

n 3 n 3 x n I n I x n2) ,

(18)

where [nl n2 n3] is the triple product nl" (n2 × n3). SignalProcessing

i=l On---~i\OOi dOi ~ - ~ i d~t~i) '

(22)

where Om/Oni defines a matrix that contains as elements the derivatives of the components of the vector m with respect to the components of another

225

X.Y. Jiang, H. Bunke/ Error analysis for photometric stereo

angle 0. and the azimuth angle 4~. in the way

vector hi, i.e.,

2 x/--xx, + yn2 On = a r c t a n - - ,

a X n"

ax~

3X. ayi

am

ay.

ay.

ani --

aXi

ayi

3y. azi '

aZn •ax~

3Z.

az.

"3Xn

az~ i = 1, 2, 3.

(23)

dO n =

1

az~ .

N o w we c o m p u t e all the partial derivatives in (22). Obviously, we have

+

am

am

- - =

/12X

311

n3~

- - :

3/2

1 2

2

Y . Z______2.

- - n I X I! 2 .

2

It is not difficult to prove (see [9]) that

Onj

i#j,

zi -Yi

0 xi

-xi 0

[ I

2

x.+y.+z.

(24)

[o z y]

Zn

~ /X2n + Y n 2

.r---r----; y" dy. z.x/ xn + y~

am

o/3

( • xo --

2 ~ 2 (Xn+Yn)/Zn

dx.

\

=

n 3 X n 1,

(30)

Xn

the errors in the values of On and ~bn can be written as 1+

ayi

4~. = arctan Y",

Zn

2

-z~

x.z.

\ , / x .2+ y .2

dz. ~ /

dx.

dy.- ~

dz.] /

1

= Xn2+y~+z2.

= Mi,

i = 1,2,3.

ynZ.._..__~

XnZ n 2 2'

2

2'

(25) - ~ )

(31)

.din,

It follows that am

--= an~

I 2 M 3 - I3M2.

am

--= an2

1

d~bn = l + ( y . / X n )

I 3 M , - I, M3,

(26) 0m -- = I1M2- I2MI. 3n3

n~ = (sin 0~ cos ~b~,sin 0; sin ~b~,cos 0~), i = 1, 2, 3.

(27)

Thus, we get an--2= (cos 0~ cos ~b~,cos 0~ sin ~bi, - s i n 0~), an i

a4,~

- ( - s i n 0~ sin ¢~, sin 0~ cos ~b~,0).

(28)

(29)

H a v i n g all the partial derivatives in (22), we can find a r e a s o n a b l e estimation o f errors in m. Since the surface n o r m a l vector m is related to the zenith

\

X.

dx.+±dy3 X.

]

1 --

In terms of zenith angle and azimuth angle the unit light source vectors can be written as

2(-

-

2

2 (--Y- d x . + x . d y . )

x.+ y. 2

1

x . + y2.

(-y,,

Xn, 0 ) ' d i n .

(32)

Hereto we have finished the derivation process. Using (31) a n d (32), we can estimate the m a g n i t u d e of errors in 0n and Sn due to small changes in various p a r a m e t e r s of the p h o t o m e t r i c stereo algorithm. C o m p a r e d with [11], our f o r m u l a s are m u c h simpler and c o m p a c t e r . In [11] three tables were established: (1) to present sample errors in 0. and ~bn due to errors in intensity m e a s u r e m e n t s only; (2) to show the errors in 0. and ~b. due to small errors in the m e a s u r e m e n t of the zenith angle or the azimuth angle o f the light sources; (3) to show the errors in 0. and ~b. due to the m a g n i t u d e of the zenith Vol. 23, No. 3, June 1991

226

x . Y. Jiang, H. Bunke/ Error analysis for photometric stereo

angle settings o f the light source in the p r e s e n c e o f errors in i m a g e intensity a n d i n p u t zenith angle. W e r e p e a t e d these c o m p u t a t i o n s to verify o u r formulas. T h a t is, we t o o k the s a m e c o n f i g u r a t i o n p a r a m e t e r s o f the p h o t o m e t r i c stereo system a n d the s a m e initial m e a s u r e m e n t errors as in [ 11 ], a n d t h e n e s t i m a t e d the errors in the c o m p u t e r surface n o r m a l s using (31) a n d (32). W e o b t a i n e d the s a m e results as in [ 11]. T h e m o s t significant result o f the sensitivity analysis is that, t y p i c a l l y , a 1° e r r o r in the surface o r i e n t a t i o n d e t e r m i n e d by the p h o t o metric stereo m e t h o d is c a u s e d b y a 1% c h a n g e in the i m a g e intensity or a 1° c h a n g e in the o r i e n t a t i o n o f a c o l l i m a t e d light source.

4. Conclusion In this p a p e r we d e v e l o p e d a new sensitivity analysis t e c h n i q u e for surface n o r m a l s d e t e r m i n e d b y the p h o t o m e t r i c stereo m e t h o d . Using the form u l a s d e r i v e d in the last section, it is p o s s i b l e to e s t i m a t e the errors in s u r f a c e o r i e n t a t i o n d u e to c h a n g e s in the v a r i o u s p a r a m e t e r s o f the p h o t o metric stereo a l g o r i t h m . As p o i n t e d o u t in [11], these sensitivity f o r m u l a s can be successfully u s e d to e x p l a i n the difference b e t w e e n t h e o r y a n d practice. H o w e v e r , this sensitivity analysis is valid o n l y for s m a l l errors. H e n c e , errors d u e to s p e c u l a r a n d m u t u a l reflection a n d s h a d o w s c a n n o t be e x p l a i n e d b y this analysis.

Signal Processing

References [1] L. Caponetti et al., "A three-dimensional vision system for bin-picking", Proc., I E E E Conf. Comput. Vision Pattern Recogniton, 1986, pp. 407-411. [2] L. Carrioli et al., "Recognition of polyhedra by photometric stereo", in: V. Cantoni, R. Creutzburg, S. Levialdi and G. Wolf, eds., Recent Issures in Pattern Analysis and Recognition, Springer, Berlin, 1989, pp. 238-248. [3] M.T. Chiaradia et al., "'A photometric approach to determine surface objects orientations for vision systems", in: V. Cappellini and R. Marconi, eds., Advances in Image Processing and Pattern Recognition, North-Holland, Amsterdam, 1986, pp. 277-283. [4] K. lkeuchi, "Determining surface orientations of specular surfaces by using the photometric stereo method", IEEE Trans. Pattern Anal Machine Intell., Vol. 3, No. 6, 1981, pp. 661-669. [5] K. Ikeuchi and B.K.P. Horn, "Numerical shape from shading and occluding boundries", Artificial Intelligence, Vol. 17, 1981, pp. 141-184. [6] K. Ikeuchi and Y. Shirai, "A model based vision system for recognition of machine parts", Proc. National Conf. on Artificial Intelligence, 1982, pp. 18-21. [7] X.Y. Jiang and H. Bunke, "Recognizing 3-D objects in needle maps", Proc. lOth lnternat. Conf. Pattern Recognition, Atlantic City, NJ, 1990, pp. 237-239. [8] B.K.P. Horn, "Understanding image intensity", Artificial Intelligence, Vol. 8, No. 11, 1977, pp. 201-231. [9] B.K.P. Horn, Robot vision, McGraw-Hill, New York, 1986, p. 461. [10] B.K.P. Horn and K. Ikeuchi, "The mechanical manipulation of randomly oriented parts", Scientific American, Vol. 251, No. 2, 1984, pp. 100-111. [ 11] R. Ray, J. Birk and R.B. Kelley, "'Error analysis of surface normals determined by radiometry", IEEE Trans. Pattern Anal. Machine Intell., Vol. 5, No. 6, 1983, pp. 631-645. [12] R.J. Woodham, "A cooperative algorithm for determining surface orientation from a single view", Proc. lnternat. Joint Conf. on Artificial Intelligence, 1977, pp. 635-641. [13] R.J. Woodham, "Photometric stereo: A reflectance map technique for determining surface orientation from image intensity", Proc. SPIE, Vol. 155, 1978, pp. 936-943.