- Email: [email protected]
Selection for gamut mapping colour constancy
Image and Vision Computing 17 (1999) 597–604
Selection for gamut mapping colour constancy Graham Finlayson*, Steven Hordley Colour and Imaging Institute, University of Derby, Derby DE22 3BL, UK Received 15 October 1997; received in revised form 28 September 1998; accepted 30 October 1998
Abstract The requirement that the sensor responses of a camera to a given surface reflectance be constant under changing illumination conditions has led to the development of the so called colour constancy algorithms. Given an image recorded under an unknown illuminant, the task for a colour constancy algorithm is to recover an estimate of the scene illuminant. One such algorithm developed by D.A. Forsyth, A novel algorithm for colour constancy, International Journal of Computer Vision 5 (1) (1990) 5–36 [1] and later extended by G.D. Finlayson, Color in perspective, IEEE Transactions on Pattern Analysis and Machine Intelligence 18(10) (1996) 1034–1038 [2] exploits the constraint that under a canonical illuminant all surface colours fall within a maximal convex set-the canonical gamut. Given a set of image colours Forsyth’s algorithm recovers the set of mappings which take these colours into the canonical gamut. This feasible set of mappings represents all illuminants, which are consistent with the recorded image colours. In this article we address the question of how best to select a single mapping from this feasible set as an estimate of the unknown illuminant. We develop our approach in the context of Finlayson’s colour-in-perspective algorithm. This algorithm performs a perspective transform on the sensor data to discard intensity information which, without unrealistic constraints (uniform illumination and no specularities) being placed on the world, cannot be recovered accurately. Unfortunately, the feasible set of mappings recovered by this algorithm is also perspectively distorted. Here, we argue that this distortion must be removed prior to carrying out map selection and show that this is easily achieved by inverting the perspective transform. A mean-selection criterion operating on non-perspective mapping space provides good colour constancy for a variety of synthetic and real images. Importantly, constancy performance surpasses all other existing methods. q 1999 Elsevier Science B.V. All rights reserved. Keywords: Gamut mapping; Dimension; Colour constancy
1. Introduction The sensor responses of rgbs measured by a camera depend on the surfaces in a scene and on the prevailing illumination conditions. Many computer vision tasks require a descriptor of the surface reflectance, which is constant under changing illumination. This requirement has led a number of researchers [1–11] to develop colour constancy algorithms whose task is to process illuminant dependent rgbs to recover surface descriptors, which are illuminant independent. Unfortunately, the colour constancy problem has proven to be very hard to solve. This is because every rgb descriptor is a function of the colour of both the illuminant and the surface. Both factors are described by three numbers, so that if there are n surfaces in an image and one prevailing illuminant we have 3n independent camera measurements but 3n 1 3 parameters to solve for (all the surfaces plus the * Corresponding author. E-mail address: [email protected] (G. Finlayson)
illuminant). Some authors [12,8,9] have suggested that additional world constraints be imposed to deal with the illposedness of this problem. For example, it is assumed that there is a white reflectance in every scene [9] or that reflectances and illuminants are exactly described by linear models with few parameters [12]. As neither these, nor other constraints, hold in the real world, assuming that they do lead to very poor colour constancy performance [13,14]. In this article, rather than assuming any additional world constraints, we use a gamut mapping algorithm to solve for the set of all possible solutions to the colour constancy problem. We then propose a new method for selecting a single answer from this solution set. The gamut mapping approach to colour constancy begun by Forsyth [1] and extended by Finlayson [2] has resulted in the most successful solution to the colour constancy problem developed so far. Forsyth founded his work on the assumption that scenes consist only of flat, matte surfaces, and that though of undetermined colour and intensity, the illumination is spatially constant. Given such a scene, the problem as Forsyth posed it, is to recover the
0262-8856/99/$ - see front matter q 1999 Elsevier Science B.V. All rights reserved. PII: S0262-885 6(98)00179-6
598
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
rgb descriptor of each surface in the scene. That is, the rgb which would be recorded by the camera when the surface is imaged under a standard or canonical illuminant. The rgb sensor responses of a camera to all possible surfaces imaged under this canonical illuminant form a closed convex set, called the canonical gamut by Forsyth. This set we denote G (C) Under the scene illuminant the recorded camera responses form a different convex set, which we denote as G (O). Forsyth recognised that if we can establish a formal relationship between gamuts under different illuminants, then we are a long way towards solving the colour constancy problem. It is well established [1,12] that under certain world restrictions (e.g. surfaces are flat Lambertian tiles presented parallel to the image plane and illumination is everywhere uniform in both the power and spectral characteristics), colour responses under different illuminants are approximately a linear transform apart. That is, there is a unique transform, MO,C which takes the set of all camera responses under the scene illuminant to the canonical gamut:
G C MO;C G O
1
Solving for colour constancy in a given scene then amounts to finding MO,C for the scene illuminant. In general however, scenes contain only a subset of all possible surface reflectances hence the corresponding set of sensor responses – the image gamut G (I) – is a subset of G (O). In such case there will be many mappings, which take the image gamut, G (I) into the canonical gamut G (C). Finding this set of mappings is the first task solved by gamut mapping algorithms. Once this set is found, an estimate of the illuminant is chosen by selecting a single mapping from this set. The first step in solving for colour constancy by gamut mapping is to find all 3 × 3 linear transforms M such that ;p [ G I
Mp [ G C
2
where p denotes a colour which belongs to the image gamut. Forsyth [1] proposed an algorithm, called MWEXT, to perform this task. However, he found this algorithm to be impractical for a number of reasons. First, a general linear map has 98 of freedom and so solving Eq. (2) is a 9-dimensional problem. Although, Forsyth formalised an algorithmic solution he found the computation to be ‘laborious’ (it certainly has very high computational complexity). Moreover, it is unclear that the full 9-dimensional freedom is really necessary since it implies that a bright red colour might be mapped to bright blue under an illuminant change, yet at the same time a bright blue would be mapped to bright red. This type of pathological colour change does not occur in nature. Indeed, ordinal relationships amongst colours are generally preserved [15]. To overcome these problems Forsyth proposed that illuminant maps M should be restricted to three parameter diagonal matrices. Although a rather arbitrary restriction when Forsyth proposed it, later work [16] has shown that in general it is valid. With this restriction the first stage in
solving for colour constancy is simplified to the problem of finding all diagonal matricies, D such that: ;p [ G I
Dp [ G C
3
Forsyth [1] developed a second algorithm, called CRULE, which performed this task. As diagonal maps have only three parameters, colour constancy now becomes a tractable 3-d problem. Forsyth presented experimental results which show that CRULE performs well provided that the assumed world restrictions-flat matte world under uniform lighting-are satisfied. However, when these assumptions do not hold, and this is the case for almost all real images, then CRULE does not work well. The reason for CRULE’s failure is that when scenes contain specular highlights, spatially varying illumination, or shape information, the linear relationship defined in Eq. (3) does not apply. Importantly, Finlayson [2] recognised that whilst these factors prevent accurate recovery of illumination intensity, they do not affect the estimate of the orientation of the illumination vector in colour space. He used this fact to develop an alternative algorithm which recovers only orientation information but which is unaffected by factors such as specular highlights present in real images. Intensity is factored out of the colour constancy problem by mapping colour vectors into a perspective space (3-d colours are mapped to 2-d chromaticities in much the same way that 3-d scene points are mapped to 2-d image locations during image formation). At a second stage Forsyth’s CRULE algorithm is applied to this perspective colour data. This new algorithm, though unable to recover intensity information is more successful at recovering orientation information from real images than Forsyth’s algorithm [2]. While Finlayson’s perspective algorithm allowed the gamut mapping idea to be applied to real-world images, it does not generate a single answer to the colour constancy problem. Rather algorithm output, is as in the case of CRULE, the set of all maps which take image colours back to canonical viewing conditions. To obtain a single answer Finlayson [2] adopted the heuristic of making the image gamut as colourful as possible (first used by Forsyth [1]). Later Barnard [17] proposed instead, taking the mean of the feasible set. Neither of these answers that turns out is acceptable. This is because in order to make gamut mapping work the 3-d rgbs were transformed to 2-d perspective colours and this in turn implies that the set of feasible maps is also perspectively distorted. In this article we propose a 3-d, non-perspective selection procedure. At the final stage we take the 2-d answer generated by gamut mapping and invert the perspective transform in order to get a 3-d mapping set. The average of this 3-d cone of solutions is shown to be a much more accurate and stable selection procedure. Indeed, it supports excellent colour constancy performance for both synthetic and real images and is a significant improvement over the other
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
constancy algorithms particularly when images contain few colours. We review Forsyth’s gamut mapping approach to colour constancy and Finlayson’s modifications to this work in Section 2. We then address the question of map selection in Section 3 and present a new version of Finlayson’s colour in perspective algorithm. In Section 4 we compare the performance of this new algorithm on both real and synthetic images with previous gamut mapping algorithms. Finally we draw some conclusions from this work in Section 5.
C {pc;1 ; pc;2 ; …; pc;n }
5
of camera responses. As each p represents the rgb response of the camera to a physically realisable surface, any convex combination, pc;x of vectors in C is also physically realisable: n X
vi pc;i
;ivi . 0
i1
Gamut mapping colour constancy, as presented here, is based on the assumption that the sensor responses of a camera to a surface viewed under two different illuminants are a diagonal matrix apart. If pc is the response to a surface under a canonical illuminant, and po the response to the same surface under an arbitrary, and possibly unknown illuminant, then pc Do;c po
good approximation to this set can be formed by considering a representative subset of surface reflectances. Each surface i imaged under the canonical illuminant c produces a rgb sensor response represented as the 3-vector pc;i . Imaging n surfaces gives us a set
pc;x
2. Gamut mapping colour constancy
599
4
Consider the response of a camera to a single surface under an unknown illuminant. Clearly, this response can be mapped to the sensor response of any surface under a canonical illuminant by applying such a diagonal transform. Fortunately, the sensor responses of a camera to all possible surface reflectances under the canonical illuminant form a closed convex set, hence the set of mappings, D 1 taking our single surface to the canonical set is also closed and convex. If we now take a second surface under the same illuminant as the first, we find a different set of mappings, D 2 taking this surface to the canonical set. However, for these mappings to be valid, they must also take the first surface into the canonical set. That is, only mappings which belong to both D 1 and D 2 represent possible illuminants, so the solution set is reduced. Adding more surfaces will reduce the size of the solution set still further. Gamut mapping works by finding the set of valid mappings for each surface in the image; the intersection of all these sets is then the feasible set. Each mapping in the feasible set takes all image sensor responses into the canonical gamut. The set I, of camera sensor responses to a flat Mondrian (a patchwork matte reflectances) viewed under an unknown, but spatially uniform, illuminant forms the input to Forsyth’s gamut mapping algorithm. The output of the algorithm is the set of mapping whose elements each of which take all colours in the image gamut, G (I) into the canonical gamut, G (C). Before describing how this set of mappings is found, we need to make the notions of canonical and image gamuts precise. The canonical gamut is defined to be the set of all sensor responses obtained by viewing all physically realisable surface reflectances under a canonical illuminant. A very
n X
vi 1
6
i1
So the canonical gamut is the set of all convex combinations of vectors in C, denoted G (C). G (C) is infinite but as it is convex we need consider only its convex hull, defined to be [18] the minimal cardinality subset of C, denoted C, such that G (C) G (C). Given a set of rgbs, I recorded under an unknown illuminant the image gamut is defined in a similar manner. Let I be the set of convex hull points of I. Any convex combination of points in this set could also be recorded under the unknown illuminant. Hence the image gamut is the set of all convex combinations of I, denoted G (I). The gamuts G (C) and G (I) taken together, provide sufficient constraint for solving the colour constancy problem as defined by Eq. (3). We now show how these sets are used to solve Eq. (3); that is to characterise the set of mappings taking image gamut to canonical gamut. Consider the uth point in I. This point can be mapped to the vth point in C by applying a diagonal matrix D u,v such that: Du;v pi;u pc;v
Du;v kk
pc;v k pi;u k
k 1; 2; 3
7
We can now define the set C=pi;u ; whose vth element is a diagonal map taking the point pi;u to the vth point of C. Representing the diagonal matrices as 3-vectors we have C=pi;u {Dp : p [ C}
Dkk
1 pi;u k
k 1; 2; 3 8
It follows that pi;u can be mapped to any point in G (C) by a convex combination of vectors in C=pi;u . We denote the set of all such convex combinations G C=pi;u Similarly, we can define a corresponding set of mappings for each point pi;u in G (I). Clearly the set of mappings, G (C/I) which take all points in the image gamut into the canonical gamut, is the intersection of all the sets, G C=pi;u which take a single point in I into the canonical gamut:
G C=I
m \
G C=pi;j
9
j1
Under the world conditions imposed by Forsyth, his
600
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
illuminants, it is possible to run Forsyth’s CRULE algorithm directly on the 2-dimensional perspective colours. Again, this algorithm produces a set of feasible mappings from which we must choose one. The question of how the selection should best be done is addressed in Section 3.
3. Selecting a mapping
Fig. 1. The letter x denotes two diagonal maps in perspective mapping space and e their mean. The mean calculated in 3-dimensions and then projected to 2-dimensions is denoted c (the correct answer). It is apparent that calculating the mean in perspective can result in large deviations from the correct answer.
CRULE algorithm delivers an accurate characterisation of the solution set for colour constancy. On real images however the algorithm is less effective. Real scenes are complicated by factors such as specularities, varying illumination and shape, that break the assumption of linear illuminant change. Finlayson [2] recognised however, that though these factors prevent accurate recovery of the illuminant intensity they do not affect recovery of the orientation of the illuminant vector in rgb space. He therefore proposed an algorithm to recover only the orientation of the illuminant vector, reducing a 3-dimensional problem to a simpler 2-dimensional one. Finlayson ignores intensity information by mapping the 3-d rgb vectors to 2-d perspective chromaticity values. If a colour is represented by a 3-vector (r,g,b) t, then its perspective chromaticity co-ordinates are calculated as r0
r b
g0
g b
1
b b
10
This produces a vector r 0 ; g 0 ; 1t in the same direction as r; g; bt but with a different intensity. Also as the third coordinate is always 1 we can represent these new vectors as 2-vectors r 0 ; g 0 t : An important feature of the perspective transform is that gamut convexity is preserved [2]. Perspective diagonal matrices are formed in an analogous manner to forming perspective colours: 2a 32 i 3 2 32 i 3 2 c 3 r a 0 0 r r 0 6 g 7 6 bi 7 6 76 i 7 6 c 7 76 7 6 0 b 0 76 g 7 6 g 7 ) 6 6 76 7 4 54 5 4 5 4 b 5 4 gi 5 0 c 0 0 g b bi g bi 2 c 3 r 6 bc 7 6 7 11 6 c 7 4g 5 bc As the diagonal matrices map perspective colours between
Finlayson’s colour in perspective algorithm results in a set of 2-D mappings which take the gamut of image colours into the canonical gamut. The final step in solving for colour constancy is to choose one mapping from this set to represent the unknown illuminant. One way of doing this [2] is to find the map that takes all image colours into the canonical gamut such that image colours are made as colourful as possible (this is achieved by finding the maximum area feasible map). A second approach by Barnard [17] is to assume that all mappings have an equal probability of being correct. It is then natural to take the mean (expected) value of the mapping set as our choice of mapping. However, neither of these selection methods is appropriate as gamut mapping was carried out in a non-linear perspective colour space. To illustrate the problem with the perspective transform consider the situation where the correct mapping in 3dimensions is (1,1,1) t, and where the set of possible solutions lies on a straight line between (3,2,1) t and (1,2,3) t. The mean vector of this 3-dimensional solution set is (2,2,2), which is a scaling from the correct answer. Also, as we have argued earlier that we can only recover the illuminant up to a scalar, this is the correct answer. The effect of the perspective transform on these vectors is illustrated in Fig. 1. Notice that the correct answer (labelled c) becomes (1,1) t and the possible solutions (delimited by x) fall on the straight line between (3,2) t and (1/3,2/3) t. The mean of the 2-d mapping set (labelled e) equals (5/3,4/3) which is quite far from the correct answer: (1,1). It is clear that the perspective transform has introduced an unwanted distortion into the mapping set and as such a 2-dimensional mean estimation selection procedure is not appropriate. To avoid this bias at the selection stage we need to reconstruct the full 3-dimensions of the mapping set (as in the above example a mean-estimator on 3-dimensional maps was seen to work well). A perspective map (a /g ,b /g ) t can be converted to a three vector (a /g ,b /g ,1) t if we remember that the mapping set lies in the plane g 1. We can then construct a cone with vertex at the origin and rays passing through the extreme points of the mapping set in the plane g 1. This cone bounds the possible variation in orientation of our feasible maps. To select a map from this cone of feasible maps we again assume that all maps are equally likely and take as our choice of mapping the mean map. The cone-based mean computation is contrasted with the perspective computation in Fig. 2. In this figure the set of all possible 3-dimensional
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
601
find an approximate solution. Essentially this involves randomly generating 3-vectors and calculating the average of all such vectors that fall inside the calculated constraint cone and the unit sphere. As the number of random vectors chosen increases, this average value approaches the mean of the cone. A similar method is employed to find the mean in 2-dimensions. We show in Section 4 that by selecting the average mapping in 3-dimensions we achieve significantly better colour constancy performance than is obtainable by doing the selection in a 2-dimensional perspective space.
4. Results
Fig. 2. The convex polygon denotes the gamut of feasible mappings calculated by 2-d gamut-mapping. The cone bounds the corresponding 3-dimensional set (constructed by inverting the perspective transform). The mean of the perspective gamut of mappings (open circle) is quite different from the mean calculated in 3-dimensions (filled circle).
diagonal maps is bounded by the cone and the average of this cone is drawn as a dashed line. The convex polygon, (dotted line), in the plane g 1 represents the perspective projection of the 3-d cone. The average of the projected set of mappings is drawn as an open circle. It is apparent that the average 3-d map perspectively projected (filled circle) is not equal to the average of the projected set of 3-d mappings and this illustrates the difficulty of carrying out estimation in perspective. Calculating the mean of a set in 3-dimensions involves calculating the following integrals: 1 Z 1 Z 1 Z a a dV b b dV g g dV V D V D V D 12 where V is the volume of the set and D is the boundary of the cone. Note that every a , b and g corresponds to a plausible diagonal map. We must be careful how we carry out the computation As the cone of possible mappings is unbounded: the map (1,1,1) is treated no differently from (100,100,100) or (1000,1000,1000). Thus, we must bound the size (length) of the maps. This is done by intersecting the cone, shown in Fig. 2, with the unit sphere. To calculate the integrals in Eq. (12) analytically turns out to be quite difficult because of the nature of the boundary constraints (a convex cone intersected with the unit sphere), so instead we use the method of Monte Carlo integration to
To evaluate our new selection method we first looked at performance using synthetic data. The reasons for this are threefold. First, performance is not affected by image noise. Second, the diagonal model of colour constancy only holds exactly for perfectly narrow-band sensors (that is sensors which respond to light only at a single wavelength); we choose such narrow-band sensors for our experiments on synthetic data. Third, we were able to evaluate selection performance over thousands of synthetic images and hence achieve a reliable performance statistic. To construct the canonical gamut we took a subset of the surface reflectances of the Munsell chips [19], imaged under the most spectrally uniform of the Judd [20] daylight phases (D55). For our camera sensors we used three perfectly narrow-band sensors: the red sensor was sensitive to light at 610 nm (and no other wavelength) and the green and blue sensors were sensitive to 540 and 450 nm light respectively. These narrow-band sensors have the advantage that they sample colour signals in a similar manner to the receptors found in the human eye [21]. Given narrow-band sensors the induced camera response of the kth sensor to a surface with reflectance function S(l ) under illuminant with spectral power distribution E(l ) is calculated [22]: p1 S l610 E l610 p2 S l540 E l540
13
p3 S l450 E l450 The 3-vector S l610 ; S l540 ; S l450 denotes the rgb of the surface and the 3-vector E l610 ; E l540 ; E l450 denotes the illuminant rgb. Modelling the interaction of light and surface by the product of their respective rgbs is a commonly used technique in computer graphics. The canonical gamut is then the convex hull of all sensor responses generated for all Munsell chips. While the range of surface reflectances of the Munsell chips are quite representative of naturally occurring reflectances, to allow for the fact that not all surface reflectances are represented we expanded the canonical gamut by 10%. A synthetic image was created by selecting between 1 and 11 surfaces from the same set of Munsell chips and a test illuminant from an illuminant set comprising the daylights;
602
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
Fig. 3. Results using synthetic data, for four different unknown illuminants, from top-to-bottom: D75, D100, a yellowish and a bluish Planckian black-body radiator. Left- and right-hand graphs show actual and worst-case error performances as a function of the number of colours in a scene.
D75 and D100, together with yellowish and bluish Planckian black-body radiators [19] (this illuminant set spans yellowish, bluish and whitish lights). The sensor responses of the camera were calculated according to Eq. (13) and these values were used as input to the colour in perspective gamut mapping procedure, which returned a set of feasible maps. To chose a single mapping from the feasible set we used each of the three selection methods: maximum [2] and mean [14] calculated in 2-d perspective and the mean in 3dimensions that was discussed in Section 3. As a measure of the accuracy of our chosen mapping we calculated the angle between it and the correct answer. Also, to give some idea of the worst case performance of each method we calculated the maximum angle possible between our chosen mapping and all the other mappings in the feasible set. To obtain a stable statistic we generated 500 scenes for each number of surfaces and unknown illuminant. Selection performance for four illuminants from the set (D75, D100, yellowish and bluish Planckian black-body radiators) is summarised in Fig. 3. The graphs on the left hand side show average angular error against the number of surfaces in the image for each of the three selection methods. The right hand graphs show the worst case error for
each of the three methods. In both the average and worst case it is clear that the 3-d mean (dotted line) does significantly better than either the maximum area (solid line) or 2d mean (dashed line). For small numbers of surfaces (between 1 and 5) the 3-d mean estimate does between
Fig. 4. The Macbeth Colour Chart.
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
603
Fig. 5. Results using real data for unknown illuminants: yellowish tungsten (top) and bluish daylight (bottom). Left- and right-hand graphs show actual and worst-case error performances as a function of the number of colours in a scene.
100% and 50% better. This level of performance increase offers a clear practical advantage for scenes where there are a small number of image colours (for example images of forests typically contain shades of only two or three colours). As the number of surfaces are increased the difference in performance narrows. This is to be expected as increasing the number of colours in the image leads to a smaller feasible set of mappings. To test the performance with real data we began by creating a measured canonical gamut. We imaged a Macbeth Color Chart (which was designed to be representative of the typical colours) under whitish cloudy sky illumination and calculated the convex hull of the rgbs that were induced in a SONY DXC-930 colour camera (which had linear response). An image of the Macbeth chart is shown in Fig. 4. The Macbeth chart was then imaged under blue sky and yellow tungsten lights. For each illuminant we then generated random scenes by selecting between 1 and 11 of the 24 checker patches and used these as input to the gamut mapping algorithm. Fig. 5 shows the results using yellow tungsten, and blue daylight as the unknown illuminant (top and bottom respectively). We again show average (left) and worst case (right) performance for each of the three selection methods. The graphs show the same trend as was obtained with the synthetic data. These clearly show that the new selection method produces a significant improvement in performance over either of the two previously used selection methods.
5. Conclusions Gamut mapping colour constancy algorithms such as Finlayson’s colour in perspective algorithm produce a set of feasible mappings representing the unknown illuminant. In this article we addressed the issue of how best to select a mapping from this set. We have presented a new selection
method and shown that this performs better than the existing methods using real and synthetic data. Our new method is significantly better, particularly for scenes with small numbers of surfaces. It should be noted that in this article we have ignored a significant feature of Finlayson’s colour in perspective algorithm, namely the illumination constraint. This constraint embodies the idea that some illuminant colours (e.g. saturated purples) and hence some illuminant maps, are very unlikely. It follows then that the size of the set of feasible maps calculated by gamut mapping algorithm might be reduced by applying this constraint. However, the problem of selection remains. As such, regardless of the initial size of the feasible map set, the selection method developed in this article should lead to improved colour constancy performance. Acknowledgements This work was supported by the EPSRC and by Hewlett– Packard Incorporated. References [1] D.A. Forsyth, A novel algorithm for colour constancy, International Journal of Computer Vision 5 (1) (1990) 5–36. [2] G.D. Finlayson, Color in Perspective, IEEE Transactions on Pattern Analysis and Machine Intelligence 18 (10) (1996) 1034–1038. [3] D.H. Brainard, W.T. Freeman, Bayesian method for recovering surface and illuminant properties from photosensor responses, Proc. of IS&T/SPIE Symposium on Electronic Imaging Science and Technology 2179 (1994) 364–376. [4] G. Buchsbaum, A spatial processor model for object colour perception, Journal of the Franklin Institute 310 (1980) 1–26. [5] M.M. D’Zmura, G. Iverson, Probabilistic color constancy, in: M.M. D’Zmura, D. Hoffman, G. Iverson, K. Romney (Eds.), Geometric Representations of Perceptual Phenomena: Articles in Honor of Tarow Indow’s 70th Birthday, Laurance Erlbaum Associates, 1994.
604
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
[6] M.M. D’Zmura, Color constancy: surface color from changing illumination, J. of the Optical Society of America, A 9 (3) (1987) 2148– 2165. [7] B.V. Funt, M.S. Drew, Color constancy computation in nearmondrian scenes using a finite dimensional linear model, in: Proc. of the Computer Vision and Pattern Recognition, IEEE Computer Society, 1988, pp. 544–549. [8] R. Gershon, A.D. Jepson, J.K. Tsotsos, From [R,G,B] to surface reflectance: computing color constant descriptors in images, Perception, 1988, pp. 755–758. [9] E.H. Land, The retinex theory of color constancy, Scientific American, 1977, pp. 108–129. [10] E.H. Land, Recent advances in retinex theory and some implications for cortical computations, Proc. Natl. Acad. Sci. USA 80 (1983) 5163–5169. [11] E.H. Land, Recent advances in retinex theory, Vision Research 26 (1) (1986) 7–21. [12] L.T. Maloney, B.A. Wandell, Color constancy: a method for recovering surface spectral reflectance, J. of the Optical Society of America, A 3 (1) (1986) 29–33. [13] G.D. Finalyson, B.V. Funt, K. Barnard, Color constancy under varying illumination, in: Proc. of the 5th International Conference on Computer Vision, IEEE Computer Society, 1995, pp. 720–725. [14] K. Barnard, G.D. Finlayson, B.V. Funt, Color constancy for scenes
[15]
[16]
[17]
[18] [19] [20]
[21]
[22]
with varying illumination, Computer vision and image understanding 65 (2) (1997) 311–321. D.H. Foster, S.M. Nascimento, Relational colour constancy from invariant cone-excitation ratios, Proc. R. Soc. Lond. B 257 (1994) 115–121. G.D. Finlayson, M.S. Drew, B.V. Funt, Spectral Sharpening: sensor transformations for improved color constancy, J. of the Optical Society of America, A 11 (5) (1994) 1553–1563. K. Barnard, Computational color constancy: taking theory into practice. master’s thesis, Simon Fraser University, School of Computing Science, 1995. F.P. Preparata, M.I. Shamos, Computational geometry: An introduction, Springer, New York, 1985. G. Wyszecki, W.S. Stiles, Color science: concepts and methods, Quantitative Data and Formulas, 2nd ed., Wiley, New York, 1982. D.B. Judd, D.L. MacAdam, G.W. Wyszecki, Spectral distribution of typical daylight as a function of correlated colour temperature, Journal of the Optical Society of America, A 54 (1964) 1031–1040. G.D. Finlayson, B.V. Funt, Coefficient channels: Derivation and relationship to other theoretical studies, COLOR research and application 21 (2) (1996) 87–96. J.A. Worthey, M.H. Brill, Heuristic analysis of von Kries color constancy, J. of the Optical Society of America, A, 3 (10) (1986) 1708–1712.
Selection for gamut mapping colour constancy Graham Finlayson*, Steven Hordley Colour and Imaging Institute, University of Derby, Derby DE22 3BL, UK Received 15 October 1997; received in revised form 28 September 1998; accepted 30 October 1998
Abstract The requirement that the sensor responses of a camera to a given surface reflectance be constant under changing illumination conditions has led to the development of the so called colour constancy algorithms. Given an image recorded under an unknown illuminant, the task for a colour constancy algorithm is to recover an estimate of the scene illuminant. One such algorithm developed by D.A. Forsyth, A novel algorithm for colour constancy, International Journal of Computer Vision 5 (1) (1990) 5–36 [1] and later extended by G.D. Finlayson, Color in perspective, IEEE Transactions on Pattern Analysis and Machine Intelligence 18(10) (1996) 1034–1038 [2] exploits the constraint that under a canonical illuminant all surface colours fall within a maximal convex set-the canonical gamut. Given a set of image colours Forsyth’s algorithm recovers the set of mappings which take these colours into the canonical gamut. This feasible set of mappings represents all illuminants, which are consistent with the recorded image colours. In this article we address the question of how best to select a single mapping from this feasible set as an estimate of the unknown illuminant. We develop our approach in the context of Finlayson’s colour-in-perspective algorithm. This algorithm performs a perspective transform on the sensor data to discard intensity information which, without unrealistic constraints (uniform illumination and no specularities) being placed on the world, cannot be recovered accurately. Unfortunately, the feasible set of mappings recovered by this algorithm is also perspectively distorted. Here, we argue that this distortion must be removed prior to carrying out map selection and show that this is easily achieved by inverting the perspective transform. A mean-selection criterion operating on non-perspective mapping space provides good colour constancy for a variety of synthetic and real images. Importantly, constancy performance surpasses all other existing methods. q 1999 Elsevier Science B.V. All rights reserved. Keywords: Gamut mapping; Dimension; Colour constancy
1. Introduction The sensor responses of rgbs measured by a camera depend on the surfaces in a scene and on the prevailing illumination conditions. Many computer vision tasks require a descriptor of the surface reflectance, which is constant under changing illumination. This requirement has led a number of researchers [1–11] to develop colour constancy algorithms whose task is to process illuminant dependent rgbs to recover surface descriptors, which are illuminant independent. Unfortunately, the colour constancy problem has proven to be very hard to solve. This is because every rgb descriptor is a function of the colour of both the illuminant and the surface. Both factors are described by three numbers, so that if there are n surfaces in an image and one prevailing illuminant we have 3n independent camera measurements but 3n 1 3 parameters to solve for (all the surfaces plus the * Corresponding author. E-mail address: [email protected] (G. Finlayson)
illuminant). Some authors [12,8,9] have suggested that additional world constraints be imposed to deal with the illposedness of this problem. For example, it is assumed that there is a white reflectance in every scene [9] or that reflectances and illuminants are exactly described by linear models with few parameters [12]. As neither these, nor other constraints, hold in the real world, assuming that they do lead to very poor colour constancy performance [13,14]. In this article, rather than assuming any additional world constraints, we use a gamut mapping algorithm to solve for the set of all possible solutions to the colour constancy problem. We then propose a new method for selecting a single answer from this solution set. The gamut mapping approach to colour constancy begun by Forsyth [1] and extended by Finlayson [2] has resulted in the most successful solution to the colour constancy problem developed so far. Forsyth founded his work on the assumption that scenes consist only of flat, matte surfaces, and that though of undetermined colour and intensity, the illumination is spatially constant. Given such a scene, the problem as Forsyth posed it, is to recover the
0262-8856/99/$ - see front matter q 1999 Elsevier Science B.V. All rights reserved. PII: S0262-885 6(98)00179-6
598
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
rgb descriptor of each surface in the scene. That is, the rgb which would be recorded by the camera when the surface is imaged under a standard or canonical illuminant. The rgb sensor responses of a camera to all possible surfaces imaged under this canonical illuminant form a closed convex set, called the canonical gamut by Forsyth. This set we denote G (C) Under the scene illuminant the recorded camera responses form a different convex set, which we denote as G (O). Forsyth recognised that if we can establish a formal relationship between gamuts under different illuminants, then we are a long way towards solving the colour constancy problem. It is well established [1,12] that under certain world restrictions (e.g. surfaces are flat Lambertian tiles presented parallel to the image plane and illumination is everywhere uniform in both the power and spectral characteristics), colour responses under different illuminants are approximately a linear transform apart. That is, there is a unique transform, MO,C which takes the set of all camera responses under the scene illuminant to the canonical gamut:
G C MO;C G O
1
Solving for colour constancy in a given scene then amounts to finding MO,C for the scene illuminant. In general however, scenes contain only a subset of all possible surface reflectances hence the corresponding set of sensor responses – the image gamut G (I) – is a subset of G (O). In such case there will be many mappings, which take the image gamut, G (I) into the canonical gamut G (C). Finding this set of mappings is the first task solved by gamut mapping algorithms. Once this set is found, an estimate of the illuminant is chosen by selecting a single mapping from this set. The first step in solving for colour constancy by gamut mapping is to find all 3 × 3 linear transforms M such that ;p [ G I
Mp [ G C
2
where p denotes a colour which belongs to the image gamut. Forsyth [1] proposed an algorithm, called MWEXT, to perform this task. However, he found this algorithm to be impractical for a number of reasons. First, a general linear map has 98 of freedom and so solving Eq. (2) is a 9-dimensional problem. Although, Forsyth formalised an algorithmic solution he found the computation to be ‘laborious’ (it certainly has very high computational complexity). Moreover, it is unclear that the full 9-dimensional freedom is really necessary since it implies that a bright red colour might be mapped to bright blue under an illuminant change, yet at the same time a bright blue would be mapped to bright red. This type of pathological colour change does not occur in nature. Indeed, ordinal relationships amongst colours are generally preserved [15]. To overcome these problems Forsyth proposed that illuminant maps M should be restricted to three parameter diagonal matrices. Although a rather arbitrary restriction when Forsyth proposed it, later work [16] has shown that in general it is valid. With this restriction the first stage in
solving for colour constancy is simplified to the problem of finding all diagonal matricies, D such that: ;p [ G I
Dp [ G C
3
Forsyth [1] developed a second algorithm, called CRULE, which performed this task. As diagonal maps have only three parameters, colour constancy now becomes a tractable 3-d problem. Forsyth presented experimental results which show that CRULE performs well provided that the assumed world restrictions-flat matte world under uniform lighting-are satisfied. However, when these assumptions do not hold, and this is the case for almost all real images, then CRULE does not work well. The reason for CRULE’s failure is that when scenes contain specular highlights, spatially varying illumination, or shape information, the linear relationship defined in Eq. (3) does not apply. Importantly, Finlayson [2] recognised that whilst these factors prevent accurate recovery of illumination intensity, they do not affect the estimate of the orientation of the illumination vector in colour space. He used this fact to develop an alternative algorithm which recovers only orientation information but which is unaffected by factors such as specular highlights present in real images. Intensity is factored out of the colour constancy problem by mapping colour vectors into a perspective space (3-d colours are mapped to 2-d chromaticities in much the same way that 3-d scene points are mapped to 2-d image locations during image formation). At a second stage Forsyth’s CRULE algorithm is applied to this perspective colour data. This new algorithm, though unable to recover intensity information is more successful at recovering orientation information from real images than Forsyth’s algorithm [2]. While Finlayson’s perspective algorithm allowed the gamut mapping idea to be applied to real-world images, it does not generate a single answer to the colour constancy problem. Rather algorithm output, is as in the case of CRULE, the set of all maps which take image colours back to canonical viewing conditions. To obtain a single answer Finlayson [2] adopted the heuristic of making the image gamut as colourful as possible (first used by Forsyth [1]). Later Barnard [17] proposed instead, taking the mean of the feasible set. Neither of these answers that turns out is acceptable. This is because in order to make gamut mapping work the 3-d rgbs were transformed to 2-d perspective colours and this in turn implies that the set of feasible maps is also perspectively distorted. In this article we propose a 3-d, non-perspective selection procedure. At the final stage we take the 2-d answer generated by gamut mapping and invert the perspective transform in order to get a 3-d mapping set. The average of this 3-d cone of solutions is shown to be a much more accurate and stable selection procedure. Indeed, it supports excellent colour constancy performance for both synthetic and real images and is a significant improvement over the other
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
constancy algorithms particularly when images contain few colours. We review Forsyth’s gamut mapping approach to colour constancy and Finlayson’s modifications to this work in Section 2. We then address the question of map selection in Section 3 and present a new version of Finlayson’s colour in perspective algorithm. In Section 4 we compare the performance of this new algorithm on both real and synthetic images with previous gamut mapping algorithms. Finally we draw some conclusions from this work in Section 5.
C {pc;1 ; pc;2 ; …; pc;n }
5
of camera responses. As each p represents the rgb response of the camera to a physically realisable surface, any convex combination, pc;x of vectors in C is also physically realisable: n X
vi pc;i
;ivi . 0
i1
Gamut mapping colour constancy, as presented here, is based on the assumption that the sensor responses of a camera to a surface viewed under two different illuminants are a diagonal matrix apart. If pc is the response to a surface under a canonical illuminant, and po the response to the same surface under an arbitrary, and possibly unknown illuminant, then pc Do;c po
good approximation to this set can be formed by considering a representative subset of surface reflectances. Each surface i imaged under the canonical illuminant c produces a rgb sensor response represented as the 3-vector pc;i . Imaging n surfaces gives us a set
pc;x
2. Gamut mapping colour constancy
599
4
Consider the response of a camera to a single surface under an unknown illuminant. Clearly, this response can be mapped to the sensor response of any surface under a canonical illuminant by applying such a diagonal transform. Fortunately, the sensor responses of a camera to all possible surface reflectances under the canonical illuminant form a closed convex set, hence the set of mappings, D 1 taking our single surface to the canonical set is also closed and convex. If we now take a second surface under the same illuminant as the first, we find a different set of mappings, D 2 taking this surface to the canonical set. However, for these mappings to be valid, they must also take the first surface into the canonical set. That is, only mappings which belong to both D 1 and D 2 represent possible illuminants, so the solution set is reduced. Adding more surfaces will reduce the size of the solution set still further. Gamut mapping works by finding the set of valid mappings for each surface in the image; the intersection of all these sets is then the feasible set. Each mapping in the feasible set takes all image sensor responses into the canonical gamut. The set I, of camera sensor responses to a flat Mondrian (a patchwork matte reflectances) viewed under an unknown, but spatially uniform, illuminant forms the input to Forsyth’s gamut mapping algorithm. The output of the algorithm is the set of mapping whose elements each of which take all colours in the image gamut, G (I) into the canonical gamut, G (C). Before describing how this set of mappings is found, we need to make the notions of canonical and image gamuts precise. The canonical gamut is defined to be the set of all sensor responses obtained by viewing all physically realisable surface reflectances under a canonical illuminant. A very
n X
vi 1
6
i1
So the canonical gamut is the set of all convex combinations of vectors in C, denoted G (C). G (C) is infinite but as it is convex we need consider only its convex hull, defined to be [18] the minimal cardinality subset of C, denoted C, such that G (C) G (C). Given a set of rgbs, I recorded under an unknown illuminant the image gamut is defined in a similar manner. Let I be the set of convex hull points of I. Any convex combination of points in this set could also be recorded under the unknown illuminant. Hence the image gamut is the set of all convex combinations of I, denoted G (I). The gamuts G (C) and G (I) taken together, provide sufficient constraint for solving the colour constancy problem as defined by Eq. (3). We now show how these sets are used to solve Eq. (3); that is to characterise the set of mappings taking image gamut to canonical gamut. Consider the uth point in I. This point can be mapped to the vth point in C by applying a diagonal matrix D u,v such that: Du;v pi;u pc;v
Du;v kk
pc;v k pi;u k
k 1; 2; 3
7
We can now define the set C=pi;u ; whose vth element is a diagonal map taking the point pi;u to the vth point of C. Representing the diagonal matrices as 3-vectors we have C=pi;u {Dp : p [ C}
Dkk
1 pi;u k
k 1; 2; 3 8
It follows that pi;u can be mapped to any point in G (C) by a convex combination of vectors in C=pi;u . We denote the set of all such convex combinations G C=pi;u Similarly, we can define a corresponding set of mappings for each point pi;u in G (I). Clearly the set of mappings, G (C/I) which take all points in the image gamut into the canonical gamut, is the intersection of all the sets, G C=pi;u which take a single point in I into the canonical gamut:
G C=I
m \
G C=pi;j
9
j1
Under the world conditions imposed by Forsyth, his
600
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
illuminants, it is possible to run Forsyth’s CRULE algorithm directly on the 2-dimensional perspective colours. Again, this algorithm produces a set of feasible mappings from which we must choose one. The question of how the selection should best be done is addressed in Section 3.
3. Selecting a mapping
Fig. 1. The letter x denotes two diagonal maps in perspective mapping space and e their mean. The mean calculated in 3-dimensions and then projected to 2-dimensions is denoted c (the correct answer). It is apparent that calculating the mean in perspective can result in large deviations from the correct answer.
CRULE algorithm delivers an accurate characterisation of the solution set for colour constancy. On real images however the algorithm is less effective. Real scenes are complicated by factors such as specularities, varying illumination and shape, that break the assumption of linear illuminant change. Finlayson [2] recognised however, that though these factors prevent accurate recovery of the illuminant intensity they do not affect recovery of the orientation of the illuminant vector in rgb space. He therefore proposed an algorithm to recover only the orientation of the illuminant vector, reducing a 3-dimensional problem to a simpler 2-dimensional one. Finlayson ignores intensity information by mapping the 3-d rgb vectors to 2-d perspective chromaticity values. If a colour is represented by a 3-vector (r,g,b) t, then its perspective chromaticity co-ordinates are calculated as r0
r b
g0
g b
1
b b
10
This produces a vector r 0 ; g 0 ; 1t in the same direction as r; g; bt but with a different intensity. Also as the third coordinate is always 1 we can represent these new vectors as 2-vectors r 0 ; g 0 t : An important feature of the perspective transform is that gamut convexity is preserved [2]. Perspective diagonal matrices are formed in an analogous manner to forming perspective colours: 2a 32 i 3 2 32 i 3 2 c 3 r a 0 0 r r 0 6 g 7 6 bi 7 6 76 i 7 6 c 7 76 7 6 0 b 0 76 g 7 6 g 7 ) 6 6 76 7 4 54 5 4 5 4 b 5 4 gi 5 0 c 0 0 g b bi g bi 2 c 3 r 6 bc 7 6 7 11 6 c 7 4g 5 bc As the diagonal matrices map perspective colours between
Finlayson’s colour in perspective algorithm results in a set of 2-D mappings which take the gamut of image colours into the canonical gamut. The final step in solving for colour constancy is to choose one mapping from this set to represent the unknown illuminant. One way of doing this [2] is to find the map that takes all image colours into the canonical gamut such that image colours are made as colourful as possible (this is achieved by finding the maximum area feasible map). A second approach by Barnard [17] is to assume that all mappings have an equal probability of being correct. It is then natural to take the mean (expected) value of the mapping set as our choice of mapping. However, neither of these selection methods is appropriate as gamut mapping was carried out in a non-linear perspective colour space. To illustrate the problem with the perspective transform consider the situation where the correct mapping in 3dimensions is (1,1,1) t, and where the set of possible solutions lies on a straight line between (3,2,1) t and (1,2,3) t. The mean vector of this 3-dimensional solution set is (2,2,2), which is a scaling from the correct answer. Also, as we have argued earlier that we can only recover the illuminant up to a scalar, this is the correct answer. The effect of the perspective transform on these vectors is illustrated in Fig. 1. Notice that the correct answer (labelled c) becomes (1,1) t and the possible solutions (delimited by x) fall on the straight line between (3,2) t and (1/3,2/3) t. The mean of the 2-d mapping set (labelled e) equals (5/3,4/3) which is quite far from the correct answer: (1,1). It is clear that the perspective transform has introduced an unwanted distortion into the mapping set and as such a 2-dimensional mean estimation selection procedure is not appropriate. To avoid this bias at the selection stage we need to reconstruct the full 3-dimensions of the mapping set (as in the above example a mean-estimator on 3-dimensional maps was seen to work well). A perspective map (a /g ,b /g ) t can be converted to a three vector (a /g ,b /g ,1) t if we remember that the mapping set lies in the plane g 1. We can then construct a cone with vertex at the origin and rays passing through the extreme points of the mapping set in the plane g 1. This cone bounds the possible variation in orientation of our feasible maps. To select a map from this cone of feasible maps we again assume that all maps are equally likely and take as our choice of mapping the mean map. The cone-based mean computation is contrasted with the perspective computation in Fig. 2. In this figure the set of all possible 3-dimensional
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
601
find an approximate solution. Essentially this involves randomly generating 3-vectors and calculating the average of all such vectors that fall inside the calculated constraint cone and the unit sphere. As the number of random vectors chosen increases, this average value approaches the mean of the cone. A similar method is employed to find the mean in 2-dimensions. We show in Section 4 that by selecting the average mapping in 3-dimensions we achieve significantly better colour constancy performance than is obtainable by doing the selection in a 2-dimensional perspective space.
4. Results
Fig. 2. The convex polygon denotes the gamut of feasible mappings calculated by 2-d gamut-mapping. The cone bounds the corresponding 3-dimensional set (constructed by inverting the perspective transform). The mean of the perspective gamut of mappings (open circle) is quite different from the mean calculated in 3-dimensions (filled circle).
diagonal maps is bounded by the cone and the average of this cone is drawn as a dashed line. The convex polygon, (dotted line), in the plane g 1 represents the perspective projection of the 3-d cone. The average of the projected set of mappings is drawn as an open circle. It is apparent that the average 3-d map perspectively projected (filled circle) is not equal to the average of the projected set of 3-d mappings and this illustrates the difficulty of carrying out estimation in perspective. Calculating the mean of a set in 3-dimensions involves calculating the following integrals: 1 Z 1 Z 1 Z a a dV b b dV g g dV V D V D V D 12 where V is the volume of the set and D is the boundary of the cone. Note that every a , b and g corresponds to a plausible diagonal map. We must be careful how we carry out the computation As the cone of possible mappings is unbounded: the map (1,1,1) is treated no differently from (100,100,100) or (1000,1000,1000). Thus, we must bound the size (length) of the maps. This is done by intersecting the cone, shown in Fig. 2, with the unit sphere. To calculate the integrals in Eq. (12) analytically turns out to be quite difficult because of the nature of the boundary constraints (a convex cone intersected with the unit sphere), so instead we use the method of Monte Carlo integration to
To evaluate our new selection method we first looked at performance using synthetic data. The reasons for this are threefold. First, performance is not affected by image noise. Second, the diagonal model of colour constancy only holds exactly for perfectly narrow-band sensors (that is sensors which respond to light only at a single wavelength); we choose such narrow-band sensors for our experiments on synthetic data. Third, we were able to evaluate selection performance over thousands of synthetic images and hence achieve a reliable performance statistic. To construct the canonical gamut we took a subset of the surface reflectances of the Munsell chips [19], imaged under the most spectrally uniform of the Judd [20] daylight phases (D55). For our camera sensors we used three perfectly narrow-band sensors: the red sensor was sensitive to light at 610 nm (and no other wavelength) and the green and blue sensors were sensitive to 540 and 450 nm light respectively. These narrow-band sensors have the advantage that they sample colour signals in a similar manner to the receptors found in the human eye [21]. Given narrow-band sensors the induced camera response of the kth sensor to a surface with reflectance function S(l ) under illuminant with spectral power distribution E(l ) is calculated [22]: p1 S l610 E l610 p2 S l540 E l540
13
p3 S l450 E l450 The 3-vector S l610 ; S l540 ; S l450 denotes the rgb of the surface and the 3-vector E l610 ; E l540 ; E l450 denotes the illuminant rgb. Modelling the interaction of light and surface by the product of their respective rgbs is a commonly used technique in computer graphics. The canonical gamut is then the convex hull of all sensor responses generated for all Munsell chips. While the range of surface reflectances of the Munsell chips are quite representative of naturally occurring reflectances, to allow for the fact that not all surface reflectances are represented we expanded the canonical gamut by 10%. A synthetic image was created by selecting between 1 and 11 surfaces from the same set of Munsell chips and a test illuminant from an illuminant set comprising the daylights;
602
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
Fig. 3. Results using synthetic data, for four different unknown illuminants, from top-to-bottom: D75, D100, a yellowish and a bluish Planckian black-body radiator. Left- and right-hand graphs show actual and worst-case error performances as a function of the number of colours in a scene.
D75 and D100, together with yellowish and bluish Planckian black-body radiators [19] (this illuminant set spans yellowish, bluish and whitish lights). The sensor responses of the camera were calculated according to Eq. (13) and these values were used as input to the colour in perspective gamut mapping procedure, which returned a set of feasible maps. To chose a single mapping from the feasible set we used each of the three selection methods: maximum [2] and mean [14] calculated in 2-d perspective and the mean in 3dimensions that was discussed in Section 3. As a measure of the accuracy of our chosen mapping we calculated the angle between it and the correct answer. Also, to give some idea of the worst case performance of each method we calculated the maximum angle possible between our chosen mapping and all the other mappings in the feasible set. To obtain a stable statistic we generated 500 scenes for each number of surfaces and unknown illuminant. Selection performance for four illuminants from the set (D75, D100, yellowish and bluish Planckian black-body radiators) is summarised in Fig. 3. The graphs on the left hand side show average angular error against the number of surfaces in the image for each of the three selection methods. The right hand graphs show the worst case error for
each of the three methods. In both the average and worst case it is clear that the 3-d mean (dotted line) does significantly better than either the maximum area (solid line) or 2d mean (dashed line). For small numbers of surfaces (between 1 and 5) the 3-d mean estimate does between
Fig. 4. The Macbeth Colour Chart.
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
603
Fig. 5. Results using real data for unknown illuminants: yellowish tungsten (top) and bluish daylight (bottom). Left- and right-hand graphs show actual and worst-case error performances as a function of the number of colours in a scene.
100% and 50% better. This level of performance increase offers a clear practical advantage for scenes where there are a small number of image colours (for example images of forests typically contain shades of only two or three colours). As the number of surfaces are increased the difference in performance narrows. This is to be expected as increasing the number of colours in the image leads to a smaller feasible set of mappings. To test the performance with real data we began by creating a measured canonical gamut. We imaged a Macbeth Color Chart (which was designed to be representative of the typical colours) under whitish cloudy sky illumination and calculated the convex hull of the rgbs that were induced in a SONY DXC-930 colour camera (which had linear response). An image of the Macbeth chart is shown in Fig. 4. The Macbeth chart was then imaged under blue sky and yellow tungsten lights. For each illuminant we then generated random scenes by selecting between 1 and 11 of the 24 checker patches and used these as input to the gamut mapping algorithm. Fig. 5 shows the results using yellow tungsten, and blue daylight as the unknown illuminant (top and bottom respectively). We again show average (left) and worst case (right) performance for each of the three selection methods. The graphs show the same trend as was obtained with the synthetic data. These clearly show that the new selection method produces a significant improvement in performance over either of the two previously used selection methods.
5. Conclusions Gamut mapping colour constancy algorithms such as Finlayson’s colour in perspective algorithm produce a set of feasible mappings representing the unknown illuminant. In this article we addressed the issue of how best to select a mapping from this set. We have presented a new selection
method and shown that this performs better than the existing methods using real and synthetic data. Our new method is significantly better, particularly for scenes with small numbers of surfaces. It should be noted that in this article we have ignored a significant feature of Finlayson’s colour in perspective algorithm, namely the illumination constraint. This constraint embodies the idea that some illuminant colours (e.g. saturated purples) and hence some illuminant maps, are very unlikely. It follows then that the size of the set of feasible maps calculated by gamut mapping algorithm might be reduced by applying this constraint. However, the problem of selection remains. As such, regardless of the initial size of the feasible map set, the selection method developed in this article should lead to improved colour constancy performance. Acknowledgements This work was supported by the EPSRC and by Hewlett– Packard Incorporated. References [1] D.A. Forsyth, A novel algorithm for colour constancy, International Journal of Computer Vision 5 (1) (1990) 5–36. [2] G.D. Finlayson, Color in Perspective, IEEE Transactions on Pattern Analysis and Machine Intelligence 18 (10) (1996) 1034–1038. [3] D.H. Brainard, W.T. Freeman, Bayesian method for recovering surface and illuminant properties from photosensor responses, Proc. of IS&T/SPIE Symposium on Electronic Imaging Science and Technology 2179 (1994) 364–376. [4] G. Buchsbaum, A spatial processor model for object colour perception, Journal of the Franklin Institute 310 (1980) 1–26. [5] M.M. D’Zmura, G. Iverson, Probabilistic color constancy, in: M.M. D’Zmura, D. Hoffman, G. Iverson, K. Romney (Eds.), Geometric Representations of Perceptual Phenomena: Articles in Honor of Tarow Indow’s 70th Birthday, Laurance Erlbaum Associates, 1994.
604
G. Finlayson, S. Hordley / Image and Vision Computing 17 (1999) 597–604
[6] M.M. D’Zmura, Color constancy: surface color from changing illumination, J. of the Optical Society of America, A 9 (3) (1987) 2148– 2165. [7] B.V. Funt, M.S. Drew, Color constancy computation in nearmondrian scenes using a finite dimensional linear model, in: Proc. of the Computer Vision and Pattern Recognition, IEEE Computer Society, 1988, pp. 544–549. [8] R. Gershon, A.D. Jepson, J.K. Tsotsos, From [R,G,B] to surface reflectance: computing color constant descriptors in images, Perception, 1988, pp. 755–758. [9] E.H. Land, The retinex theory of color constancy, Scientific American, 1977, pp. 108–129. [10] E.H. Land, Recent advances in retinex theory and some implications for cortical computations, Proc. Natl. Acad. Sci. USA 80 (1983) 5163–5169. [11] E.H. Land, Recent advances in retinex theory, Vision Research 26 (1) (1986) 7–21. [12] L.T. Maloney, B.A. Wandell, Color constancy: a method for recovering surface spectral reflectance, J. of the Optical Society of America, A 3 (1) (1986) 29–33. [13] G.D. Finalyson, B.V. Funt, K. Barnard, Color constancy under varying illumination, in: Proc. of the 5th International Conference on Computer Vision, IEEE Computer Society, 1995, pp. 720–725. [14] K. Barnard, G.D. Finlayson, B.V. Funt, Color constancy for scenes
[15]
[16]
[17]
[18] [19] [20]
[21]
[22]
with varying illumination, Computer vision and image understanding 65 (2) (1997) 311–321. D.H. Foster, S.M. Nascimento, Relational colour constancy from invariant cone-excitation ratios, Proc. R. Soc. Lond. B 257 (1994) 115–121. G.D. Finlayson, M.S. Drew, B.V. Funt, Spectral Sharpening: sensor transformations for improved color constancy, J. of the Optical Society of America, A 11 (5) (1994) 1553–1563. K. Barnard, Computational color constancy: taking theory into practice. master’s thesis, Simon Fraser University, School of Computing Science, 1995. F.P. Preparata, M.I. Shamos, Computational geometry: An introduction, Springer, New York, 1985. G. Wyszecki, W.S. Stiles, Color science: concepts and methods, Quantitative Data and Formulas, 2nd ed., Wiley, New York, 1982. D.B. Judd, D.L. MacAdam, G.W. Wyszecki, Spectral distribution of typical daylight as a function of correlated colour temperature, Journal of the Optical Society of America, A 54 (1964) 1031–1040. G.D. Finlayson, B.V. Funt, Coefficient channels: Derivation and relationship to other theoretical studies, COLOR research and application 21 (2) (1996) 87–96. J.A. Worthey, M.H. Brill, Heuristic analysis of von Kries color constancy, J. of the Optical Society of America, A, 3 (10) (1986) 1708–1712.