- Email: [email protected]
Vectorial morphological reconstruction for brightness elimination in colour images
ARTICLE IN PRESS
Real-Time Imaging 10 (2004) 379–387 www.elsevier.com/locate/rti
Vectorial morphological reconstruction for brightness elimination in colour images F. Ortiz, F. Torres Automatics, Robotics and Computer Vision Group, Department of Physics, Systems Engineering and Signal Theory, University of Alicante, P.O. Box 99, 03080 Alicante, Spain Available online 15 December 2004
Abstract In this paper, we present a new method for brightness elimination in chromatic images by means of the extension of the morphological reconstruction for colour images. We use the HSI colour space for morphological processing, after solving certain inconveniences present in this colour model. We investigate the connected filters of the mathematical morphology and, in particular, the so-called filters by reconstruction. These filters are formally defined for colour images. The power of the morphological reconstruction for brightness elimination in colour images is presented. We also describe a new algorithm to control the oversimplification of images and to reduce the high processing cost of the geodesic operations. Examples of its application are shown. The new method proposed here achieves good results, which are similar to those obtained from other techniques, yet does not require either costly multiple-view systems or stereo images. r 2004 Elsevier Ltd. All rights reserved.
1. Introduction In visual systems, images are acquired in work environments in which illumination plays an important role. Sometimes, a bad adjustment of the illumination can introduce brightness (highlights or specular reflectance) into the objects captured by the vision system. Highlights in images have long been disruptive to computer-vision algorithms. They appear as surface features, which can lead to problems, such as stereo mismatching, false segmentation and recognition errors. Furthermore, the brightness affects the visual quality of the scene in some multimedia applications. In order to reduce the problem of brightness in computer vision, Lin et al. [1] have developed a system for eliminating specularities in image sequences by means of stereo correspondence. Bajcsy et al. [2] use a chromatic space based on polar coordinates that allows the detection of specular and diffuse reflections by means of the previous knowledge of the captured scene. Corresponding author. Tel./fax: +34 965 909 750.
E-mail address: fortiz@dfists.ua.es (F. Ortiz). 1077-2014/$ - see front matter r 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.rti.2004.10.002
Wolff [3], for his part, removes highlights by taking advantage of differences in polarization between diffuse reflections and highlights. Nayar et al. [4] utilize both colour and polarization to constrain estimates the components of reflections. These approaches have produced good results but entail requirements that limit their applicability, such as the use of stereo or multipleview systems, the previous knowledge of the scene, or the assumption of a homogeneous illumination. In the other hand, Morphological filters by reconstruction are becoming increasingly popular in image processing. They constitute a powerful operation in mathematical morphology, since they permit the simplification of an image while preserving its contours. This property is very attractive for a wide range of applications in computer vision, such as simplification, segmentation, noise suppression, etc. [5–7]. In this paper, we present an extension of morphological reconstruction for a new application: brightness elimination in colour images. The colour space chosen for the processing is the HSI, which is compatible with the human perception of colours. Once the problems that arise in this space with the mathematical morphology
ARTICLE IN PRESS 380
F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
have been solved, it can be used to form a complete lattice that allows the morphological operators to be applied to colour images. To avoid problems of oversimplification (loss of information of the image) in the reconstruction and to reduce the high cost of the connected vectorial operations, we employ a technique for the detection of the area of application of the filter by means of a colour maker, i.e., in red, green or blue parts of the image. The organization of this paper is as follows: Section 2 presents the colour space chosen for mathematical processing. In Section 3, we extend the geodesic operations to colour images. In Section 4, we show the utility of the connected vectorial filters for brightness elimination in colour images, together with a new algorithm to avoid over-simplification and to obtain minimum cost of processing. Some examples of application of this new method and results are shown in Section 5. Finally, our conclusions are outlined in the last section.
2. Use of HSI colour space in mathematical morphology In the literature, many referenced colour models appear [8–12]. In general, they are three-dimensional spaces that can be classified in standardized systems (CIE-RGB, CIE-XYZ, CIE-Lab), physical systems (RGB and CMY), and perceptual systems (HSI, HSV, HLS,y), which are widely used in image processing. Important advantages of perceptual spaces are: good correlation with the human perception of colours and separability of chromatic values from achromatic values. The most representative of these colour models is the HSI (hue, saturation and intensity) and as other perceptual spaces, the HSI is derived from the RGB cube. All the HSI family of colour models use polar coordinates, as can be seen in Fig. 1. The saturation
corresponds to relative colour purity and it is proportional to radial distance, the hue represents the impression related to the dominant wavelength of the colour stimulus and it is a function of the angle in the polar coordinate system. Finally, the intensity or perceived lightness is the distance along the axis perpendicular to the polar plane. Two other colour spaces of the HSI family are HSV and HLS [12]. They differ of HSI in values of intensity and saturation. However, not all perceptual phenomena have been implemented into these models: fully saturated colours with different hues have the same values V ¼ 1 (HSV) or lightness L ¼ 0.5 (HLS) when this is not always true in human perception (V and L in range [0,1]). Several studies have been carried out on the application of mathematical morphology to colour images [13–16]. The approach most commonly adopted is based on the use of a lexicographical order [17], which imposes total order on the colour vectors. We use a lexicographical method to form a complete lattice structure with the HSI colour space. Let x ¼ (x1, x2,y,xn) and y ¼ (y1, y2,y, yn) be two arbitrary vectors (x,yAZn), an example of lexicographical order olex, will be 8 x1 oy1 or > < x1 ¼ y1 and x2 oy2 or : (1) xoy if > : x ¼ y and x ¼ y :::and:::x oy 1 2 n 1 2 n The preference or disposition of the components in the lexicographical ordering depends of the application and the properties of the image. Ordering with luminance (intensity) in the first position is the best way of preserving the contours of the objects in the image, I-H-S (lattice influenced by intensity). In situations in which the objects of interest are highly coloured or in which only objects of a specific colour are of interest, the operations with hue in the first position are the best H-I-S (lattice influenced by hue). The HSI colour family is commonly used in computer graphics. However, HSI have some features that prevent the correct formation of the necessary lattice in mathematical morphology, such as: instability of saturation in the highest and lowest level of intensity, a lack of internal order in the hue component and hue of no value in zero saturation. The use of HSI colour morphology requires the above-metioned inconveniences to be solved. In the following section we solve such drawbacks. 2.1. The adapting of the HSI colour space to form complete lattices
Fig. 1. Polar representation of HSI colour space.
The conversion from RGB to HSI is higly non-linear and complex. The formulae are given in the literature in different form, in particular for the hue H. The reason is a compromise between transformation accuracy and
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
computational simplicity. The original formula for hue, derived by Tenenbaum et al. [18] is as follows: 0:5½ðR GÞ þ ðR BÞ H ¼ cos1 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ; ðR GÞðR GÞ þ ðR GÞðG BÞ
I¼
minðR; G; BÞ ; RþGþB
RþGþB ; 3
(3)
(4)
Some instability arises in saturation for small variations of RGB values in the neighbourhood of low or high intensity. Small changes in RGB coordinates cause great variations in S coordinate. To avoid this, we must linearly reduce the saturation as the intensity increases to white point or decreases to black point. This way, we change from a cylindrical geometrical representation to an exact double-cone shape. The saturation map is now more amenable to morphological treatment. A problem also arises with the hue in trying to form a complete lattice with HSI. Hue is angle-valued, if f(p) ¼ (fI(p), fH(p), fS(p)) are the colour values of the pixel p from a colour image in the HSI colour space, then fH(p)A[0,2p). In addition, one cannot order hues from their lowest to their highest values. To order hues, Hanbury [15] and Peters [20] use a hue-valued structuring function. Hues are ordered according to the absolute value of a distance function between the image hue and a reference hue (infimum): dðf H ðpÞ; H ref Þ ( jf H ðpÞ H ref j if jf H ðpÞ H ref jpp ¼ : 2p jf H ðpÞ H ref j if jf H ðpÞ H ref j4p
changed to I-S, H-I-S must be changed to I-S, etc. Empirically, we obtain good results with STh smaller than a 5% of maximum saturation.
(2)
where if B4R, then H ¼ 3601-H. Hue is normalized to the range [0,1] by letting H ¼ H/3601. The exact derivation of these equations, resulting from converting the RGB Cartesian coordinate system in range [0,1] to the HSI cylindrical one in range [0,1], is given by Ledley et al. [19]. The coordinates of saturation S and intensity I are calculated as follows: S ¼13
381
ð5Þ
With this definition, the value of the distance will fluctuate between 0 and 180 (3601/2). In a range [0,1] the distance fluctuate between 0 and 0.5. In this case, when the hue has no value, it is represented with a value of 1 [16]. Finally, from the definition of the complete lattice, we must ensure that all of the elements in the lattice can be ordered and compared [21]. There is a problem with the hue when the saturation is zero, this means that the hue signal cannot be used to order pixels in such cases. We can also define a saturation threshold STh, which determines which pixels have a significant hue and which have not. In the lattice, if there is a pixel p with fS(p)oSTh, the lexicographical order I-H-S must be
3. Morphological filters by reconstruction in colour images Mathematical morphology is a non-linear image processing which is based on the application of lattice theory to spatial structures [22]. In mathematical morphology, we can define the morphological filters by reconstruction, which are the basis of numerous transformations of higher level ranging, from simple procedures for removing all objects connected to the image border to more sophisticated image algorithms [6,23–25]. The morphological reconstruction is based on geodesic operations, which do not require to choose a structuring element or to set its size. The approach taken with geodesic transformations is to consider two input images (mask and marker). A morphological transformation is applied to the marker image and it is then forced to remain either above or below the mask image. The use of the filters by reconstruction in colour images requires an order relationship among the pixels of the image. For the vectorial morphological processing, the lexicographical ordering will be used. As such, the infimum (4n) and supremum (3n) will be vectorial operators, and they will select pixels according to their order olex in the HSI colour space. This is the same approach successfully used in the detection of colour cells in real-time medical imaging [26] and the Gaussian noise elimination in colour images [27]. Once the orders have been defined, the morphological operators of reconstruction for colour images can be generated and applied. An elementary geodesic operation is the geodesic dilation. Let g denote a marker colour image and f a mask colour image (if olex(g)polex(f), then g4nf ¼ g). The vectorial geodesic dilation of size 1 of the marker image g with respect to the mask f can be defined as ð1Þ dð1Þ vf ðgÞ ¼ dv ðgÞ^v f ;
(6)
where dð1Þ v ðgÞ is the vectorial dilation of size 1 of the marker image g. The vectorial propagation of the marker g is limited by the mask f by the vectorial infimum 4n. The vectorial geodesic dilation of size n of a marker colour image g with respect to a mask colour image f is obtained by performing n successive geodesic dilations of g with respect to f: ð1Þ ðn1Þ dðnÞ ðgÞ nf ðgÞ ¼ dnf ½dnf
with
dð0Þ nf ðgÞ
¼ f:
(7)
ARTICLE IN PRESS 382
F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
Geodesic transformations of images always converge after a finite number of iterations [16]. The propagation of the marker image is impeded by the mask image. Morphological reconstruction of a mask image is based on this principle. The vectorial reconstruction by dilation of a mask colour image f from a marker colour image g (both with Df ¼ Dg and olex ðgÞ olex ðf Þ) can be defined as Rvf ðgÞ ¼ dðnÞ vf ðgÞ;
Fig. 2. Vectorial reconstruction by dilation, lexicographical order IH-S: (a) colour mask, (b) colour marker imposition, (c) result of the vectorial reconstruction by dilation. False colours are not present in the result.
(8)
ðnþ1Þ where n is such that dðnÞ ðgÞ: The vectorial vf ðgÞ ¼ dvf reconstruction by dilation is a geodesic dilation iterated until stability. The vectorial reconstruction is an algebraic opening (increasing, anti-extensive and idempotent) only if all the operations between pixels respect the total order, in our case, the lexicographical order. Fig. 2 illustrates the morphological operation of reconstruction for colour signals. We can see the mask (Fig. 2a) and the colour marker (orange in Fig. 2b). The propagation of the marker is made according to its order in relation to the colour mask. This is a geodesic dilation iterated until stability (Fig. 2c). In our vectorial reconstruction there are no new or false colours, but only the colours of the marker or the mask.
Fig. 3. Algorithm for brightness elimination with minimum cost and without over-simplification.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
383
4. Brightness elimination in colour images Morphological filters by reconstruction can be used to simplify the colour images while preserving contours. This property makes them very attractive for segmentation. In this paper, we use the vectorial opening by reconstruction as a filter to reduce brightness in colour images. This is an interesting application for many areas in computer vision. For the morphological operations we employ a lexicographical order olex: I-H-S (lattice influenced by intensity). The reason for choosing H as the second signal, rather than S is explained in Section 4.1. The filter’s effect is the removal of any bright components that are smaller than the structuring element used in the vectorial erosion. In the results there is a very good preservation of the contour information. In the following section, we present a new method for brightness elimination only in certain areas of interest of the image. As such, we achieve a better control of the filter by reconstruction, avoiding the over-simplification or loss of information in the image. Furthermore, we reduce significantly the processing cost of the operation. 4.1. Algorithm for minimum-cost and without oversimplification One significant improvement in the reconstruction algorithms is the possibility of choosing the areas of the images that we really want to simplify. This way, we avoid the over-simplification effect (loss of information in significant areas of the image) and we reduce the processing time required for the reconstruction, which is longer due to the multiple iterations of the filter. In colour images, the hue Href can be used as a signal to determine the areas of interest of the image (objects with brightness). We propose the following method: 1. The objects in the image to be processed f(x,y) are marked through a hue distance threshold from Href. Empirically, we obtain good results with a maximum distance of the choosen marker equal of 0.1 in range [0,1], but this threshold can be modified according to the image. The result is the binary image h(x,y). 2. The binary mask h(x,y) is processed by a closingopening filter. Holes in the objects are closed and the opening eliminates the smaller objects, which are not to be considered. We thus obtain the binary mask i(x,y). 3. For the elimination of the brightness we use a vectorial filter of opening by reconstruction (VOR) which only functions in the areas defined by the mask i(x,y). In this filter, a pyramid of vectorial erosions is done with different structuring elements of increasing sizes. We use a lexicographical order olex: I-H-S, in which Href is the hue of the marker for the first
Fig. 4. Vectorial opening by reconstruction from eroded image. Erosions done with structuring elements of increasing sizes (1–8).
Fig. 5. Over-simplification of the image ‘‘Transformer’’ in the brightness elimination process.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
384
stage. As such, when there are two pixels with the same intensity, the hue value will be observed. The new filter can be defined as follows: 0
0
ðn Þ ðn Þ ðSÞ gvf ;i ¼ fdvf v ðf Þj8f ðx; yÞ ) iðx; yÞ ¼ 1g; ðn0 Þ ðSÞ dvf ðv ðf ÞÞ
(9)
erosion replaces highlight pixels (high olex) by the surroundings chromatic pixels (low olex). Next, the vectorial geodesic dilation (iterated until stability) reconstructs the colour image without the recovering of the specularities or brightness.
0 þ1Þ ðSÞ dðn ðv ðf ÞÞ: vf
where n’ is such that ¼ The vectorial erosion of the opening by reconstruction is done with a structural element of size s. This
In Fig. 3, the different stages of the algorithm proposed are shown together with the brightness
Fig. 6. Detection and elimination of brightness in the ‘‘Scissors’’ image: (a) original image, (b) mask-image, (c) morphological elimination of brightness causing over-simplification, (d) elimination of brightness without over-simplification.
Fig. 7. Detection and elimination of brightness in the ‘‘Key ring’’ image: (a) original image, (b) mask-image, (c) morphological elimination of brightness causing over-simplification, (d) elimination of brightness without over-simplification.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
elimination of the image ‘‘Transformer’’. In Fig. 4 we can see the progressive reduction of brightness in the area of interest of the image. This reduction has been obtained by means of a pyramid of vectorial erosions that have been carried out with eight structuring elements of sizes (s) between s ¼ 1 (3 3) and s ¼ 8 (17 17). The over-simplification of the image caused by a total geodesic reconstruction (Fig. 5) is achieved with this new algorithm.
5. Results We now present the new filter effect in different colours images that have brightness in comparison with standard morphological reconstruction (Figs. 6–8). In all the examples we present the original image, the maskimage, which indicates the area to be filtered, the vectorial colour opening by reconstruction of the original image and, finally, the new vectorial colour
385
opening by reconstruction which operates only in colour areas of the original image. In conventional connected filters, the size of the structuring element required for eliminating the brightness causes the loss of detail in the images (Figs. 6c, 7c and 8c). These features cannot be retrieved in the geodesic reconstruction. This inconvenient effect does not occur with our algorithm, since the erosion and reconstruction only function in those areas in which there are brightness (Figs. 6d, 7d and 8d). Furthermore, as the function of the connected filters is restricted to specific areas in the image, the results are obtained at a very lower processing cost. If the brightness appears in more than a chromatic area, we only change the colour marker and we apply the filter again to the image. The details of the algorithm’s parameters for the colour images processed are presented in Table 1. The size (s) of the structural element required in the vectorial erosion of VOR is different for each image. The classic vectorial geodesic
Fig. 8. Detection and elimination of brightness in the ‘‘Vase’’ image: (a) original image, (b) mask-image, (c) morphological elimination of brightness causing over-simplification, (d) elimination of brightness without over-simplification.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
386 Table 1 Parameters of the new algorithm Colour image
Href (deg)
S
Scissors Key ring Vase Transformer
220 60 30 0
2 3 4 8
(5 5) (7 7) (9 9) (17 17)
References n
n’
Reduction of processing time (%)
67 43 78 97
27 19 33 59
49 62 29 51
dilation requires more iterations (n) (with greater processing time) than the new operation (n0 ). This time reduction is very significant (1 or 2 s versus a few minutes of entire reconstruction). In comparison with other non-morphological methods for brightness elimination, our morphological reconstruction does not require the previous knowledge of the scene or the assumption of a homogeneous illumination model. In addition, the brightness elimination is obtained independenly of the material of the objects on which the highlights appear, without any need of multiple-view. The results are also obtained in a shorter CPU time. This permits to achieve real-time requirements in image processing. As in the case of other methods, the only limitation to this technique is the reconstruction of highly textured areas.
6. Conclusions In this paper, we have presented the use of connected vectorial filters to eliminate brightness in colour images by means of an extension of the geodesic transformations of the mathematical morphology to colour images. We have shown that the use of the HSI colour space affords results that are quite acceptable in mathematical morphology. The inconveniences of this colour space in establishing a lattice have been overcome by using the hue distance concept and by treating its indefinition in the lexicographical order. Using the colour information of the hue signal we have been able to define a control of the filter by reconstruction, which avoids the effects of over-simplification with the subsequent loss of important areas of the image. In addition, the brightness elimination has been obtained with a very lower processing time, with respect to a classic geodesic reconstruction. Currently, we are working in brightness elimination by means of the detection of specular reflectance of the image. This would allow a more accurate selection of the processing area, independently of the hue of the objects.
[1] Lin S, Li Y, Kang S, Tong X, Shum H. Diffuse-specular separation and depth recovery from image sequences. Lecture Notes in Computer Science 2002. [2] Bajcsy R, Lee S, Leonardis A. Detection of diffuse and specular interface reflections and inter-reflections by color image segmentation. International Journal on Computer Vision 1996. [3] Wolff L. Using polarization to separate refelction components. In: Proceedings of the IEEE computer vision and pattern recognition, 1989. p. 363–9. [4] Nayar A, Fang X, Boult T. Removal of specularities using color and polarization. In: Proceedings of the IEEE computer vision and pattern recognition, 1993. p. 583–90. [5] Crespo J, Serra J, Schafer R. Theoretical aspects of morphological filters by reconstruction. Signal Processing 1995. [6] Vicent L. Morphological grayscale reconstruction in image analysis: applications and efficient algoritms. IEEE Transactions on Image Processing 1993;2(2):176–201. [7] Salembier P, Serra J. Flat Zones Filtering, Connected Operators, and Filters by Reconstruction. IEEE Transactions on Image Processing 1995;4(8):1153–60. [8] Palus H. Representations of colour images in different colour spaces. In: Sangwine S, Horne R, editors. The colour image processing handbook. London: Chapman & Hall; 1998. p. 67–90. [9] Plataniotis K, Venetsanopoulos A. Color image processing and applications. Berlin: Springer; 2000. [10] Sharma G, Yrzel M, Trussel H. Digital color processing. IEEE Transaction on Image Processing 1997;6(7):901–32. [11] Smith A. Color gamut transform pairs. Computer Graphics 1978;12(3):12–9. [12] Foley J, vanDam A, Feiner S, Hughes J. Fundamentals of interactive computer graphics. Addison Wesley: Reading, MA; 1990. [13] Comer M, Delp E. Morphological operations for colour image processing. Journal of Electronic Imaging 1999;8(3): 279–89. [14] Ortiz F, Torres F, Angulo J, Puente S. Comparative study of vectorial morphological operations in different colour spaces. In: Proceedings of SPIE, intelligent robots and computer vision XX, vol. 4572, Boston, USA, 2001. p. 259–68. [15] Hanbury A, Serra J. Morphological operators on the unit circle. IEEE Transactions on Image Processing 2001;10(12): 1842–50. [16] Ortiz F. Procesamiento morfolo´gico de ima´genes en color. Aplicacio´n a la reconstruccio´n geode´sica. Ph.D. thesis, University of Alicante, 2002. [17] Chanussot J. Approches Vectorielles ou Marginales pour le Traitement d’Images Multicomposantes. Ph.D. thesis, University of Savoie, 1998. [18] Tenenbaum J, Garvey T, Welwy S, Wolf H. An interactive facility for scene analysis research. Technical Report 87, Stanford Research Institute, AI Centre, 1974. [19] Ledley S, Buas M, Golab T. Fundamentals of true-color image processing. In: Proceedings of 10th international conference on pattern recognition, Atlantic City, 1990. p. 791–5. [20] Peters II A. Mathematical morphology for angle-valued images. In: Proceedings of SPIE, non-linear image processing VIII, vol. 3026, 1997. p. 84–94. [21] Serra J. Anamorphoses and function lattices (multivalued morphology). In: Dougherty E, editor. Mathematical Morphology in Image Processing, 1993, p. 483–521. [22] Serra J. Image analysis and mathematical morphology, vol. I, and image analysis and mathematical morphology, vol. II: theoretical advances. New york: Academic Press; 1982, 1988.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387 [23] Beucher S. Segmentation d’Images et Morphologie Mathe´matique. Ph.D. thesis, Ecole des Mines de Paris, 1990. [24] Serra J, Salembier P. Connected operators and pyramides. In: Proceedings of the SPIE, non-linear algebra and morphological image processing, vol. 2030, 1993. p. 65–75. [25] Soille P. Morphological image analysis. Principles and applications. Berlin: Springer; 1999.
387
[26] Ortiz F, Torres F, De Juan E, Cuenca N. Colour mathematical morphology for neural image analysis. Journal of Real-Time Imaging 2002;8(6):455–65. [27] Ortiz F, Torres F, Gil P. Gaussian noise elimination in colour images by vector-connected filters. In: Proceedings of the IEEE 17th International Conference on Pattern Recognition, vol. 4, 2004. p. 807–11.
Real-Time Imaging 10 (2004) 379–387 www.elsevier.com/locate/rti
Vectorial morphological reconstruction for brightness elimination in colour images F. Ortiz, F. Torres Automatics, Robotics and Computer Vision Group, Department of Physics, Systems Engineering and Signal Theory, University of Alicante, P.O. Box 99, 03080 Alicante, Spain Available online 15 December 2004
Abstract In this paper, we present a new method for brightness elimination in chromatic images by means of the extension of the morphological reconstruction for colour images. We use the HSI colour space for morphological processing, after solving certain inconveniences present in this colour model. We investigate the connected filters of the mathematical morphology and, in particular, the so-called filters by reconstruction. These filters are formally defined for colour images. The power of the morphological reconstruction for brightness elimination in colour images is presented. We also describe a new algorithm to control the oversimplification of images and to reduce the high processing cost of the geodesic operations. Examples of its application are shown. The new method proposed here achieves good results, which are similar to those obtained from other techniques, yet does not require either costly multiple-view systems or stereo images. r 2004 Elsevier Ltd. All rights reserved.
1. Introduction In visual systems, images are acquired in work environments in which illumination plays an important role. Sometimes, a bad adjustment of the illumination can introduce brightness (highlights or specular reflectance) into the objects captured by the vision system. Highlights in images have long been disruptive to computer-vision algorithms. They appear as surface features, which can lead to problems, such as stereo mismatching, false segmentation and recognition errors. Furthermore, the brightness affects the visual quality of the scene in some multimedia applications. In order to reduce the problem of brightness in computer vision, Lin et al. [1] have developed a system for eliminating specularities in image sequences by means of stereo correspondence. Bajcsy et al. [2] use a chromatic space based on polar coordinates that allows the detection of specular and diffuse reflections by means of the previous knowledge of the captured scene. Corresponding author. Tel./fax: +34 965 909 750.
E-mail address: fortiz@dfists.ua.es (F. Ortiz). 1077-2014/$ - see front matter r 2004 Elsevier Ltd. All rights reserved. doi:10.1016/j.rti.2004.10.002
Wolff [3], for his part, removes highlights by taking advantage of differences in polarization between diffuse reflections and highlights. Nayar et al. [4] utilize both colour and polarization to constrain estimates the components of reflections. These approaches have produced good results but entail requirements that limit their applicability, such as the use of stereo or multipleview systems, the previous knowledge of the scene, or the assumption of a homogeneous illumination. In the other hand, Morphological filters by reconstruction are becoming increasingly popular in image processing. They constitute a powerful operation in mathematical morphology, since they permit the simplification of an image while preserving its contours. This property is very attractive for a wide range of applications in computer vision, such as simplification, segmentation, noise suppression, etc. [5–7]. In this paper, we present an extension of morphological reconstruction for a new application: brightness elimination in colour images. The colour space chosen for the processing is the HSI, which is compatible with the human perception of colours. Once the problems that arise in this space with the mathematical morphology
ARTICLE IN PRESS 380
F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
have been solved, it can be used to form a complete lattice that allows the morphological operators to be applied to colour images. To avoid problems of oversimplification (loss of information of the image) in the reconstruction and to reduce the high cost of the connected vectorial operations, we employ a technique for the detection of the area of application of the filter by means of a colour maker, i.e., in red, green or blue parts of the image. The organization of this paper is as follows: Section 2 presents the colour space chosen for mathematical processing. In Section 3, we extend the geodesic operations to colour images. In Section 4, we show the utility of the connected vectorial filters for brightness elimination in colour images, together with a new algorithm to avoid over-simplification and to obtain minimum cost of processing. Some examples of application of this new method and results are shown in Section 5. Finally, our conclusions are outlined in the last section.
2. Use of HSI colour space in mathematical morphology In the literature, many referenced colour models appear [8–12]. In general, they are three-dimensional spaces that can be classified in standardized systems (CIE-RGB, CIE-XYZ, CIE-Lab), physical systems (RGB and CMY), and perceptual systems (HSI, HSV, HLS,y), which are widely used in image processing. Important advantages of perceptual spaces are: good correlation with the human perception of colours and separability of chromatic values from achromatic values. The most representative of these colour models is the HSI (hue, saturation and intensity) and as other perceptual spaces, the HSI is derived from the RGB cube. All the HSI family of colour models use polar coordinates, as can be seen in Fig. 1. The saturation
corresponds to relative colour purity and it is proportional to radial distance, the hue represents the impression related to the dominant wavelength of the colour stimulus and it is a function of the angle in the polar coordinate system. Finally, the intensity or perceived lightness is the distance along the axis perpendicular to the polar plane. Two other colour spaces of the HSI family are HSV and HLS [12]. They differ of HSI in values of intensity and saturation. However, not all perceptual phenomena have been implemented into these models: fully saturated colours with different hues have the same values V ¼ 1 (HSV) or lightness L ¼ 0.5 (HLS) when this is not always true in human perception (V and L in range [0,1]). Several studies have been carried out on the application of mathematical morphology to colour images [13–16]. The approach most commonly adopted is based on the use of a lexicographical order [17], which imposes total order on the colour vectors. We use a lexicographical method to form a complete lattice structure with the HSI colour space. Let x ¼ (x1, x2,y,xn) and y ¼ (y1, y2,y, yn) be two arbitrary vectors (x,yAZn), an example of lexicographical order olex, will be 8 x1 oy1 or > < x1 ¼ y1 and x2 oy2 or : (1) xoy if > : x ¼ y and x ¼ y :::and:::x oy 1 2 n 1 2 n The preference or disposition of the components in the lexicographical ordering depends of the application and the properties of the image. Ordering with luminance (intensity) in the first position is the best way of preserving the contours of the objects in the image, I-H-S (lattice influenced by intensity). In situations in which the objects of interest are highly coloured or in which only objects of a specific colour are of interest, the operations with hue in the first position are the best H-I-S (lattice influenced by hue). The HSI colour family is commonly used in computer graphics. However, HSI have some features that prevent the correct formation of the necessary lattice in mathematical morphology, such as: instability of saturation in the highest and lowest level of intensity, a lack of internal order in the hue component and hue of no value in zero saturation. The use of HSI colour morphology requires the above-metioned inconveniences to be solved. In the following section we solve such drawbacks. 2.1. The adapting of the HSI colour space to form complete lattices
Fig. 1. Polar representation of HSI colour space.
The conversion from RGB to HSI is higly non-linear and complex. The formulae are given in the literature in different form, in particular for the hue H. The reason is a compromise between transformation accuracy and
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
computational simplicity. The original formula for hue, derived by Tenenbaum et al. [18] is as follows: 0:5½ðR GÞ þ ðR BÞ H ¼ cos1 pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ; ðR GÞðR GÞ þ ðR GÞðG BÞ
I¼
minðR; G; BÞ ; RþGþB
RþGþB ; 3
(3)
(4)
Some instability arises in saturation for small variations of RGB values in the neighbourhood of low or high intensity. Small changes in RGB coordinates cause great variations in S coordinate. To avoid this, we must linearly reduce the saturation as the intensity increases to white point or decreases to black point. This way, we change from a cylindrical geometrical representation to an exact double-cone shape. The saturation map is now more amenable to morphological treatment. A problem also arises with the hue in trying to form a complete lattice with HSI. Hue is angle-valued, if f(p) ¼ (fI(p), fH(p), fS(p)) are the colour values of the pixel p from a colour image in the HSI colour space, then fH(p)A[0,2p). In addition, one cannot order hues from their lowest to their highest values. To order hues, Hanbury [15] and Peters [20] use a hue-valued structuring function. Hues are ordered according to the absolute value of a distance function between the image hue and a reference hue (infimum): dðf H ðpÞ; H ref Þ ( jf H ðpÞ H ref j if jf H ðpÞ H ref jpp ¼ : 2p jf H ðpÞ H ref j if jf H ðpÞ H ref j4p
changed to I-S, H-I-S must be changed to I-S, etc. Empirically, we obtain good results with STh smaller than a 5% of maximum saturation.
(2)
where if B4R, then H ¼ 3601-H. Hue is normalized to the range [0,1] by letting H ¼ H/3601. The exact derivation of these equations, resulting from converting the RGB Cartesian coordinate system in range [0,1] to the HSI cylindrical one in range [0,1], is given by Ledley et al. [19]. The coordinates of saturation S and intensity I are calculated as follows: S ¼13
381
ð5Þ
With this definition, the value of the distance will fluctuate between 0 and 180 (3601/2). In a range [0,1] the distance fluctuate between 0 and 0.5. In this case, when the hue has no value, it is represented with a value of 1 [16]. Finally, from the definition of the complete lattice, we must ensure that all of the elements in the lattice can be ordered and compared [21]. There is a problem with the hue when the saturation is zero, this means that the hue signal cannot be used to order pixels in such cases. We can also define a saturation threshold STh, which determines which pixels have a significant hue and which have not. In the lattice, if there is a pixel p with fS(p)oSTh, the lexicographical order I-H-S must be
3. Morphological filters by reconstruction in colour images Mathematical morphology is a non-linear image processing which is based on the application of lattice theory to spatial structures [22]. In mathematical morphology, we can define the morphological filters by reconstruction, which are the basis of numerous transformations of higher level ranging, from simple procedures for removing all objects connected to the image border to more sophisticated image algorithms [6,23–25]. The morphological reconstruction is based on geodesic operations, which do not require to choose a structuring element or to set its size. The approach taken with geodesic transformations is to consider two input images (mask and marker). A morphological transformation is applied to the marker image and it is then forced to remain either above or below the mask image. The use of the filters by reconstruction in colour images requires an order relationship among the pixels of the image. For the vectorial morphological processing, the lexicographical ordering will be used. As such, the infimum (4n) and supremum (3n) will be vectorial operators, and they will select pixels according to their order olex in the HSI colour space. This is the same approach successfully used in the detection of colour cells in real-time medical imaging [26] and the Gaussian noise elimination in colour images [27]. Once the orders have been defined, the morphological operators of reconstruction for colour images can be generated and applied. An elementary geodesic operation is the geodesic dilation. Let g denote a marker colour image and f a mask colour image (if olex(g)polex(f), then g4nf ¼ g). The vectorial geodesic dilation of size 1 of the marker image g with respect to the mask f can be defined as ð1Þ dð1Þ vf ðgÞ ¼ dv ðgÞ^v f ;
(6)
where dð1Þ v ðgÞ is the vectorial dilation of size 1 of the marker image g. The vectorial propagation of the marker g is limited by the mask f by the vectorial infimum 4n. The vectorial geodesic dilation of size n of a marker colour image g with respect to a mask colour image f is obtained by performing n successive geodesic dilations of g with respect to f: ð1Þ ðn1Þ dðnÞ ðgÞ nf ðgÞ ¼ dnf ½dnf
with
dð0Þ nf ðgÞ
¼ f:
(7)
ARTICLE IN PRESS 382
F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
Geodesic transformations of images always converge after a finite number of iterations [16]. The propagation of the marker image is impeded by the mask image. Morphological reconstruction of a mask image is based on this principle. The vectorial reconstruction by dilation of a mask colour image f from a marker colour image g (both with Df ¼ Dg and olex ðgÞ olex ðf Þ) can be defined as Rvf ðgÞ ¼ dðnÞ vf ðgÞ;
Fig. 2. Vectorial reconstruction by dilation, lexicographical order IH-S: (a) colour mask, (b) colour marker imposition, (c) result of the vectorial reconstruction by dilation. False colours are not present in the result.
(8)
ðnþ1Þ where n is such that dðnÞ ðgÞ: The vectorial vf ðgÞ ¼ dvf reconstruction by dilation is a geodesic dilation iterated until stability. The vectorial reconstruction is an algebraic opening (increasing, anti-extensive and idempotent) only if all the operations between pixels respect the total order, in our case, the lexicographical order. Fig. 2 illustrates the morphological operation of reconstruction for colour signals. We can see the mask (Fig. 2a) and the colour marker (orange in Fig. 2b). The propagation of the marker is made according to its order in relation to the colour mask. This is a geodesic dilation iterated until stability (Fig. 2c). In our vectorial reconstruction there are no new or false colours, but only the colours of the marker or the mask.
Fig. 3. Algorithm for brightness elimination with minimum cost and without over-simplification.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
383
4. Brightness elimination in colour images Morphological filters by reconstruction can be used to simplify the colour images while preserving contours. This property makes them very attractive for segmentation. In this paper, we use the vectorial opening by reconstruction as a filter to reduce brightness in colour images. This is an interesting application for many areas in computer vision. For the morphological operations we employ a lexicographical order olex: I-H-S (lattice influenced by intensity). The reason for choosing H as the second signal, rather than S is explained in Section 4.1. The filter’s effect is the removal of any bright components that are smaller than the structuring element used in the vectorial erosion. In the results there is a very good preservation of the contour information. In the following section, we present a new method for brightness elimination only in certain areas of interest of the image. As such, we achieve a better control of the filter by reconstruction, avoiding the over-simplification or loss of information in the image. Furthermore, we reduce significantly the processing cost of the operation. 4.1. Algorithm for minimum-cost and without oversimplification One significant improvement in the reconstruction algorithms is the possibility of choosing the areas of the images that we really want to simplify. This way, we avoid the over-simplification effect (loss of information in significant areas of the image) and we reduce the processing time required for the reconstruction, which is longer due to the multiple iterations of the filter. In colour images, the hue Href can be used as a signal to determine the areas of interest of the image (objects with brightness). We propose the following method: 1. The objects in the image to be processed f(x,y) are marked through a hue distance threshold from Href. Empirically, we obtain good results with a maximum distance of the choosen marker equal of 0.1 in range [0,1], but this threshold can be modified according to the image. The result is the binary image h(x,y). 2. The binary mask h(x,y) is processed by a closingopening filter. Holes in the objects are closed and the opening eliminates the smaller objects, which are not to be considered. We thus obtain the binary mask i(x,y). 3. For the elimination of the brightness we use a vectorial filter of opening by reconstruction (VOR) which only functions in the areas defined by the mask i(x,y). In this filter, a pyramid of vectorial erosions is done with different structuring elements of increasing sizes. We use a lexicographical order olex: I-H-S, in which Href is the hue of the marker for the first
Fig. 4. Vectorial opening by reconstruction from eroded image. Erosions done with structuring elements of increasing sizes (1–8).
Fig. 5. Over-simplification of the image ‘‘Transformer’’ in the brightness elimination process.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
384
stage. As such, when there are two pixels with the same intensity, the hue value will be observed. The new filter can be defined as follows: 0
0
ðn Þ ðn Þ ðSÞ gvf ;i ¼ fdvf v ðf Þj8f ðx; yÞ ) iðx; yÞ ¼ 1g; ðn0 Þ ðSÞ dvf ðv ðf ÞÞ
(9)
erosion replaces highlight pixels (high olex) by the surroundings chromatic pixels (low olex). Next, the vectorial geodesic dilation (iterated until stability) reconstructs the colour image without the recovering of the specularities or brightness.
0 þ1Þ ðSÞ dðn ðv ðf ÞÞ: vf
where n’ is such that ¼ The vectorial erosion of the opening by reconstruction is done with a structural element of size s. This
In Fig. 3, the different stages of the algorithm proposed are shown together with the brightness
Fig. 6. Detection and elimination of brightness in the ‘‘Scissors’’ image: (a) original image, (b) mask-image, (c) morphological elimination of brightness causing over-simplification, (d) elimination of brightness without over-simplification.
Fig. 7. Detection and elimination of brightness in the ‘‘Key ring’’ image: (a) original image, (b) mask-image, (c) morphological elimination of brightness causing over-simplification, (d) elimination of brightness without over-simplification.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
elimination of the image ‘‘Transformer’’. In Fig. 4 we can see the progressive reduction of brightness in the area of interest of the image. This reduction has been obtained by means of a pyramid of vectorial erosions that have been carried out with eight structuring elements of sizes (s) between s ¼ 1 (3 3) and s ¼ 8 (17 17). The over-simplification of the image caused by a total geodesic reconstruction (Fig. 5) is achieved with this new algorithm.
5. Results We now present the new filter effect in different colours images that have brightness in comparison with standard morphological reconstruction (Figs. 6–8). In all the examples we present the original image, the maskimage, which indicates the area to be filtered, the vectorial colour opening by reconstruction of the original image and, finally, the new vectorial colour
385
opening by reconstruction which operates only in colour areas of the original image. In conventional connected filters, the size of the structuring element required for eliminating the brightness causes the loss of detail in the images (Figs. 6c, 7c and 8c). These features cannot be retrieved in the geodesic reconstruction. This inconvenient effect does not occur with our algorithm, since the erosion and reconstruction only function in those areas in which there are brightness (Figs. 6d, 7d and 8d). Furthermore, as the function of the connected filters is restricted to specific areas in the image, the results are obtained at a very lower processing cost. If the brightness appears in more than a chromatic area, we only change the colour marker and we apply the filter again to the image. The details of the algorithm’s parameters for the colour images processed are presented in Table 1. The size (s) of the structural element required in the vectorial erosion of VOR is different for each image. The classic vectorial geodesic
Fig. 8. Detection and elimination of brightness in the ‘‘Vase’’ image: (a) original image, (b) mask-image, (c) morphological elimination of brightness causing over-simplification, (d) elimination of brightness without over-simplification.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387
386 Table 1 Parameters of the new algorithm Colour image
Href (deg)
S
Scissors Key ring Vase Transformer
220 60 30 0
2 3 4 8
(5 5) (7 7) (9 9) (17 17)
References n
n’
Reduction of processing time (%)
67 43 78 97
27 19 33 59
49 62 29 51
dilation requires more iterations (n) (with greater processing time) than the new operation (n0 ). This time reduction is very significant (1 or 2 s versus a few minutes of entire reconstruction). In comparison with other non-morphological methods for brightness elimination, our morphological reconstruction does not require the previous knowledge of the scene or the assumption of a homogeneous illumination model. In addition, the brightness elimination is obtained independenly of the material of the objects on which the highlights appear, without any need of multiple-view. The results are also obtained in a shorter CPU time. This permits to achieve real-time requirements in image processing. As in the case of other methods, the only limitation to this technique is the reconstruction of highly textured areas.
6. Conclusions In this paper, we have presented the use of connected vectorial filters to eliminate brightness in colour images by means of an extension of the geodesic transformations of the mathematical morphology to colour images. We have shown that the use of the HSI colour space affords results that are quite acceptable in mathematical morphology. The inconveniences of this colour space in establishing a lattice have been overcome by using the hue distance concept and by treating its indefinition in the lexicographical order. Using the colour information of the hue signal we have been able to define a control of the filter by reconstruction, which avoids the effects of over-simplification with the subsequent loss of important areas of the image. In addition, the brightness elimination has been obtained with a very lower processing time, with respect to a classic geodesic reconstruction. Currently, we are working in brightness elimination by means of the detection of specular reflectance of the image. This would allow a more accurate selection of the processing area, independently of the hue of the objects.
[1] Lin S, Li Y, Kang S, Tong X, Shum H. Diffuse-specular separation and depth recovery from image sequences. Lecture Notes in Computer Science 2002. [2] Bajcsy R, Lee S, Leonardis A. Detection of diffuse and specular interface reflections and inter-reflections by color image segmentation. International Journal on Computer Vision 1996. [3] Wolff L. Using polarization to separate refelction components. In: Proceedings of the IEEE computer vision and pattern recognition, 1989. p. 363–9. [4] Nayar A, Fang X, Boult T. Removal of specularities using color and polarization. In: Proceedings of the IEEE computer vision and pattern recognition, 1993. p. 583–90. [5] Crespo J, Serra J, Schafer R. Theoretical aspects of morphological filters by reconstruction. Signal Processing 1995. [6] Vicent L. Morphological grayscale reconstruction in image analysis: applications and efficient algoritms. IEEE Transactions on Image Processing 1993;2(2):176–201. [7] Salembier P, Serra J. Flat Zones Filtering, Connected Operators, and Filters by Reconstruction. IEEE Transactions on Image Processing 1995;4(8):1153–60. [8] Palus H. Representations of colour images in different colour spaces. In: Sangwine S, Horne R, editors. The colour image processing handbook. London: Chapman & Hall; 1998. p. 67–90. [9] Plataniotis K, Venetsanopoulos A. Color image processing and applications. Berlin: Springer; 2000. [10] Sharma G, Yrzel M, Trussel H. Digital color processing. IEEE Transaction on Image Processing 1997;6(7):901–32. [11] Smith A. Color gamut transform pairs. Computer Graphics 1978;12(3):12–9. [12] Foley J, vanDam A, Feiner S, Hughes J. Fundamentals of interactive computer graphics. Addison Wesley: Reading, MA; 1990. [13] Comer M, Delp E. Morphological operations for colour image processing. Journal of Electronic Imaging 1999;8(3): 279–89. [14] Ortiz F, Torres F, Angulo J, Puente S. Comparative study of vectorial morphological operations in different colour spaces. In: Proceedings of SPIE, intelligent robots and computer vision XX, vol. 4572, Boston, USA, 2001. p. 259–68. [15] Hanbury A, Serra J. Morphological operators on the unit circle. IEEE Transactions on Image Processing 2001;10(12): 1842–50. [16] Ortiz F. Procesamiento morfolo´gico de ima´genes en color. Aplicacio´n a la reconstruccio´n geode´sica. Ph.D. thesis, University of Alicante, 2002. [17] Chanussot J. Approches Vectorielles ou Marginales pour le Traitement d’Images Multicomposantes. Ph.D. thesis, University of Savoie, 1998. [18] Tenenbaum J, Garvey T, Welwy S, Wolf H. An interactive facility for scene analysis research. Technical Report 87, Stanford Research Institute, AI Centre, 1974. [19] Ledley S, Buas M, Golab T. Fundamentals of true-color image processing. In: Proceedings of 10th international conference on pattern recognition, Atlantic City, 1990. p. 791–5. [20] Peters II A. Mathematical morphology for angle-valued images. In: Proceedings of SPIE, non-linear image processing VIII, vol. 3026, 1997. p. 84–94. [21] Serra J. Anamorphoses and function lattices (multivalued morphology). In: Dougherty E, editor. Mathematical Morphology in Image Processing, 1993, p. 483–521. [22] Serra J. Image analysis and mathematical morphology, vol. I, and image analysis and mathematical morphology, vol. II: theoretical advances. New york: Academic Press; 1982, 1988.
ARTICLE IN PRESS F. Ortiz, F. Torres / Real-Time Imaging 10 (2004) 379–387 [23] Beucher S. Segmentation d’Images et Morphologie Mathe´matique. Ph.D. thesis, Ecole des Mines de Paris, 1990. [24] Serra J, Salembier P. Connected operators and pyramides. In: Proceedings of the SPIE, non-linear algebra and morphological image processing, vol. 2030, 1993. p. 65–75. [25] Soille P. Morphological image analysis. Principles and applications. Berlin: Springer; 1999.
387
[26] Ortiz F, Torres F, De Juan E, Cuenca N. Colour mathematical morphology for neural image analysis. Journal of Real-Time Imaging 2002;8(6):455–65. [27] Ortiz F, Torres F, Gil P. Gaussian noise elimination in colour images by vector-connected filters. In: Proceedings of the IEEE 17th International Conference on Pattern Recognition, vol. 4, 2004. p. 807–11.