- Email: [email protected]
WaterBalloons: A hybrid watershed Balloon Snake segmentation
Available online at www.sciencedirect.com
Image and Vision Computing 26 (2008) 905–912 www.elsevier.com/locate/imavis
WaterBalloons: A hybrid watershed Balloon Snake segmentation Issam Dagher *, Kamal El Tom University of Balamand, Department of Computer Engineering, P.O. Box 100, Elkoura, Lebanon Received 7 March 2006; received in revised form 3 September 2007; accepted 28 October 2007
Abstract In this paper, a new image segmentation technique called WaterBalloons is introduced. It combines both watershed segmentation and the active contour model known as Balloon Snake. The watershed transform has a major problem of over-segmentation. Solutions like region merging, use of markers, use of multi-scales have been proposed. These approaches led to other problems such as under-segmentation. The Balloon Snake in an innovative approach that detects salient objects in an image. But in general snakes are very sensitive to initialization and need user interactions and a priori knowledge of objects to segment. WaterBalloons provide the advantage of reducing watershed over-segmentation problems while preventing under-segmentation and ensure automatic initialization of traditional snakes. In addition, a method for parameter optimization of the proposed hybrid snake is introduced based on energy transitions tracking. 2007 Elsevier B.V. All rights reserved. Keywords: Watershed; Active contour model; Snake balloon; Segmentation; WaterBalloons
1. Introduction Image segmentation is the process of distinguishing objects of interest from a given image. Over-segmentation is a major issue in the watershed transform. Initialization and finding the best parameters are also key issues in the active contour models. The bubbles algorithm [1] allows the contours to be initialized at the image minima but requires fine tuned parameters. The seeded region growing method [2] depends greatly on the initial seeds. The front propagation method [3] depends on good initialization. The CBMA method [4] is a good solution to over-segmentation but it requires as well fine tuned parameters. This paper proposes a new segmentation technique by combining the watershed algorithm and the active contour model.
2. The watershed transform The watershed transform is the basic segmentation tool in mathematical morphology [5,6]. The watershed concept *
Corresponding author. Tel.: +961 6930250; fax: +961 6930278. E-mail address: [email protected] (I. Dagher).
0262-8856/$ - see front matter 2007 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2007.10.010
comes from the field of topography: In a topographic surface, the watersheds are the lines dividing two catchment basins. A gray-scale gradient image is considered as a topographic surface where the value of each pixel represents the elevation or altitude at that point. Thus, high-gradient image edges represent watersheds while low-gradient regions correspond to catchment basins. Watershed regions or catchment basins of the ‘topographic’ image are homogeneous meaning that the corresponding pixels are connected with a path of monotonically decreasing altitude. Each catchment basin is represented by the set of pixels that will drain to the same local minimum. A watershed line will separate each pair of basins, and the final union of all the lines will define the watershed segmentation. If we imagine a hole in each local minimum and immerse the topographic surface in water, the water starts filling all the catchment basins progressively. There will be an instant at which water coming from two different minima would meet and merge the two basins. To avoid that different basins merge, a dam is built at each point of contact. Following the flooding or immersion process, the union of all those dams constitutes the watersheds. The watershed transform denotes the assignment of all points in a distinct
906
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
catchment basin with the same label, while a special label is assigned to all point of the watershed. Efficient algorithms have been proposed to simulate the immersion process [7,8]. In [9] the notions of basins and contours dynamics are introduced, which require interactive dynamics thresholding. Multi-scale analysis [10] of intensity minima in the gradient magnitude image provides a mechanism for imposing a scale-based hierarchy on the watersheds regions, which can be used to label watershed boundaries according to their scale. The disadvantage of this method is that severe Gaussian blurring through scale space results in dislocated or perturbed edges. The advantage of the watershed transform is that it produces closed and adjacent contours including all image edges. Often the watershed produces a severe over-segmentation, but the crucial point is that all important object boundaries are included, and the task is reduced to eliminating the undesired ones. Some solutions of the over-segmentation are: 2.1. Region merging A simple region merging method [21] begins with the assumption that each pixel is a separate image region or belongs to a pre-segmented region. Adjacent regions are then merged based on the region’s statistics, such as the mean [11]. Fast implementations of this model group pixels by tracing ‘‘waterfalls’’ to their corresponding minima and then merge the resulting regions based on the depth or volume of the resulting pool [12]. Moreover, some ‘‘visually heterogeneous regions’’ may be ‘‘statistically homogeneous regions’’. To overcome this problem, careful intervention from the user or explicit prior knowledge on the image structure is necessary for region merging. 2.2. Multi-scale hierarchical segmentation One way to avoid the segments that result from small variations in the gradient image is to low-pass filter (blur) the image in order to remove these gradients. However, low-pass filtering will also decrease the precision. In multi-scale hierarchy [10,22], as the Gaussian scale sigma increase, the watersheds change both number and position. This means that objects constructed by merging high scale segments, will have borders that are distorted compared to the low scale borders, and this might cause problems when shape information is used to control the merging of segments. Since segments are linked over scale this is handled by replacing all high scale segments by the union of its low scale ‘‘children’’ segments linked to it [10].
has to be surrounded by pixels with higher altitude. These markers are then made to be the only allowed minima in the watershed algorithm. The watershed lines created here are defined as external markers. The external markers efficiently partition the image into regions containing one single internal marker and some background. The problem is then reduced to partitioning each of these regions into two, which can be done by some simpler segmentation algorithm. 3. Snakes In 1987, Kass et al., introduced the Active Contour Models commonly known as Snakes [13]. Active contour is defined as an energy minimizing spline. Its energy depends on its shape and location within the image. A snake can be considered as a number of control points or snaxels that are linked together and free to deform under the constraining forces. The control points v(s)=[x(s), y(s)] are traditionally placed near the edges of interest because of the poor capture range of a snake. Snake deformation is carried by minimization of an energy function so that the contour will move from an initial position until the energy will stabilize at significant edges. Furthermore, a snake works better when all the points are equally spaced and not too far apart. This is to ensure a better performance in edge tracing. Therefore, an interpolation mechanism is needed to set the appropriate limitations on the distances between control points. If points are too close or too far apart, the mechanism must delete or add points respectively in order to have a good initial position. At each step of deformation, this process of checking the initial coordinates should be done to ensure better performance of a snake. The total snake energy is defined as [13]: Z h i 1 Esnake ¼ aðsÞjvs j2 þ bðsÞjvss j2 þ cEext ðvðsÞÞds ð1Þ 2 s Finding a curve for which E has a stationary value is a problem of variational calculus and one can apply the fundamental Euler–Lagrange differential equation giving the following equation: aðsÞvss bðsÞvssss rEext ¼ 0
where the third term is normalized by the external energy weight c. Solution of the above equation gives us our final contour minimizing Esnake. One way to interpret this equation is to consider it to be a force balance equation of a system: F int þ F ext ¼ 0
2.3. Marker-controlled watershed segmentation A marker is a connected component belonging to an image. Internal markers associate with objects of interest, and external markers associate with the background. The criteria for an internal marker can for example be that it
ð2Þ
ð3Þ
where F int ¼ aðsÞvss bðsÞvssss
ð4Þ
and F ext ¼ rEext
ð5Þ
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
The final solution [13] is: 1
vt ¼ ½A þ sI ½svt1 Fðvt1 Þ
ð6Þ
where s is the time step. Splitting back vinto its x and y components: 1
ð7Þ
1
ð8Þ
xt ¼ ½A þ sI ½sxt1 FX ðxt1 ; yt1 Þ yt ¼ ½A þ sI ½syt1 FY ðxt1 ; yt1 Þ
FX(xt1, yt1) is the gradient force in x direction on point (xt1, yt1) of the image. It is retrieved from the gradient image data when snake moves on to this point. Similarly, every point on the snake will follow this process and moves to its new position. This is the complete minimizing process for snake deformation. Some of the problems faced by the Snakes are: 3.1. Initialization A snake is traditionally initialized by hand (i.e. placing the initial positions of control points) and not automatically done by computer. The control points need to be first positioned near the edges of interest due to the small capture range of image gradient. This is easily done by humans but is not an easy task for a system to intuitively realize the object of interest. However, if there is enough computer power, one can always execute many snakes all together and let it randomly pick up all the edges in an image and then select the required ones. 3.2. Generating snake parameters Producing a suitable set of parameters for snake initialization is another difficulty. Parameters/weights of energies totally control the performance of snake deformation. Choosing a set of parameters (a, b, c) that can be used for a particular object in an image is a difficult task for a computer. However, after initializing a snake, it is difficult to have a system that automatically changes the parameters with respect to time using data from the image. 3.3. Capture range If an object in the image has an edge which shows a concavity then the traditional snake will have a problem in tracing the edge at this concaved part. The reason is that the capture range at these parts of edges is too far from the snake (Fig. 1). A poor solution is to increase the ‘r’ value of the Gaussian blurring filter, thus increasing the snake capture range, but the edge of the object will be also blurred and cannot be traced accurately. A method called
907
‘Gradient Vector Flow’ (GVF) is lately proposed to overcome this problem. This method makes use of the amplification of image gradient to increase the snake capture range and deformation speed but should contain the medial axis of the object [14]. Another method to solve this problem is the Balloon Snake that will inflate or deflate the snake from its farthest positions to the final edge locations thus increasing the snake capture range. Previous combinations of watershed and snakes have not solved the over-segmentation problems. In [15], a snake zone is defined around the object boundaries where the corresponding watershed points are candidates for the true object boundary. Energy minimization or the searching process is carried out using dynamic programming. However, in most applications the difficulty is mainly in the definition (localization) of the snake zone. The watershed-based segmentation on a down-sampled image is used to get the initial contours for the snake [16]. This will guaranty automatic initialization of snakes using watershed and smoothness of watershed lines using active contours. A new energy functional is derived based on the topographic distance definition of the watershed line [17]. This segmentation will incorporate control over the smoothness of the watershed lines.
4. Balloon Snakes Cohen [18] introduced a model that makes the curve behaves like a balloon with an additional external force which moves the contour out (inflation) or in (deflation) along its normal thus eliminating the need for the initial curve to be close to the correct edges. By placing balloon snake inside/outside the object of interest and inflate/ deflate contour to its boundary, this new model is very useful for some object that is difficult to locate or too complicate to trace by original model. The Balloon Snake passes over relatively weak edges and it stops on salient edges [18,19]. In Balloon model, the original snake internal force is kept the same while a new external ‘pressure’ force term is added to the snake force function. The total Balloon Snake force is now: Fsnake ¼ ðFint þ Fext Þ þ Fpressure
ð9Þ
The external force is: Fext ¼ rEext ðnegative of the gradientÞ
Fext
¼ Gr ðx; yÞ f ðx; yÞðconvolutionÞ rEimage ¼ k 2 krEimage k
ð10Þ
The pressure force added is given by: Fig. 1. Low capture range of traditional snake.
Fpressure ¼ k 1 nðsÞ
ð11Þ
908
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
where n(s) is the normal unit vector to the curve at point v(s), and k1 is the amplitude of the force, it indicates inflation (when positive) or deflation (when negative). At the same time, the value of k1 has an influence on the speed of deformation. Fsnake ¼ ðFint þ Fext Þ þ Fpressure rEimage þ k 1 nðsÞ Fsnake ¼ Fint k 2 krEimage k
ð12Þ
The Fsnake is the force equation that needs to be minimized in Balloon Snake deformation. Due to the newly added pressure force, balloon model is able to overcome the over shrinking problem in the original snake. Also, if the object has a troublesome shape, placing a balloon inside the image and expanding its contour will solve matching the desired shape. The final solution is: vt ¼ ½A þ sI
1
½svt1 Fðvt1 Þ þ Fpressure
ð13Þ
5. WaterBalloon snake 5.1. The hybrid technique Since watershed regions are closed and disjoint regions whose unions constitute the whole image and since traditional snakes need to be initialized near the target edges, the watershed lines around each region can be considered as initial snakes. Thus, watershed regions are involved in a multiple snakes approach. The idea of this method is that the appropriate internal and external forces should drive those snakes in order to be dragged by the significant edges.
Traditional snakes could not be initialized far away from the target edges. GVF snake should contain the medial axis of the object therefore they cannot drive watershed lines. On the other hand, Balloon Snakes can be inflated or deflated until they are attracted by the object edges. Balloons are free to be initialized anywhere inside object due to the powerful external pressure energy term that will drive the snake to the correct location. Since watershed regions are over-segmented version of the object, a Balloon Snake initialized at the location of a watershed region contour will be a strong candidate for capturing the salient object edges. Watershed regions cannot cross the object edges, so Balloon Snakes will not cause any problem. They also do not have to include the object medial axis, which ensure free initialization. Fig. 2 shows the block diagram of the proposed hybrid segmentation technique. The original image is low-pass filtered or blurred to remove small noisy regions. Then, the watershed transform is applied on the gradient image to segment watershed regions. Now, contour-tracing techniques can be applied to extract watershed region contours. Those contours will constitute the initial Balloon Snakes that will deform to capture the object edges. Balloon Snakes have a large capture range so correct snake deformation can be guarantied even if the initial snake (or the contour of the watershed region) is far away from the target edges. This situation is frequent for a given candidate watershed region, so Balloon Snake is most suitable to drive the watershed contours. Once a snake captures one object, the image is relabeled so that the next snake will not be a watershed contour inside the captured object. For each snake, parameter optimization is required to ensure opti-
Fig. 2. Block diagram of WaterBalloon segmentation.
Fig. 3. Proposed segmentation strategy. (a) Initial watershed regions and lines (over-segmented image). (b) Deformation of initial watershed lines with Balloon Snakes. (c) Final Contours with the two blood cells separated.
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912 -800 -1000
-1200 -1400
Energy
mum edge localization. For example in Fig. 3 (a), the first cell will be initialized with two labels. The first snake in region labeled 1 will capture the whole region (label 1 + label 2). So the first cell will be re-labeled by only one label and only one snake is needed. This process is repeated for each snake to obtain the WaterBalloon segmentation. Illustration of the technique on the two touching blood cells is shown in Fig. 3 with a low-resolution image.
909
-1600 -1800 -2000
-2200
5.2. Optimization of snake parameters based on energy tracking
-2400 -2600
• There exist an energy transition to a high energy state and then a drop to a lower energy state. • The drop to the lower energy state occurs simultaneously in both the external and total energy terms. This is not by chance; as the edge will deform from its initial position at the weak edge to the strongest edge, it traverses a homogeneous region where gradient field is weak and no edge is encountered. In this homogeneous region, the external energy term is maximal (in absolute sense) and thus the total energy term is maximal. This explains the transition to a high-energy state in both the total and external energy curves. One can also note that the final energy state is lower than the initial one since the snake is minimizing its total energy. This behavior will be the typical for the proposed WarterBalloon technique: starting from a watershed line as an initial Balloon Snake, the deformation from the weak edge to the strong edge through the homogeneous region will always induce this energy pulse. Fig. 5 corresponds to the case where the balloon is inflated outside the target boundaries. In this case, the total energy increases considerably. Whenever this case is encountered, the parameters need to be changed and the pressure weight should be decreased. This parameters updating is continued until the energy transitions follow
-2800 0
50
100
150
200
250
200
250
iteration 0 -200
Energy
-400
-600 -800
-1000 -1200
-1400 0
50
100
150
iteration Fig. 4. Energy plots of WaterBalloon snakes. (a) External energy term behavior. (b) Total energy term behavior.
the case of Fig. 4. The optimum values obtained are then fed back to deform the snake. 3000 2500 2000 1500
Energy
A snake may not be located at the target edges due to its sensitivity to the snake parameters. Thus there is a need to parameter optimization that best localize the correct edges. The lack of knowledge about those edges is what makes optimization of the snake parameters a very hard problem. Thus a new approach for determining those parameters is proposed that is derived from the energy functional behavior of WaterBalloons. The total energy term is shown in Fig. 4(b), the energy behavior seems not to provide any information about the snake performance. The challenge was to find a relationship between this energy behavior and the optimum parameters. The key solution was to look at the total energy term and the corresponding elasticity, bending, external and pressure (Balloon) energy terms. The idea is clear in Fig. 4(a) and (b) where two conclusions are drawn:
1000 500 0 -500 -1000 -1500 0
5
10
15
20
25
30
35
40
45
iteration Fig. 5. Total energy term behavior when Balloon inflates outside object.
910
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
Fig. 6. Block diagram of optimization with energy transitions tracking.
The block diagram of the optimization stage is shown in Fig. 6 where the parameters are initialized equally to give equal importance to all energy terms. From the above discussions, a criterion can be tested to check if there is an energy pulse with transition to a high energy state and a back transition to a lower energy state or there is only a transition to a high energy state in case of a continuously inflating balloon. This criterion is based on the initial, final, and maximum values of the energy. if Efinal Einitial < T; Emax then there is an energy pulse and the Balloon Snake will stabilize (T = 0.5 is satisfactory for this application). Else there is only transition to high energies with inflating Balloon Snake.
tion until an object is captured. Next, the image is relabeled to ensure that the second watershed region will not fall in the already captured object. Once a watershed region deformed with the Balloon Snake captures a single cell part of two or more touching cells, the following deformations will separate the remaining cells (Fig. 7c and d). WaterBalloon snake was able to separate the touching objects. It should be noted that the arrow in Fig. 7(b) shows a cell which have been over-segmented by the watershed transform (seven regions). A contour extraction algorithm is applied in order to get the contours of all the seven regions (seven labels). Seven Snakes will be initialized. Starting with the first snake, its deformation will fill the first blood cell (the arrow in 7(c). The image is relabeled and there is no need for the other 6 snakes to be deformed. 6.2. Application to bovine endothelial corneal cells
6. Experimental results 6.1. Application to blood cell image Fig. 7(a) shows red blood cells image with two touching pair of cells. In traditional segmentation methods, touching objects (the blood cells) are considered as one object because these methods are insensitive to the low intensity levels of the border pixels between these objects. In Fig. 7(b), watershed transform was applied to the gradient image. In a sequential manner, a watershed region is picked up and will be subject to Balloon Snake deforma-
Fig. 8(a) shows the image of the bovine endothelial corneal cells acquired through a CCD camera attached to a microscope [20]. The arrows show an example of four cells. The objective is to segment all the 36 cells. Fig. 8(b), (c), (d), (e) shows, respectively, the convergence of the bubbles (with bubbles initialized at image minima), the front propagation, the seeded region growing (36 seeds initialized by hand), and the CBMA algorithms (fine-tuned threshold parameters). Fig. 8(f) shows the convergence of the WaterBalloons algorithm. It is clear that the touching cells have been well separated. The number of falsely segmented
Fig. 7. WaterBalloon segmentation. (a) Original image. (b) Watershed regions. (c) Balloon Snake deformation. (d) WaterBalloon segmentation with the touching cells separated.
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
911
Fig. 8. (a) Original image. (b) Bubbles algorithm. (c) Front propagation algorithm. (d) Seeded region growing algorithm. (e) CBMA algorithm. (f) WaterBalloons algorithm.
regions for the bubbles algorithm is 4, for the front propagation is 2, for the seeded region growing is 1, for the CBMA is 6, and for the WaterBalloons is zero. It should be noted that some shift of the contour from its real position have occurred in the WaterBalloons algorithm due to the blurring filter that initialized the process.
ent objects in an image. Waterballoons provide the advantage of reducing over-segmentation of Watershed while preventing under-segmentation problems. Touching objects are separated, snake parameters are optimized based on energy transitions tracking, and optimum edge localization is achieved.
7. Conclusion
References
In this paper, a hybrid Watershed Balloon Snake technique for image segmentation is introduced. The new technique combines the use of Watershed transform and Balloon Snakes in an innovative approach that detects sali-
[1] H. Tek et al., Volumetric segmentation of medical images by three dimensional bubbles, Computer Vision and Machine Understanding 65 (2) (1997) 246–258. [2] R. Adams et al., Seeded region growing, IEEE Transactions on PAMI 16 (6) (1994) 641–647.
912
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
[3] R. Malladi et al., Shape modeling with front propagating: a level set approach, IEEE Transactions on PAMI 17 (2) (1995) 158–175. [4] M.C. Andrade et al., Segmentation of microscopic images by flooding simulation: a catchment basins merging algorithm, in: Proceedings of SPIE Non linear Image processing VIII, vol. 3026, San Jose, USA, 1997, pp. 164–175. [5] R.C. Gonzales, R.E. Woods, Digital Image Processing, third ed., Prentice Hall, 2002, ISBN: 0-201-18075. [6] J. Serra, Image Analysis and Mathematical Morphology, Academic Press, London, 1982. [7] L. Vincent, P. Soille, Watershed in digital spaces: an efficient algorithm based on immersion simulations, IEEE Transactions on PAMI 13 (6) (1991) 583–598. [8] S. Beucher, F. Meyer, The morphological approach to segmentation: the watershed transformation, in: E.R. Dougherty (Ed.), Mathematical Morphology in Image Processing, Marcel Dekker, New York, 1993, pp. 433–481. [9] L. Najman, M. Schmitt, Geodesic saliency of watershed contours and hierarchical segmentation, IEEE Transactions on PAMI 18 (12) (1996) 1163–1173. [10] J.M. Gauch, Image segmentation and analysis via multiscale gradient watershed hierarchies, IEEE Transactions on Image Processing 8 (1999) 69–79. [11] A. Bleau, L.J. Leon, Watershed-based segmentation and region merging, Computer Vision and Image Understanding 77 (2000) 317–370. [12] K. Haris, S.N. Efstratiadis, N. Maglaveras, A.K. Katsaggelos, Hybrid image segmentation using watersheds and fast region merging, IEEE Transactions on Image Processing 7 (12) (1998) 1684–1699.
[13] M. Kass, A. Witkin, D. Terzopoulos, Snakes: active contour models, International Journal of Computer Vision 1 (1987) 321–331. [14] R. Nilanjan, S.T. Acton, T. Altes, E. de Lange, J. Brookeman, Merging parametric active contours within homogeneous image regions for MRI-based lung segmentation, IEEE Transactions on Medical Imaging 22 (2) (2003). [15] J. Park, J.M. Keller, Snakes on the watershed, IEEE Transactions on PAMI 23 (10) (2001). [16] V. Kiran, P.K. Bora, Watersnake: integrating the watershed and the active contour algorithms, IEEE (2003). [17] H.T. Nguyen, M. Worring, R. van den Boomgaard, Watersnakes: energy-driven watershed segmentation, IEEE Transactions on PAMI 25 (3) (2003). [18] L.D. Cohen, On active contour models and balloons, Computer Vision Graphics and Image Processing: Image Understanding 53 (2) (1991) 211–218. [19] L.D. Cohen, I. Cohen, Finite-element methods for active contour models and balloons for 2-D and 3-D images, IEEE Transactions on PAMI 15 (11) (1993) 1131–1147. [20] M.C. Andrade, An interactive algorithm for image smoothing and segmentation, Electronic Letters Computer Vision and Image Analysis 4 (1) (2004) 32–48. [21] Kostas Haris, Serafim N. Efstratiadis, Nikolaos Maglaveras, K. Aggelos, Katsaggelos: hybrid image segmentation using watersheds and fast region merging, IEEE Transactions on Image Processing 7 (12) (1998) 1684–1699. [22] I. Vanhamel, I. Pratikakis, H. Sahli, Multiscale gradient watersheds of color images, IEEE Transactions on Image Processing 12 (6) (2003) 617–626.
Image and Vision Computing 26 (2008) 905–912 www.elsevier.com/locate/imavis
WaterBalloons: A hybrid watershed Balloon Snake segmentation Issam Dagher *, Kamal El Tom University of Balamand, Department of Computer Engineering, P.O. Box 100, Elkoura, Lebanon Received 7 March 2006; received in revised form 3 September 2007; accepted 28 October 2007
Abstract In this paper, a new image segmentation technique called WaterBalloons is introduced. It combines both watershed segmentation and the active contour model known as Balloon Snake. The watershed transform has a major problem of over-segmentation. Solutions like region merging, use of markers, use of multi-scales have been proposed. These approaches led to other problems such as under-segmentation. The Balloon Snake in an innovative approach that detects salient objects in an image. But in general snakes are very sensitive to initialization and need user interactions and a priori knowledge of objects to segment. WaterBalloons provide the advantage of reducing watershed over-segmentation problems while preventing under-segmentation and ensure automatic initialization of traditional snakes. In addition, a method for parameter optimization of the proposed hybrid snake is introduced based on energy transitions tracking. 2007 Elsevier B.V. All rights reserved. Keywords: Watershed; Active contour model; Snake balloon; Segmentation; WaterBalloons
1. Introduction Image segmentation is the process of distinguishing objects of interest from a given image. Over-segmentation is a major issue in the watershed transform. Initialization and finding the best parameters are also key issues in the active contour models. The bubbles algorithm [1] allows the contours to be initialized at the image minima but requires fine tuned parameters. The seeded region growing method [2] depends greatly on the initial seeds. The front propagation method [3] depends on good initialization. The CBMA method [4] is a good solution to over-segmentation but it requires as well fine tuned parameters. This paper proposes a new segmentation technique by combining the watershed algorithm and the active contour model.
2. The watershed transform The watershed transform is the basic segmentation tool in mathematical morphology [5,6]. The watershed concept *
Corresponding author. Tel.: +961 6930250; fax: +961 6930278. E-mail address: [email protected] (I. Dagher).
0262-8856/$ - see front matter 2007 Elsevier B.V. All rights reserved. doi:10.1016/j.imavis.2007.10.010
comes from the field of topography: In a topographic surface, the watersheds are the lines dividing two catchment basins. A gray-scale gradient image is considered as a topographic surface where the value of each pixel represents the elevation or altitude at that point. Thus, high-gradient image edges represent watersheds while low-gradient regions correspond to catchment basins. Watershed regions or catchment basins of the ‘topographic’ image are homogeneous meaning that the corresponding pixels are connected with a path of monotonically decreasing altitude. Each catchment basin is represented by the set of pixels that will drain to the same local minimum. A watershed line will separate each pair of basins, and the final union of all the lines will define the watershed segmentation. If we imagine a hole in each local minimum and immerse the topographic surface in water, the water starts filling all the catchment basins progressively. There will be an instant at which water coming from two different minima would meet and merge the two basins. To avoid that different basins merge, a dam is built at each point of contact. Following the flooding or immersion process, the union of all those dams constitutes the watersheds. The watershed transform denotes the assignment of all points in a distinct
906
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
catchment basin with the same label, while a special label is assigned to all point of the watershed. Efficient algorithms have been proposed to simulate the immersion process [7,8]. In [9] the notions of basins and contours dynamics are introduced, which require interactive dynamics thresholding. Multi-scale analysis [10] of intensity minima in the gradient magnitude image provides a mechanism for imposing a scale-based hierarchy on the watersheds regions, which can be used to label watershed boundaries according to their scale. The disadvantage of this method is that severe Gaussian blurring through scale space results in dislocated or perturbed edges. The advantage of the watershed transform is that it produces closed and adjacent contours including all image edges. Often the watershed produces a severe over-segmentation, but the crucial point is that all important object boundaries are included, and the task is reduced to eliminating the undesired ones. Some solutions of the over-segmentation are: 2.1. Region merging A simple region merging method [21] begins with the assumption that each pixel is a separate image region or belongs to a pre-segmented region. Adjacent regions are then merged based on the region’s statistics, such as the mean [11]. Fast implementations of this model group pixels by tracing ‘‘waterfalls’’ to their corresponding minima and then merge the resulting regions based on the depth or volume of the resulting pool [12]. Moreover, some ‘‘visually heterogeneous regions’’ may be ‘‘statistically homogeneous regions’’. To overcome this problem, careful intervention from the user or explicit prior knowledge on the image structure is necessary for region merging. 2.2. Multi-scale hierarchical segmentation One way to avoid the segments that result from small variations in the gradient image is to low-pass filter (blur) the image in order to remove these gradients. However, low-pass filtering will also decrease the precision. In multi-scale hierarchy [10,22], as the Gaussian scale sigma increase, the watersheds change both number and position. This means that objects constructed by merging high scale segments, will have borders that are distorted compared to the low scale borders, and this might cause problems when shape information is used to control the merging of segments. Since segments are linked over scale this is handled by replacing all high scale segments by the union of its low scale ‘‘children’’ segments linked to it [10].
has to be surrounded by pixels with higher altitude. These markers are then made to be the only allowed minima in the watershed algorithm. The watershed lines created here are defined as external markers. The external markers efficiently partition the image into regions containing one single internal marker and some background. The problem is then reduced to partitioning each of these regions into two, which can be done by some simpler segmentation algorithm. 3. Snakes In 1987, Kass et al., introduced the Active Contour Models commonly known as Snakes [13]. Active contour is defined as an energy minimizing spline. Its energy depends on its shape and location within the image. A snake can be considered as a number of control points or snaxels that are linked together and free to deform under the constraining forces. The control points v(s)=[x(s), y(s)] are traditionally placed near the edges of interest because of the poor capture range of a snake. Snake deformation is carried by minimization of an energy function so that the contour will move from an initial position until the energy will stabilize at significant edges. Furthermore, a snake works better when all the points are equally spaced and not too far apart. This is to ensure a better performance in edge tracing. Therefore, an interpolation mechanism is needed to set the appropriate limitations on the distances between control points. If points are too close or too far apart, the mechanism must delete or add points respectively in order to have a good initial position. At each step of deformation, this process of checking the initial coordinates should be done to ensure better performance of a snake. The total snake energy is defined as [13]: Z h i 1 Esnake ¼ aðsÞjvs j2 þ bðsÞjvss j2 þ cEext ðvðsÞÞds ð1Þ 2 s Finding a curve for which E has a stationary value is a problem of variational calculus and one can apply the fundamental Euler–Lagrange differential equation giving the following equation: aðsÞvss bðsÞvssss rEext ¼ 0
where the third term is normalized by the external energy weight c. Solution of the above equation gives us our final contour minimizing Esnake. One way to interpret this equation is to consider it to be a force balance equation of a system: F int þ F ext ¼ 0
2.3. Marker-controlled watershed segmentation A marker is a connected component belonging to an image. Internal markers associate with objects of interest, and external markers associate with the background. The criteria for an internal marker can for example be that it
ð2Þ
ð3Þ
where F int ¼ aðsÞvss bðsÞvssss
ð4Þ
and F ext ¼ rEext
ð5Þ
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
The final solution [13] is: 1
vt ¼ ½A þ sI ½svt1 Fðvt1 Þ
ð6Þ
where s is the time step. Splitting back vinto its x and y components: 1
ð7Þ
1
ð8Þ
xt ¼ ½A þ sI ½sxt1 FX ðxt1 ; yt1 Þ yt ¼ ½A þ sI ½syt1 FY ðxt1 ; yt1 Þ
FX(xt1, yt1) is the gradient force in x direction on point (xt1, yt1) of the image. It is retrieved from the gradient image data when snake moves on to this point. Similarly, every point on the snake will follow this process and moves to its new position. This is the complete minimizing process for snake deformation. Some of the problems faced by the Snakes are: 3.1. Initialization A snake is traditionally initialized by hand (i.e. placing the initial positions of control points) and not automatically done by computer. The control points need to be first positioned near the edges of interest due to the small capture range of image gradient. This is easily done by humans but is not an easy task for a system to intuitively realize the object of interest. However, if there is enough computer power, one can always execute many snakes all together and let it randomly pick up all the edges in an image and then select the required ones. 3.2. Generating snake parameters Producing a suitable set of parameters for snake initialization is another difficulty. Parameters/weights of energies totally control the performance of snake deformation. Choosing a set of parameters (a, b, c) that can be used for a particular object in an image is a difficult task for a computer. However, after initializing a snake, it is difficult to have a system that automatically changes the parameters with respect to time using data from the image. 3.3. Capture range If an object in the image has an edge which shows a concavity then the traditional snake will have a problem in tracing the edge at this concaved part. The reason is that the capture range at these parts of edges is too far from the snake (Fig. 1). A poor solution is to increase the ‘r’ value of the Gaussian blurring filter, thus increasing the snake capture range, but the edge of the object will be also blurred and cannot be traced accurately. A method called
907
‘Gradient Vector Flow’ (GVF) is lately proposed to overcome this problem. This method makes use of the amplification of image gradient to increase the snake capture range and deformation speed but should contain the medial axis of the object [14]. Another method to solve this problem is the Balloon Snake that will inflate or deflate the snake from its farthest positions to the final edge locations thus increasing the snake capture range. Previous combinations of watershed and snakes have not solved the over-segmentation problems. In [15], a snake zone is defined around the object boundaries where the corresponding watershed points are candidates for the true object boundary. Energy minimization or the searching process is carried out using dynamic programming. However, in most applications the difficulty is mainly in the definition (localization) of the snake zone. The watershed-based segmentation on a down-sampled image is used to get the initial contours for the snake [16]. This will guaranty automatic initialization of snakes using watershed and smoothness of watershed lines using active contours. A new energy functional is derived based on the topographic distance definition of the watershed line [17]. This segmentation will incorporate control over the smoothness of the watershed lines.
4. Balloon Snakes Cohen [18] introduced a model that makes the curve behaves like a balloon with an additional external force which moves the contour out (inflation) or in (deflation) along its normal thus eliminating the need for the initial curve to be close to the correct edges. By placing balloon snake inside/outside the object of interest and inflate/ deflate contour to its boundary, this new model is very useful for some object that is difficult to locate or too complicate to trace by original model. The Balloon Snake passes over relatively weak edges and it stops on salient edges [18,19]. In Balloon model, the original snake internal force is kept the same while a new external ‘pressure’ force term is added to the snake force function. The total Balloon Snake force is now: Fsnake ¼ ðFint þ Fext Þ þ Fpressure
ð9Þ
The external force is: Fext ¼ rEext ðnegative of the gradientÞ
Fext
¼ Gr ðx; yÞ f ðx; yÞðconvolutionÞ rEimage ¼ k 2 krEimage k
ð10Þ
The pressure force added is given by: Fig. 1. Low capture range of traditional snake.
Fpressure ¼ k 1 nðsÞ
ð11Þ
908
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
where n(s) is the normal unit vector to the curve at point v(s), and k1 is the amplitude of the force, it indicates inflation (when positive) or deflation (when negative). At the same time, the value of k1 has an influence on the speed of deformation. Fsnake ¼ ðFint þ Fext Þ þ Fpressure rEimage þ k 1 nðsÞ Fsnake ¼ Fint k 2 krEimage k
ð12Þ
The Fsnake is the force equation that needs to be minimized in Balloon Snake deformation. Due to the newly added pressure force, balloon model is able to overcome the over shrinking problem in the original snake. Also, if the object has a troublesome shape, placing a balloon inside the image and expanding its contour will solve matching the desired shape. The final solution is: vt ¼ ½A þ sI
1
½svt1 Fðvt1 Þ þ Fpressure
ð13Þ
5. WaterBalloon snake 5.1. The hybrid technique Since watershed regions are closed and disjoint regions whose unions constitute the whole image and since traditional snakes need to be initialized near the target edges, the watershed lines around each region can be considered as initial snakes. Thus, watershed regions are involved in a multiple snakes approach. The idea of this method is that the appropriate internal and external forces should drive those snakes in order to be dragged by the significant edges.
Traditional snakes could not be initialized far away from the target edges. GVF snake should contain the medial axis of the object therefore they cannot drive watershed lines. On the other hand, Balloon Snakes can be inflated or deflated until they are attracted by the object edges. Balloons are free to be initialized anywhere inside object due to the powerful external pressure energy term that will drive the snake to the correct location. Since watershed regions are over-segmented version of the object, a Balloon Snake initialized at the location of a watershed region contour will be a strong candidate for capturing the salient object edges. Watershed regions cannot cross the object edges, so Balloon Snakes will not cause any problem. They also do not have to include the object medial axis, which ensure free initialization. Fig. 2 shows the block diagram of the proposed hybrid segmentation technique. The original image is low-pass filtered or blurred to remove small noisy regions. Then, the watershed transform is applied on the gradient image to segment watershed regions. Now, contour-tracing techniques can be applied to extract watershed region contours. Those contours will constitute the initial Balloon Snakes that will deform to capture the object edges. Balloon Snakes have a large capture range so correct snake deformation can be guarantied even if the initial snake (or the contour of the watershed region) is far away from the target edges. This situation is frequent for a given candidate watershed region, so Balloon Snake is most suitable to drive the watershed contours. Once a snake captures one object, the image is relabeled so that the next snake will not be a watershed contour inside the captured object. For each snake, parameter optimization is required to ensure opti-
Fig. 2. Block diagram of WaterBalloon segmentation.
Fig. 3. Proposed segmentation strategy. (a) Initial watershed regions and lines (over-segmented image). (b) Deformation of initial watershed lines with Balloon Snakes. (c) Final Contours with the two blood cells separated.
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912 -800 -1000
-1200 -1400
Energy
mum edge localization. For example in Fig. 3 (a), the first cell will be initialized with two labels. The first snake in region labeled 1 will capture the whole region (label 1 + label 2). So the first cell will be re-labeled by only one label and only one snake is needed. This process is repeated for each snake to obtain the WaterBalloon segmentation. Illustration of the technique on the two touching blood cells is shown in Fig. 3 with a low-resolution image.
909
-1600 -1800 -2000
-2200
5.2. Optimization of snake parameters based on energy tracking
-2400 -2600
• There exist an energy transition to a high energy state and then a drop to a lower energy state. • The drop to the lower energy state occurs simultaneously in both the external and total energy terms. This is not by chance; as the edge will deform from its initial position at the weak edge to the strongest edge, it traverses a homogeneous region where gradient field is weak and no edge is encountered. In this homogeneous region, the external energy term is maximal (in absolute sense) and thus the total energy term is maximal. This explains the transition to a high-energy state in both the total and external energy curves. One can also note that the final energy state is lower than the initial one since the snake is minimizing its total energy. This behavior will be the typical for the proposed WarterBalloon technique: starting from a watershed line as an initial Balloon Snake, the deformation from the weak edge to the strong edge through the homogeneous region will always induce this energy pulse. Fig. 5 corresponds to the case where the balloon is inflated outside the target boundaries. In this case, the total energy increases considerably. Whenever this case is encountered, the parameters need to be changed and the pressure weight should be decreased. This parameters updating is continued until the energy transitions follow
-2800 0
50
100
150
200
250
200
250
iteration 0 -200
Energy
-400
-600 -800
-1000 -1200
-1400 0
50
100
150
iteration Fig. 4. Energy plots of WaterBalloon snakes. (a) External energy term behavior. (b) Total energy term behavior.
the case of Fig. 4. The optimum values obtained are then fed back to deform the snake. 3000 2500 2000 1500
Energy
A snake may not be located at the target edges due to its sensitivity to the snake parameters. Thus there is a need to parameter optimization that best localize the correct edges. The lack of knowledge about those edges is what makes optimization of the snake parameters a very hard problem. Thus a new approach for determining those parameters is proposed that is derived from the energy functional behavior of WaterBalloons. The total energy term is shown in Fig. 4(b), the energy behavior seems not to provide any information about the snake performance. The challenge was to find a relationship between this energy behavior and the optimum parameters. The key solution was to look at the total energy term and the corresponding elasticity, bending, external and pressure (Balloon) energy terms. The idea is clear in Fig. 4(a) and (b) where two conclusions are drawn:
1000 500 0 -500 -1000 -1500 0
5
10
15
20
25
30
35
40
45
iteration Fig. 5. Total energy term behavior when Balloon inflates outside object.
910
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
Fig. 6. Block diagram of optimization with energy transitions tracking.
The block diagram of the optimization stage is shown in Fig. 6 where the parameters are initialized equally to give equal importance to all energy terms. From the above discussions, a criterion can be tested to check if there is an energy pulse with transition to a high energy state and a back transition to a lower energy state or there is only a transition to a high energy state in case of a continuously inflating balloon. This criterion is based on the initial, final, and maximum values of the energy. if Efinal Einitial < T; Emax then there is an energy pulse and the Balloon Snake will stabilize (T = 0.5 is satisfactory for this application). Else there is only transition to high energies with inflating Balloon Snake.
tion until an object is captured. Next, the image is relabeled to ensure that the second watershed region will not fall in the already captured object. Once a watershed region deformed with the Balloon Snake captures a single cell part of two or more touching cells, the following deformations will separate the remaining cells (Fig. 7c and d). WaterBalloon snake was able to separate the touching objects. It should be noted that the arrow in Fig. 7(b) shows a cell which have been over-segmented by the watershed transform (seven regions). A contour extraction algorithm is applied in order to get the contours of all the seven regions (seven labels). Seven Snakes will be initialized. Starting with the first snake, its deformation will fill the first blood cell (the arrow in 7(c). The image is relabeled and there is no need for the other 6 snakes to be deformed. 6.2. Application to bovine endothelial corneal cells
6. Experimental results 6.1. Application to blood cell image Fig. 7(a) shows red blood cells image with two touching pair of cells. In traditional segmentation methods, touching objects (the blood cells) are considered as one object because these methods are insensitive to the low intensity levels of the border pixels between these objects. In Fig. 7(b), watershed transform was applied to the gradient image. In a sequential manner, a watershed region is picked up and will be subject to Balloon Snake deforma-
Fig. 8(a) shows the image of the bovine endothelial corneal cells acquired through a CCD camera attached to a microscope [20]. The arrows show an example of four cells. The objective is to segment all the 36 cells. Fig. 8(b), (c), (d), (e) shows, respectively, the convergence of the bubbles (with bubbles initialized at image minima), the front propagation, the seeded region growing (36 seeds initialized by hand), and the CBMA algorithms (fine-tuned threshold parameters). Fig. 8(f) shows the convergence of the WaterBalloons algorithm. It is clear that the touching cells have been well separated. The number of falsely segmented
Fig. 7. WaterBalloon segmentation. (a) Original image. (b) Watershed regions. (c) Balloon Snake deformation. (d) WaterBalloon segmentation with the touching cells separated.
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
911
Fig. 8. (a) Original image. (b) Bubbles algorithm. (c) Front propagation algorithm. (d) Seeded region growing algorithm. (e) CBMA algorithm. (f) WaterBalloons algorithm.
regions for the bubbles algorithm is 4, for the front propagation is 2, for the seeded region growing is 1, for the CBMA is 6, and for the WaterBalloons is zero. It should be noted that some shift of the contour from its real position have occurred in the WaterBalloons algorithm due to the blurring filter that initialized the process.
ent objects in an image. Waterballoons provide the advantage of reducing over-segmentation of Watershed while preventing under-segmentation problems. Touching objects are separated, snake parameters are optimized based on energy transitions tracking, and optimum edge localization is achieved.
7. Conclusion
References
In this paper, a hybrid Watershed Balloon Snake technique for image segmentation is introduced. The new technique combines the use of Watershed transform and Balloon Snakes in an innovative approach that detects sali-
[1] H. Tek et al., Volumetric segmentation of medical images by three dimensional bubbles, Computer Vision and Machine Understanding 65 (2) (1997) 246–258. [2] R. Adams et al., Seeded region growing, IEEE Transactions on PAMI 16 (6) (1994) 641–647.
912
I. Dagher, K.E. Tom / Image and Vision Computing 26 (2008) 905–912
[3] R. Malladi et al., Shape modeling with front propagating: a level set approach, IEEE Transactions on PAMI 17 (2) (1995) 158–175. [4] M.C. Andrade et al., Segmentation of microscopic images by flooding simulation: a catchment basins merging algorithm, in: Proceedings of SPIE Non linear Image processing VIII, vol. 3026, San Jose, USA, 1997, pp. 164–175. [5] R.C. Gonzales, R.E. Woods, Digital Image Processing, third ed., Prentice Hall, 2002, ISBN: 0-201-18075. [6] J. Serra, Image Analysis and Mathematical Morphology, Academic Press, London, 1982. [7] L. Vincent, P. Soille, Watershed in digital spaces: an efficient algorithm based on immersion simulations, IEEE Transactions on PAMI 13 (6) (1991) 583–598. [8] S. Beucher, F. Meyer, The morphological approach to segmentation: the watershed transformation, in: E.R. Dougherty (Ed.), Mathematical Morphology in Image Processing, Marcel Dekker, New York, 1993, pp. 433–481. [9] L. Najman, M. Schmitt, Geodesic saliency of watershed contours and hierarchical segmentation, IEEE Transactions on PAMI 18 (12) (1996) 1163–1173. [10] J.M. Gauch, Image segmentation and analysis via multiscale gradient watershed hierarchies, IEEE Transactions on Image Processing 8 (1999) 69–79. [11] A. Bleau, L.J. Leon, Watershed-based segmentation and region merging, Computer Vision and Image Understanding 77 (2000) 317–370. [12] K. Haris, S.N. Efstratiadis, N. Maglaveras, A.K. Katsaggelos, Hybrid image segmentation using watersheds and fast region merging, IEEE Transactions on Image Processing 7 (12) (1998) 1684–1699.
[13] M. Kass, A. Witkin, D. Terzopoulos, Snakes: active contour models, International Journal of Computer Vision 1 (1987) 321–331. [14] R. Nilanjan, S.T. Acton, T. Altes, E. de Lange, J. Brookeman, Merging parametric active contours within homogeneous image regions for MRI-based lung segmentation, IEEE Transactions on Medical Imaging 22 (2) (2003). [15] J. Park, J.M. Keller, Snakes on the watershed, IEEE Transactions on PAMI 23 (10) (2001). [16] V. Kiran, P.K. Bora, Watersnake: integrating the watershed and the active contour algorithms, IEEE (2003). [17] H.T. Nguyen, M. Worring, R. van den Boomgaard, Watersnakes: energy-driven watershed segmentation, IEEE Transactions on PAMI 25 (3) (2003). [18] L.D. Cohen, On active contour models and balloons, Computer Vision Graphics and Image Processing: Image Understanding 53 (2) (1991) 211–218. [19] L.D. Cohen, I. Cohen, Finite-element methods for active contour models and balloons for 2-D and 3-D images, IEEE Transactions on PAMI 15 (11) (1993) 1131–1147. [20] M.C. Andrade, An interactive algorithm for image smoothing and segmentation, Electronic Letters Computer Vision and Image Analysis 4 (1) (2004) 32–48. [21] Kostas Haris, Serafim N. Efstratiadis, Nikolaos Maglaveras, K. Aggelos, Katsaggelos: hybrid image segmentation using watersheds and fast region merging, IEEE Transactions on Image Processing 7 (12) (1998) 1684–1699. [22] I. Vanhamel, I. Pratikakis, H. Sahli, Multiscale gradient watersheds of color images, IEEE Transactions on Image Processing 12 (6) (2003) 617–626.