- Email: [email protected]
Segmentation of plant cell pictures
Segmentation of plant cell pictures Carl Murray and Mark O’Malley
The cells are transparent. The cell walls only appear as boundaries on the extremes of a cell (when looking vertically downwards) where the light must travel a large distance through the cell wall. Due to transparency, cells below may be seen through cells above, and therefore junctions are formed where boundaries overlap. The cell shapes are reasonably predictable, however the shapes that occur due to boundary overlap are unpredictable. Due to the narrow depth of field of the microscope and transparency, only some of the boundaries are in fine focus. Those boundaries that are not in fine focus look blurred, and this is the main reason for the poor quality of the picture shown in Figure 1.
Transparent plant cells with overlapping boundaries when viewed through a microscope produce low contrast, poorly focused pictures. Multiple pictures taken at different foci are processed individually, and are combined to form a single picture. This single picture is further processed to produce the final segmented picture with boundaries of one pixel thick which preserves the geometric structure of the cells. Keywords: segmentation,
boundaries,
plant cells
The counting, sizing and analysis of plant cells is a task which is regularly undertaken in many discplines such as chemical engineering, biology and agricultural science. It is laborious, time consuming, and in general requires a very simple decision process to perform. The automation of this task using digital picture processing techniquesle3 requires three steps. First, a plant cell picture must be acquired and digitized. This picture must then be segmented4 into its constituent parts or objects. Finally, the objects are recognized” as cells, counted, sized and analysed. The segmentation step which is to be dealt with here is to identify the cell/ background boundaries. Errors in the segmentation step will make the recognition step more difficult, and will reduce overall system performance. Therefore, an effort should be made to make the segmentation step as error free as possible. The cells chosen here are Morinda Citrifolia, which are presumed to be representative of non-aggregative plant cell suspension cultures. The cells have been provided by colleagues in the Chemical Engineering Department, where research on the shear susceptibility of plant cell suspension cultures in a variety of growth and flow configurations is ongoing6. Figure 1 shows a sample picture of the cells being analysed. The principal attributes of the plant cell pictures are:
At the recognition stage techniques such as template matching’ will be of little use due to the unpredictability of the shapes. A heuristic approach is probably required, and for this reason it was deemed necessary that the segmentation process should: 0 0 0
produce a binary picture with cell boundaries at one level, background and cell interiors at the other level produce cell boundaries that are one pixel thick preserve the geometric identity of the cells
1. The cells grow in chains of various lengths consisting of up to ten cells and are of uniform width. Department of Electronic College
Dublin,
Belfield,
and Electrical Engineering, Dublin 4, Ireland
University
Paper received: 25 March 199I; revised paper received: I5 July 1992
0262%8856/93/030155-08 vol I1 no 3 april 1993
Figure I. Morinda Citrifolia plant cells taken through a microscope (magnification = 1000)
0 1993 Butterworth-Hememann
Ltd 155
l l
produce no spurious breaks or interior regions in the cell boundaries produce no spurious regions. which could be mistaken for cells.
Digital picture processing of [lant or animal cells has been investigated at length’-’ . Despite this wealth of knowledge, it is found on further investigation that the work previously done, while it is instructive and informative, cannot be applied directly to the problem considered here. Jain et al.’ investigated the segmentation of muscle cells which are not transparent and therefore do not have overlapping boundaries. The segmentation process is not addressed by Rintala and Hsu8, where the recognition of overlapping cells is tackled. Isolated single transparent cells are considered by Schnonfeld and Grebe”, and filamentous microorganisms are dealt with by Packer and Thomas”. In this paper, the particular problem of the segmentation of overlapping, microscopic transparent plant cells is addressed. The following section describes the preliminary steps taken such as digitization, pre-processing and initial attempts to segment the plant cell pictures. After the preliminaries, the ridging algorithm is described, followed by the thinning algorithm, the segmentation algorithm, results, discussion and conclusions.
64 PRELIMINARIES The pictures of the plant cells were obtained using a microscope coupled to a video camera. Dark field microscopy’ ’ was used to reduce the effect of nonuniform illumination and to enhance the cell background boundary. The video signal was digitized using a frame grabbing board mounted in an IBM PC. The resulting digital picture is stored in a 256 x 256 array. The individual pixels can have grey levels from 0 to 255, with 0 being the darkest and 255 being the lightest. The digital picture got by digitizing the picture shown in Figure 1 and filtered with a 5 x 5 smoothing mask”
Figure 2. Digital picture of Figure I, low pass filtered using a 5 x 5 smoothing mask (spatial resolution of 256 x 256 and 256 grey levels)
156
127
191
Grey level Figure 3. Grey level histogram of Figure 2
to remove high frequency spatial noise is shown in Figure 2. Initial attempts at segmentation using global thresholdingi’ and edge detection’” techniques failed. Figure 3 shows the histogram of the picture shown in Figure 2. The histogram is mono-modal, the single peak indicating the quasi-uniform background. The cell boundaries are swamped by the background and produce no detectable peak. All attempts to make the histogram bi-modal by enhancing the contrast between the background and cell boundaries failed because of the transparent nature of the cells. Figure 4, which is taken from the bottom left hand corner of Figure 2, considers the digital picture as a 3D grey level landscape where the cell boundaries are ridges. At the junctions where cell boundaries overlap as illustrated in position A of Figure 4, it was observed that one ridge dominates the other due to a higher grey level. The difference in grey level is attributed to the different depths of the overlapping cell boundaries, those cell boundaries that are in focus have a higher grey level than those that are not in focus. Pixels on the dominated ridge and close to the junction will have small gradients, and the ridge itself is severely distorted. This distortion is unpredictable as it depends on the angle of intersection of the ridges and the difference in the grey level. This domination of one ridge over the other, the small gradients and the distortion is clearly illustrated in Figure 4. The problems of focusing, low contrast and distortion at junctions makes the detection of edges very difficult. For example, the picture shown in Figure 2 is operated on by a Sobel operator” and then image and vision computing
locally choosing the detector with the most significant response could overcome this problem, but Spacek’” argues against such a technique. Noblelh declares that Canny’s approach relies on strong gradient magnitudes, and is not suitable for low contrast pictures, in particular around junctions. In summary, three main problems arose in trying to segment the cell pictures. The cell pictures have a low contrast; some of the boundaries are in fine focus whilst others are blurred, and where boundaries overlap they cause severe and unpredictable distortion to one another. Under these conditions it is unlikely that any one edge detector applied to a single picture would be capable of detecting the cell/background boundary to the accuracy that the recognition stage requires. To overcome the problems of blurring it was decided to take the same picture at different foci, to segment each picture individually, and then to combine these segmented pictures to obtain the final segmented picture. To overcome the problems of edge detection near boundary junctions, a specific edge detector I m ) was developed which would work (ridging algor’th well on those parts of the picture that were in focus, and in particular close to junctions. Figure 4. Three dimensional grey level landscape showing cell boundaries as ridges (taken from bottom left hand corner of the picture shown in Figure 2). Position A represents a boundary junction where one ridge crosses another thresholded to produce a binary picture, as shown in Figure 5. There are many large gaps in the picture shown in Figure 5, in particular near boundary junctions of overlapping cells. Canny” has developed a technique where the shape of the edge is characterized (roof, ridge and step), and an optimal detector is designed to find this shape. However, the shape of the edges here are unpredictable in particular near boundary junctions. Using more than one detector and
RIDGING
ALGORITHM
Cell boundaries are the tops of ridges” in a three dimensional landscape. The ridging algorithm uses this characteristic to find the cell boundaries. The algorithm can be divided into two distinct parts. The first part is the gradient stage. and the second part the threshold stage.
Gradient stage Consider illustrated diagonally attempts considered be on top
a pixel p, with the eight closest neighbours as in Figure 6. The pixel p, has four pairs of opposed neighbours. The gradient stage to find all the pixels which could be to be on top of a ridge. A pixel is deemed to of a ridge if:
1. It has a higher grey level than any pair of diagonally opposed neighbours; or 2. It has a higher grey level than one of the diagonally opposed neighbours, and the same grey level as the other diagonally opposed neighbour. The ridging algorithm has five steps. Steps l-4 test each pixel to see if it is on top of a ridge. At each step. if
P 9
Figure 5. Cell pcture shown in Figure 2 with Sohel operator applied and thresholded (spatial resolution 256 x 256 and 2 grey levels) vol I I no 3 april 1993
P 7
P
2
P 6
P
3
P 5
Figure 6. Pixel p, and the eight closest neighhours 157
a pixel is deemed to be on top of a ridge then the maximum difference between this pixel and the two diagonally opposed neighbours is called the ridge gradient, and is recorded. There are four ridge gradients for each pixel, and Step 5 records the maximum of these. This value is the grey level of the corresponding pixel in a picture called the gradient picture. The gradient picture is formed by applying the above five steps to all pixels in the original picture, except the border pixels which are set to zero.
Threshold
stage
The ridged picture is formed from the gradient by an automated thresholding process which steps:
picture has six
Step 1: All pixels in the gradient
picture with the grey level are tagged. Step 2: The number of separate eight-connected groups of tagged pixels* is determined. This number is called N,. The corresponding lowest tagged grey level is called G,. Step 3: From the remaining untagged pixels in the gradient picture the pixels with the highest grey level are tagged. Step 4: Steps 2 and 3 are repeated until the only untagged pixels are those with grey level of zero. Step 5: A plot of lowest tagged grey level (G,) versus the number of separate eight-connected groups of tagged pixels (NJ is illustrated in Figure 7. The position where N, first increases by a factor greater than or equal to a ridge threshold as G, is decreased by one unit is denoted by G,,- 1. The gradient threshold is G,,. A ridge threshold of two was found to give the best results. Step 6: The ridged picture is formed by taking all pixels in the gradient picture with grey level values above or equal to the gradient threshold G,, and assigning to them a grey level of 0 (logic one), all other pixels are assigned a value of 255 (logic zero). The ridged picture is thus a binary picture. highest
These six steps require an explanation. The grey level of a pixel in the gradient picture is the maximum ridge gradient of the corresponding pixel in the original picture. It was observed that noise is characterized by small and numerous groups of connected pixels with low ridge gradients and cell boundaries by large and less numerous groups of connected pixels with higher ridge gradients. As the grey level decreases from the highest value the number and size of the connected groups increases. This increase in the number of groups is slow at first as the groups are likely to be boundaries. As the grey level decreases further a rapid increase in the number of connected groups occurs due to the onset of noise. The ridging algorithm uses this rapid increase in the number of connected groups to determine the gradient threshold. Too high a value of gradient threshold and the breaks in the cell boundaries become significant; too low and the resulting ridged picture becomes immersed in noise. The routine outlined above which automatically *With reference to Figure 6, a pixel p, which is tagged is deemed to be eight-connected if any of its eight neighbouring pixels pz, p3, p4, ps, ph, p, and p8 are also tagged.
158
Position of the grodient threshold (G,)
34 L__2
4
6
0
10
12
14
16
Is
:
Lowest togged grey level (Gn) Figure 7. Number of connected groups (NJ versus lowest tagged grey level (G,). Illustration of how the gradient threshold is chosen automatically chooses the gradient threshold results in a ridged picture with some noise and some breaks. The pictures must now be thinned and the remaining noise removed.
THINNING
ALGORITHM
The thinning
algorithm
has two purposes:
1. To extract boundaries one pixel tain the geometry of the cells. 2. To remove noise. The algorithm used algorithm developed binary pictures.
Modification
was a modified by Zhang and
thick
that
main-
version of an Suen” to thin
1
The original thinning algorithm’x in determining whether a pixel should be considered as belonging to an edge as opposed to being within the body of a line uses the following test. If the number of zero one transitions in traversing the pixel’s eight immediate neighbours is equal to one then the pixel is considered a candidate for deletion. Clearly all open ended one pixel wide lines will satisfy this condition. Other tests in the original algorithm will eliminate the candidature of all such pixels, with the exception of a four-connected pixel+ ‘With reference to Figure 6 a pixel pI is deemed to be four-connected if it has the same grey level as any of its four neighboursp2, p4, ph and
PH.
image and vision computing
pictures. These intermediate pictures are combined together using a logical OR operation to form one picture called the combined picture. In the combined picture a pixel is made logic one (grey level of 0) if that pixel location is logic one in any of the intermediate pictures; otherwise, it is made logic zero (grey level of 255). The boundaries in the intermediate pictures may have slight misalignment with each other. This misalignment can result in a combined picture with small enclosed regions within the cell boundaries, and cell boundaries which are not thinned. To remove the spurious enclosed regions from the combined picture, any enclosed region of less than some number of pixels called the enclosed threshold are filed in. An enclosed threshold of eight pixels is used here and produces good results. The thinning algorithm is applied to the combined picture (with the spurious enclosed regions removed) to thin the boundaries, and the resulting picture is called the final segmentedpicture. A summary of the segmentation process developed by the authors is given in Figure 9.
Figure 8. Open ended four connected line one pixel thick
at the open end of a line, as in Figure 8. The pixel in the top left hand corner here would be deleted by the original algorithm, and on successive iterations the complete line would be deleted. A very minor modification to the original algorithm ensured that this exception does not occur and all such lines are not deleted.
RESULTS Results are presented algorithm.
Microscopic Cells
Microscope at focus fi (Dark field microscopy)
Modification 2 The second modification can be considered to be an addition to the original algorithm rather than a modification, and was introduced to remove noise. The thinned picture as produced by the original algorithm (with the first modification) is tested for open ended lines of less than some number of pixels called the tracking threshold. All such lines are deemed to be spurious and are deleted. Most of the spurious lines have two open ends and result from background noise. A few of the spurious lines have one open end with the other end forming a junction with a cell boundary. Too large a tracking threshold and some cell boundaries are removed. too low and some spurious lines remain. A tracking theshold of eight was found to give good results.
SEGMENTATION ALGORITHM It was found that the ridging algorithm assigned high ridge gradients to cell boundaries in fine focus, and low ridge gradients to cell boundaries out of focus. Therefore, the thresholding stage of the ridging algorithm may overlook some cell boundaries that are out of focus. As all cell boundaries cannot be brought into focus in a single picture due to the narrow depth of field of the microscope, several pictures of the cells at different foci were taken so that all parts of all cells were in focus in at least one picture. Each of these pictures is digitized, smoothed, and has the ridge and thinning algorithm applied to them. The resulting pictures are binary, and are referred to as intermediate r~l I I no 3 april I993
at each stage of the segmentation
Video camera Frame Grabber
Low pass filtering with 5x5 smoothing mask
Ridging algorithm
Thinning algorithm
Intermediate picture P(fi )
pCfa)
Wb)
WC)
PU)o
Combine pictures
Fill in all interior regions of less than 8 pixels in area
Thinning algorithm
Final segmented picture
Figure 9. Summary of segmentation algorithm 159
Figure 10. Gradient picture resulting from gradient stage of the ridging algorithm applied to Figure 2 (spatial resolution 256 x 256 and 20 grey levels) Gradient
Figure 12. Ridged picture resulting from the threshold stage of the ridging algorithm applied to the gradient picture in Figure 10 (gradient threshold is two; spatial resolution 256 x 256 and two grey levels)
picture
Figure 10 shows the gradient picture which results when the gradient stage of the ridging algorithm is applied to the picture shown in Figure 2. As can be seen, this is a noisy picture.
results with respect to the gradient threshold, Figure 12 shows a ridged picture which uses a lower gradient threshold of two. The lower gradient threshold has resulted in a picture with very significant amounts of noise.
Ridged picture
Figure 11 shows the ridge picture obtained by applying the threshold stage of the ridging algorithm to the gradient picture shown in Figure 10, where the gradient threshold (chosen automatically) is four. It is clearly seen that this picture has both breaks in the boundaries and some noise. To illustrate the sensitivity of the
Figure 11. Ridged picture resulting from the threshold stage of the ridging algorithm applied to the gradient picture in Figure 10 (gradient threshold (chosen automatically) is four; spatial resolution 256 x 256 and two grey levels)
160
Intermediate
picture
The thinning algorithm is applied to the picture shown in Figure 11, and the resulting intermediate picture is shown in Figure 13. The cell boundaries are one pixel thick and preserve the geometric structure of the cells. The thinning algorithm has removed a lot of the noise that was in the. ridged picture, but has also removed some of the boundaries with breaks in them.
Figure 13. Intermediate picture obtained by thinning the ridged picture in Figure 11 (spatial resolution 256 X 256 and two grey levels) image and vision computing
Combined picture The result of taking the intermediate picture shown in Figure 13 and three other intermediate pictures (each originating from pictures taken at different foci) and combining using a logical OR operation is shown in Figure 14. From this combined picture it can be seen that some of the cell boundaries not present in Figure 13 have been recovered. This is because these boundaries are in better focus in one of the three other pictures involved. The combining process has also produced small spurious interior regions due to misalignment between the four intermediate pictures. This effect is pronounced at positions A and B in Figure 14, and less so in other positions. Final segmented picture Interior regions of less than the enclosed threshold (eight pixels) are filled in, and the thinning algorithm is applied to the combined picture. The resulting picture is the final segmented picture, as shown in Figure 14, which is representative of the pictures that can be consistently produced by the segmentation algorithm. Referring back to the five requirements set out in the introduction for the segmentation algorithm, it is noted that Figure 15 fails on two counts. There are ten breaks in the cell boundaries at positions Bl to BlO. All the breaks occur at boundary junctions because of the low ridge gradients at these points. There is also a spurious line at position A which does not represent a cell boundary. Overcoming the breaks in the cell boundaries will simultaneously give a solution to removing the spurious lines. As has been said before, misalignment in the intermediate pictures may result in spurious interior regions in the boundaries. In certain circummisalignment may also produce spurious stances, lines similar to that at position A in Figure 15. The probability of occurrence of these spurious lines was
J Figure 15. Final segmented picture obtained by removing interior regions in the boundaries and thinning the combined picture in Figure 14 (spatial resolution 256 x 256 and two grey levels). Positions RI to BIO are breaks in the cell boundaries. Position A represents a spurious line which has been caused by misalignment between the four intermediate pictures
found to increase as the number of pictures at different foci which are combined increases. Therefore, there is a limit to the number of pictures’ which can be combined to produce better results. The thinning algorithm with a larger tracking threshold will eliminate these spurious lines but at the cost of possibly deleting cell boundaries which have breaks. Therefore, if the breaks can be fixed then the spurious lines can be eliminated without losing cell boundaries. DISCUSSION
Figure /4. Combined picture obtained by combining the intermediate picture in Figure 13 and three others taken at different foci (spatial resolution 256 x 256 and two grey levels). Positions A and B represent spurious interior regions caused by misalignment between the four intermediate pictures vol I I no 3 april 1993
The ridging algorithm is an edge detector which is relatively simple and requires little computational effort. It is good at detecting cell/background boundaries that are in focus, and performs well close to junctions, although as the results show, all the breaks occur at junctions. Its performance should be judged in the context of the results of the overall system, and not in isolation. Other edge detectors may be used instead of the ridging algorithm, but they may have difficulties detecting the boundaries close to junctions. There are four separate noise removal techniques used at different stages; low pass filtering, thresholding within the ridging algorithm, deletion of short disconnected lines within the thinning algorithm, and small region removal within the segmentation stage. These noise removal stages cannot be unified into a single stage. After the low pass filtering, each subsequent algorithm extracts a specifific subset of pixels disregarding other pixels as noise or more correctly, undesired. Therefore, a single noise removal stage would be insufficient as the noise to be removed only reveals itself after application of the previous algorithm. The ridge threshold. tracking threshold and the 161
enclosed threshold are all parameters whose values were found by experimentation. Consistent results are obtained without changing the values of any of these parameters for this particular picture set, i.e. Morinda Citrifolia plant cells.
7
8
CONCLUSIONS A simple but effective segmentation algorithm for overlapping, microscopic transparent plant cells has been described. Cell boundaries are ridges on a three dimensional grey level landscape, and a ridging algorithm is developed to find them. Pictures taken at different foci are combined using a logical OR operation. This is done in an effort to extract the best information from each picture. There is an upper limit to the number of pictures which may be combined successfully. The resulting segmented picture has cell boundaries which are one pixel thick and maintain the geometric structure of the cells. Small breaks may occur in the cell boundaries and the elimination of these small boundaries is the subject of further work.
REFERENCES Dugdale, R 0 and Hart, P E Pattern Classification and Scene Analysis, John Wiley, New York (1973) Rosenfeld, A and Kak, A C Digital Picture Processing (2nd edn), Academic Press, New York (1976) Gonzalez, R C and Wintz, P Digital Image Processing (2nd edn), Addison-Wesley, NJ (1987) Harlick, R M and Shapiro, L G ‘Survey: image segmentation Comput. Vision, techniques’, Graph. & Image Process., Vol 29 (1985) pp 870881 Levine, M D ‘Feature extraction: A survey’, Proc. IEEE, Vol 57 (1969) pp 1391-1407 Kieran, P, Malone, D M, MacLoughlin, P F and Wilson, G ‘Assessing the effects of fluid dynamic shear on plant cell suspension cultures’, Proc.
162
9
10
11 12
13
14
15
16
17
18
I. Chem. E. Research Colloquium, Dublin, Ireland (1988) pp 178-187 Jain, A K, Smith, S P and Baker, E ‘Segmentation of muscle cell pictures: a preliminary study’, IEEE Trans. PAMI, Vo12 No 3 (May 1980) pp 232-242 Rintala, W M and Hsu, C C ‘A feature detection program for patterns with overlapping cells’, IEEE Trans. Syst. Sci. & Cybern., Vol4 (March 1968) pp 16-23 Schnonfeld, M and Grebe, R ‘Automatic shape quanitification of freely suspended red blood cells by isodensity contour tracing and tangent counting’, Comput. Methods & Programs in Biomedicine, Vo128 (1989) pp 217-224 Packer, H L and Thomas, C R ‘Morophological measurements on filamentous microorganisms by fully automatic image analysis’, Biotechnology & Bioeng., Vol 35 (1990) pp 870-881 Spencer, M Fundamentals of Light Microscopy, Cambridge University Press, Cambridge (1982) Lee, S U and Chung, S Y ‘A comparative performance study of several global thresholding techniques for segmentation’, Comput. Vision, Graph. & Image Process., Vol 52 (1990) pp 171190 Abdou, I E and Pratt, W K ‘Qualitative design and evaluation of enhancement/thresholding edge detectors’, Proc. IEEE (May 1979) pp 753-763 Canny, J ‘A computational approach to edge detection’, IEEE Trans. PAMI, Vol 8 (1986) pp 679-698 Spacek, L A ‘Edge detection and motion detection’, Image & Vision Cornput., Vol 4 (1986) pp 43-56 Nobel, J A ‘Finding half boundaries and junctions in images’, Image & Vision Comput., Vol 10 (1992) pp 219-232 Harlick, R M ‘Ridges and valleys on digital images’, Comput. Vision, Graph. & Image Process., Vol 22 (1983) pp 28-38 Zhang, T Y and Suen, C Y ‘A fast parallel algorithm for thinning digital patterns’, Commun. ACM, Vol 27 No 3 (1984) pp 236-239
image and vision computing
The cells are transparent. The cell walls only appear as boundaries on the extremes of a cell (when looking vertically downwards) where the light must travel a large distance through the cell wall. Due to transparency, cells below may be seen through cells above, and therefore junctions are formed where boundaries overlap. The cell shapes are reasonably predictable, however the shapes that occur due to boundary overlap are unpredictable. Due to the narrow depth of field of the microscope and transparency, only some of the boundaries are in fine focus. Those boundaries that are not in fine focus look blurred, and this is the main reason for the poor quality of the picture shown in Figure 1.
Transparent plant cells with overlapping boundaries when viewed through a microscope produce low contrast, poorly focused pictures. Multiple pictures taken at different foci are processed individually, and are combined to form a single picture. This single picture is further processed to produce the final segmented picture with boundaries of one pixel thick which preserves the geometric structure of the cells. Keywords: segmentation,
boundaries,
plant cells
The counting, sizing and analysis of plant cells is a task which is regularly undertaken in many discplines such as chemical engineering, biology and agricultural science. It is laborious, time consuming, and in general requires a very simple decision process to perform. The automation of this task using digital picture processing techniquesle3 requires three steps. First, a plant cell picture must be acquired and digitized. This picture must then be segmented4 into its constituent parts or objects. Finally, the objects are recognized” as cells, counted, sized and analysed. The segmentation step which is to be dealt with here is to identify the cell/ background boundaries. Errors in the segmentation step will make the recognition step more difficult, and will reduce overall system performance. Therefore, an effort should be made to make the segmentation step as error free as possible. The cells chosen here are Morinda Citrifolia, which are presumed to be representative of non-aggregative plant cell suspension cultures. The cells have been provided by colleagues in the Chemical Engineering Department, where research on the shear susceptibility of plant cell suspension cultures in a variety of growth and flow configurations is ongoing6. Figure 1 shows a sample picture of the cells being analysed. The principal attributes of the plant cell pictures are:
At the recognition stage techniques such as template matching’ will be of little use due to the unpredictability of the shapes. A heuristic approach is probably required, and for this reason it was deemed necessary that the segmentation process should: 0 0 0
produce a binary picture with cell boundaries at one level, background and cell interiors at the other level produce cell boundaries that are one pixel thick preserve the geometric identity of the cells
1. The cells grow in chains of various lengths consisting of up to ten cells and are of uniform width. Department of Electronic College
Dublin,
Belfield,
and Electrical Engineering, Dublin 4, Ireland
University
Paper received: 25 March 199I; revised paper received: I5 July 1992
0262%8856/93/030155-08 vol I1 no 3 april 1993
Figure I. Morinda Citrifolia plant cells taken through a microscope (magnification = 1000)
0 1993 Butterworth-Hememann
Ltd 155
l l
produce no spurious breaks or interior regions in the cell boundaries produce no spurious regions. which could be mistaken for cells.
Digital picture processing of [lant or animal cells has been investigated at length’-’ . Despite this wealth of knowledge, it is found on further investigation that the work previously done, while it is instructive and informative, cannot be applied directly to the problem considered here. Jain et al.’ investigated the segmentation of muscle cells which are not transparent and therefore do not have overlapping boundaries. The segmentation process is not addressed by Rintala and Hsu8, where the recognition of overlapping cells is tackled. Isolated single transparent cells are considered by Schnonfeld and Grebe”, and filamentous microorganisms are dealt with by Packer and Thomas”. In this paper, the particular problem of the segmentation of overlapping, microscopic transparent plant cells is addressed. The following section describes the preliminary steps taken such as digitization, pre-processing and initial attempts to segment the plant cell pictures. After the preliminaries, the ridging algorithm is described, followed by the thinning algorithm, the segmentation algorithm, results, discussion and conclusions.
64 PRELIMINARIES The pictures of the plant cells were obtained using a microscope coupled to a video camera. Dark field microscopy’ ’ was used to reduce the effect of nonuniform illumination and to enhance the cell background boundary. The video signal was digitized using a frame grabbing board mounted in an IBM PC. The resulting digital picture is stored in a 256 x 256 array. The individual pixels can have grey levels from 0 to 255, with 0 being the darkest and 255 being the lightest. The digital picture got by digitizing the picture shown in Figure 1 and filtered with a 5 x 5 smoothing mask”
Figure 2. Digital picture of Figure I, low pass filtered using a 5 x 5 smoothing mask (spatial resolution of 256 x 256 and 256 grey levels)
156
127
191
Grey level Figure 3. Grey level histogram of Figure 2
to remove high frequency spatial noise is shown in Figure 2. Initial attempts at segmentation using global thresholdingi’ and edge detection’” techniques failed. Figure 3 shows the histogram of the picture shown in Figure 2. The histogram is mono-modal, the single peak indicating the quasi-uniform background. The cell boundaries are swamped by the background and produce no detectable peak. All attempts to make the histogram bi-modal by enhancing the contrast between the background and cell boundaries failed because of the transparent nature of the cells. Figure 4, which is taken from the bottom left hand corner of Figure 2, considers the digital picture as a 3D grey level landscape where the cell boundaries are ridges. At the junctions where cell boundaries overlap as illustrated in position A of Figure 4, it was observed that one ridge dominates the other due to a higher grey level. The difference in grey level is attributed to the different depths of the overlapping cell boundaries, those cell boundaries that are in focus have a higher grey level than those that are not in focus. Pixels on the dominated ridge and close to the junction will have small gradients, and the ridge itself is severely distorted. This distortion is unpredictable as it depends on the angle of intersection of the ridges and the difference in the grey level. This domination of one ridge over the other, the small gradients and the distortion is clearly illustrated in Figure 4. The problems of focusing, low contrast and distortion at junctions makes the detection of edges very difficult. For example, the picture shown in Figure 2 is operated on by a Sobel operator” and then image and vision computing
locally choosing the detector with the most significant response could overcome this problem, but Spacek’” argues against such a technique. Noblelh declares that Canny’s approach relies on strong gradient magnitudes, and is not suitable for low contrast pictures, in particular around junctions. In summary, three main problems arose in trying to segment the cell pictures. The cell pictures have a low contrast; some of the boundaries are in fine focus whilst others are blurred, and where boundaries overlap they cause severe and unpredictable distortion to one another. Under these conditions it is unlikely that any one edge detector applied to a single picture would be capable of detecting the cell/background boundary to the accuracy that the recognition stage requires. To overcome the problems of blurring it was decided to take the same picture at different foci, to segment each picture individually, and then to combine these segmented pictures to obtain the final segmented picture. To overcome the problems of edge detection near boundary junctions, a specific edge detector I m ) was developed which would work (ridging algor’th well on those parts of the picture that were in focus, and in particular close to junctions. Figure 4. Three dimensional grey level landscape showing cell boundaries as ridges (taken from bottom left hand corner of the picture shown in Figure 2). Position A represents a boundary junction where one ridge crosses another thresholded to produce a binary picture, as shown in Figure 5. There are many large gaps in the picture shown in Figure 5, in particular near boundary junctions of overlapping cells. Canny” has developed a technique where the shape of the edge is characterized (roof, ridge and step), and an optimal detector is designed to find this shape. However, the shape of the edges here are unpredictable in particular near boundary junctions. Using more than one detector and
RIDGING
ALGORITHM
Cell boundaries are the tops of ridges” in a three dimensional landscape. The ridging algorithm uses this characteristic to find the cell boundaries. The algorithm can be divided into two distinct parts. The first part is the gradient stage. and the second part the threshold stage.
Gradient stage Consider illustrated diagonally attempts considered be on top
a pixel p, with the eight closest neighbours as in Figure 6. The pixel p, has four pairs of opposed neighbours. The gradient stage to find all the pixels which could be to be on top of a ridge. A pixel is deemed to of a ridge if:
1. It has a higher grey level than any pair of diagonally opposed neighbours; or 2. It has a higher grey level than one of the diagonally opposed neighbours, and the same grey level as the other diagonally opposed neighbour. The ridging algorithm has five steps. Steps l-4 test each pixel to see if it is on top of a ridge. At each step. if
P 9
Figure 5. Cell pcture shown in Figure 2 with Sohel operator applied and thresholded (spatial resolution 256 x 256 and 2 grey levels) vol I I no 3 april 1993
P 7
P
2
P 6
P
3
P 5
Figure 6. Pixel p, and the eight closest neighhours 157
a pixel is deemed to be on top of a ridge then the maximum difference between this pixel and the two diagonally opposed neighbours is called the ridge gradient, and is recorded. There are four ridge gradients for each pixel, and Step 5 records the maximum of these. This value is the grey level of the corresponding pixel in a picture called the gradient picture. The gradient picture is formed by applying the above five steps to all pixels in the original picture, except the border pixels which are set to zero.
Threshold
stage
The ridged picture is formed from the gradient by an automated thresholding process which steps:
picture has six
Step 1: All pixels in the gradient
picture with the grey level are tagged. Step 2: The number of separate eight-connected groups of tagged pixels* is determined. This number is called N,. The corresponding lowest tagged grey level is called G,. Step 3: From the remaining untagged pixels in the gradient picture the pixels with the highest grey level are tagged. Step 4: Steps 2 and 3 are repeated until the only untagged pixels are those with grey level of zero. Step 5: A plot of lowest tagged grey level (G,) versus the number of separate eight-connected groups of tagged pixels (NJ is illustrated in Figure 7. The position where N, first increases by a factor greater than or equal to a ridge threshold as G, is decreased by one unit is denoted by G,,- 1. The gradient threshold is G,,. A ridge threshold of two was found to give the best results. Step 6: The ridged picture is formed by taking all pixels in the gradient picture with grey level values above or equal to the gradient threshold G,, and assigning to them a grey level of 0 (logic one), all other pixels are assigned a value of 255 (logic zero). The ridged picture is thus a binary picture. highest
These six steps require an explanation. The grey level of a pixel in the gradient picture is the maximum ridge gradient of the corresponding pixel in the original picture. It was observed that noise is characterized by small and numerous groups of connected pixels with low ridge gradients and cell boundaries by large and less numerous groups of connected pixels with higher ridge gradients. As the grey level decreases from the highest value the number and size of the connected groups increases. This increase in the number of groups is slow at first as the groups are likely to be boundaries. As the grey level decreases further a rapid increase in the number of connected groups occurs due to the onset of noise. The ridging algorithm uses this rapid increase in the number of connected groups to determine the gradient threshold. Too high a value of gradient threshold and the breaks in the cell boundaries become significant; too low and the resulting ridged picture becomes immersed in noise. The routine outlined above which automatically *With reference to Figure 6, a pixel p, which is tagged is deemed to be eight-connected if any of its eight neighbouring pixels pz, p3, p4, ps, ph, p, and p8 are also tagged.
158
Position of the grodient threshold (G,)
34 L__2
4
6
0
10
12
14
16
Is
:
Lowest togged grey level (Gn) Figure 7. Number of connected groups (NJ versus lowest tagged grey level (G,). Illustration of how the gradient threshold is chosen automatically chooses the gradient threshold results in a ridged picture with some noise and some breaks. The pictures must now be thinned and the remaining noise removed.
THINNING
ALGORITHM
The thinning
algorithm
has two purposes:
1. To extract boundaries one pixel tain the geometry of the cells. 2. To remove noise. The algorithm used algorithm developed binary pictures.
Modification
was a modified by Zhang and
thick
that
main-
version of an Suen” to thin
1
The original thinning algorithm’x in determining whether a pixel should be considered as belonging to an edge as opposed to being within the body of a line uses the following test. If the number of zero one transitions in traversing the pixel’s eight immediate neighbours is equal to one then the pixel is considered a candidate for deletion. Clearly all open ended one pixel wide lines will satisfy this condition. Other tests in the original algorithm will eliminate the candidature of all such pixels, with the exception of a four-connected pixel+ ‘With reference to Figure 6 a pixel pI is deemed to be four-connected if it has the same grey level as any of its four neighboursp2, p4, ph and
PH.
image and vision computing
pictures. These intermediate pictures are combined together using a logical OR operation to form one picture called the combined picture. In the combined picture a pixel is made logic one (grey level of 0) if that pixel location is logic one in any of the intermediate pictures; otherwise, it is made logic zero (grey level of 255). The boundaries in the intermediate pictures may have slight misalignment with each other. This misalignment can result in a combined picture with small enclosed regions within the cell boundaries, and cell boundaries which are not thinned. To remove the spurious enclosed regions from the combined picture, any enclosed region of less than some number of pixels called the enclosed threshold are filed in. An enclosed threshold of eight pixels is used here and produces good results. The thinning algorithm is applied to the combined picture (with the spurious enclosed regions removed) to thin the boundaries, and the resulting picture is called the final segmentedpicture. A summary of the segmentation process developed by the authors is given in Figure 9.
Figure 8. Open ended four connected line one pixel thick
at the open end of a line, as in Figure 8. The pixel in the top left hand corner here would be deleted by the original algorithm, and on successive iterations the complete line would be deleted. A very minor modification to the original algorithm ensured that this exception does not occur and all such lines are not deleted.
RESULTS Results are presented algorithm.
Microscopic Cells
Microscope at focus fi (Dark field microscopy)
Modification 2 The second modification can be considered to be an addition to the original algorithm rather than a modification, and was introduced to remove noise. The thinned picture as produced by the original algorithm (with the first modification) is tested for open ended lines of less than some number of pixels called the tracking threshold. All such lines are deemed to be spurious and are deleted. Most of the spurious lines have two open ends and result from background noise. A few of the spurious lines have one open end with the other end forming a junction with a cell boundary. Too large a tracking threshold and some cell boundaries are removed. too low and some spurious lines remain. A tracking theshold of eight was found to give good results.
SEGMENTATION ALGORITHM It was found that the ridging algorithm assigned high ridge gradients to cell boundaries in fine focus, and low ridge gradients to cell boundaries out of focus. Therefore, the thresholding stage of the ridging algorithm may overlook some cell boundaries that are out of focus. As all cell boundaries cannot be brought into focus in a single picture due to the narrow depth of field of the microscope, several pictures of the cells at different foci were taken so that all parts of all cells were in focus in at least one picture. Each of these pictures is digitized, smoothed, and has the ridge and thinning algorithm applied to them. The resulting pictures are binary, and are referred to as intermediate r~l I I no 3 april I993
at each stage of the segmentation
Video camera Frame Grabber
Low pass filtering with 5x5 smoothing mask
Ridging algorithm
Thinning algorithm
Intermediate picture P(fi )
pCfa)
Wb)
WC)
PU)o
Combine pictures
Fill in all interior regions of less than 8 pixels in area
Thinning algorithm
Final segmented picture
Figure 9. Summary of segmentation algorithm 159
Figure 10. Gradient picture resulting from gradient stage of the ridging algorithm applied to Figure 2 (spatial resolution 256 x 256 and 20 grey levels) Gradient
Figure 12. Ridged picture resulting from the threshold stage of the ridging algorithm applied to the gradient picture in Figure 10 (gradient threshold is two; spatial resolution 256 x 256 and two grey levels)
picture
Figure 10 shows the gradient picture which results when the gradient stage of the ridging algorithm is applied to the picture shown in Figure 2. As can be seen, this is a noisy picture.
results with respect to the gradient threshold, Figure 12 shows a ridged picture which uses a lower gradient threshold of two. The lower gradient threshold has resulted in a picture with very significant amounts of noise.
Ridged picture
Figure 11 shows the ridge picture obtained by applying the threshold stage of the ridging algorithm to the gradient picture shown in Figure 10, where the gradient threshold (chosen automatically) is four. It is clearly seen that this picture has both breaks in the boundaries and some noise. To illustrate the sensitivity of the
Figure 11. Ridged picture resulting from the threshold stage of the ridging algorithm applied to the gradient picture in Figure 10 (gradient threshold (chosen automatically) is four; spatial resolution 256 x 256 and two grey levels)
160
Intermediate
picture
The thinning algorithm is applied to the picture shown in Figure 11, and the resulting intermediate picture is shown in Figure 13. The cell boundaries are one pixel thick and preserve the geometric structure of the cells. The thinning algorithm has removed a lot of the noise that was in the. ridged picture, but has also removed some of the boundaries with breaks in them.
Figure 13. Intermediate picture obtained by thinning the ridged picture in Figure 11 (spatial resolution 256 X 256 and two grey levels) image and vision computing
Combined picture The result of taking the intermediate picture shown in Figure 13 and three other intermediate pictures (each originating from pictures taken at different foci) and combining using a logical OR operation is shown in Figure 14. From this combined picture it can be seen that some of the cell boundaries not present in Figure 13 have been recovered. This is because these boundaries are in better focus in one of the three other pictures involved. The combining process has also produced small spurious interior regions due to misalignment between the four intermediate pictures. This effect is pronounced at positions A and B in Figure 14, and less so in other positions. Final segmented picture Interior regions of less than the enclosed threshold (eight pixels) are filled in, and the thinning algorithm is applied to the combined picture. The resulting picture is the final segmented picture, as shown in Figure 14, which is representative of the pictures that can be consistently produced by the segmentation algorithm. Referring back to the five requirements set out in the introduction for the segmentation algorithm, it is noted that Figure 15 fails on two counts. There are ten breaks in the cell boundaries at positions Bl to BlO. All the breaks occur at boundary junctions because of the low ridge gradients at these points. There is also a spurious line at position A which does not represent a cell boundary. Overcoming the breaks in the cell boundaries will simultaneously give a solution to removing the spurious lines. As has been said before, misalignment in the intermediate pictures may result in spurious interior regions in the boundaries. In certain circummisalignment may also produce spurious stances, lines similar to that at position A in Figure 15. The probability of occurrence of these spurious lines was
J Figure 15. Final segmented picture obtained by removing interior regions in the boundaries and thinning the combined picture in Figure 14 (spatial resolution 256 x 256 and two grey levels). Positions RI to BIO are breaks in the cell boundaries. Position A represents a spurious line which has been caused by misalignment between the four intermediate pictures
found to increase as the number of pictures at different foci which are combined increases. Therefore, there is a limit to the number of pictures’ which can be combined to produce better results. The thinning algorithm with a larger tracking threshold will eliminate these spurious lines but at the cost of possibly deleting cell boundaries which have breaks. Therefore, if the breaks can be fixed then the spurious lines can be eliminated without losing cell boundaries. DISCUSSION
Figure /4. Combined picture obtained by combining the intermediate picture in Figure 13 and three others taken at different foci (spatial resolution 256 x 256 and two grey levels). Positions A and B represent spurious interior regions caused by misalignment between the four intermediate pictures vol I I no 3 april 1993
The ridging algorithm is an edge detector which is relatively simple and requires little computational effort. It is good at detecting cell/background boundaries that are in focus, and performs well close to junctions, although as the results show, all the breaks occur at junctions. Its performance should be judged in the context of the results of the overall system, and not in isolation. Other edge detectors may be used instead of the ridging algorithm, but they may have difficulties detecting the boundaries close to junctions. There are four separate noise removal techniques used at different stages; low pass filtering, thresholding within the ridging algorithm, deletion of short disconnected lines within the thinning algorithm, and small region removal within the segmentation stage. These noise removal stages cannot be unified into a single stage. After the low pass filtering, each subsequent algorithm extracts a specifific subset of pixels disregarding other pixels as noise or more correctly, undesired. Therefore, a single noise removal stage would be insufficient as the noise to be removed only reveals itself after application of the previous algorithm. The ridge threshold. tracking threshold and the 161
enclosed threshold are all parameters whose values were found by experimentation. Consistent results are obtained without changing the values of any of these parameters for this particular picture set, i.e. Morinda Citrifolia plant cells.
7
8
CONCLUSIONS A simple but effective segmentation algorithm for overlapping, microscopic transparent plant cells has been described. Cell boundaries are ridges on a three dimensional grey level landscape, and a ridging algorithm is developed to find them. Pictures taken at different foci are combined using a logical OR operation. This is done in an effort to extract the best information from each picture. There is an upper limit to the number of pictures which may be combined successfully. The resulting segmented picture has cell boundaries which are one pixel thick and maintain the geometric structure of the cells. Small breaks may occur in the cell boundaries and the elimination of these small boundaries is the subject of further work.
REFERENCES Dugdale, R 0 and Hart, P E Pattern Classification and Scene Analysis, John Wiley, New York (1973) Rosenfeld, A and Kak, A C Digital Picture Processing (2nd edn), Academic Press, New York (1976) Gonzalez, R C and Wintz, P Digital Image Processing (2nd edn), Addison-Wesley, NJ (1987) Harlick, R M and Shapiro, L G ‘Survey: image segmentation Comput. Vision, techniques’, Graph. & Image Process., Vol 29 (1985) pp 870881 Levine, M D ‘Feature extraction: A survey’, Proc. IEEE, Vol 57 (1969) pp 1391-1407 Kieran, P, Malone, D M, MacLoughlin, P F and Wilson, G ‘Assessing the effects of fluid dynamic shear on plant cell suspension cultures’, Proc.
162
9
10
11 12
13
14
15
16
17
18
I. Chem. E. Research Colloquium, Dublin, Ireland (1988) pp 178-187 Jain, A K, Smith, S P and Baker, E ‘Segmentation of muscle cell pictures: a preliminary study’, IEEE Trans. PAMI, Vo12 No 3 (May 1980) pp 232-242 Rintala, W M and Hsu, C C ‘A feature detection program for patterns with overlapping cells’, IEEE Trans. Syst. Sci. & Cybern., Vol4 (March 1968) pp 16-23 Schnonfeld, M and Grebe, R ‘Automatic shape quanitification of freely suspended red blood cells by isodensity contour tracing and tangent counting’, Comput. Methods & Programs in Biomedicine, Vo128 (1989) pp 217-224 Packer, H L and Thomas, C R ‘Morophological measurements on filamentous microorganisms by fully automatic image analysis’, Biotechnology & Bioeng., Vol 35 (1990) pp 870-881 Spencer, M Fundamentals of Light Microscopy, Cambridge University Press, Cambridge (1982) Lee, S U and Chung, S Y ‘A comparative performance study of several global thresholding techniques for segmentation’, Comput. Vision, Graph. & Image Process., Vol 52 (1990) pp 171190 Abdou, I E and Pratt, W K ‘Qualitative design and evaluation of enhancement/thresholding edge detectors’, Proc. IEEE (May 1979) pp 753-763 Canny, J ‘A computational approach to edge detection’, IEEE Trans. PAMI, Vol 8 (1986) pp 679-698 Spacek, L A ‘Edge detection and motion detection’, Image & Vision Cornput., Vol 4 (1986) pp 43-56 Nobel, J A ‘Finding half boundaries and junctions in images’, Image & Vision Comput., Vol 10 (1992) pp 219-232 Harlick, R M ‘Ridges and valleys on digital images’, Comput. Vision, Graph. & Image Process., Vol 22 (1983) pp 28-38 Zhang, T Y and Suen, C Y ‘A fast parallel algorithm for thinning digital patterns’, Commun. ACM, Vol 27 No 3 (1984) pp 236-239
image and vision computing