An Image-Processing Strategy for the Segmentation and Quantification of Microaneurysms in Fluorescein Angiograms of the Ocular Fundus

COMPUTERS AND BIOMEDICAL RESEARCH ARTICLE NO.

29, 284–302 (1996)

0021

An Image-Processing Strategy for the Segmentation and Quantification of Microaneurysms in Fluorescein Angiograms of the Ocular Fundus TIMOTHY SPENCER,* JOHN A. OLSON,† KENNETH C. MCHARDY,† PETER F. SHARP,* AND JOHN V. FORRESTER‡ *Department of Bio-Medical Physics and ‡Department of Ophthalmology, University of Aberdeen, Aberdeen, Scotland, United Kingdom; and †Diabetic Clinic, Woolmanhill, Aberdeen, Scotland, United Kingdom

Received May 22, 1995

Digital image-processing techniques can provide an objective and highly repeatable way of quantifying retinal pathology. This study describes an image-processing strategy which detects and quantifies microaneurysms present in digitized fluorescein angiograms. After preprocessing stages, a bilinear top-hat transformation and matched filtering are employed to provide an initial segmentation of the images. Thresholding this processed image results in a binary image containing candidate microaneurysms. A novel region-growing algorithm fully delineates each marked object and subsequent analysis of the size, shape, and energy characteristics of each candidate results in the final segmentation of microaneurysms. The technique is assessed by comparing the computer’s results with microaneurysm counts carried out by five clinicians, using Receiver Operating Characteristic (ROC) curves. The performance of the automated technique matched that of the clinicians’ analyses. This strategy is valuable in providing a way of accurately monitoring the progression of diabetic retinopathy.  1996 Academic Press, Inc.

1. INTRODUCTION Diabetic eye disease is the commonest cause of blindness in the UK for the age group 20 to 65 (1). Since the advent of laser therapy, panretinal photocoagulation has been effective in controlling the growth of new vessels in proliferative diabetic retinopathy, significantly reducing the incidence of sight loss. However, the commonest cause of visual impairment is from diabetic maculopathy, for which laser treatment is less effective (2). In order to evaluate treatment regimens which aim to check the progression of the disease at an earlier stage, the quantification of early diabetic retinopathy is of major importance (3). Microaneurysms are considered to be the earliest clinically detectable lesion of diabetic retinopathy and are indirect evidence of ischaemia, since each microaneurysm can be thought of as representing the occlusion of at least one 284 0010-4809/96 $18.00 Copyright  1996 by Academic Press, Inc. All rights of reproduction in any form reserved.

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

285

capillary. There is a positive correlation between the number of microaneurysms and both the severity and likely progression of retinopathy (4, 5). Initially, microaneurysm counting protocols were developed to monitor the progression of retinopathy during pharmacological trials (6, 7). However, manual counting of microaneurysms is time-consuming and open to observer error. Automated counting techniques, based on digital image-processing, offer a more objective and highly repeatable way of quantifying retinal pathology. Previous automated techniques have relied on global image-processing operations to segment microaneurysms from the background fundus and vasculature (8, 9). The processed images were thresholded at a grey-level which was, typically, a compromise between that which was low enough to detect all the microaneurysms and that which did not detect any spurious features. Binary morphological processing and shape analysis of objects in the thresholded images promoted further discrimination between microaneurysms and linear features, such as small vessel sections, improving the specificity of the techniques. Monitoring the progression of retinopathy requires a serial study of each patient-eye. Typically, images are obtained months apart. A prerequisite of any microaneurysm counting procedure is that exactly the same area of the fundus is analyzed for each patient-image: misregistration of areas-of-interest may result in erroneous microaneurysm counts. Computer-based techniques can process the same area in successive images by changing the position and shape of a region-of-interest (ROI) or by manipulating the image itself to achieve spatial registration. In this paper we present a processing strategy which segments and quantifies microaneurysms in digitized fluorescein angiograms. In addition to the use of global image-processing, we describe a novel region-growing technique which accurately delineates all those objects which could be interpreted as microaneurysms. Subsequent analysis of both the morphology and intensity characteristics of these ‘‘candidate’’ lesions enables microaneurysms to be discriminated from other microvascular abnormalities and from texture in the background retina. 2. MATERIALS Fluorescein angiograms are images taken of the fundus of the eye while a fluorescent dye passes through the blood vessels of the retina, the dye having been injected intravenously into the arm of the patient. Typically, images obtained with a fundus camera are captured on 35-mm film. Frames taken consecutively after the injection of the dye display different vascular features and are routinely used to study retinal circulation and pathology. In this study, fluorescein angiogram negatives were digitized by back-illuminating the film strip and recording the image of each frame with a 1320 3 1024 element Kodak MegaPlus C.C.D. (charge-coupled device) camera. The camera’s direct digital output was fed to the frame-grabber module of a Series-151 image-processing system (Imaging Technology Inc., U.S.A.) which enabled each frame to be captured at a resolution of 1024 3 1024 1-byte pixels. The system was controlled by a SPARCstation IPX

286

SPENCER ET AL.

FIG. 1. A digitized fluorescein angiogram frame of resolution 1024 3 1024 pixels. The image covers a 308–408 field of the fundus, centered on the fovea, corresponding to an area of p8 3 8 mm.

(Sun Microsystems, U.S.A.) through Visilog image-processing software (Noesis, France). Once digitized, images were stored on rewritable optical disks, using an LD 520 optical disc drive (Laser Magnetic Storage Int. Co., U.S.A.). A digitized fluorescein angiogram image is shown in Fig. 1. In this image of the back-illuminated negative, areas of hyperfluorescence (for example the blood vessels) appear dark in comparison to areas of little or no fluorescence. For the grey-levels in the digital image to be proportional to the intensity of the fluorescent dye they have to be intensity-inverted: analogous to producing a positive print from a film negative. In this paper, images are in the negative state unless specifically stated as positive versions. A positive version of the image in Fig. 1 can be seen in the top-left quadrant of Fig. 3. For computer processing, the best quality frame was chosen from those frames immediately following the point at which the vasculature was completely filled with fluorescent dye. The resolution

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

287

FIG. 2. Image-processing and analysis flow-diagram. IC, illumination-corrected; BLTHT, bi-linear top-hat transformation; bgnd, background; MA, microaneurysm.

of the digitized images was approximately 8–9 em per pixel, measured with respect to an optic disc diameter of 1.5–1.8 mm. 3. COMPUTER TECHNIQUES 3.1. Image-Processing The processing strategy for the segmentation and quantification of microaneurysms is shown in the flow-diagram of Fig. 2 and is described in detail below. 3.1.1. Illumination-uniformity correction. For each batch of fluorescein angiogram negatives digitized consecutively, a ‘‘flood’’ image was also captured. This was an image taken with no negative in the film-strip holder and it showed nonuniformities in both the illumination of the negative and in the lenses of the digitizing system. In order to correct for intensity-variations in the digitized angiogram image caused by the nonuniformities of the imaging system, the angiogram image was divided by this flood image and then rescaled to match the original image. Having removed these artefactual variations in image intensity, each uniformity-corrected image became the basis of all subsequent imageprocessing and analysis (henceforth to be referred to as the original image). 3.1.2. Shade-correction. Microaneurysms are part of the retinal vascular network, each one being attached to the capillary from which it has grown. Although it is possible to see the complete capillary network in fluorescein angiograms, often the finest vessels are not visible (retinal capillary lumen diameter is 3.5–6 em). This can be due to the optical properties of the patient’s eye (e.g., cataracts) or the difficulty in maintaining a sharply focussed image throughout the investigation. Furthermore, the criteria for the choice of frame for digitization may result

288

SPENCER ET AL.

FIG. 3. Image-processing stages for microaneurysm detection. The top-left quadrant shows the illumination-corrected image (positive version). The subsequent shade-corrected image is shown in the top-right quadrant. The result of applying an 11-pixel bilinear top-hat transform to the shadecorrected image is shown in the bottom-left quadrant. The bottom-right quadrant shows the result of applying an 11 3 11-pixel microaneurysm-matched filter to the previous image.

in good microaneurysm visibility but poor capillary contrast, due to washout of dye from the vessels and the retention of fluorescein by microaneurysms. For this reason, microaneurysms may well appear as isolated hyperfluorescent spheres because the capillary to which they are attached is not visible (see Figs. 1 and 3). Therefore, fluorescein angiogram images can be thought of as having two main components: the visible vascular retinal features (microaneurysms and major vessels) and the background retina (whose intensity gradually changes across the image). The texture of the background retina is largely dictated by the distribution of both the capillary bed and the pigmentation of the retinal pigment epithelium. Shade-correction was used to separate the retinal features

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

289

from the background and was achieved by subtracting an image approximating to the background from the original image. This background image was obtained by smoothing the original image with a median filter whose size was significantly greater than the largest retinal feature (9). In addition, a positive version of the background image was stored for use during region-growing and object analysis. A shade-corrected image (positive version) can be seen in the top-right quadrant of Fig. 3. 3.1.3. Bilinear top-hat transformation. Microaneurysms were segmented from the vasculature in the shade-corrected image using the bilinear top-hat transformation (10). This grey-scale morphological transformation was well suited to the task of segmenting microaneurysms from the arterioles and venules as it gave a high degree of discrimination between circular and linear features. A top-hat transformed image is shown in the bottom-left quadrant of Fig. 3. The transformation is based on a linear ‘‘structuring element’’ whose length is chosen to be just greater than the diameter of the largest circular feature to be segmented from the image. However, the longer the structuring element is, the less able it is to fit into tightly curving vessel sections, so leaving residues of vessels in the segmented image (11). Empirically, it was found that an 11-pixel linear structuring element gave the best compromise between the complete segmentation of larger microaneurysms and the best tracking of retinal vessels. The shade-corrected image was morphologically ‘‘opened’’ (12, 13) with the linear structuring element at different rotational orientations. The greater the number of rotational positions, the more likely there was to be an orientation for which the structuring element was exactly aligned with vessel sections of a given orientation. However, there was a significant increase in computation required to open an image at orientations other than the two orthogonal and two diagonal positions readily implemented on a square grid, due to the need for bilinear interpolation. To save computation time, the four main orthogonal/ diagonal orientations were used, but were augmented by 308, 608, 1208, and 1508 orientations after resampling the image for a hexagonal grid. These eight orientations were found to give a satisfactory segmentation of the images. Each opened image contained components of the vascular network corresponding to one of the eight orientations. The opened images were combined by taking the maximum pixel value at each spatial location, resulting in an image which contained vessel sections of all orientations, but with no circular features present. Finally, the top-hat operation was completed by subtracting this combined vessel image from the shade-corrected image, producing an image containing only circular features (microaneurysms). 3.1.4. Matched filtering. To improve the segmentation between microaneurysms and any remaining unwanted features, an 11 3 11 microaneurysm-matched filter of Gaussian cross-section (s 5 1 pixel) was applied to the top-hat transformed image (the result is shown in the bottom-right quadrant of Fig. 3). Although, ideally, a whole range of filters should have been used in order to optimize the detection of microaneurysms of all sizes, one filter size was found to be adequate for detecting all microaneurysms and significantly reduced the processing time

290

SPENCER ET AL.

(9). At any given location, the intensities of pixels in the resultant image were directly proportional to the degree of correlation between the matched-filter and the input-image pixels. In this respect, the image grey-levels gave an indication to the probability of a pixel corresponding to a microaneurysm. 3.1.5. Thresholding of processed image. The matched-filtered image was greylevel thresholded to produce a binary image. The threshold level was calculated to be a fixed number of grey-levels above the modal grey-level of each image. The number of grey-levels was selected so that all possible microaneurysms were detected. Spurious objects could be removed at a later stage but, clearly, any microaneurysms missing from the binary image could not be reinstated. The binary objects in the thresholded image were referred to as the candidate microaneurysms and, with the original grey-scale image, represented the input data to the region-growing algorithm. 3.2. Region Growing The region-growing algorithm took each binary object in the candidate-microaneurysm image and, using the original image to provide grey-scale information, delineated and filled-in the domain of the feature marked by the binary object. The original image was chosen to provide the grey-scale information, so that each grown binary object was directly related to the unprocessed data. A positive version of the original image was used so that the grey-levels were proportional to fluorescent intensity. The first stage of the algorithm was to reduce each discrete binary object to a single pixel so that the seed for the region-growing was located at the center of the feature it represented. The single-pixel seed was assigned to the object-pixel with the highest underlying grey-level, corresponding to the point of maximum hyperfluorescence in the retinal feature. The subsequent criterion, used to judge whether a particular neighboring pixel was to be included in the grown object, was related to the grey-level difference between the intensity of the single-pixel seed and the intensity of the spatially corresponding pixel in the background image (see shade-correction). This difference could be thought of as the greylevel ‘‘height’’ of the feature, with respect to the surrounding retina (see Fig. 4). The criterion for the inclusion of a pixel into the growing object was as follows: i $ ipeak 2 x.(ipeak 2 ibgnd ),

[1]

where i was the grey-level of the pixel under test; ipeak was the grey-level of the single-pixel seed; ibgnd was the grey-level of the background image at the same spatial position; x was a fraction between 0 and 1. In the above criterion, x dictated how much of the hyperfluorescent feature was to be included in the segmentation. For example, a value of x 5 0.75 would include all but the lowest 25% of the feature, with respect to its height. Depending on how accurately the background image represented the true grey-levels of the pixels immediately surrounding the feature, values of x approaching unity caused the region-growing to exceed the boundaries of the feature and grow indiscrimi-

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

291

FIG. 4. Typical intensity-profile of a microaneurysm, showing its grey-level ‘‘height’’ with respect to the background fundus. Ipeak , Ibgnd , and x indicate variables in the region-growing criterion.

nately into surrounding areas of the retina. To avoid this, values of x were chosen to be between 0.5 and 0.75, giving a satisfactory degree of delineation while avoiding most occurrences of over-growing. A value of x 5 0.5 was used to obtain the results of section 4. A 3 3 3 square structuring element was used to morphologically dilate (12) each binary seed in order to add to the seed its eight-connected neighbors. Pixels in the grown seed were tested for inclusion into the object and were accepted or rejected according to the above criterion. The augmented seed was then dilated and the cycle of dilation, testing, and accepting/rejecting repeated until the object stopped growing (i.e., when there were no more neighboring pixels which satisfied the criterion). Fully grown objects were accumulated in a separate binary image. The program placed an ROI around each growing-object so that the regiongrowing algorithm only operated in a small area of the image, significantly reducing processing time and avoiding growing-objects from encroaching on other objects which were in close proximity. If the domain of a grown object was found to overlap with that of another grown object already stored in the cumulative image, then it was stored in a second cumulative binary image to avoid the two objects merging. Four binary overlay planes were used to store objects at various stages of the growing process. 3.3. Analysis of Candidate Microaneurysms Objects in the two cumulative images (nonoverlapping objects and overlapping objects) were analyzed to finally segment the microaneurysms from other retinal features. A number of criteria were used to judge whether a grown object was

292

SPENCER ET AL.

indeed a microaneurysm, based on information derived from both its size and shape, using the delineated binary object, and its pixel grey-levels. 3.3.1. Binary object-parameters. Upper and lower limits were set for the size of the perimeter of a microaneurysm, in order to restrict lesions to an acceptable size range. Technically, there should be no lower limit set for the size of a microaneurysm appearing in a fluorescein angiogram. However, since the lesion gradually transmutes from a small swelling on the side of a capillary, to a classical spherical balloon with a diameter many times greater than its feeding vessel, a lower limit must be set to avoid the detection of other microvascular abnormalities and capillary sections seen end-on. In common with previous studies (4, 6, 7), we have only counted microaneurysms with diameters of 25 em or greater. The upper limit for perimeter was chosen to be that of the largest microaneurysm likely to appear in any image of this magnification. Aspect-ratio (defined as the ratio of length to breadth) was used as measure of the shape of an object. Classically, microaneurysms are spherical lesions (14) and so appear as hyperfluorescent circles in fluorescein angiograms (with an aspect-ratio of unity). However, many lesions are elliptical or are elongated in some way and so the maximum aspect ratio imposed on the objects had to tolerate these slight deviations from the ideal. Complexity of shape was measured by calculating perimeter2 /(4f.area) (15). An upper limit was set for the complexity of a microaneurysm, a circle having a complexity value of unity in the continuous case (in a digital image, values just below unity are possible). An upper limit of just over unity was set to enable microaneurysms which had a capillary ‘‘tail’’ attached to them, or which had a slightly irregular shape, to be included in the segmentation. 3.3.2. Grey-scale object-parameters. The mean ‘‘energy’’ of an object was calculated by summing the grey-levels of those pixels in the original image (positive version) which coincided with the overlying binary object and dividing this total energy by the area of the object. A minimum mean-energy level was used to reject objects which did not have enough fluorescence (with respect to their size) to be considered microaneurysms. The total energy of a retinal object contains a contribution from the background fluorescence at that location in the fundus. Therefore, a microaneurysm located in an area of high background fluorescence displays a greater energy than an identical microaneurysm situated in an area of low fluorescence, such as the fovea. A measure of object energy independent of the background fluorescence was obtained by calculating the difference in the mean-energies of the object and the background retina at that location. The mean-energy of the background retina was obtained by summing the grey-levels of the pixels in the background image which coincided with the object, and dividing this total energy by the number of pixels in the object. The mean-energy difference was then calculated by subtracting the mean background energy from the mean object energy. This parameter effectively gave a measure of the fluorescence of an object with respect to the background fluorescence at that point, that is to say a measure of object contrast. A minimum difference was set so that only features with a significant

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

293

energy-contrast were accepted as microaneurysms: objects with low energy-differences often corresponded to patches of retinal texture. 3.4. Combining Criteria for Segmenting Microaneurysms from Candidate Objects In order to find the correct combination of criteria which would satisfactorily define microaneurysms, as they appeared in digitized fluorescein angiograms, lesions from thirteen images were studied. Of the thirteen images taken from diabetic patients, seven displayed microaneurysms; the remaining images showed no lesions and appeared ophthalmoscopically normal. The angiogram negatives, and prints made from these, were used by two of the authors to identify microaneurysms which were to be used as a ‘‘teaching set.’’ The locations of these lesions were then identified in the digital images, their exact positions being marked on an overlying transparent sheet. The computer program was run for all the images, resulting in the storage of a binary image containing candidate microaneurysms associated with each original grey-scale image. For those images known to contain microaneurysms, the binary candidate-microaneurysm image was overlayed onto the original grey-scale image and those objects corresponding to microaneurysms in the teaching set were removed, to be stored in a separate definite-microaneurysm image. A data-collecting program was run which used the candidate-microaneurysm image (and definite-microaneurysm image if applicable) with the original and background images to tabulate all the relevant data associated with each object, i.e., size, complexity, aspect-ratio, mean energy, and energy difference. Histograms of the object-data were constructed and are shown in Figs. 5 and 6. The graphs show that no one parameter can be used to discriminate between definite microaneurysms and the other candidate objects. Histograms 5a–c, relating to size and shape data, show that larger, linear, and more complex objects can be easily discriminated from compact, circular microaneurysms whose sizes fall between well-defined limits. Figure 6 shows histograms relating to grey-scale parameters. Histogram 6a shows that the energy of a candidate object is not a good parameter for segmenting microaneurysms. The mean-energy of an object is a much better parameter for this, as can be seen in histogram 6b: microaneurysms have a high energy/unit-area compared to larger, less fluorescent features. The difference between the mean-energy of an object and its background can also be used to discriminate between microaneurysms and spurious, low-contrast features (as shown in histogram 6c). The data was also fed into a spread-sheet program. By studying the data, limits for object-parameters were identified which could be used to discriminate between definite microaneurysms and spurious objects. The spread-sheets were able to show which spurious objects failed to be removed (labeled as falsepositives, FPs) and which definite microaneurysms (true-positives, TPs) failed to be labeled as such (resulting in false-negatives). The maximum and minimum parameter-values for microaneurysms were iden-

294

SPENCER ET AL.

FIG. 5. Histograms from six images showing the distribution of binary object-parameters for microaneurysms and other candidate features. The individual histograms are as follows: 5a, perimeter histogram; 5b, aspect-ratio histogram; 5c, complexity histogram.

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

295

FIG. 6. Histograms from six images showing the distribution of grey-scale object-parameters for microaneurysms and other candidate features. The individual histograms are as follows: 6a, energy histogram; 6b, mean-energy histogram; 6c, mean-energy difference histogram.

296

SPENCER ET AL. TABLE 1 TRUE-POSITIVE AND FALSE-POSITIVE RATES AS OBTAINED BY SIX FILTER COMBINATIONS APPLIED TO ALL THIRTEEN IMAGES Filter 1

Image A B C D E F G H I J K L M Totals % Sensitivity % Specificity % FP rate a

TPs FPs

Filter 2

Filter 3

Filter 4

TPs FPs TPs FPs TPs FPs

Filter 5

Filter 6

Totals

TPs FPs TPs FPs MAs a Objects

18 11 16 13 2 0 0 0 0 0 1 0 0

77 14 37 29 4 34 106 95 95 52 36 43 110

18 10 12 10 2 0 0 0 0 0 1 0 0

31 6 10 14 1 6 19 30 16 9 5 5 23

18 10 11 10 2 0 0 0 0 0 1 0 0

24 6 8 14 1 5 13 22 11 8 5 3 18

18 10 10 10 2 0 0 0 0 0 1 0 0

21 4 7 12 1 3 11 15 7 6 2 1 15

16 8 8 9 2 0 0 0 0 0 1 0 0

12 3 4 7 1 0 5 8 2 1 0 0 6

13 7 6 7 2 0 0 0 0 0 1 0 0

6 2 2 2 0 0 0 0 0 0 0 0 0

18 17 19 17 2 0 0 0 0 0 1 0 0

546 33 310 231 20 314 963 516 528 368 293 449 509

61

732

53

175

52

138

51

105

44

49

36

12

74

5080

82.4 85.6 14.4

71.6 96.6 3.4

70.3 97.3 2.7

68.9 97.9 2.1

59.5 99.0 1.0

48.6 99.8 0.2

MAs, microaneurysms.

tified. These were logically combined as a filter, so that an object had to satisfy all the parameter-limits for it to be a microaneurysm. This rule is shown in the Boolean statement of Filter 1: Filter 1: (10 # p # 62) & (ar # 2.3) & (c # 2.1) & (me . 114) & (med . 14) & (e . 2180) & (ed . 30) where p is an object’s perimeter, in pixel-units; ar is aspect-ratio; c is complexity; me is mean energy, in grey-levels; med is mean-energy difference; e is energy; and ed is energy difference. A logical AND is represented by &. By definition, this filter correctly identified the microaneurysms present in the candidate-microaneurysm image. However, as Table 1 shows, a significant number of other objects were also labeled as microaneurysms in error (falsepositives). A more stringent filter was made by ‘‘tightening’’ the parameter-limits and is described below: Filter 2: (10 # p # 60) & (ar # 1.6) & (c # 1.1) & (me . 140) & (med . 16) & (e . 2180) & (ed . 30) This filter reduced the false-positive rate, but also reduced the sensitivity (truepositive rate). This indicated that more complicated filters had to be used if a

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

297

more accurate segmentation was to be achieved. The spread sheets were studied further to find suitable rules, the aim being to find a filter combination which accepted all the definite microaneurysms while rejecting all the remaining candidates (i.e., 100% TP rate and 0% FP rate). The other filters are described below and their TP and FP rates are shown in Table 1: Filter 3: Filter 2 and one of Rules R1–R4 had to be satisfied Filter 4: Filter 2 and two of Rules R1–R4 had to be satisfied Filter 5: Filter 2 and three of Rules R1–R4 had to be satisfied Filter 6: Filter 2 and four of Rules R1–R4 had to be satisfied Rule 1: (med . 30) OR (me . 210) Rule 2: ( p . 20) OR (ar , 1.3) Rule 3: (med . 30) OR (ar , 1.4) Rule 4: [( p . 20) & (med . 20)] OR [ed . 750] Another, less stringent, filter was constructed to obtain a seventh TP : FP pair for the computer technique: Filter 7: (10 # p # 62) & (ar # 2.3) & (c # 2.1) & (me . 50) & (med . 14) & (e . 200) & (ed . 300) Filter 7 was applied to a candidate-microaneurysm image derived from a binary image obtained using a lower threshold of the matched-filtered image, in an attempt to detect some of the microaneurysms previously missed. These filters, utilizing all the object-parameters combined in different ways, were applied to the spread-sheets to assess their effectiveness at segmenting the definite-microaneurysms from the remaining candidates. Table 1 shows the TP and FP rates for the thirteen images after the application of Filters 1–6. The table shows that there is no one filter which could be considered superior in its segmentation of the candidate-microaneurysms, there being a direct trade-off between sensitivity and specificity. Depending on the application, a filter could be chosen to suit a particular requirement relating to high specificity, sensitivity, or minimum error. The optimum filter combination would then be implemented in the original program so that definite microaneurysms were automatically segmented from the candidate-microaneurysm image: objects not satisfying the criteria would be removed from the binary image and the remaining objects, deemed to be microaneurysms, counted and this number stored in a results file. 3.5. Processing of Sequential Images To accurately monitor changes in the numbers of microaneurysms in images derived from different investigations of the same patient-eye, it is essential that exactly the same area of the fundus is analyzed in each case. Positioning an ROI to cover the same area of the fundus in successive images is a difficult task, because optical distortions in the images mean that irregularly shaped ROIs would have to be used. A better approach is to spatially register the images and

298

SPENCER ET AL.

then use the same ROI, at the same position, for all images. The images were registered interactively, using the Visilog ‘‘warping’’ facility: 6 to 10 points were marked on the reference image at anatomically significant points and these were also located on the image to be warped. The quadratic equation, which described the mapping of the reference points to the markers on the second image, was applied to the second image, which was subsequently warped to match the reference image. Similarly, the same warping transformation was applied to the matched-filtered image associated with the second image. It was not sufficient to simply translate, rotate, and magnify the second image to match the first, because the optical properties of the eye and the fundus camera are such that there can be a significant amount of geometric distortion in the periphery of the images, particularly in those taken with a large field of view. The distortions are effectively local changes in magnification. Once registered, a 512 3 512 pixel ROI was defined in the reference image and second image at the same location, nominally centered on the fovea. To achieve a degree of parity between their overall image contrasts, which may have differed due to different photographic exposures, the image contrast was adjusted so that the two images had the same minimum and maximum greylevels. Region-growing was then carried out on these stretched original images. The objects remaining after the analysis and Boolean filtering of the regiongrown objects, i.e., those deemed to be definite microaneurysms, were stored for each of the pair of registered images. One of the possible problems resulting from discrepancies in contrast between images in a pair, is that the threshold level for one of the matched-filtered images may be too high; i.e., after the thresholding stage, some microaneurysms detected in one image are not present in the other. To make sure that a grown object appearing in one image, but not in another, was genuinely due to a microaneurysm disappearing (or appearing) and not to an error in the processing, each grown microaneurysm was used as a seed for region-growing in the other image of the pair. If the resultant object satisfied the microaneurysm criteria then it was added to the cumulative binary image containing the other microaneurysms for that image. 3.6. Assessment of Computer Technique To assess the computer technique, microaneurysm counts by both computer and clinician were compared. High quality 100 3 80 prints of the four images containing the greatest number of microaneurysms (images A, B, C, and D in Table 1) were made from the original angiogram negatives. These prints were given to five ophthalmologists (two consultants and three registrars), who were asked to locate and mark each lesion within an ROI (P4 3 4 mm). These images were chosen because they displayed the greatest range of microaneurysms in terms of size, shape, and brightness (energy). The automated counting technique operated within the same ROI (512 3 512 pixels) and located microaneurysms using the seven filter combinations described in section 3.4.

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

299

FIG. 7. Free Response ROC curves for four images analyzed by both clinicians and computer. Each square represents the combined microaneurysm-count from the four images by one clinician. The triangles represent the filters 1–7 (right to left, respectively) as described in section 3.4.

4. RESULTS The clinicians’ and computer’s results were judged with respect to gold standards compiled by the authors from the 100 3 80 prints of the negatives A, B, C, and D (see Table 1). The results were plotted as a Receiver Operating Characteristic (ROC) curve, showing how the true-positive rate varied with falsepositive rate for each observer (16). A ‘‘free response’’ ROC curve had to be used because the number of false-positives detected could not be expressed as a percentage (the total number of possible false-positive responses being unknown) (17). The computer’s results were shown by plotting true-positive : false-positive pairs for different filter combinations. The results can be seen in Fig. 7. The computer’s performance satisfactorily matched that of the clinicians’ results. The ceiling of 82% sensitivity reached by the computer technique was due to there being 13 microaneurysms absent from the candidate-microaneurysm images. These lesions were not detected at the thresholding stage because of the following reasons: 4 microaneurysms were lost from merging with adjacent lesions; 4 microaneurysms were too big and diffuse (due to excessive leaking of fluorescein) to be detected by the matched-filter; a conglomeration of 3 microaneurysms was too big to be detected by the bilinear top-hat transform or the matched-filter; and a conglomeration of two microaneurysms formed an object that was too linear to be segmented by the bilinear tophat transform. 5. DISCUSSION In common with many image-segmentation tasks, previous attempts at quantifying microaneurysms present in fluorescein angiograms have relied on global grey-scale operators to provide most of the discrimination between object and

300

SPENCER ET AL.

background pixels (8–10). Processed images have subsequently been grey-level thresholded at a level low enough to maximize the detection of microaneurysms, while being high enough to avoid the inclusion of large numbers of spurious background features. This has resulted in the inevitable conflict between high sensitivity (requiring low thresholds) and high specificity (requiring relatively high thresholds). Threshold levels were invariably chosen to maximize sensitivity: binary morphological operators and shape analysis being employed in an attempt to increase specificity, by identifying those binary objects which were part of the vasculature but had been picked-up by the thresholding stage in error. Although, on the whole, this binary processing stage was effective at removing potential vessel-bound false-positives, genuine microaneurysms attached to vessels were also removed, in error. In addition, these processing strategies were not able to overcome the rejection of conglomerations of microaneurysms (because their overall shape was too complex), low contrast microaneurysms or leaky microaneurysms (which had a blurred outline). In the latter two cases the microaneurysms were not even present in the thresholded image because, to include them, the threshold would have been at a level which would have detected considerably more background noise resulting from retinal texture. If they had been included, any further analysis of the thresholded objects which could have led to their segmentation was hampered by the fact that they were only indirectly related to the underlying greyscale features. For example, a binary object may have represented a vessel residue, resulting from a bilinear top-hat transform, and so did not represent any genuine anatomical or pathological phenomenon. Even when a binary object did not merely represent a processing artefact, then it was unlikely that its pixels would correspond to the entire domain of the original grey-scale feature it marked. For example, the binary object representing a microaneurysm may only constitute a few pixels at the center of the lesion: typically, in a matched-filtered image, a few bright pixels will indicate the points of highest correlation. For this reason it would be inappropriate, for segmentation purposes, to use grey-scale information from those pixels in the original image corresponding to the binary object. The region-growing approach acknowledges the fact that binary objects, obtained by thresholding a processed image, can only serve the purpose of marking the location of those features which are most likely to be true microaneurysms. Having reduced each binary object to a single pixel seed at the center of the hyperfluorescent feature, the region-growing algorithm then delineates each feature, marking all the pixels within its domain. This results in a binary object whose size and shape are directly related to the underlying feature, facilitating the use of grey-level information for the purpose of characterizing and identifying microaneurysms or other pathology/anatomy. In all cases, vessel-connected microaneurysms were significantly brighter than the attached capillary and so the region-growing algorithm successfully segmented the feature from the image. Blob-like microvascular abnormalities displaying grey-levels similar to their connecting vessels were not included as mi-

IMAGE-PROCESSING STRATEGY FOR SEGMENT MICROANEURYSMS

301

croaneurysms because the seed grew into the vessel itself, giving a complex shape: such features were more likely to be dilated capillary sections than classical microaneurysms (14). In the same respect, conglomerations of microaneurysms could be correctly segmented as individual lesions if they displayed a high enough object : background contrast. Small, low contrast microaneurysms remained difficult to discriminate from small, discrete patches of background fluorescence. However, the region-growing technique was able to reject patches of texture if they had a sufficiently complex outline or if their mean energy (with respect to the background) was too low: the classical model of a microaneurysm (as a fluorescein-filled sphere) dictates that a lesion of a given diameter will hold a known amount of fluorescein, corresponding to a particular mean-energy with respect to the background fundus. 6. CONCLUSIONS Monitoring the progression of early diabetic retinopathy is a prerequisite for assessing the efficacy of new drugs and treatment regimens. Quantification of retinal pathology by digital image-processing offers an accurate and highly repeatable technique for monitoring fundal changes. The region-growing approach to the quantification of microaneurysms in digitized fluorescein angiograms represents an improvement on existing techniques, in terms of its true-positive rate for a given false-positive rate. The complete delineation of each microaneurysm, as opposed to a mere marking of its location, facilitates the use of other parameters, such as size, energy, and morphology, in the study of the natural history of these lesions. ACKNOWLEDGMENTS The authors thank Mr. George Cameron and Dr. Phillip Ross for their computing support; Mrs. Alison Farrow for the ophthalmic photography; and Mr. Raymond Hutcheon for photographing the computer images.

REFERENCES 1. GHAFOUR, I. S., ALLAN, D., AND FOULDS, W. S. Common causes of blindness and visual handicap in the West of Scotland. Br. J. Ophthalmol. 67, 209 (1983). 2. KOHNER, E. M. Diabetic retinopathy. BMJ 307, 1195 (1993). 3. PHILLIPS, R. P., SPENCER, T., ROSS, P. G. B., SHARP, P. F., AND FORRESTER, J. V. Quantification of diabetic maculopathy by digital imaging of the fundus. Eye 5, 130 (1991). 4. KOHNER, E. M., SLEIGHTHOLM, M., AND KROC COLLABORATIVE STUDY GROUP. Does microaneurysm count reflect the severity of early diabetic retinopathy? Ophthalmology 93, 586 (1984). 5. KLEIN, R., MEUER, S. M., MOSS, S. E., AND KLEIN, B. E. K. The relationship of retinal microaneurysm counts to the four-year progression of diabetic retinopathy. Arch. Ophthalmol. 107, 1780 (1986). 6. BAUDOIN, C., MANESCHI, F., QUENTEL, G., SOUBRANE, G., HAYES, T., JONES, G., COSCAS, G., AND KOHNER, E. M. Quantitative evaluation of fluorescein angiograms: Microaneurysm counts. Diabetes 32 (Suppl. 2), 8 (1983).

302

SPENCER ET AL.

7. DAMAD STUDY GROUP. Effect of aspirin alone and aspirin plus dipyridamole in early diabetic retinopathy. A multicentre randomised controlled clinical trial. Diabetes 38, 491 (1989). 8. LAY, B., BAUDOIN, C., AND KLEIN, J.-C. Automatic detection of microaneurysms in retinopathy fluoro-angiogram. Proc. SPIE 432, 165 (1983). 9. SPENCER, T., PHILLIPS, R. P., SHARP, P. F., AND FORRESTER, J. V. Automated detection and quantification of microaneurysms in fluorescein angiograms. Graefe’s Arch. Clin. Exp. Ophthalmol. 230, 36 (1992). 10. LAY, B., AND BAUDOIN, C. Computer analysis of angiofluorographies. In ‘‘Proceedings of the 7th International Conference on Pattern Recognition’’ (M. D. Levine, Ed.), pp. 927–929. Comput. Soc. Press, USA, 1984. 11. SPENCER, T. Digital imaging of the retina. Ph.D. thesis, University of Aberdeen, Scotland, UK. 12. SERRA, J. Introduction to mathematical morphology. Comput. Vision. Graph. Image. Processing 35, 283 (1986). 13. STERNBERG, S. R. Greyscale morphology. Comput. Vision. Graph. Image. Processing 35, 333 (1986). 14. FRYCZKOWSKI, A. W., CHAMBERS, R. B., CRAIG, E. J., WALKER, J., AND DAVIDORF, F. H. Scanning electron microscope study of microaneurysms in the diabetic retina. Ann. Ophthalmol. 23, 130 (1991). 15. HARALICK, R. M. A measure for circularity of digital figures. IEEE Trans. Syst. Man. Cybern. SMC-4, 394 (1974). 16. METZ, C. E. ROC methodology in radiologic imaging. Invest. Radiol. 21, 720 (1986). 17. BUNCH, P. C., HAMILTON, J. F., SANDERSON, G. K., AND SIMMONS, A. H. A free response approach to the measurement and characterisation of radiographic observer performance. Proc. SPIE 127, 124 (1977).