- Email: [email protected]
Computer-assisted, interactive fundus image processing for macular drusen quantitation1
Computer-assisted, Interactive Fundus Image Processing for Macular Drusen Quantitation David S. Shin, MD, Noreen B. Javornik, MS, Jeffrey W. Berger, MD, PhD Purpose: To design and validate a software package to quantitate the area subtended by drusen in color fundus photographs for the conduct of efficient, accurate clinical trials in age-related macular degeneration. Design: Algorithm and software development. Comparisons with manual methodologies. Participants: Evaluation and testing on color fundus photographs from patient records and from eyes enrolled in the Choroidal Neovascularization Prevention Trial. Methods: Fundus photographs of eyes with drusen were digitized. The green channel was selected for maximum contrast and preprocessed with filtering and shade correction to minimize noise, improve contrast, and correct for illumination and background inhomogeneities. Local thresholding and region-growing algorithms identified drusen. Multiple levels of supervision were incorporated to maximize robustness, accuracy, and validity. Validation studies compared computer-assisted with manual grading by an experienced grader. Intraclass correlation coefficients were calculated as a measure of the concordance between manual and computerassisted fundus gradings. Main Outcome Measures: Drusen area and concordance with manual grading. Results: Automated supervised image analysis offers extreme robustness and accuracy. Most images were segmented with little or no supervision, with processing times on the order of 5 seconds. More complicated images required supervision and a total analysis time varying from 20 seconds to 5 minutes, with most of this time devoted to inspection and comparison. Interactive image processing affords arbitrarily close concordance with manual drusen identification, with calculated intraclass correlation coefficients of 0.92 and 0.93 for comparison of manual with automated, supervised grading by two observers. Conclusions: Automated supervised fundus image analysis is an efficient, robust, valid technique for drusen quantitation from color fundus photographs. This approach should prove useful in the conduct of efficient accurate clinical trials for age-related macular degeneration. Ophthalmology 1999;106:1119 –1125 Age-related macular degeneration (AMD) is the most common cause of severe visual loss in developed countries. Both evaluation and treatment of AMD are predicated on clinical examination, including careful analysis of fundus features, such as drusen, choroidal neovascularization (CNV), and retinal pigment epithelial atrophy. Patient management and clinical trials rely on image analysis, image comparison, and change detection. Laser photocoagulation of CNV is the only treatment of proven benefit for AMD.1 However, only a small fraction of patients are eligible for laser therapy, and the high rates of
Originally received: August 19, 1998. Revision accepted: February 26, 1999. Manuscript no. 98442. From Computer Vision Laboratory, Scheie Eye Institute, University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania. Presented in part at the Association for Research in Vision and Ophthalmology annual meeting, Fort Lauderdale, Florida, 1998, and at the American Academy of Ophthalmology annual meeting, New Orleans, Louisiana, October 1998. Supported by a Career Development Award from Research to Prevent Blindness, Inc. (JWB) and NIH K-08 EY00374 (JWB). The authors have no proprietary interest in any aspect of this study. Reprint requests to Jeffrey W. Berger, MD, PhD, Retina Service Scheie Eye Institute, 51 North 39th St., Philadelphia, PA 19104. E-mail: [email protected].
persistence and recurrence of CNV after laser therapy limit the efficacy of laser photocoagulation for preservation of visual function. Moreover, although a subset of patients with subfoveal CNV are eligible for laser therapy, these patients will suffer irreversible loss of central vision after foveal laser photocoagulation. Accordingly, major efforts are underway to prophylax against the development of the vision-limiting sequelae of AMD, particularly CNV and retinal pigment epithelial atrophy, and to treat existing CNV with alternate, perhaps more selective, forms of therapy. Eyes with drusen may enjoy excellent visual function but are at significant risk for the development of CNV and retinal pigment epithelial atrophy.2 Stimulated by the observation that light laser application may provoke drusen resorption and predicated on the assumption that drusen reduction may confer protection from the vision-limiting sequelae of AMD, a number of groups are exploring the utility of laser prophylaxis.3– 6 Indeed, the National Eye Institute recently announced its intention to fund the Complications of AMD Prevention Trial, a planned multicenter clinical trial to evaluate prophylactic laser therapy for eyes with bilateral, large drusen at risk for vision loss attributable to the development of CNV and geographic atrophy. Quantitation and highly accurate image comparison of fundus features in AMD, including drusen, will potentiate efficient and accurate clinical trials. However, current meth-
1119
Ophthalmology Volume 106, Number 6, June 1999 ods for evaluating fundus features in AMD are subjective, laborious, and semiquantitative at best.6 –9 Computer-assisted image analysis offers the potential for greater accuracy, objectivity, and reproducibility, but various challenges make robust algorithm design nontrivial. Prior efforts have resulted in algorithms with suboptimal performance.10 –12 Nevertheless, improvements in computer performance and advances in algorithmic design have facilitated progress toward the design and implementation of computerized reading centers for coordination of clinical trials.13–16 In this report, we describe the design, implementation, and validation of an interactive image processing approach for robust accurate segmentation and quantitation of macular drusen in color fundus photographs.
Methods Acknowledging the limitations of computer vision algorithms in unconstrained environments, we have been developing computerassisted techniques for robust, accurate, reproducible fundus feature quantitation and change detection.13–16 The paradigm of interactive image processing allows for highly robust, accurate function, capitalizing on computer performance with various levels of supervision.17 Computers serve well to count and quantify, but, depending on the image quality and features, are less accurate at feature identification. Ideally, complete automation allows for total objectivity and reduced user effort. However, spurious results may be generated by complete, autonomous, computerized analysis. We suggest that robustness, accuracy, and repeatability are priorities in the design of a computer-assisted fundus image analysis system. A small reduction in automation and objectivity is acceptable if gains are realized in accuracy, robustness, and repeatability; a system that is autonomous but only 90% accurate is less valuable than a partially supervised approach affording nearly 100% accuracy. We propose that “automated, supervised” image analysis offers maximum robustness, accuracy, and reproducibility in fundus image analysis, and we describe a custom, application-specific, in-house-developed software package for drusen quantitation.
Image Selection and Digitization Color fundus photographs were obtained from patient records and from the Reading Center of the Choroidal Neovascularization Prevention Trial (CNVPT). For algorithmic development purposes, high-quality, high-contrast images were selected initially, with subsequent development enabled by analysis of poorer quality images. Validation studies were performed on images selected at random from CNVPT records. Thirty-five millimeter color slides were digitized at 1000 pixels per inch, 8 bits per color channel, 3 color channels per pixel (red, green, blue [RGB]), with a Microtek ScanMaker 35t slide scanner (Redondo Beach, CA).
Preprocessing Routinely acquired fundus photographs are of highly variable quality with components of random image noise as well as uneven illumination and lens edge artifacts. The objective of image preprocessing is to minimize noise, maximize contrast, and correct for uneven illumination, lens edge artifact, and natural background pigmentation heterogeneity to prepare optimally the image for subsequent segmentation—the extraction of features of interest, in this case, drusen.18 –23 Color images may be decomposed into
1120
various color representations including RGB, CYMK (cyan, yellow, magenta, black), IHS (intensity, hue, saturation), and others. Clinical experience suggests that drusen are often better discerned on monochromatic “red-free” (green) images when compared with color fundus photographs. We have explored drusen visibility as a function of color image decomposition in various color spaces. In no case was contrast improved when compared with the green channel of the color fundus photograph (JW Berger, DS Shin, RS Kaiser, unpublished results, 1997). Moreover, and perhaps somewhat surprisingly, monochromatically acquired red-free images were not superior to images derived by extraction of the green channel in RGB color space. Accordingly, for maximum contrast, the green channel was selected (Fig 1). Image noise was reduced by application of a 3- ⫻ 3-pixel mean filter followed by 5- ⫻ 5-pixel median filter. These operations were chosen empirically to produce an image with reduced noise suitable for segmentation. Shade correction for uneven illumination and pigmentation was accomplished by dividing the original image by the image convolved with an 85- ⫻ 85-pixel mean filter. The latter image is one without sharp detail but with local brightness, reflecting slowly varying brightness differences over the image, as might result from pigmentation and illumination heterogeneity across the image. Each pixel in the original image could then be normalized with respect to the local brightness image as represented by the image convolved with a large mean filter. The size of the mean filter was chosen empirically to be large enough to average high-frequency intensity fluctuations, as might be associated with fundus features including vessels and drusen, yet smaller than the image size to capture some degree of low-frequency brightness variation.
Segmentation The region of interest for segmentation was identified by manually identifying the fovea with a mouse click. The user could then specify a 1500-m radius, 3000-m radius, or variably sized, user-selected circular region of interest. Other noncircular regions of interest could be specified but were not found to be useful for clinical grading and comparison. Analyses in this report reflect quantitation within an approximately 1500-m radius region of interest. Drusen segmentation was performed on the preprocessed, green channel image. To improve the ability of our algorithms to detect drusen, multiple local-thresholding operations were performed rather than a single global threshold operation. The region of interest was divided into smaller square areas varying from 20 to 100 pixels on a side in increments of 10 pixels. Hence, each pixel was analyzed as part of a square area (or that part of the square area within the region of interest) varying in size from 20 to 100 pixels. Empirically, it was determined that drusen were present only if the skewness24,25 of the set of pixel intensities within a local area was greater than ⫺0.5. A positive skew reflects a greater preponderance of higher intensity outliers—the drusen. Accordingly, only those areas with skewness greater than ⫺0.5 were analyzed, and the threshold was set as the mean pixel value across the region of interest plus a term proportional to the product of the local area pixel brightness standard deviation.
Supervision Two levels of interactivity are incorporated. First, the user may adjust a slider to control the overall “sensitivity,” in which the sensitivity value is inversely proportional to the excess of the local threshold compared to the mean pixel value in the local area. Therefore, a low sensitivity increases the value of the threshold, and fewer pixels are identified. Second, for more complex images,
Shin et al 䡠 Computer-assisted Drusen Quantitation the user may add or remove drusen erroneously segmented or add and remove pixels by mouse. An intensity-based region-growing algorithm was also implemented to expand incompletely identified drusen. Specifically, for some drusen, only the brightest central core was identified, with the periphery of the druse falling below the threshold and therefore not included. Pixels were added to the identified drusen if the neighboring pixels were sufficiently similar to the drusen pixels. The similarity criterion was adjustable with a slider.
Change Detection The ability to quantify change depends on a highly accurate comparison of fundus photographs over corresponding regions of interest. The region of interest specified in one fundus photograph may be stored and applied to follow-up photographs after accurate image registration.15 The change in the area subtended by drusen may then be quantified longitudinally.
Validation Method validation requires demonstration of intraobserver reproducibility and high correlation between drusen identified manually by an experienced grader and the output of our computer algorithm. Ten images of five patients randomly selected from the patient records of the CNVPT were analyzed. Two images of each eye separated in time by 1 year were selected, with the better image of each stereoscopic pair chosen for analysis. One of us (NBJ), an experienced fundus photograph grader, quantitated the area subtended by drusen within a 1500-m radius circular region of interest with two methods. First, the digitized image was displayed on a computer screen and custom, separate software allowed for totally manual grading. The grader outlined each druse manually with a mouse on a 21-inch computer monitor while viewing repeatedly the photographic stereo pair on a lightbox located directly adjacent to the computer. The grader outlined all drusen; the software merely totaled the number of pixels and the fractional area subtended within the region of interest corresponding to the manually identified drusen. We term this value as manual. Second, the software identified the drusen without supervision on the scanned photographs. This parameter is termed automated, unsupervised. Third, the grader was able to supervise fully the process by adding or removing pixels and/or adding or removing drusen by mouse as facilitated by our graphic user interface. The results of this grading are termed automated, supervised. For comparison, a second grader performed automated, supervised analysis independently. Each grading was performed independently without knowledge of the results of prior gradings. Intraclass correlation coefficients were calculated as a measure of the concordance between manual and computer-assisted gradings.24
Results Segmentation Figure 2A depicts a typical image for drusen segmentation. The preprocessed image is shown in Figure 2B, and the results of automated segmentation are depicted in Figure 2C. Drusen occupy 17% of the circular region of interest. Note that preprocessing removes artifacts, maximizes contrast, and enhances drusen visibility for subsequent segmentation. For most images, the software performs quite well in an automated fashion without supervision.
Total computation time for automated analysis is approximately 5 seconds.
Supervision The graphical user interface for supervision of the segmentation process is demonstrated in Figures 3 and 4. Capabilities are incorporated to select an image for analysis, define the region of interest, and interact with the segmentation process by adding or removing pixels as outlined by mouse, removing a druse by clicking anywhere within the errantly identified drusen, and adding a druse with region-growing algorithms. The user may perform automated analysis, compare the output with grading/clinical judgment, and then readjust the sensitivity iteratively to produce results commensurate with subjective drusen identification. More detailed supervision including add/remove pixels/drusen can be performed at any time. The user can toggle easily between the original fundus photograph, the preprocessed image, and the color image with drusen identified to facilitate inspection and supervision. A typical color fundus photograph is shown in Figure 3A, the region of interest is defined in Figure 3B, the preprocessed image is shown in Figure 3C, and the results of automated segmentation are shown in Figure 3D. A large, central druse with pigment figures confounds automated drusen recognition (Figs 3C, 3D). Figure 4 illustrates supervision, allowing for accurate drusen quantitation of the image depicted in Figure 3. Pixels erroneously identified as drusen are removed (Figs 4A, 4B), and menu-driven interactivity allows for mouse-controlled supervision and refinement (Fig 4C). Here, drusen occupy 43% of the region of interest. Depending on the level of interaction required for supervision, the total time required (not computer time, but real time) for complete analysis was approximately 1 minute (range, 20 seconds for straightforward supervision to approximately 5 minutes for complicated images).
Change Detection The software allows for straightforward overlay of a region of interest in one photograph onto a follow-up photograph for longitudinal comparison. Corrections are included to account for discrepancies in rotation, translation, and magnification between the two images.15 An example of longitudinal comparison is depicted in Figure 5. The eye depicted in Figure 2 was followed. At 12 months, there is visible central drusen reduction (Fig 5A, compare with Fig 2A). The region of interest from the analysis in Figure 2 is stored and applied to the image in Figure 5A, allowing for accurate image comparison (Fig 5B). The area subtended by drusen is reduced from 17% to 7%.
Validation Table 1 summarizes validation data. The column for unsupervised segmentation is presented for comparison purposes only; no validity is assigned to unsupervised analysis. The manual and supervised columns allow for comparison between the current gold standard—manual grading by an expert observer—and our algorithms for automated, supervised drusen segmentation. The last column, area subtended by drusen as quantified by our software under the supervision of a second grader, allows for evaluation of interobserver variability. Based on all ten images tabulated (Table 1), intraclass correlation coefficients between manual and automated, supervised gradings were 0.93 and 0.92, respectively, for the two supervisors. There is close agreement between manual and computer-assisted methods, and there is a high concordance between the two supervisors.
1121
Ophthalmology Volume 106, Number 6, June 1999 Figure 1. Representative color fundus photograph of an eye with age-related macular degeneration and drusen (A) with corresponding red (B) and green (C) channel projections. Note the increased visibility of the choroidal markings in the red channel partially obscuring the drusen. Greater contrast is noted in the green channel (C).
Figure 2. Color fundus photograph of an eye with agerelated macular degeneration and drusen (A). The preprocessed image (B) and the final drusen segmentation (C) in the circular region of interest are shown.
Figure 3. Example of supervision. The color fundus image is loaded into the graphical user interface-driven environment (A); the region of interest is selected (B); the image is automatically preprocessed, resulting in improved contrast and drusen visibility (C); and drusen are identified automatically (D). The large, central druse confounds accurate analysis. Supervision of this analysis is depicted in Figure 4. Colors surrounding the identified drusen are arbitrary. (Fig 3 continues.)
1122
Shin et al 䡠 Computer-assisted Drusen Quantitation
Figure 3. (continued).
Figure 4. Example of supervision (continued). Identified pixels not corresponding to drusen are circled (red, A) and removed (B). A menudriven interface allows for addition or removal of pixels or drusen or both (C). Here, the large, central druse is outlined with a mouse.
Figure 5. Image comparison. Same eye as depicted in Figure 2, now 12 months later (A). The region of interest from the earlier analysis (Fig 2) can be stored and overlaid onto the follow-up image after correcting for translation, rotation, and scale differences between the images (B). Drusen segmentation then allows for accurate image comparison (compare B with Fig 2C).
1123
Ophthalmology Volume 106, Number 6, June 1999 Table 1. Concordance between Manual and Automated Supervised Drusen Quantitation Manual Image (%) 1a 1b 2a 2b 3a 3b 4a 4b 5a 5b
13 3 61 27 36 36 34 45 19 27
Automated, Unsupervised (%)
Automated, Supervised (%)
Automated, Supervised (2nd observer, %)
8 2 34 41 40 34 28 28 8 10
9 2 60 31 42 34 35 45 10 (10)*
8 2 54 34 37 36 26 36 16 (16)*
The percent of the region of interest subtended by drusen is given for patients 1–5 initially (a) and following 12 months of follow-up (b). Data are presented for manual, automated, unsupervised and automated, supervised grading by an experienced fundus grader, and for automated, supervised analysis by a second observer. * Image 5b was of poor quality.
Discussion Robust, computerized quantification of pathology in AMD will potentiate epidemiologic studies and clinical trials. Current techniques for drusen area measurement require the observer to estimate the area subtended by drusen in standard regions of interest.6,7,9 However, these methods are coarse, highly subjective, laborious, and poorly suited to image comparison. Several groups have attempted automated segmentation and quantitation of macular drusen.10 –12 However, acceptable performance has heretofore not been achieved. For example, Kirkpatrick et al10 compared their algorithm to manual counting. The authors achieved a sensitivity of 60% and 35% for color fundus photographs and scanning laser ophthalmoscope images, respectively, when a specificity of 90% was enforced. Accordingly, computer-assisted methodology has not gained widespread acceptance and has not yet been incorporated into large clinical trials. Accepting the limitations of computer vision algorithmic performance in an unconstrained environment, particularly when exposed to images of poor contrast and quality, we propose a paradigm for automated, supervised fundus image analysis. In addition, we demonstrate the applicability of this approach to robust, reproducible, accurate image segmentation for drusen quantitation in AMD. Image preprocessing allows for correction of uneven illumination, poor contrast, and image noise and potentiates accurate drusen recognition (Figs 2–5). Our algorithms perform very well without supervision (Table 1); however, acceptance into the clinical environment requires nearly 100% accuracy and robustness. Therefore, tools for supervision are incorporated into the environment, permitting refinement of the segmentation. In theory, supervision affords segmentation, yielding arbitrarily close agreement with manual grading; with effort, the user may totally overhaul the output generated by automated drusen recognition.
1124
Differences between manual grading (in which the computer merely counts the user-identified areas) and computerassisted, automated, supervised analysis (Table 1) reflect intraobserver variability in repeated grading of fundus images. Recognizing the limited quantitation abilities of human graders, most studies invoke coarse grading schemes (e.g., estimating the area subtended by drusen as ⬍25%, between 25% and 50%, or ⬎50%). By incorporating supervision, our software capitalizes on both computer and human image feature recognition and the obvious superiority of computerbased methodologies for quantitation. Whereas an expert observer is superior to a computer in drusen recognition, particularly for complex drusen morphologies, computers are more accurate and faster in quantitation tasks. Automated, supervised fundus image analysis is an efficient, robust, valid technique for drusen quantitation from color fundus photographs. This approach should prove useful in the conduct of efficient, accurate clinical trials to evaluate novel therapeutic and prophylactic approaches for the vision-limiting complications of AMD. Acknowledgments. The authors thank the members of the Choroidal Neovascularization Prevention Trial Study Group; Stuart L. Fine, MD, Study Chairman; and Allen C. Ho, MD, Reading Center Director, for access to their data. In addition, the authors thank Maureen G. Maguire, PhD, for assistance with statistical analyses.
References 1. Berger JW, Maguire MG, Fine SL. The Macular Photocoagulation Study. In: Kertes PJ, Conway MD, eds. Clinical Trials in Ophthalmology: A Summary and Practice Guide. Baltimore: Williams & Wilkins, 1996; Chap. 5. 2. Bressler SB, Maguire MG, Bressler NM, Fine SL. Relationship of drusen and abnormalities of the retinal pigment epithelium to the prognosis of neovascular macular degeneration. The Macular Photocoagulation Study Group. Arch Ophthalmol 1990;108:1442–7. 3. Figuero MS, Regueras A, Bertrand J. Laser photocoagulation to treat macular soft drusen in age-related macular degeneration. Retina 1994;14:391– 6. 4. Guymer RH, Gross–Jendroska M, Owens SL, et al. Laser treatment in subjects with high-risk clinical features of agerelated macular degeneration. Arch Ophthalmol 1997;115: 595– 603. 5. Little H, Showman JM, Brown BW. A pilot randomized controlled study on the effect of laser photocoagulation of confluent soft macular drusen. Ophthalmology 1997;104:623– 31. 6. The laser treatment in eyes with large drusen. Short-term effects seen in a pilot randomized clinical trial. Choroidal Neovascularization Prevention Trial Research Group. Ophthalmology 1998;105:11–23. 7. Bird AC, Bressler NM, Bressler SB, et al. An international classification and grading system for age-related maculopathy and age-related macular degeneration. The International ARM Epidemiological Study Group. Surv Ophthalmol 1995;39:367–74. 8. Chamberlin JA, Bressler NM, Bressler SB, et al. The use of fundus photographs and fluorescein angiograms in the identi-
Shin et al 䡠 Computer-assisted Drusen Quantitation
9. 10.
11. 12. 13. 14. 15.
fication and treatment of choroidal neovascularization in the Macular Photocoagulation Study. The Macular Photocoagulation Study Group. Ophthalmology 1989;96:1526 –34. Klein R, Klein BEK, Jensen SC, Meuer SM. The five-year incidence and progression of age-related maculopathy: the Beaver Dam Eye Study. Ophthalmology 1997;104:7–21. Kirkpatrick JNP, Spencer T, Manivannan A, et al. Quantitative image analysis of macular drusen from fundus photographs and scanning laser ophthalmoscope images. Eye 1995; 9:48 –55. Morgan WH, Cooper RL, Constable IJ, Eikelboom RH. Automated extraction and quantification of macular drusen from fundal photographs. Aust N Z J Ophthalmol 1994;22:7–12. Peli E, Lahav M. Drusen measurement from fundus photographs using computer image analysis. Ophthalmology 1986; 93:1575– 80. Berger JW. Quantitative, spatio-temporal image analysis of fundus features in age-related macular degeneration. Proc SPIE 1998;3246:48 –53. Berger JW. Quantitative, image sequence analysis of fundus fluorescein angiography. Ophthalmic Surg Lasers 1999;30: 72–3. Shin DS, Kaiser RS, Lee MS, Berger JW. Fundus image change analysis: geometric and radiometric normalization. Proc SPIE, In press.
16. Shin DS, Berger JW. Digital Fundus Imaging and Analysis. In: Berger JW, Fine SL, Maguire MG, eds. Age-Related Macular Degeneration. Mosby, 1999; chap. 11. 17. Batchelor B, Waltz F. Interactive image processing for machine vision. London; New York: Springer–Verlag, 1993, p. 71–94. 18. Akita K, Kuga H. A computer method of understanding ocular fundus images. Pattern Recognition 1982;15:431– 43. 19. Daily MJ. Color Image Segmentation. In: Mahdavieh Y, Gonzalez RC, eds. Advances in Image Analysis. Bellingham, Wash., USA: SPIE Optical Engineering Press, 1992;131– 61. 20. Goldbaum MH, Katz NP, Chaudhuri S, et al. Digital Image Processing for Ocular Fundus Images. Ophthalmology Clinics of North America 1990;3:447– 66. 21. Haralick RM, Shapiro LG. Image Segmentation Techniques. Computer Vision, Graphics, and Image Processing 1985;29: 100 –32. 22. Katz NP, Goldbaum MH, Nelson M, Chaudhuri S. An image processing system for automatic retina diagnosis. Proc SPIE 1988;902:131–7. 23. Parker JR. Algorithms for image processing and computer vision. New York: Wiley Computer Pub., 1997. 24. Snedecor GW, Cochran WG. Statistical Methods. Ames, Iowa: Iowa State University Press, 1973. 25. Dawson–Saunders B, Trapp RG. Basic and clinical biostatistics, 2nd ed. Norwalk, CT: Appleton & Lange, 1994.
1125
Originally received: August 19, 1998. Revision accepted: February 26, 1999. Manuscript no. 98442. From Computer Vision Laboratory, Scheie Eye Institute, University of Pennsylvania School of Medicine, Philadelphia, Pennsylvania. Presented in part at the Association for Research in Vision and Ophthalmology annual meeting, Fort Lauderdale, Florida, 1998, and at the American Academy of Ophthalmology annual meeting, New Orleans, Louisiana, October 1998. Supported by a Career Development Award from Research to Prevent Blindness, Inc. (JWB) and NIH K-08 EY00374 (JWB). The authors have no proprietary interest in any aspect of this study. Reprint requests to Jeffrey W. Berger, MD, PhD, Retina Service Scheie Eye Institute, 51 North 39th St., Philadelphia, PA 19104. E-mail: [email protected].
persistence and recurrence of CNV after laser therapy limit the efficacy of laser photocoagulation for preservation of visual function. Moreover, although a subset of patients with subfoveal CNV are eligible for laser therapy, these patients will suffer irreversible loss of central vision after foveal laser photocoagulation. Accordingly, major efforts are underway to prophylax against the development of the vision-limiting sequelae of AMD, particularly CNV and retinal pigment epithelial atrophy, and to treat existing CNV with alternate, perhaps more selective, forms of therapy. Eyes with drusen may enjoy excellent visual function but are at significant risk for the development of CNV and retinal pigment epithelial atrophy.2 Stimulated by the observation that light laser application may provoke drusen resorption and predicated on the assumption that drusen reduction may confer protection from the vision-limiting sequelae of AMD, a number of groups are exploring the utility of laser prophylaxis.3– 6 Indeed, the National Eye Institute recently announced its intention to fund the Complications of AMD Prevention Trial, a planned multicenter clinical trial to evaluate prophylactic laser therapy for eyes with bilateral, large drusen at risk for vision loss attributable to the development of CNV and geographic atrophy. Quantitation and highly accurate image comparison of fundus features in AMD, including drusen, will potentiate efficient and accurate clinical trials. However, current meth-
1119
Ophthalmology Volume 106, Number 6, June 1999 ods for evaluating fundus features in AMD are subjective, laborious, and semiquantitative at best.6 –9 Computer-assisted image analysis offers the potential for greater accuracy, objectivity, and reproducibility, but various challenges make robust algorithm design nontrivial. Prior efforts have resulted in algorithms with suboptimal performance.10 –12 Nevertheless, improvements in computer performance and advances in algorithmic design have facilitated progress toward the design and implementation of computerized reading centers for coordination of clinical trials.13–16 In this report, we describe the design, implementation, and validation of an interactive image processing approach for robust accurate segmentation and quantitation of macular drusen in color fundus photographs.
Methods Acknowledging the limitations of computer vision algorithms in unconstrained environments, we have been developing computerassisted techniques for robust, accurate, reproducible fundus feature quantitation and change detection.13–16 The paradigm of interactive image processing allows for highly robust, accurate function, capitalizing on computer performance with various levels of supervision.17 Computers serve well to count and quantify, but, depending on the image quality and features, are less accurate at feature identification. Ideally, complete automation allows for total objectivity and reduced user effort. However, spurious results may be generated by complete, autonomous, computerized analysis. We suggest that robustness, accuracy, and repeatability are priorities in the design of a computer-assisted fundus image analysis system. A small reduction in automation and objectivity is acceptable if gains are realized in accuracy, robustness, and repeatability; a system that is autonomous but only 90% accurate is less valuable than a partially supervised approach affording nearly 100% accuracy. We propose that “automated, supervised” image analysis offers maximum robustness, accuracy, and reproducibility in fundus image analysis, and we describe a custom, application-specific, in-house-developed software package for drusen quantitation.
Image Selection and Digitization Color fundus photographs were obtained from patient records and from the Reading Center of the Choroidal Neovascularization Prevention Trial (CNVPT). For algorithmic development purposes, high-quality, high-contrast images were selected initially, with subsequent development enabled by analysis of poorer quality images. Validation studies were performed on images selected at random from CNVPT records. Thirty-five millimeter color slides were digitized at 1000 pixels per inch, 8 bits per color channel, 3 color channels per pixel (red, green, blue [RGB]), with a Microtek ScanMaker 35t slide scanner (Redondo Beach, CA).
Preprocessing Routinely acquired fundus photographs are of highly variable quality with components of random image noise as well as uneven illumination and lens edge artifacts. The objective of image preprocessing is to minimize noise, maximize contrast, and correct for uneven illumination, lens edge artifact, and natural background pigmentation heterogeneity to prepare optimally the image for subsequent segmentation—the extraction of features of interest, in this case, drusen.18 –23 Color images may be decomposed into
1120
various color representations including RGB, CYMK (cyan, yellow, magenta, black), IHS (intensity, hue, saturation), and others. Clinical experience suggests that drusen are often better discerned on monochromatic “red-free” (green) images when compared with color fundus photographs. We have explored drusen visibility as a function of color image decomposition in various color spaces. In no case was contrast improved when compared with the green channel of the color fundus photograph (JW Berger, DS Shin, RS Kaiser, unpublished results, 1997). Moreover, and perhaps somewhat surprisingly, monochromatically acquired red-free images were not superior to images derived by extraction of the green channel in RGB color space. Accordingly, for maximum contrast, the green channel was selected (Fig 1). Image noise was reduced by application of a 3- ⫻ 3-pixel mean filter followed by 5- ⫻ 5-pixel median filter. These operations were chosen empirically to produce an image with reduced noise suitable for segmentation. Shade correction for uneven illumination and pigmentation was accomplished by dividing the original image by the image convolved with an 85- ⫻ 85-pixel mean filter. The latter image is one without sharp detail but with local brightness, reflecting slowly varying brightness differences over the image, as might result from pigmentation and illumination heterogeneity across the image. Each pixel in the original image could then be normalized with respect to the local brightness image as represented by the image convolved with a large mean filter. The size of the mean filter was chosen empirically to be large enough to average high-frequency intensity fluctuations, as might be associated with fundus features including vessels and drusen, yet smaller than the image size to capture some degree of low-frequency brightness variation.
Segmentation The region of interest for segmentation was identified by manually identifying the fovea with a mouse click. The user could then specify a 1500-m radius, 3000-m radius, or variably sized, user-selected circular region of interest. Other noncircular regions of interest could be specified but were not found to be useful for clinical grading and comparison. Analyses in this report reflect quantitation within an approximately 1500-m radius region of interest. Drusen segmentation was performed on the preprocessed, green channel image. To improve the ability of our algorithms to detect drusen, multiple local-thresholding operations were performed rather than a single global threshold operation. The region of interest was divided into smaller square areas varying from 20 to 100 pixels on a side in increments of 10 pixels. Hence, each pixel was analyzed as part of a square area (or that part of the square area within the region of interest) varying in size from 20 to 100 pixels. Empirically, it was determined that drusen were present only if the skewness24,25 of the set of pixel intensities within a local area was greater than ⫺0.5. A positive skew reflects a greater preponderance of higher intensity outliers—the drusen. Accordingly, only those areas with skewness greater than ⫺0.5 were analyzed, and the threshold was set as the mean pixel value across the region of interest plus a term proportional to the product of the local area pixel brightness standard deviation.
Supervision Two levels of interactivity are incorporated. First, the user may adjust a slider to control the overall “sensitivity,” in which the sensitivity value is inversely proportional to the excess of the local threshold compared to the mean pixel value in the local area. Therefore, a low sensitivity increases the value of the threshold, and fewer pixels are identified. Second, for more complex images,
Shin et al 䡠 Computer-assisted Drusen Quantitation the user may add or remove drusen erroneously segmented or add and remove pixels by mouse. An intensity-based region-growing algorithm was also implemented to expand incompletely identified drusen. Specifically, for some drusen, only the brightest central core was identified, with the periphery of the druse falling below the threshold and therefore not included. Pixels were added to the identified drusen if the neighboring pixels were sufficiently similar to the drusen pixels. The similarity criterion was adjustable with a slider.
Change Detection The ability to quantify change depends on a highly accurate comparison of fundus photographs over corresponding regions of interest. The region of interest specified in one fundus photograph may be stored and applied to follow-up photographs after accurate image registration.15 The change in the area subtended by drusen may then be quantified longitudinally.
Validation Method validation requires demonstration of intraobserver reproducibility and high correlation between drusen identified manually by an experienced grader and the output of our computer algorithm. Ten images of five patients randomly selected from the patient records of the CNVPT were analyzed. Two images of each eye separated in time by 1 year were selected, with the better image of each stereoscopic pair chosen for analysis. One of us (NBJ), an experienced fundus photograph grader, quantitated the area subtended by drusen within a 1500-m radius circular region of interest with two methods. First, the digitized image was displayed on a computer screen and custom, separate software allowed for totally manual grading. The grader outlined each druse manually with a mouse on a 21-inch computer monitor while viewing repeatedly the photographic stereo pair on a lightbox located directly adjacent to the computer. The grader outlined all drusen; the software merely totaled the number of pixels and the fractional area subtended within the region of interest corresponding to the manually identified drusen. We term this value as manual. Second, the software identified the drusen without supervision on the scanned photographs. This parameter is termed automated, unsupervised. Third, the grader was able to supervise fully the process by adding or removing pixels and/or adding or removing drusen by mouse as facilitated by our graphic user interface. The results of this grading are termed automated, supervised. For comparison, a second grader performed automated, supervised analysis independently. Each grading was performed independently without knowledge of the results of prior gradings. Intraclass correlation coefficients were calculated as a measure of the concordance between manual and computer-assisted gradings.24
Results Segmentation Figure 2A depicts a typical image for drusen segmentation. The preprocessed image is shown in Figure 2B, and the results of automated segmentation are depicted in Figure 2C. Drusen occupy 17% of the circular region of interest. Note that preprocessing removes artifacts, maximizes contrast, and enhances drusen visibility for subsequent segmentation. For most images, the software performs quite well in an automated fashion without supervision.
Total computation time for automated analysis is approximately 5 seconds.
Supervision The graphical user interface for supervision of the segmentation process is demonstrated in Figures 3 and 4. Capabilities are incorporated to select an image for analysis, define the region of interest, and interact with the segmentation process by adding or removing pixels as outlined by mouse, removing a druse by clicking anywhere within the errantly identified drusen, and adding a druse with region-growing algorithms. The user may perform automated analysis, compare the output with grading/clinical judgment, and then readjust the sensitivity iteratively to produce results commensurate with subjective drusen identification. More detailed supervision including add/remove pixels/drusen can be performed at any time. The user can toggle easily between the original fundus photograph, the preprocessed image, and the color image with drusen identified to facilitate inspection and supervision. A typical color fundus photograph is shown in Figure 3A, the region of interest is defined in Figure 3B, the preprocessed image is shown in Figure 3C, and the results of automated segmentation are shown in Figure 3D. A large, central druse with pigment figures confounds automated drusen recognition (Figs 3C, 3D). Figure 4 illustrates supervision, allowing for accurate drusen quantitation of the image depicted in Figure 3. Pixels erroneously identified as drusen are removed (Figs 4A, 4B), and menu-driven interactivity allows for mouse-controlled supervision and refinement (Fig 4C). Here, drusen occupy 43% of the region of interest. Depending on the level of interaction required for supervision, the total time required (not computer time, but real time) for complete analysis was approximately 1 minute (range, 20 seconds for straightforward supervision to approximately 5 minutes for complicated images).
Change Detection The software allows for straightforward overlay of a region of interest in one photograph onto a follow-up photograph for longitudinal comparison. Corrections are included to account for discrepancies in rotation, translation, and magnification between the two images.15 An example of longitudinal comparison is depicted in Figure 5. The eye depicted in Figure 2 was followed. At 12 months, there is visible central drusen reduction (Fig 5A, compare with Fig 2A). The region of interest from the analysis in Figure 2 is stored and applied to the image in Figure 5A, allowing for accurate image comparison (Fig 5B). The area subtended by drusen is reduced from 17% to 7%.
Validation Table 1 summarizes validation data. The column for unsupervised segmentation is presented for comparison purposes only; no validity is assigned to unsupervised analysis. The manual and supervised columns allow for comparison between the current gold standard—manual grading by an expert observer—and our algorithms for automated, supervised drusen segmentation. The last column, area subtended by drusen as quantified by our software under the supervision of a second grader, allows for evaluation of interobserver variability. Based on all ten images tabulated (Table 1), intraclass correlation coefficients between manual and automated, supervised gradings were 0.93 and 0.92, respectively, for the two supervisors. There is close agreement between manual and computer-assisted methods, and there is a high concordance between the two supervisors.
1121
Ophthalmology Volume 106, Number 6, June 1999 Figure 1. Representative color fundus photograph of an eye with age-related macular degeneration and drusen (A) with corresponding red (B) and green (C) channel projections. Note the increased visibility of the choroidal markings in the red channel partially obscuring the drusen. Greater contrast is noted in the green channel (C).
Figure 2. Color fundus photograph of an eye with agerelated macular degeneration and drusen (A). The preprocessed image (B) and the final drusen segmentation (C) in the circular region of interest are shown.
Figure 3. Example of supervision. The color fundus image is loaded into the graphical user interface-driven environment (A); the region of interest is selected (B); the image is automatically preprocessed, resulting in improved contrast and drusen visibility (C); and drusen are identified automatically (D). The large, central druse confounds accurate analysis. Supervision of this analysis is depicted in Figure 4. Colors surrounding the identified drusen are arbitrary. (Fig 3 continues.)
1122
Shin et al 䡠 Computer-assisted Drusen Quantitation
Figure 3. (continued).
Figure 4. Example of supervision (continued). Identified pixels not corresponding to drusen are circled (red, A) and removed (B). A menudriven interface allows for addition or removal of pixels or drusen or both (C). Here, the large, central druse is outlined with a mouse.
Figure 5. Image comparison. Same eye as depicted in Figure 2, now 12 months later (A). The region of interest from the earlier analysis (Fig 2) can be stored and overlaid onto the follow-up image after correcting for translation, rotation, and scale differences between the images (B). Drusen segmentation then allows for accurate image comparison (compare B with Fig 2C).
1123
Ophthalmology Volume 106, Number 6, June 1999 Table 1. Concordance between Manual and Automated Supervised Drusen Quantitation Manual Image (%) 1a 1b 2a 2b 3a 3b 4a 4b 5a 5b
13 3 61 27 36 36 34 45 19 27
Automated, Unsupervised (%)
Automated, Supervised (%)
Automated, Supervised (2nd observer, %)
8 2 34 41 40 34 28 28 8 10
9 2 60 31 42 34 35 45 10 (10)*
8 2 54 34 37 36 26 36 16 (16)*
The percent of the region of interest subtended by drusen is given for patients 1–5 initially (a) and following 12 months of follow-up (b). Data are presented for manual, automated, unsupervised and automated, supervised grading by an experienced fundus grader, and for automated, supervised analysis by a second observer. * Image 5b was of poor quality.
Discussion Robust, computerized quantification of pathology in AMD will potentiate epidemiologic studies and clinical trials. Current techniques for drusen area measurement require the observer to estimate the area subtended by drusen in standard regions of interest.6,7,9 However, these methods are coarse, highly subjective, laborious, and poorly suited to image comparison. Several groups have attempted automated segmentation and quantitation of macular drusen.10 –12 However, acceptable performance has heretofore not been achieved. For example, Kirkpatrick et al10 compared their algorithm to manual counting. The authors achieved a sensitivity of 60% and 35% for color fundus photographs and scanning laser ophthalmoscope images, respectively, when a specificity of 90% was enforced. Accordingly, computer-assisted methodology has not gained widespread acceptance and has not yet been incorporated into large clinical trials. Accepting the limitations of computer vision algorithmic performance in an unconstrained environment, particularly when exposed to images of poor contrast and quality, we propose a paradigm for automated, supervised fundus image analysis. In addition, we demonstrate the applicability of this approach to robust, reproducible, accurate image segmentation for drusen quantitation in AMD. Image preprocessing allows for correction of uneven illumination, poor contrast, and image noise and potentiates accurate drusen recognition (Figs 2–5). Our algorithms perform very well without supervision (Table 1); however, acceptance into the clinical environment requires nearly 100% accuracy and robustness. Therefore, tools for supervision are incorporated into the environment, permitting refinement of the segmentation. In theory, supervision affords segmentation, yielding arbitrarily close agreement with manual grading; with effort, the user may totally overhaul the output generated by automated drusen recognition.
1124
Differences between manual grading (in which the computer merely counts the user-identified areas) and computerassisted, automated, supervised analysis (Table 1) reflect intraobserver variability in repeated grading of fundus images. Recognizing the limited quantitation abilities of human graders, most studies invoke coarse grading schemes (e.g., estimating the area subtended by drusen as ⬍25%, between 25% and 50%, or ⬎50%). By incorporating supervision, our software capitalizes on both computer and human image feature recognition and the obvious superiority of computerbased methodologies for quantitation. Whereas an expert observer is superior to a computer in drusen recognition, particularly for complex drusen morphologies, computers are more accurate and faster in quantitation tasks. Automated, supervised fundus image analysis is an efficient, robust, valid technique for drusen quantitation from color fundus photographs. This approach should prove useful in the conduct of efficient, accurate clinical trials to evaluate novel therapeutic and prophylactic approaches for the vision-limiting complications of AMD. Acknowledgments. The authors thank the members of the Choroidal Neovascularization Prevention Trial Study Group; Stuart L. Fine, MD, Study Chairman; and Allen C. Ho, MD, Reading Center Director, for access to their data. In addition, the authors thank Maureen G. Maguire, PhD, for assistance with statistical analyses.
References 1. Berger JW, Maguire MG, Fine SL. The Macular Photocoagulation Study. In: Kertes PJ, Conway MD, eds. Clinical Trials in Ophthalmology: A Summary and Practice Guide. Baltimore: Williams & Wilkins, 1996; Chap. 5. 2. Bressler SB, Maguire MG, Bressler NM, Fine SL. Relationship of drusen and abnormalities of the retinal pigment epithelium to the prognosis of neovascular macular degeneration. The Macular Photocoagulation Study Group. Arch Ophthalmol 1990;108:1442–7. 3. Figuero MS, Regueras A, Bertrand J. Laser photocoagulation to treat macular soft drusen in age-related macular degeneration. Retina 1994;14:391– 6. 4. Guymer RH, Gross–Jendroska M, Owens SL, et al. Laser treatment in subjects with high-risk clinical features of agerelated macular degeneration. Arch Ophthalmol 1997;115: 595– 603. 5. Little H, Showman JM, Brown BW. A pilot randomized controlled study on the effect of laser photocoagulation of confluent soft macular drusen. Ophthalmology 1997;104:623– 31. 6. The laser treatment in eyes with large drusen. Short-term effects seen in a pilot randomized clinical trial. Choroidal Neovascularization Prevention Trial Research Group. Ophthalmology 1998;105:11–23. 7. Bird AC, Bressler NM, Bressler SB, et al. An international classification and grading system for age-related maculopathy and age-related macular degeneration. The International ARM Epidemiological Study Group. Surv Ophthalmol 1995;39:367–74. 8. Chamberlin JA, Bressler NM, Bressler SB, et al. The use of fundus photographs and fluorescein angiograms in the identi-
Shin et al 䡠 Computer-assisted Drusen Quantitation
9. 10.
11. 12. 13. 14. 15.
fication and treatment of choroidal neovascularization in the Macular Photocoagulation Study. The Macular Photocoagulation Study Group. Ophthalmology 1989;96:1526 –34. Klein R, Klein BEK, Jensen SC, Meuer SM. The five-year incidence and progression of age-related maculopathy: the Beaver Dam Eye Study. Ophthalmology 1997;104:7–21. Kirkpatrick JNP, Spencer T, Manivannan A, et al. Quantitative image analysis of macular drusen from fundus photographs and scanning laser ophthalmoscope images. Eye 1995; 9:48 –55. Morgan WH, Cooper RL, Constable IJ, Eikelboom RH. Automated extraction and quantification of macular drusen from fundal photographs. Aust N Z J Ophthalmol 1994;22:7–12. Peli E, Lahav M. Drusen measurement from fundus photographs using computer image analysis. Ophthalmology 1986; 93:1575– 80. Berger JW. Quantitative, spatio-temporal image analysis of fundus features in age-related macular degeneration. Proc SPIE 1998;3246:48 –53. Berger JW. Quantitative, image sequence analysis of fundus fluorescein angiography. Ophthalmic Surg Lasers 1999;30: 72–3. Shin DS, Kaiser RS, Lee MS, Berger JW. Fundus image change analysis: geometric and radiometric normalization. Proc SPIE, In press.
16. Shin DS, Berger JW. Digital Fundus Imaging and Analysis. In: Berger JW, Fine SL, Maguire MG, eds. Age-Related Macular Degeneration. Mosby, 1999; chap. 11. 17. Batchelor B, Waltz F. Interactive image processing for machine vision. London; New York: Springer–Verlag, 1993, p. 71–94. 18. Akita K, Kuga H. A computer method of understanding ocular fundus images. Pattern Recognition 1982;15:431– 43. 19. Daily MJ. Color Image Segmentation. In: Mahdavieh Y, Gonzalez RC, eds. Advances in Image Analysis. Bellingham, Wash., USA: SPIE Optical Engineering Press, 1992;131– 61. 20. Goldbaum MH, Katz NP, Chaudhuri S, et al. Digital Image Processing for Ocular Fundus Images. Ophthalmology Clinics of North America 1990;3:447– 66. 21. Haralick RM, Shapiro LG. Image Segmentation Techniques. Computer Vision, Graphics, and Image Processing 1985;29: 100 –32. 22. Katz NP, Goldbaum MH, Nelson M, Chaudhuri S. An image processing system for automatic retina diagnosis. Proc SPIE 1988;902:131–7. 23. Parker JR. Algorithms for image processing and computer vision. New York: Wiley Computer Pub., 1997. 24. Snedecor GW, Cochran WG. Statistical Methods. Ames, Iowa: Iowa State University Press, 1973. 25. Dawson–Saunders B, Trapp RG. Basic and clinical biostatistics, 2nd ed. Norwalk, CT: Appleton & Lange, 1994.
1125