- Email: [email protected]
Tortuosity of arterioles and venules in quantifying plus disease
Tortuosity of arterioles and venules in quantifying plus disease Suzanne C. Johnston, MD, David K. Wallace, MD, MPH, Sharon F. Freedman, MD, Tammy L. Yanovitch, MD, and Zheen Zhao, PhD BACKGROUND
METHODS
RESULTS
CONCLUSIONS
Plus disease is the major criterion for laser treatment of retinopathy of prematurity. ROPtool is a computer program that traces retinal blood vessels and measures their tortuosity. Our objectives were to determine (1) whether examiners could accurately discriminate between arterioles and venules and (2) whether tortuosity sufficient for plus disease and pre-plus disease was assessed most accurately by considering arterioles, venules, or both. One hundred retinal vessels were identified in 25 images randomly selected from 184 total images. Three pediatric ophthalmologists independently designated vessels as arteriole or venule. Seventy-seven images that had at least 1 traceable arteriole and venule in each quadrant were analyzed by ROPtool, and the results were compared with the consensus of 3 expert examiners. Receiver operating characteristics (ROC) curves were generated and areas under the curves calculated to quantify the diagnostic utility of ROPtool’s assessment of tortuosity of arterioles, venules, and both. Three pediatric ophthalmologists agreed on the designation of arteriole or venule for 83 of 100 blood vessels. With the use of expert consensus as the reference standard, areas under the ROC curves for identification of tortuosity sufficient for plus disease were 0.91, 0.70, and 0.93 for arterioles, venules, and both, respectively. Areas under the ROC curves for identification of tortuosity sufficient for pre-plus disease were 0.91, 0.63, and 0.90 for arterioles, venules, and both, respectively. When considering whether tortuosity is sufficient for plus or pre-plus disease, the assessment of either arterioles alone or of arterioles and venules together resulted in high diagnostic accuracy. ( J AAPOS 2009;13:181-185)
P
lus disease is an important prognostic indicator in retinopathy of prematurity (ROP) as well as a crucial sign used to determine when treatment of severe ROP is needed. Plus disease was defined in the International Classification of Retinopathy of Prematurity (ICROP) as being present when ‘‘the vascular changes are so marked that the posterior veins are enlarged and the arterioles tortuous.’’1 The Committee to Revisit the International Classification of Retinopathy of Prematurity defined pre-plus disease as ‘‘vascular abnormalities of the posterior pole that are insufficient for the diagnosis of plus disease but that demonstrate more arterial tortuosity and more venous dilation than normal.’’2 The diagnosis of plus disease generally is based on an examiner’s judgment of dilation and tortuosity in comparison with a standard photograph
Author affiliations: Duke University Eye Center, Durham, North Carolina Presented at the 34th Annual Meeting of the American Association for Pediatric Ophthalmology and Strabismus, Washington, D.C., April 2-6, 2008. Submitted April 2, 2008. Revision accepted October 30, 2008. Reprint requests: David K. Wallace, MD, MPH, Duke University Eye Center DUMC 3802, Durham, NC 27710 (email: [email protected]). Copyright Ó 2009 by the American Association for Pediatric Ophthalmology and Strabismus. 1091-8531/2009/$36.00 1 0 doi:10.1016/j.jaapos.2008.10.019
Journal of AAPOS
and was first used for the Cryotherapy for Retinopathy of Prematurity (CRYO-ROP) study.3 However, it may be difficult to accurately and consistently diagnose plus disease. Using RetCam images, Chiang and colleagues4 found that 22 experts agreed on the same 3-level diagnosis in only 4 of 34 images (12%). Wallace and colleagues5 found that 3 experts disagreed on the presence or absence of plus disease for 18 of 67 RetCam images (27%) with pre-plus or worse. Because of the importance of plus disease and the subjective nature of its diagnosis, interest has grown in development of an objective method for measuring it. ROPtool is one of a few computer programs that have been developed to quantify vascular tortuosity and dilation.6-8 A previous study demonstrated that ROPtool’s assessment of arteriole tortuosity sufficient for plus disease compared favorably with that of individual examiners.7 In that study, both arterioles and venules were included, and tortuosity values were averages of all analyzed blood vessels. Although definitions of plus and pre-plus disease imply that arteriolar tortuosity and venular dilation are the most important factors in grading plus and pre-plus disease, we do not know whether examiners actually use arteriolar tortuosity, venular tortuosity, or both to determine the amount of vascular tortuosity. This information would give greater insight into how examiners grade plus disease and which blood vessels should be analyzed when future
181
182
Johnston et al
versions of ROPtool or other programs quantify plus disease. Previous versions of ROPtool averaged tortuosity of all retinal blood vessels selected in a given quadrant. ROPtool has recently been upgraded, allowing its operator to label vessels as either arterioles or venules. However, measuring the tortuosity of arterioles and venules separately depends on our ability to accurately distinguish them. Therefore, the initial phase of this study was designed to determine whether examiners could agree consistently on vessel classification as either arteriole or venule. If there was reasonable agreement, then the relative importance of arterioles and venules in assessing tortuosity sufficient for plus and pre-plus disease could be examined. The next step, and the primary purpose of this study, was to determine the accuracy of ROPtool in measuring tortuosity sufficient for plus and pre-plus disease when considering (1) arterioles only, (2) venules only, and (3) both arterioles and venules.
Methods and Materials This study used 184 images that had been used in a previously published study.7 The images had been cropped to simulate the view through a 28D lens. This sample of images included a larger proportion of plus and pre-plus disease than would typically be encountered during routine examinations. Our Institutional Review Board granted an exemption because none of the infants could be identified from the retinal images. For this study, images were evaluated by more than one group of examiners. The first group, referred to as ‘‘pediatric ophthalmologists’’ (PO), consisted of 3 authors (DW, SF, TY). All of these authors have experience examining infants with ROP, and their role was to classify vessels as arterioles or venules in the first phase of the study. The second group consisted of three ‘‘experts in ROP’’ (EROP), who graded each image (as part of a previous study)7 as having tortuosity sufficient for plus disease, pre-plus disease, or neither. Their grades were used as the reference standard in the second phase of this study. Our first step in this study was to determine whether experienced ROP examiners (PO) could agree on the classification of retinal blood vessels as arterioles or venules. Of the 184 RetCam images, 25 were selected with the use of a random number generator. In each of these 25 photographs, one vessel from each quadrant was chosen and numbered by one of the authors (SJ). These photographs were then forwarded to each of the 3 authors (PO). They classified each vessel as an arteriole, a venule, likely an arteriole, or likely a venule, and a percentage agreement was then calculated. To determine whether these factors were associated with accurate identification of vessels, one of the authors (SJ) reviewed each of the 25 images and classified each as having excellent, good, or fair quality and light, medium, or dark fundus pigmentation. Expert consensus (EROP), which had been previously determined, was used to identify each image as having tortuosity sufficient for plus disease, pre-plus disease, or neither. Figure 1 shows a typical fundus image after analysis by ROPtool. In our previous study, the tortuosity of each vessel and of each quadrant had been measured, but arterioles and venules
Volume 13 Number 2 / April 2009 had not been distinguished. For this study, one of the authors (SJ) labeled each traced vessel of each image as either an arteriole or a venule. Vessels that were difficult to classify were reviewed by a second author (DW). If the 2 authors agreed that the vessel was likely an arteriole or venule, it was classified as such. If there was disagreement, or if it was determined that the vessel could not be classified, then it was excluded from the study. Arteriolar tortuosity, venular tortuosity, and combined arteriolar and venular tortuosity were calculated for each quadrant of each image. When 2 or more of the same vessel type (eg, 2 arterioles) were analyzed in a single quadrant, an average was calculated to represent the value for the quadrant. Those quadrants that did not contain both an arteriole and venule were excluded from quadrant-level analyses (where each quadrant was treated separately), and their respective images were excluded from eye-level analyses (where each eye served as the unit of analysis). Receiver operating characteristic (ROC) curves were constructed to assess the diagnostic value of using arterioles alone, venules alone, or both arterioles and venules when diagnosing tortuosity sufficient for plus or pre-plus disease. ROC curves plot sensitivity on the y-axis and 1-specificity (the false-positive rate) on the x-axis for multiple threshold values calculated by a diagnostic test such as ROPtool. They demonstrate the tradeoff between sensitivity and specificity, and larger areas under the curves reflect greater diagnostic accuracy. A perfect diagnostic test has an area under the curve of 1.0 and reaches the upper left corner of the graph. A poor diagnostic test (that is, equivalent to flipping a coin) has an area under the curve of 0.5. The reference standard was consensus of 3 ROP experts (EROP), as described in a previous publication.7 The analyses were first performed for eye-level data for tortuosity sufficient for plus disease, and they were repeated for eye-level data for pre-plus disease, for quadrant-level data for plus disease, and for quadrant-level data for pre-plus disease. An eye-level analysis represents analysis of all 4 quadrants in a single eye. To include an eye in an eye-level analysis, we required at least one traceable arteriole and one traceable venule in each of 4 quadrants. Eye-level grades were based on a combination of the quadrant-level grades for each image. For example, an eye-level grade of tortuosity sufficient for plus disease was present if at least 2 of the 4 quadrants in a single eye had tortuosity sufficient for plus disease.
Results All 3 PO examiners agreed on the classification of arteriole or venule for 83 of 100 quadrants (83%). For those quadrants of images with excellent quality, there was 86% agreement (24/28). For those quadrants of images with good or fair quality, there was 85% (41/48) and 75% (18/ 24) agreement, respectively. Those quadrants of images with light, medium, and dark fundus pigmentation had 80% (16/20), 88% (28/32), and 81% (39/48) agreement, respectively. Quadrants of images with (1) no plus or preplus disease, (2) pre-plus disease, and (3) plus disease had 83% (63/76), 92% (11/12), and 75% (9/12) agreement, respectively.
Journal of AAPOS
Volume 13 Number 2 / April 2009
FIG 1. Appearance of image after analysis by ROPtool. Small circle is optic nerve border; large dotted circle is 28 D circle; straight lines are borders of quadrants; wiggly lines are traced vessels whose tortuosity has been calculated.
For the second phase of the study, all 184 images were initially reviewed by one of the authors (SJ). Five images were excluded from both quadrant-level and eye-level analyses because it was the consensus of 2 authors (SJ and DW) that arterioles and venules could not reliably be differentiated from one another. Typically, these images did not have sufficient clarity as the result of poor focus. Seventyseven of the remaining 179 images (43%) had sufficient quality, focus, and centering that at least one arteriole and one venule could be traced by ROPtool in all 4 quadrants, and only these images were included in the eye-level analysis. Excluded images had at least one quadrant where an arteriole or venule could not be identified by ROPtool. Using as the reference standard expert consensus (EROP) as to whether there was eye-level plus tortuosity, pre-plus tortuosity, or neither, areas under the ROC curves for identification of tortuosity sufficient for plus disease were 0.97, 0.80, and 0.97 for arterioles, venules, and both, respectively. Areas under the ROC curves for identification of tortuosity sufficient for eye-level pre-plus disease were 0.92, 0.69, and 0.93 for arterioles, venules, and both, respectively. Five hundred and forty quadrants that had at least one traceable arteriole and one traceable venule were included in the quadrant-level analyses; all other quadrants were excluded. Using expert consensus (EROP) as to whether there was quadrant-level plus tortuosity, pre-plus tortuosity, or neither as the reference standard, areas under the ROC curves for identification of tortuosity sufficient for plus disease were 0.91, 0.70, and 0.93 for arterioles,
Journal of AAPOS
Johnston et al
183
FIG 2. Receiver operating characteristic curves for ROPtool’s detection of quadrant-level tortuosity sufficient for plus disease using arterioles alone and venules alone in comparison with the consensus of 3 experts.
venules, and both, respectively. Areas under the ROC curves for identification of quadrant-level tortuosity sufficient for pre-plus disease were 0.91, 0.63, and 0.90 for arterioles, venules, and both respectively. Figure 2 shows quadrant-level ROC curves for arterioles alone and venules alone in determining tortuosity sufficient for plus disease. Each point on the ROC curve represents a different numeric threshold (or ‘‘cut-off point’’) for tortuosity, and each point is associated with a distinct sensitivity and specificity. The ROC curves based on quadrant-level data show that the optimum cut-off point (ie, maximizing sensitivity without greatly compromising specificity) is 16 tortuosity units for arteriolar plus tortuosity, resulting in a sensitivity of 0.90 and a specificity of 0.78. For arteriolar pre-plus tortuosity (curves not shown), the optimum cut-off point is 12 tortuosity units, resulting in a sensitivity of 0.83 and a specificity of 0.83. Although choosing a different cutoff point could improve specificity, it would compromise the excellent sensitivity that is important when diagnosing plus disease.
Discussion PO examiners had a reasonably good agreement rate of 83% for designating retinal blood vessels as either arterioles or venules. Thus, it is feasible to separate the analysis of arteriolar and venular tortuosity. Although a larger study would be needed to determine whether there are factors predictive of agreement, our preliminary data suggest that image quality, fundus pigmentation, and disease severity had little to no effect on agreement when classifying vessels. Differentiating arterioles from venules is important beyond the refinement of ROPtool for several reasons. First,
184
Johnston et al
plus disease originally was defined by ICROP as being present when ‘‘the vascular changes are so marked that the posterior veins are enlarged and the arterioles tortuous.’’1 This statement implies that arteries and veins should be considered separately when evaluating for the presence of plus disease. Second, our results give some insight into how experts judge plus disease. We found that expert opinion agreed more with ROPtool’s measurement of arteriolar tortuosity than with venular tortuosity, suggesting that experts primarily focus on arterioles when judging vascular tortuosity. Third, ROPtool is one of several computer programs that are being used to quantify plus disease.9-13 For any program to be refined and applied to retinal images, it is important to know whether differentiating arterioles from venules is feasible, and whether doing so affects calculations obtained from these programs. Finally, quantification of vascular changes has application beyond retinopathy of prematurity. For example, arteriolar narrowing has been associated with risk of developing hypertension,14 diabetes,15 and coronary artery disease.16 Because retinal arterioles and venules respond differently to systemic disease, it may be important to distinguish arteriolar from venular changes to most accurately determine disease risk. This study showed that the use of arterioles alone or using arterioles and venules together provides very good diagnostic accuracy for ROPtool’s determination of tortuosity sufficient for plus disease or pre-plus disease. However, venule tortuosity alone did not have good diagnostic accuracy. These results are consistent with the definitions of plus and pre-plus disease, which highlight the importance of tortuosity of arterioles and dilation of venules. Venules are important in determining plus disease, but it seems that their primary contribution is to vessel dilation rather than tortuosity. A dilation measurement has recently been added to ROPtool capabilities, but it has not yet been validated. Other programs are available that can grade venular dilation.9-13 Once tortuosity is known, it is possible that the measurement of dilation may have little influence in the overall assessment of plus disease. Kylstra and colleagues (Kylstra JA; et al. IOVS 1995; 36:ARVO Abstract 77) found that computer-assisted measurement of plus disease based only on tortuosity showed high sensitivity (85%) and specificity (91%) when compared with expert evaluation. Yanovitch and colleagues17 found that both dilation and tortuosity are important to consider in the clinical diagnosis of plus disease. When severe plus disease develops, both arterioles and venules become tortuous and dilated, so it might be expected that venular tortuosity would have fared better in this study. There are several possible explanations for the poor diagnostic accuracy of venular tortuosity alone. First, arteriolar tortuosity often is present without venular tortuosity, especially when plus disease is not severe. Second, our assessment of diagnostic accuracy was based on agreement with the consensus of experts, and it is possible that the experts focused on arteriolar tortuosity when determin-
Volume 13 Number 2 / April 2009 ing the reference standard for comparison. The most tortuous vessel in each quadrant is likely to be an arteriole, and that vessel may ‘‘catch the eye’’ of examiners and greatly influence their tortuosity judgment. Third, when venular tortuosity does occur, the amount of tortuosity may be relatively less than that observed with arterioles. Consequently, there was a smaller spread of tortuosity values with venules, resulting in greater misclassification of plus disease by ROPtool. It is possible that tortuosity is less in venules than in arterioles because veins are more distensible than arteries. If plus disease is the result of increased flow through arteriovenous shunts (one plausible hypothesis),18 then venules may respond at least initially by dilating instead of becoming tortuous. There are several limitations to this study. First, many images could not be used because ROPtool would not analyze both an arteriole and a venule in all 4 quadrants because of insufficient quality, focus, and/or centering. Second, it is possible that some vessels were not classified correctly as arteriole or venule. To limit this misclassification, we excluded images when we could not be reasonably certain of the vessel type after review by two authors. Third, there is correlation between tortuosity of vessels in the same eye or image, so values for quadrant-level analyses are not completely independent of each other. For this reason, standard statistical comparisons of areas under the curves could not be performed. Fourth, we assessed tortuosity only, and diagnosing plus disease depends on both dilation and tortuosity. We are currently refining and testing ROPtool’s dilation measure, and it is possible that including dilation will improve diagnostic accuracy. Fifth, because we analyzed RetCam images, our results cannot be applied directly to findings by indirect ophthalmoscopy. In conclusion, our results suggest that considering arterioles alone in computer-assisted diagnosis is probably sufficient for determining plus or pre-plus tortuosity. These data will be used to modify ROPtool so that it focuses on arterioles when measuring tortuosity. Much more work lies ahead to determine the contribution of dilation to the overall assessment of plus disease and to validate ROPtool’s overall assessment of plus and pre-plus disease using both tortuosity and dilation.
Acknowledgments We acknowledge Michael Chiang and Graham Quinn for serving as expert graders. We acknowledge Michael Chiang, Anna Ells, and the PHOTO-ROP study group for sharing RetCam photographs.
References 1. The Committee for the Classification of Retinopathy of Prematurity. An international classification of retinopathy of prematurity. Arch Ophthalmol 1984;102:1130-34. 2. The International Classification of Retinopathy of Prematurity revisited. Arch Ophthalmol 2005;123:991-9. 3. Multicenter trial of cryotherapy for retinopathy of prematurity. Preliminary results. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Arch Ophthalmol 1988;106:471-9.
Journal of AAPOS
Volume 13 Number 2 / April 2009 4. Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol 2007;125:875-80. 5. Wallace DK, Quinn GE, Freedman SF, Chiang MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS 2008;12:352-6. 6. Wallace DK, Zhao Z, Freedman SF. A pilot study using ‘‘ROPtool’’ to quantify plus disease in retinopathy of prematurity. J AAPOS 2007;11: 381-7. 7. Wallace DK, Freedman SF, Zhao Z, Jung SH. Accuracy of ROPtool vs individual examiners in assessing retinal vascular tortuosity. Arch Ophthalmol 2007;125:1523-30. 8. Wallace DK, Jomier J, Aylward SR, Landers MB, 3rd. Computer-automated quantification of plus disease in retinopathy of prematurity. J AAPOS 2003;7:126-30. 9. Gelman R, Martinez-Perez ME, Vanderveen DK, Moskowitz A, Fulton AB. Diagnosis of plus disease in retinopathy of prematurity using retinal image multiscale analysis. Invest Ophthalmol Vis Sci 2005; 46:4734-8. 10. Heneghan C, Flynn J, O’Keefe M, Cahill M. Characterization of changes in blood vessel width and tortuosity in retinopathy of prematurity using image analysis. Medical Image Analysis 2002;6:407-29. 11. Swanson CR, Cocker KD, Parker KH, Moseley MJ, Wren SME, Fielder AR. Semi-automated computer analysis of vessel growth in preterm infants without and with ROP. Br J Ophthalmol 2003;87:1474-7.
Journal of AAPOS
Johnston et al
185
12. Johnson KS, Mills MD, Karp KA, Grunwald JE. Semiautomated analysis of retinal vessel diameter in retinopathy of prematurity patients with and without plus disease. Am J Ophthalmol 2007;143: 723-35. 13. Wilson CM, Cocker KD, Moseley MJ, Paterson C, Clay ST, Schulenburg WE, et al. Computerized analysis of retinal vessel width and tortuosity in premature infants. Invest Ophthalmol Vis Sci 2008; 49:3577-85. 14. Wong TY, Klein R, Sharrett AR, Duncan BB, Couper DJ, Klein BE, et al. Retinal arteriolar diameter and risk for hypertension. Ann Intern Med 2004;17(140):248-55. 15. Wong TY, Klein R, Sharrett AR, Schmidt MI, Pankow JS, Couper DJ, et al. Retinal arteriolar narrowing and risk of diabetes mellitus in middle-aged persons. JAMA 2002;287:2528-33. 16. Wong TY, Klein R, Sharrett AR, Duncan BB, Couper DJ, Tielsch JM, et al. Retinal arteriolar narrowing and risk of coronary heart disease in men and women. The Atherosclerosis Risk in Communities Study. JAMA 2002;6(287):1153-9. 17. Yanovitch TL, Freedman SF, Wallace DK. The Importance of dilation and tortuosity in diagnosing plus disease. Arch Ophthalmol 2009; in press. 18. Schaffer DB, Palmer EA, Plotsky DF, Metz HS, Flynn JT, Tung B, et al, for the Prognostic factors in the natural course of retinopathy of prematurity. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Ophthalmology 1993;100:230-37.
METHODS
RESULTS
CONCLUSIONS
Plus disease is the major criterion for laser treatment of retinopathy of prematurity. ROPtool is a computer program that traces retinal blood vessels and measures their tortuosity. Our objectives were to determine (1) whether examiners could accurately discriminate between arterioles and venules and (2) whether tortuosity sufficient for plus disease and pre-plus disease was assessed most accurately by considering arterioles, venules, or both. One hundred retinal vessels were identified in 25 images randomly selected from 184 total images. Three pediatric ophthalmologists independently designated vessels as arteriole or venule. Seventy-seven images that had at least 1 traceable arteriole and venule in each quadrant were analyzed by ROPtool, and the results were compared with the consensus of 3 expert examiners. Receiver operating characteristics (ROC) curves were generated and areas under the curves calculated to quantify the diagnostic utility of ROPtool’s assessment of tortuosity of arterioles, venules, and both. Three pediatric ophthalmologists agreed on the designation of arteriole or venule for 83 of 100 blood vessels. With the use of expert consensus as the reference standard, areas under the ROC curves for identification of tortuosity sufficient for plus disease were 0.91, 0.70, and 0.93 for arterioles, venules, and both, respectively. Areas under the ROC curves for identification of tortuosity sufficient for pre-plus disease were 0.91, 0.63, and 0.90 for arterioles, venules, and both, respectively. When considering whether tortuosity is sufficient for plus or pre-plus disease, the assessment of either arterioles alone or of arterioles and venules together resulted in high diagnostic accuracy. ( J AAPOS 2009;13:181-185)
P
lus disease is an important prognostic indicator in retinopathy of prematurity (ROP) as well as a crucial sign used to determine when treatment of severe ROP is needed. Plus disease was defined in the International Classification of Retinopathy of Prematurity (ICROP) as being present when ‘‘the vascular changes are so marked that the posterior veins are enlarged and the arterioles tortuous.’’1 The Committee to Revisit the International Classification of Retinopathy of Prematurity defined pre-plus disease as ‘‘vascular abnormalities of the posterior pole that are insufficient for the diagnosis of plus disease but that demonstrate more arterial tortuosity and more venous dilation than normal.’’2 The diagnosis of plus disease generally is based on an examiner’s judgment of dilation and tortuosity in comparison with a standard photograph
Author affiliations: Duke University Eye Center, Durham, North Carolina Presented at the 34th Annual Meeting of the American Association for Pediatric Ophthalmology and Strabismus, Washington, D.C., April 2-6, 2008. Submitted April 2, 2008. Revision accepted October 30, 2008. Reprint requests: David K. Wallace, MD, MPH, Duke University Eye Center DUMC 3802, Durham, NC 27710 (email: [email protected]). Copyright Ó 2009 by the American Association for Pediatric Ophthalmology and Strabismus. 1091-8531/2009/$36.00 1 0 doi:10.1016/j.jaapos.2008.10.019
Journal of AAPOS
and was first used for the Cryotherapy for Retinopathy of Prematurity (CRYO-ROP) study.3 However, it may be difficult to accurately and consistently diagnose plus disease. Using RetCam images, Chiang and colleagues4 found that 22 experts agreed on the same 3-level diagnosis in only 4 of 34 images (12%). Wallace and colleagues5 found that 3 experts disagreed on the presence or absence of plus disease for 18 of 67 RetCam images (27%) with pre-plus or worse. Because of the importance of plus disease and the subjective nature of its diagnosis, interest has grown in development of an objective method for measuring it. ROPtool is one of a few computer programs that have been developed to quantify vascular tortuosity and dilation.6-8 A previous study demonstrated that ROPtool’s assessment of arteriole tortuosity sufficient for plus disease compared favorably with that of individual examiners.7 In that study, both arterioles and venules were included, and tortuosity values were averages of all analyzed blood vessels. Although definitions of plus and pre-plus disease imply that arteriolar tortuosity and venular dilation are the most important factors in grading plus and pre-plus disease, we do not know whether examiners actually use arteriolar tortuosity, venular tortuosity, or both to determine the amount of vascular tortuosity. This information would give greater insight into how examiners grade plus disease and which blood vessels should be analyzed when future
181
182
Johnston et al
versions of ROPtool or other programs quantify plus disease. Previous versions of ROPtool averaged tortuosity of all retinal blood vessels selected in a given quadrant. ROPtool has recently been upgraded, allowing its operator to label vessels as either arterioles or venules. However, measuring the tortuosity of arterioles and venules separately depends on our ability to accurately distinguish them. Therefore, the initial phase of this study was designed to determine whether examiners could agree consistently on vessel classification as either arteriole or venule. If there was reasonable agreement, then the relative importance of arterioles and venules in assessing tortuosity sufficient for plus and pre-plus disease could be examined. The next step, and the primary purpose of this study, was to determine the accuracy of ROPtool in measuring tortuosity sufficient for plus and pre-plus disease when considering (1) arterioles only, (2) venules only, and (3) both arterioles and venules.
Methods and Materials This study used 184 images that had been used in a previously published study.7 The images had been cropped to simulate the view through a 28D lens. This sample of images included a larger proportion of plus and pre-plus disease than would typically be encountered during routine examinations. Our Institutional Review Board granted an exemption because none of the infants could be identified from the retinal images. For this study, images were evaluated by more than one group of examiners. The first group, referred to as ‘‘pediatric ophthalmologists’’ (PO), consisted of 3 authors (DW, SF, TY). All of these authors have experience examining infants with ROP, and their role was to classify vessels as arterioles or venules in the first phase of the study. The second group consisted of three ‘‘experts in ROP’’ (EROP), who graded each image (as part of a previous study)7 as having tortuosity sufficient for plus disease, pre-plus disease, or neither. Their grades were used as the reference standard in the second phase of this study. Our first step in this study was to determine whether experienced ROP examiners (PO) could agree on the classification of retinal blood vessels as arterioles or venules. Of the 184 RetCam images, 25 were selected with the use of a random number generator. In each of these 25 photographs, one vessel from each quadrant was chosen and numbered by one of the authors (SJ). These photographs were then forwarded to each of the 3 authors (PO). They classified each vessel as an arteriole, a venule, likely an arteriole, or likely a venule, and a percentage agreement was then calculated. To determine whether these factors were associated with accurate identification of vessels, one of the authors (SJ) reviewed each of the 25 images and classified each as having excellent, good, or fair quality and light, medium, or dark fundus pigmentation. Expert consensus (EROP), which had been previously determined, was used to identify each image as having tortuosity sufficient for plus disease, pre-plus disease, or neither. Figure 1 shows a typical fundus image after analysis by ROPtool. In our previous study, the tortuosity of each vessel and of each quadrant had been measured, but arterioles and venules
Volume 13 Number 2 / April 2009 had not been distinguished. For this study, one of the authors (SJ) labeled each traced vessel of each image as either an arteriole or a venule. Vessels that were difficult to classify were reviewed by a second author (DW). If the 2 authors agreed that the vessel was likely an arteriole or venule, it was classified as such. If there was disagreement, or if it was determined that the vessel could not be classified, then it was excluded from the study. Arteriolar tortuosity, venular tortuosity, and combined arteriolar and venular tortuosity were calculated for each quadrant of each image. When 2 or more of the same vessel type (eg, 2 arterioles) were analyzed in a single quadrant, an average was calculated to represent the value for the quadrant. Those quadrants that did not contain both an arteriole and venule were excluded from quadrant-level analyses (where each quadrant was treated separately), and their respective images were excluded from eye-level analyses (where each eye served as the unit of analysis). Receiver operating characteristic (ROC) curves were constructed to assess the diagnostic value of using arterioles alone, venules alone, or both arterioles and venules when diagnosing tortuosity sufficient for plus or pre-plus disease. ROC curves plot sensitivity on the y-axis and 1-specificity (the false-positive rate) on the x-axis for multiple threshold values calculated by a diagnostic test such as ROPtool. They demonstrate the tradeoff between sensitivity and specificity, and larger areas under the curves reflect greater diagnostic accuracy. A perfect diagnostic test has an area under the curve of 1.0 and reaches the upper left corner of the graph. A poor diagnostic test (that is, equivalent to flipping a coin) has an area under the curve of 0.5. The reference standard was consensus of 3 ROP experts (EROP), as described in a previous publication.7 The analyses were first performed for eye-level data for tortuosity sufficient for plus disease, and they were repeated for eye-level data for pre-plus disease, for quadrant-level data for plus disease, and for quadrant-level data for pre-plus disease. An eye-level analysis represents analysis of all 4 quadrants in a single eye. To include an eye in an eye-level analysis, we required at least one traceable arteriole and one traceable venule in each of 4 quadrants. Eye-level grades were based on a combination of the quadrant-level grades for each image. For example, an eye-level grade of tortuosity sufficient for plus disease was present if at least 2 of the 4 quadrants in a single eye had tortuosity sufficient for plus disease.
Results All 3 PO examiners agreed on the classification of arteriole or venule for 83 of 100 quadrants (83%). For those quadrants of images with excellent quality, there was 86% agreement (24/28). For those quadrants of images with good or fair quality, there was 85% (41/48) and 75% (18/ 24) agreement, respectively. Those quadrants of images with light, medium, and dark fundus pigmentation had 80% (16/20), 88% (28/32), and 81% (39/48) agreement, respectively. Quadrants of images with (1) no plus or preplus disease, (2) pre-plus disease, and (3) plus disease had 83% (63/76), 92% (11/12), and 75% (9/12) agreement, respectively.
Journal of AAPOS
Volume 13 Number 2 / April 2009
FIG 1. Appearance of image after analysis by ROPtool. Small circle is optic nerve border; large dotted circle is 28 D circle; straight lines are borders of quadrants; wiggly lines are traced vessels whose tortuosity has been calculated.
For the second phase of the study, all 184 images were initially reviewed by one of the authors (SJ). Five images were excluded from both quadrant-level and eye-level analyses because it was the consensus of 2 authors (SJ and DW) that arterioles and venules could not reliably be differentiated from one another. Typically, these images did not have sufficient clarity as the result of poor focus. Seventyseven of the remaining 179 images (43%) had sufficient quality, focus, and centering that at least one arteriole and one venule could be traced by ROPtool in all 4 quadrants, and only these images were included in the eye-level analysis. Excluded images had at least one quadrant where an arteriole or venule could not be identified by ROPtool. Using as the reference standard expert consensus (EROP) as to whether there was eye-level plus tortuosity, pre-plus tortuosity, or neither, areas under the ROC curves for identification of tortuosity sufficient for plus disease were 0.97, 0.80, and 0.97 for arterioles, venules, and both, respectively. Areas under the ROC curves for identification of tortuosity sufficient for eye-level pre-plus disease were 0.92, 0.69, and 0.93 for arterioles, venules, and both, respectively. Five hundred and forty quadrants that had at least one traceable arteriole and one traceable venule were included in the quadrant-level analyses; all other quadrants were excluded. Using expert consensus (EROP) as to whether there was quadrant-level plus tortuosity, pre-plus tortuosity, or neither as the reference standard, areas under the ROC curves for identification of tortuosity sufficient for plus disease were 0.91, 0.70, and 0.93 for arterioles,
Journal of AAPOS
Johnston et al
183
FIG 2. Receiver operating characteristic curves for ROPtool’s detection of quadrant-level tortuosity sufficient for plus disease using arterioles alone and venules alone in comparison with the consensus of 3 experts.
venules, and both, respectively. Areas under the ROC curves for identification of quadrant-level tortuosity sufficient for pre-plus disease were 0.91, 0.63, and 0.90 for arterioles, venules, and both respectively. Figure 2 shows quadrant-level ROC curves for arterioles alone and venules alone in determining tortuosity sufficient for plus disease. Each point on the ROC curve represents a different numeric threshold (or ‘‘cut-off point’’) for tortuosity, and each point is associated with a distinct sensitivity and specificity. The ROC curves based on quadrant-level data show that the optimum cut-off point (ie, maximizing sensitivity without greatly compromising specificity) is 16 tortuosity units for arteriolar plus tortuosity, resulting in a sensitivity of 0.90 and a specificity of 0.78. For arteriolar pre-plus tortuosity (curves not shown), the optimum cut-off point is 12 tortuosity units, resulting in a sensitivity of 0.83 and a specificity of 0.83. Although choosing a different cutoff point could improve specificity, it would compromise the excellent sensitivity that is important when diagnosing plus disease.
Discussion PO examiners had a reasonably good agreement rate of 83% for designating retinal blood vessels as either arterioles or venules. Thus, it is feasible to separate the analysis of arteriolar and venular tortuosity. Although a larger study would be needed to determine whether there are factors predictive of agreement, our preliminary data suggest that image quality, fundus pigmentation, and disease severity had little to no effect on agreement when classifying vessels. Differentiating arterioles from venules is important beyond the refinement of ROPtool for several reasons. First,
184
Johnston et al
plus disease originally was defined by ICROP as being present when ‘‘the vascular changes are so marked that the posterior veins are enlarged and the arterioles tortuous.’’1 This statement implies that arteries and veins should be considered separately when evaluating for the presence of plus disease. Second, our results give some insight into how experts judge plus disease. We found that expert opinion agreed more with ROPtool’s measurement of arteriolar tortuosity than with venular tortuosity, suggesting that experts primarily focus on arterioles when judging vascular tortuosity. Third, ROPtool is one of several computer programs that are being used to quantify plus disease.9-13 For any program to be refined and applied to retinal images, it is important to know whether differentiating arterioles from venules is feasible, and whether doing so affects calculations obtained from these programs. Finally, quantification of vascular changes has application beyond retinopathy of prematurity. For example, arteriolar narrowing has been associated with risk of developing hypertension,14 diabetes,15 and coronary artery disease.16 Because retinal arterioles and venules respond differently to systemic disease, it may be important to distinguish arteriolar from venular changes to most accurately determine disease risk. This study showed that the use of arterioles alone or using arterioles and venules together provides very good diagnostic accuracy for ROPtool’s determination of tortuosity sufficient for plus disease or pre-plus disease. However, venule tortuosity alone did not have good diagnostic accuracy. These results are consistent with the definitions of plus and pre-plus disease, which highlight the importance of tortuosity of arterioles and dilation of venules. Venules are important in determining plus disease, but it seems that their primary contribution is to vessel dilation rather than tortuosity. A dilation measurement has recently been added to ROPtool capabilities, but it has not yet been validated. Other programs are available that can grade venular dilation.9-13 Once tortuosity is known, it is possible that the measurement of dilation may have little influence in the overall assessment of plus disease. Kylstra and colleagues (Kylstra JA; et al. IOVS 1995; 36:ARVO Abstract 77) found that computer-assisted measurement of plus disease based only on tortuosity showed high sensitivity (85%) and specificity (91%) when compared with expert evaluation. Yanovitch and colleagues17 found that both dilation and tortuosity are important to consider in the clinical diagnosis of plus disease. When severe plus disease develops, both arterioles and venules become tortuous and dilated, so it might be expected that venular tortuosity would have fared better in this study. There are several possible explanations for the poor diagnostic accuracy of venular tortuosity alone. First, arteriolar tortuosity often is present without venular tortuosity, especially when plus disease is not severe. Second, our assessment of diagnostic accuracy was based on agreement with the consensus of experts, and it is possible that the experts focused on arteriolar tortuosity when determin-
Volume 13 Number 2 / April 2009 ing the reference standard for comparison. The most tortuous vessel in each quadrant is likely to be an arteriole, and that vessel may ‘‘catch the eye’’ of examiners and greatly influence their tortuosity judgment. Third, when venular tortuosity does occur, the amount of tortuosity may be relatively less than that observed with arterioles. Consequently, there was a smaller spread of tortuosity values with venules, resulting in greater misclassification of plus disease by ROPtool. It is possible that tortuosity is less in venules than in arterioles because veins are more distensible than arteries. If plus disease is the result of increased flow through arteriovenous shunts (one plausible hypothesis),18 then venules may respond at least initially by dilating instead of becoming tortuous. There are several limitations to this study. First, many images could not be used because ROPtool would not analyze both an arteriole and a venule in all 4 quadrants because of insufficient quality, focus, and/or centering. Second, it is possible that some vessels were not classified correctly as arteriole or venule. To limit this misclassification, we excluded images when we could not be reasonably certain of the vessel type after review by two authors. Third, there is correlation between tortuosity of vessels in the same eye or image, so values for quadrant-level analyses are not completely independent of each other. For this reason, standard statistical comparisons of areas under the curves could not be performed. Fourth, we assessed tortuosity only, and diagnosing plus disease depends on both dilation and tortuosity. We are currently refining and testing ROPtool’s dilation measure, and it is possible that including dilation will improve diagnostic accuracy. Fifth, because we analyzed RetCam images, our results cannot be applied directly to findings by indirect ophthalmoscopy. In conclusion, our results suggest that considering arterioles alone in computer-assisted diagnosis is probably sufficient for determining plus or pre-plus tortuosity. These data will be used to modify ROPtool so that it focuses on arterioles when measuring tortuosity. Much more work lies ahead to determine the contribution of dilation to the overall assessment of plus disease and to validate ROPtool’s overall assessment of plus and pre-plus disease using both tortuosity and dilation.
Acknowledgments We acknowledge Michael Chiang and Graham Quinn for serving as expert graders. We acknowledge Michael Chiang, Anna Ells, and the PHOTO-ROP study group for sharing RetCam photographs.
References 1. The Committee for the Classification of Retinopathy of Prematurity. An international classification of retinopathy of prematurity. Arch Ophthalmol 1984;102:1130-34. 2. The International Classification of Retinopathy of Prematurity revisited. Arch Ophthalmol 2005;123:991-9. 3. Multicenter trial of cryotherapy for retinopathy of prematurity. Preliminary results. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Arch Ophthalmol 1988;106:471-9.
Journal of AAPOS
Volume 13 Number 2 / April 2009 4. Chiang MF, Jiang L, Gelman R, Du YE, Flynn JT. Interexpert agreement of plus disease diagnosis in retinopathy of prematurity. Arch Ophthalmol 2007;125:875-80. 5. Wallace DK, Quinn GE, Freedman SF, Chiang MF. Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. J AAPOS 2008;12:352-6. 6. Wallace DK, Zhao Z, Freedman SF. A pilot study using ‘‘ROPtool’’ to quantify plus disease in retinopathy of prematurity. J AAPOS 2007;11: 381-7. 7. Wallace DK, Freedman SF, Zhao Z, Jung SH. Accuracy of ROPtool vs individual examiners in assessing retinal vascular tortuosity. Arch Ophthalmol 2007;125:1523-30. 8. Wallace DK, Jomier J, Aylward SR, Landers MB, 3rd. Computer-automated quantification of plus disease in retinopathy of prematurity. J AAPOS 2003;7:126-30. 9. Gelman R, Martinez-Perez ME, Vanderveen DK, Moskowitz A, Fulton AB. Diagnosis of plus disease in retinopathy of prematurity using retinal image multiscale analysis. Invest Ophthalmol Vis Sci 2005; 46:4734-8. 10. Heneghan C, Flynn J, O’Keefe M, Cahill M. Characterization of changes in blood vessel width and tortuosity in retinopathy of prematurity using image analysis. Medical Image Analysis 2002;6:407-29. 11. Swanson CR, Cocker KD, Parker KH, Moseley MJ, Wren SME, Fielder AR. Semi-automated computer analysis of vessel growth in preterm infants without and with ROP. Br J Ophthalmol 2003;87:1474-7.
Journal of AAPOS
Johnston et al
185
12. Johnson KS, Mills MD, Karp KA, Grunwald JE. Semiautomated analysis of retinal vessel diameter in retinopathy of prematurity patients with and without plus disease. Am J Ophthalmol 2007;143: 723-35. 13. Wilson CM, Cocker KD, Moseley MJ, Paterson C, Clay ST, Schulenburg WE, et al. Computerized analysis of retinal vessel width and tortuosity in premature infants. Invest Ophthalmol Vis Sci 2008; 49:3577-85. 14. Wong TY, Klein R, Sharrett AR, Duncan BB, Couper DJ, Klein BE, et al. Retinal arteriolar diameter and risk for hypertension. Ann Intern Med 2004;17(140):248-55. 15. Wong TY, Klein R, Sharrett AR, Schmidt MI, Pankow JS, Couper DJ, et al. Retinal arteriolar narrowing and risk of diabetes mellitus in middle-aged persons. JAMA 2002;287:2528-33. 16. Wong TY, Klein R, Sharrett AR, Duncan BB, Couper DJ, Tielsch JM, et al. Retinal arteriolar narrowing and risk of coronary heart disease in men and women. The Atherosclerosis Risk in Communities Study. JAMA 2002;6(287):1153-9. 17. Yanovitch TL, Freedman SF, Wallace DK. The Importance of dilation and tortuosity in diagnosing plus disease. Arch Ophthalmol 2009; in press. 18. Schaffer DB, Palmer EA, Plotsky DF, Metz HS, Flynn JT, Tung B, et al, for the Prognostic factors in the natural course of retinopathy of prematurity. Cryotherapy for Retinopathy of Prematurity Cooperative Group. Ophthalmology 1993;100:230-37.