- Email: [email protected]
Evaluation of Screening for Retinopathy of Prematurity by ROPtool or a Lay Reader
Evaluation of Screening for Retinopathy of Prematurity by ROPtool or a Lay Reader Ashkan M. Abbey, MD,1,2 Cagri G. Besirli, MD, PhD,3 David C. Musch, PhD, MPH,3,4 Chris A. Andrews, PhD,3 Antonio Capone, Jr., MD,1,2 Kimberly A. Drenser, MD, PhD,1,2 David K. Wallace, MD, MPH,5 Susan Ostmo, MS,6 Michael Chiang, MD,6,7 Paul P. Lee, MD, JD,3 Michael T. Trese, MD1,2 Purpose: To determine if (1) tortuosity assessment by a computer program (ROPtool, developed at the University of North Carolina, Chapel Hill, and Duke University, and licensed by FocusROP) that traces retinal blood vessels and (2) assessment by a lay reader are comparable with assessment by a panel of 3 retinopathy of prematurity (ROP) experts for remote clinical grading of vascular abnormalities such as plus disease. Design: Validity and reliability analysis of diagnostic tools. Participants: Three hundred thirty-five fundus images of prematurely born infants. Methods: Three hundred thirty-five fundus images of prematurely born infants were obtained by neonatal intensive care unit nurses. A panel of 3 ROP experts graded 84 images showing vascular dilatation, tortuosity, or both and 251 images showing no evidence of vascular abnormalities. These images were sent electronically to an experienced lay reader who independently graded them for vascular abnormalities. The images also were analyzed using the ROPtool, which assigns a numerical value to the level of vascular abnormality and tortuosity present in each of 4 quadrants or sectors. The ROPtool measurements of vascular abnormalities were graded and compared with expert panel grades with a receiver operating characteristic (ROC) curve. Grades between human readers were cross-tabulated. The area under the ROC curve was calculated for the ROPtool, and sensitivity and specificity were computed for the lay reader. Main Outcome Measures: Measurements of vascular abnormalities by ROPtool and grading of vascular abnormalities by 3 ROP experts and 1 experienced lay reader. Results: The ROC curve for ROPtool’s tortuosity assessment had an area under the ROC curve of 0.917. Using a threshold value of 4.97 for the second most tortuous quadrant, ROPtool’s sensitivity was 91% and its specificity was 82%. Lay reader sensitivity and specificity were 99% and 73%, respectively, and had high reliability (k, 0.87) in repeated measurements. Conclusions: ROPtool had very good accuracy for detection of vascular abnormalities suggestive of plus disease when compared with expert physician graders. The lay reader’s results showed excellent sensitivity and good specificity when compared with those of the expert graders. These options for remote reading of images to detect vascular abnormalities deserve consideration in the quest to use telemedicine with remote reading for efficient delivery of high-quality care and to detect infants requiring bedside examination. Ophthalmology 2015;-:1e6 ª 2015 by the American Academy of Ophthalmology.
The role of telemedicine in ophthalmology continues to expand at a rapid pace. Several previous studies have shown that telemedicine can be used safely and effectively in the screening of infants with retinopathy of prematurity (ROP).1e7 Recent studies show that nonphysician readers can perform ROP screening and assess the need for bedside examination with good accuracy.5,7 ROPtool (developed at the University of North Carolina, Chapel Hill, and Duke University, and licensed by FocusROP)6,8e17 is a computer program that traces vessels in retinal photographs to generate a numerical value for dilation and tortuosity. The software’s detection of vascular abnormalities may be a useful indicator of the necessity for a bedside examination for a premature infant by an experienced ROP diagnostician. In this study, we assessed whether the presence of these vascular abnormalities could be determined effectively by a lay reader or by using the ROPtool. 2015 by the American Academy of Ophthalmology Published by Elsevier Inc.
Methods This study involved only coded private information that could not be linked to a specific individual by the investigators directly or indirectly through a coding system. Fundus images were analyzed without any patient identifiers, and no human subjects were involved in the study. In accordance with Office for Human Research Protections guidance on this subject (see http://www.hhs.gov/ohrp/humansubjects/guidance/cdebiol.htm), institutional review board approval was not required, because the data cannot be tracked to a human subject.
Data Acquisition Three hundred thirty-five fundus photographs of premature infants were obtained by neonatal intensive care unit nurses. One hundred ninety-five of these images were obtained from the PhotoROP study.3 Posterior pole images of both eyes were used in 32 infants. Each eye had an average of 6.1 images; however, repeat images http://dx.doi.org/10.1016/j.ophtha.2015.09.048 ISSN 0161-6420/15
1
Ophthalmology Volume -, Number -, Month 2015 occurred at least 1 week apart. One hundred forty additional posterior pole images of control eyes not involved in the PhotoROP study also were included.
Image Grading These images were sent electronically using the FocusROP software program (available at: http://www.focusrop.com) to 3 ROP experts (M.T.T., A.C., K.A.D.) who graded the vessel characteristics in each of 4 quadrants or sectors around the optic nerve. Experts were defined as practicing pediatric retina specialists who met 1 or more of 3 criteria: having been a study center principal investigator for the Cryotherapy for Retinopathy of Prematurity (CRYO-ROP) or Early Treatment for Retinopathy of Prematurity (ETROP) studies, having been a certified investigator for either study, or having coauthored 5 or more peer-reviewed ROP manuscripts.18 The experts read all of the images together and rendered consensus grades. They graded each section from 0 to 4 relative to vascular abnormalities such as vessel tortuosity and dilatation as follows: 0 ¼ none; 1 ¼ questionable; 2 ¼ mild dilatation; 3 ¼ moderate dilatation, mild tortuosity; and 4 ¼ obvious dilatation and tortuosity. This classification as read by the ROP experts was treated as the gold standard for vascular status. Images were classified as control images (n ¼ 251) if all quadrants or sectors were 0 and as vascular abnormality images (n ¼ 84) if any quadrant or sector was more than 0. For the purposes of this validation study for detection of vascular abnormalities, identification of 1 quadrant or sector as more than 0 was considered abnormal. All of these images (control and vascular abnormality) were sent to a trained, experienced lay reader (S.O.) who graded the images using the same 0-to-4 scale. The lay reader had been an ROP study coordinator for 3 years, during which time she participated in weekly ROP rounds, which included fundus photography and discussions with an ROP expert after each examination. She coordinates an ROP project that involves review and selection of fundus photographs from multiple clinical sites. Three months later, the lay reader was sent 50 images for regrading to assess intraobserver variability. The 50 images were selected randomly from the 195 PhotoROP study images.
ROPtool These images also were analyzed with the ROPtool. When analyzing a fundus photograph using the ROPtool, the operator first clicks on the superior, inferior, nasal, and temporal borders of the optic nerve, and then on the center of the macula (fovea). The program then defines the posterior pole as a circle whose radius extends from the center of the optic nerve to the fovea. This circle is divided electronically into quadrants, and the borders of these quadrants can be adjusted by the operator such that each quadrant may not represent a 90 angle, but actually forms 4 sectors. Within each quadrant or sector, the most dilated and the most tortuous major vessels are selected, and the ROPtool automatically traces these vessels and measures their tortuosity and dilation (Fig 1). Tortuosity is a ratio of the total length of the vessel divided by a smooth curve generated from points spaced approximately 40 pixels apart on the vessel.17 An average vessel has 4 to 5 points from which a smooth curve is generated. A value for each quadrant or sector for tortuosity and dilation is based on values for the most tortuous and most dilated major vessels in that quadrant or section. The second largest of the 4 tortuosity quadrant or sector measurements was used to define the tortuosity score for each eye.
2
Figure 1. Screenshot from the ROPtool software demonstrating 4 sectors and their respective tortuosity values, which are based on the most tortuous vessels in each sector. The second largest of the 4 tortuosity sector measurements was used to define the overall tortuosity score for each eye.
Statistical Analysis Standard graphical (boxplots) and numerical (quartiles) summaries were used to summarize the distributions of ROPtool tortuosity within the gold standard categories and the 5-point scale categories. Quadrants or sectors with missing values were excluded. A receiver operating characteristic (ROC) curve and the area under the ROC curve were used to assess the predictive value of ROPtool measurements of tortuosity versus the gold standard. Standard errors were estimated by nonparametric bootstrap assuming independence of observations. Logistic regression modeling was used to build predictive models for the gold standard using both tortuosity and dilatation of all quadrants or sectors. Quadrant or sector grades by the expert panel and by the lay reader were cross-tabulated. Sensitivity and specificity were computed to assess ability of determining disease status. The unweighted k statistic was used to assess intraobserver variability in the lay reader’s repeated grading. k Values range from þ1 (complete agreement) to 0 (observed agreement equal to chance alone) to negative values (observed agreement less than chance agreement). To check the possibility that the source of image acquisition affected outcomes, the analysis was repeated excluding the non-PhotoROP images. No substantive differences were noted. Additional sensitivity analyses using the PhotoROP images were conducted allowing for dependence among images from the same infant. Dependence among images may reduce the effective sample
Table 1. Distribution of Tortuosity Assessed with the ROPtool within Control Eyes and Case Eyes (Eyes with Evidence of Vascular Abnormalities) Tortuosity
No. read by ROPtool Unable to be read by ROPtool Maximum tortuosity Third quartile tortuosity Median tortuosity First quartile tortuosity Minimum tortuosity
Controls
Cases
238.00 13.00 14.25 6.01 4.86 3.99 2.09
74.00 10.00 46.77 20.19 13.05 8.80 3.54
Abbey et al
ROP Screening by ROPtool or a Lay Reader
Figure 2. Boxplots displaying distribution of tortuosity assessed by the ROPtool within control (Ctrl) eyes and case eyes (i.e., eyes with evidence of vascular abnormalities). Controls from 2 data sources also are graphed separately.
size. In particular, we implemented a cluster bootstrap that resampled at the infant level (rather than the image level) for the 195 PhotoROP images on which we had study identification numbers. The net effect was to increase the length of the area under the ROC curve confidence interval by 23%. Analyses were performed in the R statistical computing environment version 3.1 (R Foundation for Statistical Computing, Vienna, Austria).
Results Tortuosity measurements were compared between images of control eyes and eyes with any vascular abnormality (Table 1). The ROPtool cannot read some poor-quality images if the vessels are blurred or indistinct. If the vessels are clear and distinct, then ROPtool generally can trace them. Thirteen control eyes and 10 eyes with evidence of vascular abnormalities were unable to be read by the ROPtool and therefore were excluded, leaving a total of 238 control eyes and 74 case eyes for analysis. The median quadrant tortuosity value was higher for eyes classified with the vascular abnormality of plus disease
Figure 3. Boxplots displaying distribution of tortuosity assessed with the ROPtool within 5 grades by retinopathy of prematurity experts.
(13.1; interquartile range, 8.8e20.2) than for those classified as controls (4.9; interquartile range, 4.0e6.0; Table 1). The PhotoROP control eyes and the additional control eyes had similar tortuosity values (Fig 2).
3
Ophthalmology Volume -, Number -, Month 2015 in the internal grades when trying to distinguish between questionable and mild. The lay reader’s grading of plus disease (Table 3) had a sensitivity of 99% (83 of 84) and specificity of 73% (183 of 251). Intraobserver agreement based on repeated grading of 200 quadrants of 50 images was very high (k ¼ 0.87).
Discussion
Figure 4. Receiver operating characteristic curve depicting the ability of the second largest tortuosity assessed with the ROPtool to differentiate control eyes and case eyes (i.e., eyes with evidence of vascular abnormalities). The marked point indicates the lowest tortuosity value (4.11) with sensitivity at least 95%, which has specificity 64%. AUC ¼ area under the receiver operating characteristic curve.
The tortuosity values generated by ROPtool showed a predictable difference based on the grades given to the quadrants or sectors by the ROP experts (Fig 3). Tortuosity measurements were higher for quadrants or sectors classified as obvious dilatation and tortuosity (grade 4) and were lower for quadrants classified as none (grade 0). In the 3 middle categories (questionable, mild, and moderate), tortuosity measurements were similar. The specificity and sensitivity of ROPtool in differentiating control eyes and eyes with evidence of vascular abnormalities were plotted in a ROC curve (Fig 4). With respect to the second largest tortuosity quadrant, the area under the ROC curve was 0.917 (95% confidence interval, 0.879e0.955). When a cutoff tortuosity value of 4.11 (highlighted point in Fig 4) was chosen to give 95% sensitivity or more, the ROPtool correctly identified 71 of 74 eyes with the vascular abnormality of plus disease (96% sensitivity) and 152 of 238 control eyes (64% specificity). When a cutoff tortuosity value of 4.97 was chosen to give 90% or more sensitivity, the ROPtool correctly identified 67 of 74 case eyes (91% sensitivity) and 194 of 238 control eyes (82% specificity). If an unreadable image was categorized automatically as a case and a cut-off value was chosen to give 95% or more sensitivity, the sensitivity was 96% (81 of 84) and the specificity was 61% (152 of 251). Additional modeling strategies beyond the use of the second largest tortuosity quadrant or sector were analyzed with the ROC curve. These strategies included using 2 or more tortuosity measurements jointly, using dilatation and tortuosity in combination, and using quadrant or sector information. However, none of these models improved discriminatory power. Cross-tabulation of quadrant or sector grades (Table 2) showed good agreement between the lay reader and the expert panel for grades at the ends of the scale (i.e., 0 and 4), but more variability
4
For decades, the standard protocol for ROP screening has involved bedside examination by a physician, who typically made a drawing to document the findings of a clinical examination. However, there can be considerable variation in drawings between physicians, and monitoring newborns over time in neonatal intensive care units often involves more than 1 examiner because of examiner availability. Such inconsistencies and subjectivity may contribute to screening errors or missed diagnoses.19 Retinopathy of prematurity is a time-dependent disease that can progress very rapidly from early stages to retinal detachment causing blindness. Therefore, it is critical to have reliable, timely examinations to monitor its progression. These examinations can be documented accurately and consistently by photographic screening, which can reduce the cost and burden of screening by physicians and may help to reduce subjectivity and improve appropriate diagnosis and management. Numerous previous studies have shown that photographic documentation with remote interpretation can yield results that are comparable with bedside examination.20e26 Retinopathy of prematurity diagnosis and treatment tends to be a very litigious area of medicine, and as a result it is one of the most costly areas in which to be insured for both ophthalmologists and neonatologists.19 Photographic documentation may provide excellent medicolegal protection for the screening physician. It can be particularly helpful in demonstrating that treatment was administered before the occurrence of retinal detachment in the earlier stages of ROP. The presence of the vascular abnormality of plus disease is a defining characteristic of type 1, or treatment-warranted, ROP, and this characteristic can be detected photographically in the posterior pole. Prior studies have demonstrated that photographic screening is able to detect treatment-warranted ROP with a high degree of sensitivity and specificity. In this study, reliance on ROPtool’s measurement of tortuosity yielded a highly sensitive (96%) detection of plus disease (specificity, 64%) at the selected tortuosity value of Table 2. Distribution of Lay Reader Quadrant Grades within Gold Standard Grades Lay Reader Grades Gold Standard Grades 0 1 2 3 4
(none) (questionable) (mild) (moderate) (obvious)
Data are number (%).
0 867 15 8 0 0
(83) (27) (9) (0) (0)
1 5 2 2 1 7
2
3
(0) 111 (11) 62 (6) 2 (4) 16 (29) 21 (38) 2 (2) 14 (15) 62 (67) 7 (3) 4 (10) 23 (59) 11 (8) 3 (4) 30 (35) 45
4
Total
(<1) 1047 (4) 56 (8) 93 (28) 39 (53) 85
Abbey et al
ROP Screening by ROPtool or a Lay Reader
Table 3. Distribution of Lay Reader Abnormality Assessment within Gold Standard Case and Control Groups Lay Reader Grades Gold Standard Groups
Negative
Positive
Total
Controls Cases
183 (73) 1 (1)
68 (27) 83 (99)
251 84
Negative indicates all quadrants were graded 0. Positive indicates at least 1 quadrant with a nonzero grade. Data are number (%).
4.11. With respect to tortuosity, the area under the ROC curve was 0.917, indicating that the ROPtool tortuosity measurement has very good discriminatory power to differentiate between control eyes and eyes with plus disease. Using dilation and tortuosity in combination did not improve discriminatory power. Relying solely on tortuosity measurements, the ROPtool was quite accurate in determining the presence of a vascular abnormality and the need for a bedside examination. Reliance on a trained and experienced lay reader, who was sent the images, also yielded very high sensitivity (99%) and specificity (73%). A previous study that compared the accuracy of ROP screening between specialist nurses and an expert pediatric ophthalmologist demonstrated a slightly lower sensitivity (91.7%) and slightly higher specificity (80.6%).7 Although the sensitivity results indicated that false-negative results were minimal, which is a critically important finding for both the lay reader and ROPtool approaches, both yielded specificity values (lay reader, 73%; ROPtool, 64%) that would result in many false-positive results. These false-positive readings could be adjudicated by an expert in ROP evaluation by remote viewing of the retinal images, with the attendant added time and cost. This type of telemedicine screening for ROP has several barriers to implementation. Currently, cameras available for appropriate telemedicine screening are very expensive and cost between $70 000 and $145 000 (United States dollars). Backup cameras also are necessary, which makes that price range inaccessible to many hospitals. Fortunately, several cameras with increased illumination and resolution at reduced cost are in development. Software also has been developed that can minimize image handling and reduce human error. Some physicians fear that such rigorous photographic documentation could provide greater ease for malpractice proceedings against them. However, the combination of photographic documentation and computer-based image analysis should give physicians added confidence in their ability to interpret clinical images accurately and to seek expert opinions. This will lead to improved outcomes and less medical error, thus providing less subjective criteria for early treatment and better care for the infant at risk of blindness resulting from ROP. Finally, several clear barriers to the integration of telemedicine in ROP screening include privacy and licensure constraints, as well as Food and Drug Administration regulations that govern the allocation of devices and software in healthcare.
Two limitations of ROPtool should be noted. The software could not read a small number of poor-quality images. Furthermore, image analysis performed by ROPtool is not fully automated; an operator must adjust the sectors and identify the vessels to be analyzed. The role of telemedicine in ROP, as well as in other ophthalmic diseases, continues to expand. Appropriate screening is undoubtedly the most important aspect of the management of ROP. Computerized interpretation of images using software such as ROPtool reduces the subjectivity of detecting vascular abnormalities that trigger the need for a bedside examination. This technology will aid in the quest for high-quality care for infants.
References 1. Chiang MF, Wang L, Busuioc M, et al. Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol 2007;125:1531–8. 2. Lorenz B, Spasovska K, Elflein H, Schneider N. Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP). Six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol 2009;247:1251–62. 3. Photographic Screening for Retinopathy of Prematurity (Photo-ROP) Cooperative Group. The photographic screening for retinopathy of prematurity study (photo-ROP). Primary outcomes. Retina 2008;28(3 Suppl):S47–54. 4. Dhaliwal C, Wright E, Graham C, et al. Wide-field digital retinal imaging versus binocular indirect ophthalmoscopy for retinopathy of prematurity screening: a two-observer prospective, randomised comparison. Br J Ophthalmol 2009;93:355–9. 5. Williams SL, Wang L, Kane SA, et al. Telemedical diagnosis of retinopathy of prematurity: accuracy of expert versus nonexpert graders. Br J Ophthalmol 2010;94:351–6. 6. Quinn GE, Ying G, Daniel E, et al. Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity. JAMA Ophthalmol 2014;132:1178–84. 7. Shah SP, Wu Z, Iverson S, Dai S. Specialist nurse screening for retinopathy of prematurity: a pilot study. Asia Pac J Ophthalmol (Phila) 2013;2:300–4. 8. Cabrera MT, Freedman SF, Kiely AE, et al. Combining ROPtool measurements of vascular tortuosity and width to quantify plus disease in retinopathy of prematurity. J AAPOS 2011;15:40–4. 9. Wittenberg LA, Jonsson NJ, Chan RV, Chiang MF. Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity. J Pediatr Ophthalmol Strabismus 2012;49:11–9. quiz 0, 20. 10. Kiely AE, Wallace DK, Freedman SF, Zhao Z. Computerassisted measurement of retinal vascular width and tortuosity in retinopathy of prematurity. Arch Ophthalmol 2010;128: 847–52. 11. Wallace DK, Freedman SF, Zhao Z. Evolution of plus disease in retinopathy of prematurity: quantification by ROPtool. Trans Am Ophthalmol Soc 2009;107:47–52. 12. Wallace DK, Freedman SF, Zhao Z. A pilot study using ROPtool to measure retinal vascular dilation. Retina 2009;29:1182–7. 13. Johnston SC, Wallace DK, Freedman SF, et al. Tortuosity of arterioles and venules in quantifying plus disease. J AAPOS 2009;13:181–5. 14. Ahmad S, Wallace DK, Freedman SF, Zhao Z. Computerassisted assessment of plus disease in retinopathy of
5
Ophthalmology Volume -, Number -, Month 2015
15.
16. 17. 18.
19. 20.
prematurity using video indirect ophthalmoscopy images. Retina 2008;28:1458–62. Wallace DK. Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc 2007;105:594–615. Wallace DK, Freedman SF, Zhao Z, Jung SH. Accuracy of ROPtool vs individual examiners in assessing retinal vascular tortuosity. Arch Ophthalmol 2007;125:1523–30. Wallace DK, Zhao Z, Freedman SF. A pilot study using “ROPtool” to quantify plus disease in retinopathy of prematurity. J AAPOS 2007;11:381–7. Chiang MF, Gelman R, Williams SL, et al. Plus disease in retinopathy of prematurity: development of composite images by quantification of expert opinion. Invest Ophthalmol Vis Sci 2008;49:4064–70. Ophthalmic Mutual Insurance Company Retinopathy of Prematurity Task Force. Retinopathy of Prematurity: Creating a Safety Net. Ophthalmic Mutual Insurance Company 2013. Chiang MF, Starren J, Du YE, et al. Remote image based retinopathy of prematurity diagnosis: a receiver operating characteristic analysis of accuracy. Br J Ophthalmol 2006;90: 1292–6.
21. Ells AL, Holmes JM, Astle WF, et al. Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology 2003;110:2113–7. 22. Fijalkowski N, Zheng LL, Henderson MT, et al. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): five years of screening with telemedicine. Ophthalmic Surg Lasers Imaging Retina 2014;45:106–13. 23. Schwartz SD, Harrison SA, Ferrone PJ, Trese MT. Telemedical evaluation and management of retinopathy of prematurity using a fiberoptic digital fundus camera. Ophthalmology 2000;107:25–8. 24. Wu C, Petersen RA, VanderVeen DK. RetCam imaging for retinopathy of prematurity screening. J AAPOS 2006;10: 107–11. 25. Chiang MF, Keenan JD, Starren J, et al. Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol 2006;124:322–7. 26. Early Treatment for Retinopathy of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the Early Treatment for Retinopathy of Prematurity Randomized Trial. Arch Ophthalmol 2003;121: 1684–94.
Footnotes and Financial Disclosures Originally received: July 2, 2015. Final revision: September 29, 2015. Accepted: September 30, 2015. Available online: ---.
K.A.D.: Equity owner e ROPtool D.K.W.: Equity owner e ROPtool Manuscript no. 2015-1127.
1
Department of Ophthalmology, William Beaumont Hospital, Royal Oak, Michigan.
2
Associated Retinal Consultants, Royal Oak, Michigan. Department of Ophthalmology and Visual Sciences, Kellogg Eye Center, University of Michigan, Ann Arbor, Michigan.
M.T.T.: Equity owner e ROPtool Author Contributions: Conception and design: Abbey, Besirli, Musch, Andrews, Capone, Drenser, Wallace, Ostmo, Chiang, Lee, Trese
3
Analysis and interpretation: Abbey, Besirli, Musch, Andrews, Capone, Drenser, Ostmo, Chiang, Lee, Trese
4
Data collection: Abbey, Besirli, Musch, Andrews, Capone, Drenser, Wallace, Ostmo, Chiang, Trese
Department of Epidemiology, University of Michigan, Ann Arbor, Michigan.
5
Obtained funding: none
6
Overall responsibility: Abbey, Besirli, Musch, Andrews, Capone, Drenser, Wallace, Ostmo, Chiang, Trese
Department of Ophthalmology, Duke University Eye Center, Durham, North Carolina. Department of Ophthalmology, Oregon Health and Science University, Portland, Oregon.
7
Department of Medical Informatics & Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon. Financial Disclosure(s): The author(s) have made the following disclosure(s): A.C.: Equity owner e ROPtool, developed at the University of North Carolina, Chapel Hill, and Duke University, and licensed by FocusROP.
6
Abbreviations and Acronyms: ROC ¼ receiver operating characteristic; ROP ¼ retinopathy of prematurity. Correspondence: Michael T. Trese, MD, Associated Retinal Consultants, 3535 West 13 Mile Road, Suite 344, Royal Oak, MI 48073. E-mail: [email protected].
The role of telemedicine in ophthalmology continues to expand at a rapid pace. Several previous studies have shown that telemedicine can be used safely and effectively in the screening of infants with retinopathy of prematurity (ROP).1e7 Recent studies show that nonphysician readers can perform ROP screening and assess the need for bedside examination with good accuracy.5,7 ROPtool (developed at the University of North Carolina, Chapel Hill, and Duke University, and licensed by FocusROP)6,8e17 is a computer program that traces vessels in retinal photographs to generate a numerical value for dilation and tortuosity. The software’s detection of vascular abnormalities may be a useful indicator of the necessity for a bedside examination for a premature infant by an experienced ROP diagnostician. In this study, we assessed whether the presence of these vascular abnormalities could be determined effectively by a lay reader or by using the ROPtool. 2015 by the American Academy of Ophthalmology Published by Elsevier Inc.
Methods This study involved only coded private information that could not be linked to a specific individual by the investigators directly or indirectly through a coding system. Fundus images were analyzed without any patient identifiers, and no human subjects were involved in the study. In accordance with Office for Human Research Protections guidance on this subject (see http://www.hhs.gov/ohrp/humansubjects/guidance/cdebiol.htm), institutional review board approval was not required, because the data cannot be tracked to a human subject.
Data Acquisition Three hundred thirty-five fundus photographs of premature infants were obtained by neonatal intensive care unit nurses. One hundred ninety-five of these images were obtained from the PhotoROP study.3 Posterior pole images of both eyes were used in 32 infants. Each eye had an average of 6.1 images; however, repeat images http://dx.doi.org/10.1016/j.ophtha.2015.09.048 ISSN 0161-6420/15
1
Ophthalmology Volume -, Number -, Month 2015 occurred at least 1 week apart. One hundred forty additional posterior pole images of control eyes not involved in the PhotoROP study also were included.
Image Grading These images were sent electronically using the FocusROP software program (available at: http://www.focusrop.com) to 3 ROP experts (M.T.T., A.C., K.A.D.) who graded the vessel characteristics in each of 4 quadrants or sectors around the optic nerve. Experts were defined as practicing pediatric retina specialists who met 1 or more of 3 criteria: having been a study center principal investigator for the Cryotherapy for Retinopathy of Prematurity (CRYO-ROP) or Early Treatment for Retinopathy of Prematurity (ETROP) studies, having been a certified investigator for either study, or having coauthored 5 or more peer-reviewed ROP manuscripts.18 The experts read all of the images together and rendered consensus grades. They graded each section from 0 to 4 relative to vascular abnormalities such as vessel tortuosity and dilatation as follows: 0 ¼ none; 1 ¼ questionable; 2 ¼ mild dilatation; 3 ¼ moderate dilatation, mild tortuosity; and 4 ¼ obvious dilatation and tortuosity. This classification as read by the ROP experts was treated as the gold standard for vascular status. Images were classified as control images (n ¼ 251) if all quadrants or sectors were 0 and as vascular abnormality images (n ¼ 84) if any quadrant or sector was more than 0. For the purposes of this validation study for detection of vascular abnormalities, identification of 1 quadrant or sector as more than 0 was considered abnormal. All of these images (control and vascular abnormality) were sent to a trained, experienced lay reader (S.O.) who graded the images using the same 0-to-4 scale. The lay reader had been an ROP study coordinator for 3 years, during which time she participated in weekly ROP rounds, which included fundus photography and discussions with an ROP expert after each examination. She coordinates an ROP project that involves review and selection of fundus photographs from multiple clinical sites. Three months later, the lay reader was sent 50 images for regrading to assess intraobserver variability. The 50 images were selected randomly from the 195 PhotoROP study images.
ROPtool These images also were analyzed with the ROPtool. When analyzing a fundus photograph using the ROPtool, the operator first clicks on the superior, inferior, nasal, and temporal borders of the optic nerve, and then on the center of the macula (fovea). The program then defines the posterior pole as a circle whose radius extends from the center of the optic nerve to the fovea. This circle is divided electronically into quadrants, and the borders of these quadrants can be adjusted by the operator such that each quadrant may not represent a 90 angle, but actually forms 4 sectors. Within each quadrant or sector, the most dilated and the most tortuous major vessels are selected, and the ROPtool automatically traces these vessels and measures their tortuosity and dilation (Fig 1). Tortuosity is a ratio of the total length of the vessel divided by a smooth curve generated from points spaced approximately 40 pixels apart on the vessel.17 An average vessel has 4 to 5 points from which a smooth curve is generated. A value for each quadrant or sector for tortuosity and dilation is based on values for the most tortuous and most dilated major vessels in that quadrant or section. The second largest of the 4 tortuosity quadrant or sector measurements was used to define the tortuosity score for each eye.
2
Figure 1. Screenshot from the ROPtool software demonstrating 4 sectors and their respective tortuosity values, which are based on the most tortuous vessels in each sector. The second largest of the 4 tortuosity sector measurements was used to define the overall tortuosity score for each eye.
Statistical Analysis Standard graphical (boxplots) and numerical (quartiles) summaries were used to summarize the distributions of ROPtool tortuosity within the gold standard categories and the 5-point scale categories. Quadrants or sectors with missing values were excluded. A receiver operating characteristic (ROC) curve and the area under the ROC curve were used to assess the predictive value of ROPtool measurements of tortuosity versus the gold standard. Standard errors were estimated by nonparametric bootstrap assuming independence of observations. Logistic regression modeling was used to build predictive models for the gold standard using both tortuosity and dilatation of all quadrants or sectors. Quadrant or sector grades by the expert panel and by the lay reader were cross-tabulated. Sensitivity and specificity were computed to assess ability of determining disease status. The unweighted k statistic was used to assess intraobserver variability in the lay reader’s repeated grading. k Values range from þ1 (complete agreement) to 0 (observed agreement equal to chance alone) to negative values (observed agreement less than chance agreement). To check the possibility that the source of image acquisition affected outcomes, the analysis was repeated excluding the non-PhotoROP images. No substantive differences were noted. Additional sensitivity analyses using the PhotoROP images were conducted allowing for dependence among images from the same infant. Dependence among images may reduce the effective sample
Table 1. Distribution of Tortuosity Assessed with the ROPtool within Control Eyes and Case Eyes (Eyes with Evidence of Vascular Abnormalities) Tortuosity
No. read by ROPtool Unable to be read by ROPtool Maximum tortuosity Third quartile tortuosity Median tortuosity First quartile tortuosity Minimum tortuosity
Controls
Cases
238.00 13.00 14.25 6.01 4.86 3.99 2.09
74.00 10.00 46.77 20.19 13.05 8.80 3.54
Abbey et al
ROP Screening by ROPtool or a Lay Reader
Figure 2. Boxplots displaying distribution of tortuosity assessed by the ROPtool within control (Ctrl) eyes and case eyes (i.e., eyes with evidence of vascular abnormalities). Controls from 2 data sources also are graphed separately.
size. In particular, we implemented a cluster bootstrap that resampled at the infant level (rather than the image level) for the 195 PhotoROP images on which we had study identification numbers. The net effect was to increase the length of the area under the ROC curve confidence interval by 23%. Analyses were performed in the R statistical computing environment version 3.1 (R Foundation for Statistical Computing, Vienna, Austria).
Results Tortuosity measurements were compared between images of control eyes and eyes with any vascular abnormality (Table 1). The ROPtool cannot read some poor-quality images if the vessels are blurred or indistinct. If the vessels are clear and distinct, then ROPtool generally can trace them. Thirteen control eyes and 10 eyes with evidence of vascular abnormalities were unable to be read by the ROPtool and therefore were excluded, leaving a total of 238 control eyes and 74 case eyes for analysis. The median quadrant tortuosity value was higher for eyes classified with the vascular abnormality of plus disease
Figure 3. Boxplots displaying distribution of tortuosity assessed with the ROPtool within 5 grades by retinopathy of prematurity experts.
(13.1; interquartile range, 8.8e20.2) than for those classified as controls (4.9; interquartile range, 4.0e6.0; Table 1). The PhotoROP control eyes and the additional control eyes had similar tortuosity values (Fig 2).
3
Ophthalmology Volume -, Number -, Month 2015 in the internal grades when trying to distinguish between questionable and mild. The lay reader’s grading of plus disease (Table 3) had a sensitivity of 99% (83 of 84) and specificity of 73% (183 of 251). Intraobserver agreement based on repeated grading of 200 quadrants of 50 images was very high (k ¼ 0.87).
Discussion
Figure 4. Receiver operating characteristic curve depicting the ability of the second largest tortuosity assessed with the ROPtool to differentiate control eyes and case eyes (i.e., eyes with evidence of vascular abnormalities). The marked point indicates the lowest tortuosity value (4.11) with sensitivity at least 95%, which has specificity 64%. AUC ¼ area under the receiver operating characteristic curve.
The tortuosity values generated by ROPtool showed a predictable difference based on the grades given to the quadrants or sectors by the ROP experts (Fig 3). Tortuosity measurements were higher for quadrants or sectors classified as obvious dilatation and tortuosity (grade 4) and were lower for quadrants classified as none (grade 0). In the 3 middle categories (questionable, mild, and moderate), tortuosity measurements were similar. The specificity and sensitivity of ROPtool in differentiating control eyes and eyes with evidence of vascular abnormalities were plotted in a ROC curve (Fig 4). With respect to the second largest tortuosity quadrant, the area under the ROC curve was 0.917 (95% confidence interval, 0.879e0.955). When a cutoff tortuosity value of 4.11 (highlighted point in Fig 4) was chosen to give 95% sensitivity or more, the ROPtool correctly identified 71 of 74 eyes with the vascular abnormality of plus disease (96% sensitivity) and 152 of 238 control eyes (64% specificity). When a cutoff tortuosity value of 4.97 was chosen to give 90% or more sensitivity, the ROPtool correctly identified 67 of 74 case eyes (91% sensitivity) and 194 of 238 control eyes (82% specificity). If an unreadable image was categorized automatically as a case and a cut-off value was chosen to give 95% or more sensitivity, the sensitivity was 96% (81 of 84) and the specificity was 61% (152 of 251). Additional modeling strategies beyond the use of the second largest tortuosity quadrant or sector were analyzed with the ROC curve. These strategies included using 2 or more tortuosity measurements jointly, using dilatation and tortuosity in combination, and using quadrant or sector information. However, none of these models improved discriminatory power. Cross-tabulation of quadrant or sector grades (Table 2) showed good agreement between the lay reader and the expert panel for grades at the ends of the scale (i.e., 0 and 4), but more variability
4
For decades, the standard protocol for ROP screening has involved bedside examination by a physician, who typically made a drawing to document the findings of a clinical examination. However, there can be considerable variation in drawings between physicians, and monitoring newborns over time in neonatal intensive care units often involves more than 1 examiner because of examiner availability. Such inconsistencies and subjectivity may contribute to screening errors or missed diagnoses.19 Retinopathy of prematurity is a time-dependent disease that can progress very rapidly from early stages to retinal detachment causing blindness. Therefore, it is critical to have reliable, timely examinations to monitor its progression. These examinations can be documented accurately and consistently by photographic screening, which can reduce the cost and burden of screening by physicians and may help to reduce subjectivity and improve appropriate diagnosis and management. Numerous previous studies have shown that photographic documentation with remote interpretation can yield results that are comparable with bedside examination.20e26 Retinopathy of prematurity diagnosis and treatment tends to be a very litigious area of medicine, and as a result it is one of the most costly areas in which to be insured for both ophthalmologists and neonatologists.19 Photographic documentation may provide excellent medicolegal protection for the screening physician. It can be particularly helpful in demonstrating that treatment was administered before the occurrence of retinal detachment in the earlier stages of ROP. The presence of the vascular abnormality of plus disease is a defining characteristic of type 1, or treatment-warranted, ROP, and this characteristic can be detected photographically in the posterior pole. Prior studies have demonstrated that photographic screening is able to detect treatment-warranted ROP with a high degree of sensitivity and specificity. In this study, reliance on ROPtool’s measurement of tortuosity yielded a highly sensitive (96%) detection of plus disease (specificity, 64%) at the selected tortuosity value of Table 2. Distribution of Lay Reader Quadrant Grades within Gold Standard Grades Lay Reader Grades Gold Standard Grades 0 1 2 3 4
(none) (questionable) (mild) (moderate) (obvious)
Data are number (%).
0 867 15 8 0 0
(83) (27) (9) (0) (0)
1 5 2 2 1 7
2
3
(0) 111 (11) 62 (6) 2 (4) 16 (29) 21 (38) 2 (2) 14 (15) 62 (67) 7 (3) 4 (10) 23 (59) 11 (8) 3 (4) 30 (35) 45
4
Total
(<1) 1047 (4) 56 (8) 93 (28) 39 (53) 85
Abbey et al
ROP Screening by ROPtool or a Lay Reader
Table 3. Distribution of Lay Reader Abnormality Assessment within Gold Standard Case and Control Groups Lay Reader Grades Gold Standard Groups
Negative
Positive
Total
Controls Cases
183 (73) 1 (1)
68 (27) 83 (99)
251 84
Negative indicates all quadrants were graded 0. Positive indicates at least 1 quadrant with a nonzero grade. Data are number (%).
4.11. With respect to tortuosity, the area under the ROC curve was 0.917, indicating that the ROPtool tortuosity measurement has very good discriminatory power to differentiate between control eyes and eyes with plus disease. Using dilation and tortuosity in combination did not improve discriminatory power. Relying solely on tortuosity measurements, the ROPtool was quite accurate in determining the presence of a vascular abnormality and the need for a bedside examination. Reliance on a trained and experienced lay reader, who was sent the images, also yielded very high sensitivity (99%) and specificity (73%). A previous study that compared the accuracy of ROP screening between specialist nurses and an expert pediatric ophthalmologist demonstrated a slightly lower sensitivity (91.7%) and slightly higher specificity (80.6%).7 Although the sensitivity results indicated that false-negative results were minimal, which is a critically important finding for both the lay reader and ROPtool approaches, both yielded specificity values (lay reader, 73%; ROPtool, 64%) that would result in many false-positive results. These false-positive readings could be adjudicated by an expert in ROP evaluation by remote viewing of the retinal images, with the attendant added time and cost. This type of telemedicine screening for ROP has several barriers to implementation. Currently, cameras available for appropriate telemedicine screening are very expensive and cost between $70 000 and $145 000 (United States dollars). Backup cameras also are necessary, which makes that price range inaccessible to many hospitals. Fortunately, several cameras with increased illumination and resolution at reduced cost are in development. Software also has been developed that can minimize image handling and reduce human error. Some physicians fear that such rigorous photographic documentation could provide greater ease for malpractice proceedings against them. However, the combination of photographic documentation and computer-based image analysis should give physicians added confidence in their ability to interpret clinical images accurately and to seek expert opinions. This will lead to improved outcomes and less medical error, thus providing less subjective criteria for early treatment and better care for the infant at risk of blindness resulting from ROP. Finally, several clear barriers to the integration of telemedicine in ROP screening include privacy and licensure constraints, as well as Food and Drug Administration regulations that govern the allocation of devices and software in healthcare.
Two limitations of ROPtool should be noted. The software could not read a small number of poor-quality images. Furthermore, image analysis performed by ROPtool is not fully automated; an operator must adjust the sectors and identify the vessels to be analyzed. The role of telemedicine in ROP, as well as in other ophthalmic diseases, continues to expand. Appropriate screening is undoubtedly the most important aspect of the management of ROP. Computerized interpretation of images using software such as ROPtool reduces the subjectivity of detecting vascular abnormalities that trigger the need for a bedside examination. This technology will aid in the quest for high-quality care for infants.
References 1. Chiang MF, Wang L, Busuioc M, et al. Telemedical retinopathy of prematurity diagnosis: accuracy, reliability, and image quality. Arch Ophthalmol 2007;125:1531–8. 2. Lorenz B, Spasovska K, Elflein H, Schneider N. Wide-field digital imaging based telemedicine for screening for acute retinopathy of prematurity (ROP). Six-year results of a multicentre field study. Graefes Arch Clin Exp Ophthalmol 2009;247:1251–62. 3. Photographic Screening for Retinopathy of Prematurity (Photo-ROP) Cooperative Group. The photographic screening for retinopathy of prematurity study (photo-ROP). Primary outcomes. Retina 2008;28(3 Suppl):S47–54. 4. Dhaliwal C, Wright E, Graham C, et al. Wide-field digital retinal imaging versus binocular indirect ophthalmoscopy for retinopathy of prematurity screening: a two-observer prospective, randomised comparison. Br J Ophthalmol 2009;93:355–9. 5. Williams SL, Wang L, Kane SA, et al. Telemedical diagnosis of retinopathy of prematurity: accuracy of expert versus nonexpert graders. Br J Ophthalmol 2010;94:351–6. 6. Quinn GE, Ying G, Daniel E, et al. Validity of a telemedicine system for the evaluation of acute-phase retinopathy of prematurity. JAMA Ophthalmol 2014;132:1178–84. 7. Shah SP, Wu Z, Iverson S, Dai S. Specialist nurse screening for retinopathy of prematurity: a pilot study. Asia Pac J Ophthalmol (Phila) 2013;2:300–4. 8. Cabrera MT, Freedman SF, Kiely AE, et al. Combining ROPtool measurements of vascular tortuosity and width to quantify plus disease in retinopathy of prematurity. J AAPOS 2011;15:40–4. 9. Wittenberg LA, Jonsson NJ, Chan RV, Chiang MF. Computer-based image analysis for plus disease diagnosis in retinopathy of prematurity. J Pediatr Ophthalmol Strabismus 2012;49:11–9. quiz 0, 20. 10. Kiely AE, Wallace DK, Freedman SF, Zhao Z. Computerassisted measurement of retinal vascular width and tortuosity in retinopathy of prematurity. Arch Ophthalmol 2010;128: 847–52. 11. Wallace DK, Freedman SF, Zhao Z. Evolution of plus disease in retinopathy of prematurity: quantification by ROPtool. Trans Am Ophthalmol Soc 2009;107:47–52. 12. Wallace DK, Freedman SF, Zhao Z. A pilot study using ROPtool to measure retinal vascular dilation. Retina 2009;29:1182–7. 13. Johnston SC, Wallace DK, Freedman SF, et al. Tortuosity of arterioles and venules in quantifying plus disease. J AAPOS 2009;13:181–5. 14. Ahmad S, Wallace DK, Freedman SF, Zhao Z. Computerassisted assessment of plus disease in retinopathy of
5
Ophthalmology Volume -, Number -, Month 2015
15.
16. 17. 18.
19. 20.
prematurity using video indirect ophthalmoscopy images. Retina 2008;28:1458–62. Wallace DK. Computer-assisted quantification of vascular tortuosity in retinopathy of prematurity (an American Ophthalmological Society thesis). Trans Am Ophthalmol Soc 2007;105:594–615. Wallace DK, Freedman SF, Zhao Z, Jung SH. Accuracy of ROPtool vs individual examiners in assessing retinal vascular tortuosity. Arch Ophthalmol 2007;125:1523–30. Wallace DK, Zhao Z, Freedman SF. A pilot study using “ROPtool” to quantify plus disease in retinopathy of prematurity. J AAPOS 2007;11:381–7. Chiang MF, Gelman R, Williams SL, et al. Plus disease in retinopathy of prematurity: development of composite images by quantification of expert opinion. Invest Ophthalmol Vis Sci 2008;49:4064–70. Ophthalmic Mutual Insurance Company Retinopathy of Prematurity Task Force. Retinopathy of Prematurity: Creating a Safety Net. Ophthalmic Mutual Insurance Company 2013. Chiang MF, Starren J, Du YE, et al. Remote image based retinopathy of prematurity diagnosis: a receiver operating characteristic analysis of accuracy. Br J Ophthalmol 2006;90: 1292–6.
21. Ells AL, Holmes JM, Astle WF, et al. Telemedicine approach to screening for severe retinopathy of prematurity: a pilot study. Ophthalmology 2003;110:2113–7. 22. Fijalkowski N, Zheng LL, Henderson MT, et al. Stanford University Network for Diagnosis of Retinopathy of Prematurity (SUNDROP): five years of screening with telemedicine. Ophthalmic Surg Lasers Imaging Retina 2014;45:106–13. 23. Schwartz SD, Harrison SA, Ferrone PJ, Trese MT. Telemedical evaluation and management of retinopathy of prematurity using a fiberoptic digital fundus camera. Ophthalmology 2000;107:25–8. 24. Wu C, Petersen RA, VanderVeen DK. RetCam imaging for retinopathy of prematurity screening. J AAPOS 2006;10: 107–11. 25. Chiang MF, Keenan JD, Starren J, et al. Accuracy and reliability of remote retinopathy of prematurity diagnosis. Arch Ophthalmol 2006;124:322–7. 26. Early Treatment for Retinopathy of Prematurity Cooperative Group. Revised indications for the treatment of retinopathy of prematurity: results of the Early Treatment for Retinopathy of Prematurity Randomized Trial. Arch Ophthalmol 2003;121: 1684–94.
Footnotes and Financial Disclosures Originally received: July 2, 2015. Final revision: September 29, 2015. Accepted: September 30, 2015. Available online: ---.
K.A.D.: Equity owner e ROPtool D.K.W.: Equity owner e ROPtool Manuscript no. 2015-1127.
1
Department of Ophthalmology, William Beaumont Hospital, Royal Oak, Michigan.
2
Associated Retinal Consultants, Royal Oak, Michigan. Department of Ophthalmology and Visual Sciences, Kellogg Eye Center, University of Michigan, Ann Arbor, Michigan.
M.T.T.: Equity owner e ROPtool Author Contributions: Conception and design: Abbey, Besirli, Musch, Andrews, Capone, Drenser, Wallace, Ostmo, Chiang, Lee, Trese
3
Analysis and interpretation: Abbey, Besirli, Musch, Andrews, Capone, Drenser, Ostmo, Chiang, Lee, Trese
4
Data collection: Abbey, Besirli, Musch, Andrews, Capone, Drenser, Wallace, Ostmo, Chiang, Trese
Department of Epidemiology, University of Michigan, Ann Arbor, Michigan.
5
Obtained funding: none
6
Overall responsibility: Abbey, Besirli, Musch, Andrews, Capone, Drenser, Wallace, Ostmo, Chiang, Trese
Department of Ophthalmology, Duke University Eye Center, Durham, North Carolina. Department of Ophthalmology, Oregon Health and Science University, Portland, Oregon.
7
Department of Medical Informatics & Clinical Epidemiology, Oregon Health and Science University, Portland, Oregon. Financial Disclosure(s): The author(s) have made the following disclosure(s): A.C.: Equity owner e ROPtool, developed at the University of North Carolina, Chapel Hill, and Duke University, and licensed by FocusROP.
6
Abbreviations and Acronyms: ROC ¼ receiver operating characteristic; ROP ¼ retinopathy of prematurity. Correspondence: Michael T. Trese, MD, Associated Retinal Consultants, 3535 West 13 Mile Road, Suite 344, Royal Oak, MI 48073. E-mail: [email protected].