- Email: [email protected]
Positive predictive value and screening performance of GoCheck Kids in a primary care university clinic
Journal Pre-proof Positive predictive value and screening performance of GoCheck Kids in a primary care university clinic Megan X. Law, MD, Mariana Flores Pimentel, MD, Catherine E. Oldenburg, ScD, MPH, Alejandra G. de Alba Campomanes, MD, MPH PII:
S1091-8531(20)30010-0
DOI:
https://doi.org/10.1016/j.jaapos.2019.11.006
Reference:
YMPA 3126
To appear in:
Journal of AAPOS
Received Date: 31 July 2019 Revised Date:
13 November 2019
Accepted Date: 17 November 2019
Please cite this article as: Law MX, Pimentel MF, Oldenburg CE, de Alba Campomanes AG, Positive predictive value and screening performance of GoCheck Kids in a primary care university clinic, Journal of AAPOS (2020), doi: https://doi.org/10.1016/j.jaapos.2019.11.006. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. Copyright © 2020, American Association for Pediatric Ophthalmology and Strabismus. Published by Elsevier Inc. All rights reserved.
Positive predictive value and screening performance of GoCheck Kids in a primary care university clinic Megan X. Law, MD, Mariana Flores Pimentel, MD, Catherine E. Oldenburg, ScD, MPH, and Alejandra G. de Alba Campomanes, MD, MPH Author affiliations: University of California, San Francisco. San Francisco, California Presented as a poster at the 45th Annual Meeting of the American Association for Pediatric Ophthalmology and Strabismus, San Diego, CA, March 27-31, 2019. Submitted July 21, 2019. Revision accepted November 17, 2019. Correspondence: Alejandra G. de Alba Campomanes, MD, MPH, 10 Koret Way, San Francisco, CA 94143-0730 (email: [email protected]). Word count: 2,491 Abstract only: 247
Abstract Purpose To determine the positive predictive value (PPV) of GoCheck Kids, a smartphone-based photoscreener, to detect refractive amblyopia risk factors (ARFs) in children 3-48 months of age. Methods The medical records of all children ≤48 months of age who failed GoCheck Kids photoscreening at a University of California, San Francisco, pediatric clinic between February 2017 and August 2018 and subsequently examined at the pediatric ophthalmology clinic were reviewed retrospectively. The PPV of GoCheck Kids was determined, where a true positive represents an abnormal cycloplegic refractive error according to the 2013 American Association of Pediatric Ophthalmology and Strabismus Vision Screening Committee criteria. For patients ≤12 months of age, refractive error thresholds were based on the 2017 American Academy of Ophthalmology Preferred Practice Patterns Pediatric Eye Evaluation guidelines. Results A total of 2,963 children were screened with GoCheck Kids. Of these, 172 (5.8%) failed the screening, of whom 115 (67%) were evaluated in the pediatric ophthalmology clinic. The mean age was 24.9 ± 11.1months (range, 3-48). Fifty-seven patients met ARF criteria yielding a PPV of 50% (95% CI, 41%-60%). The PPV was higher in patients of Latino/hispanic ethnicity (75%; 95% CI, 57%-100%; P < 0.01) and changed significantly with increasing age (P = 0.03). Patients who were screened between age 3-12 months had the lowest PPV at 26% (95% CI, 14%-47%). Conclusions Modifying refractive error thresholds based on patient age and prevalence of ARFs in a population may improve the PPV of GoCheck Kids in a community-based screening program.
Amblyopia, a leading cause of childhood vision impairment in the United States, is most effectively managed when detected early.1 Instrument-based photoscreening, first introduced in the mid-1990s, has been endorsed by the American Academy of Pediatrics (AAP) and the American Association of Pediatric Ophthalmology and Strabismus (AAPOS) to screen for amblyopia risk factors (ARFs) in children as young as 12 months of age.2,3 GoCheck Kids (GoCheck Kids, Gobiquity Mobile Health, Scottsdale, AZ), a smartphone-based photoscreening app, has been shown to detect refractive ARFs, yet its performance in young children has not been thoroughly evaluated. The purpose of this study was to determine the positive predictive value (PPV) of GoCheck Kids to detect refractive ARFs in children 3-48 months of age. Subjects and Methods The retrospective study was approved by the University of California, San Francisco (UCSF) Institutional Review Board and conformed to the requirements of the US Health Insurance Portability and Accountability Act of 1996. In 2017, GoCheck Kids was adopted as the primary vision screening method at UCSF for children age ≤48 months of age. The medical records of all screened children in this age group who presented to the UCSF pediatric ophthalmology clinic after an abnormal GoCheck Kids photoscreening result between February 2017 and August 2018 were reviewed. Photoscreening was conducted at a UCSF pediatric or family medicine clinic during a routine well-child check, using the GoCheck Kids app on a Nokia Lumia 1020 smartphone (Nokia Corporation, Espoo, Finland). All screeners (medical assistants and nurses) had successfully completed a training tutorial on the GoCheck Kids app. GoCheck Kids enables users to capture a flash image in portrait orientation with cues to adjust room lighting and proper distance from the subject. The app emits animal noises to attract the subject and facilitate proper fixation. Additional
photographs were taken if the initial image was not gradable. A noncycloplegic estimation of myopia or hyperopia along the meridian of flash eccentricity was determined based on the appearance of a light crescent relative to the pupillary reflex. Abnormal results (“risk factor identified” either by an automated interpretation or manual interpretation by a GoCheck Kids specialist) prompted a referral. The GoCheck Kids refractive criteria to trigger “risk factor identified” were >1.24 D of hyperopia, >0.70 D of anisometropia, or >2.0 D of myopia, irrespective of the subject’s age. The criteria were preset, and could not be modified by the tester. Patients who followed up at the ophthalmology clinic received a comprehensive examination by a pediatric ophthalmologist or optometrist, including the following tests and measurements: visual acuity, intraocular pressure, stereopsis, ocular alignment, and ocular motility. Anterior segment and dilated fundoscopic examination were also performed. Cycloplegic retinoscopy was performed in all children after instillation of 2 drops of cyclopentolate 1% and tropicamide 1% 5 minutes apart. Portable autorefraction (Retinomax K+3 autorefractometer, Luneau, Luneau SAS 1, Prunay le Gillon, France) was performed on all children able to cooperate with the test. Preexisting patients of the pediatric ophthalmology clinic with ongoing eye care were excluded from the study. To determine the PPV of the GoCheck Kids app, a “true positive” was defined as an abnormal cycloplegic refractive error based on the 2013 AAPOS Vision Screening Committee guidelines to detect ARFs (Table 1). For patients age ≤12 months of age, the 2017 American Academy of Ophthalmology (AAO) Preferred Practice Patterns Pediatric Eye Evaluation guidelines were used to define a “true positive” result,4 because the 2013 criteria do not include this age group. Subanalysis was also performed using AAPOS guidelines for 12- to 30-montholds. PPVs were calculated overall and by subgroup as the proportion of subjects who failed
GoCheck Kids screening, with 95% confidence intervals calculated using an intercept-only binomial generalized estimating equation to account for clustering (2 eyes screened within 1 patient). The PPV was first calculated overall and then by subgroup, including age (in categories of 3-12 months, 13-30 months, and 31-48 months to match AAPOS screening categories), child’s sex, child’s race/ethnicity (white, African American, Asian, and/or Hispanic), and by specific risk factor (myopia, hyperopia, or anisometropia). To assess whether the PPV varied across subgroups, we built binomial generalized estimating equations with the subgroup variable predicting disease positivity. Differences between children included in the study and those lost to follow-up were assessed using a t test for continuous variables and Fisher exact test for categorical variables. All analyses were conducted in R version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria). Results A total of 2,963 children were screened with GoCheck Kids during the study period (Figure 1). Of the total, 172 children (5.8%) had an abnormal photoscreening result and were referred, of whom 115 (67%) returned for a comprehensive eye examination. The mean age, with standard deviation was 24.9 ± 11.1 months (range, 3-48): 31 patients were 3-12 months of age, 42 patients were 13-30 months of age, and 42 patients were 31-48 months of age. The main reasons for referral were hyperopia and anisometropia (54%), followed by hyperopia only (18%), myopia and anisometropia (11%), anisometropia only (9%), and myopia only (8%). The demographic characteristics and screening differences of the 115 children in our study and the 57 children lost to follow-up are presented in Table 2. There were no significant differences between groups except that those lost to follow-up were more likely to fail GoCheck Kids screening from
hyperopia (86% vs 71%; P = 0.05). A total of 57 patients met criteria for ARFs, yielding a PPV of 50% (95% CI, 41%-60%). See Table 3. The PPV changed significantly with increasing age (P = 0.03). The PPV was lowest for patients 3-12 months of age at 26% (95% CI, 14%-47%). PPV in this age group has not been investigated in previous reports. Furthermore, we found that the PPV was higher (75%) in patients of Latino/Hispanic ethnicity (95% CI, 57%-100%; P < 0.01). In these patients, 10 of 12 “true positives” (83%) were due to astigmatic refractive errors. The PPV varied little by the type of risk factor identified by GoCheck Kids. However, for anisometropia, the PPV was very low (9%; P = 0.06) for predicting true anisometropia. The PPV for myopia as the risk factor identified by GoCheck Kids varied according to age, with a statistically significant lower PPV in patients ≤12 months of age (P = 0.01). In our cohort, 16 children (28%) met criteria for astigmatism: 9 were referred for hyperopia and anisometropia, 3 for myopia and anisometropia, 2 for hyperopia only, and 1 each for myopia only and anisometropia only. Of the 115 patients who failed GoCheck Kids screening and were evaluated in clinic, 58 were false positives. The mean age of these patients was 23 months. All 58 patients (100%) had subthreshold hyperopia. Thirty-six patients (62%) had subthreshold astigmatism: 29 had withthe-rule (90° ± 15°) astigmatism, 6 had against-the-rule astigmatism (180°˚ ± 15°), and 1 had oblique astigmatism. Of the 115 subjects, 22 (19%) had more than a single photoscreening attempt: 19 (86%) had 2 images taken, and 3 (14%) had 3 images taken. Twelve patients (55%) had a risk factor identified on their additional photographs, 7 (32%) had a not-gradable photograph, and 3 (14%) had no risk factor identified on their subsequent photographs.
Discussion Automated photoscreeners are an important tool for identifying ARFs in young children. The PPV of GoCheck Kids to detect ARFs in our cohort was 50%, which is lower than the PPV calculated in previous GoCheck Kids validation studies and the PPV reported for other photoscreeners: Plusoptix (51%-97.9%), Spot (67%-87%), and iScreen (87%-94%).5-9 Arnold and colleagues10 reported that in 217 subjects 1-6 years of age examined after an abnormal result from the GoCheck Kids app (on the Nokia 1020 smartphone), the PPV was 68% using 2013 AAPOS guidelines. The GoCheck Kids referral thresholds were >2.0 D for hyperopia, >3.0 D for myopia, and >1.5 D for anisometropia, which were higher than the thresholds in our study. In that study, none of the children were younger than 12 months of age, and a significant proportion of the children (25%) were 4 years or older. We found that PPV varies by age, which may explain the difference in PPV between studies. A similar study by Arnold and colleagues11 in 2018 on the efficacy of GoCheck Kids to detect ARFs in 287 patients at 4 private pediatric ophthalmology clinics revealed a PPV of 69% for automated grading and 76% for manual grading by GoCheck Kids, according to 2013 AAPOS criteria. It is unclear what the GoCheck Kids referral thresholds were for this study. Children were stratified as either 1-3 years or 4-6 years, and PPV was reported on all patients combined. Furthermore, GoCheck Kids screening was performed using an iPhone 7 rather than a Nokia 1020 smartphone, as in our study. In 2018 Peterseim and colleagues12 published their validation study of GoCheck Kids (using the Nokia 1020 smartphone), in which 206 patients were evaluated at a single pediatric ophthalmology practice. The PPV was 57% (95% CI, 50%-63%), with the inclusion of 21 patients 6-12 months of age. As in our study, they found increased PPV when older children
were screened; additionally, a similar proportion of their patients (17%) required more than a single screening attempt.12 Our study shows that PPV varies significantly according to patient age, with the youngest cohort having the lowest PPV (26%). The PPV in this age group was unchanged when we applied the AAPOS 2013 criteria for age 13-20 months. Compared to the 2017 AAO Preferred Practice Patterns ARF targets for hyperopia, myopia, and anisometropia in children <12 months of age (>6 D, >5 D, and >2.5 D, resp.), GoCheck Kids referral criteria for hyperopia, myopia, and anisometropia (>1.24 D, >2 D, and >0.70 D, resp.) are substantially lower. The discrepancy in threshold levels likely explains the low probability of GoCheck Kids to detect ARFs in the youngest age group. Hispanic ethnicity was also associated with a higher PPV (75%; 95% CI, 57%-100%), with 83% meeting ARF criteria for astigmatism on cycloplegic refraction. In this group, 9 of 10 subjects had with-the-rule astigmatism, and 1 patient had against-the-rule astigmatism. Of note, prevalence (pretest probability) of astigmatism in the Hispanic population is high. According to the Multi-Ethnic Pediatric Eye Disease Study, the prevalence of >1.5 D of astigmatism in Hispanic children is 40% in 6-11 months, 16% in 12-23 months, 15% in 24-35 months, and 13% in 36-47 months (Table 4).13-15 The “false positive” patients tended to be young (mean age, 23 months), with subthreshold with-the-rule astigmatism and mild hyperopia. Our results suggest that GoCheck Kids referral thresholds may need to be refined by patient age, the pretest probability based on the prevalence of ARFs in a population, and the single-axis flash method, which accounts for both sphere and with-the-rule astigmatism within the same threshold value. In our cohort, 22 subjects required multiple photographs. Various factors could influence the testability of a photoscreener, including appropriate patient fixation, proper lighting
conditions, and user familiarity with the testing device. Additional studies could elucidate whether there is a learning curve for personnel performing GoCheck Kids screening that may independently affect the frequency of repeat scans. The mean age of patients with multiple photographs was 20 months (range, 4-38). Seventeen subjects with multiple scans (77%) failed the initial GoCheck Kids screening but “passed” subsequent evaluation in clinic. These findings suggest that there may be an association between taking multiple scans and having a false positive GoCheck Kids screen. Moreover, it is possible that young children are less likely than older ones to be engaged by the smartphone’s fixation target, reducing the photoscreener’s effectiveness. Our study has several limitations, including its retrospective nature and limited follow-up of patients who failed the GoCheck Kids screening. Of the referred patients, 33% did not receive a comprehensive eye examination; thus, it is unclear whether they had a true ARF. However, a comparison between our patients and those lost to follow-up show no statistically significant baseline differences, including age and ethnicity, that would indicate a selection bias. The only statistically significant difference was the rate of hyperopia detection, but we found that this type of ARF detected by GoCheck Kids did not seem to significantly affect the PPV to detect any type of ARF and is unlikely to bias our estimation. However, other unmeasured characteristics could have introduced bias in our estimates, and we cannot rule out informative censoring. It is possible that patients lost to follow-up are more likely from lower socioeconomic class, which has been associated with an increased incidence of refractive error and amblyopia.16 Another limitation is that the pediatric ophthalmologists and optometrists in our study were not masked to the results of the GoCheck Kids screen. Lastly, because we only evaluated patients with a result of “risk factor identified” on GoCheck Kids screening, patients with not-gradable scans were not
referred. Patients unable to complete screening tests are known to be at higher risk for significant refractive errors and amblyopia17; therefore, our PPV may be an underestimation of the true PPV in our population. This is the first study to report the PPV of GoCheck Kids screening in the primary care setting with a large cohort of young patients age 12 months or younger. Although the AAP and AAPOS recommend initial screening between 1 and 3 years of age, it is important to recognize that earlier screening with GoCheck Kids, while feasible, may have limited utility to detect ARFs. As new vision screening technologies penetrate the market, it is important to evaluate their performance independently in clinical practice.
References 1.
Kirk VG, Clausen NM, Armitage RW, Arnold RW. Preverbal photoscreening for amblyogenic risk factors and outcomes in amblyopia treatment: Early objective screening and visual acuities. Arch Ophthalmol 2008;126:489-92.
2.
Donahue SP, Arthur B, Neely DE, Arnold RW, Silbert D, Ruben JB; POS Vision Screening Committee. Guidelines for automated preschool vision screening: a 10-year, evidence-based update. J AAPOS 2013;17:4-8.
3.
Donahue S, Nixon C; Section of Ophthalmology, American Academy of Pediatrics; Committee on Practice and Ambulatory Medicine, American Academy of Pediatrics; American Academy of Ophthalmology; American Association for Pediatric Ophthalmology and Strabismus; American Association of Certified Orthoptists. Visual system assessment in infants, children, and young adults by pediatricians. Pediatrics 2016;137:28-30.
4.
American Academy of Ophthalmology Preferred Practice Patterns. Pediatric Eye Evaluation.2017. Available online: https://www.aaojournal.org/article/S01616420(17)32958-5/pdf. Accessed March 2, 2019.
5.
Arnold RW, Armitage MD. Performance of Four new photoscreeners on pediatric patients with high risk amblyopia. J Pediatr Ophthalmol Strabismus 2014;51:46-52.
6.
Lowry EA, Wang W, Nyong’o O. Objective vision screening in 3-year-old children at a multispecialty practice. J AAPOS 2015;19:16-20.
7.
Matta NS, Arnold RW, Singman EL, Silbert DI. Comparison between the plusoptiX and MTI photoscreeners. Arch Ophthalmol 2009;127:1591-5.
8.
Peterseim MMW, Papa CE, Wilson ME, et al. The effectiveness of the Spot Vision
Screener in detecting amblyopia risk factors. J AAPOS 2014;18:539-42. 9.
Kennedy RA, Thomas DE. Evaluation of the iScreen digital screening system for amblyogenic factors. Can J Ophthalmol 2000;35:258-62.
10.
Arnold RW, Arnold AW, Hunt-Smith TT, Grendahl RL, Winkle RK. The positive predictive value of smartphone photoscreening in pediatric practices. J Pediatr Ophthalmol Strabismus 2018;55:393-6.
11.
Arnold RW, O’Neil JW, Cooper KL, Silbert DI, Donahue SP. Evaluation of a smartphone photoscreening app to detect refractive amblyopia risk factors in children aged 1-6 years. Clin Ophthalmol 2018;12:1533-7.
12.
Peterseim MMW, Rhodes RS, Patel RN, et al. Effectiveness of the Go Check Kids vision screener in detecting amblyopia risk factors. Am J Ophthalmol 2018;187:87-91.
13.
Wen G, Tarczy-Hornoch K, McKean-Cowdin, et al; Multi-Ethnic Pediatric Eye Disease Study Group. Prevalence of myopia, hyperopia, and astigmatism in non-Hispanic White and Asian children: Multi-Ethnic Pediatric Eye Disease Study. Ophthalmology 2013;120:2109-16.
14.
Fozailoff AF, Tarczy-Hornoch K, Cotter S, et al; Writing Committee for the MEPEDS Study Group. Prevalence of astigmatism in 6- to 72-month-old African American and Hispanic children: The Multi-Ethnic Pediatric Eye Disease Study. Ophthalmology 2011;118:284-93.
15.
Multi-Ethnic Pediatric Eye Disease Study Group. Prevalence of myopia and hyperopia in 6-to 72-month-old African American and Hispanic children: the Multi-Ethnic Pediatric Eye Disease Study. Ophthalmology 2010;117:140-47.e3.
16.
Williams C, Northstone K, Howard M, Harvey I, Harrad RA, Sparrow JM. Prevalence
and risk factors for common vision problems in children: Data from the ALSPAC study. Br J Ophthalmol 2008;92:959-64. 17.
Maguire MG; The Vision in Preschoolers Study Group. Children unable to perform screening tests in vision in preschoolers study: Proportion with Ocular conditions and impact on measures of test accuracy. Invest Ophthalmol Vis Sci 2007;48:83-7.
Legends FIG 1. GoCheck Kids screening and follow-up.
Table 1. Amblyopia risk factor targets (cycloplegic refraction) from the 2013 American Association for Pediatric Ophthalmology and Strabismus Vision Screening Committee guidelines and the 2017 American Academy of Ophthalmology (AAO) Preferred Practice Patterns—Guidelines for Refractive 2,4 Correction in Infants and Young Children Age, months a 3-12 13-30 31-48 a
Astigmatism >3.0 D >2.0 D >2.0 D
Hyperopia >6.0 D >4.5 D >4.0 D
Anisometropia >2.5 D >2.5 D >2.0 D
Myopia > 5.0 D > 3.5 D > 3.0 D
Refractive targets in this age group are from the 2017 AAO Preferred Practice Patterns; they are not listed in the 2013 AAPOS guidelines.
Table 2. Demographic characteristics of patients who failed GoCheck Kids screening Study parameter Follow-up No. 115 Age, months, mean ± SD (range) 24.9 ± 11.1 ( 3-27) Female sex, % 40.0 Race and ethnicity, % White 39.1 Asian 28.7 African American 11.3 Hispanic 13.9 Type of refractive error by GoCheck Kids, % Myopia 19.1 Hyperopia 71.3 Anisometropia 73.9 SD, standard deviation.
Lost to follow-up P value 57 27.0 ± 10.7 (12-43) 0.23 49.1 0.33 33.3 15.8 19.3 15.8
0.57 0.10 0.23 0.92
14.0 86.0 73.7
0.54 0.05 1.00
Table 3. Positive predictive value of GoCheck Kids by age, race/ethnicity, and risk factor identified Study parameter N Age group a All 115 3-12 months 31 13-30 months 42 31-48 months 42 Race/Ethnicity White 45 Asian 33 African 13 American b Latino/a 16
Hyperopia Myopia Anisometropia
82 22 85
PPV
95% CI
0.50 0.26 0.60 0.57
0.41, 0.14, 0.46, 0.43,
0.44 0.42 0.62
0.32, 0.62 0.28, 0.64 0.38, 0.98
0.75 PPV for same ARF 0.36 0.36 0.09
0.60 0.47 0.77 0.75
0.57, 1.00 PPV for any ARF 0.45 c 0.55 0.45
ARF, amblyopia risk factor; PPV, positive predictive value. a
P value for age, 0.03. P value for ethnicity (Latino/a vs. not): 0.006. c P value for the interaction between myopia and age: 0.01. b
Table 4. Reported prevalence of hyperopia, myopia and astigmatism by race and ethnicity from the Multi-Ethnic Pediatric Eye Disease 13-15 Study
White
Asian
African American
Hispanic
Age, months 6-11 12-23 24-35 36-47 6-11 12-23 24-35 36-47 6-11 12-23 24-35 36-47 6-11 12-23 24-35 36-47
Hyperopia >2 D, % 30 24 20 27 17 10 10 16 18 20 18 20 35 22 24 29
Myopia >2 D, % 0 0 0.4 0.3 0 0.8 0.7 0 2 2 1 2 0.7 1.7 1.8 0.8
Astigmatism >1.5 D, % 18 6 4 5 13 7 9 5 25 13 13 6 40 16 15 13
S1091-8531(20)30010-0
DOI:
https://doi.org/10.1016/j.jaapos.2019.11.006
Reference:
YMPA 3126
To appear in:
Journal of AAPOS
Received Date: 31 July 2019 Revised Date:
13 November 2019
Accepted Date: 17 November 2019
Please cite this article as: Law MX, Pimentel MF, Oldenburg CE, de Alba Campomanes AG, Positive predictive value and screening performance of GoCheck Kids in a primary care university clinic, Journal of AAPOS (2020), doi: https://doi.org/10.1016/j.jaapos.2019.11.006. This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. Copyright © 2020, American Association for Pediatric Ophthalmology and Strabismus. Published by Elsevier Inc. All rights reserved.
Positive predictive value and screening performance of GoCheck Kids in a primary care university clinic Megan X. Law, MD, Mariana Flores Pimentel, MD, Catherine E. Oldenburg, ScD, MPH, and Alejandra G. de Alba Campomanes, MD, MPH Author affiliations: University of California, San Francisco. San Francisco, California Presented as a poster at the 45th Annual Meeting of the American Association for Pediatric Ophthalmology and Strabismus, San Diego, CA, March 27-31, 2019. Submitted July 21, 2019. Revision accepted November 17, 2019. Correspondence: Alejandra G. de Alba Campomanes, MD, MPH, 10 Koret Way, San Francisco, CA 94143-0730 (email: [email protected]). Word count: 2,491 Abstract only: 247
Abstract Purpose To determine the positive predictive value (PPV) of GoCheck Kids, a smartphone-based photoscreener, to detect refractive amblyopia risk factors (ARFs) in children 3-48 months of age. Methods The medical records of all children ≤48 months of age who failed GoCheck Kids photoscreening at a University of California, San Francisco, pediatric clinic between February 2017 and August 2018 and subsequently examined at the pediatric ophthalmology clinic were reviewed retrospectively. The PPV of GoCheck Kids was determined, where a true positive represents an abnormal cycloplegic refractive error according to the 2013 American Association of Pediatric Ophthalmology and Strabismus Vision Screening Committee criteria. For patients ≤12 months of age, refractive error thresholds were based on the 2017 American Academy of Ophthalmology Preferred Practice Patterns Pediatric Eye Evaluation guidelines. Results A total of 2,963 children were screened with GoCheck Kids. Of these, 172 (5.8%) failed the screening, of whom 115 (67%) were evaluated in the pediatric ophthalmology clinic. The mean age was 24.9 ± 11.1months (range, 3-48). Fifty-seven patients met ARF criteria yielding a PPV of 50% (95% CI, 41%-60%). The PPV was higher in patients of Latino/hispanic ethnicity (75%; 95% CI, 57%-100%; P < 0.01) and changed significantly with increasing age (P = 0.03). Patients who were screened between age 3-12 months had the lowest PPV at 26% (95% CI, 14%-47%). Conclusions Modifying refractive error thresholds based on patient age and prevalence of ARFs in a population may improve the PPV of GoCheck Kids in a community-based screening program.
Amblyopia, a leading cause of childhood vision impairment in the United States, is most effectively managed when detected early.1 Instrument-based photoscreening, first introduced in the mid-1990s, has been endorsed by the American Academy of Pediatrics (AAP) and the American Association of Pediatric Ophthalmology and Strabismus (AAPOS) to screen for amblyopia risk factors (ARFs) in children as young as 12 months of age.2,3 GoCheck Kids (GoCheck Kids, Gobiquity Mobile Health, Scottsdale, AZ), a smartphone-based photoscreening app, has been shown to detect refractive ARFs, yet its performance in young children has not been thoroughly evaluated. The purpose of this study was to determine the positive predictive value (PPV) of GoCheck Kids to detect refractive ARFs in children 3-48 months of age. Subjects and Methods The retrospective study was approved by the University of California, San Francisco (UCSF) Institutional Review Board and conformed to the requirements of the US Health Insurance Portability and Accountability Act of 1996. In 2017, GoCheck Kids was adopted as the primary vision screening method at UCSF for children age ≤48 months of age. The medical records of all screened children in this age group who presented to the UCSF pediatric ophthalmology clinic after an abnormal GoCheck Kids photoscreening result between February 2017 and August 2018 were reviewed. Photoscreening was conducted at a UCSF pediatric or family medicine clinic during a routine well-child check, using the GoCheck Kids app on a Nokia Lumia 1020 smartphone (Nokia Corporation, Espoo, Finland). All screeners (medical assistants and nurses) had successfully completed a training tutorial on the GoCheck Kids app. GoCheck Kids enables users to capture a flash image in portrait orientation with cues to adjust room lighting and proper distance from the subject. The app emits animal noises to attract the subject and facilitate proper fixation. Additional
photographs were taken if the initial image was not gradable. A noncycloplegic estimation of myopia or hyperopia along the meridian of flash eccentricity was determined based on the appearance of a light crescent relative to the pupillary reflex. Abnormal results (“risk factor identified” either by an automated interpretation or manual interpretation by a GoCheck Kids specialist) prompted a referral. The GoCheck Kids refractive criteria to trigger “risk factor identified” were >1.24 D of hyperopia, >0.70 D of anisometropia, or >2.0 D of myopia, irrespective of the subject’s age. The criteria were preset, and could not be modified by the tester. Patients who followed up at the ophthalmology clinic received a comprehensive examination by a pediatric ophthalmologist or optometrist, including the following tests and measurements: visual acuity, intraocular pressure, stereopsis, ocular alignment, and ocular motility. Anterior segment and dilated fundoscopic examination were also performed. Cycloplegic retinoscopy was performed in all children after instillation of 2 drops of cyclopentolate 1% and tropicamide 1% 5 minutes apart. Portable autorefraction (Retinomax K+3 autorefractometer, Luneau, Luneau SAS 1, Prunay le Gillon, France) was performed on all children able to cooperate with the test. Preexisting patients of the pediatric ophthalmology clinic with ongoing eye care were excluded from the study. To determine the PPV of the GoCheck Kids app, a “true positive” was defined as an abnormal cycloplegic refractive error based on the 2013 AAPOS Vision Screening Committee guidelines to detect ARFs (Table 1). For patients age ≤12 months of age, the 2017 American Academy of Ophthalmology (AAO) Preferred Practice Patterns Pediatric Eye Evaluation guidelines were used to define a “true positive” result,4 because the 2013 criteria do not include this age group. Subanalysis was also performed using AAPOS guidelines for 12- to 30-montholds. PPVs were calculated overall and by subgroup as the proportion of subjects who failed
GoCheck Kids screening, with 95% confidence intervals calculated using an intercept-only binomial generalized estimating equation to account for clustering (2 eyes screened within 1 patient). The PPV was first calculated overall and then by subgroup, including age (in categories of 3-12 months, 13-30 months, and 31-48 months to match AAPOS screening categories), child’s sex, child’s race/ethnicity (white, African American, Asian, and/or Hispanic), and by specific risk factor (myopia, hyperopia, or anisometropia). To assess whether the PPV varied across subgroups, we built binomial generalized estimating equations with the subgroup variable predicting disease positivity. Differences between children included in the study and those lost to follow-up were assessed using a t test for continuous variables and Fisher exact test for categorical variables. All analyses were conducted in R version 3.5.1 (R Foundation for Statistical Computing, Vienna, Austria). Results A total of 2,963 children were screened with GoCheck Kids during the study period (Figure 1). Of the total, 172 children (5.8%) had an abnormal photoscreening result and were referred, of whom 115 (67%) returned for a comprehensive eye examination. The mean age, with standard deviation was 24.9 ± 11.1 months (range, 3-48): 31 patients were 3-12 months of age, 42 patients were 13-30 months of age, and 42 patients were 31-48 months of age. The main reasons for referral were hyperopia and anisometropia (54%), followed by hyperopia only (18%), myopia and anisometropia (11%), anisometropia only (9%), and myopia only (8%). The demographic characteristics and screening differences of the 115 children in our study and the 57 children lost to follow-up are presented in Table 2. There were no significant differences between groups except that those lost to follow-up were more likely to fail GoCheck Kids screening from
hyperopia (86% vs 71%; P = 0.05). A total of 57 patients met criteria for ARFs, yielding a PPV of 50% (95% CI, 41%-60%). See Table 3. The PPV changed significantly with increasing age (P = 0.03). The PPV was lowest for patients 3-12 months of age at 26% (95% CI, 14%-47%). PPV in this age group has not been investigated in previous reports. Furthermore, we found that the PPV was higher (75%) in patients of Latino/Hispanic ethnicity (95% CI, 57%-100%; P < 0.01). In these patients, 10 of 12 “true positives” (83%) were due to astigmatic refractive errors. The PPV varied little by the type of risk factor identified by GoCheck Kids. However, for anisometropia, the PPV was very low (9%; P = 0.06) for predicting true anisometropia. The PPV for myopia as the risk factor identified by GoCheck Kids varied according to age, with a statistically significant lower PPV in patients ≤12 months of age (P = 0.01). In our cohort, 16 children (28%) met criteria for astigmatism: 9 were referred for hyperopia and anisometropia, 3 for myopia and anisometropia, 2 for hyperopia only, and 1 each for myopia only and anisometropia only. Of the 115 patients who failed GoCheck Kids screening and were evaluated in clinic, 58 were false positives. The mean age of these patients was 23 months. All 58 patients (100%) had subthreshold hyperopia. Thirty-six patients (62%) had subthreshold astigmatism: 29 had withthe-rule (90° ± 15°) astigmatism, 6 had against-the-rule astigmatism (180°˚ ± 15°), and 1 had oblique astigmatism. Of the 115 subjects, 22 (19%) had more than a single photoscreening attempt: 19 (86%) had 2 images taken, and 3 (14%) had 3 images taken. Twelve patients (55%) had a risk factor identified on their additional photographs, 7 (32%) had a not-gradable photograph, and 3 (14%) had no risk factor identified on their subsequent photographs.
Discussion Automated photoscreeners are an important tool for identifying ARFs in young children. The PPV of GoCheck Kids to detect ARFs in our cohort was 50%, which is lower than the PPV calculated in previous GoCheck Kids validation studies and the PPV reported for other photoscreeners: Plusoptix (51%-97.9%), Spot (67%-87%), and iScreen (87%-94%).5-9 Arnold and colleagues10 reported that in 217 subjects 1-6 years of age examined after an abnormal result from the GoCheck Kids app (on the Nokia 1020 smartphone), the PPV was 68% using 2013 AAPOS guidelines. The GoCheck Kids referral thresholds were >2.0 D for hyperopia, >3.0 D for myopia, and >1.5 D for anisometropia, which were higher than the thresholds in our study. In that study, none of the children were younger than 12 months of age, and a significant proportion of the children (25%) were 4 years or older. We found that PPV varies by age, which may explain the difference in PPV between studies. A similar study by Arnold and colleagues11 in 2018 on the efficacy of GoCheck Kids to detect ARFs in 287 patients at 4 private pediatric ophthalmology clinics revealed a PPV of 69% for automated grading and 76% for manual grading by GoCheck Kids, according to 2013 AAPOS criteria. It is unclear what the GoCheck Kids referral thresholds were for this study. Children were stratified as either 1-3 years or 4-6 years, and PPV was reported on all patients combined. Furthermore, GoCheck Kids screening was performed using an iPhone 7 rather than a Nokia 1020 smartphone, as in our study. In 2018 Peterseim and colleagues12 published their validation study of GoCheck Kids (using the Nokia 1020 smartphone), in which 206 patients were evaluated at a single pediatric ophthalmology practice. The PPV was 57% (95% CI, 50%-63%), with the inclusion of 21 patients 6-12 months of age. As in our study, they found increased PPV when older children
were screened; additionally, a similar proportion of their patients (17%) required more than a single screening attempt.12 Our study shows that PPV varies significantly according to patient age, with the youngest cohort having the lowest PPV (26%). The PPV in this age group was unchanged when we applied the AAPOS 2013 criteria for age 13-20 months. Compared to the 2017 AAO Preferred Practice Patterns ARF targets for hyperopia, myopia, and anisometropia in children <12 months of age (>6 D, >5 D, and >2.5 D, resp.), GoCheck Kids referral criteria for hyperopia, myopia, and anisometropia (>1.24 D, >2 D, and >0.70 D, resp.) are substantially lower. The discrepancy in threshold levels likely explains the low probability of GoCheck Kids to detect ARFs in the youngest age group. Hispanic ethnicity was also associated with a higher PPV (75%; 95% CI, 57%-100%), with 83% meeting ARF criteria for astigmatism on cycloplegic refraction. In this group, 9 of 10 subjects had with-the-rule astigmatism, and 1 patient had against-the-rule astigmatism. Of note, prevalence (pretest probability) of astigmatism in the Hispanic population is high. According to the Multi-Ethnic Pediatric Eye Disease Study, the prevalence of >1.5 D of astigmatism in Hispanic children is 40% in 6-11 months, 16% in 12-23 months, 15% in 24-35 months, and 13% in 36-47 months (Table 4).13-15 The “false positive” patients tended to be young (mean age, 23 months), with subthreshold with-the-rule astigmatism and mild hyperopia. Our results suggest that GoCheck Kids referral thresholds may need to be refined by patient age, the pretest probability based on the prevalence of ARFs in a population, and the single-axis flash method, which accounts for both sphere and with-the-rule astigmatism within the same threshold value. In our cohort, 22 subjects required multiple photographs. Various factors could influence the testability of a photoscreener, including appropriate patient fixation, proper lighting
conditions, and user familiarity with the testing device. Additional studies could elucidate whether there is a learning curve for personnel performing GoCheck Kids screening that may independently affect the frequency of repeat scans. The mean age of patients with multiple photographs was 20 months (range, 4-38). Seventeen subjects with multiple scans (77%) failed the initial GoCheck Kids screening but “passed” subsequent evaluation in clinic. These findings suggest that there may be an association between taking multiple scans and having a false positive GoCheck Kids screen. Moreover, it is possible that young children are less likely than older ones to be engaged by the smartphone’s fixation target, reducing the photoscreener’s effectiveness. Our study has several limitations, including its retrospective nature and limited follow-up of patients who failed the GoCheck Kids screening. Of the referred patients, 33% did not receive a comprehensive eye examination; thus, it is unclear whether they had a true ARF. However, a comparison between our patients and those lost to follow-up show no statistically significant baseline differences, including age and ethnicity, that would indicate a selection bias. The only statistically significant difference was the rate of hyperopia detection, but we found that this type of ARF detected by GoCheck Kids did not seem to significantly affect the PPV to detect any type of ARF and is unlikely to bias our estimation. However, other unmeasured characteristics could have introduced bias in our estimates, and we cannot rule out informative censoring. It is possible that patients lost to follow-up are more likely from lower socioeconomic class, which has been associated with an increased incidence of refractive error and amblyopia.16 Another limitation is that the pediatric ophthalmologists and optometrists in our study were not masked to the results of the GoCheck Kids screen. Lastly, because we only evaluated patients with a result of “risk factor identified” on GoCheck Kids screening, patients with not-gradable scans were not
referred. Patients unable to complete screening tests are known to be at higher risk for significant refractive errors and amblyopia17; therefore, our PPV may be an underestimation of the true PPV in our population. This is the first study to report the PPV of GoCheck Kids screening in the primary care setting with a large cohort of young patients age 12 months or younger. Although the AAP and AAPOS recommend initial screening between 1 and 3 years of age, it is important to recognize that earlier screening with GoCheck Kids, while feasible, may have limited utility to detect ARFs. As new vision screening technologies penetrate the market, it is important to evaluate their performance independently in clinical practice.
References 1.
Kirk VG, Clausen NM, Armitage RW, Arnold RW. Preverbal photoscreening for amblyogenic risk factors and outcomes in amblyopia treatment: Early objective screening and visual acuities. Arch Ophthalmol 2008;126:489-92.
2.
Donahue SP, Arthur B, Neely DE, Arnold RW, Silbert D, Ruben JB; POS Vision Screening Committee. Guidelines for automated preschool vision screening: a 10-year, evidence-based update. J AAPOS 2013;17:4-8.
3.
Donahue S, Nixon C; Section of Ophthalmology, American Academy of Pediatrics; Committee on Practice and Ambulatory Medicine, American Academy of Pediatrics; American Academy of Ophthalmology; American Association for Pediatric Ophthalmology and Strabismus; American Association of Certified Orthoptists. Visual system assessment in infants, children, and young adults by pediatricians. Pediatrics 2016;137:28-30.
4.
American Academy of Ophthalmology Preferred Practice Patterns. Pediatric Eye Evaluation.2017. Available online: https://www.aaojournal.org/article/S01616420(17)32958-5/pdf. Accessed March 2, 2019.
5.
Arnold RW, Armitage MD. Performance of Four new photoscreeners on pediatric patients with high risk amblyopia. J Pediatr Ophthalmol Strabismus 2014;51:46-52.
6.
Lowry EA, Wang W, Nyong’o O. Objective vision screening in 3-year-old children at a multispecialty practice. J AAPOS 2015;19:16-20.
7.
Matta NS, Arnold RW, Singman EL, Silbert DI. Comparison between the plusoptiX and MTI photoscreeners. Arch Ophthalmol 2009;127:1591-5.
8.
Peterseim MMW, Papa CE, Wilson ME, et al. The effectiveness of the Spot Vision
Screener in detecting amblyopia risk factors. J AAPOS 2014;18:539-42. 9.
Kennedy RA, Thomas DE. Evaluation of the iScreen digital screening system for amblyogenic factors. Can J Ophthalmol 2000;35:258-62.
10.
Arnold RW, Arnold AW, Hunt-Smith TT, Grendahl RL, Winkle RK. The positive predictive value of smartphone photoscreening in pediatric practices. J Pediatr Ophthalmol Strabismus 2018;55:393-6.
11.
Arnold RW, O’Neil JW, Cooper KL, Silbert DI, Donahue SP. Evaluation of a smartphone photoscreening app to detect refractive amblyopia risk factors in children aged 1-6 years. Clin Ophthalmol 2018;12:1533-7.
12.
Peterseim MMW, Rhodes RS, Patel RN, et al. Effectiveness of the Go Check Kids vision screener in detecting amblyopia risk factors. Am J Ophthalmol 2018;187:87-91.
13.
Wen G, Tarczy-Hornoch K, McKean-Cowdin, et al; Multi-Ethnic Pediatric Eye Disease Study Group. Prevalence of myopia, hyperopia, and astigmatism in non-Hispanic White and Asian children: Multi-Ethnic Pediatric Eye Disease Study. Ophthalmology 2013;120:2109-16.
14.
Fozailoff AF, Tarczy-Hornoch K, Cotter S, et al; Writing Committee for the MEPEDS Study Group. Prevalence of astigmatism in 6- to 72-month-old African American and Hispanic children: The Multi-Ethnic Pediatric Eye Disease Study. Ophthalmology 2011;118:284-93.
15.
Multi-Ethnic Pediatric Eye Disease Study Group. Prevalence of myopia and hyperopia in 6-to 72-month-old African American and Hispanic children: the Multi-Ethnic Pediatric Eye Disease Study. Ophthalmology 2010;117:140-47.e3.
16.
Williams C, Northstone K, Howard M, Harvey I, Harrad RA, Sparrow JM. Prevalence
and risk factors for common vision problems in children: Data from the ALSPAC study. Br J Ophthalmol 2008;92:959-64. 17.
Maguire MG; The Vision in Preschoolers Study Group. Children unable to perform screening tests in vision in preschoolers study: Proportion with Ocular conditions and impact on measures of test accuracy. Invest Ophthalmol Vis Sci 2007;48:83-7.
Legends FIG 1. GoCheck Kids screening and follow-up.
Table 1. Amblyopia risk factor targets (cycloplegic refraction) from the 2013 American Association for Pediatric Ophthalmology and Strabismus Vision Screening Committee guidelines and the 2017 American Academy of Ophthalmology (AAO) Preferred Practice Patterns—Guidelines for Refractive 2,4 Correction in Infants and Young Children Age, months a 3-12 13-30 31-48 a
Astigmatism >3.0 D >2.0 D >2.0 D
Hyperopia >6.0 D >4.5 D >4.0 D
Anisometropia >2.5 D >2.5 D >2.0 D
Myopia > 5.0 D > 3.5 D > 3.0 D
Refractive targets in this age group are from the 2017 AAO Preferred Practice Patterns; they are not listed in the 2013 AAPOS guidelines.
Table 2. Demographic characteristics of patients who failed GoCheck Kids screening Study parameter Follow-up No. 115 Age, months, mean ± SD (range) 24.9 ± 11.1 ( 3-27) Female sex, % 40.0 Race and ethnicity, % White 39.1 Asian 28.7 African American 11.3 Hispanic 13.9 Type of refractive error by GoCheck Kids, % Myopia 19.1 Hyperopia 71.3 Anisometropia 73.9 SD, standard deviation.
Lost to follow-up P value 57 27.0 ± 10.7 (12-43) 0.23 49.1 0.33 33.3 15.8 19.3 15.8
0.57 0.10 0.23 0.92
14.0 86.0 73.7
0.54 0.05 1.00
Table 3. Positive predictive value of GoCheck Kids by age, race/ethnicity, and risk factor identified Study parameter N Age group a All 115 3-12 months 31 13-30 months 42 31-48 months 42 Race/Ethnicity White 45 Asian 33 African 13 American b Latino/a 16
Hyperopia Myopia Anisometropia
82 22 85
PPV
95% CI
0.50 0.26 0.60 0.57
0.41, 0.14, 0.46, 0.43,
0.44 0.42 0.62
0.32, 0.62 0.28, 0.64 0.38, 0.98
0.75 PPV for same ARF 0.36 0.36 0.09
0.60 0.47 0.77 0.75
0.57, 1.00 PPV for any ARF 0.45 c 0.55 0.45
ARF, amblyopia risk factor; PPV, positive predictive value. a
P value for age, 0.03. P value for ethnicity (Latino/a vs. not): 0.006. c P value for the interaction between myopia and age: 0.01. b
Table 4. Reported prevalence of hyperopia, myopia and astigmatism by race and ethnicity from the Multi-Ethnic Pediatric Eye Disease 13-15 Study
White
Asian
African American
Hispanic
Age, months 6-11 12-23 24-35 36-47 6-11 12-23 24-35 36-47 6-11 12-23 24-35 36-47 6-11 12-23 24-35 36-47
Hyperopia >2 D, % 30 24 20 27 17 10 10 16 18 20 18 20 35 22 24 29
Myopia >2 D, % 0 0 0.4 0.3 0 0.8 0.7 0 2 2 1 2 0.7 1.7 1.8 0.8
Astigmatism >1.5 D, % 18 6 4 5 13 7 9 5 25 13 13 6 40 16 15 13