Challenges and Successes of Immunization Registry Reminders at Inner-City Practices

Challenges and Successes of Immunization Registry Reminders at Inner-City Practices Matilde M. Irigoyen, MD; Sally Findley, PhD; Dongwen Wang, PhD; Shaofu Chen, MD, PhD; Frank Chimkin, MBA, MSW; Oscar Pena, BA; Eneida Mendonca, MD, PhD Objectives.–To assess the effectiveness of two serial registry reminder protocols and the interactive effects of reminders with child characteristics on immunization rates. Methods.–At an inner city practice network in New York City we randomized 1662 children aged 6 weeks-15 months due or late for a diphtheria-tetanus-pertussis (DTaP) to 3 groups: continuous reminders (as needed), limited reminders (up to 3) and controls, for 6 months. Reminders were triggered by the hospital registry and immunizations were tracked with both the hospital and city registries. Analyses were based on intention to treat. Results.–At randomization, the study groups were comparable (9.2 months of age, 77% Latino, 86% Medicaid, 49.3% up-to date). A quarter of the children were sent false reminders, 15% had incorrect contact information, and 15% had missed opportunities for vaccination. In the univariate analysis, reminders improved coverage rates, but only for the children sent continu-

ous reminders (51.2% vs. 44.9% controls, p ⬍ .01). Multivariate analysis showed reminders had no independent effect on immunization outcomes. Age, up-to-date and Medicaid status at randomization were strong predictors of a child receiving any subsequent immunization. However, reminders interacted synergistically with Medicaid to increase the likelihood of receiving an immunization. Conclusion.–At an inner city practice network, registry reminders were not effective at improving immunization outcomes due to major system barriers. Immunization registries are powerful vehicles for identifying children in need of immunizations and generating reminders but system challenges must be addressed if this promise is to be achieved in inner city practices. KEY WORDS: childhood immunizations; registry-based reminders; inner city Ambulatory Pediatrics 2006;6:100 –104

R

contact information.20 Additionally, discontinuity of primary care contributes to record scatter and incomplete immunization data.21–23 Reminder effectiveness could also depend on access to services and availability of insurance, but this has not yet been documented. Use of serial reminders could be one way to address some of these problems in inner-city practices, yet, to date the few studies examining the effect of serial reminders have had mixed results.9,17 The objective of this study was to conduct a randomized controlled trial to assess the relative effectiveness of two serial registry reminder protocols at an inner-city practice network. To begin to address the access issues, we also examined how child and practice characteristics interacted with reminders to affect outcomes. A secondary objective was to test a single vaccine algorithm to trigger the reminders.

eminders have promise for improving immunization coverage in primary care settings 1,2 and being valuable tools in narrowing the gap in immunization coverage between poor, minority children and their more affluent counterparts.3 Yet reminders are little used for immunizations because of lack of resources or difficulty in identifying children who need immunizations.4 Registries can address some of these barriers, since they enable identification of children due for immunizations and can generate reminders.5 Registry-generated reminders could be widely applicable because, nationwide, nearly half of children ⬍6 years participate in a registry.6 Early studies of the effectiveness of immunization registry reminders were encouraging.7–13 However, effective implementation of registry reminders in inner-city practices has proven particularly challenging.14 –19 In these communities, high rates of mobility lead to inaccurate

METHODS From the Children’s Hospital of New York Presbyterian, New York, NY (Dr Irigoyen); the Division of General Pediatrics, Department of Pediatrics, College of Physicians & Surgeons, Columbia University, NY, NY (Dr Irigoyen, Dr Findley, Dr Chen, Mr Chimkin and Mr Pena); the Heilbrunn Department of Population and Family Health, Mailman School of Public Health, Columbia University, NY, NY (Drs Irigoyen and Findley); and the Department of Medical Informatics, College of Physicians & Surgeons, Columbia University, NY, NY (Drs Wang and Mendonca). Address correspondence to Matilde M. Irigoyen, Columbia University, 622 W. 168th Street, VC 402, New York, NY 10032 (e-mail: [email protected]). Received for publication June 13, 2005; accepted October 15, 2005.

AMBULATORY PEDIATRICS Copyright © 2006 by Ambulatory Pediatric Association

Study Setting The study was conducted at a network of five community-based pediatric practices affiliated with an academic health center in an inner-city community in New York City. The network is the major pediatric health care provider in the community (60 000 pediatric visits in 2001, 10 000 –25 000 per practice). Most visits (85%) were covered by Medicaid (⬍20% managed care). At the time of the study, the network did not have reminder systems in place.

100

Volume 6, Number 2 March–April 2006

AMBULATORY PEDIATRICS

Immunization Registry Reminders at Inner-City Practices

101

Figure. Randomization and immunization registry algorithm.

EzVAC, a provider-based registry, is a web-based system launched in 2000 to consolidate immunization records for a hospital health care system. EzVAC is linked to the hospital billing system and includes all children born or receiving care at the hospital or affiliated practices. Immunizations are entered at the point of service. On a weekly basis, EzVAC reports to the New York Citywide Immunization Registry (CIR), a population-based registry. New York City providers are mandated to report to CIR; over 90% of providers report to CIR, and of these, 80% do so regularly (Amy Metroka, personal communication, November 2003 email). At the time of the study, the registry linkage was unidirectional from EzVAC to CIR. Providers could look up CIR case by case, but in practice relied on EzVAC for immunization decisions. Thus, to reproduce current practice, we only used EzVAC to trigger the reminders. To assess the internal validity of EzVAC, we compared immunizations in EzVAC and the paper medical records. Before the study and using Centers for Disease Control and Prevention methodology, we audited the records of 3343 randomly sampled 6 –35-month-olds with at least one visit to the network.24 For the 4:3:1:3 immunization series (diphtheria-tetanus-acellular pertussis

[DTaP], polio, measles-mumps-rubella [MMR], and Haemophilus influenza b [Hib]), we found 99% agreement between EzVAC and the paper records for 6 –11-montholds and 76% for 12–23-month-olds. At the end of the study, we found 95% agreement among 125 randomly selected children ⬍24 months (data not shown). We also examined the external validity of EzVAC by comparing immunizations from EzVAC at randomization to those in both EzVAC and CIR at the end of the study (see Results). Study Design, Sample, and Randomization We used a randomized controlled design. Children were eligible for randomization if they were aged 6 weeks to 15 months, had made at least one visit to the network, and were due or late for a DTaP dose (Figure). The eligibility criteria used only one vaccine, DTaP, because its schedule overlaps with other immunizations and DTaP 4 is the dose with the lowest coverage in the 4:3:1:3 series nationwide.25 We calculated a desired sample size of 1662 to achieve a power of 80% and a Type I error of 5%.26 Due to limitations in practice capacity, randomization was carried out weekly, sequentially building the sample until the desired number was attained. We programmed EzVAC to

102

Irigoyen et al

identify eligible children every week, randomly sample a fraction (12%) of those eligible, and then randomly assign those sampled to 1 of 3 study groups: continuous reminders (as many as needed), limited reminders (up to 3), and controls. Randomization was conducted from July 29, 2001 to September 9, 2001, but the 345 children enrolled in the 2 weeks before September 11, 2001 were excluded, and randomization was extended to the weeks of September 23, 2001 and September 30, 2001. Once randomized, children remained in the study group for a period of 6 months. The final sample included 1662 of 13 886 eligible children: 549 randomized to continuous reminders, 552 to limited reminders, and 561 to control group. Every week, EzVAC determined whether children in the two reminder groups needed a repeat reminder for the previous dose or a reminder for a new dose. The reminders consisted of registry-generated postcards with the photograph of a baby and a standard bilingual English/Spanish message: “Dear parent: Our records indicate that . . . now needs 1 or more immunizations. If you haven’t already, please call our clinic at . . . to make an appointment.” Postcards were returned for 13.6% of children who were sent reminders (150 out of 1101). At the end of the study, two address verification surveys confirmed the reliability of the postcard return rate. The study was given exempt status by the Columbia University Medical Center Institutional Review Board. Measures We measured two immunization outcomes at 3 and 6 months postrandomization: (1) received any subsequent immunization and (2) age-appropriate, up-to-date (UTD) status. The practices followed the schedule of the Advisory Committee on Immunization Practices.27 We measured the following independent variables, all obtained from the child’s record in EzVAC: gender, date of birth, ethnicity (Latino vs other), insurance status (Medicaid vs other), practice site (small vs larg), and initial UTD status. Age-appropriate UTD rates are reported for DTaP and 4:3:1:3 as follows: 1 DTaP, 1 polio, and 1 Hib by 3 months; 2 DTaP, 2 polio, and 2 Hib by 5 months; 3 DTaP, 2 polio, and 2 Hib by 7 months; 3 DTaP, 2 polio, 1 MMR, and 2 Hib by 16 months; and 4 DTaP, 3 polio, 1 MMR, and 3 Hib by 19 months. For 29 children in the study who had a DTaP dose ordered but not given due to vaccine shortage, we simulated DTaP as given on the date ordered. We also calculated the rate of missed opportunities to vaccinate, defined as the failure to simultaneously administer any vaccination for which a child was eligible.24 Immunizations were tracked with EzVAC. At the end of the study, for each child not UTD in EzVAC, we looked for additional immunizations in their records in CIR. Of immunizations documented at the end of the study, 84% were in EzVAC (in network); 16% were only in CIR (out of network). Statistical Analysis We performed outcome analyses by intention to treat. Student’s t test was used to evaluate the differences be-

AMBULATORY PEDIATRICS

tween outcomes across study groups. Multivariate logistic regression models were used to estimate predictors of receiving an immunization and being UTD 3 months postrandomization. The models included 1) child characteristics at study randomization: age (0 –5 months, 6 –11 months, and 12–18 months), Latino ethnicity, Medicaid, and UTD status; 2) practice size (large vs small: large ⬎10 000 visits/year); and 3) study group. We estimated models for each study group (continuous reminders, limited reminders, and control) and compared the coefficients between the models. Analyses were conducted with SPSS version 11.0 (SPSS Inc, Chicago, Ill). RESULTS At randomization, there were no significant differences among the groups with regard to age (mean 9. 2 months, SD 4.4 months), gender (53% males), ethnicity (77% Latino), Medicaid (86%), and 4:3:1:3 UTD rate (48.1% control group, 50.2% limited reminder group, 49.5% continuous reminder group). Children were sent an average of 3.4 reminders in the continuous group, 2.6 reminders in the limited group, and 0 reminders in the control group. Of the 3323 reminders sent, 36% were for DTaP 1, 27% for DTaP 2, 22% for DTaP 3, and 15% for DTaP 4. In the univariate analysis, reminders had no significant effect on the likelihood of receiving a subsequent immunization, but did affect coverage rates. Reminders significantly increased immunization coverage for both DTaP and the 4:3:1:3 series, but only for the children sent continuous reminders (3 months postrandomization: 51.2% continuous reminders vs 44.9% controls, P ⬍ .01; 6 months postrandomization: 44.1% continuous reminders vs 39.2% controls, P ⬍ .05; Table 1). Coverage for children sent limited reminders was not significantly better than controls. Missed opportunities averaged 15% and did not differ by study group or practice size. The multivariate analysis showed that age, UTD, and Medicaid status at randomization were strong predictors of a child receiving any subsequent immunization, but practice size had no significant effect on immunization outcomes (Table 2). After controlling for child characteristics and practice size, reminders had no independent effect on the probability of children receiving any immunization or being UTD 3 and 6 months postrandomization. However, reminders reduced the effect of prior UTD status on receiving immunizations for the limited reminder group. In addition, reminders affected the relation between Medicaid and immunizations. In the control group, children on Medicaid were no more likely to receive an immunization than children without Medicaid, but in the reminder groups children on Medicaid were more likely to receive an immunization (limited reminders, OR ⫽ 5.2; CI, 1.6-16.6; continuous reminders, OR ⫽ 3.1; CI, 1.1-9.0). The predictive power of the multivariate models ranged from 68% to 75%. At 3 months, children sent continuous reminders had 6.3% (95% CI 4.0-12.1%) higher rates of immunization than the controls (51.2% vs. 44.9% CI: 47.0-55.4% vs 40.8-49.0%, p ⬍ .01).

AMBULATORY PEDIATRICS

Immunization Registry Reminders at Inner-City Practices

103

Table 1. Immunization Status by Study Group (N ⫽ 1662) At Randomization

3 Months Postrandomization

6 Months Postrandomization

Study Group

n

Up-to-date 4:3:1:3‡ %

Received Any Immunization %

Up-to-date DTaP† %

Up-to-date 4:3:1:3‡ %

Received Any Immunization %

Up-to-date DTaP %

Up-to-date 4:3:1:3 %

Continuous reminders Limited reminders Control

549 552 561

49.5 50.2 48.1

46.8 47.8 45.1

56.5 54.9 52.9

51.2** 46.4 44.9

55.7 56.0 54.4

50.1* 48.2 45.8

44.1* 42.0 39.2

†Diphtheria-tetanus-acellular pertussis. ‡4 DTaP:3 polio:1 MMR:3 Hib. *p ⬍ .05. **p ⬍ .01.

immunization data, inaccurate patient contact information, and provider missed opportunities. In our study, 1 in 4 children was sent false reminders, 1 in 6 could not be reached due to incorrect addresses, and 1 in 6 was not vaccinated due to provider missed opportunities. If registry-based reminders are to be effective, the assessment of immunization status must be accurate at the reminder date; contact information needs to be current to insure that families actually receive the reminders, and lastly, providers must be primed and ready to vaccinate when children return for the requested immunizations. The first challenge to registry reminders in inner-city practices is to complete documentation of immunizations. With incomplete documentation, reminders triggered by the registry have unacceptable rates for false reminders, which we documented through registry linkages. Record scatter was the major source of incomplete data. Linkage between registries is part of the solution to this problem,28 but the completeness of data in the registry also depends on the consistency and timeliness of provider reporting. In

Misclassification of Vaccine Doses and False Reminders A quarter of the children (n ⫽ 425, 25.6%) were misclassified as due or late for a DTaP dose and were sent false reminders. Of these, a small number of children (n ⫽ 68) had been vaccinated at the network, but data entry to EzVAC had been delayed. Most false reminders were sent to children (n ⫽ 357) who were vaccinated by out-ofnetwork providers. The misclassified children were evenly distributed across study groups. The average rate for missing data for a DTaP dose was 30%. Applying the calculation 1 ⫺ (1 ⫺ missing rate per dose) ⴱⴱ (number of doses),28 the misclassification error for the four DTaP dose series would be 76%. DISCUSSION Immunization registry reminders triggered by a provider-based registry were not effective at an inner-city practice network in New York City. The lack of effectiveness can be attributed to three major system barriers: incomplete

Table 2. Multivariate Models of Immunization Status (3 Months Postrandomization) Received Any Immunization

Characteristics at Randomization Child Age (reference: 12–17 mo) 1–5 mo 6–11 mo Ethnicity (reference; not Latino) Latino Insurance (reference: not Medicaid) Medicaid Up-to-date at randomization Small practice Study group

Control Group

Limited Reminders

Up-to-date

Continuous Reminders

Control Group

Limited Reminders

Continuous Reminders

OR

95% CI*

OR

95% CI

OR

95% CI

OR

95% CI

OR

95% CI

OR

95% CI

7.16 1.37

3.14–16.3 0.79–2.38

4.65 0.84

2.03–10.6 0.50–1.44

3.02 1.27

1.53–6.00 0.73–2.20

1.33 1.60

0.67–2.62 0.87–2.94

1.25 1.20

0.62–2.51 0.68–2.14

0.54 2.16

0.28–1.07 1.10–4.22

1.08

0.63–1.85

0.80

0.46–1.40

1.01

0.60–1.69

0.96

0.55–1.70

1.14

0.65–2.00

0.79

0.45–1.39

2.32 0.60–8.92 2.96 1.76–4.98 1.20 0.74–1.94 ... ... Overall correct ⫽ 0.69 Sensitivity ⫽ 0.76 Specificity ⫽ 0.59 n ⫽ 561

*CI indicates confidence interval.

5.20 1.62–16.6 1.85 1.11–3.11 1.44 0.91–2.26 1.11 0.81–1.53 Overall correct ⫽ 0.68 Sensitivity ⫽ 0.79 Specificity ⫽ 0.54 n ⫽ 552

3.07 1.05–9.02 3.16 1.89–5.30 1.21 0.75–1.94 1.01 0.73–1.40 Overall correct ⫽ 0.69 Sensitivity ⫽ 0.78 Specificity ⫽ 0.59 n ⫽ 549

1.94 0.49–7.73 7.77 4.45–13.6 0.93 0.56–1.53 ... ... Overall correct ⫽ 0.73 Sensitivity ⫽ 0.76 Specificity ⫽ 0.69 n ⫽ 561

1.90 0.66–5.45 2.16 0.73–6.38 5.83 3.37–10.1 10.8 5.77–20.3 0.92 0.58–1.47 0.78 0.47–1.30 0.94 0.67–1.30 1.25 0.89–1.76 Overall correct Overall correct ⫽ 0.71 ⫽ 0.75 Sensitivity ⫽ Sensitivity ⫽ 0.75 0.82 Specificity ⫽ Specificity ⫽ 0.67 0.65 n ⫽ 552 n ⫽ 549

104

Irigoyen et al

our study, despite using two registries as sources of data, immunization coverage and false reminder rates are likely to be underestimated. This may be especially true for older children due to the cumulative nature of missing data.28 Strategies to enhance provider reporting through incentives such as linkage to the Vaccines for Children Program merit further study. Another solution is to facilitate timely provider documentation of immunization through webbased entry of data at the point of service. The second challenge to registry reminders in inner-city practices is to maintain accurate contact information for a population with high rates of mobility and discontinuity of care. Alternative procedures for updating addresses in registries may include linkages to Medicaid and managed care plans, which may have more current contact information. The third challenge to registry reminders is to decrease provider missed opportunities.14 –16,29 Quality improvement strategies to reduce missed opportunities need to be coupled with reminders to improve their effectiveness. An intriguing finding of our study was the variation in reminder impact, depending on the child’s age, prior immunization status, and Medicaid. In this inner-city community, there was a synergism between Medicaid and reminders. Families on Medicaid have greater access to health care,22 and this may have enhanced their responsiveness to reminders. Future reminder interventions could further explore mechanisms for taking advantage of this synergism among the Medicaid-insured. A study innovation was the use of a simplified reminder algorithm based on DTaP. Given the increased probability of false reminders with each additional vaccine for which reminders are sent, the use of a single vaccine indicator could help reduce false reminders. In conclusion, this study identified system challenges in meeting the promise of registry-based reminders to improve immunization coverage in inner-city practices. ACKNOWLEDGMENTS This study was supported by grant U66/CCU212961 from the National Immunization Program, Centers for Disease Control and Prevention (Dr Irigoyen). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the funding agency.

REFERENCES 1. Briss PA, Rodewald LE, Hinman AR, et al. Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. The Task Force on Community Preventive Services. Am J Prev Med. 2000;18:97–140. 2. Szilagyi PG, Bordley C, Vann JC, et al. Effect of patient reminder/ recall interventions on immunization rates: a review. JAMA. 2000;284:1820 –1827. 3. Chu SY, Barker LE, Smith PJ. Racial/ethnic disparities in preschool immunizations: United States, 1996-2001. Am J Public Health. 2004;94:973–977. 4. Tierney CD, Yusuf H, McMahon SR, et al. Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics. Pediatrics. 2003;112:1076 –1082. 5. Freeman VA, DeFriese GH. The challenge and potential of childhood immunization registries. Annu Rev Public Health. 2003;24: 227–246. 6. Centers for Disease Control and Prevention (CDC). Immunization Registry Progress, 2002. MMWR Morb Mortal Wkly Rep. 2002;51:760.

AMBULATORY PEDIATRICS 7. Alemi F, Alemagno SA, Goldhagen J, et al. Computer reminders improve on-time immunization rates. Medical Care. 1996;34(suppl): OS45–OS51. 8. Dini EF, Linkins RW, Chaney M. Effectiveness of computergenerated telephone messages in increasing clinic visits. Arch Pediatr Adolesc Med. 1995;149:902–905. 9. Dini EF, Linkins RW, Sigafoos J. The impact of computer-generated messages on childhood immunization coverage. Am J Prev Med. 2000;19:68 –70. 10. Harper PG, Madlon-Kay DJ, Luxenberg MG, Tempest R. A clinic system to improve preschool vaccinations in a low socioeconomic status population. Arch Pediatr Adolesc Med. 1997;151:1220 –1223. 11. Lieu TA, Capra AM, Makol J, et al. Effectiveness and costeffectiveness of letters, automated telephone messages, or both for underimmunized children in a health maintenance organization. Pediatrics. 1998;101(4):E3. 12. Linkins RW, Dini EF, Watson G, Partiarca PA. A randomized trial of the effectiveness of computer-generated telephone messages in increasing immunization visits among preschool children. Arch Pediatr Adolesc Med. 1994;148:908 –914. 13. Stehr-Green PA, Dini EF, Lindegren ML, Patriarca PA. Evaluation of telephoned computer-generated reminders to improve immunization coverage at inner-city clinics. Public Health Rep. 1993;108:426 – 430. 14. Daley MF, Steiner JF, Brayden RM, et al. Immunization registrybased recall for a new vaccine. Ambul Pediatr. 2002;2:438 – 443. 15. Daley MF, Steiner JF, Kempe A, et al. Quality improvement in immunization delivery following an unsuccessful immunization recall. Ambul Pediatr. 2004;4:217–223. 16. Kempe A, Lowery NE, Pearson KA, et al. Immunization recall: effectiveness and barriers to success in an urban teaching clinic. J Pediatr. 2001;139:630 – 635. 17. LeBaron CW, Starnes DM, Rask KJ. The impact of reminder-recall interventions on low vaccination coverage in an inner-city population. Arch Pediatr Adolesc Med. 2004;158:255–261. 18. Rodewald LE, Santoli JM. The challenge of vaccinating vulnerable children. J Pediatr. 2001;139:613– 616. 19. Davis MM, Szilagyi PG. Can quality improvement reach into pockets of need for childhood immunizations? Ambul Pediatr. 2004;4:224 –225. 20. Findley SE, Irigoyen M, Schulman A. Children on the move and vaccination coverage in a low-income, urban, Latino population. Am J Public Health. 1999;89:1728 –1731. 21. Irigoyen M, Findley S, Chen S, et al. Early continuity of care and immunization coverage. Ambulatory Pediatrics. 2004;4:199 –203. 22. Newacheck PW, Stoddard JJ, Hughes DC, Pearl M. Health insurance and access to primary care for children. N Engl J Med. 1998;338:513–519. 23. Yusuf H, Adams M, Rodewald L, et al. Fragmentation of immunization history among providers and parents of children in selected underserved areas. Am J Prev Med. 2002;23:106 –112. 24. Centers for Disease Control and Prevention (CDC). Clinic assessment software application. Available at: http://www.cdc.gov/nip/ casa. Accessed July 1, 2000. 25. Barker L, Santoli J, and McCauley M. Morbidity and Mortality Weekly Report (MMWR), July 30, 2004/53(29);658 – 661. 26. Irigoyen M, Findley S, Earle B, et al. Impact of appointment reminders on vaccination coverage at an urban clinic. Pediatrics. 2000;106: 916 –923. 27. Centers for Disease Control and Prevention (CDC). Recommended Childhood Immunization Schedule - United States - 2001. MMWR Morb Mortal Wkly Rep. 2001;50:7–10, 19. 28. Rodewald L, Maes E, Stevenson J, et al. Immunization performance measurement in a changing immunization environment. Pediatrics. 1999;103:889 – 897. 29. Szilagyi PG, Rodewald LE. Missed opportunities for immunizations: a review of the evidence. J Public Health Manag Pract. Winter 1996;2:18 –25.