Immunization registry accuracy

Immunization Registry Accuracy Improvement with Progressive Clinical Application Arthur J. Davidson, MD, MSPH, Paul Melinkovich, MD, Brenda L. Beatty, MSPH, Vijayalaxmi Chandramouli, MS, Simon J. Hambidge, MD, PhD, Stephanie L. Phibbs, MPH, Patricia Braun, MD, Charles W. LeBaron, MD, John F. Steiner, MD, MPH Background: Healthcare systems have been challenged to ensure the timely administration of immunizations. Immunization registries have been proposed to improve the accuracy and completeness of immunization information and to promote effective practice. Methods:

Comparison of randomly selected samples from two birth cohorts (1993 and 1998) from Denver Health Medical Center. Chart review and immunization registry information for these groups were compared; a composite immunization was recorded and up-to-date (UTD) status established. Registry data were compared with this composite using a sensitivity measure to assess completeness and accuracy.

Results:

Among 818 children in the 1993 cohort and 1043 children in the 1998 cohort, there were 6386 and 6886 valid immunizations, respectively. The registry recorded 71.4% and 97.7% of these for the 1993 and 1998 cohorts, respectively (p ⬍0.001). The apparent UTD rate, as measured with registry data alone, improved from 37% to 79% between the two time frames (p ⬍0.001). Composite UTD status was 83.1% and 78.9% (1993 vs 1998, respectively). Accurate registry-defined UTD status improved from 44.4% to 100% between the two intervals.

Conclusions: Immunization registry accuracy improved dramatically for recorded immunizations and UTD status. However, after 3 years of registry use, the overall proportion of children who were UTD had not significantly improved. The mere presence of a registry does not ensure more complete vaccination coverage. Other registry-based strategies, including use of the data for reminder, recall, and audit, may further improve immunization coverage. (Am J Prev Med 2003;24(3):276 –280) © 2003 American Journal of Preventive Medicine

Introduction

C

hildhood immunizations have reduced the incidence of many infectious diseases.1–3 The challenge for healthcare systems is the development of mechanisms to ensure timely administration and thorough documentation of immunizations.4 –7 Parental recall of their child’s immunization history inadequately measures and potentially interferes with provider efforts for timely immunizations.8,9 Providers with incomplete information due to record scattering are similarly hampered in their vaccination efforts.10,11

From the Department of Preventive Medicine and Biometrics (Davidson, Steiner), Department of Family Medicine (Davidson), Division of General Internal Medicine (Chandramouli, Phibbs, Steiner), and Division of Pediatrics (Melinkovich, Beatty, Hambidge, Braun), University of Colorado Health Sciences Center, Denver, Colorado; Denver Community Health Services (Davidson, Melinkovich, Hambidge, Braun), Denver, Colorado; and National Immunization Program, Centers for Disease Control and Prevention (LeBaron), Atlanta, Georgia Address correspondence to: Arthur J. Davidson, MD, MSPH, Denver Public Health, 605 Bannock Street, Denver CO 80204. E-mail: [email protected].

276

Computer-based registries have been proposed to improve the accuracy of immunization information and to promote effective immunization practices.12 The concept of immunization registries/tracking systems has existed for more than 30 years.13,14 Their relative success in improving immunization coverage and documentation4 is cause for their inclusion in national immunization coverage strategies15–18 by monitoring and providing feedback to caregivers,19 patients/parents,20 and case management workers.21 An effective immunization registry needs to ensure complete and accurate data entry for all vaccinations provided at all facilities for any individual.17,22 Several studies have suggested that registry data capture and accuracy may be too low to provide the projected benefits.23–25 One study suggested that registry implementation at the point of service improved data quality.26 Does the completeness or accuracy of an immunization registry improve with time and progressive clinical application? This study was undertaken to determine how implementation of a point-of-service immunization registry over a 3-year interval affected

Am J Prev Med 2003;24(3) © 2003 American Journal of Preventive Medicine • Published by Elsevier

0749-3797/03/$–see front matter doi:10.1016/S0749-3797(02)00638-4

data completeness and accuracy for two birth cohorts of children. If improved accuracy was observed, a second objective was to determine if that change was associated with improved up-to-date (UTD) immunization status.

Methods Registry Denver Health (DH), an integrated, citywide network of 25 healthcare delivery sites—that is, community health centers (CHCs), school-based clinics, an urban public hospital (Denver Health Medical Center [DHMC]), and a public health department—provides care to medically underserved and indigent citizens of Denver, Colorado. In January 1996, an immunization registry, using ADIOS (Automated Data Integration Operating System) Software (JK Inc., Denver, Colorado, 1996, no longer available) was implemented throughout all DH facilities to track vaccine administration. Previously, immunizations administered by public health clinics were maintained in separate paper files. At registry inception, this unified electronic record was declared the systemwide legal report/repository, with direct recording of immunization events and history in a common database regardless of site. The registry was initially populated with 1993–1995 administrative (billing) data to facilitate tracking and reporting without having to enter every prior immunization event. Subsequently, nursing staff entered all data at the time of vaccine administration. By 1997, the registry had 126,890 immunizations for 16,997 children born after September 1, 1992; the registry grew to 551,622 immunizations for 64,329 children by January 2001. Final immunization registry analysis files for the earlier and later cohorts were generated on September 29, 1997, and November 1, 2000, respectively. Registry accuracy was assessed by comparing randomly selected, active-user samples from two DHMC birth cohorts: (1) 1993 cohort born in 1993 (n ⫽2653) and (2) 1998 cohort born between July 1, 1998 and June 30, 1999 (n ⫽2844). Active users were children with one or more CHC outpatient visits and an entry into the immunization registry. Active users comprised 82% for both the 1993 and 1998 DH birth cohorts. The earlier (1993) cohort was based on an a priori samplesize calculation for random sampling from all ten CHCs as part of an observational retrospective study. The later (1998) cohort was an a priori cluster sample-size calculation for random assignment of all children attending three CHCs assigned to the control arm of an immunization intervention study. The Colorado Multiple Institutional Review Board approved both studies (protocol numbers 97-075 and 97-507, respectively). All ambulatory care records (e.g., primary care, urgent care, specialty care, and documented immunizations) are unified in one outpatient chart residing at one clinic location. When children transfer their care to a different community clinic within the network, their medical charts are transferred as well. All charts use agency-wide, standard, age-specific, well-child and acute-care encounter forms. Abstraction was completed using a standardized data entry form. One of two project managers oriented and trained all abstractors on chart interpretation and data entry methods. Any immunization administered and documented before 1 year of life,

Table 1. Comparison of demographic characteristics and immunizations recorded for 1993 and 1998 cohorts, Denver Health, Denver, Colorado 1993 cohort nⴝ818 (%)

Demographic characteristic Race African American Latino White Other Immunizations All sources combined Unique immunizations

1998 cohort nⴝ1043 (%)

11 75 10 4

5 84 8 3

10,780 6,386

12,779 6,886

␹2 p value 0.001 0.001 NS NS NA NA

NS, not significant; NA, not applicable.

identified by chart review and/or immunization registry, was considered valid for inclusion. Final immunization registry analysis files for the earlier and later cohorts were generated on September 29, 1997 (3 years and 9 months after birth of last cohort child) and November 1, 2000 (1 year and 4 months after birth of last cohort child), respectively.

Assessment of Accuracy Patient-specific electronic registry and chart abstraction data were sorted by antigen type (e.g., diphtheria/tetanus/acellular pertussis [DTaP]) and date of immunization. A composite database was constructed and processed by each patient to unduplicate same-day immunizations and eliminate unnecessary27 immunizations. UTD immunization status was determined at 12 months of age by the 3:2:3 regimen (3 DTaP, 2 polio [either oral or inactivated], and 3 Haemophilis influenzae b [Hib]). Although the recommended schedule for both cohorts included Hepatitis B vaccination,28 Vaccine for Children (VFC) distribution for the relatively new vaccine was limited in 1993. Thus, this immunization was excluded from analysis.

Statistical Procedures Analysis with SAS (version 6.12, SAS Institute, Cary, North Carolina) tested associations between categorical or proportional data with chi-square tests. The concept of “sensitivity,” a common measure of the accuracy of a diagnostic test, was adopted to assess the accuracy of the registry. For this purpose, the sensitivity of the registry was defined as the proportion of children UTD by the composite measure that was identified as UTD by the registry.

Results There were 818 active users identified in the 1993 cohort and 1043 children in the 1998 cohort. Between the two cohorts, there was similar ethnic/race distribution, with a larger percentage of Latino patients and less African Americans in the later cohort (Table 1). Combining all data sources, 10,780 immunizations were administered to the 1993 cohort and 12,779 for Am J Prev Med 2003;24(3)

277

Table 2. Comparison of immunizations recorded and UTD status for 1993 versus 1998 DHMC cohorts by chart review, immunization registry, and composite, Denver, Colorado

Comparison category Immunizations recorded Chart review Immunization registry Composite UTD status Chart review Immunization registry Composite

1993 Cohort nⴝ818 n (%)

1998 Cohort nⴝ1084 n (%)

6314 4559 6386

6048 6731 6886

672 (82.2) 302 (36.9) 680 (83.1)

598 (57.3) 823 (78.9) 823 (78.9)

DHMC, Denver Health Medical Center; UTD, up-to-date.

the 1998 cohort. After data cleaning and merging, there were 6386 and 6886 unique and valid immunizations administered to the 1993 and 1998 cohorts, respectively. Were each child to receive the complete series of eight immunizations, the expected totals would be 6544 and 8344 immunizations for the 1993 and 1998 cohorts, respectively. In the earlier timeframe, 4559 (71.4%) of 6386 composite immunizations were recorded in the ADIOS registry. Three years after registry implementation, 6731 (97.7%) of 6886 composite immunizations were recorded (Table 2). The number of immunizations uniquely identified from chart review diminished significantly from the earlier to later intervals (i.e., 1827 [28.6%] vs 155 [2.2%] immunizations), suggesting that data-entry omissions/errors to the registry had not significantly increased. Alternatively, the number of omissions for the earlier period may have been compounded by inaccurate billing information prior to registry inception. The composite UTD rate for children was 680/818 (83.1%) in the 1993 cohort and 823/1043 (78.9%) for the 1998 cohort (␹2, p ⬍0.005), as shown in Table 2. Using only immunizations recorded by chart review, the UTD rates were 672/818 (82.2%) for 1993 cohort and 598/1043 (57.3%) for the 1998 cohort (␹2, p ⬍0.001); using the immunization registry alone, the UTD rates were 36.9% and 78.9% (1993 vs 1998; ␹2, p ⬍0.001).Immunization registry accuracy was measured comparing the “gold standard” of composite immunizations (from the chart and registry review) with UTD assessment as determined by the registry alone.

Discussion Compared with a composite chart and registry database, the accuracy of the immunization registry improved dramatically for immunizations recorded and UTD status. However, after 3 years of registry use, the overall proportion of children who were UTD had not 278

significantly improved. For the later (1998) cohort, the immunization registry was more complete (more vaccinations captured) and achieved greater agreement with the composite UTD status, while medical records (via chart review) became less complete (fewer vaccinations captured). Thus, the electronic registry has strong potential to assist public health officials and others in the conduct of immunization audits and reminder/ recall. Probably the greatest factor contributing to improved accuracy was declaring the registry as the official record, which requires clinics to immediately document vaccinations when given. Contrary to other registry implementations, where entry may be completed weeks to months after a immunization was given, appropriate assessments of coverage and reminder/recall may be accomplished in near real time. Commitment at the system level required absolute sponsorship and maintenance responsibility for the immunization registry concept despite logistical setbacks. Three years after inception, the product DH purchased was orphaned and support vanished when the vendor went bankrupt. Source-code acquisition and product support became major information services priorities. This agency-wide commitment to support the registry was essential to gain provider trust and meet documentation requirements for a registry that serves as the authorized (legal) immunization record. Improved registry accuracy for UTD status reflects increased provider trust and system reliance over time. In a previous study across a spectrum of care providers, provider trust varied by specialty. Nurses and nurse practitioners were the most optimistic about the power of a registry. Prior unsuccessful registry experiences adversely impacted perceptions and may ultimately affect implementation and clinical utility.29 A strategic DH-registry–implementation decision was to depend almost exclusively on nurses’ data entry. If the system had relied on busy physicians for data entry, diminished data accuracy might have led to a self-perpetuating cycle of skepticism about the registry and continued deterioration of data quality. Substantial research suggests that tracking and reminder systems significantly increase immunization rates18,30 –32 and are cost effective.18,33 Payne et al.17 demonstrated improved data quality by integrating electronic data entry by clinicians in place of paper charting. Identifying children whose records indicate incomplete immunization series and generation of patient31,33,34 and provider35 reminder/recall efforts are key immunization registry features. Data entry omissions may be corrected in the registry over time. Samuels et al.36 observed improved registry accuracy for one practice through changing information flow and concentrating immunization tracking as one staff member’s responsibility. Although such improved accuracy is associated with additional costs (range of

American Journal of Preventive Medicine, Volume 24, Number 3

estimated annual per child costs for large registries, $3.91 to $5.26), these have been offset by reduced manual processing of immunization data and capacity for reminder/recall, audits, and automation of national immunization surveys.37,38 While previous studies have shown computerized registries to be more accurate than parental records,14,39 the keys to accuracy are inclusion of as many providers as possible who order and administer immunizations and a high degree of collaboration across all health-sector providers. The DH public healthcare system, which cares for nearly 3000 additional newborn children each year, has made participation and collaboration priorities throughout community health centers, school-based clinics, and public health immunization clinics. Without such cooperation and shared priorities, these results may not generalize to immunization registries established in less integrated or more fragmented healthcare systems. These study results should be applicable to the private sector if a jurisdiction’s registry is built on such a collaborative foundation. During recent years, growing Medicaid managed-care options may result in greater record scatter. A registry that serves the entire spectrum of care becomes increasingly more important. Limitations to the establishment of a registry (such as this) include the burden of data entry for previous immunization history. Having populated the registry with administrative data, the inherent level of inaccuracy in billing data may inadequately capture all immunization events. The early population of the registry with inaccurately coded billing data is different from real-time documentation and may have contributed to some of the major findings. Therefore, a period of nearly 2 years, from registry implementation to preparation of the analysis dataset for the earlier cohort, is suggested. Ample time had passed, during which many patients who returned for care would have had their registry information updated. In addition, previous assessments of DH chart and billing data showed a 5% to 10% discordance rate. As administrative systems improve and become more integrated, standard pointof-service mechanisms for providers to electronically submit immunization information will be essential. Although randomly selected, the three 1998 CHCs may have applied the registry in a different manner than the larger 1993 cohort, which included all ten CHCs. From another intervention study (data not shown), registry use was uniformly applied across all facilities in the later time period. Although the immunization registry’s accuracy improved, this was tempered by the fact that the comparison data source (chart review) decreased in accuracy and completeness over the interval. If paper record keeping declined, it would make the registry appear more accurate. The greater than twofold rise in accuracy of UTD status for the registry (44% to 100%) was

greater than would otherwise be explained by decreased, medical-record, UTD-status accuracy (99% to 73%). This would suggest that additional factors (other than the deteriorating state of paper record keeping) actually resulted in improved registry accuracy. Notably, the UTD rate at 1 year for 6-month immunizations was lower in the later cohort. Potential reasons include (1) immunization received elsewhere (data omissions) or data entry errors22 or (2) providers deviating from current recommendations with increased numbers of injectable immunizations in the recent standard 6-month regimen. This finding suggests that the mere presence of a registry does not ensure more complete vaccination coverage. During the study period, the registry was used in a sporadic fashion to generate reminder/recall lists, as not all clinics had fully invested in such efforts. Only through use of the data and application of reminder, recall, and audit features can registries improve data quality and coverage. Implementation of an immunization registry requires careful review of its purpose, function, and value within the healthcare organization12 and the broader community.40,41 Progress toward the elusive, ultimate goal of a network of interoperable local and state databases15 includes minimum design principles using Health Level 7 standards.42 System designers need to accommodate various data sources and resolve apparently inconsistent immunization history dates and events when merging data between systems (e.g., adjacent local jurisdictions or local and state) until standards and real-time bi-directional data flow are established. Given the incompleteness and/or inaccuracies of any single immunization data source8,22,43 and the impact of record scattering,10 merging available paper and electronic data is paramount for registry success. This study has demonstrated improved immunizationregistry accuracy over time. Yet further research is needed to define strategies for correctly combining data as we work toward interoperable systems, and to define methods that better utilize registries to enhance the UTD immunization status of populations. This study was funded by the National Immunization Program via contract with Colorado Department of Public Health and Environment and Association of Teachers of Preventive Medicine/Centers for Disease Control and Prevention cooperative agreement number TS 252-13/15. We are indebted to Louis Espinoza-Organista, Susan Macaskill-Deloach, and Kristin Miller for their efforts in data collection and to Drs. Robert Linkens, Susan Chu, and Jeanne Santoli for their thoughtful comments on an earlier version of the manuscript.

References 1. Centers for Disease Control and Prevention. Final 1994 reports of notificable diseases. MMWR Morb Mortal Wkly Rep 1995;44:537–43.

Am J Prev Med 2003;24(3)

279

2. Centers for Disease Control and Prevention. Impact of vaccines universally recommended for children—United States, 1900 –1998. MMWR Morb Mortal Wkly Rep 1999;48:243–8. 3. Vazquez M, LaRussa PS, Gershon AA, Stienberg SP, Freudigman K, Shapiro ED. The effectiveness of the varicella vaccine in clinical practice. N Engl J Med 2001;344:955–60. 4. Abramson JS, O’Shea TM, Ratledge DL, Lawless MR, Givner LB. Development of a vaccine tracking system to improve the rate of age-appropriate primary immunization in children of lower socioeconomic status. J Pediatr 1995;126:583–6. 5. Christy C, McConnochie KM, Zernik N, Brzoza S. Impact of an algorithmguided nurse intervention on the use of immunization opportunities. Arch Pediatr Adolesc Med 1997;151:384 –91. 6. Hinman AR. What will it take to fully protect all American children with vaccines? Am J Dis Child 1991;145:536 –7. 7. Berwick D. Harvesting knowledge from improvement. JAMA 1996;275:877– 8. 8. Watson M, Feldman K, Sugar N, Sommer C, Thomas E, Lin T. Inadequate history as a barrier to immunization. Arch Pediatr Adolesc Med 1996;150: 135–9. 9. Fierman AH, Rosen CM, Legano LA, Lim SW, Mendelsohn AL, Dreyer BP. Immunization status as determined by patients’ hand-held cards vs. medical records. Arch Pediatr Adolesc Med 1996;150:863–6. 10. Yawn BP, Edmonson L, Huber L, Poland GA, Jacobson RM, Jacobsen SJ. The impact of a simulated immunization registry on perceived childhood immunization status. Am J Manag Care 1998;4:185–92. 11. Stokley S, Rodewald LE, Maes EF. The impact of record scattering on the measurement of immunization coverage. Pediatrics 2001;107:91–6. 12. Sinn JS, Kronenburg MA, Morrow AL. The purpose and functions of immunization information systems within health care organizations. Arch Pediatr Adolesc Med 1997;151:615–20. 13. Saunders J. Results and cost of a computer-assisted scheme. Br J Prev Soc Med 1970;24:187–91. 14. Ortega AN, Andrews SF, Katz SH, et al. Comparing a computer-based childhood vaccination registry with parental vaccination cards: a population-based study of Delaware children. Clin Pediatr 1997;36:217–21. 15. National Vaccine Advisory Committee. Developing a national childhood immunization information system: registries, reminders and recall. Washington, DC: Subcommittee on Vaccination Registries, U.S. Department of Health and Human Services, U.S. Public Health Service, 1994. 16. Gostin LO, Lassarini Z. Childhood immunization registries: a national review of public health information systems and the protection of privacy. JAMA 1995;274:1793–9. 17. Payne T, Kanvik S, Seward R, et al. Development and validation of an immunization tracking system in a large health maintenance organization. Am J Prev Med 1993;9:96 –100. 18. Loeser H, Zvagulis I, Hercz L, Pless IB. The organization and evaluation of a computer-assisted centralized immunization registry. Am J Public Health 1983;73:1298 –301. 19. McDonald CJ, Siu LH, Smith DM, et al. Reminders to physicians from an introspective computer medical record. Ann Intern Med 1984;100:130 –8. 20. Paunio M, Virtanen M, Peltola H, et al. Increase of vaccination coverage by mass media and individual approach: intensified measles, mumps and rubella prevention program in Finland. Am J Epidemiol 1991;133:1152–60. 21. Wood D, Halfon N, Donald-Sherbourne C, et al. Increasing immunization rates among inner-city, African American children: a randomized trial of case management. JAMA 1998;279:29 –34. 22. Wilton R, Pennisi AJ. Evaluating the accuracy of transcribed computerstored immunization data. Pediatrics 1994;94:902–6. 23. Samuels RC, Appel L, Reddy S, Tilson M. Tracking of immunizations: computers may not be the magic bullet. Presented at annual conference of Ambulatory Pediatric Association, May 5, 1996, Washington, DC.

280

24. Stonehocker-Quick L, Rotharmel P, Webb W, Meagher G, Hoekstra EJ. How valid is registry data for AFIX? Presented at National Immunization Conference, May 21, 1997, Detroit MI, sponsored by Centers for Disease Control and Prevention. 25. LeBaron CW, Mercer JT, Massourdi MS, et al. Changes in clinic vaccination coverage after institution of measurement and feedback in 4 states and 2 cities. Arch Pediatr Adolesc Med 1999;153:879 –86. 26. Adams WG, Conners WP, Mann AM, Palfrey S. Immunization entry at the point of service improves quality, saves time and is well-accepted. Pediatrics 2000;106:489 –92. 27. Murphy TV, Pastor P, Medley FB. Factors associated with unnecessary immunization given to children. Pediatr Infect Dis J 1997;16:47–52. 28. Centers for Disease Control and Prevention. Hepatitis B virus: a comprehensive strategy for eliminating transmission in the United States through universal childhood vaccination: recommendation of the Immunization Practices Advisory Committee. MMWR Morb Mortal Wkly Rep 1991;40:1– 19. 29. Christakis DA, Stewart L, Bibus D, et al. Providers’ perceptions of an immunization registry. Am J Prev Med 1999;17:147–50. 30. Linkins RW, Feikema SM. Immunization registries: the cornerstone of childhood immunization in the 21st century. Pediatr Ann 1998;27:349 –54. 31. Tollestrup K, Hubbard B. Evaluation of a follow-up system in a county health department’s immunization clinic. Am J Prev Med 1991;7:24 –8. 32. Klachko DM, Wright DL, Gardner DW. Effect of a microcomputer-based registry on adult immunizations. J Fam Pract 1989;29:169 –72. 33. Lieu T, Black S, Ray R, et al. Computer-generated recall letters for underimmunized children: how cost-effective? Pediatr Infect Dis J 1997;16: 28 –33. 34. Stehr-Green P, Dini E, Lindegren M, Patriaca P. Evaluation of telephoned computer-generated reminders to improve immunization coverage at inner-city clinics. Public Health Rep 1993;108:426 –30. 35. Dini E, Chaney M, Moolenaar R, LeBaron C. Information as intervention: how Georgia used vaccination coverage deata to double public sector vaccination coverage in seven years. J Public Health Manag Pract 1996;2: 45–9. 36. Samuels RC, Appel L, Reddy SI, Tilson RS. Improving accuracy in a computerized registry. Ambul Pediatr 2002;2:187–92. 37. Rask KJ, Wells KJ, Kohler SA, Rust CT, Cangialose CB. Measuring immunization registry costs: promises and pitfalls. Am J Prev Med 2000;18:262–7. 38. Horne PR, Saarlas KN, Hinman AR. Costs of immunization registries: experiences from the All Kids Count II Projects. Am J Prev Med 2000;19: 94 –8. 39. Renfrew BL, Kempe A, Lowery NE, Chandramouli V, Steiner JF, Berman S. The impact of immunization record aggregation on up-to-date rates: implications for immunization registries in rural areas. J Rural Health 2001;17:122–6. 40. DeFriese G, Flaherty K, Freeman V, et al. Developing child immunization registries: the All Kids Count Program. In: To improve health and health care. Princeton, NJ: Robert Woods Johnson Foundation, 1997:187–208. 41. Wood D, Saarlas KN, Inkelas M, Matvas BT. Immunization registries in the United States: implications for the practice of public health in a changing health care system. Annu Rev Public Health 1999;20:231–55. 42. Centers for Disease Control and Prevention. Implementation guide for immunization transactions using version 2.3.1 of the Health Level Seven (HL7) standard protocol, version 2.0. Atlanta, GA: National Immunization Program, Data Management Division, Systems Development Branch, June 1999. 43. Suarez L, Simpson D, Smith D. Errors and correlates in parental recall of child immunizations: effects on vaccination coverage estimates. Pediatrics 1997;99:(E3), 723.

American Journal of Preventive Medicine, Volume 24, Number 3