- Email: [email protected]
Measuring the success of the U.S. childhood immunization system1
Commentary
Measuring the Success of the U.S. Childhood Immunization System Peter G. Szilagyi, MD, MPH
L
et’s play a pretend game. Pretend you were asked by the parents of preschool children in the United States to evaluate the success of this nation’s childhood immunization program. These parents have asked: “Is it working?” and “What works well?” and “What needs to be fixed?” How would you go about answering these questions? As a scientific thinker, you consider the methods to use. First, you realize that you are really being asked to evaluate the quality of care of a major public health program. You are familiar with the growing body of evidence about measuring quality of health care. Because you recognize the advantages of basing evaluations on strong conceptual frameworks, you select the classic health services research model proposed and revised by Donabedian,1 Starfield,2 and Brook et al.3 involving the assessment of the structure, process, and outcomes of care. You rephrase the parents’ questions as: “How successful are the structure of the U.S. immunization delivery system, the process of immunization delivery, and the outcomes of the program?” You also consider the quality of immunization delivery with respect to important subgroups, including children of different ages and from different socioeconomic levels, racial and ethnic backgrounds, and geographic regions. Furthermore, you recognize that vaccinations are not the only preventive health measure for children, so you consider the quality of the immunization delivery system with respect to overall pediatric preventive care. The next step is to define measurable indicators of structure, process, and outcomes with respect to the U.S. immunization program,4 and then accurately measure them. Because childhood immunization recommendations are changing constantly, and new vaccines are developed at ever-increasing rates, you recognize that a single “snapshot” will be insufficient to provide the most-useful knowledge; therefore, you must repeat From the Division of General Pediatrics, University of Rochester School of Medicine and Dentistry, Rochester, New York Address correspondence to: Peter Szilagyi, MD, MPH, Box 632, Strong Memorial Hospital, 601 Elmwood Avenue, Rochester, NY 14642. E-mail: [email protected]. Address reprint requests to: Centers for Disease Control and Prevention, National Immunization Program Resource Center, 1600 Clifton Road NE, Mailstop E-34, Atlanta, Georgia 30333. Fax: (404) 639-8828.
these measurements over time because quality of care can change. After much consultation with experts, you develop key indicators of quality. For structural measures, these include (but are not limited to): ● ●
Who provides vaccinations to children Financing, structural arrangements, and delivery systems
Important process measures include (but are not limited to): ●
● ● ●
Quality of immunization delivery for specific vaccines, including new vaccines (such as missed opportunities, and use of effective strategies to improve rates) Performance by various types of providers Efficiency of immunization delivery Record keeping, scattering of care, and continuity of care
Outcome measures are the most important. These include (but are not limited to) how well the U.S. immunization program has: ● ● ● ● ● ●
Prevented vaccine-preventable diseases Achieved high vaccination coverage rates Achieved success across the board, with no disparities across groups of children Led to improved receipt of overall preventive care among children Minimized adverse outcomes Provided lessons useful in evaluating other aspects of health care
Now obviously I have written this “pretend game” to make the point that in order to effectively evaluate the quality of a major national program such as the U.S. childhood immunization system, there must be a systematic mechanism to measure key aspects of the program. The National Immunization Survey (NIS) provides one such mechanism by systematically collecting data about the structure, process, and outcomes of the U.S. childhood immunization program. This supplement to the American Journal of Preventive Medicine showcases the NIS and highlights several articles that address important topics regarding quality of the immunization program.
Am J Prev Med 2001;20(4S) 0749-3797/01/$–see front matter © 2001 American Journal of Preventive Medicine • Published by Elsevier Science Inc. PII S0749-3797(01)00288-4
3
From these and related articles using data from the NIS, we have learned much about structural aspects of the program. For example, 85% of childhood vaccinations in the private sector are provided by pediatricians,5 and the majority of pediatricians and family physicians now participate in the Vaccines for Children (VFC) program6 (which has markedly reduced the cost of vaccinations for patients and providers). In addition, during the past several years there has been a major shift in immunization delivery, with greater numbers of children being vaccinated in the private sector, and fewer in the public sector.7 A major cause of this dramatic shift involves vaccine-financing reforms such as the VFC program and First Dollar Insurance laws7 (the latter, present in nearly half the states, requires insurers to cover childhood immunizations in their benefit packages). This shift in immunization delivery back to medical homes in the private sector presents important challenges for both the private and public sectors. Private providers are now faced with being largely responsible for (and increasingly accountable for) both administration of immunizations and other aspects of preventive care for children. This is a big challenge and the opportunities to incorporate strategies that are really effective8,9 have never been greater. The public sector still vaccinates many children, including disenfranchised and hard-to-reach patients, and is now also assuming newer functions including assessment of local vaccination rates, policy development, and assurance of immunization delivery.10 The NIS has also helped us better understand the process of immunization delivery. For example, a study in this supplement examined in detail the process of visits for vaccinations, and found that most children who were behind in vaccinations needed only one visit in order to become up-to-date—surely an attainable goal.11 Other studies using the NIS have focused on the scattering of immunization records.12 Every primary care provider understands this issue of record scattering and the difficulty of optimally managing patients with insufficient information about prior vaccinations. It is a worldwide problem, and leads to missed opportunities for vaccinations for undervaccinated patients, and unnecessary vaccinations in other situations.13 Some newer strategies such as computerized immunization registries and managed care databases may alleviate this problem. Future analyses of NIS will inform us about the degree to which record scattering occurs. Perhaps most importantly, the NIS provides invaluable data about the outcomes of our immunization program. National and state-specific data on vaccination coverage rates, published in annual surveillance tables such as those found in this supplement, have helped focus efforts, increase activities where needed, and maintain the momentum in our never-ending battle to achieve higher and higher vaccination rates. Coupled with exciting data about declining rates of 4
vaccine-preventable diseases,14 the rising national vaccination rates represent one of the great healthcare achievements of our time.15 But before we pat ourselves on the back too heartily, we must remember that other studies, including several in this supplement, highlight disparities in immunization coverage rates, with lower rates (see coverage tables in Luman et al.16) being found among children who are impoverished,17 from racial18 and ethnic minority groups, and from rural regions (for varicella vaccine19). Reducing such disparities has become a major component of the Healthy People 2010 national goals,20 and we should not rest until disparities in childhood healthcare measures are eliminated. The NIS is a window through which we can examine the delivery and receipt of preventive care for children. Childhood vaccination rates have been highly correlated with other measures of preventive care, and childhood immunizations represent an important marker for overall quality of pediatric care.21 Thus, improvements in vaccination rates are likely to reflect overall improvements in quality of primary care,22 and examination of problems in immunization delivery for certain groups of children sheds light on the overall quality of health care for these populations. The NIS is an important yardstick with which we can measure key aspects of the quality of our national immunization program. It has been said that one measure of the goodness of a society is how well it treats its children. Using childhood vaccinations as one measure, and the NIS as a yardstick, let’s keep working to treat all of our children well. This is not a pretend game, but a real marker of the goodness of our society.
References 1. Donabedian A. The definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration Press, 1980. 2. Starfield B. Health services research: a working model. N Engl J Med 1973;289:132– 6. 3. Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med 1996;335:966 –70. 4. Rodewald LE, Maes E, Stevenson J, Lyons B, Stokley S, Szilagyi PG. Immunization performance measurement in a changing immunization environment. Pediatrics 1999;103:889 –97. 5. Rodewald L, Maes E, Coronado V, et al. Medical home and provider specialty of preschool immunization providers: results from the 1997 National Immunization Survey. 32nd National Immunization Conference, Conference Abstracts. Washington, DC: U.S. Department of Health and Human Services, 1998:117. 6. Santoli JM, Rodewald LE, Maes EF, Battaglia MP, Coronado VG. Vaccines for Children program, United States, 1997. Pediatrics 1999;104:e15. 7. Szilagyi PG, Humiston SG, Shone LP, Kolasa MS, Rodewald LE. Decline in physician referrals to health department clinics for immunizations: the role of vaccine financing. Am J Prev Med 2000;18:318 –24. 8. Task Force on Community Preventive Services. Vaccine-preventable diseases: improving vaccination coverage in children, adolescents, and adults. MMWR Morb Mort Wkly Rep 1999;48:1–15. 9. Briss PA, Rodewald LE, Hinman AR, et al. Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med 2000;18(suppl 1):97–140. 10. Szilagyi PG, Humiston SG, Shone LP, Barth R, Kolasa MS, Rodewald LE. Impact of vaccine financing on vaccinations delivered by Health Department Clinics. Am J Public Health 2000;90:739 – 45.
American Journal of Preventive Medicine, Volume 20, Number 4S
11. Luman ET, Stokley S, Daniels D, Klevens RM. Vaccination visits in early childhood: just one more visit to be fully vaccinated. Am J Prev Med 2001;20(suppl 4):32– 40. 12. Stokley S, Rodewald LE, Maes EF. The impact of record scattering on the measurement of immunization coverage. Pediatrics 2001;107:91– 6. 13. Feikema SM, Klevens RM, Washington ML, Barker L. Extraimmunization among U.S. children. JAMA 2000;283:1311–7. 14. Centers for Disease Control and Prevention. Notifiable diseases/deaths in selected cities weekly information. MMWR Morb Mort Wkly Rep 1999;48: 1183–90. 15. Centers for Disease Control and Prevention. Ten great public health achievements—United States, 1900 –1999. MMWR Morb Mort Wkly Rep 1999;48:241–3. 16. Luman ET, Barker LE, Simpson DM, Rodewald LE, Szilagyi P, Zhao Z. National, state, and urban-area vaccination-coverage levels among children aged 19 –35 months, United States, 1999. Am J Prev Med 2001;20(suppl 4):88 –153. 17. Klevens RM, Luman ET. U.S. children living in and near poverty: risk of vaccine-preventable diseases. Am J Prev Med 2001;20(suppl 4):41– 6.
18. Daniels D, Jiles RB, Klevens RM, Herrera GA. Undervaccinated AfricanAmerican preschoolers: a case of missed opportunities. Am J Prev Med 2001;20(suppl 4):61– 8. 19. Stokley S, Smith PJ, Klevens RM, Battaglia MP. Vaccination status of children living in rural areas in the United States: Are they protected? Am J Prev Med 2001;20(suppl 4):55– 60. 20. U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion. Healthy People 2010, 2nd ed. With understanding and improving health and objectives for improving health, 2 vols. Washington DC: U.S. Department of Health and Human Services, 2000. Available at: http://www.health.gov/healthypeople/default.htm. Accessed March 18, 2000. 21. Rodewald LE, Szilagyi PG, Shiuh T, et al. Is underimmunization a marker for insufficient utilization of preventive and primary care? Arch Pediatr Adolesc Med 1995;149:393–7. 22. Rodewald LE, Szilagyi PG, Humiston SG, et al. The effects of an outreachbased recall system and a policy to reduce missed immunization opportunities on immunization coverage and primary care. Pediatrics 1999;103: 31– 8.
Am J Prev Med 2001;20(4S)
5
Measuring the Success of the U.S. Childhood Immunization System Peter G. Szilagyi, MD, MPH
L
et’s play a pretend game. Pretend you were asked by the parents of preschool children in the United States to evaluate the success of this nation’s childhood immunization program. These parents have asked: “Is it working?” and “What works well?” and “What needs to be fixed?” How would you go about answering these questions? As a scientific thinker, you consider the methods to use. First, you realize that you are really being asked to evaluate the quality of care of a major public health program. You are familiar with the growing body of evidence about measuring quality of health care. Because you recognize the advantages of basing evaluations on strong conceptual frameworks, you select the classic health services research model proposed and revised by Donabedian,1 Starfield,2 and Brook et al.3 involving the assessment of the structure, process, and outcomes of care. You rephrase the parents’ questions as: “How successful are the structure of the U.S. immunization delivery system, the process of immunization delivery, and the outcomes of the program?” You also consider the quality of immunization delivery with respect to important subgroups, including children of different ages and from different socioeconomic levels, racial and ethnic backgrounds, and geographic regions. Furthermore, you recognize that vaccinations are not the only preventive health measure for children, so you consider the quality of the immunization delivery system with respect to overall pediatric preventive care. The next step is to define measurable indicators of structure, process, and outcomes with respect to the U.S. immunization program,4 and then accurately measure them. Because childhood immunization recommendations are changing constantly, and new vaccines are developed at ever-increasing rates, you recognize that a single “snapshot” will be insufficient to provide the most-useful knowledge; therefore, you must repeat From the Division of General Pediatrics, University of Rochester School of Medicine and Dentistry, Rochester, New York Address correspondence to: Peter Szilagyi, MD, MPH, Box 632, Strong Memorial Hospital, 601 Elmwood Avenue, Rochester, NY 14642. E-mail: [email protected]. Address reprint requests to: Centers for Disease Control and Prevention, National Immunization Program Resource Center, 1600 Clifton Road NE, Mailstop E-34, Atlanta, Georgia 30333. Fax: (404) 639-8828.
these measurements over time because quality of care can change. After much consultation with experts, you develop key indicators of quality. For structural measures, these include (but are not limited to): ● ●
Who provides vaccinations to children Financing, structural arrangements, and delivery systems
Important process measures include (but are not limited to): ●
● ● ●
Quality of immunization delivery for specific vaccines, including new vaccines (such as missed opportunities, and use of effective strategies to improve rates) Performance by various types of providers Efficiency of immunization delivery Record keeping, scattering of care, and continuity of care
Outcome measures are the most important. These include (but are not limited to) how well the U.S. immunization program has: ● ● ● ● ● ●
Prevented vaccine-preventable diseases Achieved high vaccination coverage rates Achieved success across the board, with no disparities across groups of children Led to improved receipt of overall preventive care among children Minimized adverse outcomes Provided lessons useful in evaluating other aspects of health care
Now obviously I have written this “pretend game” to make the point that in order to effectively evaluate the quality of a major national program such as the U.S. childhood immunization system, there must be a systematic mechanism to measure key aspects of the program. The National Immunization Survey (NIS) provides one such mechanism by systematically collecting data about the structure, process, and outcomes of the U.S. childhood immunization program. This supplement to the American Journal of Preventive Medicine showcases the NIS and highlights several articles that address important topics regarding quality of the immunization program.
Am J Prev Med 2001;20(4S) 0749-3797/01/$–see front matter © 2001 American Journal of Preventive Medicine • Published by Elsevier Science Inc. PII S0749-3797(01)00288-4
3
From these and related articles using data from the NIS, we have learned much about structural aspects of the program. For example, 85% of childhood vaccinations in the private sector are provided by pediatricians,5 and the majority of pediatricians and family physicians now participate in the Vaccines for Children (VFC) program6 (which has markedly reduced the cost of vaccinations for patients and providers). In addition, during the past several years there has been a major shift in immunization delivery, with greater numbers of children being vaccinated in the private sector, and fewer in the public sector.7 A major cause of this dramatic shift involves vaccine-financing reforms such as the VFC program and First Dollar Insurance laws7 (the latter, present in nearly half the states, requires insurers to cover childhood immunizations in their benefit packages). This shift in immunization delivery back to medical homes in the private sector presents important challenges for both the private and public sectors. Private providers are now faced with being largely responsible for (and increasingly accountable for) both administration of immunizations and other aspects of preventive care for children. This is a big challenge and the opportunities to incorporate strategies that are really effective8,9 have never been greater. The public sector still vaccinates many children, including disenfranchised and hard-to-reach patients, and is now also assuming newer functions including assessment of local vaccination rates, policy development, and assurance of immunization delivery.10 The NIS has also helped us better understand the process of immunization delivery. For example, a study in this supplement examined in detail the process of visits for vaccinations, and found that most children who were behind in vaccinations needed only one visit in order to become up-to-date—surely an attainable goal.11 Other studies using the NIS have focused on the scattering of immunization records.12 Every primary care provider understands this issue of record scattering and the difficulty of optimally managing patients with insufficient information about prior vaccinations. It is a worldwide problem, and leads to missed opportunities for vaccinations for undervaccinated patients, and unnecessary vaccinations in other situations.13 Some newer strategies such as computerized immunization registries and managed care databases may alleviate this problem. Future analyses of NIS will inform us about the degree to which record scattering occurs. Perhaps most importantly, the NIS provides invaluable data about the outcomes of our immunization program. National and state-specific data on vaccination coverage rates, published in annual surveillance tables such as those found in this supplement, have helped focus efforts, increase activities where needed, and maintain the momentum in our never-ending battle to achieve higher and higher vaccination rates. Coupled with exciting data about declining rates of 4
vaccine-preventable diseases,14 the rising national vaccination rates represent one of the great healthcare achievements of our time.15 But before we pat ourselves on the back too heartily, we must remember that other studies, including several in this supplement, highlight disparities in immunization coverage rates, with lower rates (see coverage tables in Luman et al.16) being found among children who are impoverished,17 from racial18 and ethnic minority groups, and from rural regions (for varicella vaccine19). Reducing such disparities has become a major component of the Healthy People 2010 national goals,20 and we should not rest until disparities in childhood healthcare measures are eliminated. The NIS is a window through which we can examine the delivery and receipt of preventive care for children. Childhood vaccination rates have been highly correlated with other measures of preventive care, and childhood immunizations represent an important marker for overall quality of pediatric care.21 Thus, improvements in vaccination rates are likely to reflect overall improvements in quality of primary care,22 and examination of problems in immunization delivery for certain groups of children sheds light on the overall quality of health care for these populations. The NIS is an important yardstick with which we can measure key aspects of the quality of our national immunization program. It has been said that one measure of the goodness of a society is how well it treats its children. Using childhood vaccinations as one measure, and the NIS as a yardstick, let’s keep working to treat all of our children well. This is not a pretend game, but a real marker of the goodness of our society.
References 1. Donabedian A. The definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration Press, 1980. 2. Starfield B. Health services research: a working model. N Engl J Med 1973;289:132– 6. 3. Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care. N Engl J Med 1996;335:966 –70. 4. Rodewald LE, Maes E, Stevenson J, Lyons B, Stokley S, Szilagyi PG. Immunization performance measurement in a changing immunization environment. Pediatrics 1999;103:889 –97. 5. Rodewald L, Maes E, Coronado V, et al. Medical home and provider specialty of preschool immunization providers: results from the 1997 National Immunization Survey. 32nd National Immunization Conference, Conference Abstracts. Washington, DC: U.S. Department of Health and Human Services, 1998:117. 6. Santoli JM, Rodewald LE, Maes EF, Battaglia MP, Coronado VG. Vaccines for Children program, United States, 1997. Pediatrics 1999;104:e15. 7. Szilagyi PG, Humiston SG, Shone LP, Kolasa MS, Rodewald LE. Decline in physician referrals to health department clinics for immunizations: the role of vaccine financing. Am J Prev Med 2000;18:318 –24. 8. Task Force on Community Preventive Services. Vaccine-preventable diseases: improving vaccination coverage in children, adolescents, and adults. MMWR Morb Mort Wkly Rep 1999;48:1–15. 9. Briss PA, Rodewald LE, Hinman AR, et al. Reviews of evidence regarding interventions to improve vaccination coverage in children, adolescents, and adults. Am J Prev Med 2000;18(suppl 1):97–140. 10. Szilagyi PG, Humiston SG, Shone LP, Barth R, Kolasa MS, Rodewald LE. Impact of vaccine financing on vaccinations delivered by Health Department Clinics. Am J Public Health 2000;90:739 – 45.
American Journal of Preventive Medicine, Volume 20, Number 4S
11. Luman ET, Stokley S, Daniels D, Klevens RM. Vaccination visits in early childhood: just one more visit to be fully vaccinated. Am J Prev Med 2001;20(suppl 4):32– 40. 12. Stokley S, Rodewald LE, Maes EF. The impact of record scattering on the measurement of immunization coverage. Pediatrics 2001;107:91– 6. 13. Feikema SM, Klevens RM, Washington ML, Barker L. Extraimmunization among U.S. children. JAMA 2000;283:1311–7. 14. Centers for Disease Control and Prevention. Notifiable diseases/deaths in selected cities weekly information. MMWR Morb Mort Wkly Rep 1999;48: 1183–90. 15. Centers for Disease Control and Prevention. Ten great public health achievements—United States, 1900 –1999. MMWR Morb Mort Wkly Rep 1999;48:241–3. 16. Luman ET, Barker LE, Simpson DM, Rodewald LE, Szilagyi P, Zhao Z. National, state, and urban-area vaccination-coverage levels among children aged 19 –35 months, United States, 1999. Am J Prev Med 2001;20(suppl 4):88 –153. 17. Klevens RM, Luman ET. U.S. children living in and near poverty: risk of vaccine-preventable diseases. Am J Prev Med 2001;20(suppl 4):41– 6.
18. Daniels D, Jiles RB, Klevens RM, Herrera GA. Undervaccinated AfricanAmerican preschoolers: a case of missed opportunities. Am J Prev Med 2001;20(suppl 4):61– 8. 19. Stokley S, Smith PJ, Klevens RM, Battaglia MP. Vaccination status of children living in rural areas in the United States: Are they protected? Am J Prev Med 2001;20(suppl 4):55– 60. 20. U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion. Healthy People 2010, 2nd ed. With understanding and improving health and objectives for improving health, 2 vols. Washington DC: U.S. Department of Health and Human Services, 2000. Available at: http://www.health.gov/healthypeople/default.htm. Accessed March 18, 2000. 21. Rodewald LE, Szilagyi PG, Shiuh T, et al. Is underimmunization a marker for insufficient utilization of preventive and primary care? Arch Pediatr Adolesc Med 1995;149:393–7. 22. Rodewald LE, Szilagyi PG, Humiston SG, et al. The effects of an outreachbased recall system and a policy to reduce missed immunization opportunities on immunization coverage and primary care. Pediatrics 1999;103: 31– 8.
Am J Prev Med 2001;20(4S)
5