A comparison of multiple data sources to identify vaccinations for veterans with spinal cord injuries and disorders

Journal of the American Medical Informatics Association

Case Report

Volume 11

Number 5 Sep / Oct 2004

377

j

A Comparison of Multiple Data Sources to Identify Vaccinations for Veterans with Spinal Cord Injuries and Disorders FRANCES M. WEAVER, PHD, MICHAEL HATZAKIS, MD, PHD, CHARLESNIKA T. EVANS, MPH, BRIDGET SMITH, MPA, SHERRI L. LAVELA, MPH, MBA, CAROLYN WALLACE, PHD, MARCIA W. LEGRO, PHD, BARRY GOLDSTEIN, MD, PHD A b s t r a c t Monitoring vaccination activity requires regular access to information about patient vaccination status. This report describes our experience using multiple Department of Veterans Affairs (VA) data sources to determine availability and completeness of vaccination information for veterans with spinal cord injuries and disorders (SCI&D). Administrative and clinical databases were limited to coding vaccine administration, undercounted vaccinations, and were unable to account for whether the vaccine was offered and the reasons for nonreceipt. Medical record review provided more detail but was labor intensive and costly. Patient surveys provided the richest information but were costly, time-consuming, and based on a sample of patients. Agreement was poor between data sources. This report suggests that while VA is well positioned to use national databases for clinical care decisions and to inform policy, vaccination data were incomplete. Electronic records must include data that are consistently entered and validated before they can be useful for care management and decision making. j

J Am Med Inform Assoc. 2004;11:377–379. DOI 10.1197/jamia.M1516.

Annual influenza vaccination and a single pneumococcal polysaccharide vaccination (PPV) are considered best practice for the prevention of influenza and pneumonia and resulting respiratory complications for identified high-risk groups.1–4 The proportion of patients vaccinated is often used as a quality-of-care indicator. Therefore, it is important that this infor-

Affiliations of the authors: Spinal Cord Injury Quality Enhancement Research Initiative, Midwest Center for Health Services and Policy Research, Hines Veterans Administration Hospital, Hines, IL, and Institute for Health Services and Policy Research, Northwestern University, Chicago, IL (FMW); Departments of Rehabilitation Medicine and Biomedical and Health Informatics, University of Washington School of Medicine, and Rehabilitation Care Services, Puget Sound VA Health Care System, Seattle, WA (MH); Hines VA Hospital, Hines, IL (CTE, BS, SLL); Puget Sound VA Health Care System, Seattle, WA (CW, MWL); and Spinal Cord Injury Quality Enhancement Research Initiative, Spinal Cord Injury and Disorders Strategic Healthcare Group, Puget Sound VA Health Care System, and Department of Rehabilitation Medicine, University of Washington, Seattle, WA (BG). Funded by the Department of Veterans Affairs Quality Enhancement Research Initiative (QUERI) Program within the Health Services Research and Development (HSR&D) Service, Washington, DC (SCI 98-001; SCT 01-169). The authors thank Scott Miskevics, BS, and Jibby Chittet, MPH, for their contributions to this report. The views presented in the paper are those of the authors and do not necessarily reflect the views of the Department of Veterans Affairs. Correspondence and reprints: Frances M. Weaver, PhD, Midwest Center for Health Services and Policy Research, 151H, VA Hospital, Hines, IL 60141; e-mail: . Received for publication: 12/15/03; accepted for publication: 04/04/04.

mation is accurate and easily accessible to monitor performance. Vaccinating patients is a clinical care process that involves offering and administering the vaccine, documentation of care, and the ability to both audit and provide feedback regarding vaccine activities. Ideally, documentation will indicate whether the patient was vaccinated, when and where, and the reason for nonvaccination. A computerized medical record system allows storage of the results of an encounter in a standardized format. It has been demonstrated that practitioners who use an electronic medical record system to document clinical activities produced more complete information than those who used paper records.5

Case Description People with spinal cord injuries and disorders (SCI&D) are at high risk of respiratory complications because injury often results in weakened respiratory muscles, impaired cough, and less effective clearance of secretions.6,7 It has been shown that persons with SCI&D who contracted influenza or pneumonia were 37 times more likely to die of influenza or pneumonia than comparable individuals from the general population.8 Respiratory vaccinations in this population are considered best practice. Data from Department of Veterans Affairs (VA)’s External Peer Review Program (EPRP) indicate very low influenza vaccination and PPV rates among veterans with SCI&D ages 65 years and older. Rates in 1998 were 26% for influenza vaccination and 25% for PPV, while rates in the general veteran population were much higher: 71% for influenza and 73% for PPV.9 The VA SCI Quality Enhancement Research Initiative (SCI QUERI) implemented strategies to increase the number of veterans with SCI&D who receive yearly influenza vaccines

378

WEAVER ET AL., Comparing Data Sources for Vaccinations

and a single PPV. To assess the impact of our efforts, we identified data sources that provide documentation of vaccine activities. This report reviews our experience with each of these data sources and compares them with patient self-report of vaccination status.

Method This project involved examination of multiple data sources to identify documentation of vaccine delivery for veterans with SCI&D for two years [fiscal years (FY) 2000 and 2001]. Sources included national-level administrative and clinical databases, medical record review, and patient surveys. In this case, patient self-report was considered the gold standard.

National Data Sources We identified two national sources for vaccine activity data. Administrative data available through VA’s computerized National Patient Care Database (NPCD) contain information on all episodes of care provided in VA facilities.10 We examined several diagnostic and procedure codes related to vaccine delivery using the NPCD. [Access to NPCD is available to VA employees by obtaining a time-sharing user account on VA’s mainframe computer (http://www.virec.research. med.va.gov/databases/access_and_training.htm). An individual must submit a request for a user account through their local VA facility point of contact. Access to VA data by non-VA employees is provided in accordance with multiple regulations (see Web site). Influenza vaccine codes included ICD-9 codes of 99.52 (vaccination against influenza), V04.8 (need for vaccination against influenza), and CPT codes 90657-9 (influenza vaccines) and 90471-2 (vaccine, illness not specified). PPV codes included ICD-9 codes 99.55 (PPV), V06.6, V06.8, V03.82 (vaccination against pneumonia and influenza) and CPT code 90732 (vaccines against diseases including pneumonia).] The numbers of influenza vaccinations documented for 2000 and 2001 were low (15.5% and 18.5%, respectively), and number of PPVs documented was extremely low (,4%) across years. A second source of national information was vital statistics data available from clinical databases. (Vitals data may be obtained through a written request to the Diabetes QUERI and the Health Care Analysis and Information Group. Similar to the request for EPRP data, the investigator must submit a signed data release agreement, a copy of the study proposal, and approval from the human studies institutional review board.) Local VA facility data on vital statistics were extracted from the VA Computerized Patient Record System as part of the data collection requirements of the Diabetes QUERI to build and maintain a cohort of veterans with diabetes. All veterans, not just those with diabetes, were included. Several variables including dates of influenza vaccination and PPV were extracted. SCI QUERI requested data for two years (FY 2000 and 2001) for veterans with SCI&D. Influenza vaccinations were reported for 17% and 18% of cases in 2000 and 2001, respectively. PPV dates were noted for 21% of veterans in 2000 and 25% in 2001. Upon closer examination, we identified several limitations to these data. First, no data were available for two of the 23 VA SCI centers in 2000, and in 2001, data were missing for four centers. Second, 20% of patients in 2000 and 21% of patients in 2001 had two different dates listed for influenza vaccination in the same year, suggesting either that these data were entered twice or that some patients received

the vaccine twice in one year. For PPV, some cases did not have dates of receipt listed (1.1% in 2000 and 2.4% in 2001). These discrepancies raise questions about data completeness and accuracy.

Medical Record Review Another source of vaccination information was available through the VA’s Office of Quality and Performance.* Data are collected by trained nurse data abstractors who travel to each VA facility to review a sample of medical records to gather information on several performance measures including vaccinations. Using chart review, the abstractor can identify whether a veteran received a vaccine, regardless of location. The rates for influenza vaccination for 2000 and 2001 were low (28% and 33%); PPV rates were somewhat better at 40% in 2000 and 50% in 2001. When these rates are compared with the administrative data, which only count when a vaccine is given in VA, the proportion of veterans who received vaccines was higher using chart reviews for the same time periods.

Patient Surveys For the past three years, we have surveyed veterans with SCI&D about influenza vaccine receipt. Response rates to the surveys were good across years (72%–90%). Self-reported rates of influenza vaccination were 65% in 2000 and 57% in 2001. PPV self-reported rates were only accessed in 2001; 59% indicated that they had received the vaccine. Limitations to using surveys to gather vaccine data include the resources required to gather these data (labor, supplies, postage and/or phone charges). In addition, surveys are based on patient recall, which may be affected by a number of factors including confusion as to which vaccine was received or when or where the vaccine was received. Nonetheless, we used patient self-report as our gold standard for comparison with other sources of vaccine information.

Example We compared these different data sources for vaccination documentation during FY 2001. Vaccination rates documented between the administrative data (NPCD) and the vitals data were low (, 25%; Table 1). In the large majority of cases, vaccination was not documented in either database. In less than 12% of cases, an influenza vaccine was documented in both places, and for PPV, documentation was less than 2%. Documentation was more likely to occur in the vital statistics database than the administrative database. Agreement between these databases for influenza vaccine, using kappa statistics, was moderate (0.47). For PPVs, the level of agreement was poor (kappa coefficient = 0.06). We also performed McNemar’s test, which indicated a significant level of disagreement (p , 0.001) between the databases for documentation of both influenza vaccinations and PPVs. The second comparison between patient survey and administrative data revealed that a higher percentage of patients reported having received a vaccine than was documented in the NPCD (Table 2). For 15% of those receiving the *EPRP data are available through a data use agreement. Recent information is available on the VA Intranet site (http://vaww. oqp.med.va.gov/default.htm). Investigators must submit a signed data use agreement, a research protocol, institutional review board approval, and a list of EPRP variables of interest. Limits include data only available for a sample of patients; sample varies by year.

Journal of the American Medical Informatics Association

Volume 11

Table 1 j Comparison of NPCD to Vitals Data for Documentation of Influenza and PPV Vaccination in 2001 (N = 12,424)

379

Number 5 Sep / Oct 2004

Table 2 j Comparison of NPCD to Patient Self-report Survey for Influenza Vaccination and PPV in 2001 (N = 1,440)

NPCD Vitals Data Influenza Reported Not reported PPV Reported Not reported

NCPD

Documented

Not Documented

1,446 (11.6%) 486 (3.9%)

1,712 (13.8%) 8,780 (70.7%)

183 (1.5%) 54 (0.4%)

3,687 (29.7%) 8,500 (68.4%)

Patient Survey Influenza Reported Not reported PPV Reported Not reported

Documented

Not Documented

216 (15.0%) 21 (1.5%)

696 (48.3%) 507 (35.2%)

30 (2.1%) 7 (0.5%)

836 (59.0%) 543 (38.3%)

Kappa statistics: influenza = 0.47; PPV = 0.06. NPCD = National Patient Care Database; PPV = Polysaccharide vaccination.

Kappa statistics: Influenza = 0.16; PPV = 0.02. NPCD = National Patient Care Database; PPV = Polysaccharide vaccination.

influenza vaccine, there was agreement between the patient and the administrative data that the patient received the vaccine. However, the agreement was only 2.1% for those receiving the PPV. The kappa coefficients for receipt of either influenza vaccine (0.16) or PPV (0.02) and McNemar’s test (p , 0.0001) suggest poor overall agreement between databases. Receipt of a vaccine was rarely documented in the administrative database if the patient did not report receiving the vaccine. However, it should be noted that approximately 75% of the time when patients reported that they had received the influenza vaccination, they said they received it from a VA facility. Thus, despite having received it at a VA facility, this information was not getting into the electronic medical record as a procedure code. However, there are several other places within the electronic medical record that vaccine activity may be recorded; it is possible that a thorough search of the record would find evidence of vaccination.

process. The VA is in a good position to implement these changes because the medical record system is completely computerized. CCRs are used, mechanisms exist to link information across facilities, and providers are familiar with computerized entry and retrieval of medical record information. In multiple areas, including diabetes care and respiratory vaccinations, the VA has exceeded the benchmarks of quality care established in non-VA health systems. These achievements are evidence that VA can create a well-integrated clinical information system to manage individuals with chronic diseases. However, local system variability makes the process more difficult. Increased standardization and the ability to easily search across data collection modules are needed to ensure accurate capture of care delivery. As other organizations move toward electronic medical records, the need for standardization and integration of data elements cannot be overemphasized.

Data from EPRP were not available at the patient-identifiable level for 2001. However, the self-reported rate of 62% from the patient survey is the same as the EPRP influenza vaccination rate of 62% identified in a review of a sample medical records during 2001. This suggests that both sources would provide similar findings regarding vaccination rates.

References

Discussion Although there are multiple methods to track vaccine activity on an organizational level, the most cost-efficient mechanism is to track vaccine activity through the use of existing information technology infrastructures including the electronic medical record. The limitation of this approach is that the validity of audits on vaccine delivery activity will only be as good as the data that are entered in the computerized record. While it appears that vaccine activity is likely to be recorded somewhere in the medical record, a specific retrievable procedure code for vaccination may not. Use of computerized clinical reminders (CCRs) or encounter templates on a system-wide basis may yield the more accurate results but requires significant organizational unity to agree to use a single data entry mechanism or set of data entry standards. CCRs allow standardized documentation of vaccine delivery, recording of reasons why vaccines are not delivered, receipt of vaccines outside the VA, and refusal of vaccines. CCRs also provide a mechanism for data to be used for audit and feedback about a clinician, facility, or national performance. Organizational changes are required to facilitate accurate and complete capture of all data elements relevant to the vaccination

j

1. CDC-ACIP. Prevention and control of influenza: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Recomm Rep. 2003;52(RR08):1–36. 2. CDC-ACIP. Prevention of pneumococcal disease: recommendations of the Advisory Committee on Immunization Practices (ACIP). MMWR Recomm Rep. 1997;46(RR-8):1–25. 3. Nordin J, Mulloohy J, Poblete S, et al. Influenza vaccine effectiveness in preventing hospitalizations and deaths in persons 65 years or older in Minnesota, New York and Oregon: data from 3 health plans. J Infect Dis. 2001;184:665–70. 4. Waites KB, Canupp KC, Edwards K, et al. Immunogenicity of pneumococcal vaccine in persons with spinal cord injury. Arch Phys Med Rehabil. 1998;79:1504–9. 5. De Lusignan S, Stephens PN, Adal N, Majeed A. Does feedback improve the quality of computerized medical records in primary care? J Am Med Inform Assoc. 2002;9:395–401. 6. Wilmot CB, Hall KM. The respiratory system. In: Whiteneck GG, Charlifue SW, Gerhart KA, et al. (eds). Aging with Spinal Cord Injury. New York, NY: Demos Publications; 1993, pp 93–104. 7. DeVivo MJ, Krause JS, Lammertse DP. Recent trends in mortality and causes of death among persons with spinal cord injury. Arch Phys Med Rehabil. 1999;80:1411–9. 8. DeVivo MJ, Black KJ, Stover SL. Causes of death during the first 12 years after spinal cord injury. Arch Phys Med Rehabil. 1993; 74:248–54. 9. Veterans Health Administration. Spinal Cord Injury Study Performance Report, March 1999. Washington DC. Office of Quality and Performance, Department of Veterans Affairs, 1999. 10. Murphy PA, Cowper DC, Seppala G, Stroupe KT, Hynes DM. Veterans Health Administration inpatient and outpatient care data: an overview. Eff Clin Pract. 2002;5(3 suppl):E4.