The Completeness, Validity, and Timeliness of AIDS Surveillance Data

The Completeness, Validity, and Timeliness of AIDS Surveillance Data R. MONINA KLEVENS, DDS, MPH, PATRICIA L. FLEMING, PHD, JIANMIN LI, DPE, C. GREG GAINES, PHD, KATHLEEN GALLAGHER, DSC, MPH, SANDRA SCHWARCZ, MD, JOHN M. KARON, PHD, AND JOHN W. WARD, MD

PURPOSE: To assess the completeness, validity, and timeliness of the AIDS surveillance system after the 1993 change in the surveillance case definition. METHODS: To assess completeness of AIDS case reporting, three study sites conducted a comparison of their AIDS surveillance registries with an independent source of information. To evaluate validity, the same sites conducted record reviews on a sample of reported AIDS cases, we then compared agreement between the original report and the record review for sex, race, and mode of transmission. To evaluate timeliness, we calculated the median delay from time of diagnosis to case report, before and after the change in case definition, in each of the three study sites. RESULTS: After expansion of the case definition, completeness of AIDS case reporting in hospitals ( 93%) and outpatient settings ( 90%) was high. Agreement between the information provided on the original case report and the medical record was  98% for sex,  83% for each race/ethnicity group; and  67% for each risk group. The median reporting delay after the change was four months, but varied by site from three to six months. CONCLUSIONS: The completeness, validity, and timeliness of the AIDS surveillance system remains high after the 1993 change in the surveillance case definition. These findings might be useful for programs implementing integrated HIV and AIDS surveillance systems. Ann Epidemiol 2001;11:443–449. © 2001 Elsevier Science Inc. All rights reserved. KEY WORDS: Acquired Immunodeficiency Syndrome, Population Surveillance, Evaluation Studies, Reproducibility of Results.

INTRODUCTION Epidemiologic surveillance is the ongoing and systematic collection, analysis, and interpretation of health data (1). Surveillance is needed to identify health events that require public health action and to evaluate the effectiveness of prevention programs (1). Timely dissemination of the data is essential for public health officials to apply results to disease prevention (2). Periodic evaluation of surveillance systems ensures that they operate efficiently and that resources are allocated to only important public health problems (2). From the Centers for Disease Control and Prevention, National Center for HIV, STD, and TB Prevention, Division of HIV/AIDS Prevention, Atlanta, GA (R.M.K., P.L.F., J.M.K., J.W.W.); TRW, Atlanta, GA (J.L.); Louisiana Department of Health and Human Services, New Orleans, LA (C.G.G.); Massachusetts Department of Public Health, Boston, MA (K.G.); and San Francisco Department of Public Health, San Francisco, CA (S.S.). Address correspondences to: R. Monina Klevens, D.D.S., M.P.H., Centers for Disease Control and Prevention, National Center for HIV, STD, and TB Prevention, Division of HIV/AIDS Prevention, 1600 Clifton Road, MS E-47, Atlanta, GA 30333. Address reprint requests to: Office of Communications, Centers for Disease Control and Prevention, National Center for HIV, STD, and TB Prevention, Division of HIV/AIDS Prevention, 1600 Clifton Road, MS E-06, Atlanta, GA 30333. Received August 29, 2000; revised April 19, 2001; accepted April 23, 2001. © 2001 Elsevier Science Inc. All rights reserved. 655 Avenue of the Americas, New York, NY 10010

In the United States, surveillance for Acquired Immunodeficiency Syndrome (AIDS) has been ongoing since 1981. Data from AIDS surveillance have been used to describe the behavioral, demographic, and clinical characteristics of persons reported with AIDS. In addition to monitoring trends in the epidemic, surveillance information has been used to target prevention programs, estimate needs for HIV-related medical and social services (3) and determine the distribution of federal funds for prevention and treatment of HIV (4). Because the quality of AIDS surveillance data will determine the adequacy of these activities, the completeness, validity, and timeliness should be high. In 1993, in the U.S., the AIDS surveillance case definition was expanded to include CD4 T-lymphocyte counts of less than 200 cells/L or less than 14% of total lymphocytes and three additional clinical conditions (pulmonary tuberculosis, recurrent pneumonia, and invasive cervical cancer) among HIV-infected persons (5). The objective of this expansion was to enable surveillance to better reflect persons with severe HIV-related immunosuppression, that is, persons at greatest risk of HIV-related morbidity. Clinicians were already using measurement of CD4 T-lymphocytes to guide clinical management of HIV-infected persons. The 1993 revision had a major impact; in the first three months of 1993, there was a 204% increase in the number 1047-2797/01/$–see front matter PII S1047-2797(01)00256-3

444

Klevens et al. AIDS SURVEILLANCE EVALUATION

Selected Abbreviations and Acronyms AIDS  acquired immunodeficiency syndrome HIV  human immunodeficiency virus CDC  Centers for Disease Control and Prevention

of cases reported compared to the same period in 1992 (6). Even for cases meeting earlier case definitions, reporting increased 21% when compared to the same period in 1992 (7). Unlike previous definitions, the 1993 case definition allowed AIDS reporting based on a laboratory marker (CD4 count) that resulted in the reporting of persons at an earlier stage of their disease (8). The impact of this change on the validity and completeness of AIDS case reporting is unknown. Timeliness and completeness of case reports may be higher in states with laboratory-initiated reporting (9). Underreporting might occur because asymptomatic, HIV infected persons with severe immunosuppression may be more frequently diagnosed in outpatient settings, where traditionally, active case finding has been less emphasized. Prior to 1993, periodic evaluations of the U.S. AIDS surveillance system documented that reporting of AIDS was highly complete ( 80%) (10–12), and timely (about 55% of cases were reported within three months of diagnosis) (13). The study reported here is the first multisite study to evaluate the completeness, validity, and timeliness of data reported to U.S. AIDS surveillance after the 1993 expansion in the case definition. As more states move to integrating HIV and AIDS surveillance (14), the methods and findings from this study may be useful.

METHODS All 50 states and the District of Columbia (D.C.) report AIDS cases to CDC using a uniform surveillance case definition and case report form (15). The states and D.C. have public health regulations that require health care providers, health care facilities, laboratories, or others to confidentially report persons diagnosed with AIDS to designated health department officials. In addition, over 30 states require reports of persons infected with HIV who have not developed AIDS. HIV and AIDS activities combine passive and active reporting, that is, facilities and providers submit case reports, but health department personnel prompt reporting and conduct case finding activities using alternative sources such as death certificate reviews among others (15). CDC has published recommended program practices and performance standards to assist health departments in conducting HIV/AIDS surveillance (14). Three sites participated in this evaluation: Louisiana, Massachusetts, and San Francisco. They were selected by a competitive process in which CDC announced funds for evaluation activities, and many health departments ap-

AEP Vol. 11, No. 7 October 2001: 443–449

plied. The methods used for evaluating validity and timeliness were standardized in the three sites, and thus data were pooled; in contrast, methods to evaluate completeness were different in Massachusetts compared to Louisiana and San Francisco, because alternative datasets were more accessible in Massachusetts. Completeness of Reporting Traditionally, methods used in the evaluation of reporting completeness consist of a comparison of the site’s AIDS surveillance registry with an alternate source of data not routinely used for case finding. The two sources of data are then matched to determine the number of cases in both data sources. Completeness is calculated as the proportion of persons detected by the alternate source that were also reported to the AIDS surveillance registry (12). Although this is a simple approach, it is actually difficult to estimate the true number of cases of AIDS in a population (16). In epidemiology, capture-recapture methods are frequently used to estimate the extent of incomplete case count from a population, using information from overlapping lists of cases from independent sources. Capture-recapture methods can formally measure the degree of undercount in the individual sources of data (e.g., the AIDS surveillance registry) as well as estimate a total number of cases, i.e., the ascertainment corrected cases (17). Louisiana and San Francisco developed an alternate database for this evaluation by generating a list of health care facilities from telephone directories. Facilities were categorized into three types: hospitals, outpatient clinics, and private physicians’ offices. In both Louisiana and San Francisco, the number of facilities selected for follow-up was determined by the availability of human resources to conduct the field activities. In Louisiana, 32 hospitals (15% of all hospitals), nine private infectious disease specialists (15%), and 70 physicians (2%) from all other specialties were selected at random. For the San Francisco evaluation of completeness, a more detailed description, analysis, and interpretation is described elsewhere (18). Briefly, two hospitals (18%) were selected at random. For outpatient clinics and private physicians’ offices, a weight was assigned based on the number of AIDS cases they reported in the previous year. Thus, facilities were more likely to be selected if they had reported many cases. Four outpatient clinics (14%) and 12 private physicians specializing in either infectious diseases or internal medicine (14%) were selected at random. Within each sampled facility, Louisiana and San Francisco used electronic files and written logs of patients visiting the facilities in 1994 to identify persons most likely to be infected with HIV. To accomplish this, electronic searches for diagnoses of HIV infection, opportunistic illness associated with HIV infection, procedures related to treatment of HIV/AIDS and prescriptions for zidovudine

AEP Vol. 11, No. 7 October 2001: 443–449

were conducted. Then, medical records were manually reviewed to determine whether they met the AIDS surveillance case criteria. In San Francisco, after the first review of medical records of potential eligibles, an extra 5% random sample of the records of all patients seen in 1994 in the clinics and physicians offices was drawn, and these records were also reviewed. This was an attempt to decrease the risk of missing unreported cases. Completeness was calculated as the proportion of persons detected by the alternate source of patients that were reported to AIDS surveillance (12). In Massachusetts, the health department used their Uniform Hospital Discharge Data Set and their Medicaid claims data set to evaluate completeness. Details of the Massachusetts evaluation of completeness are described elsewhere (19). Subsets of these two databases were created using diagnosis, procedure and pharmacy codes that are frequent among persons with HIV infection. The AIDS surveillance database was then cross-matched to each of these databases. In addition, the hospital discharge and Medicaid datasets were matched to each other and then matched to the AIDS surveillance registry to determine the number of cases found in all three sources. Capture-recapture methodology was used to assess completeness of AIDS case reporting by first estimating the total number of cases using each individual two-source analysis (17). Once this number was estimated, completeness was calculated using the below equation. Completeness was calculated separately for each of the two alternate sources as well as the combined dataset (19). Total Number of AIDS Cases Reported to the AIDS Registry ----------------------------------------------------------------------------------------------------------------------------------------------------Ascertainment Corrected Number of Cases

Validity of Data Elements To evaluate the validity of basic data elements routinely collected through AIDS surveillance, a random sample of 200 adult cases reported to CDC from November 1, 1993 through October 30, 1994 was drawn for each of the three sites. An additional 100 cases for each site were sampled and provided in random order to allow replacement if needed. Staff from each of the participating sites went to the facility reporting the case, blinded to all information about the case sampled, except for name, date of birth, and date of case report. For each case, study staff completed a new case report form, abstracting information pre-dating the date of each case report, from the medical chart. For each case we compared the sex, race/ethnicity, and mode of transmission on the original case report with data collected in the record review. We measured the percent of cases for which the data on the original case report were confirmed on the record review. Exact confidence intervals for the percentages were calculated assuming a binomial distribution, using S-Plus version 3.3 (MathSoft, Seattle, WA); likelihood ratio tests for variation among the proportions were calculated using

Klevens et al. AIDS SURVEILLANCE EVALUATION

445

logistic regression (20) in SAS statistical software version 6.12 (SAS Institute Inc, Cary, NC). For each variable, an unweighted Cohen’s kappa statistic was calculated to assess the agreement beyond that which would be expected by chance alone (21). Timeliness To compare timeliness of reporting before and after the change in the surveillance case definition, we selected persons diagnosed with AIDS in 1991 and 1995 in each of the three participating areas. We chose these years to avoid the effects of surveillance activities conducted in preparation for the 1993 change, the activities related to implementing new software, and to catch up with the influx of newly reported cases. We calculated reporting delay as the time between the month and year of AIDS diagnosis and the month and year of report to CDC. We compared median delay for cases diagnosed in 1991 (before the change) and in 1995 (after the change) by site, reported through June 2000. We used the Wilcoxon rank-sum test to assess statistically significant differences (22).

RESULTS In Louisiana, data were received representing 488 unique individuals with AIDS in hospitals, 2262 in clinics, and 154 in private physician’s offices. Overall, 2865/2904 (99% complete) had been previously reported to surveillance; completeness was highest in private physicians’ offices where 153/154 (99% complete) cases were found. In hospitals and clinics, completeness was 98% (480/490) and 99% (2234/2262), respectively. In San Francisco, 810 records were reviewed in hospitals, 1902 records in outpatient clinics, and 531 records in private physicians’ offices. Overall, data from 1353 unique individuals with AIDS were matched with the AIDS database; 1285 (95% complete) had been previously reported and the remaining 68 were unreported cases. By facility type, completeness was 100% in hospitals, 99% in clinics (1879/1902), and 92% in private physicians’ offices (486/531). In Massachusetts, the estimated total number of cases was 8463 when using the hospital discharge data, 8292 when using the Medicaid data, and 8384 when using both data sets combined. A total of 7834 patients had been previously reported to the AIDS surveillance registry. Using these numbers we estimated completeness to be 93% (7834/8463) in the hospital setting and 95% in the Medicaid setting. Overall completeness was estimated as 93% (7834/8384). Validity In contrast to the methods described above for the evaluation of completeness, the methods for validity were stan-

446

Klevens et al. AIDS SURVEILLANCE EVALUATION

AEP Vol. 11, No. 7 October 2001: 443–449

dardized in the three sites. Louisiana followed up 179 case reports, Massachusetts 208, and San Francisco 171. Data from all three sites were combined because methodologies used and levels of agreement were similar across sites. Cases who were followed up were similar to other reported cases with regard to age and sex. However, despite random sampling, fewer of those followed up were white (51.9% and 60.6%; p  0.01) and fewer were men who have sex with men (41.8% and 58.1%; p  0.01) (Table 1). Table 2 describes the concordance between the data collected from the original case report and medical record review. Concordance was highest for sex ( 98%, kappa  0.974), followed by race/ethnicity ( 83%, kappa  0.908) and mode of exposure to HIV ( 67%, kappa  0.732). Levels of agreement between the original case report and record review were similar for the different categories of sex, but not race and exposure category. Concordance was lowest for the exposure categories of heterosexual contact and persons with dual risk male to male sex and injection drug use. We did not measure agreement for the 40 persons initially reported with no risk, since this risk category is an interim classification for which routine investigations are conducted to determine a mode of exposure. Timeliness Overall in the three sites, the delay in reporting of cases diagnosed in 1995 compared to cases diagnosed in 1991 deTABLE 1. Characteristics of persons with AIDS sampled for evaluation study and with follow-up, compared to those of all other persons with AIDS reported to CDC from November 1993 through October 1994

Characteristic Sex Male Female Race/ethnicity White Black Hispanic Asian/Pacific Islander American Indian/ Alaska Native Unknown Exposure category Men who have sex with men (MSM) Injection drug users (IDU) MSM and IDU Hemophilia Heterosexual contact Transfusion recipient No risk reported/other a

Persons with follow-up (N  543)

Others (N  4763)

p-valuea

87.7% 12.3%

88.5% 11.5%

0.57

51.9% 28.4% 14.7% 1.1%

60.6% 24.9% 12.2% 1.7%

0.2% 3.7%

0.3% 0.3%

0.001

41.8% 24.9% 8.1% 0.4% 6.5% 1.8% 16.5%

58.1% 19.4% 7.5% 0.5% 6.5% 1.1% 6.9%

0.001

p-values are for the chi-square of each corresponding contingency table.

creased from a median of six months to four months (p  0.0001). However, the pattern was different in each site. In California, there was a substantive decrease in the median delay for cases diagnosed in 1991 compared to those diagnosed in 1995, from 14 months to three months (p  0.0001). In Massachusetts, the decrease in delay was less: from seven months to six months (p  0.367). In Louisiana, there was a small increase in the median reporting delay, from two months to three months (p  0.003).

DISCUSSION We evaluated the completeness, validity, and timeliness of the AIDS surveillance system in three surveillance programs, after the 1993 change in the surveillance case definition. Reporting was highly complete, both in hospitals ( 93%) and in outpatient settings ( 90%). Agreement between demographic variables as originally reported and from follow-up abstraction of records was high, but less so for mode of transmission, which varied from 67% to 87%. Timeliness of reporting improved overall. After the 1987 change in case definition, there were three multisite evaluations. In one evaluation, researchers used hospital discharge datasets and found reporting was 92% complete (12). In another, the AIDS registry was compared with death certificates in different cities: completeness varied between 83–100% (23). For the remaining evaluation, a summary of findings from various cities and methods, completeness ranged from 62% (compared to a list of zidovudine recipients), to 98% (compared with hospital discharge data), concluding that completeness was higher in hospitals than in outpatient settings (11). Our evaluation indicates that since expansion of the case definition, the percentage of completeness remains high in hospitals and may have increased in outpatient settings. Our findings suggest that participating sites are meeting the performance standards that have currently been established for completeness ( 85%) (14). Completeness is high in hospitals because, since the beginning of the epidemic, health department staff have had close contact with hospital personnel to collect case reports. We cannot say for sure why we observed higher completeness from outpatient settings, but there may be a combination of factors involved. For example, surveillance programs conducted outreach activities to train providers about the change in the definition, and as a result of the awareness of potential underreporting from outpatient settings, surveillance programs reallocated resources to enhance reporting from these settings. Louisiana initiated direct laboratory reporting, and may have identified some unreported cases. Furthermore, there was new community awareness of the direct linkage between the number of cases reported in a state and Ryan White CARE funding for HIV treatment and prevention programs.

AEP Vol. 11, No. 7 October 2001: 443–449

Klevens et al. AIDS SURVEILLANCE EVALUATION

447

TABLE 2. Percent of cases originally reported that were validated, by sex, race, and mode of exposure to HIV Variable Sex Male Female Race White Black Hispanic Asian Mode of exposure to HIVa MSM IDU MSM  IDU Heterosexual Transfusion

Original

Validated

Percent agreement (95% CI)

Likelihood Ratio 2

Kappa statistic

467 67

465 66

99.6 (95.9,100.0) 98.5 (95.5,100.0)

0.9 (1 df)

0.974

287 151 87 6

273 148 76 5

95.1 (92.3,97.6) 98.0 (95.4,100.0) 87.4 (80.5,94.3) 83.3 (50.0,100.0)

12.2 (3 df)

0.908

255 143 44 43 8

212 124 30 29 7

83.1 (78.4,87.5) 86.7 (81.1,92.3) 68.2 (54.5,81.8) 67.4 (53.5,81.4) 87.5 (62.5,100.0)

13.9 (4 df)

0.732

a

Men who have sex with men (MSM), Injection drug users (IDU), Transfusion includes also recipients of blood transfusion, blood components, or tissue, hemophilia and coagulation disorders. Abbreviations: df, degrees of freedom; CI, confidence interval.

Surveillance for HIV and AIDS is, in many aspects, different than surveillance for other health events. The demand for accurate and timely data to characterize the AIDS epidemic, coupled with more resources than available for many other surveillance programs, may have led to high completeness of reporting. Evaluations of tuberculosis surveillance indicate reporting is highly complete, but less timely (24). Several other surveillance systems have similarly high levels of completeness: paralytic poliomyelitis (94% complete) (25), spinal cord injury (89% complete) (26), and surgical wound infections (84% complete) (27). Most surveillance systems in the United States have lower completeness: Neisseria meningitidis and Haemophilus influenzae (50% complete) (28), measles (29% complete) (29), and birth defects reporting (28% complete) (30), or are not evaluated. We also found that validity, as measured by level of agreement, was high for basic variables collected with each case report such as sex, race/ethnicity, and mode of exposure to HIV. Other studies have found that mode of exposure collected in AIDS surveillance is highly valid (31). Our findings are consistent with other studies (32, 33) that the validity of heterosexual exposure to HIV may be less than for other exposure categories. This may be due to the difficulty in documenting the HIV risk behaviors of the sex partners of heterosexual persons (31), of finding evidence that a person with AIDS had sexual contact with an HIVinfected person, or a problem of recall. Validity of exposure to HIV for persons reporting both male-to-male sex and injection drug use was also low; given that persons with multiple HIV risk behaviors can potentially bridge the transmission of HIV between groups of persons with other risk behaviors, the comprehensive assessment of HIV risk behaviors is important. For any exposure category, there is also the possibility that the person completing the initial case report form had additional information that was not

recorded in the medical chart. At an individual level, providers need risk information to counsel to reduce risk to uninfected partners. At the population level, risk is essential to target prevention activities. Ultimately, the validity of mode of transmission for surveillance depends on how well health care providers elicit and record patient risk behaviors in medical records. Our findings with regard to race are similar to those published previously; although concordance is good, it varied among racial and ethnic categories (34). We found that Hispanics were more frequently misclassified than persons of other racial/ethnic groups. Trends in the HIV/AIDS epidemic indicate increasing proportions of black and Hispanic persons (15). Findings from this study suggest that the proportion of Hispanics might be underestimated in surveillance data. There were decreases in reporting delays after the change in case definition in two of the three sites in the evaluation; in one of these, the decrease was dramatic. In the remaining site there was a small increase in reporting delay. This might be due to the implementation of laboratory reporting simultaneous with the change in the case definition in that site. It is possible that laboratory reporting identified previously unreported cases, which could create the longer reporting delay. The generalizability of findings from this evaluation is limited because only three sites participated in the study. In addition, persons followed up in the study were less likely to be white and men who have sex with men. However, because the three sites that participated in this evaluation conduct surveillance using different methods, the consistency of findings suggests some level of generalizability. For example, Louisiana has laboratory-initiated CD4 reporting and conducts HIV surveillance in addition to surveillance for AIDS. Massachusetts collects information on

448

Klevens et al. AIDS SURVEILLANCE EVALUATION

AIDS cases from relatively passive reports from health care providers, and San Francisco conducts active and continuous follow-up at most of the hospitals and clinics that treat HIV-infected persons. Other factors, unrelated to surveillance methods, create variations, such as human and other resources; however, certain basic methods are standard at AIDS reporting sites in the United States. All sites, for example, conduct active surveillance at their largest hospitals, and conduct reviews of death certificates. Therefore, methods and results from this study should be useful to many HIV/AIDS surveillance programs. In their evaluation of completeness, Louisiana and San Francisco relied on the voluntary participation of the private physicians sampled. Although this method has been used successfully in previous evaluations in Oregon (35) and New York City (10) it might overestimate completeness of case reporting if physicians who refused to participate are less likely to report AIDS cases to the health department. The findings from our study suggest that even if this were true, completeness of reporting in outpatient settings is high (90%) as demonstrated by matching with Medicaid claim files (Massachusetts) where this bias would not be present. In addition, routine review of death certificates by surveillance staff to find unreported cases results in only a small proportion of cases, suggesting that most are reported while patients are in health care. Massachusetts was able to access two valuable alternative data sources to evaluate completeness: the Massachusetts Uniform Hospital Discharge Data Set and the Medicaid claims dataset. They compared these using capture-recapture methods and made several assumptions, discussed in more detail by Jara and coworkers (19). One assumption is that the population used is closed such that the probability of being “captured” in any given data source is constant. To ensure this assumption was met, the data sets were limited in time to one year and to persons who were residents of the Commonwealth. Another assumption is that the probability of an individual being in any one source is equal for all individuals; this assumption was not met, but the impact is probably small. Finally, there is the assumption of independence of the data sources. Jara and coworkers measured the independence of the sources applying standard criteria and found a low likelihood of dependence (19). They concluded that this, perhaps the most important assumption, was met. Beyond the measured results of this evaluation, the methods used to assess completeness, validity, and timeliness might be useful, especially during implementation of HIV surveillance. As in many evaluations of surveillance systems, we used medical record reviews. One disadvantage of using medical records is that persons with HIV infection might seek health care services from more than one facility; thus, if records from all facilities are not reviewed, some information will be missed.

AEP Vol. 11, No. 7 October 2001: 443–449

Our results suggest that despite a major change in the AIDS surveillance case definition in 1993, AIDS surveillance remained complete and valid and that timeliness of case reporting improved. Recently, the CDC recommended that all state and territories conduct surveillance for HIV infection as an extension of current AIDS surveillance activities and that periodic evaluations of HIV and AIDS surveillance continue to be conducted (14). Methods used in this study could be helpful in evaluations of HIV surveillance. The authors thank W. Meade Morgan, Ph.D., for assistance with sampling and statistical guidance, and Robert L. Frey, Ph.D., for his work in the analysis of timeliness.

REFERENCES 1. Thacker SB. Historical Development. In: Teutsch SM, Churchill RE, eds. Principals and Practice of Public Health Surveillance. New York: Oxford University Press; 1994:3. 2. CDC Guidelines for evaluating surveillance systems. MMWR. 1988; 34: 1–18. 3. Buehler JW, Berkelman RL, Curran JW. Reporting of AIDS: Tracking HIV morbidity and mortality. JAMA. 1989;262:2896–2897. 4. Health Resources Services Administration. Bureau of Health Resources Development Division of HIV services. A compilation of the Ryan White CARE legislative act of 1990, as amended by the Ryan White CARE act amendments of 1996. Rockville, MD; 1996 5. Centers for Disease Control and Prevention. 1993 Revised classification system for HIV infection and expanded surveillance case definition for AIDS among adolescents and adults. Morb Mortal Wkly Rep. 1992;41(RR-17):1–19. 6. Centers for Disease Control and Prevention. Impact of the expanded AIDS surveillance case definition on AIDS case reporting. Morb Mortal Wkly Rep. 1993;42:308–310. 7. Centers for Disease Control and Prevention. Update: Impact of the expanded AIDS surveillance case definition for adolescents and adults on case reporting — United States, 1993. Morb Mortal Wkly Rep. 1994;43:160–161, 167–170. 8. Chang SW, Katz MH, Hernandez SR. The new AIDS case definition: Implications for San Francisco. JAMA 1992;267:973–975. 9. Klevens RM, Fleming PL, Li J, Karon J. Impact of laboratory-initiated reporting of CD4 T-lymphocytes on U.S. AIDS surveillance. J Acquir Immune Defic Syndr Hum Retrovirol. 1997;14:56–60. 10. Greenberg AE, Hindin R, Nicholas AG, Bryan EL, Thomas PA. The completeness of AIDS case reporting in New York City. JAMA. 1993; 269:2995–3001. 11. Buehler JW, Berkelman RL, Stehr-Green JK. The completeness of AIDS surveillance. J Acquir Immune Defic Syndr. 1992;5:257–264. 12. Rosenblum L, Buehler JW, Morgan MW, Casta S, Hidalgo J, Holmes R, et al. The completeness of AIDS case reporting, 1988: A multisite collaborative surveillance project. Am J Public Health. 1992; 82:1495–1499. 13. Centers for Disease Control and Prevention: HIV/AIDS Surveillance Report. Atlanta, GA: CDC; 1992:22. 14. Centers for Disease Control and Prevention. Guidelines for National Human Immunodeficiency Virus Case Surveillance, including monitoring for human immunodeficiency virus infection and acquired immunodeficiency syndrome. MMWR. 1999;48(RR-13):1–28. 15. Centers for Disease Control and Prevention: HIV/AIDS Surveillance Update. Atlanta, GA: CDC; 2000;1(1):4.

AEP Vol. 11, No. 7 October 2001: 443–449

16. Evans BG. Estimating underreporting of AIDS: Straightforward in theory — difficult in practice. AIDS. 1991;5:1261–1262. 17. Hook EB, Regal RR. Capture-recapture methods in epidemiology: Methods and limitations. Epidemiol Rev. 1995;17:243–264. 18. Schwarcz SK, Hsu LC, Parisi MK, Katz MH. The impact of the 1993 AIDS case definition on the completeness and timeliness of AIDS surveillance. AIDS. 1999;13(9):1109–1114. 19. Jara MM, Gallagher KM, Scheiman S. Estimation of Completeness of AIDS Case Reporting in Massachusetts. Epidemiology. 2000;11:209–213. 20. Hosmer DW, Lemeshow S. Applied Logistic Regression. New York: Wiley; 1989:14. 21. Cohen J. A coefficient of agreement for nominal scales. Ed Psychol Meas. 1960;20:37–46. 22. Rosner B. Fundamentals of Biostatistics. 4th ed. Belmont, CA: Duxbury Press; 1995; 562–566. 23. Hardy AM, Starcher ET, Morgan WM, Druker J, Kristal A, Day JM, et al. Review of death certificates to assess completeness of AIDS case reporting. Public Health Rep. 1987;102:386–391. 24. Curtis AB, McCray E, McKenna M, Onorato IM. Completeness and timeliness of tuberculosis case reporting. A multistate study. Am J Prev Med. 2001;20(2):108–112. 25. Prevots DR, Sutter RW, Strebel PM, Weibel RE, Cochi SL. Completeness of reporting for paralytic poliomyelitis, United States, 1980 through 1991. Implications for estimating the risk of vaccine-associated disease. Arch Pediatr Adolesc Med. 1994;148(5):479–485. 26. Thurman DJ, Burnett CL, Jeppson L, Beaudoin DE, Sniezek JE. Surveillance of spinal cord injuries in Utah, USA. Paraplegia 1994;32(10):665–669.

Klevens et al. AIDS SURVEILLANCE EVALUATION

449

27. Cardo DM, Galk PS, Mayhall CG. Validation of surgical wound surveillance. Infect Control Hosp Epidemiol. 1993;14(4):211–215. 28. Standaert SM, Lefkowitz LB, Horan JM, Hutcheson RH, Schaffner W. The reporting of communicable diseases: A controlled study of Neisseria meningitidis and Haemophilus influenzae infections. Clin Infect Dis. 1995;20(1):30–36. 29. Ewert DP, Westman S, Frederick PD, Waterman SH. Measles reporting completeness during a community-wide epidemic in inner-city Los Angeles. Public Health Rep. 1995;110(2):161–165. 30. Watkins ML, Edmonds L, McClearn A, Mullins L, Mulinare J, Khoury M. The surveillance of birth defects: The usefulness of the revised US standard birth certificate. Am J Public Health. 1996;86(5):731–734. 31. Klevens RM, Fleming PL, Neal JJ, Li J, and the MTV Study Group. Is there really a heterosexual AIDS epidemic in the United States? Findings from a multisite validation study, 1992–1995. Am J Epidemiol. 1999;149:75–84. 32. Serraino D, Francheschi S, Dal Maso L, Cozzi-Lepri A, Tirellio U, Rezza G. The classification of AIDS cases: Concordance between two AIDS surveillance systems in Italy. Am J Public Health. 1995; 85:1112–1114. 33. Murphy JT, Mueller GE, Whitman S. Redefining the growth of the heterosexual HIV/AIDS epidemic in Chicago. J Acquir Immune Defic Syndr Hum Retrovirol. 1997;16:122–126. 34. Kelley JJ, Chu S, Diaz T, Leary LS, Buehler JW. Race/ethnicity misclassification of persons reported with AIDS. Ethn Health. 1996;1(1):87– 94. 35. Modesitt SK, Hulman S, Fleming D. Evaluation of active versus passive AIDS surveillance in Oregon. Am J Public Health. 1990; 80:463–464.