Use of insurance claims databases to evaluate the outcomes of ophthalmic surgery

SURVEY OF OPHTHALMOLOGY

VOLUME

42.

NUMBER

3

l

NOVEMBER-DECEMBER

1997

ELSEVIER

PUBLIC HEALTH AND THE EYE DONALD

FONG AND JOHANNA

SEDDON,

Use of Insurance Claims Outcomes of Ophthalmic ANNE L. COLEMAN,

EDITORS

Databases Surgery

MD, PHD,’ and HAL MORGENSTERN,

‘Department of Ophthalmology, School of Medicine and 2Depatiment University of Calqomia, Los Angeles, Calijornia, USA

to Evaluate

the

PHD*

of Epidemiology,

School of Public Health,

Abstract. This article reviews methodological

issues in research using Medicare claims data and reviews how those issues affect the interpretation of study results. Although studies using Medicare claims data can improve our knowledge of surgical outcomes, the ability to infer that one type of surgery or one surgeon is better than another is limited by such factors as bias due to inaccurate record keeping, confounding of treatment effects by other covariates, effect modification due to factors associated with treatment, and selective loss of follow-up of patients. Five studies on the outcomes of cataract surgery and capsulotomy’.“-‘4 are reviewed, providing illustrations of methodological strengths and weaknesses of this type of research. (Surv Ophthalmol42:271-278, 199’7. 0 1997by Elsevier Science Inc. All rights reserved.)

Key words. claims data

l

Medicare

l

outcomes analysis

Although the use of existing databases to evaluate clinical ophthalmic outcomes is an important area of current research, there are important re?sons to scrutinize the interpretation of results from such studies. Large databases offer the potential to learn about successand complication rates associated with various ophthalmic procedures by pooling evidence from many patients, but it is easy to imagine such databases being used to argue, for example, that certain clinicians should not practice medicine based on above-average complication rates. With potential risks and rewards in mind, we consider a number of inherent methodological issuessurrounding the use of existing databaseswhen making causal inferences. Databases collected for billing purposes (insurance claims databases) constitute the major source of information for this new line of research. Many of

these databases,which are maintained by insurance companies and health maintenance organizations, are not publicly available. The Health Care Finance Administration, which maintains records on Medicare billing, does make public-use tapes available. These tapes have served as the basisfor many of the studies evaluating ophthalmic surgery outcomes from existing claims data.*,“-” There

are several

types of administrative

databases

maintained by the Health Care Finance Administration, such as Medical Provider Analysis and Review, Part B Hospital Outpatient Facility file (outpatient file), Part B Medicare Annual Data Beneficiary file, and Physician/Supplier Part B. The Medical Provider Analysis and Review file contains 100% of Medicare beneficiaries using hospital inpatient services, while the outpatient file contains 100% of Medicare benefi-

271 0039~6”.57/97/$17.00

PII s0039-6257(97)00095-7

272

Surv Ophthalmo142 (3) November-December 199’7

ciaries using hospital outpatient services. The Part B Medicare Annual Data Beneficiary file contains a 5% sample of physician and ambulatory surgical center bills for Medicare beneficiaries and wasavailable until 1990. Since 1990, the Physician/Supplier Part B file has contained a 5% sample of physician and ambulatory surgical center bills. A 100% sample of physician and ambulatory surgical center bills is not available through the public-use files, becausethe file size is too large, according to the Health Care Finance Administration. Thus, most published studies using Medicare claims data have accessto the records of 100% of the casesof hospital inpatient and outpatient servicesand 5% of the casesof ambulatory surgery centers. TABLE Synojxis ofFive Studies Using Medicare

COLEMAN

Review

AND MORGENSTERN

of Five Studies Using Claims Data

Medicare

We will use five studies on the outcomes of cataract surgery and capsulotom~“-‘4 to show the strengths and limitations of using large, available databasesto assess the quality of care (Table I). These five studies were chosen because they were among the first pub lished studies using Medicare claims in ophthalmology and because they utilize well-established methods in health services research. Three of the studies compared the risk of retinal detachment,‘” endophthalmitis,14cornea1 edema, and corneal transplantation* among eyes that have had extracapsular cataract exI

Claims Data to Evaluate

the Outcomes of Cataract Surgery

No. of Cases Database Source (year) reference

Type of Surgery

(No. of Procedures)

Study 1’”

MEDPAR (1984)I3

ICCE ECCE PHACO ANTVIT

1,366(99,971) 1,481(195,587) 277 (28,474) 165 (3,634)

Study 214 MEDPAR (1984)r4

ICCE ECCE PHACO ANTVIT

170 236 34 21

(99,971) (195,587) (28,474) (3,634)

Study 3’ MEDPAR (1984)’

ICCE ECCE PHACO ANTVIT

1,016 1,029 156 74

(100,376) (196,386) (28,562) (3,670)

Outcome 4year risk of retinal detachment repair per 100 patients 1.55 0.90 1.17 5.00

l-year risk of endophthalmitis per 100patients 0.17 0.12 0.12 0.58 4year risk of cornea1 transplant or edema per 100 patients 1.40 0.63 0.62 2.42

Relativerisk of retinal detachmentrepair (3 year)

Study 4’”

per 100 patients

MEDPAR (1986-1987)” BMAD (1986-1987) Outpatient (19861987)

ECCE/PHACO No capsuIotomy

Casesnot published (43,394) Reference

Capsulotomyperformed Casesnot published (13,709) at least1 day after cataract surgery Study 5” MEDPAR (1986-1987)” BMAD

ECCE/PHACO

44 (57,103)

3.9

l-year risk of endophthalmitis per 100 patients 0.081

(1986-1987)

Outpatient (1986-1987)

ECCE/PHAC

261 (57,103)

J-year risk of retinal detachment repair per 100 patients 0.81

MEDPAR = medical provider analysis and review; ICCE = intracapsular cataract extraction; ECCE = extracapsular cataract extraction; PHACO = phacoemulsification; ANTVIT = cataract extraction with anterior vitrectomy; BMAD = part B medicare annual data beneficiary file; Outpatient = part B hospital outpatient facility file.

INSURANCE

CLAIMS

DATABASES

FOR OPHTHALMIC

traction (ECCE) , intracapsular cataract extraction (ICCE), phacoemulsification (PHACO), and any type of cataract surgery with an anterior vitrectomy in inpatient facilities in 1984 (Table 1). Another study evaluated the occurrence of retinal detachments in eyes with or without a posterior capsulotomy after ECCE or PHACO in hospital inpatient and outpatient facilities and ambulatory surgical centers.” The fifth study compared the risk of retinal detachment and endophthalmitis in eyes in which cataract surgery had been performed in inpatient and outpatient facilities and ambulatory surgical centers between 1986 and 1987 with the risk of retinal detachment and endophthalmitis in eyes that had cataract surgery in inpatient facilities in 1984.” All analyseswere based on retrospective follow-up data derived from the billing statements of ophthalmologists throughout the United States. Subject? were not randomized to type of cataract procedure. Early work by Mitchell and coworkers’g discusses the challenges of identifying the study population, defining the time interval after the index event (cataract surgery), and defining complications or outcomes in studies using Medicare claims data. Our focus is on a different set of issuesrelated to study interpretation.

Advantages LARGE

SAMPLE

of Insurance

Claims Databases

SIZE

One of the main advantages of the Medicare claims database is that it is large; for example, in 1984 there were 338,141 Medicare beneficiaries aged 65 years and older who were hospitalized for cataract extractions.‘ L”.‘~ Studies with large sample sizes enable investigators to estimate effects of interest more precisely. This advantage is particularly important when the outcome of interest is a rare event. For example, an outcome of interest may be the risk of endophthalmitis after different cataract procedures. In studies using smaller databases, the risk of endophthalmitis after ICCE was 0.086% (95% confidence interval [CI], 0.056-0.116%)7 and after ECCE, 0.072% (95% CI, 0.043-0.12%).s Javitt and coworkers,” using the cataract extractions in Medicare beneficiaries, estimated the l-year risk of endophthalmitis in 1984 to be 0.17% (95% CI, O.I5-0.20%) for ICCE and 0.12% (95% CI, O.lO-0.14%) for ECCE. The narrower 95% CIs in the study by Javitt and coworkers’” reflect the greater precision of the estimates in that study compared with the studies using smaller databases.‘,‘” It is interesting to note that the nominal 95% CIs for risk of endophthalmitis after ICCE from the smaller and larger studies do not overlap. This lack of overlap may be explained by differences in the sampling frame (retrospective chart review from a single clinical popu-

273

SURGERY

lation versus retrospective analysis of claims data of Medicare beneficiaries) and selection methods (culture-positive endophthalmitis versus International Classification of DiseasesDiagnosiscodes for endophthalmitis) of each study. A general comment about the use of large databasesis that a smaller portion of total estimation error is due to sampling variability (random error) compared to most clinical research, which may have a larger portion of error due to sampling variability becauseof the smaller sample size. When Javitt and coworkers” used the International Classification of DiseasesDiagnosis codes for endophthalmitis in a cohort of 57,103 Medicare beneficiaries who underwent ECCE between 1986 and 1987, the l-year risk of endophthalmitis was 0.81% (95% CI, 0.057-0.10%/o).This 95% CI overlaps the 95% CI for l-year risk of endophthalmitis in 1984. As Javitt and coworkers” note, however, there appears to be a decline in the rate of endophthalmitis from 1984 to the period between 1986 and 1987 when the same method was used to ascertain cases. If more-detailed patient characteristics could be collected, as in a case-control study, it would then be interesting to investigate whether this decline is secondary to changes in cataract surgical technique, differences in the treatment of endophthalmitis, or differences in the study population between the 2 years (inpatient population in 1984 versus mainly outpatient population between 1986 and 1987). POPULATION-BASED

DESIGN

Another advantage of studies using large databases,such as that from Medicare, is the populationbased design, because nearly all relevant outcomes can be identified for a well-defined population at risk. With this approach, the difference in the risk of adverse outcomes arising from two particular cataract procedures estimates the excess risk of outcomes of one procedure compared with the other in that population. This absolute measure of effect may be useful in public policy decisions of resource allocation in the source population. For example, ICCE is associated with 5 more casesof endophthalmitis per 10,000 cataract extractions than is ECCE.‘” If assignment to ICCE versus ECCE were random, one might expect one additional case of endophthalmitis if 2,000 additional ICCEs were performed per year compared with ECCEs in the Medicare population. Such information would be expected to influence Health Care Finance Administration policy decisions or managed care organizations, especially as resources become scarce. GENERALIZABILITY

AND LACK OF SELECTION

BIAS

Generalizability involves the extrapolation of results of a study to other populations and the pooling

274

Surv Ophthalmol42

(3) November-December

of information from different studies. Selection bias occurs when the manner in which subjects are included in the study or analysis distorts effect estimates, such as the relative risk.17,20 Selection bias may reduce the generalizability of a study. For example, in nonrandomized follow-up studies, the selection of subjects does not usually introduce selection bias because the outcome does not influence subject selection.22 In addition, in studies using Medicare claims data,2*“-‘4 selection bias is assumed to be small because reporting to Medicare is apt to be nearly universal, because doctors want to be paid. Thus, studies using insurance claims data have the potential to be more generalizable than randomized clinical trials, because claims data analysis likely reflects community practice. Randomized clinical trials often enroll selected subjects who receive care that has been standardized but does not necessarily reflect community practice. In clinical trials, randomization is used to help prevent or reduce selection bias. Randomized studies have the appealing property that any associations between treatment assignment and background characteristics arise as a result of chance variation and thus are expected to be smaller as the sample size increases.5Randomization of treatment assignments can be expected to balance the distribution of characteristics between groups; in the absence of randomization of treatment assignments, the primary strategy for improving comparability is to control for potential confounders in the analysis. The Medicare public-use tapes are assembled by random sampling of the Medicare patient population, but this type of random sampling should be distinguished from randomization of treatment assignments. For example, the 5% random sample of Medicare beneficiaries in 1986 and 1987”,” allows investigators to generalize from the sample to the entire 1986 to 1987 cohort, but it does not improve the comparability between treatment groups. Similarly, even with random sampling of subjects, there may still be lack of generalizability in a study because of effect modification, the variation of the magnitude of the treatment effect with different levels of another risk factor.” Because the probability of having inpatient surgery may depend on the type of cataract surgery and covariates, such as age, race, and sex of the patient; experience of the surgeon; the presence of ocular comorbidity, such asglaucoma or prior retinal detachment surgery; and the presence of systemic comorbidity, such ashypertension or diabetes, studies using the Medicare claims database from inpatient surgery only”,‘“,‘” may not be generalizable to all subjects with cataracts who are candidates for cataract surgery. Although studies based on Medicare claims data from inpatient and outpatient surgery”,‘”

1997

COLEMAN AND MORGENSTERN

may be extrapolated to patients who had either type of surgery, the results still may not be generalizabie to all cataract patients who were candidates for cataract surgery, because the study population was limited to individuals older than 65 years of age who were not members of health maintenance organizations and who had continuous part A and B coverage from 1986 to 1988. Thus, the findings may not be generalizable to patients less than 65 years old who are candidates for cataract surgery; however, the results may be generalizable to patients 65 years or older because they reflect community practice and depend on a population-based data analysis.

Potential Areas of Bias and Confounding MISCLASSIFICATION BIAS Misclassification due to errors in both diseasedetection and coding carries the potential to bias estimates of desired effects (e.g., risk or rate ratios) even in the presence of complete population data.” Misclassification bias can arise not only from error in measuring the outcomes of cataract surgery and capsulotomy, but also from errors in the Medicare databasein coding which cataract procedure was actually done. It is useful to distinguish two types of misclassification bias, nondifferential and differential, the determination of which depends on whether misclassification on the outcome is independent of the type of cataract procedure. If the errors in diagnosis or coding of the outcome are independent of the type of cataract procedure, then the misclassification is nondifferential. Nondifferential misclassification usually biasesthe estimate of effect toward the null value (no effect). For example, if there is a common rate of errors in coding retinal detachments for ICCE and ECCE, then the difference in detachment rates between the two cataract procedures (the rate difference) would likely be closer to the null value of zero, and the rate ratio (or relative risk) of ICCE and ECCE would likely be closer to the null value of one. Differential misclassification bias occurs when errors in diagnosing or coding the outcome are dependent on the type of cataract procedure. For example, because there may be vitreous loss in eyes after ICCE, clinicians may more aggressively evaluate the retina of eyes after ICCE than after ECCE. There may, therefore, be more retinal tears or detachments diagnosed in eyes after ICCE than after ECCE.” This differential misclassification would cause an apparent excess risk of retinal detachment in eyes with ICCE, because there would be more casesof retinal detachment diagnosed in eyes after ICCE than in eyes after ECCE. In general, differential misclassification can bias effect estimates either away from or toward the null value of no effect, leading to overes-

INSURANCE

CLAIMS

DATABASES

FOR OPHTHALMIC

SURGERY

timates or underestimates of rate differences and rate ratios.” Misclassification bias can also occur from lack of relevant information in the database, such as which eye had the index surgery and which eye had the sub sequent outcome. The involved eye is coded in only 10% of the Medicare insurance claims,” and some subjects in the cohort may have already had cataract extraction in one eye prior to undergoing cataract extraction in the other eye. Javitt and coworkers” reported that in a 5% random sample of inpatient and outpatient Medicare claims between 1985 and 1988, 29.8% of the cohort had bilateral cataract surgeries. The possibility of bilateral cataract surgery is important, because for these subjects a disease outcome may have been mistakenly attributed to the surgery that qualified them for inclusion in the cohort.‘,‘“.‘4 If, for example, this mistaken attribution of an adverse outcome wasdone more frequently for patients who had ICCE in their first surgery and ECCE in the eye that qualified them for inclusion in the study, which might arise asa consequence of trends in prevailing physician practice, then there would be differential misclassification bias. This bias could affect estimatesof risk ratios or risk difFerences for endophthalmitis, penetrating keratoplasty, cornea1 edema, or retinal detachment surgery. Errors in disease diagnosis or coding can be very difficult to correct in statistical analyses. One approach to this problem is to attempt to estimate the magnitude and direction of misclassification bias. Because there is a 20.8% error in coding of hospital discharge data in the Medicare claims database,3,8 the magnitude of misclassification error (differential or nondifferential) may be large and may greatly affect the estimated effects of treatments on outcomes. In addition, it may vary across studies. However, the large measurement error in coding of hospital discharge data is not noted in the coding of cataract procedures.“.‘” Javitt and coworkers”’ reviewed 802 paid Medicare claims for cataract surgery and reported the positive predictive value of cataract surgery in the Medicare claims database to be 0.99 (95% CI, 0.93-1.00). With information on sensitivity and specificity of disease coding, or with information on positive predictive value, statistical corrections may be performed that incorporate measurement error rates into outcome rates of interest. In one illustration of this, the relative risk of endophthalmitis in ICCE compared with ECCE, based on 1984 data, was 0.17/0.12 = 1.4214;the corrected relative risk obtained using the positive predictive value of the original coding of cataract surgery is 1.42, which is the same as the crude estimate of relative risk.{” Thus, the magnitude of misclassification bias is an empirical issue. Based on published estimates,

275

misclassification bias might not be as great a concern in evaluating cataract procedures as in evaluating other medical procedures or diagnoses,where coding problems are more severe. CONFOUNDING

Conclusions drawn from studies utilizing a database created primarily for purposes other than research need to be interpreted cautiously because of the lack of randomization and because of treatment differences within treatment groups. Subjects in the Medicare claims database are not randomized to different cataract procedures; rather, their treatment assignment depends on a combination of their specific ophthalmic condition and their physician’s judgment. A key issuein the interpretation of results from nonrandomized studies of intended effects (e.g., the beneficial effects of medical and surgical interventions) is the high likelihood of “confounding by indication.“15 In other words, it is very likely that certain medical/surgical procedures are deliberately performed on those patients who are the most severely (or least severely) ill, i.e., those who are at greatest (or lowest) risk of developing adverse outcomes. Variables, such as disease severity, the presence of comorbidities, and general health status, are referred to as extraneous prognostic factors if they increase the probability of an individual having a certain outcome. These factors may be distributed unevenly among the different cataract procedures. This uneven distribution is likely to bias (confound) the estimation of treatment effects on the risk of endophthalmitis, penetrating keratoplasty, cornea1 edema, and retinal detachment surgery. To be a confounder, a covariate must fulfill the following three criteria: 1. It must be a risk factor, or a proxy for a risk factor, for the disease outcome in the reference group, which may be untreated or receiving a different treatment than the “treated” group. 2. It must be associated with treatment status in the total population at risk, that is, it must be disproportionately represented in the “treated” group relative to the reference group; and 3. The association between the risk factor and treatment status must not be due entirely to the effect of the treatment on the risk factor. For example, the risk factor cannot be an “intermediate variable” in a causal pathway between treatment and outcome. For example, in the study of the risk of retinal detachment repair after diierent cataract procedures,‘” race was treated as a confounder, because race is a risk factor for retinal detachment surgery in the reference group (ICCE) , blacks were more likely to un-

276

Surv Ophthalmol42

(3) November-December

dergo ECCE than were whites, and the type of cataract surgery does not influence the race of the subjects. Race was thus appropriately controlled as a confounder in the statistical analysis. The selection of race as a confounder was possible through relations noted in the database. It is important to realize that the use of one’s data to select confounders can be faulty and can result in bias in the estimation of the effect of interest.’ Controlling for a covariate that has no association with the outcome will result in a loss of power if it is associated with treatment status; however, this is generally not a central concern in analyses of large databases because of the substantial sample size. The best way to choose confounders is to identify potential risk factors that are believed to meet the three criteria listed above. When the confounders have been measured with little or no error, confounding can be controlled with stratification or statistical modeling. If the confounders have been measured with error, then analytic control for confounding may be incomplete, and estimates of effects of interest, such as the risk ratio, may then be biased.18

Effect

Modification

Extraneous risk factors may also modify the effect of treatment on diseaseoutcome, i.e., the magnitude of the treatment effect may vary over levels of another risk factor, which is referred to as an effect modifier.” The presence of statistical interaction between treatment indicators and risk factors in statistical analyses implies effect modification. As with any investigation into the presence of interactions, questions about effect modification depend in part on the scale of the outcome in the analysis;for example, a risk factor might be an effect modifier in an additive or linear model, but the same factor might not be an effect modifier in a multiplicative or logistic model. The assessmentof effect modification is relevant for judging the appropriateness of statistical models, for evaluating possible biological interaction effects, and for enhancing generalization of study results to other populations. Age is an example of an effect modifier in the study of the risk of retinal detachment repair after a capsulotomy.” Compared with subjects over 85 years in age, the estimated effect of capsulotomy on retinal detachment is three times higher than those 75 to 84 years old and seven times higher than those 65 to 75 years old. This effect modification due to age is relevant to assessinga possible biological interaction and to generalizing the results. A potential biological interaction of age and capsulotomy may be due to the increased incidence of posterior vitreous detachment with increasing age. In addition, recognizing age as an effect modifier is important in comparing the risk of complications

1997

COLEMAN

AND MORGENSTERN

over time, because the age distribution of the population may change. Standardizing complication rates to a target population with fixed characteristics is needed to develop interpretable comparisons between populations that have different distributions of characteristics or effect modifiers.

Selective

Loss to Follow-up

Selective loss to follow-up occurs when withdrawal from the study is related to both outcome risk and treatment status (ICCE, ECCE, PHACO, anterior vitrectomy, and/or capsulotomy) .” If these withdrawals occur differentially, for those with and without the different disease outcomes, then selection bias results. Selective loss to follow-up is difficult to measure, and, in general, the greater the number of withdrawals from observation, the greater the potential for bias. Patients with specific diseaseoutcomes (endophthalmitis, penetrating keratoplasty from cornea1 edema, and retinal detachment) may have received treatment outside of the Medicare system, although this type of attrition probably is rare. Because the postoperative period for following-up surgical patients is 3 months, it is unlikely that patients have received treatment outside of the Medicare system during this time. Whether or not patients have more incentive to continue their medical care after the 3-month postoperative period at a Veterans Administration Hospital or at a health maintenance organization is unknown. Of 57,103 ECCE casesin 1986 and 1987, 8,282 (14.5%) were lost to follow-up due to death, additional intraocular surgery, or transfer of care to a health maintenance organization.” Although death is an inevitable cause of loss to followup, especially in an elderly cohort, such as Medicare beneficiaries, there is concern that patients who are ill and more likely to die may be more likely to have one type of cataract procedure such as ICCE versus another. For example, if more subjects with ICCE than ECCE die, then the risk of complications for ICCE may be artificially low; that is, if the subjects with ICCE had lived, they may have had more complications than reported. Becauseselective lossto follow-up is difficult to measure, it is helpful for investigators to report attrition rates from measured events, such as death or the occurrence of additional intraocular surgery, because readers can then speculate on the direction and magnitude of potential bias.

Review

of Validation

Studies

Two current studies have helped to confirm the validity of the results of studies using Medicare claims data In a population-based, case-control study, Tielsch and coauthors” used a 5% sample of inpatient and outpatient Medicare claims data files from 1988 to 1991 to identify subjects who had retinal de-

INSURANCE

CLAIMS

DATABASES

FOR OPHTHALMIC

tachment after ECCE. They then identified controls as subjects who underwent cataract extraction but did not have a subsequent retinal detachment. The ophthatmologists who provided care to these patients were then contacted and asked to complete a medical chart review and brief clinical data collection form. After exclusions due to ineligibility and/or lack of information, there were 291 cases of confirmed retinal detachment and 807 controls. Tielsch and coauthors found an odds ratio of 3.8:l.O for retinal detachment after Nd:YAG laser capsulotomy. This odds ratio was very similar to the risk ratio of 3.9: 1.O that was reported previously by Javitt and coauthors.” Thus, the potential misclassification error due to lack of information on laterality in the Medicare claims data analysis appears minimal. In addition, this study provided information on risk factors for retinal detachment that was not available in the claims data analysis, such as the presence of lattice degeneration or prior ocular trauma. In the casecontrol study 149 of 691 (22%) potential casesof retinal detachment were not confirmed. This potential error rate was similar to that reported in other studies on the coding of hospital discharge data in the Medicare claims database.“,’ Another study that supports the conclusions reported in the studies using the Medicare claims database was the evaluation of the risk of retinal detachment following cataract surgery in Denmark, in which 19,043 subjects were identified from the Danish Hospital Registry with similar diagnostic criteria asused in the Medicare claims data analysis.2’Norregaard and coauthors”’ reported a 4year cumulative risk of retinal detachment after ECCE of 0.98% (95% CI, 0.751.21) while Javitt and coauthors’” reported a similar 4year cumulative risk of 0.90%. Although the study by Norregaard and coauthors had many of the samepotential biasesas the studies that used Medicare claims data analysis, the similarity of the results of the two studiesstrengthens one’s beliefs in their conclusions.

Implications

for Comparison Success Rates

277

SURGERY

of Provider

The same issuesthat arise in trying to interpret evidence about possible differences in treatment efficacy would apply to comparisons of provider (e.g., hospital or individual clinician) successrates. An additional caveat is that the modest sample sizesavailable for individual providers make sampling variability a genuine concern, especially when comparing event rates for relatively rare complications. In addition, differences in provider specializations imply differences in case mix between patients that can exaggerate the effects of potential confounders to the point that TWO providers may exhibit little or no overlap in the kinds of patients they treat. In such instances,

comparisons of raw event rates or comparisons of event rates that incorporate statistical adjustment for confounders become difficult to interpret. These issuesare familiar to researchers in other fields, such as social science and education, where investigators rely heavily on observational data’” and they have important implications for health-related research. Although it is possible to include within- and between-provider variability in a hierarchical model (sometimes referred to as an empirical-Bayes approach) ,’ such modeling does not solve the problems posed by unobserved confounders. In addition, even when there is significant between-provider variability in treatment successrates, interval estimates for individual providers tend to overlap substantially due to the small sample size associatedwith individual providers. Because publication of provider comparisons or ranking could substantially influence individual reputations, and because the accuracy of such comparisons may be questionable due to inherent sources of bias and confounding, we believe that scientists would be well advised to establish clear boundaries between investigations of treatment efftcacy, which may be sensitive to a number of factors as we have described, and investigations of provider efficacy, which tend to be less precise and involve a greater reliance on modeling assumptions.

Summary Studies using Medicare claims databases can contribute greatly to our understanding of the effectiveness and drawbacks of ophthalmic intervention. Nevertheless, the estimation of treatment effects on diseaseoutcomes may be biased due to limitations of the study design, the collection of data, and analytic methods. The authors”,“-‘” of the studies analyzing the association of cataract surgery or capsulotomy with endophthalmitis, penetrating keratoplasty, corneal edema, and retinal detachment surgery should be commended for providing substance to the discussion of important clinical questions. But asJavitt has suggested,” if resources such as the Medicare claims database are to be used to assessthe potential usefulness of a procedure by providers or consumers, data analysesshould consider issues,such asmisclassification, confounding, effect modification, lack of generalizability, and selective loss to follow-up. These issueswill continue to complicate and limit causal inferences regarding treatment effectiveness and individual clinician skill. In terms of the design of data-collection efforts, more flexible and informative analyses would be possible if more detail were available on potential confounders and sources of attrition Improvements, such as the coding of the eye that receives treatment, seem attainable and offer the possibility of substantial gains in insight.

278

Surv Ophthalmo142

(3) November-December

1997

COLEMAN

References AB: Bacterial endophthalmitis after 1. Allen HF, Mangiaracine cataract extraction. II: Incidence in 36,000 consecutive operations with special reference to preoperative topical antibiotics. Arch Ophthalmol91:3-7, 1974 outcomes of cata2. Canner JK, Javitt JC, McBean MA: National ract surgery. III: Cornea1 edema and transplant following inpatient surgery. Arch Ophthalmol. 110:1137-1142, 1992 3. Fisher ES, Whaley FS, Krushat M, et al: The accuracy of Medicare’s hospital claims data: Progress has been made, but problems remain. Am J Pub Health 82:243-248, 1992 S: The effect of misclassification in the presence 4. Greenland of covariates. Am J Epidemiol 112:564-569, 1980 5. Greenland S: Randomization, statistics, and causal inference. Epidemiology 1:421-429, 1990 6. Greenland S: Methods for epidemiologic analyses of multiple exposures: A review and comparative study of maximum-likelihood, preliminary-testing, and empirical-Bayes regression. Stat Med 12:717-736, 1993 7. Greenland S, Robins JM: Identifiability, exchangeability, and epidemiologic confounding. Int J Epidemiol 15:412-418, 1986 8. Hsia DC, Krushat WM, Fagan AB, et al: Accuracy of diagnostic coding for Medicare patients under the prospective-payment system. N Engl J Med 318:352-355,1988 9. Javitt JC: Outcomes of eye care from Medicare data. Arch Ophthalmol 109:1079-1080,199l 10. Javitt JC, McBean AM, Sastry SS, DiPaolo F Jr: Accuracy of coding in Medicare part B claims: Cataract as a case study. Arch Ophthalmol 111:605-607,1993 11. Javitt JC, Street DA, Tielsch JM, et al: National outcomes of cataract extraction: Retinal detachment and endophthalmitis after outpatient cataract surgery. Ophthalmology lOl:lOO105,1994 12. Javitt JC, Tielsch JM, Canner JK, et al: National outcomes of cataract extraction: Increased risk of retinal complications as-

13.

14.

15.

16. 17. 18. 19. 20.

21.

22.

23.

AND MORGENSTERN

sociated with Nd:YAG laser capsulotomy. Ophthalmology 99:1487-1498, 1992 JavittJC, Vitale S, Canner JK, et al: National outcomes of cataract extraction. I: Retinal detachment after inpatient surgery. Ophthalmology 98:895-902,199l Javitt JC, Vitale S, Canner JK, et al: National outcomes of cataract extraction. II: Endophthalmitis following inpatient surgery. Arch Ophthalmol 109:1085-1089, 1991 Kattan H, Flynn HW Jr, Pflugfelder S, et al: Nosocomial endophthalmitis survey: Current incidence of infection following intraocular surgery. Ophthalmology 98:227-238, 1991 Kelsey JL, Thompson WD, Evans AS Methods in Observdtional Epidemiology. New York, Oxford University Press, 1986 Kleinbaum DG, Kupper LL, Morgenstern H (eds): Epidemiologic Research. New York, Van Nostrand Reinhold, 1982 Miettinen OS: The need for randomization in the study of intended effects. Stat Med 2:267-271, 1983 Mitchell JB, Bubolz T, Paul JE, et al: Using Medicare claims for outcomes research. Med Care 32:JS38-JS51. 1994 Morgenstern H, Thomas D: Principles of study design in environmental epidemiology. Environ Health Perspect 101 (Suppl 4):23-38, 1993 Norregaard JC, Thoning H, Andersen TF, et al: Risk of retinal detachment following cataract extraction: Results from the International Cataract Surgery Outcomes Study. Br J Ophthalmol80:689-693, 1996 Tielsch JM, Legro MW, Cassdrd SD, et al: Risk factors for retinal detachment after cataract surgery: A population-based case-control study. Ophthalmology 103:1537-1545, 1996 Wainer H: Eelworms, Bullet Holes, and Geraldine Ferraro in The Role of Models in Nonexperimental Social Science: Two Debates. Washington, DC, American Educational Research Association and American Statistical Association, 1992

Reprint address: Anne Coleman, stitute, 100 Stein Plaza, Los Angeles,

MD, PhD, Jules CA 90095-7004,

Stein Eye InUSA.