- Email: [email protected]
Indices for monitoring hospital outcomes in developed countries
Heat&hh/icy, 21 (1992) I-15 01992 Elwvier science Publishers B.V. All rights reserved. 0168-8510/92/$5.00
1
HPE 00459
Indices for monitoring hospital outcomes in developed countries Susan I. DesHarnais and Kit N. Simpson Department of Health Policy and Administration, School of Public Health, Univemity of North Carolina at Chapel Hill, North Carolina, U.S.A. Accepted
6 November
1991
Summary We discuss some of the chsllenges facing hospitals in developed nations, with spedal attention to the need to monitor and evaluate hospital performance. In particular, there is a need for quallty Indicators that measure the effects of treatment, and are rlsk+rdjusted, so that valid comparisons of outcomes can be made across hospitals that treat different types of patients. Until recently, only very crude quality indicators have been available for comparing the performance of different hospitals. We describe three risk-adJusted Indices for comparing the outcomes of hospltsl care, focuslng on the construction and vslldation of these measures. We discuss the uses of these tools for identlfylng problems and for monltodng outcomes of cam within a hospital, Including screening medlcal records for peer review, ldentlfylng variations in outcomes ecross various subgroups of physicians, and comparing changes in outcomes following various changes In the delivery system. Possible apptlcstions at the regions4 national and international levels are then discussed, with special emphasis on the use of these tools for messuring the size of the gap between the efllcacy of a technology, as determined through rsndomlxed controlled trials under stdngent protocols, and the ef&tlveness of the same technology once It Is exported, and then used under true pmctlca condltlons In another country. Quallty measurement; Hospital performance; Outcome; Monitoring; Risk adJustment
Addtws for comsspond~ Susan I. DesHamais, Ph.D. Health Policy and AdmlnistmtionCE37400, McGavran-GreenbegHall, UNC Chapel Hill, Chapel Hill, NC 27599-7400, U.S.A.
2
Introduction Health care experts from virtually all developed nations are reporting changes in the health care systems in their home countries. Changes are occurring in the structure and financing of most systems, and there are also changes in disease prevalence in many countries. The or&r of the day in health care sectors seem to be rapid, constant change, with a majority of these changes related to the provision of hospital care. Physicians are adopting new medical treatment technologies, economists are trying to assure efficient allocation of resources, managers are working to improve their hospitals’ effectiveness, and politicians are cutting budgets. The above actors are often assuming that hospitals are robust systems for producing medical care, and that continuous ‘tinkering’ will not cause unacceptable changes in the quality of care, with quality being defined by us throughout this discussion as health outcomes, i.e., the effects of treatment on the health status of the patients. Recent U.S. reports of grossly decreased access and increased costs [l] demonstrate, however, that the balance between cost, quality and access to hospital care is quite delicate. Belatedly, we are recognizing the need to develop systems for monitoring changes in the outcomes of hospital care, so that we can identify, analyze, and correct unacceptable variations in performance across hospitals.
Challenges facing hospitals in developed nations Hospitals across developed nations are being challenged to respond to major health care issues: the aging of the population; the increase in chronic disease caused by lifestyle factors; the need to incorporate prevention in a system where most available resources are needed for treatment [2]; and the escalating cost of providing care [3]. It is generally recognized that no country is able to afford all the medical care that its citizens need [4]. As a result, hospitals in most countries are under pressure to decrease their total costs without harming the quality of care provided. Faced with limited resources, it is necessary to assure that up-to-date medical technology is purchased and made available, but without bankrupting the system [5,6]. Therefore, decisions must be made on which new technologies to adopt, and how to monitor changes in outcomes of care once new technologies are adopted. The goal in most countries is to develop hospital information systems that provide a parsimonious set of quality indicators which can be used to monitor and manage change. Information is required on resource inputs, the process of providing care, the outputs of the organization, and patient outcomes [7,8]. These indicators must be sensitive, and they must enable hospitals to avoid unacceptable variations in cost, efficiency, effectiveness, and quality. The information system must be capable of providing data for planning and evaluation of care at the various levels of a country, and preferably cross-
3
nationally [9]. Information systems should also be able to provide data that can be used to protect the hospital from unfair comparisons with other institutions. Although hospital information systems have made great improvements in their ability to capture resource utilization and cost data [lo], until recently, it has not been possible to provide comparative data that allow for the monitoring and review of the quality of care provided in different hospitals [ll]. Only very crude indicators of health outcomes have been available for comparison of the health of populations served by different hospitals and systems [12,13]. This inability to measure the outcomes of care has led to a fragmented approach to improving the quality, effticiency and effectiveness of Care.
Governments, elected officials and purchasers of care need valid, yet inexpensive measures of quality, as they attempt to compare the performance of various hospitals. Birch and Maynard from the U.K. specifically state that: “me] deign of existing performance indicators should be changed in order to move towards a more careful consideration of what the hospital is achieving as opposed to simply what it is using. In particular [we need] to usesdata on the quality of a hospital’s output, as indicated by readmission rates and post-opcratin complication rates. Clearly, these rates are of major concern to patients and, hence, are indicators of the degree to which objectives are being satisfied.” [14]
Ideally, we want to compare rates of both positive and negative outcomes across hospitals. We would like to be able to evaluate the effects of hospital care directly, by measuring the changes in patients’ health status following treatment. This, however, is diflicult to do since there is no practical way to obtain data on patient health status before and after treatment for a large national sample of hospitals. While we may not be able to directly measure quality in terms of changes on patient health status following treatment, we are able to compare the rates of certain adverse consequences of treatment across hospitals. Under the assumption that the hospitals with the lower rates of adverse events are producing better patient outcomes, our measures of adverse outcomes are used as proxies for positive measures of outcomes. There are many historical precedents for measuring rates of adverse events, rather than using direct or ‘positive’ measures of quality. Death has been used as an indicator by many investigators, including Wennberg, Flood, Luft, Knaus, Kelly, and Hebel [15-201. Readmissions have been examined by Fethke, Anderson, Good@, ROOS,Riley, Zook, Holloway, and Smith, [21-281 among others. Complication rates have been studied by ROOS,Brook, Chilton, Adams [28-311 and others. The system of indicators presented in this paper is configured so that it provides three important ‘negative’ indicators of hospital quality: death rates, readmission rates, and complication rates. These three outcome measures are adjusted for patient risk at the time of admission. This risk-adjustment is essential for valid comparisons across hospitals. While differences in patient outcomes across hospitals can be viewed as partially a result of hospital
4
performance, there are several other factors that also influence outcomes, Differences in outcomes across hospitals or over time may be due to variations in the types of patients treated; the severity of the illness which is designated as the principal diagnosis; the types and complexity of the patients’ comorbidities; the patients’ age distribution, and the social and financial condition of the patients treated at a given hospital. In order to measure provider performance with some accuracy, one must control for these other factors which affect outcomes. If a population group treated in a given hospital has higher than average risk factors, then the rates of adverse outcomes would be expected to be high for these patients. While it is impossible to control completely for all risk factors, given our existing data sets and measurement tools, it is possible to use existing data to approximate some of these control variables. A method for doing so is described next. Indices of risk-adjusted outcomes: mortality, readmissions, and complications In two recent articles we described in detail the methods used to construct the Risk-Adjusted Mortality Index (RAMI) [32], and the construction and validation of the three measures of hospital performance reported in this paper [33]. In order to construct indices of hospital performance, two separate but related problems had to be solved: it was essential to take into account differences across hospitals in the types of patients treated (case mix); and it was necessary to use an appropriate severity measure to take into account differences across hospitals in the severity of illness within each of the disease categories (case complexity). Unless these adjustments are made, one cannot make valid comparisons across hospitals, since those hospitals treating more ‘difficult’ cases will appear to have worse outcomes. In developing these indices we used information contained in an existing database to develop proxies for many of the factors other than provider performance which are related to health outcomes. By means of indirect standardization to the large, national all-payor database collected by the Commission on Professional and Hospital Activities (CPHA), we obtained empirical information to calculate and model the effects of various risk factors for each type of patient. Even though some risk factors are only approximated by this method, the approach is useful. The models are practical, insofar as they use existing databases, and they are comprehensive, insofar as they measure hospital performance for all payors and virtually all types of cases treated in a given hospital. Three different risk-adjusted indices of hospital performance were developed: - The Risk-Adjusted Mortality Index (RAMI) - The Risk-Adjusted Readmission Index (RARI) - The Risk-Adjusted Complications Index (RACI) The risk factors for each of these quality measures were originally modeled
5
using a national CPHA data file, containing six million cases from 1983, from 776 short-term U.S. general hospitals. This cohort of 776 CPHA hospitals was generally representative of the 5663 US hospitals in the universe. However, the for-profit hospitals and Southern hospitals were somewhat underrepresented, while major teaching hospitals are overrepresented. Nevertheless, it is important to realize that the discharges are the units of analysis for modelling the risk of each outcome. There is reason to believe that these six million discharges used for modelling risk factors are representative of all discharges treated in the U.S.. The three indices were later rebased using data from 1988. The following cases were excluded from the analysis file: - all cases transferred to other short-term hospitals (referral centers, specialty hospitals). - all cases of newborn infants. Since most critically ill newborns are transferred to neonatal centers, it was not possible to model risk factors for this group in a valid manner. Moreover, newborn weight, a critical variable in predicting infant deaths, was not available. - all cases with stays of less than one day who were discharged alive (presumed to be outpatient surgery and other outpatient cases). We ,then used an empirical approach to model the risk of death, readmission, and complications associated with each reason for admission. Since it was apparent that the patient characteristics associated with the risk of each outcome would vary from one disease to another, separate models were developed for each disease category. The first step in constructing each measure was the aggregation of Diagnosis Related Groups (DRGs) into clusters which group those DRGs with the same clinical condition. This clustering was necessary for our purposes because some of the factors associated with an increased risk of death, readmission, or complication within a clinical condition were used as the basis for DRG divisions (age, comorbidities/complications). For instance, DRGs 89, 90, and 91 are all simple pneumonia and pleurisy, but the DRGs differ by the age of the patient. We needed to regroup the DRGs into clusters, to determine how age, comorbidities, and other factors were associated with an increased risk of adverse outcomes within each disease condition. This clustering procedure resulted in 316 categories, each representing a different clinical condition. Once this clustering was done, we proceeded to develop the three different indices: RAMI, RARI, and RACI. The risk models were based on clinical factors which were recorded in the CPHA dataset. CPHA abstracts contain, among other items, the following data elements for each patient discharge: age, race, hospital identification, dates of admission and discharge, discharge status (alive, dead), principal diagnosis, secondary diagnoses (up to I l), and principal surgical procedure.
6
Construction of the three indices The Risk-Adjusted Mortality Index (RAM) Two different types of models were employed when modelling the risk of mortality for each patient. This was done so that the most appropriate technique would be used for each disease or condition. These models are: - The Contingency Table Model: For each of the 252 DRG clusters where the death rate was less than 5% (83% of all discharges; 28% of all deaths), a Contingency Table Model was developed, since the estimating procedures for logistic regressions are inadequate in this situation. - The Logistic Model: For each of the 64 clusters where the death rate was 5% or more (17% of all discharges; 72% of all deaths), a logistic model was developed. The Contingency Table Model is based on deriving a separate 2 x 3 table for each of the DRG clusters with relatively low death rates. The age of patients (O-64; 65-74; 75+) and the presence of comorbidities were used as classifying variables, since these were the best predictors of death in the more complicated models that we tried to construct. The death rate was determined empirically within each of the six cells. (See Table 1 for an example of a contingency table.) Complications were excluded as risk factors, because we wanted to measure the severity of a patient’s illness at the time of admission, in order to assess the risk of the patient’s primary medical problem and related comorbidities prior to medical care intervention. However, the HCFA list of comorbidities and complications (CCs) contains both comorbidities (secondary diagnoses present at the time of admissions) and complications (conditions that arose during the course of treatment). Because we wanted to measure comorbidities, but not complications, we attempted to separate the two. Our medical consultant designated 70 of the conditions on the HCFA CC list as most likely to be complications of surgery, or iatrogenic events. This list includes problems such as transfusion reactions, accidental operation lacerations, etc. This was a conservative approach, insofar as certain conditions such as pneumonia and urinary tract infections, which may or may not be complications, were assumed to be comorbidities. Because the structure of the ICD-PCM codes explicitly identifies surgical and OB/GYN complications, but not medical complications, we’were unable to distinguish whether some medical problems arose during treatment. Therefore, hospitals were credited for having sicker patients if we could not be sure whether a condition was present at the time of admission. However, obvious complications were removed as ‘risk factors’ in the models, since hospitals should not be credited for a more complex case mix on the basis of a high frequency of iatrogenic events among their patients. Thus, the Contingency Table Model was used to distinguish six risk categories, within each of the relatively low death rate clusters, based on three age groups and the presence of comorbidities. These six risk categories do show
substantial differences in death rates across the six cells for virtually all of the clusters. LogSc Models were used to assess the relative effects of various clinical factors on the probability of death. These models were constructed for each of the 64 DRG clusters which had relatively high death rates (5% or more). Thus, we controlled for the patient’s immediate problem or illness by modelling each DRG cluster separately. In our Logistic Models the dependent variable is discharge status (alive or dead). The independent variables in the logistic regressions are proxies for the patient’s condition at the time of admission to the hospital. The construction of the independent variables for the Logistic Models was a two-step process: 1. Preliminary assessments were performed to estimate (empirically) the relative risk of each surgical procedure and each secondary diagnosis within each of the Major Diagnostic Categories (MDC). This preliminary assessment allowed us to assign a risk score to each procedure and each secondary diagnosis within each MDC. This approach was necessary because the risk associated with a given secondary diagnosis varied across the different MDCs. 2. The risk factors associated with each procedure and with each secondary diagnosis were then placed into a series of tables, which could be used to look up the risks associated with these variables for each patient record. The next step was to develop the Logistic Models. Within each of the 64 clusters the following independent variables were included as proxies for the patient’s condition when admitted to the hospital: - patient’s age - patient’s race - risk of death associated with the principal diagnosis, if more than one principal diagnosis is possible within the DRG - risk of death associated with first operative procedure (surgical patients only) - whether there were any secondary diagnosis - presence of any cancer except skin cancer as a secondary diagnosis - risk associated with the comorbidity having the highest risk of death (excluding complications) - number of secondary diagnoses (except complications) where the risk of death was greater for the secondary diagnosis than the overall risk associated with the DRG cluster itself. The overall risk of death for each patient is thus derived from the combination of the risks associated with the patient’s primary diagnosis, principal procedure, age, and certain secondary diagnoses. Using either the Contingency Table Models or the Logistic Models, as appropriate, we were able to calculate the probability of death for any given patient. By processing patient records through the models and accumulating the probabilities of death for groups of cases, we can thus aggregate patients, and then predict the
8
number of deaths that would have occurred had this group of patients been given ‘average’ care (standardized to the six million observations we used to derive risks). Risk-Adjusted Readmissions Index (RARI) The Risk-Adjusted Readmissions Index (RARI) was developed in a manner parallel to the Contingency Table approach used in the Risk-Adjusted Mortality Index (RAMI). For RARI, however, the dependent variable is whether an unanticipated readmission to the same hospital occurred within 30 days of discharge. Clearly many readmissions are scheduled, and thus do not represent poor hospital performance. Since we were interested in using the RARI as a measure of adverse outcomes, we excluded those types of m-admissions which would ordinarily be either scheduled (bilateral elective surgery; chemotherapy; diagnostic admission followed by surgical admission) or unavoidable (multiple admissions for AIDS patients, cancer patients, etc.). Our clinical consultant compiled a list of such exclusions, which can be obtained from the authors, by request. In addition, cases that were transferred to another short-term hospital, cases that died during the first admission, and newborns were excluded from the RARI. After cases were grouped into DRG clusters, 2 x 3 Contingency Tables (Table 1) were constructed for each DRG cluster, based on the age of the patients and whether comorbidities or complications occurred during the original stay. (Note that in RARI we included complications as risk factors for readmissions.) We then calculated the probability of readmission for each cell of each DRG cluster model. This was done empirically, using indirect standardization from the large 1983 CPHA database. It should be noted that not all of the hospitals in the six million database used unit record numbering, which was necessary to link episodes of hospitalization. Therefore, a somewhat smaller subset of hospitals was used to model readmissions. Once the 2 x 3 tables were calculated for each DRG cluster, we used the tables to estimate the number of readmissions that would normally occur for any hospital, given its case mix and case complexity, i.e., the distribution of patients across DRG clusters, and age/CC distribution within each DRG cluster. Table 1 Expected death rates by age and comorbldlty DRG cluster 72: coronaty bypass with awdlac catheterlzatlon
AkF
No comorbidity
~65 65-69 70+
1.3 ;:;:
With comorbidities Z-f 14:1
9
The Risk-Adjusted Compllcatlons Index (RACI) The Risk-Adjusted Complications Index (RACI) was constructed in a manner similar to the RARI, described in the previous section. Once again 2 x 3 Contingency Tables were used within each cluster, but this time the dependent variable was whether or not a complication occurred during the hospital stay. The independent variables were age and the presence or absence of comorbidities. (Note that the presence or absence of complications was not used as an independent variable for RAM1 or IUCI, but was for RARI.) The list of complications we considered was developed by our clinical consultant to represent many postsurgical and postdelivery complications. We excluded newborns, all cases that died, and all cases that were transferred to another short-term hospital. Thus, the RACI represents an effort to look at complications that occurred during the hospital stay, but that did not result in death. Once the three indices were constructed, individual patient records could be ‘scored’, i.e., three values could be assigned to each case: - The RAM1 score, which is the probability of death, given the patient’s risk factors; - The RARI score, which is the probability of an unscheduled readmission occurring within 30 days of discharge, given the patient’s risk factors; and - The RACI score, which is the probability of any of a group of postsurgical or post-delivery complications occurring, given the patient’s risk factors. For example, an 80-year-old patient who was hospitalized for a major large bowel procedure with secondary diagnoses of diabetes and hypertension might have the following scores: probability of death (RAMI) = 0.23; probability of readmission to the same hospital within 30 days (RARI) = 0.31; probability of post-surgical complications (RACI) = 0.19. Once the three scores are calculated for each record, one can then aggregate the individual (patient) records in any way that one wishes. By comparing the actual (observed) rates of these adverse events to the predicted rates, one can determine whether the ratio of actual to predicted events is higher than expected, as expected, or lower than expected. Thus, we have three different risk-adjusted measures of hospital quality which, in fact, measure three different (uncorrelated) aspects of hospital performance [34]. These measures are constructed from existing data sets, and are thus easy and inexpensive to use. Uses of the risk-adjusted
indices
If we assume that we may soon have uniform classification and coding systems across the service areas of hospitals (a situation that may extend crossnationally with the adoption of the ED-10 classification system by many
10
nations over the next several years), and that the coding for this classification is reliable, then our indices may be used in many ways, including: (a) Identifying patient records for peer review within a hospital
Brennan et al. [351found that nearly four percent of hospitalizations result in adverse events, and that about 30% of these events are due to negligence. The indices may be used to monitor overall hospital rates of adverse outcomes, and to control preventable adverse events. For example, once the expected values of RAMI, RARI, and RACI are inserted on each patient’s record, one can print out a listing of all cases where the expected value was low, but the adverse outcome did in fact occur. This listing should be an excellent tool for identifying cases for peer review within a hospital. (b) Comparing variations in outcomes across various subgroups of physicians, services, or hospitals The indices may be used to identify those physicians, services, or hospitals
that consistently have poorer outcomes than expected. The individuals, units, or hospitals identified in this manner may then be targeted for in-depth record review and intervention, if needed. (c) Comparing outcomes in various regions of a country Roos et al. [36] have identified large variations in physician practice patterns
in different regions of a country, even after adjustment for patient characteristics. Such differences exist within countries as well as among countries [371, and are supposedly due to differences in physician judgement. Practice pattern variations may be modified through mechanisms such as Consensus Conferences [38] and technology assessment studies [39], but we suspect that information on variations in outcomes will be needed to change practice patterns permanently. While some information on variations in outcomes may be provided eventually by randomized clinical trials [40], the indicators discussed in this paper can be used to identify differences in outcomes between high and low use areas for high risk, high cost, or high discomfort procedures such as angiography, endoscopy, and endarterectomy, where there is substantial disagreement between expert physicians on both the theoretical and practical indications for use [41]. (d) Monitoring changes in outcomes following a change in the delivery system
Changes in programs, funding, or organization of health services may influence quality, and raise questions about efficiency/quality/equity tradeoffs. Such questions may be more easily and inexpensively examined if the hospitals involved have risk-adjusted outcome indicator scores routinely encoded on their patient records. Some issues related to current or proposed changes in the Dutch, Danish, and British health care systems may serve to illustrate the usefulness of risk-adjusted outcome indices for health policy assessment.
11
The Dekker Report in Holland recommended increased competition, cost control, and decentralized decision-making [8]. Are the efficiencies from this change bought at the price of decreased quality of care? Are the hospitals that do well financially, providing care of less quality? Are they skimming patients? that is, are undesirable patient groups being limited access to these institutions, and as a result receiving care later with poorer outcomes? Mammography screening was recently implemented in several Danish counties [42]. Among the issues raised by one chief of surgery was the risk of breast caucer surgery patients ‘crowding out’ other patients. He was concerned that the increase in surgery and biopsies required by new cases found by screening would over stress the already strained capacity of the main referral hospital, and other patients would suffer. Was he right? Do we see an increase in acuity of other surgical patients? Is there an increase in death rates? Do we see an effect on the complication rate? And are these effects spilling over to the less specialized hospitals in the region? Will the changes in Britain resulting from the 1989 white paper [43] influence outcomes of care for patients served by hospitals who are in the self-governing group, or those in budget holding practices? If waiting lists are getting longer for some diseases and procedures, is this a&cting outcomes once patients receive services? Are they sicker when they get admitted, and do they have a higher death and readmissions rate? Many of these countries are experiencing changes in their health care systems that may exacerbate variations in access and quality, such as changes toward decentralixed decision-making [44,451, increased privatization [46l, increased competition for patients or resources [47J, deregulation and use of market mechanisms [48,49], and increased emphasis on accountability for resource use and outcomes [50-521. These changes may mean that the traditional information management systems used by hospitals for assuring access, maintaining quality, managing costs, maximSng efficiency, and assuring system responsiveness may be inadequate to the tasks required of them in the future. Indeed, these information management systems may have been inadequate all along. The indicators and methods used for program evaluation and technology assessment need to be improved if the contemplated changes are to have a greater chance of being successful. The risk-adjusted measures may prove to be valuable tools for these types of analyses. (e) Measuring the size of the gap between the efficacv of a technology (as determined through rmtdomized controlled trials under stringent protocols, or as determined by expert clinical consensus panels) and the effectiveness of thti technology once it is imported, and then used un&r true practice conditions Policy, administrative and clinical decision-makers at all levels need information that allows them to increase the use of rational decisionmaking in order to manage changes successfully [53,54]. In order to improve hospital performance, many decision-makers are choosing to import some of their hard and soft medical technology, management technology, and
12
financing methods from abroad, with many of these innovations coming from the United States [55J. An increase in the diffusion of innovation across national borders is especially evident in the European Community (EC) countries, as a result of the intensified assessment and planning which is taking place prior to 1992. It is, however, essential to use hospital indicators and quality monitoring systems to evaluate the technical innovations imported from abroad. If this is not done, there is a risk of inadvertently importing less effective technological, management, or payment innovations. This risk is especially high if a change originated in the US, since the US system does not base patient access to new technology on the effectiveness of that technology. Instead, patient access to new technology in the US is determined by the patient’s ability to pay, either by having insurance benefits available or by paying directly (out-of-pocket). Thus, in the US innovations are disseminated through the marketplace, rather than based on criteria of effectiveness and cost used in many other systems
WI. Whenever a health care system outside of the US is considering the adoption of technology that could result in changes in access or quality, it is especially important to have a system of indicators in place to monitor the effects of such changes. Data derived from controlled clinical trials and/or consensus conferences provide us with valuable information on the efficacy of a procedure, i.e., its outcomes under ideal practice conditions. While this is useful information, clearly there are differences in outcomes when one departs from the ideal. In particular, we need to know if various technologies or procedures that look promising in clinical trials then Mfil their promise under different practice conditions. For example, can a procedure be ‘exported’ successfully to another setting where the healthcare personnel are trained differently, or where patients have delayed treatment for many months or years? These types of questions can be answered empirically, once we are able to control for certain patient risk factors and different mixes of patients. (f) Analyzing differences in outcomes across nations
Since the health care delivery systems differ across nations in many respects (including the methods of financing care, the methods of determining eligibility for certain services, and the technologies that are used for diagnosis and treatment), one might regard these differences as a sort of natural experiment. Using the risk-adjusted indices, one could potentially analyze differences in outcomes across nations, after controlling for differences in patient risk factors. We could determine if patients in some nations are sicker when they are admitted to the hospital, and if so, what effect this has on outcomes. We could also assess differences in outcomes when different technologies are used, controlling for certain patient risk factors.
13
Conclusions It is doubtful that the share of national expenditures consumed by health care in developed nations will be allowed to increase in the foreseeable future. Thus, issues of efficient provision of high quality care will become increasingly important. We must remember that while efficiency measures relate resource inputs to outputs, in health care we do not value health care outputs, only outcomes. The use of valid measures of risk-adjusted outcomes of medical care is important for improving the quality of the care provided. The real benefits of widespread use of such indicators may, however, lie in the fact that they enable us to keep our attention on the overriding goal of health care: what is the best way to produce better health care outcomes?
Acknowledgement We would like to acknowledge the contribution of Dale Schumacher, M.D., President of the Commission on Professional and Hospital Activities in Ann Arbor, Michigan. Dr. Schumacher was most helpful in the development of the ideas presented in this paper.
References 1 U.S. Bipartkn Commission on Comprchcmivc Health Care, A call for action - The Pepper Commission, fmal report, !kptembcr 1990, S. Prt. 101-114, U.S. Government Printing Office, Washington, 1990. 2 Kampcr-Jocrgenaen, F., Cau#s of differences in utilization of health services, Scandinavian Journal of social Medicine, 30 (1984) 57-66. 3 Schkbcr, G. and Poullicr, G., International health spending and utilization trends, Health AfTairs, 7 (1988) 105-l 12. in medical care practices, Health Care Financing 4 McPherson, K., International ditkcnw Review Annual Supplement (1989) P-20. 5 Ncuhauscr, D., Budgetary incentives for the appropriate use of medical technology, Conference paper, World Health Organkation Regional Office for Europe, Koln and Dusseldorf 8-10 October 1984. 6 Andrcasen, P.B., Consensus conferences in different countries: aims and pcrspectiv~~, International Journal of Technology Assessment in Health Care, 4 (1988) 305-308. 7 France, F.H.R., AIban, A., Barber, B. et al., Hospital comparisons using a Euro health data base for resource management and strategic planning, Health Policy, 17 (1991) 165-177. in the Dutch health care 8 Kirkman-Liff, B. and van de Vcn, W.P.M.M., Improving eftkhq system: current innovations and future options, Health Policy, 13 (1989) 35-53. 9 Abel-Smith, B., editor, European Healthcare Analysis, Eurocare (1989). 10 PhiIl@s, D. and Kawachi, I., Resource manamcnt in New Zealand hospitals: the RUS system - an inustration of its applications, Health Policy 16 (1990) 19-25. 11 Moorhead, MA., Asusing quality of care: another step fomwrd, American Journal of Public Health, (1989) 415-416. 12 Klein, R., Rationing health care, British Medical Journal, 289 (1984) 143144. 13 Simpson, K.N. and Vuuy, J.E., National indicators for health for a& Social Indicators Research, 20 (1988) 533548.
14 14 Birch, S. and Maynard, A., Performance indicators and performance assessment in the UK National Health Service: implications for management and planning, Jntemational Journal of Health Planning and Management, 1 (1986) 143-156. 15 Wemrberg, J.E., et al., Use of claims data systems to evaluate health care outcomes: mortality and reoperation following prostatectomy, Journal of American Medical Association, 257 (1987) 933. 16 Flood, A.B., et al., Effectiveness in professional organizations: the impact of surgeons and surgical staE organizations on the quality of care in hospitals, Health Servkes Research, 17 (1982) 341. 17 LUR, H.S. and Hunt, S.S., Evaluating individual hospital quality through outcome statistics, Journal of American Medical Association, 255 (1986) 2780. 18 Knaus, W.A., et al., An evaluation of outcome from intensive care in major medical centers, Annual Internal Medicine, 104 (1986) 410. 19 KeBy, J.V. and Hellinger, F.J., Physician and hospital factors associated with mortality of surgical patients, Medical Care, 24 (1986) 785. 20 Hebel, J.R., et al., Assessment of hospital performance by use of death rates: a recent case history, Journal of American Medical Association, 248 (1982) 3131. 21 Fethke, C.C., Smith, I.M. and Johnson, R., ‘Risk’ factors a&cting readmission of the elderly into the health care system, Medical Care, 24 (1986) 429. 22 Anderson, G.F. and Steinberg, E.P., Predicting hospital readmissions in the Medicare population, Inquiry, 22 (1985) 251. 23 Gooding, J. and Jette, A.M., Hospital readmissions among the elderly, Journal of American Geriatric Society, 33 (1985) 595. 24 Riley, G. and Lubitz, J., Outcomes of surgery in the Medicare aged population: rehospitalization after surgery, Health Care Financing Review, 8 (1986) 23. 25 Zook, C.J., Slavickis, S.F. and Moore, F.D., Repeated hospitalization for the same disease: a multiplier of national health costs, Milbank Quarterly, 58 (1980) 454. 26 Holloway, J.J., Thomas, W. and Shapiro, L., Clinical and sociodemographic risk factors for readmission of Medicare beneficiaries, Health Care Financing Review, 10 (1988) 27. 27 Smith, D.M., Norton, J.A. and McDonald, C.J., Nonelective readmissions of medical patients, Journal of Chronic Disease, 38 (1985) 213. 28 Roos, L.L., Cageorge, S.M., Au&en, E. and Lohr, K.N., Using computers to identify complications atIer surgery, American Journal of Public Health, 75 (1985) 1288. 29 Brook, R.H., Fii, A., Solomon, D.H., Kosecoff, J. and Chassin, M., A review of the literature on cholecystectomy: fmdings, complications, utilization rates, costs, efiicacy, and indications, Santa Monica, CA: Rand Corporation, R-3204/2CWF/PMT/HF, 1975. 30 Chilton, C.P., Morgan, R.J., England, H.R., Paris, A.M.I. and Blandy, J.P., A critical evaluation of the results of transurethral resection of the prostate, British Journal of Urology, 50 (1978) 542. 31 Adams, D.F., Fraser, D.B. and Abrams, H.L., The complications of coronary arteriography, Circulation, 48 (1973) 609. 32 DesHamais, S., et al., The risk-adjusted mortality in&x: a new measure of hospital performance, Medical Care, 25 (1988) 1129. 33 DesHamais, S.I., McMahon Jr., L.F., Wroblewski, R.T. and Hogan, A.J., Measuring hospital performance - the development and validation of risk-adjusted indexes of mortality, readmission, and complications, Medical Care, 28 (1990) 1127-1141. 34 DesHarnais, S.I., McMahon, L.F. and Wroblewski, R., Measuring outcomes of hospital care using multiple risk-adjusted indexes, Health Services Research, forthcoming October 1991. 35 Bremran, T.R., Leape, L.L., Laird, N.M., et al., Jncidence of adverse events and negligence in hospital&d patients, New England Journal of Medicine, (1991) 376376. 36 Roos, N.P., Wennberg, J.E., Malenka, D.J., Fisher, E.S., McPherson, K., An&men, T.F., Cohen, M.M. and Ramsey, E., Mortality and reoperation atIer open and transurethralresection of the prostate for benign prostatic hyperplasia, New England Journal of Medicine, 320 (1989) 1120-1124.
15 37 Lobr, K.N., Understanding variations in the use of mrvicex are there clinical explanations? Health Affairs, 3 (1984) 139-148. 38 Goodman, C. and Baratz, S.R., Improving consensus development for health technology assessment an international perspective, Council on Health Care Technology, Institute of Medicine, National Academy Press, Washington, D.C. (1990). 39 Mosteller, F. and Burdick, E., Current issues in health care technology assessment, International Journal of Technology Assessment in Health Care, 5 (1989) 123-136. 40 Paintin, D.B., Effective care in pregnancy and childbirth, British Journal of Obstetrics and Gynaecology, 97 (1990) 967-973. 41 Park, R.E., Fink, A., Brook, R. et al., Physician ratings of appropriate indications for three procedures: theoretical indications vs. indications used in practice, American Journal of Public Health, 79 (1989) 445-447. 42 Rapport vedroerende org amsering af brystkraeRscrcening i Vejle amt, Vejle Amt Sundhedsforvaltningen, Vejle, Denmark. Mimeographed (1990). 43 Weiner, J.P. and Ferriss, D.M.. GP budget holding in the UK: lessons from American HMGs, Health Policy, 16 (1990) 209-220. 44 &tad, J.I., Health services and decentral.ized government: tbe case of primary health services in Norway, International Journal of Health Sewices, 20 (1990) 545-559. 45 Brogren, P. and Brommels, M., Central and local control in nordic health care: the public organization spectrum revised, International Journal of Health Planning and Management, 5 (1990) 27-39. 46 Jansen, R. and J. van der Ma&, Privatization in health care: concepts, motives and policies, He&b Policy, 14 (1990) 191-202. 47 Maarse, J.A.M., et al., Changing relations between hospitals and primary health care: new challenges for hospital management, International Journal of Health Planning and Management, 5 (1990) 53-57. 48 van den Heuvel, R., 1992 and health insurance in the European community, paper presented to a conference on Health Care in Europe after 1992, Erasmus University, Rotterdam, October 1618, 1989. 49 Bakker, D.H., The district concept for primary health care planning: attempts at implementation in the Netherlands, Health Policy, 13 (1989) 55-64. 50 Lister, J. The politics of medicine in Britain and the United States, New England Journal of Medicine, 315 (1986) 168-174. 51 Girakies, M.D.R., The equity and efficiency principle in the financing system of the NT-IS in Portugal, Health Policy, 14 (1990) 1328. 52 Brommeh, M., A ‘stethoscope’ for the physician manager conditions for clinical accountability in tbe Nordic setting, International Journal of Health Planning and Management, 2 (1989) 97106. 53 Banta, H.D. and Andreasen, P.B., The political dimension in health care technology assessment programs. International Journal of Technology Assessment in Health Care, 6 (1990) 115-123. 54 Simpson, K. and Snider, L., Informing mammography coverage &bate: results of meta analysis, computer modeging, and issue analysis, International Journal of Technology Assessment in Health Care, 7 (1991) forthcoming. 55 Drummond, M. and Maynard, A., Efficiency in the National Health Service: lessons from abroad, Health Policy, 9 (1988) 59-74. 56 Churchill, L.R., Rationing health care in America: perceptions and principles of justice, University of Notre Dame Press, Notre Dame, Indiana, 1987.
1
HPE 00459
Indices for monitoring hospital outcomes in developed countries Susan I. DesHarnais and Kit N. Simpson Department of Health Policy and Administration, School of Public Health, Univemity of North Carolina at Chapel Hill, North Carolina, U.S.A. Accepted
6 November
1991
Summary We discuss some of the chsllenges facing hospitals in developed nations, with spedal attention to the need to monitor and evaluate hospital performance. In particular, there is a need for quallty Indicators that measure the effects of treatment, and are rlsk+rdjusted, so that valid comparisons of outcomes can be made across hospitals that treat different types of patients. Until recently, only very crude quality indicators have been available for comparing the performance of different hospitals. We describe three risk-adJusted Indices for comparing the outcomes of hospltsl care, focuslng on the construction and vslldation of these measures. We discuss the uses of these tools for identlfylng problems and for monltodng outcomes of cam within a hospital, Including screening medlcal records for peer review, ldentlfylng variations in outcomes ecross various subgroups of physicians, and comparing changes in outcomes following various changes In the delivery system. Possible apptlcstions at the regions4 national and international levels are then discussed, with special emphasis on the use of these tools for messuring the size of the gap between the efllcacy of a technology, as determined through rsndomlxed controlled trials under stdngent protocols, and the ef&tlveness of the same technology once It Is exported, and then used under true pmctlca condltlons In another country. Quallty measurement; Hospital performance; Outcome; Monitoring; Risk adJustment
Addtws for comsspond~ Susan I. DesHamais, Ph.D. Health Policy and AdmlnistmtionCE37400, McGavran-GreenbegHall, UNC Chapel Hill, Chapel Hill, NC 27599-7400, U.S.A.
2
Introduction Health care experts from virtually all developed nations are reporting changes in the health care systems in their home countries. Changes are occurring in the structure and financing of most systems, and there are also changes in disease prevalence in many countries. The or&r of the day in health care sectors seem to be rapid, constant change, with a majority of these changes related to the provision of hospital care. Physicians are adopting new medical treatment technologies, economists are trying to assure efficient allocation of resources, managers are working to improve their hospitals’ effectiveness, and politicians are cutting budgets. The above actors are often assuming that hospitals are robust systems for producing medical care, and that continuous ‘tinkering’ will not cause unacceptable changes in the quality of care, with quality being defined by us throughout this discussion as health outcomes, i.e., the effects of treatment on the health status of the patients. Recent U.S. reports of grossly decreased access and increased costs [l] demonstrate, however, that the balance between cost, quality and access to hospital care is quite delicate. Belatedly, we are recognizing the need to develop systems for monitoring changes in the outcomes of hospital care, so that we can identify, analyze, and correct unacceptable variations in performance across hospitals.
Challenges facing hospitals in developed nations Hospitals across developed nations are being challenged to respond to major health care issues: the aging of the population; the increase in chronic disease caused by lifestyle factors; the need to incorporate prevention in a system where most available resources are needed for treatment [2]; and the escalating cost of providing care [3]. It is generally recognized that no country is able to afford all the medical care that its citizens need [4]. As a result, hospitals in most countries are under pressure to decrease their total costs without harming the quality of care provided. Faced with limited resources, it is necessary to assure that up-to-date medical technology is purchased and made available, but without bankrupting the system [5,6]. Therefore, decisions must be made on which new technologies to adopt, and how to monitor changes in outcomes of care once new technologies are adopted. The goal in most countries is to develop hospital information systems that provide a parsimonious set of quality indicators which can be used to monitor and manage change. Information is required on resource inputs, the process of providing care, the outputs of the organization, and patient outcomes [7,8]. These indicators must be sensitive, and they must enable hospitals to avoid unacceptable variations in cost, efficiency, effectiveness, and quality. The information system must be capable of providing data for planning and evaluation of care at the various levels of a country, and preferably cross-
3
nationally [9]. Information systems should also be able to provide data that can be used to protect the hospital from unfair comparisons with other institutions. Although hospital information systems have made great improvements in their ability to capture resource utilization and cost data [lo], until recently, it has not been possible to provide comparative data that allow for the monitoring and review of the quality of care provided in different hospitals [ll]. Only very crude indicators of health outcomes have been available for comparison of the health of populations served by different hospitals and systems [12,13]. This inability to measure the outcomes of care has led to a fragmented approach to improving the quality, effticiency and effectiveness of Care.
Governments, elected officials and purchasers of care need valid, yet inexpensive measures of quality, as they attempt to compare the performance of various hospitals. Birch and Maynard from the U.K. specifically state that: “me] deign of existing performance indicators should be changed in order to move towards a more careful consideration of what the hospital is achieving as opposed to simply what it is using. In particular [we need] to usesdata on the quality of a hospital’s output, as indicated by readmission rates and post-opcratin complication rates. Clearly, these rates are of major concern to patients and, hence, are indicators of the degree to which objectives are being satisfied.” [14]
Ideally, we want to compare rates of both positive and negative outcomes across hospitals. We would like to be able to evaluate the effects of hospital care directly, by measuring the changes in patients’ health status following treatment. This, however, is diflicult to do since there is no practical way to obtain data on patient health status before and after treatment for a large national sample of hospitals. While we may not be able to directly measure quality in terms of changes on patient health status following treatment, we are able to compare the rates of certain adverse consequences of treatment across hospitals. Under the assumption that the hospitals with the lower rates of adverse events are producing better patient outcomes, our measures of adverse outcomes are used as proxies for positive measures of outcomes. There are many historical precedents for measuring rates of adverse events, rather than using direct or ‘positive’ measures of quality. Death has been used as an indicator by many investigators, including Wennberg, Flood, Luft, Knaus, Kelly, and Hebel [15-201. Readmissions have been examined by Fethke, Anderson, Good@, ROOS,Riley, Zook, Holloway, and Smith, [21-281 among others. Complication rates have been studied by ROOS,Brook, Chilton, Adams [28-311 and others. The system of indicators presented in this paper is configured so that it provides three important ‘negative’ indicators of hospital quality: death rates, readmission rates, and complication rates. These three outcome measures are adjusted for patient risk at the time of admission. This risk-adjustment is essential for valid comparisons across hospitals. While differences in patient outcomes across hospitals can be viewed as partially a result of hospital
4
performance, there are several other factors that also influence outcomes, Differences in outcomes across hospitals or over time may be due to variations in the types of patients treated; the severity of the illness which is designated as the principal diagnosis; the types and complexity of the patients’ comorbidities; the patients’ age distribution, and the social and financial condition of the patients treated at a given hospital. In order to measure provider performance with some accuracy, one must control for these other factors which affect outcomes. If a population group treated in a given hospital has higher than average risk factors, then the rates of adverse outcomes would be expected to be high for these patients. While it is impossible to control completely for all risk factors, given our existing data sets and measurement tools, it is possible to use existing data to approximate some of these control variables. A method for doing so is described next. Indices of risk-adjusted outcomes: mortality, readmissions, and complications In two recent articles we described in detail the methods used to construct the Risk-Adjusted Mortality Index (RAMI) [32], and the construction and validation of the three measures of hospital performance reported in this paper [33]. In order to construct indices of hospital performance, two separate but related problems had to be solved: it was essential to take into account differences across hospitals in the types of patients treated (case mix); and it was necessary to use an appropriate severity measure to take into account differences across hospitals in the severity of illness within each of the disease categories (case complexity). Unless these adjustments are made, one cannot make valid comparisons across hospitals, since those hospitals treating more ‘difficult’ cases will appear to have worse outcomes. In developing these indices we used information contained in an existing database to develop proxies for many of the factors other than provider performance which are related to health outcomes. By means of indirect standardization to the large, national all-payor database collected by the Commission on Professional and Hospital Activities (CPHA), we obtained empirical information to calculate and model the effects of various risk factors for each type of patient. Even though some risk factors are only approximated by this method, the approach is useful. The models are practical, insofar as they use existing databases, and they are comprehensive, insofar as they measure hospital performance for all payors and virtually all types of cases treated in a given hospital. Three different risk-adjusted indices of hospital performance were developed: - The Risk-Adjusted Mortality Index (RAMI) - The Risk-Adjusted Readmission Index (RARI) - The Risk-Adjusted Complications Index (RACI) The risk factors for each of these quality measures were originally modeled
5
using a national CPHA data file, containing six million cases from 1983, from 776 short-term U.S. general hospitals. This cohort of 776 CPHA hospitals was generally representative of the 5663 US hospitals in the universe. However, the for-profit hospitals and Southern hospitals were somewhat underrepresented, while major teaching hospitals are overrepresented. Nevertheless, it is important to realize that the discharges are the units of analysis for modelling the risk of each outcome. There is reason to believe that these six million discharges used for modelling risk factors are representative of all discharges treated in the U.S.. The three indices were later rebased using data from 1988. The following cases were excluded from the analysis file: - all cases transferred to other short-term hospitals (referral centers, specialty hospitals). - all cases of newborn infants. Since most critically ill newborns are transferred to neonatal centers, it was not possible to model risk factors for this group in a valid manner. Moreover, newborn weight, a critical variable in predicting infant deaths, was not available. - all cases with stays of less than one day who were discharged alive (presumed to be outpatient surgery and other outpatient cases). We ,then used an empirical approach to model the risk of death, readmission, and complications associated with each reason for admission. Since it was apparent that the patient characteristics associated with the risk of each outcome would vary from one disease to another, separate models were developed for each disease category. The first step in constructing each measure was the aggregation of Diagnosis Related Groups (DRGs) into clusters which group those DRGs with the same clinical condition. This clustering was necessary for our purposes because some of the factors associated with an increased risk of death, readmission, or complication within a clinical condition were used as the basis for DRG divisions (age, comorbidities/complications). For instance, DRGs 89, 90, and 91 are all simple pneumonia and pleurisy, but the DRGs differ by the age of the patient. We needed to regroup the DRGs into clusters, to determine how age, comorbidities, and other factors were associated with an increased risk of adverse outcomes within each disease condition. This clustering procedure resulted in 316 categories, each representing a different clinical condition. Once this clustering was done, we proceeded to develop the three different indices: RAMI, RARI, and RACI. The risk models were based on clinical factors which were recorded in the CPHA dataset. CPHA abstracts contain, among other items, the following data elements for each patient discharge: age, race, hospital identification, dates of admission and discharge, discharge status (alive, dead), principal diagnosis, secondary diagnoses (up to I l), and principal surgical procedure.
6
Construction of the three indices The Risk-Adjusted Mortality Index (RAM) Two different types of models were employed when modelling the risk of mortality for each patient. This was done so that the most appropriate technique would be used for each disease or condition. These models are: - The Contingency Table Model: For each of the 252 DRG clusters where the death rate was less than 5% (83% of all discharges; 28% of all deaths), a Contingency Table Model was developed, since the estimating procedures for logistic regressions are inadequate in this situation. - The Logistic Model: For each of the 64 clusters where the death rate was 5% or more (17% of all discharges; 72% of all deaths), a logistic model was developed. The Contingency Table Model is based on deriving a separate 2 x 3 table for each of the DRG clusters with relatively low death rates. The age of patients (O-64; 65-74; 75+) and the presence of comorbidities were used as classifying variables, since these were the best predictors of death in the more complicated models that we tried to construct. The death rate was determined empirically within each of the six cells. (See Table 1 for an example of a contingency table.) Complications were excluded as risk factors, because we wanted to measure the severity of a patient’s illness at the time of admission, in order to assess the risk of the patient’s primary medical problem and related comorbidities prior to medical care intervention. However, the HCFA list of comorbidities and complications (CCs) contains both comorbidities (secondary diagnoses present at the time of admissions) and complications (conditions that arose during the course of treatment). Because we wanted to measure comorbidities, but not complications, we attempted to separate the two. Our medical consultant designated 70 of the conditions on the HCFA CC list as most likely to be complications of surgery, or iatrogenic events. This list includes problems such as transfusion reactions, accidental operation lacerations, etc. This was a conservative approach, insofar as certain conditions such as pneumonia and urinary tract infections, which may or may not be complications, were assumed to be comorbidities. Because the structure of the ICD-PCM codes explicitly identifies surgical and OB/GYN complications, but not medical complications, we’were unable to distinguish whether some medical problems arose during treatment. Therefore, hospitals were credited for having sicker patients if we could not be sure whether a condition was present at the time of admission. However, obvious complications were removed as ‘risk factors’ in the models, since hospitals should not be credited for a more complex case mix on the basis of a high frequency of iatrogenic events among their patients. Thus, the Contingency Table Model was used to distinguish six risk categories, within each of the relatively low death rate clusters, based on three age groups and the presence of comorbidities. These six risk categories do show
substantial differences in death rates across the six cells for virtually all of the clusters. LogSc Models were used to assess the relative effects of various clinical factors on the probability of death. These models were constructed for each of the 64 DRG clusters which had relatively high death rates (5% or more). Thus, we controlled for the patient’s immediate problem or illness by modelling each DRG cluster separately. In our Logistic Models the dependent variable is discharge status (alive or dead). The independent variables in the logistic regressions are proxies for the patient’s condition at the time of admission to the hospital. The construction of the independent variables for the Logistic Models was a two-step process: 1. Preliminary assessments were performed to estimate (empirically) the relative risk of each surgical procedure and each secondary diagnosis within each of the Major Diagnostic Categories (MDC). This preliminary assessment allowed us to assign a risk score to each procedure and each secondary diagnosis within each MDC. This approach was necessary because the risk associated with a given secondary diagnosis varied across the different MDCs. 2. The risk factors associated with each procedure and with each secondary diagnosis were then placed into a series of tables, which could be used to look up the risks associated with these variables for each patient record. The next step was to develop the Logistic Models. Within each of the 64 clusters the following independent variables were included as proxies for the patient’s condition when admitted to the hospital: - patient’s age - patient’s race - risk of death associated with the principal diagnosis, if more than one principal diagnosis is possible within the DRG - risk of death associated with first operative procedure (surgical patients only) - whether there were any secondary diagnosis - presence of any cancer except skin cancer as a secondary diagnosis - risk associated with the comorbidity having the highest risk of death (excluding complications) - number of secondary diagnoses (except complications) where the risk of death was greater for the secondary diagnosis than the overall risk associated with the DRG cluster itself. The overall risk of death for each patient is thus derived from the combination of the risks associated with the patient’s primary diagnosis, principal procedure, age, and certain secondary diagnoses. Using either the Contingency Table Models or the Logistic Models, as appropriate, we were able to calculate the probability of death for any given patient. By processing patient records through the models and accumulating the probabilities of death for groups of cases, we can thus aggregate patients, and then predict the
8
number of deaths that would have occurred had this group of patients been given ‘average’ care (standardized to the six million observations we used to derive risks). Risk-Adjusted Readmissions Index (RARI) The Risk-Adjusted Readmissions Index (RARI) was developed in a manner parallel to the Contingency Table approach used in the Risk-Adjusted Mortality Index (RAMI). For RARI, however, the dependent variable is whether an unanticipated readmission to the same hospital occurred within 30 days of discharge. Clearly many readmissions are scheduled, and thus do not represent poor hospital performance. Since we were interested in using the RARI as a measure of adverse outcomes, we excluded those types of m-admissions which would ordinarily be either scheduled (bilateral elective surgery; chemotherapy; diagnostic admission followed by surgical admission) or unavoidable (multiple admissions for AIDS patients, cancer patients, etc.). Our clinical consultant compiled a list of such exclusions, which can be obtained from the authors, by request. In addition, cases that were transferred to another short-term hospital, cases that died during the first admission, and newborns were excluded from the RARI. After cases were grouped into DRG clusters, 2 x 3 Contingency Tables (Table 1) were constructed for each DRG cluster, based on the age of the patients and whether comorbidities or complications occurred during the original stay. (Note that in RARI we included complications as risk factors for readmissions.) We then calculated the probability of readmission for each cell of each DRG cluster model. This was done empirically, using indirect standardization from the large 1983 CPHA database. It should be noted that not all of the hospitals in the six million database used unit record numbering, which was necessary to link episodes of hospitalization. Therefore, a somewhat smaller subset of hospitals was used to model readmissions. Once the 2 x 3 tables were calculated for each DRG cluster, we used the tables to estimate the number of readmissions that would normally occur for any hospital, given its case mix and case complexity, i.e., the distribution of patients across DRG clusters, and age/CC distribution within each DRG cluster. Table 1 Expected death rates by age and comorbldlty DRG cluster 72: coronaty bypass with awdlac catheterlzatlon
AkF
No comorbidity
~65 65-69 70+
1.3 ;:;:
With comorbidities Z-f 14:1
9
The Risk-Adjusted Compllcatlons Index (RACI) The Risk-Adjusted Complications Index (RACI) was constructed in a manner similar to the RARI, described in the previous section. Once again 2 x 3 Contingency Tables were used within each cluster, but this time the dependent variable was whether or not a complication occurred during the hospital stay. The independent variables were age and the presence or absence of comorbidities. (Note that the presence or absence of complications was not used as an independent variable for RAM1 or IUCI, but was for RARI.) The list of complications we considered was developed by our clinical consultant to represent many postsurgical and postdelivery complications. We excluded newborns, all cases that died, and all cases that were transferred to another short-term hospital. Thus, the RACI represents an effort to look at complications that occurred during the hospital stay, but that did not result in death. Once the three indices were constructed, individual patient records could be ‘scored’, i.e., three values could be assigned to each case: - The RAM1 score, which is the probability of death, given the patient’s risk factors; - The RARI score, which is the probability of an unscheduled readmission occurring within 30 days of discharge, given the patient’s risk factors; and - The RACI score, which is the probability of any of a group of postsurgical or post-delivery complications occurring, given the patient’s risk factors. For example, an 80-year-old patient who was hospitalized for a major large bowel procedure with secondary diagnoses of diabetes and hypertension might have the following scores: probability of death (RAMI) = 0.23; probability of readmission to the same hospital within 30 days (RARI) = 0.31; probability of post-surgical complications (RACI) = 0.19. Once the three scores are calculated for each record, one can then aggregate the individual (patient) records in any way that one wishes. By comparing the actual (observed) rates of these adverse events to the predicted rates, one can determine whether the ratio of actual to predicted events is higher than expected, as expected, or lower than expected. Thus, we have three different risk-adjusted measures of hospital quality which, in fact, measure three different (uncorrelated) aspects of hospital performance [34]. These measures are constructed from existing data sets, and are thus easy and inexpensive to use. Uses of the risk-adjusted
indices
If we assume that we may soon have uniform classification and coding systems across the service areas of hospitals (a situation that may extend crossnationally with the adoption of the ED-10 classification system by many
10
nations over the next several years), and that the coding for this classification is reliable, then our indices may be used in many ways, including: (a) Identifying patient records for peer review within a hospital
Brennan et al. [351found that nearly four percent of hospitalizations result in adverse events, and that about 30% of these events are due to negligence. The indices may be used to monitor overall hospital rates of adverse outcomes, and to control preventable adverse events. For example, once the expected values of RAMI, RARI, and RACI are inserted on each patient’s record, one can print out a listing of all cases where the expected value was low, but the adverse outcome did in fact occur. This listing should be an excellent tool for identifying cases for peer review within a hospital. (b) Comparing variations in outcomes across various subgroups of physicians, services, or hospitals The indices may be used to identify those physicians, services, or hospitals
that consistently have poorer outcomes than expected. The individuals, units, or hospitals identified in this manner may then be targeted for in-depth record review and intervention, if needed. (c) Comparing outcomes in various regions of a country Roos et al. [36] have identified large variations in physician practice patterns
in different regions of a country, even after adjustment for patient characteristics. Such differences exist within countries as well as among countries [371, and are supposedly due to differences in physician judgement. Practice pattern variations may be modified through mechanisms such as Consensus Conferences [38] and technology assessment studies [39], but we suspect that information on variations in outcomes will be needed to change practice patterns permanently. While some information on variations in outcomes may be provided eventually by randomized clinical trials [40], the indicators discussed in this paper can be used to identify differences in outcomes between high and low use areas for high risk, high cost, or high discomfort procedures such as angiography, endoscopy, and endarterectomy, where there is substantial disagreement between expert physicians on both the theoretical and practical indications for use [41]. (d) Monitoring changes in outcomes following a change in the delivery system
Changes in programs, funding, or organization of health services may influence quality, and raise questions about efficiency/quality/equity tradeoffs. Such questions may be more easily and inexpensively examined if the hospitals involved have risk-adjusted outcome indicator scores routinely encoded on their patient records. Some issues related to current or proposed changes in the Dutch, Danish, and British health care systems may serve to illustrate the usefulness of risk-adjusted outcome indices for health policy assessment.
11
The Dekker Report in Holland recommended increased competition, cost control, and decentralized decision-making [8]. Are the efficiencies from this change bought at the price of decreased quality of care? Are the hospitals that do well financially, providing care of less quality? Are they skimming patients? that is, are undesirable patient groups being limited access to these institutions, and as a result receiving care later with poorer outcomes? Mammography screening was recently implemented in several Danish counties [42]. Among the issues raised by one chief of surgery was the risk of breast caucer surgery patients ‘crowding out’ other patients. He was concerned that the increase in surgery and biopsies required by new cases found by screening would over stress the already strained capacity of the main referral hospital, and other patients would suffer. Was he right? Do we see an increase in acuity of other surgical patients? Is there an increase in death rates? Do we see an effect on the complication rate? And are these effects spilling over to the less specialized hospitals in the region? Will the changes in Britain resulting from the 1989 white paper [43] influence outcomes of care for patients served by hospitals who are in the self-governing group, or those in budget holding practices? If waiting lists are getting longer for some diseases and procedures, is this a&cting outcomes once patients receive services? Are they sicker when they get admitted, and do they have a higher death and readmissions rate? Many of these countries are experiencing changes in their health care systems that may exacerbate variations in access and quality, such as changes toward decentralixed decision-making [44,451, increased privatization [46l, increased competition for patients or resources [47J, deregulation and use of market mechanisms [48,49], and increased emphasis on accountability for resource use and outcomes [50-521. These changes may mean that the traditional information management systems used by hospitals for assuring access, maintaining quality, managing costs, maximSng efficiency, and assuring system responsiveness may be inadequate to the tasks required of them in the future. Indeed, these information management systems may have been inadequate all along. The indicators and methods used for program evaluation and technology assessment need to be improved if the contemplated changes are to have a greater chance of being successful. The risk-adjusted measures may prove to be valuable tools for these types of analyses. (e) Measuring the size of the gap between the efficacv of a technology (as determined through rmtdomized controlled trials under stringent protocols, or as determined by expert clinical consensus panels) and the effectiveness of thti technology once it is imported, and then used un&r true practice conditions Policy, administrative and clinical decision-makers at all levels need information that allows them to increase the use of rational decisionmaking in order to manage changes successfully [53,54]. In order to improve hospital performance, many decision-makers are choosing to import some of their hard and soft medical technology, management technology, and
12
financing methods from abroad, with many of these innovations coming from the United States [55J. An increase in the diffusion of innovation across national borders is especially evident in the European Community (EC) countries, as a result of the intensified assessment and planning which is taking place prior to 1992. It is, however, essential to use hospital indicators and quality monitoring systems to evaluate the technical innovations imported from abroad. If this is not done, there is a risk of inadvertently importing less effective technological, management, or payment innovations. This risk is especially high if a change originated in the US, since the US system does not base patient access to new technology on the effectiveness of that technology. Instead, patient access to new technology in the US is determined by the patient’s ability to pay, either by having insurance benefits available or by paying directly (out-of-pocket). Thus, in the US innovations are disseminated through the marketplace, rather than based on criteria of effectiveness and cost used in many other systems
WI. Whenever a health care system outside of the US is considering the adoption of technology that could result in changes in access or quality, it is especially important to have a system of indicators in place to monitor the effects of such changes. Data derived from controlled clinical trials and/or consensus conferences provide us with valuable information on the efficacy of a procedure, i.e., its outcomes under ideal practice conditions. While this is useful information, clearly there are differences in outcomes when one departs from the ideal. In particular, we need to know if various technologies or procedures that look promising in clinical trials then Mfil their promise under different practice conditions. For example, can a procedure be ‘exported’ successfully to another setting where the healthcare personnel are trained differently, or where patients have delayed treatment for many months or years? These types of questions can be answered empirically, once we are able to control for certain patient risk factors and different mixes of patients. (f) Analyzing differences in outcomes across nations
Since the health care delivery systems differ across nations in many respects (including the methods of financing care, the methods of determining eligibility for certain services, and the technologies that are used for diagnosis and treatment), one might regard these differences as a sort of natural experiment. Using the risk-adjusted indices, one could potentially analyze differences in outcomes across nations, after controlling for differences in patient risk factors. We could determine if patients in some nations are sicker when they are admitted to the hospital, and if so, what effect this has on outcomes. We could also assess differences in outcomes when different technologies are used, controlling for certain patient risk factors.
13
Conclusions It is doubtful that the share of national expenditures consumed by health care in developed nations will be allowed to increase in the foreseeable future. Thus, issues of efficient provision of high quality care will become increasingly important. We must remember that while efficiency measures relate resource inputs to outputs, in health care we do not value health care outputs, only outcomes. The use of valid measures of risk-adjusted outcomes of medical care is important for improving the quality of the care provided. The real benefits of widespread use of such indicators may, however, lie in the fact that they enable us to keep our attention on the overriding goal of health care: what is the best way to produce better health care outcomes?
Acknowledgement We would like to acknowledge the contribution of Dale Schumacher, M.D., President of the Commission on Professional and Hospital Activities in Ann Arbor, Michigan. Dr. Schumacher was most helpful in the development of the ideas presented in this paper.
References 1 U.S. Bipartkn Commission on Comprchcmivc Health Care, A call for action - The Pepper Commission, fmal report, !kptembcr 1990, S. Prt. 101-114, U.S. Government Printing Office, Washington, 1990. 2 Kampcr-Jocrgenaen, F., Cau#s of differences in utilization of health services, Scandinavian Journal of social Medicine, 30 (1984) 57-66. 3 Schkbcr, G. and Poullicr, G., International health spending and utilization trends, Health AfTairs, 7 (1988) 105-l 12. in medical care practices, Health Care Financing 4 McPherson, K., International ditkcnw Review Annual Supplement (1989) P-20. 5 Ncuhauscr, D., Budgetary incentives for the appropriate use of medical technology, Conference paper, World Health Organkation Regional Office for Europe, Koln and Dusseldorf 8-10 October 1984. 6 Andrcasen, P.B., Consensus conferences in different countries: aims and pcrspectiv~~, International Journal of Technology Assessment in Health Care, 4 (1988) 305-308. 7 France, F.H.R., AIban, A., Barber, B. et al., Hospital comparisons using a Euro health data base for resource management and strategic planning, Health Policy, 17 (1991) 165-177. in the Dutch health care 8 Kirkman-Liff, B. and van de Vcn, W.P.M.M., Improving eftkhq system: current innovations and future options, Health Policy, 13 (1989) 35-53. 9 Abel-Smith, B., editor, European Healthcare Analysis, Eurocare (1989). 10 PhiIl@s, D. and Kawachi, I., Resource manamcnt in New Zealand hospitals: the RUS system - an inustration of its applications, Health Policy 16 (1990) 19-25. 11 Moorhead, MA., Asusing quality of care: another step fomwrd, American Journal of Public Health, (1989) 415-416. 12 Klein, R., Rationing health care, British Medical Journal, 289 (1984) 143144. 13 Simpson, K.N. and Vuuy, J.E., National indicators for health for a& Social Indicators Research, 20 (1988) 533548.
14 14 Birch, S. and Maynard, A., Performance indicators and performance assessment in the UK National Health Service: implications for management and planning, Jntemational Journal of Health Planning and Management, 1 (1986) 143-156. 15 Wemrberg, J.E., et al., Use of claims data systems to evaluate health care outcomes: mortality and reoperation following prostatectomy, Journal of American Medical Association, 257 (1987) 933. 16 Flood, A.B., et al., Effectiveness in professional organizations: the impact of surgeons and surgical staE organizations on the quality of care in hospitals, Health Servkes Research, 17 (1982) 341. 17 LUR, H.S. and Hunt, S.S., Evaluating individual hospital quality through outcome statistics, Journal of American Medical Association, 255 (1986) 2780. 18 Knaus, W.A., et al., An evaluation of outcome from intensive care in major medical centers, Annual Internal Medicine, 104 (1986) 410. 19 KeBy, J.V. and Hellinger, F.J., Physician and hospital factors associated with mortality of surgical patients, Medical Care, 24 (1986) 785. 20 Hebel, J.R., et al., Assessment of hospital performance by use of death rates: a recent case history, Journal of American Medical Association, 248 (1982) 3131. 21 Fethke, C.C., Smith, I.M. and Johnson, R., ‘Risk’ factors a&cting readmission of the elderly into the health care system, Medical Care, 24 (1986) 429. 22 Anderson, G.F. and Steinberg, E.P., Predicting hospital readmissions in the Medicare population, Inquiry, 22 (1985) 251. 23 Gooding, J. and Jette, A.M., Hospital readmissions among the elderly, Journal of American Geriatric Society, 33 (1985) 595. 24 Riley, G. and Lubitz, J., Outcomes of surgery in the Medicare aged population: rehospitalization after surgery, Health Care Financing Review, 8 (1986) 23. 25 Zook, C.J., Slavickis, S.F. and Moore, F.D., Repeated hospitalization for the same disease: a multiplier of national health costs, Milbank Quarterly, 58 (1980) 454. 26 Holloway, J.J., Thomas, W. and Shapiro, L., Clinical and sociodemographic risk factors for readmission of Medicare beneficiaries, Health Care Financing Review, 10 (1988) 27. 27 Smith, D.M., Norton, J.A. and McDonald, C.J., Nonelective readmissions of medical patients, Journal of Chronic Disease, 38 (1985) 213. 28 Roos, L.L., Cageorge, S.M., Au&en, E. and Lohr, K.N., Using computers to identify complications atIer surgery, American Journal of Public Health, 75 (1985) 1288. 29 Brook, R.H., Fii, A., Solomon, D.H., Kosecoff, J. and Chassin, M., A review of the literature on cholecystectomy: fmdings, complications, utilization rates, costs, efiicacy, and indications, Santa Monica, CA: Rand Corporation, R-3204/2CWF/PMT/HF, 1975. 30 Chilton, C.P., Morgan, R.J., England, H.R., Paris, A.M.I. and Blandy, J.P., A critical evaluation of the results of transurethral resection of the prostate, British Journal of Urology, 50 (1978) 542. 31 Adams, D.F., Fraser, D.B. and Abrams, H.L., The complications of coronary arteriography, Circulation, 48 (1973) 609. 32 DesHamais, S., et al., The risk-adjusted mortality in&x: a new measure of hospital performance, Medical Care, 25 (1988) 1129. 33 DesHamais, S.I., McMahon Jr., L.F., Wroblewski, R.T. and Hogan, A.J., Measuring hospital performance - the development and validation of risk-adjusted indexes of mortality, readmission, and complications, Medical Care, 28 (1990) 1127-1141. 34 DesHarnais, S.I., McMahon, L.F. and Wroblewski, R., Measuring outcomes of hospital care using multiple risk-adjusted indexes, Health Services Research, forthcoming October 1991. 35 Bremran, T.R., Leape, L.L., Laird, N.M., et al., Jncidence of adverse events and negligence in hospital&d patients, New England Journal of Medicine, (1991) 376376. 36 Roos, N.P., Wennberg, J.E., Malenka, D.J., Fisher, E.S., McPherson, K., An&men, T.F., Cohen, M.M. and Ramsey, E., Mortality and reoperation atIer open and transurethralresection of the prostate for benign prostatic hyperplasia, New England Journal of Medicine, 320 (1989) 1120-1124.
15 37 Lobr, K.N., Understanding variations in the use of mrvicex are there clinical explanations? Health Affairs, 3 (1984) 139-148. 38 Goodman, C. and Baratz, S.R., Improving consensus development for health technology assessment an international perspective, Council on Health Care Technology, Institute of Medicine, National Academy Press, Washington, D.C. (1990). 39 Mosteller, F. and Burdick, E., Current issues in health care technology assessment, International Journal of Technology Assessment in Health Care, 5 (1989) 123-136. 40 Paintin, D.B., Effective care in pregnancy and childbirth, British Journal of Obstetrics and Gynaecology, 97 (1990) 967-973. 41 Park, R.E., Fink, A., Brook, R. et al., Physician ratings of appropriate indications for three procedures: theoretical indications vs. indications used in practice, American Journal of Public Health, 79 (1989) 445-447. 42 Rapport vedroerende org amsering af brystkraeRscrcening i Vejle amt, Vejle Amt Sundhedsforvaltningen, Vejle, Denmark. Mimeographed (1990). 43 Weiner, J.P. and Ferriss, D.M.. GP budget holding in the UK: lessons from American HMGs, Health Policy, 16 (1990) 209-220. 44 &tad, J.I., Health services and decentral.ized government: tbe case of primary health services in Norway, International Journal of Health Sewices, 20 (1990) 545-559. 45 Brogren, P. and Brommels, M., Central and local control in nordic health care: the public organization spectrum revised, International Journal of Health Planning and Management, 5 (1990) 27-39. 46 Jansen, R. and J. van der Ma&, Privatization in health care: concepts, motives and policies, He&b Policy, 14 (1990) 191-202. 47 Maarse, J.A.M., et al., Changing relations between hospitals and primary health care: new challenges for hospital management, International Journal of Health Planning and Management, 5 (1990) 53-57. 48 van den Heuvel, R., 1992 and health insurance in the European community, paper presented to a conference on Health Care in Europe after 1992, Erasmus University, Rotterdam, October 1618, 1989. 49 Bakker, D.H., The district concept for primary health care planning: attempts at implementation in the Netherlands, Health Policy, 13 (1989) 55-64. 50 Lister, J. The politics of medicine in Britain and the United States, New England Journal of Medicine, 315 (1986) 168-174. 51 Girakies, M.D.R., The equity and efficiency principle in the financing system of the NT-IS in Portugal, Health Policy, 14 (1990) 1328. 52 Brommeh, M., A ‘stethoscope’ for the physician manager conditions for clinical accountability in tbe Nordic setting, International Journal of Health Planning and Management, 2 (1989) 97106. 53 Banta, H.D. and Andreasen, P.B., The political dimension in health care technology assessment programs. International Journal of Technology Assessment in Health Care, 6 (1990) 115-123. 54 Simpson, K. and Snider, L., Informing mammography coverage &bate: results of meta analysis, computer modeging, and issue analysis, International Journal of Technology Assessment in Health Care, 7 (1991) forthcoming. 55 Drummond, M. and Maynard, A., Efficiency in the National Health Service: lessons from abroad, Health Policy, 9 (1988) 59-74. 56 Churchill, L.R., Rationing health care in America: perceptions and principles of justice, University of Notre Dame Press, Notre Dame, Indiana, 1987.