- Email: [email protected]
Comorbidity scores for administrative data benefited from adaptation to local coding and diagnostic practices
Journal of Clinical Epidemiology 64 (2011) 1426e1433
Comorbidity scores for administrative data benefited from adaptation to local coding and diagnostic practices Alex Bottle*, Paul Aylin Dr Foster Unit, Department of Primary Care and Public Health, Imperial College London, 1st Floor, Jarvis House, 12 Smithfield Street, London EC1A 9LA, UK Accepted 6 April 2011; Published online 20 July 2011
Abstract Objective: The Charlson and Elixhauser indices are the most commonly used comorbidity indices with risk prediction models using administrative data. Our objective was to compare the original Charlson index, a modified set of Charlson codes after advice from clinical coders, and a published modified Elixhauser index in predicting in-hospital mortality. Study Design and Setting: Logistic regression using two separate years of administrative hospital data for all acute nonspecialist public hospitals in England. Results: For all admissions combined, discrimination was similar for the Charlson index using the original codes and weights and the Charlson index using the original codes but England-calibrated weights (c 5 0.73), although model fit was superior for the latter. The new Charlson codes improved discrimination (c 5 0.76), model fit, and consistency of recording between admissions. The modified Elixhauser had the best performance (c 5 0.80). For admissions for acute myocardial infarction and chronic obstructive pulmonary disease, the weights often differed, although the patterns were broadly similar. Conclusion: Recalibration of the original Charlson index yielded only modest benefits overall. The modified Charlson codes and weights offer better fit and discrimination for English data over the original version. The modified Elixhauser performed best of all, but its weights were perhaps less consistent across the different patient groups considered here. Ó 2011 Elsevier Inc. All rights reserved. Keywords: Comorbidity; Charlson; Elixhauser; Administrative data; Mortality; Risk modeling; Casemix; Hospitals
1. Introduction Patient comorbidity is a common potential confounder in health services research and can be estimated from administrative data sets. It is often included in risk models of mortality and other outcomes in provider profiling [1]. The two most commonly used indices are the Charlson [2] and Elixhauser [3], originally described using International Classification of Diseases, Ninth Revision (ICD-9) with US data. Both were designed for use with databases that cannot distinguish between comorbidities and complications and tried to limit their set of conditions to those of a chronic nature or a few acute conditions that are relatively unlikely to be potential complications. The Charlson index is now more than 20 years old, and both indices need calibrating on the data set of interest (external validation); to our knowledge, very little has been published from the United Kingdom. The weights (or scores) for these two indices
may be inappropriate for the United Kingdom because of differing populations and coding practices. We have to date been using an Australian version of Charlson [1] in our risk adjustment models [4,5], but discussions with clinical coders raised questions over the suitability of some codes when used in the United Kingdom. Recent work in Canada [6] found that a modified Elixhauser [7] outperforms Charlson; the authors also concluded that it needed external validation. Our first aim was to derive new empirical weights based on English hospital administrative data for the original Charlson codes, a modified set of Charlson codes following discussions with coders, and the modified Elixhauser. Given that secondary diagnosis recording is known to vary between hospitals, our second aim was to assess the consistency of recording of codes used in the modified Charlson and modified Elixhauser across admissions belonging to the same patient.
2. Methods * Corresponding author. Tel.: +44 (0)207 3328964; fax +44 (0)207 3328888. E-mail address: [email protected] (A. Bottle). 0895-4356/$ - see front matter Ó 2011 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2011.04.004
We took 2 years of Secondary Uses Service data from Hospital Episodes Statistics (HES), the administrative data
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
What is new? Key findings Despite being around 25 years old and derived in the United States, the original weights for the Charlson comorbidity index perform nearly as well as the weights recalibrated for current English data. Some extra codes used more in England explained more variation, especially at high-coding hospitals, and had higher consistency of recording across admissions for the same patient. The Elixhauser score performed best of all, but its weights varied more across patient groups. What this adds to what was known The Charlson index needed some extra codes for use in England but still performed well, although the Elixhauser index did best. What is the implication Comorbidity index codes and weights may need to be adapted to suit local patterns.
set that covers all admissions to National Health Service (NHS) (public) hospitals in England, covering the period 2007e08 to 2008e09. HES currently contains 20 diagnostic fields, coded using standard International Classification of Diseases, Tenth Revision (ICD-10). The first of these is the primary diagnosis or main problem treated, with the others available to capture comorbidities and complications; there are no present-on-admission flags. We included only inpatients at the 147 acute nonspecialist hospital trusts, as other hospitals have a very different casemix and mental health units tend to have poor-quality data. A trust can comprise several hospital sites. We included only inpatients, both planned and unplanned, but not day case surgery or regular day/night attenders. For each year separately, we constructed logistic regression models for in-hospital mortality with indicator variables for each component of the three comorbidity indices to be compared: original Charlson (ICD-10 version by Sundararajan et al. [3]), a modified Charlson using coder advice, and the modified Elixhauser [6,7]. The second year acted as the validation data set for the first year. We then used the methods described by Sullivan et al. [8] to convert the parameter estimates from the regression models into an index. The number of points assigned to each comorbidity variable equaled its regression coefficient divided by the coefficient in the model with the smallest absolute value, rounding this ratio to the nearest whole number. Each comorbidity variable is, therefore, compared with the ‘‘weakest’’ variable in the model, that is, whose parameter
1427
estimate had the smallest absolute value. Any variables with P O 0.05 were excluded for this purpose and assigned a score of zero. A comorbidity variable given 2 points is, therefore, considered twice as ‘‘strong’’ as one assigned 1 point and would be associated with double the log odds of mortality. Each patient’s comorbidity score was calculated by adding up their total points. Their risk of death would be derived using a simple formula given by van Walraven [6]. The specifications for the modified Charlson index are given in Table A1 (See Table A1 on the journal’s web site at www.elsevier.com). These represent the consensus of opinion at the coding department at a single hospital trust. We did not attempt a formal revision of the index that combined, for example, coding with medical opinion obtained independently. As the relations between comorbidity and mortality may vary by condition, we repeated the above models using first 5 years of unplanned admissions for acute myocardial infarction (AMI, ICD-10 I21, I22) and second 5 years of unplanned admissions for chronic obstructive pulmonary disease (COPD, ICD-10 J40eJ44), 2004e05 to 2008e09 in both cases. As for all inpatients, we selected each patient’s first admission for the condition of interest during the period. Five years of data were needed for these single conditions to obtain robust scores, although the levels of coding changed during the period. In a second analysis for 2008e09, we calculated for each hospital trust their mean coding depth, defined as the number of diagnostic fields (maximum 14) with nonmissing ICD-10 codes divided by the number of admissions. Trusts were then split into ‘‘high coders’’ (above the upper quartile) and ‘‘low coders’’ (below the upper quartile). This was because it is suspected that indices will perform better if the coding is high. For each regression model, the following were obtained: receiver operating characteristic curve c statistic (measure of discrimination), McFadden adjusted R-squared (measure of goodness of fit), and a calibration plot (the Hosmere Lemeshow test is often used for calibration, but we and others have found that it can detect unimportant differences with large data sets such as ours [9]). For the plot, we ordered patients by their predicted risk taken from the model with the modified Charlson index and divided into percentiles. The predicted risks for each index were taken from models including 5-year age group and sex as well as the index as covariates and were plotted against the actual crude case fatality rate. Without present-on-admission flags that a few countries have, UK data sets cannot easily distinguish comorbidities from complications acquired in hospital after admission. However, comorbidities once diagnosed should be recorded in every subsequent admission for the same patient. In contrast, complications or acute conditions will often only appear once. To try to assess the consistency within hospitals of recording of such codes, we inspected the last admission in 2007e08 for each patient admitted in 2008e09 and
1428
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
noted the proportion of patients with the comorbidity code present in the two consecutive admissions.
3. Results In 2008e09, there were a total of 5,432,220 inpatient admissions: 5,130,552 (94.5%) of these were to our list of acute nonspecialist hospital trusts (this proportion was similar in 2007e08). Of these admissions, 73.2% had one or more secondary diagnoses entered (70.9% in 2007e08). Table 1 shows the distribution for the 2 years of the number of comorbidities by each score. Compared with the original Charlson codes, the amended Charlson gives a nonzero comorbidity score to a further 2.1% of patients and Elixhauser a further 15.6% of patients. Comorbidity recording was a little higher in 2008e09 than in 2007e08, with about an extra 2% of patients having nonzero scores. Of the Charlson components, the most commonly recorded was chronic lung disease (7.7%). It was common for patients with any of the Charlson variables recorded to have had one or more admissions in the previous year. The most consistently recorded variable across two consecutive admissions for the same patient was diabetes without longterm complications at 77% of such patients (See Table A2 on the journal’s web site at www.elsevier.com). The suggested changes to the Charlson specification by our clinical coders led to a large increase in this consistency for AMI (from 31% to 56%) and dementia (from 38% to 51%), but only a slight increase for the other conditions. For human immunodeficiency virus (HIV), however, a decrease in consistency was seen. The suggested changes flagged more admissions for each condition, with AMI and dementia more
than trebling in numbers and more than 50% rises in the numbers flagged with renal disease and HIV. The most commonly recorded Elixhauser variable in 2008e09 was hypertension without complications (15.8%). Many of the component variables had frequencies of less than 1% (See Table A3 on the journal’s web site at www. elsevier.com). The consistency of recording of these variables across two successive admissions for the same patient varied from 7.2% to 90.1%, with three of the 31 variables being recorded in !10% of the patient’s previous admission (blood loss anemia, peptic ulcer disease, excluding bleeding and weight loss). 3.1. All inpatients combined Most of the Elixhauser weights derived from 2008e09 data were about half the value of those derived from 2007e08, but the relative importance was unchanged (Table 2). In 2007e08, the condition with the smallest coefficient given a weight of 1 was diabetes with long-term complications, but in 2008e09 it was valvular disease. The pattern for HES and Canadian data was also similar, with a few exceptions. Blood loss anemia and valvular disease had a negative weight in Canada, but we gave them small positive weights in both years in England. Depression had a weight of 3 in Canada but did not reach statistical significance in either year in our models. The HES-based weights for 2008e09 were very different from the original weights that Charlson et al. obtained in their original study (Table 3). The range of scores for both the original and the modified set of codes were much wider for HES. In relative terms, the biggest change was seen for HIV, to which Charlson et al. gave the highest weight, whereas we
Table 1. Distribution of comorbidity scores and case fatality rates for 2007e08 and 2008e09
Number of comorbidities in each index
2007e08
2007e08
2008e09
2008e09
Number of admissions (% of total)
Number of deaths (case fatality rate, %)
Number of admissions (% of total)
Number of deaths (case fatality rate, %)
Charlson (original) 0 1 2 3 4þ
4,381,059 789,251 182,957 35,697 6,200
(81.2) (14.6) (3.4) (0.7) (0.1)
61,413 49,508 23,509 6,944 1,655
(1.4) (6.3) (12.8) (19.5) (26.7)
4,326,053 853,718 203,832 41,115 7,502
(79.6) (15.7) (3.8) (0.8) (0.1)
57,760 49,902 24,513 7,530 1,905
(1.3) (5.8) (12.0) (18.3) (25.4)
Charlson (amended) 0 1 2 3 4þ
4,277,656 821,116 227,918 55,370 13,104
(79.3) (15.2) (4.2) (1.0) (0.2)
52,772 49,861 27,558 9,707 3,131
(1.2) (6.1) (12.1) (17.5) (23.9)
4,208,053 884,255 256,590 66,247 17,075
(77.5) (16.3) (4.7) (1.2) (0.3)
48,262 49,993 28,623 10,980 3,752
(1.1) (5.7) (11.2) (16.6) (22.0)
Modified Elixhauser 0 1 2 3 4þ
3,546,811 1,047,469 513,919 200,258 86,707
(65.7) (19.4) (9.5) (3.7) (1.6)
32,194 39,779 35,384 21,259 14,413
(0.9) (3.8) (6.9) (10.6) (16.6)
3,422,930 1,105,463 565,218 230,809 107,800
(63.0) (20.4) (10.4) (4.2) (2.0)
27,512 38,243 35,619 22,976 17,260
(0.8) (3.5) (6.3) (10.0) (16.0)
Total
5,395,164 (100)
5,432,220 (100)
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
1429
Table 2. Comparison of scores for modified Elixhauser index for English HES data and published Canadian data
Comorbidity variable Alcohol abuse Cardiac arrhythmias Blood loss anemia Congestive heart failure Chronic pulmonary disease Coagulopathy Deficiency anemia Depression Diabetes, complicated Diabetes, uncomplicated Drug abuse Fluid and electrolyte disorders AIDS/HIV Hypertension, complicateda Hypertension, uncomplicateda Hypothyroidism Liver disease Lymphoma Metastatic cancer Obesity Other neurological disorders Paralysis Peptic ulcer disease excluding bleeding Psychoses Pulmonary circulation disorders Peripheral vascular disorders Renal failure Rheumatoid arthritis/collagen vascular diseases Solid tumor without metastasis Valvular disease Weight loss
Score from Canadian dataa
HES-based score 2007e08, all inpatients
HES-based score 2008e09, all inpatients
HES-based score for AMI admissions
HES-based score for COPD admissions
0 5 2 7 3 3 2 3 0 0 7 5 0 0 0 0 11 9 12 4 6 7 0 0 4 2 5 0 4 1 6
0 15 4 17 6 10 5 0 1 3 12 22 1 2 5 3 18 15 25 4 12 12 8 6 14 10 17 5 11 2 15
0 9 3 10 4 5 3 0 0 2 6 13 0 1 4 2 10 9 15 2 8 7 5 4 8 6 10 3 6 1 9
8 7 0 14 1 4 4 3 4 1 26 22 36 6 4 0 20 8 13 14 16 12 0 5 7 5 14 0 11 2 14
1 3 0 5 0 4 1 0 0 2 9 9 0 2 1 1 4 3 8 0 2 4 0 0 6 3 7 0 5 0 6
Abbreviations: AMI, acute myocardial infarction; COPD, chronic obstructive pulmonary disease; HES, Hospital Episodes Statistics; AIDS, acquired immunodeficiency syndrome; HIV, human immunodeficiency virus. a Canadian score was published by van Walraven et al. Only one weight for hypertension was given in that study.
found its presence not to be statistically significantly associated with higher mortality and with the modified codes was actually associated with a lower than average risk. 3.2. Admissions for AMI and COPD The Elixhauser weights for AMI and COPD sometimes differed from those for all inpatients combined either in magnitude or occasionally in direction (Table 2). Many were consistent in direction, such as congestive heart failure, fluid and electrolyte disorders, various cancers, renal failure, and pulmonary circulatory disease; the drug abuse codes were consistently associated with a lower risk. Alcohol abuse and HIV were given large negative scores for AMI but not for all inpatients (just 48 patients admitted for AMI had HIV recorded). There appeared to be more consistency across the three sets of weights for the amended Charlson, although the magnitudes sometimes differed greatly, as for severe liver disease, although its weight was derived from just 203 patients (Table 3). The weights were generally largest for AMI and smallest for COPD.
3.3. Model performance Table 4 shows the model performance for the three sets of scores. For all inpatients combined, both discrimination and model fit were best for the Elixhauser and lowest for the original Charlson index with the original codes and weights. Discrimination was very similar using the original Charlson index and weights and using the original Charlson codes but empirical HES-based weights, although model fit was superior with the latter. Model performance was a little better in high-coding hospitals than in low-coding hospitals, with c statistics around 0.015 higher and adjusted R-squared statistics around 0.01 higher (figures not shown). Fig. 1 compares the predicted case fatality rate for each score (rates also adjusted for age and sex) against the actual rate for 2008e09 by risk percentile, with patients put into ascending order of predicted risk using the modified Charlson score. At low risk, all three scores overpredict. Calibration improves by medium risk, with each score better than the others at different parts of the curve, and then all three scores underpredict at high risk until overpredicting again in the highest risk percentile.
2 0 2 2 0 1 2 0 1 0 2 1 1 1 1 2 5 7 0 14 12 2 0 9 0 0 1 7 2 3 5 2 9 27 (2.53e2.66) (1.72e1.86) (3.55e3.75) (6.59e6.85) (0.78e0.90) (1.34e1.39) (5.00e5.20) (0.62e1.55) (2.35e2.74) (1.74e1.79) (6.05e6.37) (1.81e1.89) (1.23e1.36) (2.89e3.35) (2.25e2.39) (3.69e3.84) (10.66e12.32) 2.60 1.78 3.65 6.72 0.84 1.36 5.10 0.98 2.54 1.76 6.21 1.85 1.29 3.11 2.32 3.76 11.46 4 2 5 7 1 1 6 0 4 2 7 2 1 4 3 5 9 5 3 7 8 0 2 8 0 4 3 8 5 1 5 4 7 11 2 1 1 1 2 1 1 6 1 1 3 1 2 1 1 2 3
Abbreviations: AMI, acute myocardial infarction; COPD, chronic obstructive pulmonary disease; HES, Hospital Episodes Statistics; HIV, human immunodeficiency virus; CI, confidence interval.
Comorbidity variable
Cancer Connective tissue disorder Cerebral vascular accident Dementia Diabetes with long-term complications Diabetes without long-term complications Congestive heart failure HIV Mild or moderate liver disease Pulmonary disease Metastatic cancer AMI Paraplegia Peptic ulcer Peripheral vascular disease Renal disease Severe liver disease
Original Charlson codes, new HES-based weights, all inpatients 2008e09
Amended Charlson codes, new HES-based weights, all inpatients 2008e09
Odds ratios and CIs for amended Charlson codes
Amended Charlson codes, new HES-based weights for AMI admissions, 2004e05 to 2008e09
Amended Charlson codes, new HES-based weights for COPD admissions, 2004e05 to 2008e09
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
Original Charlson weights
Table 3. Comparison of scores for Charlson index: weights from original Charlson et al. study and new weights from English HES data for all inpatients, AMI, and COPD
1430
4. Discussion 4.1. Summary of results In terms of discrimination and consistency of recording, the modified Charlson performed better than the original version but its discrimination was lower than the modified Elixhauser. The original Charlson codes and weights had similar discrimination as but inferior R-squared statistics to the modified set of codes and empirical weights. Model performance was much better for all inpatients combined than for either AMI or COPD admissions alone, but the relative performance of the comorbidity indices was the same. Although the suggested code for HIV led to a greater prevalence, we believe that it should not be adopted because of concerns over inconsistency of recording and heterogeneity of risk. It also affected the weight considerably. The consistency using the codes B21eB24 as in the original Charlson was nearly 100%. However, this may have been because of the fact that, because of the data provider considering sexually transmitted diseases particularly sensitive, we had to give different anonymized patient identifiers for records with these conditions from the anonymized identifiers we gave for records with all other conditions (HIV was defined as only B21eB24 for this purpose). The consistency was, therefore, inflated because records belonging to these patients without the ‘‘sensitive’’ conditions were made to appear to belong to other patients. Several Elixhauser codes appear to be for acute events, some of which could be complications although Elixhauser et al. tried to exclude these. This was reflected in their low rates of recording overall and in successive admissions for the same patient, in particular blood loss anemia and fluid and electrolyte disorders. These conditions were among the least prevalent, and their influence will be small. There are some significant omissions within Elixhauser, such as dementia, which has a relatively high score within Charlson. The Charlson index also includes some acute events, however, although its components were more consistently recorded, which suggests that this potential problem was minimized. All indices performed better with higher levels of recording (at the ‘‘high-coding’’ hospitals), and there were small year-on-year improvements in both coding depth and model discrimination (figures not shown). As others have noted, the comorbidity scores by themselves predict death only moderately well, as such factors as age, primary diagnosis, and urgency of admission will usually be at least as important. Some comorbidities were actually associated with a lower risk of mortality and were given negative scores. In real casemix models, comorbidity scores would not be the only covariate, and other factors such as age will account for some of the variation explained by the comorbidity scores. The calibration plot presented here included age and sex in the model, and it is likely (and to be hoped) that the addition of other variables would improve the fit to the observed case fatality rates.
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
1431
Table 4. Summary of model performance for the three scores for all inpatients, AMI, and COPD Original Charlson codes and weights
Original Charlson codes, empirical weights
Amended Charlson codes, empirical weights
Modified Elixhauser, empirical weights
C statistic All inpatients AMI admissions COPD admissions
0.719 0.624 0.577
0.726 0.641 0.601
0.757 0.654 0.611
0.799 0.668 0.646
Adjusted R-squared statistic All inpatients AMI admissions COPD admissions
0.100 0.046 0.020
0.106 0.051 0.030
0.123 0.058 0.032
0.142 0.058 0.044
Measure
Abbreviations: ICD-10, International Classification of Diseases, Tenth Revision; AMI, acute myocardial infarction (ICD-10 I21, I22); COPD, chronic obstructive pulmonary disease (ICD-10 J40eJ44).
4.2. Comparison with Canadian study of findings for the modified Elixhauser index HES had lower prevalences compared with Canada for all Elixhauser conditions with the sole exception of hypothyroidism. The rankings of comorbidity variables looked similar for all inpatients combined, with both studies deriving some negative weights. The HES scores were often higher, although the smallest significant odds ratio, the log of which was used to base all the weights on, was similar: it was for valvular disease in both studies, with 0.91 in Canada compared with our 1.14 (which equates to 0.88 when reciprocated). One potential explanation of this would be if HES recorded the Elixhauser codes for more severe disease or in iller patients than in the Canadian sample. This is plausible given the lower HES prevalence of these conditions but cannot be tested using these data alone. The pattern of weights for admissions for AMI and/or COPD differed for some comorbidities from the pattern for all inpatients combined.
4.3. Issues and limitations This study benefits from the use of previously validated methods and a large sample size with national coverage in each year of data. We considered all inpatients combined and two commonly used single diagnoses as examples to show how the sets of weights might differ across different patient groups, although we did not intend the analysis to be exhaustive. The level of coding is clearly important. The c and R-squared statistics were higher in 2008e09 than the previous year and were higher for ‘‘high-coding’’ trusts than for ‘‘low-coding’’ trusts. This was also found by Martins and Blais [10] using Brazilian data, who calculated the c statistic using an increasing number of secondary diagnosis codes, from two to four. Comorbidity data will usually not be missing at random, as they vary between hospitals and will be influenced by issues such as documentation of patient notes by physicians and financial incentives such as diagnosis-related groups or healthcare-related groups. In 2007, the audit commission began a quality assurance
Fig. 1. Crude rate compared with predicted rate for original Charlson, modified Charlson Elixhauser score by risk percentile, 2008e09 data.
1432
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
program for the inpatient data used in reimbursement. Its most recent report found an improvement in the average coding error rate by hospital trust from 16% to 11% in 3 years, with variations by specialty [11]. Current guidance states that any comorbidity that affects the management of the patient and contributes to an accurate clinical picture within the current episode of care must be recorded, but others should not be. Deciding on the relevance of a given comorbidity may not be straightforward, and in the first stage of their review, the NHS Classifications Service working group produced a list of 61 comorbidities that, from April 2010, must always be recorded [12]. This would capture most of the Charlson index and a number of the Elixhauser codes, which should improve the models’ performance. If the comorbidity used as the base is uncommon and can vary in frequency between years, then the set of scores will also vary between years. If these weights are applied to another single year or another country’s hospital administration data set, this does not matter; however, as long as the ranking of the comorbidity variables is the same with regards to mortality. For example, if the mortality in another data set of hospitalized patients recorded as having HIV is as high as suggested by Charlson’s original weights rather than as suggested by our weights, then recalibration for that data set will be required. Also, if two or more years of data are combined, for example, to increase the sample size as for AMI and COPD in this study, then the choice of weights will be an issue unless the level of coding is similar throughout the time period. The implicit assumption is that the relation between the comorbidity and the outcome is constant throughout the time period. We have not incorporated information from previous admissions into the comorbidity score as has been tried by others [13e15]. This has been found to be of some benefit, but we wanted to focus on all patients combined in this study, including those with no prior admissions. The coding suggestions that we incorporated into our ‘‘modified Charlson’’ index were the combined opinions of members of the coding department at a single English hospital. We did not attempt to formalize this process, but with the exception of the additional HIV code, the suggestions seem to have led to an improved risk model. However, others may disagree with these choices and there remain several challenges considered briefly in the next section.
model discrimination between AMI, COPD, and all admissions combined and, although the pattern of weights was broadly similar across the three groups for each index, there were some notable exceptions. There was perhaps more variation in scores for the Elixhauser components than for the Charlson ones. It would be interesting to compare the results using total mortality, for example, at 1 year, as was done by Li et al. This requires linkage between the hospital record and death registrations, which currently incurs a considerable time lag in the United Kingdom. We have not considered interactions between comorbidity variables. This might best be done using machine learning methods, which are perhaps better suited to this than logistic regression. As has been suggested by others including Elixhauser, it would be worthwhile but not straightforward to look for markers of ‘‘well’’ patients in the form of perhaps a number of secondary diagnosis codes added by keen coders at some hospitals. Finally, it is notable that two of our most important comorbidities within Charlson, dementia and cerebrovascular disease, are not in Elixhauser as her study found no relation with mortality. This suggests that there is scope for combining elements of the two scores.
4.4. Future work
References
As with some previous studies, we have fitted each model on all inpatient admissions combined, but it is possible that the mortality pattern associated with the comorbidities could differ between primary diagnoses. Li et al. [16] found that the change from ICD-9 to ICD-10 specification made very little difference but that for their five patient groups, discrimination using either Charlson or Elixhauser ranged from 0.62 to 0.82. We also found differences in
Acknowledgments Our unit is largely funded via a research grant by Dr Foster Intelligence, an independent health care information company and joint venture with the NHS Information Center. The Dr Foster Unit is affiliated with the Center for Patient Safety and Service Quality at Imperial College Healthcare NHS Trust, which is funded by the National Institute of Health Research. We are grateful for support from the National Institute for Health Research Biomedical Research Center funding scheme.
Appendix Supplementary material Supplementary material can be found, in the online version, at 10.1016/j.jclinepi.2011.04.004.
[1] Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol 2004;57:1288e94. [2] Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40(5):373e83. [3] Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998;36:8e27. [4] Bottle A, Aylin P. Intelligent information: a national system for monitoring clinical performance. Health Serv Res 2008;43:10e31.
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433 [5] Aylin P, Bottle A, Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334:1044e9. [6] van Walraven C, Austin PC, Jennings A, Quan H, Forster AJ. A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data. Med Care 2009;47: 626e33. [7] Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005;43:1130e9. [8] Sullivan LM, Massaro JM, D’Agostino RB Sr. Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med 2004;23:1631e60. [9] Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med 1997;16:965e80. [10] Martins M, Blais R. Evaluation of comorbidity indices for inpatient mortality prediction models. J Clin Epidemiol 2006;59:665e9. [11] Audit Commission. Improving data quality in the NHS. Audit Commission 2010. Available at URL: http://www.audit-commission.gov.
[12] [13]
[14]
[15]
[16]
1433
uk/nationalstudies/health/pbr/pbr2010/Pages/default.aspx. Accessed October 2010. NHS Classifications Service 2010. Coding of comorbidities. Coding Clinic, 2010;6(6). Stukenborg GJ, Wagner DP, Connors AF Jr. Comparison of the performance of two comorbidity measures, with and without information from prior hospitalizations. Med Care 2001;39: 727e39. Lee DS, Donovan L, Austin PC, Gong Y, Liu PP, Rouleau JL, et al. Comparison of coding of heart failure and comorbidities in administrative and clinical data for use in outcomes research. Med Care 2005;43:182e8. Preen DB, D’Arcy C, Holman J, Spilsbury K, Semmens JB, Brameld KJ. Length of comorbidity lookback period affected regression model performance of administrative health data. J Clin Epidemiol 2006;59:940e6. Li B, Evans D, Faris P, Dean S, Quan H. Risk adjustment performance of Charlson and Elixhauser comorbidities in ICD-9 and ICD-10 administrative databases. BMC Health Serv Res 2008;8:12.
Comorbidity scores for administrative data benefited from adaptation to local coding and diagnostic practices Alex Bottle*, Paul Aylin Dr Foster Unit, Department of Primary Care and Public Health, Imperial College London, 1st Floor, Jarvis House, 12 Smithfield Street, London EC1A 9LA, UK Accepted 6 April 2011; Published online 20 July 2011
Abstract Objective: The Charlson and Elixhauser indices are the most commonly used comorbidity indices with risk prediction models using administrative data. Our objective was to compare the original Charlson index, a modified set of Charlson codes after advice from clinical coders, and a published modified Elixhauser index in predicting in-hospital mortality. Study Design and Setting: Logistic regression using two separate years of administrative hospital data for all acute nonspecialist public hospitals in England. Results: For all admissions combined, discrimination was similar for the Charlson index using the original codes and weights and the Charlson index using the original codes but England-calibrated weights (c 5 0.73), although model fit was superior for the latter. The new Charlson codes improved discrimination (c 5 0.76), model fit, and consistency of recording between admissions. The modified Elixhauser had the best performance (c 5 0.80). For admissions for acute myocardial infarction and chronic obstructive pulmonary disease, the weights often differed, although the patterns were broadly similar. Conclusion: Recalibration of the original Charlson index yielded only modest benefits overall. The modified Charlson codes and weights offer better fit and discrimination for English data over the original version. The modified Elixhauser performed best of all, but its weights were perhaps less consistent across the different patient groups considered here. Ó 2011 Elsevier Inc. All rights reserved. Keywords: Comorbidity; Charlson; Elixhauser; Administrative data; Mortality; Risk modeling; Casemix; Hospitals
1. Introduction Patient comorbidity is a common potential confounder in health services research and can be estimated from administrative data sets. It is often included in risk models of mortality and other outcomes in provider profiling [1]. The two most commonly used indices are the Charlson [2] and Elixhauser [3], originally described using International Classification of Diseases, Ninth Revision (ICD-9) with US data. Both were designed for use with databases that cannot distinguish between comorbidities and complications and tried to limit their set of conditions to those of a chronic nature or a few acute conditions that are relatively unlikely to be potential complications. The Charlson index is now more than 20 years old, and both indices need calibrating on the data set of interest (external validation); to our knowledge, very little has been published from the United Kingdom. The weights (or scores) for these two indices
may be inappropriate for the United Kingdom because of differing populations and coding practices. We have to date been using an Australian version of Charlson [1] in our risk adjustment models [4,5], but discussions with clinical coders raised questions over the suitability of some codes when used in the United Kingdom. Recent work in Canada [6] found that a modified Elixhauser [7] outperforms Charlson; the authors also concluded that it needed external validation. Our first aim was to derive new empirical weights based on English hospital administrative data for the original Charlson codes, a modified set of Charlson codes following discussions with coders, and the modified Elixhauser. Given that secondary diagnosis recording is known to vary between hospitals, our second aim was to assess the consistency of recording of codes used in the modified Charlson and modified Elixhauser across admissions belonging to the same patient.
2. Methods * Corresponding author. Tel.: +44 (0)207 3328964; fax +44 (0)207 3328888. E-mail address: [email protected] (A. Bottle). 0895-4356/$ - see front matter Ó 2011 Elsevier Inc. All rights reserved. doi: 10.1016/j.jclinepi.2011.04.004
We took 2 years of Secondary Uses Service data from Hospital Episodes Statistics (HES), the administrative data
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
What is new? Key findings Despite being around 25 years old and derived in the United States, the original weights for the Charlson comorbidity index perform nearly as well as the weights recalibrated for current English data. Some extra codes used more in England explained more variation, especially at high-coding hospitals, and had higher consistency of recording across admissions for the same patient. The Elixhauser score performed best of all, but its weights varied more across patient groups. What this adds to what was known The Charlson index needed some extra codes for use in England but still performed well, although the Elixhauser index did best. What is the implication Comorbidity index codes and weights may need to be adapted to suit local patterns.
set that covers all admissions to National Health Service (NHS) (public) hospitals in England, covering the period 2007e08 to 2008e09. HES currently contains 20 diagnostic fields, coded using standard International Classification of Diseases, Tenth Revision (ICD-10). The first of these is the primary diagnosis or main problem treated, with the others available to capture comorbidities and complications; there are no present-on-admission flags. We included only inpatients at the 147 acute nonspecialist hospital trusts, as other hospitals have a very different casemix and mental health units tend to have poor-quality data. A trust can comprise several hospital sites. We included only inpatients, both planned and unplanned, but not day case surgery or regular day/night attenders. For each year separately, we constructed logistic regression models for in-hospital mortality with indicator variables for each component of the three comorbidity indices to be compared: original Charlson (ICD-10 version by Sundararajan et al. [3]), a modified Charlson using coder advice, and the modified Elixhauser [6,7]. The second year acted as the validation data set for the first year. We then used the methods described by Sullivan et al. [8] to convert the parameter estimates from the regression models into an index. The number of points assigned to each comorbidity variable equaled its regression coefficient divided by the coefficient in the model with the smallest absolute value, rounding this ratio to the nearest whole number. Each comorbidity variable is, therefore, compared with the ‘‘weakest’’ variable in the model, that is, whose parameter
1427
estimate had the smallest absolute value. Any variables with P O 0.05 were excluded for this purpose and assigned a score of zero. A comorbidity variable given 2 points is, therefore, considered twice as ‘‘strong’’ as one assigned 1 point and would be associated with double the log odds of mortality. Each patient’s comorbidity score was calculated by adding up their total points. Their risk of death would be derived using a simple formula given by van Walraven [6]. The specifications for the modified Charlson index are given in Table A1 (See Table A1 on the journal’s web site at www.elsevier.com). These represent the consensus of opinion at the coding department at a single hospital trust. We did not attempt a formal revision of the index that combined, for example, coding with medical opinion obtained independently. As the relations between comorbidity and mortality may vary by condition, we repeated the above models using first 5 years of unplanned admissions for acute myocardial infarction (AMI, ICD-10 I21, I22) and second 5 years of unplanned admissions for chronic obstructive pulmonary disease (COPD, ICD-10 J40eJ44), 2004e05 to 2008e09 in both cases. As for all inpatients, we selected each patient’s first admission for the condition of interest during the period. Five years of data were needed for these single conditions to obtain robust scores, although the levels of coding changed during the period. In a second analysis for 2008e09, we calculated for each hospital trust their mean coding depth, defined as the number of diagnostic fields (maximum 14) with nonmissing ICD-10 codes divided by the number of admissions. Trusts were then split into ‘‘high coders’’ (above the upper quartile) and ‘‘low coders’’ (below the upper quartile). This was because it is suspected that indices will perform better if the coding is high. For each regression model, the following were obtained: receiver operating characteristic curve c statistic (measure of discrimination), McFadden adjusted R-squared (measure of goodness of fit), and a calibration plot (the Hosmere Lemeshow test is often used for calibration, but we and others have found that it can detect unimportant differences with large data sets such as ours [9]). For the plot, we ordered patients by their predicted risk taken from the model with the modified Charlson index and divided into percentiles. The predicted risks for each index were taken from models including 5-year age group and sex as well as the index as covariates and were plotted against the actual crude case fatality rate. Without present-on-admission flags that a few countries have, UK data sets cannot easily distinguish comorbidities from complications acquired in hospital after admission. However, comorbidities once diagnosed should be recorded in every subsequent admission for the same patient. In contrast, complications or acute conditions will often only appear once. To try to assess the consistency within hospitals of recording of such codes, we inspected the last admission in 2007e08 for each patient admitted in 2008e09 and
1428
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
noted the proportion of patients with the comorbidity code present in the two consecutive admissions.
3. Results In 2008e09, there were a total of 5,432,220 inpatient admissions: 5,130,552 (94.5%) of these were to our list of acute nonspecialist hospital trusts (this proportion was similar in 2007e08). Of these admissions, 73.2% had one or more secondary diagnoses entered (70.9% in 2007e08). Table 1 shows the distribution for the 2 years of the number of comorbidities by each score. Compared with the original Charlson codes, the amended Charlson gives a nonzero comorbidity score to a further 2.1% of patients and Elixhauser a further 15.6% of patients. Comorbidity recording was a little higher in 2008e09 than in 2007e08, with about an extra 2% of patients having nonzero scores. Of the Charlson components, the most commonly recorded was chronic lung disease (7.7%). It was common for patients with any of the Charlson variables recorded to have had one or more admissions in the previous year. The most consistently recorded variable across two consecutive admissions for the same patient was diabetes without longterm complications at 77% of such patients (See Table A2 on the journal’s web site at www.elsevier.com). The suggested changes to the Charlson specification by our clinical coders led to a large increase in this consistency for AMI (from 31% to 56%) and dementia (from 38% to 51%), but only a slight increase for the other conditions. For human immunodeficiency virus (HIV), however, a decrease in consistency was seen. The suggested changes flagged more admissions for each condition, with AMI and dementia more
than trebling in numbers and more than 50% rises in the numbers flagged with renal disease and HIV. The most commonly recorded Elixhauser variable in 2008e09 was hypertension without complications (15.8%). Many of the component variables had frequencies of less than 1% (See Table A3 on the journal’s web site at www. elsevier.com). The consistency of recording of these variables across two successive admissions for the same patient varied from 7.2% to 90.1%, with three of the 31 variables being recorded in !10% of the patient’s previous admission (blood loss anemia, peptic ulcer disease, excluding bleeding and weight loss). 3.1. All inpatients combined Most of the Elixhauser weights derived from 2008e09 data were about half the value of those derived from 2007e08, but the relative importance was unchanged (Table 2). In 2007e08, the condition with the smallest coefficient given a weight of 1 was diabetes with long-term complications, but in 2008e09 it was valvular disease. The pattern for HES and Canadian data was also similar, with a few exceptions. Blood loss anemia and valvular disease had a negative weight in Canada, but we gave them small positive weights in both years in England. Depression had a weight of 3 in Canada but did not reach statistical significance in either year in our models. The HES-based weights for 2008e09 were very different from the original weights that Charlson et al. obtained in their original study (Table 3). The range of scores for both the original and the modified set of codes were much wider for HES. In relative terms, the biggest change was seen for HIV, to which Charlson et al. gave the highest weight, whereas we
Table 1. Distribution of comorbidity scores and case fatality rates for 2007e08 and 2008e09
Number of comorbidities in each index
2007e08
2007e08
2008e09
2008e09
Number of admissions (% of total)
Number of deaths (case fatality rate, %)
Number of admissions (% of total)
Number of deaths (case fatality rate, %)
Charlson (original) 0 1 2 3 4þ
4,381,059 789,251 182,957 35,697 6,200
(81.2) (14.6) (3.4) (0.7) (0.1)
61,413 49,508 23,509 6,944 1,655
(1.4) (6.3) (12.8) (19.5) (26.7)
4,326,053 853,718 203,832 41,115 7,502
(79.6) (15.7) (3.8) (0.8) (0.1)
57,760 49,902 24,513 7,530 1,905
(1.3) (5.8) (12.0) (18.3) (25.4)
Charlson (amended) 0 1 2 3 4þ
4,277,656 821,116 227,918 55,370 13,104
(79.3) (15.2) (4.2) (1.0) (0.2)
52,772 49,861 27,558 9,707 3,131
(1.2) (6.1) (12.1) (17.5) (23.9)
4,208,053 884,255 256,590 66,247 17,075
(77.5) (16.3) (4.7) (1.2) (0.3)
48,262 49,993 28,623 10,980 3,752
(1.1) (5.7) (11.2) (16.6) (22.0)
Modified Elixhauser 0 1 2 3 4þ
3,546,811 1,047,469 513,919 200,258 86,707
(65.7) (19.4) (9.5) (3.7) (1.6)
32,194 39,779 35,384 21,259 14,413
(0.9) (3.8) (6.9) (10.6) (16.6)
3,422,930 1,105,463 565,218 230,809 107,800
(63.0) (20.4) (10.4) (4.2) (2.0)
27,512 38,243 35,619 22,976 17,260
(0.8) (3.5) (6.3) (10.0) (16.0)
Total
5,395,164 (100)
5,432,220 (100)
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
1429
Table 2. Comparison of scores for modified Elixhauser index for English HES data and published Canadian data
Comorbidity variable Alcohol abuse Cardiac arrhythmias Blood loss anemia Congestive heart failure Chronic pulmonary disease Coagulopathy Deficiency anemia Depression Diabetes, complicated Diabetes, uncomplicated Drug abuse Fluid and electrolyte disorders AIDS/HIV Hypertension, complicateda Hypertension, uncomplicateda Hypothyroidism Liver disease Lymphoma Metastatic cancer Obesity Other neurological disorders Paralysis Peptic ulcer disease excluding bleeding Psychoses Pulmonary circulation disorders Peripheral vascular disorders Renal failure Rheumatoid arthritis/collagen vascular diseases Solid tumor without metastasis Valvular disease Weight loss
Score from Canadian dataa
HES-based score 2007e08, all inpatients
HES-based score 2008e09, all inpatients
HES-based score for AMI admissions
HES-based score for COPD admissions
0 5 2 7 3 3 2 3 0 0 7 5 0 0 0 0 11 9 12 4 6 7 0 0 4 2 5 0 4 1 6
0 15 4 17 6 10 5 0 1 3 12 22 1 2 5 3 18 15 25 4 12 12 8 6 14 10 17 5 11 2 15
0 9 3 10 4 5 3 0 0 2 6 13 0 1 4 2 10 9 15 2 8 7 5 4 8 6 10 3 6 1 9
8 7 0 14 1 4 4 3 4 1 26 22 36 6 4 0 20 8 13 14 16 12 0 5 7 5 14 0 11 2 14
1 3 0 5 0 4 1 0 0 2 9 9 0 2 1 1 4 3 8 0 2 4 0 0 6 3 7 0 5 0 6
Abbreviations: AMI, acute myocardial infarction; COPD, chronic obstructive pulmonary disease; HES, Hospital Episodes Statistics; AIDS, acquired immunodeficiency syndrome; HIV, human immunodeficiency virus. a Canadian score was published by van Walraven et al. Only one weight for hypertension was given in that study.
found its presence not to be statistically significantly associated with higher mortality and with the modified codes was actually associated with a lower than average risk. 3.2. Admissions for AMI and COPD The Elixhauser weights for AMI and COPD sometimes differed from those for all inpatients combined either in magnitude or occasionally in direction (Table 2). Many were consistent in direction, such as congestive heart failure, fluid and electrolyte disorders, various cancers, renal failure, and pulmonary circulatory disease; the drug abuse codes were consistently associated with a lower risk. Alcohol abuse and HIV were given large negative scores for AMI but not for all inpatients (just 48 patients admitted for AMI had HIV recorded). There appeared to be more consistency across the three sets of weights for the amended Charlson, although the magnitudes sometimes differed greatly, as for severe liver disease, although its weight was derived from just 203 patients (Table 3). The weights were generally largest for AMI and smallest for COPD.
3.3. Model performance Table 4 shows the model performance for the three sets of scores. For all inpatients combined, both discrimination and model fit were best for the Elixhauser and lowest for the original Charlson index with the original codes and weights. Discrimination was very similar using the original Charlson index and weights and using the original Charlson codes but empirical HES-based weights, although model fit was superior with the latter. Model performance was a little better in high-coding hospitals than in low-coding hospitals, with c statistics around 0.015 higher and adjusted R-squared statistics around 0.01 higher (figures not shown). Fig. 1 compares the predicted case fatality rate for each score (rates also adjusted for age and sex) against the actual rate for 2008e09 by risk percentile, with patients put into ascending order of predicted risk using the modified Charlson score. At low risk, all three scores overpredict. Calibration improves by medium risk, with each score better than the others at different parts of the curve, and then all three scores underpredict at high risk until overpredicting again in the highest risk percentile.
2 0 2 2 0 1 2 0 1 0 2 1 1 1 1 2 5 7 0 14 12 2 0 9 0 0 1 7 2 3 5 2 9 27 (2.53e2.66) (1.72e1.86) (3.55e3.75) (6.59e6.85) (0.78e0.90) (1.34e1.39) (5.00e5.20) (0.62e1.55) (2.35e2.74) (1.74e1.79) (6.05e6.37) (1.81e1.89) (1.23e1.36) (2.89e3.35) (2.25e2.39) (3.69e3.84) (10.66e12.32) 2.60 1.78 3.65 6.72 0.84 1.36 5.10 0.98 2.54 1.76 6.21 1.85 1.29 3.11 2.32 3.76 11.46 4 2 5 7 1 1 6 0 4 2 7 2 1 4 3 5 9 5 3 7 8 0 2 8 0 4 3 8 5 1 5 4 7 11 2 1 1 1 2 1 1 6 1 1 3 1 2 1 1 2 3
Abbreviations: AMI, acute myocardial infarction; COPD, chronic obstructive pulmonary disease; HES, Hospital Episodes Statistics; HIV, human immunodeficiency virus; CI, confidence interval.
Comorbidity variable
Cancer Connective tissue disorder Cerebral vascular accident Dementia Diabetes with long-term complications Diabetes without long-term complications Congestive heart failure HIV Mild or moderate liver disease Pulmonary disease Metastatic cancer AMI Paraplegia Peptic ulcer Peripheral vascular disease Renal disease Severe liver disease
Original Charlson codes, new HES-based weights, all inpatients 2008e09
Amended Charlson codes, new HES-based weights, all inpatients 2008e09
Odds ratios and CIs for amended Charlson codes
Amended Charlson codes, new HES-based weights for AMI admissions, 2004e05 to 2008e09
Amended Charlson codes, new HES-based weights for COPD admissions, 2004e05 to 2008e09
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
Original Charlson weights
Table 3. Comparison of scores for Charlson index: weights from original Charlson et al. study and new weights from English HES data for all inpatients, AMI, and COPD
1430
4. Discussion 4.1. Summary of results In terms of discrimination and consistency of recording, the modified Charlson performed better than the original version but its discrimination was lower than the modified Elixhauser. The original Charlson codes and weights had similar discrimination as but inferior R-squared statistics to the modified set of codes and empirical weights. Model performance was much better for all inpatients combined than for either AMI or COPD admissions alone, but the relative performance of the comorbidity indices was the same. Although the suggested code for HIV led to a greater prevalence, we believe that it should not be adopted because of concerns over inconsistency of recording and heterogeneity of risk. It also affected the weight considerably. The consistency using the codes B21eB24 as in the original Charlson was nearly 100%. However, this may have been because of the fact that, because of the data provider considering sexually transmitted diseases particularly sensitive, we had to give different anonymized patient identifiers for records with these conditions from the anonymized identifiers we gave for records with all other conditions (HIV was defined as only B21eB24 for this purpose). The consistency was, therefore, inflated because records belonging to these patients without the ‘‘sensitive’’ conditions were made to appear to belong to other patients. Several Elixhauser codes appear to be for acute events, some of which could be complications although Elixhauser et al. tried to exclude these. This was reflected in their low rates of recording overall and in successive admissions for the same patient, in particular blood loss anemia and fluid and electrolyte disorders. These conditions were among the least prevalent, and their influence will be small. There are some significant omissions within Elixhauser, such as dementia, which has a relatively high score within Charlson. The Charlson index also includes some acute events, however, although its components were more consistently recorded, which suggests that this potential problem was minimized. All indices performed better with higher levels of recording (at the ‘‘high-coding’’ hospitals), and there were small year-on-year improvements in both coding depth and model discrimination (figures not shown). As others have noted, the comorbidity scores by themselves predict death only moderately well, as such factors as age, primary diagnosis, and urgency of admission will usually be at least as important. Some comorbidities were actually associated with a lower risk of mortality and were given negative scores. In real casemix models, comorbidity scores would not be the only covariate, and other factors such as age will account for some of the variation explained by the comorbidity scores. The calibration plot presented here included age and sex in the model, and it is likely (and to be hoped) that the addition of other variables would improve the fit to the observed case fatality rates.
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
1431
Table 4. Summary of model performance for the three scores for all inpatients, AMI, and COPD Original Charlson codes and weights
Original Charlson codes, empirical weights
Amended Charlson codes, empirical weights
Modified Elixhauser, empirical weights
C statistic All inpatients AMI admissions COPD admissions
0.719 0.624 0.577
0.726 0.641 0.601
0.757 0.654 0.611
0.799 0.668 0.646
Adjusted R-squared statistic All inpatients AMI admissions COPD admissions
0.100 0.046 0.020
0.106 0.051 0.030
0.123 0.058 0.032
0.142 0.058 0.044
Measure
Abbreviations: ICD-10, International Classification of Diseases, Tenth Revision; AMI, acute myocardial infarction (ICD-10 I21, I22); COPD, chronic obstructive pulmonary disease (ICD-10 J40eJ44).
4.2. Comparison with Canadian study of findings for the modified Elixhauser index HES had lower prevalences compared with Canada for all Elixhauser conditions with the sole exception of hypothyroidism. The rankings of comorbidity variables looked similar for all inpatients combined, with both studies deriving some negative weights. The HES scores were often higher, although the smallest significant odds ratio, the log of which was used to base all the weights on, was similar: it was for valvular disease in both studies, with 0.91 in Canada compared with our 1.14 (which equates to 0.88 when reciprocated). One potential explanation of this would be if HES recorded the Elixhauser codes for more severe disease or in iller patients than in the Canadian sample. This is plausible given the lower HES prevalence of these conditions but cannot be tested using these data alone. The pattern of weights for admissions for AMI and/or COPD differed for some comorbidities from the pattern for all inpatients combined.
4.3. Issues and limitations This study benefits from the use of previously validated methods and a large sample size with national coverage in each year of data. We considered all inpatients combined and two commonly used single diagnoses as examples to show how the sets of weights might differ across different patient groups, although we did not intend the analysis to be exhaustive. The level of coding is clearly important. The c and R-squared statistics were higher in 2008e09 than the previous year and were higher for ‘‘high-coding’’ trusts than for ‘‘low-coding’’ trusts. This was also found by Martins and Blais [10] using Brazilian data, who calculated the c statistic using an increasing number of secondary diagnosis codes, from two to four. Comorbidity data will usually not be missing at random, as they vary between hospitals and will be influenced by issues such as documentation of patient notes by physicians and financial incentives such as diagnosis-related groups or healthcare-related groups. In 2007, the audit commission began a quality assurance
Fig. 1. Crude rate compared with predicted rate for original Charlson, modified Charlson Elixhauser score by risk percentile, 2008e09 data.
1432
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433
program for the inpatient data used in reimbursement. Its most recent report found an improvement in the average coding error rate by hospital trust from 16% to 11% in 3 years, with variations by specialty [11]. Current guidance states that any comorbidity that affects the management of the patient and contributes to an accurate clinical picture within the current episode of care must be recorded, but others should not be. Deciding on the relevance of a given comorbidity may not be straightforward, and in the first stage of their review, the NHS Classifications Service working group produced a list of 61 comorbidities that, from April 2010, must always be recorded [12]. This would capture most of the Charlson index and a number of the Elixhauser codes, which should improve the models’ performance. If the comorbidity used as the base is uncommon and can vary in frequency between years, then the set of scores will also vary between years. If these weights are applied to another single year or another country’s hospital administration data set, this does not matter; however, as long as the ranking of the comorbidity variables is the same with regards to mortality. For example, if the mortality in another data set of hospitalized patients recorded as having HIV is as high as suggested by Charlson’s original weights rather than as suggested by our weights, then recalibration for that data set will be required. Also, if two or more years of data are combined, for example, to increase the sample size as for AMI and COPD in this study, then the choice of weights will be an issue unless the level of coding is similar throughout the time period. The implicit assumption is that the relation between the comorbidity and the outcome is constant throughout the time period. We have not incorporated information from previous admissions into the comorbidity score as has been tried by others [13e15]. This has been found to be of some benefit, but we wanted to focus on all patients combined in this study, including those with no prior admissions. The coding suggestions that we incorporated into our ‘‘modified Charlson’’ index were the combined opinions of members of the coding department at a single English hospital. We did not attempt to formalize this process, but with the exception of the additional HIV code, the suggestions seem to have led to an improved risk model. However, others may disagree with these choices and there remain several challenges considered briefly in the next section.
model discrimination between AMI, COPD, and all admissions combined and, although the pattern of weights was broadly similar across the three groups for each index, there were some notable exceptions. There was perhaps more variation in scores for the Elixhauser components than for the Charlson ones. It would be interesting to compare the results using total mortality, for example, at 1 year, as was done by Li et al. This requires linkage between the hospital record and death registrations, which currently incurs a considerable time lag in the United Kingdom. We have not considered interactions between comorbidity variables. This might best be done using machine learning methods, which are perhaps better suited to this than logistic regression. As has been suggested by others including Elixhauser, it would be worthwhile but not straightforward to look for markers of ‘‘well’’ patients in the form of perhaps a number of secondary diagnosis codes added by keen coders at some hospitals. Finally, it is notable that two of our most important comorbidities within Charlson, dementia and cerebrovascular disease, are not in Elixhauser as her study found no relation with mortality. This suggests that there is scope for combining elements of the two scores.
4.4. Future work
References
As with some previous studies, we have fitted each model on all inpatient admissions combined, but it is possible that the mortality pattern associated with the comorbidities could differ between primary diagnoses. Li et al. [16] found that the change from ICD-9 to ICD-10 specification made very little difference but that for their five patient groups, discrimination using either Charlson or Elixhauser ranged from 0.62 to 0.82. We also found differences in
Acknowledgments Our unit is largely funded via a research grant by Dr Foster Intelligence, an independent health care information company and joint venture with the NHS Information Center. The Dr Foster Unit is affiliated with the Center for Patient Safety and Service Quality at Imperial College Healthcare NHS Trust, which is funded by the National Institute of Health Research. We are grateful for support from the National Institute for Health Research Biomedical Research Center funding scheme.
Appendix Supplementary material Supplementary material can be found, in the online version, at 10.1016/j.jclinepi.2011.04.004.
[1] Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA. New ICD-10 version of the Charlson comorbidity index predicted in-hospital mortality. J Clin Epidemiol 2004;57:1288e94. [2] Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis 1987;40(5):373e83. [3] Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998;36:8e27. [4] Bottle A, Aylin P. Intelligent information: a national system for monitoring clinical performance. Health Serv Res 2008;43:10e31.
A. Bottle, P. Aylin / Journal of Clinical Epidemiology 64 (2011) 1426e1433 [5] Aylin P, Bottle A, Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334:1044e9. [6] van Walraven C, Austin PC, Jennings A, Quan H, Forster AJ. A modification of the Elixhauser comorbidity measures into a point system for hospital death using administrative data. Med Care 2009;47: 626e33. [7] Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care 2005;43:1130e9. [8] Sullivan LM, Massaro JM, D’Agostino RB Sr. Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med 2004;23:1631e60. [9] Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med 1997;16:965e80. [10] Martins M, Blais R. Evaluation of comorbidity indices for inpatient mortality prediction models. J Clin Epidemiol 2006;59:665e9. [11] Audit Commission. Improving data quality in the NHS. Audit Commission 2010. Available at URL: http://www.audit-commission.gov.
[12] [13]
[14]
[15]
[16]
1433
uk/nationalstudies/health/pbr/pbr2010/Pages/default.aspx. Accessed October 2010. NHS Classifications Service 2010. Coding of comorbidities. Coding Clinic, 2010;6(6). Stukenborg GJ, Wagner DP, Connors AF Jr. Comparison of the performance of two comorbidity measures, with and without information from prior hospitalizations. Med Care 2001;39: 727e39. Lee DS, Donovan L, Austin PC, Gong Y, Liu PP, Rouleau JL, et al. Comparison of coding of heart failure and comorbidities in administrative and clinical data for use in outcomes research. Med Care 2005;43:182e8. Preen DB, D’Arcy C, Holman J, Spilsbury K, Semmens JB, Brameld KJ. Length of comorbidity lookback period affected regression model performance of administrative health data. J Clin Epidemiol 2006;59:940e6. Li B, Evans D, Faris P, Dean S, Quan H. Risk adjustment performance of Charlson and Elixhauser comorbidities in ICD-9 and ICD-10 administrative databases. BMC Health Serv Res 2008;8:12.