Assessing illness severity: Does clinical judgment work?

J CbmnDis Vol. 39, No. 6, Printed in GreatBritain

pp. 439-452,

0021-9681/86 $3.00+ 0.00 Pergamon Journals Ltd

1986

ASSESSING ILLNESS SEVERITY: DOES CLINICAL JUDGMENT WORK? MARY E. CHARLSON, FREDERICL. SAX, C. RONALD MACKENZIE, SUZANNE D. FIELDS, ROBERT L. BRAHAMand R. G. DOUGLAS, JR Clinical Epidemiology Unit, Division of General Internal Medicine, Department of Medicine, Cornell University, New York, U.S.A. (Received

in revised form 30 October

1985)

Abstract-Accurate classificationof clinical severityis importantfor interpreting casemix in clinical studies and for stratifying patients for clinical trials. To evaluate whether clinical judgment might be an effective method of estimating severity, all 604 patients admitted to the medical service in a one month period were rated at the time of admission by the responsible resident as to how sick they were. Within the 13 comorbid disease groups, and within the 15 basic categories of reason for admission, the physicians’ severity ratings were the most significant predictor of in-hospital mortality. Death rates rose from 0% in those rated as not ill, to 2% in the mildly ill, to 6% in the moderately ill, to 23% in the severely ill, and to 58% in those rated as moribund (p i 0.001). Sickness ratings also predicted time to death: mildly ill patients died after prolonged hospitalizations, while the moribund died shortly after admission. The patients’ age, sex, race, the number of comorbid diseases or problems did not predict mortality. Patients with serious comorbidity (metastases, AIDS, or cirrhosis) had a higher mortality rate than other patients (p c 0.001); however, the severity ratings predicted outcomes within this group (p < 0.001) as well as among those without such serious comorbidity (p < 0.001). Patients who were admitted with acute neurologic (p < 0.05) or acute cardiovascular (p < 0.01) events did have an independently worse prognosis. In conclusion, physicians’ estimates or sickness provided an accurate estimate of illness severity, with mortality rates that essentially tripled from one stratum to the next. Clinical judgment may suffice to classify the clinical severity of patients at the time of enrollment in prospective trials and can provide a useful method of controlling for casemix.

I. INTRODUCTION IN STUDIES of therapeutic efficacy or effectiveness, investigators must insure that patients assigned to different treatments have equal susceptibility for the development of adverse outcomes, such as morbidity and mortality. If such prognostic similarity is not achieved, then differences in outcomes may be attributable to inherent prognostic disparities rather than to the effects of treatment. In studies of acute illnesses, patients assigned to different treatments must have a comparable risk of death. Accuracy in the assessment of illness severity is critical to the validity of such studies. In a few medical conditions, scales have been developed to rate the severity of the illness, for example, the Killip and Norris classifications [l, 21 for acute myocardial infarction, and the Glasgow Coma Scale [3]. Several scales exist for assessing severity of illness in surgical patients, including the Injury Severity Score [4], the Trauma Index [5], the Comprehensive Injury scale [6], and the burn severity indices [7]. Recently, a new method (APACHE) has been developed for evaluating severity of illness among

Dr Charlsonis a Henry .I. Kaiser Foundation Faculty Scholar in General Internal Medicine. Dr Sax is a Fellow at the National Heart, Lung and Blood Institute, National Institutes of Health, Bethesda. Maryland. Dr Fields is a Henry J. Kaiser Foundation Fellow in General Internal Medicine. Address Reprint Requests to: Dr Charlson, 515 E. 71st St, New York, NY 10021. U.S.A. 439

MARY

440

E.

CHARLSONet al.

patients admitted to critical care units [8,9]. The majority of studies of medical patients, however, involve non-critically ill patients with conditions that do not have disease-specific scales to estimate prognosis. As a result, controversies commonly arise as to whether the patients did indeed have equal susceptibility to adverse outcomes, and therefore, whether the results are valid. In the day to day practice of medicine, however, physicians rely on their estimates of severity to make many clinical decisions. For example, decisions to admit patients to the hospital, to see them in the emergency room, and to initiate or change therapy are made on the basis of physicians’ assessments of how sick patients are. Despite our reliance on such clinical judgment in practice of medicine, the accuracy of such physicians’ assessments of clinical severity has never been systematically appraised. In this study, our objective was to assess whether clinical judgment might be an effective method of assessing clinical severity. If so, then physicians’ estimates of severity could provide a method for interpreting casemix in clinical studies and for stratifying patients for clinical trials. II.

METHODS

A. Assembly of Population The intended population for this study consisted of all patients admitted to the medical service at New York Hospital-Cornell Medical Center during a one month period in 1985. During this period, 607 patients were admitted to the medical service. B. Prospective Evaluations At the time of admission, the admitting resident rated how sick the patient was, using a 9 point ordinal scale. Junior residents (PGY II) made 82% of the assessments, and senior residents (PGY III), 18%. The specific question was: How sick is the patient now?

Not ill Mildly Moderately Severely Moribund 1____________2____________3____________4____________5____________6____________~____________~__ (n) Mortality

(22) 0%

(43) 0%

(125) 2%

(60) 3%

(179) 6%

(62) 21%

(82) 24%

(18) 55%

(13) 62%

The number of patients with each rating and the mortality rate for the group is listed under the scale. The nine ratings were combined to form five groups for this analysis; specifically, ratings of 1 and 2 formed not ill; 3 and 4, mildly ill; 5 moderately ill; 6 and 7, severely ill; and 8 and 9, moribund. The residents also rated other patient characteristics, namely, stability, complexity, long term prognosis, functional ability, and recorded the time and place of admission, and the reason for admission. The residents’ evaluations were collected on the morning after admission. Each patient was re-evaluated at the time of any transfer to or from the critical care units. Each patient was followed until discharge. C. Retrolective Assembly of Data After discharge, the patients’ hospital records were reviewed and data were collected about the patients’ age, race, sex, insurance coverage, and occurrence of recent hospitalizations. The urgency of admission (emergent, urgent or elective), place of admission (ward or private service, critical care units) were also recorded. The patients’ clinical characteristics such as the number and severity of comorbid diseases, the number of active problems on admission, the number of medications on admission were also noted. The subsequent course of the patient, including procedures, consults, complications, unit transfers, arrests, death and discharge were also recorded, All 607 patients were evaluated on admission; however, the charts of three patients could not be located. Therefore, the analysis is confined to 604 patients or 99.5% of the total cohort.

Assessing Illness Severity

441

D. Classljication of Reasons for Admission

The reason(s) for admission were coded from the residents’ responses, which were not recorded according to any specific criteria. The residents did not record “reason” according to any specific taxonomy; thus, the reason for admission may have been a symptom, a sign, a pathologic lesion or a pathophysiologic process. Further, the reasons were not necessarily mutually exclusive. In short, the reasons for admission were a collection of phenomena that were perceived as precipitants of admission to hospital. To analyze estimates of illness severity in the context of over 100 different reasons for admission, a method was devised to create larger strata that were reasonably homogeneous clinically and prognostically. An empirical two stage approach was employed; as such, its use is limited to the patients studied and is by no means comprehensive. First, conditions were clustered based on the affected system. Within systems, groups were created based on the qualitative similarity of the phenomena from the perspective of their life-threatening potential, and less importantly, from their clinical similarity (diagnostic and therapeutic requirements). These groups listed (as a, b, c, etc. under each affected system). The first group usually contains conditions which pose the greatest short-term threat to life, while the last contains primarily diagnostic problems (the least threat). Second the reasons for admission within each affected system were labelled “bad” if the observed mortality was greater than that for the cohort as a whole (> 11Oh), or “not bad,” if < 11%. If other cutoffs were used (e.g. 9, 13, 15 and 17%) the results did not differ substantially, and the conclusions were unchanged. There were 15 basic categories of reason for admission created by this retrospective process. 1. Neurologic: Bad

(a) Intracranial hemorrhage (2), cerebrovascular accident (11) transient ischemic attack (1), head trauma (2) subdural hematoma (1); mortality 35% (17). (b) Change in mental status (21) found unresponsive (6); mortality 19% (27). 2. Neurologic:' Not bad (a) Seizures (11) possible cord compression (8), lower extremity weakness (1), paraspinal mass (1); mortality 10% (21). (b) Syncope (21) dizziness (4); mortality 4% (25). (c) Diplopia (1) cerebellar signs (5), headache (1), recurrent blindness (1) ataxia (2); mortality 10% (10). 3. Cardiovascular: Bad (a) Cardiac arrest (1), hypotension (5); mortality 50% (6) (b) Acute myocardial infarction (lo), pulmonary edema (3) pericardial effusion (1); mortality 21% (14). 4. Cardiovascular: Not bad (a) Acute arrhythmia (11) atria1 fibrillation (14); mortality 4% (25). (b) Chest pain (72); mortality 6% (72). (c) Congestive heart failure (16); mortality 6% (16). (d) Severe hypertension (4) pheochromocytoma (I), pulmonary hypertension vein thrombosis (5) gangrene (1); mortality 8% (12).

(I), deep

5. Infectious: Bad

(a) Sepsis (4) meningitis (I), acute abdomen (2), orbital cellulitis (2); mortality 33% (9). (b) Fever (29), nadir fever (1), possible sepsis (4); mortality 18% (34). (c) Fever and mental status changes (8); mortality 25% (8).

442

MARY E. CHARLSONet al.

6. Infectious: Not bad

(a) Septic arthritis (I), urinary tract infection (2), leg cellulitis (8), zoster (3) candida esophagitis (I), osteomyelitis (1), prosthetic joint infection (I), dental abscess (1); mortality 11% (18). 7. Pulmonary: Bad (a) Acute respiratory failure (6); mortality 33% (6). (b) Acute dyspnea (34); mortality 21% (34). 8. Pulmonary: Not bad (a) Pneumonia (28), infiltrate (7); mortality 5% (35). (b) Bronchitis (7), asthma (14); mortality 0% (21). (c) Hemoptysis (1), nodule (3), pneumothorax (1), r/o pulmonary embolus (lo), effusion (3); mortality 11% (18). 9. Gastrointestinal: Bad (a) Bleeding (42); mortality 19% (42). (b) Decompensation of chronic liver disease (9); mortality 22% (9). 10. Gastrointestinal: Not bad (a) Pancreatitis (2), diarrhea (lo), flare of inflammatory bowel disease (3); mortality 7% (15). (b) Hepatomegaly (2), dysphagia (1), jaundice (2), abdominal pain (7), foreign body esophagus (1), bowel obstruction (2), biliary colic (1); mortality 6% (16). 11. Metabolic: Bad

(a) Dehydration (9), hypoglycemia calcemia (1); mortality 24% (21).

(2), hyponatremia

(3), hyperglycemia

(6), hyper-

12. Metabolic: Not bad (a) Renal failure (l), decreased renal function (5), diabetic ketoacidosis (6); mortality 8% (12). (b) Alcohol withdrawal (2), alcoholism (9), overdose (8), substance abuse (3); mortality 0% (22). 13. Hematologic: Not bad (a) Pancytopenia (20) thrombocytopenia (5); mortality 8% (25). (b) Sickle cell anemia (5) refractory anemia (6); mortality 0% (11). (c) Anemia (7); Mortality 0% (7). 14. Miscellaneous: Bad (a) Placement, Failure to thrive (27); mortality 26% (27). 15. Miscellaneous: Not bad (a) Procedures 0% (58); chronic disease (49); 0% (107). (b) Chemotherapy 8% (24). E. Classification of Comorbidity

All comorbid diseases were recorded. For example a patient with known AIDS admitted with pneumocystis pneumonia would be classified as pneumonia for reason for admission and AIDS as co-morbidity. Conditions that had completely resolved (i.e. history of pneumonia) or a history of operation for currently inactive conditions (i.e. history of cholecystectomy) were not counted as comorbidity. For the more common conditions, such as ischemic heart disease, diabetes, hypertension, data were collected characterizing

443

Assessing Illness Severity

the disease seriousness. For rare conditions, such as multiple sclerosis, no such data was collected. In the analysis, while some conditions (e.g. valvular heart disease) were not subdivided, the most common conditions were split into two or three levels, labelled A, B and C from least to most serious. The designations A, B, and C were assigned on the bases of standard clinical judgment not on the basis of the observed mortality. For most comorbid conditions, the in-hospital mortality did not differ between the A and B groups, so they were combined. Many of the C groups did do worse, so they are listed separately. However, some of the C groups (Cangina and Cdiabetes) did not have a higher mortality in this study; nonetheless, both groups of C patients would conventionally be rated as having more serious disease, so the labels were retained. The basic comorbid conditions are listed below. The abbreviations that are listed are used in subsequent equations. For each comorbid condition, the number of patients and the observed mortality is listed. (1)

Congestive heart failure C CHF: End-stage cardiomyopathy;

mortality 100% (I). AB CHF: All other patients with congestive failure; mortality

10% (90).

(2) Ischemic heart disease MI: previous myocardial infarctions; mortality; 10% (71). CANGINA: Rest, nocturnal or accelarated angina; mortality 0% (19). ABANGINA: Stable exertional angina or possible MI; mortality 5% (100). (3) Valvular heart disease AORTIC: Hemodynamically significant aortic insufficiency or stenosis; mortality 0% (12). MITRAL: Hemodynamically significant mitral insufficiency or stenosis; mortality 11% (9). VALVE: Other valvular disease, including idiopathic hypertrophic subaortic stenosis; mortality 0% (4). PROST: Prosthetic valve; mortality 14% (7). (4) Peripheral vascular disease

CPVD: Thoracic or abdominal aneurysm or gangrene; mortality 29% (7). ABPVD: Intermittent claudication; mortality 8% (25). (5) Hypertension CHBP: Diastolic greater than 120; mortality 16% (6). ABHBP: Diastolic less than 120; mortality 11% (155). (6) Diabetes mellitus CDIAB: Diabetes with retinopathy, neuropathy or nephropathy; mortality 0% (10). ABDIAB: Diabetes without these conditions; mortality 16% (37). (7) Pulmonary CPULM: Patients with a p0, < 50 or pCOz > 50 or constant oxygen; mortality 20% (5). ABPULM: Other patients with COPD or asthma; mortality 11% (83). (8) Renal CRENAL: Uremia or dialysis; mortality 16% (6). ABRENAL: Moderate to severe renal insufficiency; mortality 9% (33). (9) Liver disease CLIVER: Cirrhosis with varices; mortality 38% (8). ABLIVER: Cirrhosis or chronic hepatitis; mortality 14% (14).

444

MARY E. CHARLSONet al.

(10) Cancer

LYMPH: Lymphoma; mortality 9% (33). LEUK: Leukemia; mortality 0% (17). NMETS: Metastatic solid tumor; mortality 29% (52). CA: History of solid tumor, no evidence of metastases; mortality 9% (33). ( I I ) Acquired immune de$ciency syndrome AIDS: Definite or probable AIDS; mortality

38% (18).

(12) Neurologic PLEGIC: Hemiparesis, regardless of cause; mortality 20% (15). TIA: History of CVA without residua or transient ischemic attacks; mortality 6% (35). NEURO: Other neurologic problems, including Parkinsons and seizures; mortality 13% (32). (13) Gastrointestinal ULCER: Peptic ulcer disease; mortality 19% (32). BLEED: History of gastrointestinal bleeding; mortality 13% (31). INFLAM: Ulcerative colitis or regional enteritis; mortality 8% (12). In order to characterize the extent of comorbidity, weighted for its severity, we adapted the method described by Harrell [lo]. The comorbid diseases were clustered based on their clinical similarities and the weights were assigned using the observed mortality rates, with some corrections dictated by clinical judgment. A weight of one denotes comorbid conditions with a mortality rate close to that for the entire cohort (11%). This allowed the development of indices of myocardial damage, vascular damage, cancer damage, gastrointestinal damage, etc. which were weighted as follows: Myocardial Damage = 3CCHF + ABCHF + MI + CANGINA + ABANGINA Vascular Damage = ZCPVD + 2PLEGIC + ABPVD + TIA Valvular Damage = AORTIC + MITRAL + PROST + VALVE Hypertension Damage = 1.SCHBP + ABHBP Pulmonary Damage = 2CPULM + ABPULM Diabetes Damage = 1.5 (ABDIAB + CDIAB) Cancer Damage = 2METS + CA + LEUK + LYMPH Liver Damage = 1.SABLIVER + 3CLIVER GI Damage = BLEED + INFLAM + ULCER Renal Damage = 1.SCRENAL + ABRENAL Immune Damage = 3AIDS These separate indices were then added to form a composite comorbidity damage index, which would reflect the severity as well as the number of the separate comorbid diseases. F. Statistical Analysis

Analysis of categorical data was performed using the FUNCAT procedure in SAS [I 1] and through LOGIST, a program for logistic regression developed by Harrell [12]. The GLM procedure in SAS was used for ordinary least squares regression and for analysis of covariance [13]. Hazard rates, or the average daily risk of mortality were calculated as 2 qi/Hi (1 + pi), where qi is the probability of dying during the interval, and pi = 1 - qi [14]. hi in this study was days in hospital. Cox regression analysis (proportional hazards) was performed using PHGLM in SAS [15]. 111.

RESULTS

A. Severity ratings, demographic and admission characteristics

The overal mortality rate was 10.9%. The mortality rates rose significantly, essentially tripling from one stratum to the next, across the sickness ratings from 0% in those rated

Assessing Illness Severity TABLE

445

1. MORTALITY ACCORDING TO SEVERITY AND DEMOGRAPHIC AND ~DhilSSl0N CHARACTERlSTlcS NO1 ill

Mildly ill

Moderately ill

Severely ill

Moribund

Total

Age c49 5G63 6474 215

0% (22) 0% (22) 0% (12) 0% (9)

6% 2% 0% 0%

(47) (54) (39) (45)

5% 5% 4% 10%

(42) (38) (48) (51)

18% 27% 19% 27%

(34) (37) (36) (37)

33% (6) 100% (6) 50% (8) 54% (11)

Sex Female Male

0% (28) 0% (37)

1% 3%

(94) (91)

5% 7%

(81) (98)

23% 23%

(65) (79)

40% (20) 91%(11)

10% (288) 12% (316)

Race White Black & hispanic

0% (54) 0% (I 1)

2% (155) 3% (30)

7% (146) 3% (33)

21% (119) 32% (25)

57% (23) 62% (8)

10% (497) 14% (107)

Medicare Medicaid Private None

0% (24) 0% (10) 0% (26) 0% (5)

0% 7% 4% 0%

(85) (15) (75) (IO)

9% 0% 5% 0%

(89) (19) (59) (12)

27% 0% 26% 7%

(64) (9) (57) (14)

55% (18) 50% (4) 71% (7) 50% (2)

13% (280) 5% (57) 12% (224) 5% (43)

Diagnostic Therapeutic Both

0% (26) 0% (17) 0% (18)

2% 2% 3%

(52) (54) (79)

0% (28) 7% (42) 7% (108)

8% 23% 25%

(12) (44) (85)

50% (2) 57% (14) 58% (12)

3% (120) 12% (171) 13% (302)

6% (138) 11% (19) 0% (12) 10% (10)

20% (122) 50% (10) 33% (3) 33% (9)

56% (27) 100% (I)

3% 7% 10%

(69) (69) (41)

19% 24% 25%

(42) (70) (32)

77% (9) 33%(12) 70% (IO)

I I % (232)

6% 7% 5%

(68) (56) (55)

32% 32% 13%

(58) (38) (48)

57% (7) SS%(ll) 62% (13)

12% (232) 12% (185) 9% (187)

20.2 * 33.8 7 (l-22)

15.7 + 19.2 IO (5-18)

9% 12% 9% 7%

(151) (157) (143) (153)

Insurance

Urgency Emergent Urgent Elective Transfer

0% (32) O%((ll) 0% (22) -

< 1% (124) 1% (27) 0% (31) 0% (3)

Problems 62 34 >5

0% (44) 0% (18) 0% (3)

2% (100) 3% (63) 0% (22)

0% (30) 0% (19) 0% (16)

6% 0% 0%

5.6 If- 4.6 4 (2-7)

IIS&

18.6 f 20.2 13 (7-22)

solely for nursing

home placement

Number O-i 2-3 ,4

(3)

(443) 15% (68) 1% (68) 24% (25) 8% (264) 9% (108)

of medicines

Length of stay (days) Mean (k SD) Median Ranget

67%

I 1%

*Eleven patients admitted tlnterquartile range.

(69) (61) (55)

11.4 9 (5-14)

21.2 _+22.9 I4 (7-26) omitted.

as not ill; to 2% in the mildly ill; to 6% in the moderately ill; to 23% in the severely ill and to 58% in those rated as moribund (x2 = 69.21; p < 0.0001). Table 1 shows the mortality rates according to severity ratings and demographic as well as other characteristics. The age, sex, race, and economic (insurance) status of the patient did not influence mortality. The mortality rates were lower among patients admitted for diagnostic purposes, than for patients admitted for either therapeutic purposes, or for both. However, the difference was due to a larger proportion of sicker patients in the latter two groups; the mortality rates within the severity strata did not differ significantly. Mortality rates were slightly lower among patients admitted emergently than those admitted “urgently” (p = 0.04); the rates within the severity strata were also higher. This was primarily due to a disproportionate number of patients with metastatic cancer in the urgent admission group. Mortality rates did not differ according to the number of active problems on admission, nor the number of medicines taken by the patient. Finally, the median length of stay was lowest in the not ill (4 days), and rose to 9 days in the mildly ill. The moderately and severely ill had a median length of stay of 13-14 days. A preponderance of early deaths in the moribund patients reduced their median length of stay to 7 days. B. Severity ratings and the time to death The sickness ratings also were predictive of the time to death, as shown in Table 2. Among the mildly ill patients, none died during the first two weeks of hospitalization. Deaths among patients rated as mildly ill occurred after prolonged hospitalization. Among

446

MARY E. CHARLSON et al.

TABLE

2.

AVERAGE

DAILY

RISK

OF

DEATH

ACCORDING

TO

LENGTH

OF

HOSPITALIZATION

Length of hospitalization Not ill Mildly ill Moderately ill Severely ill Moribund

1-2 weeks

3-4 weeks

5-6 weeks

7-8 weeks

0% 0% 0.3% 1.2% 5.4%

0% 0.3% 0.4% 0.9% 0.8%

1.7% 0.6% 0.9% 2.6%

oo/.(s)* 0% (14)f 0.9% (18)t: 0% (3)6

(65) (185) (179) (144) (31)

(3) (44) (76) (66) (12)

(12) (36) (66) (7)

Numbers in parentheses represent patients per group remaining in hospital at the beginning of the interval. *The one patient in hospital in this group after nine weeks survived hospitalization. tThere are no deaths among the 7 patients remaining in the hospital after nine weeks. SThere are 4 deaths among the 10 patients in this group remaining in the hospital after 9 weeks. $There is 1 death among the 2 patients remaining in the hospital after 9 weeks.

the 44 mildly ill patients who remained in the hospital for 3-4 weeks, the daily risk of death rose to 0.3%; the weekly risk would be approximately 2%. Among the 12 mildly ill patients who remained in the hospital 5-6 weeks, the risk rose to 1.7% per day. Among the moderately ill, the risk or hazard rates rose from 0.3% per day in the first 2 weeks to 0.6% per day after 5-6 weeks. Among the severely ill, the hazard rates were almost constant over time, averaging about 1% per day; this was the only group whose hazard rates did not change significantly with time. Among the moribund, the hazard rate was 5.4% per day during the first two weeks. In short, the mildly ill died only after prolonged hospitalization; the moribund, immediately, and the moderately and severely ill had intermediate patterns. Importantly, there is a constant relationship of the hazard rates in weeks l-2 and 334 after admission, suggesting that the physicians initial estimates remain accurate through this time period. For example, there is a stepwise increase hazard rates across the severity groups in the first two weeks (0, 0.3, 1.2 and 5.4%) and in the third and fourth weeks (0, 0.3,0.4,0.9 and 0.8%). This suggests that the rank order of physicians estimates of severity on admission in terms of probability of death basically hold for the first four weeks of hospitalization. C. Severity ratings and the reasons for admission

The reasons for admission were grouped into 15 major categories, according to the strategy outlined in the Methods Section. The objective of the strategy was to create groups of patients that were clinically and prognostically similar and to assess whether the sickness ratings create gradients in mortality across groups of prognostically similar patients. Table 3 shows the mortality rates according to these categories and the severity ratings. The mortality rates differed significantly across the severity ratings for most of the basic categories of reason for admission. For example, among patients admitted with “bad” neurologic reasons for admission (including intracranial hemorrhage, head trauma, subdural hematoma, cerebrovascular accident, etc.), none of the not ill or mildly ill patients died, but 7% of the moderately ill; 33% of the severely ill; and 80% of the moribund. In total, 35% (209) of patients had “bad” reasons for admission, such as myocardial infarction, pulmonary edema, respiratory failure, cardiac arrest, acute cerebrovascular accident, coma, sepsis, severe dehydration, decompensated cirrhosis, altered mental status, and acute dyspnea. As was intended by the classification of reasons for admission as bad or not bad, patients in the bad group had a 22% mortality, significantly higher than the 5% mortality among the 395 patients with “not bad” reasons for admission (x2 = 40; p < 0.001). However, controlling for the severity ratings, only admission with acute neurologic events (p = 0.04) and with bad cardiovascular events (p = O.Ol), such as hypotension, arrest, myocardial infarction or pulmonary edema were significant independent predictors of mortality.

Assessing Illness Severity

447

TABLE 3. MORTALITY RATES ACCORDING TO CLINICAL SEVERITY AND REASON FOR ADMISSION

Reason for admission Neurologic I. Bad 2. Not bad Total Cardiovascular 3. Bad 4. Not bad Total

Not ill 0%

Mildly ill (2)

Gastrointestinal 9. Bad IO. Not bad Total

(5)

.Cherely ill

7% (14)

33%(18)

g+ 0 0%

0%

(2)

(7)

9%(ll)

8% (13)

30% (20) 6% (125) 9% (145)

p < 0.05 p < 0.05 p < 0.001

22%

(50)

+g# 0

p < 0.02 NS p < 0.01

10% (IO)

0%

(4)

0%

(2)

O%(ll)

14% (14)

+gg D

A 57% (7)

23% (40) 9% (74) 14% (114)

NS p < 0.05 p < 0.01

40% (15)

100% (3)

20%

(51)

zg 0

m

iz# 0

p c 0.01 NS p < 0.001

(21)

p < 0.05 NS p < 0.05

(43)

p < 0.05

26% (27) 2% (133)

p < 0.05 NS

*

21% (19)

5% (20)

$+ 0

22%

80% (5)

(6)

D 71% (7)

A 71% (7)

z+ 0

oo/,

(9)

57% (7)

Metabolic II. Bad 12. Not bad Total

fl 0

Hematologic 13. Not bad

0%

(6)

0% (19)

0% (IO)

17%

(6)

50% (2)

Miscellaneous 14. Bad 15. Not bad

0% (3) 0% (42)

0% (3) 4% (46)

11% (9) 0% (26)

44% II%

(9) (9)

67% (3)

0%

(I)

0%

(6)

fi 0

13%

(8)

-!g 0

p
33%

80% (5)

Total

25% (44) 7% (56) 15% (100)

+g$ 0

ti 0

Moribund

A 80% (5)

E++0

Infections 5. Bad 6. Not bad Total Pulmonary I. Bad 8. Not bad Total

0%

Moderately ill

100%

(2)

50% (4) +$j+ 0

i!Fg 0

24% iF+ 0 5%

In this analysis, a patient who has two or more reasons for admission was counted twice; however, the results do not differ significantly if the data is analyzed restricting the reasons for admission to one per patient (the one with the highest mortality rate). D. Severity ratings and comorbidity 1. The damage index. The comorbidity damage index, as described in the Methods, is a scale that estimates the burden of comorbid disease in a given patient, by taking into account the number of separate comorbid diseases, weighted for their severity. When the 15 categories of reason for admission were stratified by the comorbidity damage index, the mortality rates did not differ between patients who had higher or lower damage index scores. Mortality rates differed according to the comorbidity damage index scores only between those with ratings of O-l and those with ratings of 2-5; the mortality rates were 8 and 15%, respectively (x2 = 7.4; p < 0.01). Table 4 shows the mortality rates for patients according to the comorbidity damage index, and according to whether the reason for admission was “bad” or “not bad”. Fewer patients with “bad” reasons for admission were rated as not ill than those with “not bad” reasons (3 vs 15%; p < 0.001). More patients with “bad” reasons for admission were rated as moribund than those with “not bad” reasons (13 vs 1%; p < 0.001). However, within each of the four damage-reason groups, there was a wide spectrum of severity ratings. Within each of the damage-reason groups, patients with different severity ratings had significant differences in mortality. With the exception of two groups which had only four patients (both moribund groups in the “not bad”), the severity ratings resulted in stepwise increases in mortality with the damage-reason groups. Among patients with “bad” reasons for admission, the patients who had damage index scores of 22 did fare worse than those with a lower score (29 vs 14%, p < 0.02). This is due to the extremely high mortality rate among moribund patients with damage scores > 2. In short, patients with “bad” reasons for admission fared worse, but with this one

448

MARY E. CHARL~ON et al.

TABLE4. MORTALITYRATESBY SEVERITY, TYPEOF REASONFOR ADMISSION AND COMORBIDITY DAMAGEINDEX Reason: “not bad”?

Reason: “bad”*

Severity

-Comorbidity damage1 index 12

Not ill Mildly ill Moderately ill Severely ill Moribund

0% 4% 6% 30% 40%

Total

14% (97)

(4) (25) (35) (23) (IO)

Comorbidity damage index 22 0% 6% II% 33% 82%

(3) (17) (36) (39) (17)

29% (112)

Total 0% 5% 8% 32% 48%

(7) (42) (71) (62) (27)

22% (2091

Comorbidity damage index <2 0% 2% 4% 18% 0%

(39) (90) (56) (39) (3)

5% (227)

Comorbidity damage index 22 0% 0% 6% 14% 0%

Total

(19) (53) (52) (43) (I)

0% (58) I% (143) 5% (108) 17% (82) 0% (4)

5% (168)

5% (395)

*Bad reasons for admission include all bad conditions, i.e. myocardial infarction, cerebrovascular accident, severe dehydration, GI bleeding, pulmonary edema, and acute dyspnea, as listed in the Methods. If a patient had two reasons for admission and one was bad, they would be classified as bad in this Table. tNot bad reasons for admission include all not bad conditions, such as chest pain, syncope, pneumonia, jaundice, as listed in the Methods. $Tbe comorbidity damage index assesses the number and severity of the comorbid condition.

exception, the burden of comorbid disease, as measured by the comorbidity damage index, did not make a difference in mortality. 2. Specific comorbid diseases. The mortality rates according to the major comorbid diseases and the severity ratings are shown in Table 5. Within almost every category of comorbid disease, the severity ratings produced significant gradients in mortality rates. Patients with metastatic disease (29%), AIDS (39%), and those with liver disease (23%) (predominantly cirrhosis with portal hypertension) had significantly higher mortality rates (p < 0.001; p -C0.001; and p < 0.01, respectively) than those without such diseases (11%). These three diseases explained the finding that patients with a damage index score 22 fared worse with “bad” reasons for admission. When patients with these three conditions were removed, patients with higher damage index scores did not fare worse than those with lower scores. Thus, patients with any given comorbid diseases had a broad spectrum of reasons for admission. For most of the comorbid conditions, about 25% of the patients were admitted for reasons related to their underlying disease. However, 50% of patients with chronic pulmonary disease were admitted for acute or non-acute pulmonary problems, and 36% of patients with liver disease were admitted for decompensated cirrhosis. E. Reason for admission, comorbidity and severity

The physicians’ rating of illness severity was the most significant predictor of mortality. Patients admitted with acute neurologic (p = 0.04) or cardiovascular (p = 0.01) events and TABLE5. MORTALIN RATES:CLINICALSEVERITY ANDCOMORBID DISEASE Not ill

Comorbid disease Metastatic cancer Other cancer* AIDS Chronic liver diseaset Peptic ulcer disease Myocardial diseaset Cerebrovascular diseas4 Peripheral vascular disease11 Angina (alone) Chronic pulmonary disease Diabetes meltitus Chronic renal disease Hypertension Valvular heart disease

0% (4) 8;

#

0;

(I) 0% (4) 0% (4) 0% 0% 0% 0% 0% 0% 0%

(2) (17) (3) (9) (I) (16) (3)

Mildly ill

Moderately ill

Severely ill

Moribund 100% (5) 50% (4) 50% (2) 100% (I) 100% (2) 87% (8) 67% (3)

29% 10% 38% 23% 19% 15% 10%

50% (2) 50% (4) 57% (7) 100% (I) 100% (2) 55% (9) 100% (2)

13% (31) 4% (119) II% (88) 11% (47) 10% (39) I I% (161) 6% (32)

1% 0% 25% 0% 0% 0% 0%

(14) (18) (4) (2) (8) (20) (16)

20% (IS) 4% (26) 33% (3) 16% (6) 0% (13) 8% (25) 0% (13)

42% (14) 19% (27) 50% (8) 25% (12) 44% (9) I I% (18) 30% (IO)

0% 0% 4% 0% 0% 0% 0%

(6) (34) (25) (12) (II) (43) (II)

10% (IO) 3% (38) ll%,(26) 0% (17) 7% (14) 4% (51) 0% (II)

18% (II) 8% (26) 7% (27) 50% (8) 9%(ll) 24% (42) 0% (5)

Total (52) (82) (18) (22) (32) (75) (46)

P
p p p p p

*Includes lymphoma, leukemia, and solid tumors without metastases. tlncludes cirrhosis with and without portal hypertension and chronic hepatitis. @rcludes congestive heart failure and/or history of myocardial infarction, excluding patients with valvular disease. §Includes cerebrovascular accident with or without residual, and transient ischemic attack. 1IIncludes intermittent claudication, s/p vascular bypass, thoracic or abdominal aneurysm, and gangrene.

NS < 0.001 < 0.01 co.02 < 0.001 < 0.001 NS

Assessing Illness Severity 6. MORTALITY

TABLE

RATES

BY

SEVERITY, COMORBIDITY,

AND

449

REASON

FOR

ADMNON

WITHOUT

SERIOUS

COMORBIDITY* Reason

NO1

for

Mildly

ill

admission

Moderately

ill

Severely

ill

ill

Moribund

Total

Bad reasons (CV Not

neuro)t

0%

(3)

0%

(5)

6%

(18)

30%

(20)

77%

(9)

reasonsf

0%

(3)

3%

(32)

7%

(43)

24%

(29)

45%

(I I)

0%

(53)

<1%(130)

3%

(97)

10%

(69)

0%

(59)

I%

and

Other

bad

bad reasons$

Total

(167)

4% (158)

25%

(55)

17%(118)

0%

(4)

3% (353)

17%(118)

50%

(24)

9% (526)

With serious comorbidity Bad reasons (CV Not

0%

and neuro)t

Other

bad bad

0% comorbidity

(AIDS) tBad

50%

(4)

100%

(I)

33%

(9)

29%

(7)

55%

(9)

83%

(6)

48%

(27)

(6)

7%

(13)

18%

(II)

46%

(I9

25%

(42)

11%

(18)

19%

(21)

50%

(26)

(7)

35%

(78)

includes

patients

with

m&static

solid tumors,

acquired

71% immune

deficiency

syndrome

and cirrhosis.

reasons (CV

cardiac

(3)

(5)

-

00/,(5)

reason@

Total *Serious

0%

20%

(I)

reasons1

arrest,

and neuro)

myocardial

and cardiovascular #All

other

§All

not bad

reasons

bad reasons reasons

includes infarction,

admitted

pulmonary

with cerebrovascular

edema,

etc. as listed

accident,

subdural

in the A sections

hematoma,

of the neurologic

for admission.

for admission, for

patients

admission,

as listed as listed

in the methods. in the methods.

those with metastatic disease, AIDS or chronic liver disease (p = 0.02) had significantly worse outcomes. These findings held, regardless of the multivariate technique that was employed (i.e. logistic regression, ordinary least squares regression, or categorical analysis). Table 6 shows the mortality rates according to severity ratings and the reason for admission among patients with serious comorbidity (metastatic disease, AIDS and chronic liver disease) and those without serious comorbidity. Among patients without such comorbid diseases, the reason for admission did make a difference in terms of mortality, specifically, patients with “bad” reasons fared worse. Within the group of patients with either of these three comorbid conditions, the mortality was high, regardless of the specific reason for admission. When severity, reason for admission and comorbidity were taken into account, no other feature was a significant predictor of mortality (e.g. age, race, sex, the urgency of admission, the number of acute problems, and occurrence of recent hospitalizations). When the mortality rates in the severity strata were adjusted, using analysis of covariance for the reason for admission and comorbidity, the adjusted mortality rates were 0% in the not ill, 2% in the mildly ill, 3% in the moderately ill, 19% in the severely ill and 53% in the moribund. The differences in mortality rates between severity strata after adjustment were significant (p c 0.001). The adjusted mortality rates are almost identical to the unadjusted rates, with the exception of the moderately ill group (which is 3% less after adjustment). IV.

DISCUSSION

The primary objective of a measure of illness severity is to assess the probability of mortality. Accurate methods of predicting the short term clinical course of patients have proven importance for both clinical practice and research. The Killip classification for acute myocardial infarction is the best known disease-specific index of illness severity [l]. This index is utilized widely because it is clinically sensible and is a powerful predictor of mortality [16, 171. Even those investigators who have argued that it is preferable to use such dimensional measures as cardiac index and pulmonary capillary wedge pressure to characterize infarct severity have had to concede that the clinical assessments are as accurate prognostically as the hemodynamic assessments [17]. For studies of therapies designed to alter the probability of dying from acute myocardial infarction, the dangers of ignoring prognostically cogent features are well documented [ 181.For example, Mitchell

450

MARY E. CHARLSONet al.

argued in his commentary about acute myocardial infarction that the key question about a trial of nitroprusside [ 191 was: “Were the groups comparable in all respects other than the regimen under scrutiny? The answer would have been no, because 23% of the control groups were in heart failure on admission but only 15% of the nitroprusside group [20].” The prognostic imbalance between the two treatment groups opened the trial to damaging criticism about the validity of its results. This could have been avoided had patients been randomized within prognostically comparable strata. The importance of stratification for “sickness” or illness severity in the design of studies has been eloquently argued by Feinstein: “These clinical distinctions, if not analyzed will be major sources of unrecognized bias in any clinical survey or trial. Vaguely aware of the bias, the statistician may assume he can remove it by random allocation of therapy without attempting to stratify the patients into different clinical subgroups. . . or that he can remove the non-random bias if the population sampled is large enough or sufficiently analysed by various numerologic formulas . . . no amount of statistical manipulation can possibly identify what was initially left unspecified [21].” Today, more physican investigators have faith in the validity of using multivariate techniques to adjust prognostic imbalances, while more statisticians would agree with Feinstein that such adjustments are problematic [22]. As Simon argues with respect to cancer trials: “the variability in prognosis among patients is greater than the size of the treatment differences usually seen. Consequently, failure to understand and adequately account for patient heterogeneity easily leads to unreliable claims and inefficient trials [23].” For a few conditions, disease-specific measures of illness severity have been developed. Apart from those designed for patients with acute myocardial infarction [ 1,2], the majority of indices for short term outcomes have been designed for either surgical patients [4-71 or for the critically ill [8,9]. The APACHE system was designed to duplicate physicians’ assessments of critically ill patients. This index employs physiologic variables, primarily laboratory data, weighted for the extent of abnormality, to calculate a physiologic score which is used to measure the severity of illness [24]. While it has proven utility in critical care units, this index has not been demonstrated to be a valid measure of short term outcomes in other settings, although some studies are underway [25]. Several systems have been developed to assess illness severity retrospectively; they are based primarily on discharge, diagnosis, and status. One system for measuring illness severity involves grouping discharge ICDA codes [26,27] into four groups: no problems, problems in one organ system, problems in multiple organ systems, and death. Another system involves the evaluation of seven variables, including stage of principal diagnosis, response to therapy, and complications, on a four point ordinal scale; the rater then selects an overall severity rating for the admission by “implicitly integrating” the values of these variables [28]. While such systems may be useful in the studies of resource utilization, they miss the clinical nuances required in evaluating patients for studies of therapeutic modalities. At present, the primary method of assessing casemix for studies of health care utilization is the diagnosis-related groups (DRG) system developed at Yale [29]; the DRG’s were defined so that length of stay and total charges would be similar within each group. The system is designed to insure comparable resource utilization within each group, not comparable clinical outcomes. In fact, the use of DRG’s to control for casemix in evaluating clinical outcomes may be misleading. For example, lower resource utilization may occur in conditions that lead to early demise. Therefore, measures of casemix designed for assessment of resource utilization cannot be assumed to be valid as methods of measuring illness severity in clinical studies [30]. Similarly, the traditional indices of morbidity and mortality or our method based on physician estimates of clinical severity should not be used as the sole measure of casemix for studies of resources, cost or reimbursement. Specifically, we are not suggesting that physicians’ estimates of clinical severity should be used as a measure of casemix for reimbursement. In this study, physicians’ estimates of chnical severity were remarkably accurate predictors of mortality, with mortality tripling between each successive severity strata. In

Assessing Illness Severity

451

a study of patients admitted to an intensive care unit, Detsky et al. found that mortality was overpredicted (survival underestimated) in the sickest patients. Admitting residents estimated the actual probability that a patient would die, and from these estimates, five groups were created and from best to worst, the observed mortality rates were 3, 6, 20, 37 and 76% [31]. This is the only other study that evaluates the accuracy of physicians’ ability to predict risk of death through estimates of illness severity, and the essential trends were similar to those found in this study. A physician’s ability to accurately assess illness severity is critical to the practice of medicine. Many therapeutic decisions hinge on the accuracy of physicians’ clinical judgment of severity. As crucial as such assessments are, physicians are never explicitly taught how to judge how sick patients are. In fact, as Feinstein argues: “We often do not define the word (sickness), but we know what it communicates: a sense of urgency in the need for treatment, or a poor prognosis, or a difference between two patients with the same diagnosed disease . . . [32]” With several disease-specific exceptions, in the main, we do not know how physicians make such assessments [33]. This study does not address the issue of what elements physicians use in their assessments of how sick patients are. It does suggest that physicians estimates are, in the aggregate, sufficiently accurate to make further exploration of this issue worthwhile. Our data also suggest that the reason for admission and comorbidity are necessary but not sufficient to assess risk of mortality. For example, if patients admitted with sepsis, acute myocardial infarction, respiratory failure or variceal bleeding are invariably rated as sickest, then the clinical judgment involved in assessing illness severity could be reduced to a set of straightforward decision rules based on reason for admission, weighted by the extent and severity of comorbidity. Our data demonstrates that within any reason for admission, controlling for the burden of comorbidity, there can be a wide spectrum of illness severity and concomitantly differing outcomes. Physicians’ estimates of severity may, in fact, illustrate the old axiom that the whole may be more than the sum of the parts [34,35]. An index of clinical severity would be extremely important for clinical research, primarily for use in studies designed to evaluate therapeutic efficacy or effectiveness. A reliable index would allow randomization of patients stratified by severity, and eliminate the fear that the two groups under study were not truly balanced in terms of their short term risk. Further, the results of any therapeutic intervention could then be analyzed in each of the severity groups, making it clearer which patients derive benefit and to which patients the results do apply. If clinical severity was assessed by physicians responsible for the patients, but not involved in the actual conduct of the trial (blind assessors), such clinical judgments could be used for stratifying patients prior to randomization, and trial results could be analyzed directly within severity strata. Before such clinical assessments of severity could be used in clinical trials, studies would have to demonstrate that such estimates are reproducible between physicians. Dr Joseph Ruggiero, Dr David Miller and the medical housestaff of The New York Hospital for their cooperation and assistance in the conduct of this study.

Acknowledgemenrs-To

REFERENCES I. 2. 3. 4. 5.

Killip T, Kimball JT: Treatment of myocardial infarction in a coronary care unit a two year experience with 250 patients. Am J Cardiol 20: 457-464, 1967 Norris RM, Brandt PW, Caughey DE ef al: A new coronary prognostic index. Lancet I: 274-278, 1969 Jennet B, Bond M: Assessment of outcome after severe brain damage. Lancet I: 48&484, 1975 Baker SP, O’Neill B, Haddon W et al: The injury severity score: A method for describing patients with multiple injuries and evaluating emergency care. J Trauma 14: 187-196, 1974 Kirkpatrick J, Youmans R: Trauma index: An aid in the evaluation of injury victims. J Trauma 1I: 71 l-714, 1971

6. 7.

Committee on Medical Aspects of Automative Safety: Rating the Severity of Tissue Damage II. The Comprehensive Scale. JAMA 220: 277-280, 1972 Gustafson C, Holloway DG: A decision theory approach to measuring severity of illness. Health !3erv Res IO: 97-104, 1975

452 8. 9. IO. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. 32. 33. 34. 35.

MARY E. CHARL~ONet al. Scheffler RM, Knaus WA, Wagner DP et al: Severity of illness and the relationship between intensive care and survival. Am J Pub1 Health 72: 449-454, 1982 Wagner DP, Knaus WA, Draper EA: Statistical validation of a severity of illness measure. Am J Pub1 Health 73: 878-884, 1983 Harrell FE, Lee KL, Califf RM et al: Regression modelling strategies for improved prognostic prediction. Stat Med 3: 143-152, 1984 Statistical Analysis System Institute: the FUNCAT Procedure. SAS User’s Guide. Cary, North Carolina: SAS Institute, 1979. pp. 221-236 Harrell F: The LOGIST Procedure. Supplemental Library User’s Guide. Cary, North Carolina: Statistical Analysis System (SAS) Institute, 1983. pp. 181-202 Statistical Analysis System Institute: The GLM Procedure. SAS User’s Guide. Cary, North Carolina: SAS Institute, 1979. pp. 245-264 Biomedieai Computer Programs, P&&s. Dixon WJ, Brown MB (Eds). Berkeley: University of California Press, 1979. p. 747 Harrell F: The PHGLM Procedure. Supplemental Library User’s Guide. Cary, North Carolina: Statistical Analysis System (SAS) Institute, 1983. pp. 267-294 Eaton LW, Weiss JL, Bulkley BH et al: Regional cardiac dilatation after acute myocardial infarction. N Engl J Med 300: 5762, 1979 Forrester JS, Diamond G, Chatterjee K. et al: Medical therapy of acute myocardial infarction by application of hemodynamic subsets. N Engl J Med 295: 13561362, 1976 Gifford RH, Feinstein AR: A critique of methodology in studies of anticoagulant therapy for acute myocardial infarction. N Engl J Med 280: 351-357, 1969 Mitchell JR: “But will it help by patients with myocardial infarction ?” The implications of recent trials for everyday country folk. Br Med J 285: 1140-l 148, 1982 Durrer JD, Lie KI, Van Capelle FJ er al: Effect of sodium nitroprusside on mortality in acute myocardial infarction. N Engl J Med 306: 1121-1128, 1982 Feinstein AR: Clinical Judgment. Baltimore: Williams and Wilkins, 1967. p. 224 Simon R: Use of regression models: statistical aspects. Cancer Clinical Triak Design, Practice and Analysis. Buyse ME, Staquet MJ, Sylvester RJ (Eds). Oxford: Oxford Press, 1983. pp. 445-466 Simon R: Importance of prognostic factors in cancer clinical trials. Cancer Treat Rep 68: 185192, 1984 Knaus WA, Zimmerman JE, Wagner DP et al: APACHE-Acute Physiology and Chronic Health Evaluation: A physiologically based classification system. Crit Care Med 9: 591-597, 1981 Knaus WA, Wagner DP, Draper EA: The value of measuring severity of disease in clinical research on acutely ill patients. J Cbron Dis 37: 455-463, 1984 Gonnell JS, Hornbrook MC, Louis DZ: Staging of disease: A case-mix measurement. JAMA 251: 637-644, 1984 The International Classification of Diseases: Clinical Modifications, 9th revision. Ann Arbor, Michigan: Commission on Professional and Hospital Activities, 1978 Horn S: Measuring severity of illness: comparisons across institutions. Am J Pub1 Health 73: 25-31, 1983 Fetter RB, Shin Y, Freeman JL et al: Case mix definition by diagnosis-related groups. Med Care 18 (Suppl): 1-53, 1980 Kreitzer SL, Loebner ES, Roveti GC: Severity of illness; The DRG’s missing link. Q Rev BuII 8: 21-34, 1982 Detsky AS, Stricker SC, Mulley AG er al: Prognosis, Survival and the Expenditure of Hospital Resources for Patients in an Intensive-Care Unit. N EngI J Med 305: 667-672, 1981 Feinstein AR CIIaIcaI Judgment. Baltimore: Williams and Wilkins, 1967, p. 224 Jalan KN. Prescott RJ, Sircus N. et al: Ulcerative colitis a clinical study of 399 patients. J R COB Surg 16: 338-355, 1971 Krischer JP: Indexes of severity: underlying concepts Health Serv Res 11: 143-157, 1976 Krischer JP: Indexes of severity; conceptual development. Health Serv Res 14: 56-67, 1979