- Email: [email protected]
Predictive validity of the nursing severity index in patients with musculoskeletal disease
J Clin
Pergamon
0895-435q94)00142-1
Vol. 48. No. 2. DD. 179-188. 1995 Copyright (’ 1995 El&r Science Ltd Printed in Great Brilain. All rights reserved
Epidemiol
089%4356/95
PREDICTIVE VALIDITY INDEX IN PATIENTS GARY E. ROSENTHAL’, C. SETH LANDEFELD+’
$9.50 + 0.00
OF THE NURSING SEVERITY WITH MUSCULOSKELETAL DISEASE
EDWARD J. HALLORAN2, MARYLOU KILEY3, and THE NURSES OF UNIVERSITY HOSPITALS OF CLEVELAND
‘Section of Clinical Epidemiology, Division of General Internal Medicine, Department of Medicine, Cleveland Veterans Administration Medical Center, Case Western Reserve University School of Medicine, Cleveland, OH, U.S.A., %chool of Nursing, University of North Carolina, Chapel Hill, NC, U.S.A., ‘Frances Payne Bolton School of Nursing, Case Western Reserve University, Cleveland, OH, U.S.A. and “Section of Clinical Epidemiology, Divisions of General Internal Medicine and Geriatrics, Department of Medicine, Cleveland Veterans Administration Medical Center and University Hospitals of Cleveland, Case Western Reserve University School of Medicine, Cleveland, OH, U.S.A. (Received in revised form 13 July 1994)
Abstract-Prior studies have not examined the validity of severity of illness instruments in patients at low risk for mortality. We, therefore, examined the predictive validity of a newly developed instrument, the Nursing Severity Index in 5347 adult medical and surgical patients with musculoskeletal diagnoses admitted to an academic medical center in 1985-88. The Index is based on aggregating 34 clinical observations which were recorded by primary nurses during patient care; observations reflect biologic, functional, cognitive and psychosocial abnormalities. Other data, including patient demographic data and outcomes were obtained from hospital data bases. We found that, among all study patients, admission Nursing Severity Index scores were highly related (p < 0.001) to in-hospital death rates-which were 0, 0.4, 0.8, 2.6, 6.7 and 23.5% in six hierarchical strata defined by the Index-and to nursing home discharge rates. In multivariate analyses, adjusting for diagnosis and other important covariates, each strata was associated with a 2.5-fold increased risk of mortality and a 1.6-fold increased risk of nursing home discharge. In addition, the Nursing Severity Index was an independent predictor (p < 0.001) of hospital charges and length of stay. We conclude that the Nursing Severity Index assesses multiple dimensions of illness, can be easily recorded during routine patient care, and accurately predicts hospital outcomes in an important ‘low risk’ group of patients. The validity of the Nursing Severity Index in other clinical subgroups should be further studied. Health services research Quality of health care ment Mortality Length of stay
INTRODUCTION
Efforts to measure and improve the quality and efficiency of hospital care have been increasingly based on the examination of patient outcomes [I, 21. These efforts are exemplified by several initiatives, including the yearly release of hospital mortality data for Medicare patients
Outcome and process assess-
by the Health Care Financing Administration (HCFA) [3,4], the release of comparative hospital reports by Pennsylvania [5], Iowa [6] and New York [7] and the emergence of regional provider-purchaser coalitions in Cleveland [8] and other cities. Because hospitals differ in the types of patients that they care for, the validity of out179
Gary E. Rosenthalet al.
180
comes-based efforts depends on the availability of accurate and easily applied methods of adjusting for severity of illness and other underlying patient characteristics that impact hospital outcomes. Although several instruments for measuring severity of illness (i.e. a patient’s underlying risk for adverse events or propensity for resource utilization) have been proposed [9917], few have been subjected to rigorous independent scrutiny [ 1G-211. In addition, current instruments principally assess pathophysiologic aspects of disease and have generally excluded other important predictors of outcomes, such as comorbid illness and physical, social and emotional functioning [22228]. Furthermore, the validity of these instruments has been established primarily in patients whose risk of death is relatively high (e.g. patients in intensive care units or hospitalized for pneumonia or stroke) [7, 11, 16, 17,20,21]; thus, little is known about the applicability of current instruments to patients in whom the risk of mortality is low (e.g. orthopedic patients) or to patients whose functional limitations and/or psycho-social needs may be out of proportion to acute physiologic aberrations. To address these limitations, we recently reported the development and validation of a new severity instrument, the Nursing Severity Index, in a heterogeneous sample of hospital patients [29]. The Nursing Severity Index is unique among severity measures in that it assesses functional, cognitive, psychosocial and biologic abnormalities and is based on the clinical observations of nurses. We conducted the current study to examine the predictive validity of the Nursing Severity Index in a ‘low risk’ cohort of hospital patients who are likely to have significant functional impairmentpatients with musculoskeletal disease. We hypothesized that the Nursing Severity Index would stratify this cohort with respect to several important hospital outcomes, including inhospital mortality, length of stay, hospital charges and discharge to nursing homes. METHODS
Patients The study was conducted at University Hospitals of Cleveland, an 874-bed major teaching hospital of the Schools of Medicine and Nursing of Case Western Reserve University. The study included patients aged 18 years and
older who were admitted between 3 November 1985 and 3 1 December 1988 and who were classified into diagnosis related groups (DRGs) for musculoskeletal disease (DRGs 209-2.56, 471). Of the 7345 eligible patients admitted during this period, admission Nursing Severity Index assessments were available for 5347 patients (73%), who form the study population. The 5347 study patients and the 1998 other patients were similar in mean age (53.6 vs 53.8 years, respectively) and gender (58% vs 58% female, respectively) and had nearly identical mean lengths of stay (8.6 vs 8.7 days, respectively); the two groups differed somewhat in race (82% vs 77% white, respectively; p < O.OOl), type of health insurance (54% vs 47% with non-governmental insurance plans, respectively; p < O.OOl), and the proportion undergoing a surgical procedure (72% vs 66%, respectively; p < 0.001). Based on DRGs, patients were classified as surgical (DRGs 2099234, 471) or medical (DRGs 235-256). Data Patient age, sex, race, marital status, type of health insurance, length of stay, total hospital charges, discharge vital status, discharge destination, DRG and DRG weight were obtained from computerized data bases. The DRG weight is a relative measure of expected resource use associated with each DRG that is used to determine hospital reimbursement for Medicare and other patients; patients were classified into approximate quartiles, based on the DRG weight assigned to their DRG. Type of health insurance was categorized into four groups according to the primary payer:
6) Medicare; (ii) private (Blue Cross/Blue Shield and other private carriers; (iii) managed care (health maintenance organizations and preferred provider organizations); and (3 Medicaid or uninsured. Patients were categorized as DRG outliers if their length of stay exceeded published Medicare outlier thresholds for each DRG during each year of the study. For each patient, Nursing Severity Index assessments were recorded by primary ward nurses on the day of admission or the next hospital day. All Nursing Severity Index assessments were made by clinical nurses in the course of routine patient care and were recorded in one
Severity Adjustment
to two minutes using bar-code data entry. For patients in whom more than one assessment was available, only the initially recorded assessment was analyzed. The Nursing Severity Index is based on the presence or absence of 34 specific nursing diagnoses (see Appendix), which were found to be independently related to in-hospital mortality, [29]. Nursing diagnoses are widely used in patient management [30,31] and can be recorded with similar reliability as physical examination findings, diagnostic test interpretations and medical diagnoses [29]. Explicit clinical criteria were developed for each of the diagnoses used and all nurses underwent formal training to standardize application of the diagnoses. Nursing Severity Index scores equal the number of the 34 diagnoses noted to be present by the clinical nurse. In previous analyses [29], weighting individual nursing diagnoses according to coefficients from regression analyses did not significantly improve the performance of the Index, as measured by receiver operating characteristic (ROC) curve analysis (see below). Among the 5347 study patients, scores ranged from 0 to 24. In our analyses, scores were categorized into six hierarchical strata: O-3,&6, 7-9, 10-12, 13-15 and 216. Analysis Analyses were performed using mainframe SAS. Associations between categorical variables were tested using the x2 test. Differences between groups for continuous variables were tested using the t-test or Wilcoxon signed rank test. Bivariate associations between Nursing Severity Index scores and dichotomous outcomes (death, nursing home discharge and DRG outlier status) were determined using the x2 test, with the modification for linear trend. Performance of the index for dichotomous outcomes was measured using ROC curve area analysis [32], which examines the ability of an index to discriminate between patients with and without an outcome. ROC curve areas range from 0.5, which indicates no discrimination, to 1.O, which indicates perfect discrimination. Bivariate associations between Nursing Severity Index scores and two measures of hospital resource use (length of stay and charges), were determined using simple linear regression; performance was assessed by the total variance (R*) in length of stay and charges explained by the index.
181
in Low Risk Patients
Logistic regression analysis [33] was used to examine the independent association between the Nursing Severity Index and in-hospital death, controlling for several potential confounders, including age, sex, race, marital status, type of health insurance and the observed DRG-specific mortality rate. By controlling for DRG-specific mortality, the logistic regression model provides a conservative estimate of the association between the Nursing Severity Index and in-hospital death. Although it would have been preferable to use dummy variables to represent individual diagnoses, the relative number of deaths made it impractical to include the necessary number of dummy variables. Based on the logistic regression coefficient, the odds ratio for the difference in mortality risk between the six hierarchical Nursing Severity Index strata was estimated. Similar analyses were used to examine independent associations between the Nursing Severity Index and nursing home discharge and DRG outlier status; these analyses controlled for DRG-specific rates of nursing home discharge or DRG outlier status. In-hospital deaths were excluded from all analyses of nursing home discharges. Multiple linear regression [34] was used to examine independent associations between the Nursing Severity Index and length of stay and charges, controlling for the above potential confounders. The effect of diagnosis was controlled for using DRG weight. Because of their skewed distributions, both length of stay and charges were log transformed. Based on the anti log of the regression coefficients, percent differences in charges or length of stay associated with the six Nursing Severity Index strata were estimated.
RESULTS
The mean age of the 5347 study patients was 53.6 years (Table 1); 58% were female and 82% were white. Based on DRGs, 72% of patients were classified as surgical and 28% were classified as medical. Among surgical patients, the most common procedures were back and neck procedures (DRGs 214215; N = 1018) major joint replacement procedures (DRGs 209 and 471; N = 966) and hip and femur procedures, excluding major joint replacement (DRGs 21O-2 11; N = 297); among medical patients, the most common diagnoses were medical back problems (DRG 243; N = 672),
182
Gary E. Rosenthal et al. Table 1. Characteristics
of 5347 study patients Patient group
Mean Mean Mean Mean Mean
age (years) length of stay (days)t hospital charges ($)t Nursing Severity Index scores? DRG weight?
All (N =5347)
Medical (N = 1514)
+ + + + 1.38 &
53.6 k 18.6 5.5 + 5.8 6192&6951 6.6 & 3.0 0.76 + 0.19
53.6 8.6 12,698 6.9
18.9 7.8 11,503 3.0
0.75
Surgical (N =3833) 53.6 9.9 15,268 7.0 1.64
f * + +
19.0 9.1 11,924 2.9 & 0.74
Number (%) Sex:? Male Female Race:? White Nonwhite Marital status:* Married Single In-hospital deatht DRG length of stay outlier Nursing home discharget
2219 (41.5) 3128 (58.5)
566 (37.4) 948 (62.6)
1653 (43.1) 2180 (56.9)
4377 (8 1.9) 970 (18.1)
1140 (75.3) 374 (24.7)
3237 (84.4) 596 (15.6)
2995 (56.0) 2352 (44.0) 62 (1.2)
100 (1.9) 213 (4.0)
*The difference between medical and surgical patients is significant, tThe difference between medical and surgical patients is significant, SAnalysis excludes in-hospital deaths, as described in the Methods.
pathological fractures and musculoskeletal malignancies (DRG 239; N = 193) and connective tissue diseases (DRGs 240-241; N = 148). Medical patients, compared to surgical patients, were more likely to be female, nonwhite and single (Table 1). In addition, medical patients had a higher rate of in-hospital death and lower mean length of stay and charges. Similar proportions of medical and surgical patients were discharged to nursing homes and were DRG length of stay outliers. Relationship between the Nursing Severity Index and in -hospital death Admission Nursing Severity Index scores were strongly associated with in-hospital death rates, which were 0, 0.4, 0.8, 2.6, 6.7 and 23.5% (p < 0.001) respectively, in patients with scores of O-3, 4-6, 7-9, 10-12, 13-15 and 216. Furthermore, the 62 patients who died had significantly higher Nursing Severity Index scores than the 5285 patients who survived (11.1 k 3.7 vs 6.9 f 2.9, respectively; p < 0.001). Admission Nursing Severity Index scores were also strongly associated (p < 0.001) with in-hospital death among medical [Table 2(A)] and surgical [Table 2(B)] patients, when examined separately. In a further stratified analysis, Nursing Severity Index scores were associated with inhospital death (p < 0.001) among patients in ‘lower’ risk DRGs (observed mortality rate <2%) and in ‘higher’ risk DRGs (observed
809 705 34 24 48
(53.4) (46.6) (2.2) (1.6) (3.2)
2186 (57.0)
1647 (43.0) 28 (0.7) 76 (2.0) 165 (4.3)
p < 0.05. p < 0.001.
mortality 22%; Table 3). Among all patients, the Nursing Severity Index had an ROC curve area of 0.809 &- 0.033 for predicting in-hospital death. In sub-group analyses, the Nursing Severity Index had similar (p > 0.1) performance among medical and surgical patients and among patients in higher and lower risk DRGs. In a multiple logistic regression model, adjusting for DRG-specific mortality, age, race, sex, marital status and type of health insurance, the Nursing Severity Index remained a strong independent predictor of in-hospital death; the model estimated that each of the six hierarchical strata was associated with a 2.5-fold difference in the risk of death (regression coefficient, 0.93 f 0.13; p < 0.0001). For example, the model predicted that, independent of other factors, the risk of death was 16.3 times greater in a patient with a score of 14 than in a patient with a score of five. Relationship between the Nursing Severity Index and nursing home discharge Admission Nursing Severity Index scores were associated (p < 0.001) with the rate of discharge to nursing homes; among all patients, rates of discharge to nursing homes were 1.3, 1.3, 4.8, 9.9, 9.6 and 19.2%, respectively, in the six hierarchical strata of Index scores. The above associations were similar in both medical [Table 2(A)] and surgical [Table 2(B)] patients, as well as in analyses stratified according to
Severity Adjustment
in Low Risk Patients
183
Table 2 (A) Rates q/in-hospital death, discharge to nursing homes, and DRG outlier SIUIUS according to admksion Nursing Severity Index scores among I514 medical patients Strata of Nursing Severity Index scores* O-3 4-6 7-9 IO-12 13-15 3 I6
(N (N (N (N (N (N
= = = = = =
In-hospital
Nursing home dischargestf
death?
DRG outlierst
~ Number (%)
216) 588) 483) 166) 44) 17)
I (0.5) 6 (1.0) 15 (3.2) 19 (12.3) 4 (10.8) 3 (27.3)
0 (0.0) 2 (0.3) 7 (1.4) I2 (7.2) l(15.9) 6 (35.3)
2 4 7 3 3 5
(0.9) (0.7) (1.4) (1.8) (6.8) (29.4)
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. i-The relationship between the Nursing Severity Index strata and the outcome is significant, p < 0.001. IAnalysis excludes in-hospital deaths, as described in the Methods. (B) Rates of in-hospital death, discharge to nursing homes, and DRG outlier stutus according to admission Nursing Severity Index scores among 3833 surgical patients Strata of Nursing Severity Index scores* o-3 44 7-9 IO-12 13-15 2 I6
(N (N (N (N (N (N
= = = = = =
In-hospital
Nursing home dischargestf
death?
DRG outlierst
~ Number (X)
395) 1416) 1243) 628) 134) 17)
0 5 7 9 5 2
(0.0) (0.4) (0.6) (1.4) (3.7) (I 1.8)
7 (1.8) I9 (1.4) 67 (5.4) 58 (9.4) 12 (9.3) 2 (13.3)
4 (1.0) I6 (1.1) 32 (2.6) 15 (2.4) 5 (3.7) 4 (23.5)
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. tThe relationship between the Nursing Severity Index strata and the outcome is significant, p < 0.001. IAnalysis excludes in-hospital deaths, as described in the Methods.
patient age (the strongest predictor of nursing home discharge). Among patients < 75 years of age, rates of discharge to nursing homes were 0.5, 0.6, 1.7, 3.2, 4.0 and lO.O%, respectively, in the six strata of Index scores 0, < 0.001); among patients 75 years and older, rates of discharge to nursing homes were 11.1, 6.8, 20.7, 33.3, 26.8 and 50.0%, respectively, in the six strata (p < 0.001). Among all patients, the Nursing
Severity Index had an ROC curve area of 0.733 + 0.020 for predicting nursing home discharge. In sub-group analyses, the Nursing Severity Index had similar (p > 0.1) performance among patient < 75 years and 75 years and older. In a logistic regression model, adjusting for diagnosis, age, race, sex, marital status and health insurance, the Nursing Severity Index was an independent predictor of nursing home
Table 3. In-hospital death rates according to admission Nursing Severity Index scores in ‘lower’ and ‘higher’ risk DRG among all study patients, Based on observed death rates, DRGs were classified as lower (< 2%) or higher (22%) risk DRG risk classification Strata of Nursing Severity Index score* o-3 46 7-9 IO-12 13-15 216
Lower riskt (N = 4685)
Higher riskt (N = 662)
~ Total patients/Number 572/O 1848/5 l480/7 645/4 122/3 IS/l
of deaths (%)
(0.0) (0.3) (0.5) (0.6) (2.5) (5.6)
Total patients/Number
of deaths (%)
39/o (0.0) 15612 (1.3) 246/7 (2.8) 149/17 (I 1.4) 56/9 (16.1) 16/7 (43.8)
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. tThe relationship between the Nursing Severity Index strata and in-hospital death rates is significant, p < 0.001.
184
Gary E. Rosenthal et al. Table 4. Mean length of stay and total hospital charges according to admission Nursing Severity Index scores among medical and surgical patients Medical patients Strata of Nursing Severity Index scores*
Mean length of stay (days)?
o-3 4-6 7-9 IO-12 13-15 >I6
3.6 4.6 6.0 7.4 9.5 16.0
Surgical patients
Mean charges (%)t 4245 5218 646 I 800 I 10,797 27,373
Mean length of stay (da&t 7.8 9.1 10.2 II.1 13.1 21.9
Mean charges (S)t 11,565 14,103 15.83 I 17,650 18,597 35.052
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. tThe relationship between the Nursing Severity Index strata and length of stay or total hospital charges is significant, p < 0.001. -
discharge; the model estimated that, among all patients, each of the six hierarchical strata was associated with a 1.6-fold difference in the risk of nursing home discharge (regression coefficient, 0.49 + 0.08; p < 0.0001). Relationship between the Nursing Severity Index and length of stay, charges and DRG outlier status Admission Nursing Severity Index scores were directly related to length of stay and total hospital charges. Among all patients, mean length of stay were 6.3, 7.8, 9.1, 10.3, 12.2 and 19.0 days in the six hierarchical strata, respectively 0, < 0.001); mean hospital charges were $8977, $11,496, $13,210, $15,633, $17,422 and $31,212 in the six strata, respectively @ < 0.001). These direct relationships were similar in medical and surgical patients (Table 4) and in groups stratified by DRG weight (see Methods) quartiles (Table 5). Among all patients, the Nursing Severity Index explained 9.8% of the variance in hospital charges and 8.8% of the variance in length of stay.
In multiple linear regression models, adjusting for DRG weight, age, sex, race, marital status and health insurance, the Nursing Severity Index remained independently related (p < 0.001) to length of stay and charges in both medical and surgical patients. Among medical patients, each Nursing Severity Index stratum was associated with a 25.4% increase in length of stay and a 21.3% increase in total hospital charges (e.g. the multivariate model estimated that, after adjusting for all other variables, charges were 79% greater for a patient with an admission score of 14 than for a patient with a score of five). Among surgical patients, each Nursing Severity Index stratum was associated with a 5.6% increase in length of stay and a 6.1% increase in charges. Finally, admission Nursing Severity Index scores were associated (p -C0.001) with having an excessively long length of stay, according to DRG length of stay outlier criteria; among all patients, rates of DRG outliers were 1.0, 1.O, 2.3, 2.3, 4.5 and 26.5%, respectively, in the six strata of Index scores. Associations were similar in medical [Table 2(A)] and surgical [Table 2(B)]
Table 5. Mean length of stay according to admission Nursing Severity Index scores in quartiles of expected resource use among all study patients. Patients were categorized into quartiles on the basis of DRG weights, as described in the Methods. Similar relationships (p < 0.001) in the DRG weight quartiles were found for hospital charges (results not shown)
Strata of Nursing Severity Index scores* O-3 4-6 7-9 IO-12 13-15 > 16
Quartile 1.1 (DRG weights 0.40&0.68) -
Quartile Ilt (DRG weights 0.69-l .08)
Quartile IlIt (DRG weights I .09-l .97)
Quartile IVP (DRG weights 1.984.15)
Mean length of stay (days) 3.1 3.8 4.7 5.4 9.1 17.5
3.9 5.1 5.7 6.8 8.7 II.7
8.4 9.2 10.3 10.0 10.5 20.1
13.4 13.7 14.4 14.4 15.8 26.7
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. tThe relationship between the Nursing Severity Index strata and length of stay is significant, p < 0.001.
Severity Ad.justment in Low Risk Patients
patients. In a logistic regression model, adjusting for potential confounders, the Nursing Severity Index remained an independent predictor; the multivariate model estimated that, among all patients, each of the six hierarchical strata was associated with a l&fold difference in the risk of being a DRG outlier (regression coefficient, 0.54 f 0.09; p 0.0001). DlSCUSSION
185
on acute pathophysiologic abnormalities and/or medical diagnoses [l&16]. The narrow spectrum of disease described by these methods may be particularly problematic for patients with musculoskeletal disease who are often affected by functional and psychosocial impairments. The 34 nursing diagnoses which comprise the Nursing Severity Index (see Appendix) assess functional status (e.g. continence, mobility and cognitive impairments) and psychosocial parameters (e.g. mood, family relationships and ability to cope with disease), as well as biologic aspects of disease (e.g. presence of infection and volume status). The Nursing Severity Index also differs from other instruments in that it utilizes simple clinical observations that can be recorded while providing care. Thus, the Index avoids the expense of retrospective review of medical records, required by several instruments our empirical analyses [lo]. Furthermore, indicate that the performance of the Nursing Severity Index among patients with musculoskeletal disease was similar to the performance of other severity measures in populations of general medical and surgical patients [IB, 19,291, with respect to mortality, length of stay and hospital charges. We suggest that future studies compare the predictive validity of the Nursing Severity Index and other generic severity of illness measures in patients with musculoskeletal disease.
Growing concern by the federal government and other payers about the cost of health care has led to a national effort to measure the effectiveness of hospital care through the use of patient outcomes data [l, 21. The yearly release of hospital mortality data by HCFA [3,4] and recent hospital comparative reports from Pennsylvania [5], Iowa [6] and New York [7] exemplify this effort. However, the interpretation of outcomes data and assessment of the hospital effectiveness requires valid and practical instruments for adjusting for differences in severity of illness between patients. Although several generic measures are currently in use, the validity of these instruments in specific patient groups has been poorly studied. This is particularly true for hospital patients in whom the overall risk of mortality is low. We, therefore, examined the predictive validity of a new instrument, the Nursing Severity Index, in 5347 medical and surgical patients with musculoskeletal disease admitted to an aca- Methodologic considerations In examining our findings, potential methoddemic medical center. Our analyses revealed that admission Nursing Severity Index assess- ologic limitations related to the assembly of the study sample, the classification of patients using ments accurately stratified patients according to rates of in-hospital mortality, discharge to the Nursing Severity Index, designation of the nursing homes, and of having excessively long ‘hospital admission’ period, and the generalizability of study findings should be considered. lengths of stay (i.e. DRG length of stay outFirst, selection bias in the assembly of the study liers); in multivariate analyses, adjusting for groups is possible because the groups were important covariates, the Nursing Severity Index remained a strong independent predictor non-random samples of consecutive admissions. of each of the three outcomes. In addition, we However, the study group included a large found that the Nursing Severity Index was an proportions of eligible patients (72%) and independent predictor of hospital charges and patients studied did not differ markedly from length of stay, even after adjusting for DRG those not studied. Second, although the nursing weight, a measure used to determine hospital diagnoses which comprise the Nursing Severity Index are widely used and are defined by explicit reimbursement, and other covariates. clinical criteria [30], misclassification of patients Comparison of the Nursing Severity Index to according to specific nursing diagnoses probinstruments ably occurs. However, inter-rater reliabilities of The Nursing Severity Index measures mulnursing diagnoses [29] are similar to physical tiple domains of health and, thus, is based on a examination findings, diagnostic test interpretbroader concept of illness than current severity ations, and medical diagnoses [35538]-data of illness instruments which are primarily based used in other severity measures. Third. desig-
186
Gary E. Rosenthal et al.
nation of the length of the ‘admission period’ is important to assessing the predictive validity of severity instruments because patients’ prognoses may change during hospitalization. In examining the Nursing Severity Index, we analyzed data collected on the first or second hospital day, a period that is similar to or shorter than that used in other methods [lo-121. Finally, the generalizability of our findings to other diagnostic categories in which the risk of mortality is low and to patients in other types of hospitals (e.g. community, non-teaching) should be established. Implications In recent years, efforts to measure the quality and efficiency of hospital care have focused on the analysis of patient outcomes [l, 21. For example, an analysis detailing ‘severityadjusted’ hospital mortality rates, lengths of stay, and charges for 66 specific DRGs among 26 hospitals was recently released by the Iowa Health Data Commission [6]; included are analyses of several musculoskeletal DRGs (209, 210, 215, 239 and 243). Proper interpretation of outcomes data, however, requires that methods used to adjust for differences in severity of illness be rigorously evaluated. Although generic severity of illness measures and arthritisspecific health status instruments have been developed, the current study represents the first critical evaluation of one such instrument in a large group of patients hospitalized with musculoskeletal diagnoses. In addition to establishing the predictive validity of the Nursing Severity Index, the current study demonstrates that the data necessary for severity adjustment can be accurately and concisely obtained during routine clinical practice. Furthermore, because Nursing Severity Index data can be gathered and used concurrently during hospitalization, the Index is complementary to the development of case managed care strategies which hospitals are implementing in an increasing fashion to improve patient outcomes and to deliver diagnostic, therapeutic and rehabilitative interventions more effectively [39]. As the demand by payers for information about the effectiveness of hospital care increases and as the proportion of hospital care that is reimbursed prospectively increases, the need for valid instruments that can account for differences in patient severity becomes more critical. To avoid the creation of disincentives for hospitals to care for sicker patients [40], it is impera-
tive that severity measures be implemented that have been validated in specific clinical subgroups. Finally, our finding that the Nursing Severity Index was an independent predictor of nursing home discharge in patients with musculoskeletal disease has implications for the future assessment of hospital outcomes in clinical subgroups, in whom the risk of mortality is low. To our knowledge, the predictive validity of severity measures with respect to nursing home placement has not previously been examined. Because nursing home discharge occurs more frequently than in-hospital death in patients with muscuioskeletal disease, it may represent a more sensitive outcome indicator. Indeed, data from patients with hip fracture suggests that hospital quality may be directly related to rates of nursing home discharge [41]. Future studies should address the relationship of severityadjusted rates of nursing home discharge to quality of care and to the assessment of hospital outcomes. Conclusion We conclude that in a large group of ‘low risk’ medical and surgical patients with musculoskeletal conditions, the Nursing Severity Index was easily measured during routine patient care and that it accurately predicted the risk of in-hospital death, the rate of discharge to nursing homes and hospital resource use. These results support the validity of the Nursing Severity Index as a method of assessing and comparing hospital outcomes in patients with musculoskeletal disease. Furthermore, these results suggest that the use of the Nursing Severity Index may avoid the creation of disincentives for hospitals to care for more severely ill patients. Therefore, we feel that the Nursing Severity Index merits further testing in patients with other diagnoses and in other hospitals. Acknowledgemenfs-We thank Paul Jones, Ph.D. and Roland W. Moskowitz, M.D. for their helpful reviews of this manuscript. Supported by NIH Grant AR-20618. U.S.P.H.S. Northeast Ohio Multipurpose Arthritis Center.
REFERENCES 1. Relman AS. Assessment and accountability: the third revolution in health care. New Engl J Med 1988; 319: 1220-1222. 2. Roper WL. Winkenwerder W. Hackbarth GM, Krakauer H. Effectiveness in health care: an initiative to evaluate and improve medical practice. New Engl J Med 1988; 319: 1197-1202.
Severity Adjustment
5. 6. 7. 8. 9.
10.
11.
12.
13. 14. 15. 16.
17.
18.
19. 20.
21. 22.
23.
New York Times. US releasing lists of hospitals with abnormal mortality rates. Wednesday, March 12, 1986. Sullivan LW, Toby W. Medicare Hospital Information Report 1988, 1989, 1990: Volumes l-55. Washington DC: Health Care Financing Administration, U.S. Department of Health and Human Services; 1992. Hospital Effectiveness Report. Harrisburg, PA: Pennsylvania Health Care Containment Council; 1991. Iowa Hospital Resource and Outcome Report, July 1990 through Juw 1991, DRG Specific Version. Des Moines, IA: Iowa Health Data Commission; 1992. Hannan EL, Kilburn H, O’Donnell JF, Lukacik G, Shields EP. Adult open heart surgery in New York state. JAMA 1990; 264: 2768-2774. Shaller DV, Woods P. Reforming the market for value in health care: the Cleveland experience. J Occup Med 1991; 33: 358-362. McMahon LF, Billi JE. Measurement of severity of illness and the Medicare prospective payment system: state of the art and future directions. J Gen Intern Med 1988; 3: 482490. Iezzoni LI. Measuring the severity of illness and case mix. In: Providing Quality Care: the Challenge to Clinicians Goldfield N, Nash DB (Eds). Philadelphia, PA. American College of Physicians; 1989. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system for severely ill patients. Crit Care Med 1985; 13: 818-829. Brewster AC, Karlin BG, Hyde LA, Jacobs CM, Bradbury RC, Chae YM. MEDISGROUPS: a clinically based approach to classifying hospital patients at admission. Inquiry 1985; 22: 3777387. Horn SD, Horn RA. The computerized severity index: a new tool for case-mix management. J Med Syst 1986; 10: 73378. Gonneila JS. Hornbrook MC. Staging a disease: a case-mix measurement. JAMA 1984; 251: 637Wj44. Young WW, Swinkola RB, Zorn DM. The measurement of hospital case mix. Med Care 1985; 20: 501-512. Daley J, Jencks S, Draper D, Lenhart G, Thomas N, Walker J. Predicting hospital-associated mortality for Medicare patients: a method for patients-with stroke, pneumonia, acute myocardial infarction, and congestive heart failure. JAMA 1988; 260: 3617-3624. Knaus WA, Wagner DP, Draper EA, et al. The APACHE III prognostic system: risk prediction of hospital mortality for critically ill hospitalized patients. Chest 1991; 100: 1619-1636. Thomas JW, Ashcraft MLF, Zimmerman J. An evaluation of alternative severity of illness measures for use by university hospitals. Ann Arbor, MI: Department of Health Services Management and Policy, School of Public Health, University of Michigan; 1986. Iezzoni LI, Moskowitz MA. A clinical assessment of MedisGroups. JAMA 1988; 260: 3 159-3 163. Alemi F, Rice J, Hankins R. Predicting in-hospital survival of myocardial infarction: a comparative study of various severity measures. Med Care 1990; 28: 762-775. Blumberg MS. Biased estimates of acute myocardial infarction mortality using MedisGroups admission severity groups. JAMA 1991; 265: 2965-2970. Lichtenstein MJ, Federspiel CF, Shaffner W. Factors associated with early demise in nursing home residents: a case control study. J Am Geriatr Sot 1985; 33: 315-319. Reuben DB, Mor V, Hiris J. Clinical symptoms and length of survival in patients with terminal cancer. Arch Intern Med 1988; 148: 15861591.
in Low Risk Patients 24.
25. 26.
27.
28. 29.
30. 31. 32. 33. 34. 35. 36. 37.
38.
39. 40.
41.
187
Narain P, Rubenstein LZ, Wieland CD, Rosbrook B, ef al. Predictors of immediate and 6-month outcomes in hospitalized elderly patients: the importance of functional status, J Am Geriatr Sot 1988; 36: 775-783. Greenfield S, Aronow HU, Elashoff RM, Watanabe D. Flaws in mortality data: the hazards of ignoring comorbid disease. JAMA 1988; 260: 2253-2255. Reuben DB, Rubenstein LV, Hirsch SH, Hays RD. Value of functional status as a predictor of mortality: results of a prospective study. Am J Med 1992; 93: 6633669. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis 1987; 40: 373-383. Kaplan MH, Feinstein AR. The importance of classifying initial comorbidity in evaluating the outcome of diabetes mellitus. J Chron Dis 1974; 27: 387404. Rosenthal GE, Halloran EJ, Kiley M, Pinkley C, Landefeld CS. Development and validation of the Nursing Severity Index: a new method for measuring severity of illness using nursing diagnoses. Med Care 1992; 30: 1127-I 141. Carpenito LJ. Nursing Diagnosis: Application to Clinical Practice, 3rd edn. Philadelphia, PA: JB Lippincott Co.; 1989. Gordon M. Nursing Diagnosis: Process and Application. New York: McGraw-Hill; 1982. Hanley JA, McNeil BJ. The meaning and use of the area nnder a receiver operating characteristic (ROC) curve. Radiology 1982; 148: 8399843. The LOGISTIC procedure. SAS/STAT Users Guide, Version 6, 4th Edn, Vol. 2. Cary, NC: SAS Institute Inc.; 1990. The GLM procedure. SAS/STAT Users Guide, Version 6, 4th Edn. Vol. 2. Carv, NC: SAS Institute Inc.; 1990. Koran LM. The reliability of clinical methods, data, and judgments (part one). New Engl J Med 1975; 293: 642. Koran LM. The reliability of clinical methods, data, and judgments (part two). New Engl J Med 1975; 293: 695. Iezzoni LI, Burnside S, Sickles L, Moskowitz MA, Sawitz E, Levine PA. Coding of acute myocardial infarction: clinical and policy implications. Ann Intern Med 1988; 109: 745. Anderson RE, Hill RB, Key CR. The sensitivity and specificity of clinical diagnostics during five decades: toward an understanding of necessary fallibility. JAMA 1989; 261: 1610. Zander K. Nursing case management: strategic management of cost and quality outcomes. J Nursing Admin 1988; 18: 23330. Horn SD, Bulkley G, Sharkey PD, Chamber AF, Horn RA, Schramm CJ. Interhospital differences in severity of illness: problems for prospective payment based on diagnosis-related groups (DRGs). New Engl J Med 1985; 313: 20-24. Fitzgerald JF, Moore PS, Dittus RS. The care of elderly patients with hip fracture: changes since implementation of the prospective payment system. New Engl J Med 1988; 319: 1392-1397.
APPENDIX The 34 individual nursing diagnoses which comprise the Nursing Severity Index are listed below. The diagnoses are grouped into five subscales, based on Gordon’s model of “dysfunctional health patterns” [31] which reflect specific aspects of nursing care. Each diagnosis is opera-
188
Gary E. Rosenthal et al.
tionalized by clinical nurses, using explicit and standardized criteria [30]. (I) Overall health and perceptions Potential for injury Infection/contagion Prolonged disease/disability Instability Impaired life support systems (II) Nutrition and metabolism Excess fluid volume Fluid volume deficit Bleeding Less nutrition than required Potential skin impairment Alterations in oral mucous membranes Altered body temperature (III) Urinary and fecal function Urinary incontinence Other altered urinary elimination Constipation
pattern
Diarrhea Bowel incontinence (IV) Activity and exercise Activity intolerance Ineffective airway clearance Altered breathing pattern Impaired gas exchange Decreased cardiac output Altered health maintenance Impaired mobility Self-care deficit ( V) Psycho -social concerns Disturbed self-concept Depression Grieving Altered family process Social isolation Impaired verbal communication Ineffective individual coping Potential for growth in family coping Spiritual distress
Pergamon
0895-435q94)00142-1
Vol. 48. No. 2. DD. 179-188. 1995 Copyright (’ 1995 El&r Science Ltd Printed in Great Brilain. All rights reserved
Epidemiol
089%4356/95
PREDICTIVE VALIDITY INDEX IN PATIENTS GARY E. ROSENTHAL’, C. SETH LANDEFELD+’
$9.50 + 0.00
OF THE NURSING SEVERITY WITH MUSCULOSKELETAL DISEASE
EDWARD J. HALLORAN2, MARYLOU KILEY3, and THE NURSES OF UNIVERSITY HOSPITALS OF CLEVELAND
‘Section of Clinical Epidemiology, Division of General Internal Medicine, Department of Medicine, Cleveland Veterans Administration Medical Center, Case Western Reserve University School of Medicine, Cleveland, OH, U.S.A., %chool of Nursing, University of North Carolina, Chapel Hill, NC, U.S.A., ‘Frances Payne Bolton School of Nursing, Case Western Reserve University, Cleveland, OH, U.S.A. and “Section of Clinical Epidemiology, Divisions of General Internal Medicine and Geriatrics, Department of Medicine, Cleveland Veterans Administration Medical Center and University Hospitals of Cleveland, Case Western Reserve University School of Medicine, Cleveland, OH, U.S.A. (Received in revised form 13 July 1994)
Abstract-Prior studies have not examined the validity of severity of illness instruments in patients at low risk for mortality. We, therefore, examined the predictive validity of a newly developed instrument, the Nursing Severity Index in 5347 adult medical and surgical patients with musculoskeletal diagnoses admitted to an academic medical center in 1985-88. The Index is based on aggregating 34 clinical observations which were recorded by primary nurses during patient care; observations reflect biologic, functional, cognitive and psychosocial abnormalities. Other data, including patient demographic data and outcomes were obtained from hospital data bases. We found that, among all study patients, admission Nursing Severity Index scores were highly related (p < 0.001) to in-hospital death rates-which were 0, 0.4, 0.8, 2.6, 6.7 and 23.5% in six hierarchical strata defined by the Index-and to nursing home discharge rates. In multivariate analyses, adjusting for diagnosis and other important covariates, each strata was associated with a 2.5-fold increased risk of mortality and a 1.6-fold increased risk of nursing home discharge. In addition, the Nursing Severity Index was an independent predictor (p < 0.001) of hospital charges and length of stay. We conclude that the Nursing Severity Index assesses multiple dimensions of illness, can be easily recorded during routine patient care, and accurately predicts hospital outcomes in an important ‘low risk’ group of patients. The validity of the Nursing Severity Index in other clinical subgroups should be further studied. Health services research Quality of health care ment Mortality Length of stay
INTRODUCTION
Efforts to measure and improve the quality and efficiency of hospital care have been increasingly based on the examination of patient outcomes [I, 21. These efforts are exemplified by several initiatives, including the yearly release of hospital mortality data for Medicare patients
Outcome and process assess-
by the Health Care Financing Administration (HCFA) [3,4], the release of comparative hospital reports by Pennsylvania [5], Iowa [6] and New York [7] and the emergence of regional provider-purchaser coalitions in Cleveland [8] and other cities. Because hospitals differ in the types of patients that they care for, the validity of out179
Gary E. Rosenthalet al.
180
comes-based efforts depends on the availability of accurate and easily applied methods of adjusting for severity of illness and other underlying patient characteristics that impact hospital outcomes. Although several instruments for measuring severity of illness (i.e. a patient’s underlying risk for adverse events or propensity for resource utilization) have been proposed [9917], few have been subjected to rigorous independent scrutiny [ 1G-211. In addition, current instruments principally assess pathophysiologic aspects of disease and have generally excluded other important predictors of outcomes, such as comorbid illness and physical, social and emotional functioning [22228]. Furthermore, the validity of these instruments has been established primarily in patients whose risk of death is relatively high (e.g. patients in intensive care units or hospitalized for pneumonia or stroke) [7, 11, 16, 17,20,21]; thus, little is known about the applicability of current instruments to patients in whom the risk of mortality is low (e.g. orthopedic patients) or to patients whose functional limitations and/or psycho-social needs may be out of proportion to acute physiologic aberrations. To address these limitations, we recently reported the development and validation of a new severity instrument, the Nursing Severity Index, in a heterogeneous sample of hospital patients [29]. The Nursing Severity Index is unique among severity measures in that it assesses functional, cognitive, psychosocial and biologic abnormalities and is based on the clinical observations of nurses. We conducted the current study to examine the predictive validity of the Nursing Severity Index in a ‘low risk’ cohort of hospital patients who are likely to have significant functional impairmentpatients with musculoskeletal disease. We hypothesized that the Nursing Severity Index would stratify this cohort with respect to several important hospital outcomes, including inhospital mortality, length of stay, hospital charges and discharge to nursing homes. METHODS
Patients The study was conducted at University Hospitals of Cleveland, an 874-bed major teaching hospital of the Schools of Medicine and Nursing of Case Western Reserve University. The study included patients aged 18 years and
older who were admitted between 3 November 1985 and 3 1 December 1988 and who were classified into diagnosis related groups (DRGs) for musculoskeletal disease (DRGs 209-2.56, 471). Of the 7345 eligible patients admitted during this period, admission Nursing Severity Index assessments were available for 5347 patients (73%), who form the study population. The 5347 study patients and the 1998 other patients were similar in mean age (53.6 vs 53.8 years, respectively) and gender (58% vs 58% female, respectively) and had nearly identical mean lengths of stay (8.6 vs 8.7 days, respectively); the two groups differed somewhat in race (82% vs 77% white, respectively; p < O.OOl), type of health insurance (54% vs 47% with non-governmental insurance plans, respectively; p < O.OOl), and the proportion undergoing a surgical procedure (72% vs 66%, respectively; p < 0.001). Based on DRGs, patients were classified as surgical (DRGs 2099234, 471) or medical (DRGs 235-256). Data Patient age, sex, race, marital status, type of health insurance, length of stay, total hospital charges, discharge vital status, discharge destination, DRG and DRG weight were obtained from computerized data bases. The DRG weight is a relative measure of expected resource use associated with each DRG that is used to determine hospital reimbursement for Medicare and other patients; patients were classified into approximate quartiles, based on the DRG weight assigned to their DRG. Type of health insurance was categorized into four groups according to the primary payer:
6) Medicare; (ii) private (Blue Cross/Blue Shield and other private carriers; (iii) managed care (health maintenance organizations and preferred provider organizations); and (3 Medicaid or uninsured. Patients were categorized as DRG outliers if their length of stay exceeded published Medicare outlier thresholds for each DRG during each year of the study. For each patient, Nursing Severity Index assessments were recorded by primary ward nurses on the day of admission or the next hospital day. All Nursing Severity Index assessments were made by clinical nurses in the course of routine patient care and were recorded in one
Severity Adjustment
to two minutes using bar-code data entry. For patients in whom more than one assessment was available, only the initially recorded assessment was analyzed. The Nursing Severity Index is based on the presence or absence of 34 specific nursing diagnoses (see Appendix), which were found to be independently related to in-hospital mortality, [29]. Nursing diagnoses are widely used in patient management [30,31] and can be recorded with similar reliability as physical examination findings, diagnostic test interpretations and medical diagnoses [29]. Explicit clinical criteria were developed for each of the diagnoses used and all nurses underwent formal training to standardize application of the diagnoses. Nursing Severity Index scores equal the number of the 34 diagnoses noted to be present by the clinical nurse. In previous analyses [29], weighting individual nursing diagnoses according to coefficients from regression analyses did not significantly improve the performance of the Index, as measured by receiver operating characteristic (ROC) curve analysis (see below). Among the 5347 study patients, scores ranged from 0 to 24. In our analyses, scores were categorized into six hierarchical strata: O-3,&6, 7-9, 10-12, 13-15 and 216. Analysis Analyses were performed using mainframe SAS. Associations between categorical variables were tested using the x2 test. Differences between groups for continuous variables were tested using the t-test or Wilcoxon signed rank test. Bivariate associations between Nursing Severity Index scores and dichotomous outcomes (death, nursing home discharge and DRG outlier status) were determined using the x2 test, with the modification for linear trend. Performance of the index for dichotomous outcomes was measured using ROC curve area analysis [32], which examines the ability of an index to discriminate between patients with and without an outcome. ROC curve areas range from 0.5, which indicates no discrimination, to 1.O, which indicates perfect discrimination. Bivariate associations between Nursing Severity Index scores and two measures of hospital resource use (length of stay and charges), were determined using simple linear regression; performance was assessed by the total variance (R*) in length of stay and charges explained by the index.
181
in Low Risk Patients
Logistic regression analysis [33] was used to examine the independent association between the Nursing Severity Index and in-hospital death, controlling for several potential confounders, including age, sex, race, marital status, type of health insurance and the observed DRG-specific mortality rate. By controlling for DRG-specific mortality, the logistic regression model provides a conservative estimate of the association between the Nursing Severity Index and in-hospital death. Although it would have been preferable to use dummy variables to represent individual diagnoses, the relative number of deaths made it impractical to include the necessary number of dummy variables. Based on the logistic regression coefficient, the odds ratio for the difference in mortality risk between the six hierarchical Nursing Severity Index strata was estimated. Similar analyses were used to examine independent associations between the Nursing Severity Index and nursing home discharge and DRG outlier status; these analyses controlled for DRG-specific rates of nursing home discharge or DRG outlier status. In-hospital deaths were excluded from all analyses of nursing home discharges. Multiple linear regression [34] was used to examine independent associations between the Nursing Severity Index and length of stay and charges, controlling for the above potential confounders. The effect of diagnosis was controlled for using DRG weight. Because of their skewed distributions, both length of stay and charges were log transformed. Based on the anti log of the regression coefficients, percent differences in charges or length of stay associated with the six Nursing Severity Index strata were estimated.
RESULTS
The mean age of the 5347 study patients was 53.6 years (Table 1); 58% were female and 82% were white. Based on DRGs, 72% of patients were classified as surgical and 28% were classified as medical. Among surgical patients, the most common procedures were back and neck procedures (DRGs 214215; N = 1018) major joint replacement procedures (DRGs 209 and 471; N = 966) and hip and femur procedures, excluding major joint replacement (DRGs 21O-2 11; N = 297); among medical patients, the most common diagnoses were medical back problems (DRG 243; N = 672),
182
Gary E. Rosenthal et al. Table 1. Characteristics
of 5347 study patients Patient group
Mean Mean Mean Mean Mean
age (years) length of stay (days)t hospital charges ($)t Nursing Severity Index scores? DRG weight?
All (N =5347)
Medical (N = 1514)
+ + + + 1.38 &
53.6 k 18.6 5.5 + 5.8 6192&6951 6.6 & 3.0 0.76 + 0.19
53.6 8.6 12,698 6.9
18.9 7.8 11,503 3.0
0.75
Surgical (N =3833) 53.6 9.9 15,268 7.0 1.64
f * + +
19.0 9.1 11,924 2.9 & 0.74
Number (%) Sex:? Male Female Race:? White Nonwhite Marital status:* Married Single In-hospital deatht DRG length of stay outlier Nursing home discharget
2219 (41.5) 3128 (58.5)
566 (37.4) 948 (62.6)
1653 (43.1) 2180 (56.9)
4377 (8 1.9) 970 (18.1)
1140 (75.3) 374 (24.7)
3237 (84.4) 596 (15.6)
2995 (56.0) 2352 (44.0) 62 (1.2)
100 (1.9) 213 (4.0)
*The difference between medical and surgical patients is significant, tThe difference between medical and surgical patients is significant, SAnalysis excludes in-hospital deaths, as described in the Methods.
pathological fractures and musculoskeletal malignancies (DRG 239; N = 193) and connective tissue diseases (DRGs 240-241; N = 148). Medical patients, compared to surgical patients, were more likely to be female, nonwhite and single (Table 1). In addition, medical patients had a higher rate of in-hospital death and lower mean length of stay and charges. Similar proportions of medical and surgical patients were discharged to nursing homes and were DRG length of stay outliers. Relationship between the Nursing Severity Index and in -hospital death Admission Nursing Severity Index scores were strongly associated with in-hospital death rates, which were 0, 0.4, 0.8, 2.6, 6.7 and 23.5% (p < 0.001) respectively, in patients with scores of O-3, 4-6, 7-9, 10-12, 13-15 and 216. Furthermore, the 62 patients who died had significantly higher Nursing Severity Index scores than the 5285 patients who survived (11.1 k 3.7 vs 6.9 f 2.9, respectively; p < 0.001). Admission Nursing Severity Index scores were also strongly associated (p < 0.001) with in-hospital death among medical [Table 2(A)] and surgical [Table 2(B)] patients, when examined separately. In a further stratified analysis, Nursing Severity Index scores were associated with inhospital death (p < 0.001) among patients in ‘lower’ risk DRGs (observed mortality rate <2%) and in ‘higher’ risk DRGs (observed
809 705 34 24 48
(53.4) (46.6) (2.2) (1.6) (3.2)
2186 (57.0)
1647 (43.0) 28 (0.7) 76 (2.0) 165 (4.3)
p < 0.05. p < 0.001.
mortality 22%; Table 3). Among all patients, the Nursing Severity Index had an ROC curve area of 0.809 &- 0.033 for predicting in-hospital death. In sub-group analyses, the Nursing Severity Index had similar (p > 0.1) performance among medical and surgical patients and among patients in higher and lower risk DRGs. In a multiple logistic regression model, adjusting for DRG-specific mortality, age, race, sex, marital status and type of health insurance, the Nursing Severity Index remained a strong independent predictor of in-hospital death; the model estimated that each of the six hierarchical strata was associated with a 2.5-fold difference in the risk of death (regression coefficient, 0.93 f 0.13; p < 0.0001). For example, the model predicted that, independent of other factors, the risk of death was 16.3 times greater in a patient with a score of 14 than in a patient with a score of five. Relationship between the Nursing Severity Index and nursing home discharge Admission Nursing Severity Index scores were associated (p < 0.001) with the rate of discharge to nursing homes; among all patients, rates of discharge to nursing homes were 1.3, 1.3, 4.8, 9.9, 9.6 and 19.2%, respectively, in the six hierarchical strata of Index scores. The above associations were similar in both medical [Table 2(A)] and surgical [Table 2(B)] patients, as well as in analyses stratified according to
Severity Adjustment
in Low Risk Patients
183
Table 2 (A) Rates q/in-hospital death, discharge to nursing homes, and DRG outlier SIUIUS according to admksion Nursing Severity Index scores among I514 medical patients Strata of Nursing Severity Index scores* O-3 4-6 7-9 IO-12 13-15 3 I6
(N (N (N (N (N (N
= = = = = =
In-hospital
Nursing home dischargestf
death?
DRG outlierst
~ Number (%)
216) 588) 483) 166) 44) 17)
I (0.5) 6 (1.0) 15 (3.2) 19 (12.3) 4 (10.8) 3 (27.3)
0 (0.0) 2 (0.3) 7 (1.4) I2 (7.2) l(15.9) 6 (35.3)
2 4 7 3 3 5
(0.9) (0.7) (1.4) (1.8) (6.8) (29.4)
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. i-The relationship between the Nursing Severity Index strata and the outcome is significant, p < 0.001. IAnalysis excludes in-hospital deaths, as described in the Methods. (B) Rates of in-hospital death, discharge to nursing homes, and DRG outlier stutus according to admission Nursing Severity Index scores among 3833 surgical patients Strata of Nursing Severity Index scores* o-3 44 7-9 IO-12 13-15 2 I6
(N (N (N (N (N (N
= = = = = =
In-hospital
Nursing home dischargestf
death?
DRG outlierst
~ Number (X)
395) 1416) 1243) 628) 134) 17)
0 5 7 9 5 2
(0.0) (0.4) (0.6) (1.4) (3.7) (I 1.8)
7 (1.8) I9 (1.4) 67 (5.4) 58 (9.4) 12 (9.3) 2 (13.3)
4 (1.0) I6 (1.1) 32 (2.6) 15 (2.4) 5 (3.7) 4 (23.5)
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. tThe relationship between the Nursing Severity Index strata and the outcome is significant, p < 0.001. IAnalysis excludes in-hospital deaths, as described in the Methods.
patient age (the strongest predictor of nursing home discharge). Among patients < 75 years of age, rates of discharge to nursing homes were 0.5, 0.6, 1.7, 3.2, 4.0 and lO.O%, respectively, in the six strata of Index scores 0, < 0.001); among patients 75 years and older, rates of discharge to nursing homes were 11.1, 6.8, 20.7, 33.3, 26.8 and 50.0%, respectively, in the six strata (p < 0.001). Among all patients, the Nursing
Severity Index had an ROC curve area of 0.733 + 0.020 for predicting nursing home discharge. In sub-group analyses, the Nursing Severity Index had similar (p > 0.1) performance among patient < 75 years and 75 years and older. In a logistic regression model, adjusting for diagnosis, age, race, sex, marital status and health insurance, the Nursing Severity Index was an independent predictor of nursing home
Table 3. In-hospital death rates according to admission Nursing Severity Index scores in ‘lower’ and ‘higher’ risk DRG among all study patients, Based on observed death rates, DRGs were classified as lower (< 2%) or higher (22%) risk DRG risk classification Strata of Nursing Severity Index score* o-3 46 7-9 IO-12 13-15 216
Lower riskt (N = 4685)
Higher riskt (N = 662)
~ Total patients/Number 572/O 1848/5 l480/7 645/4 122/3 IS/l
of deaths (%)
(0.0) (0.3) (0.5) (0.6) (2.5) (5.6)
Total patients/Number
of deaths (%)
39/o (0.0) 15612 (1.3) 246/7 (2.8) 149/17 (I 1.4) 56/9 (16.1) 16/7 (43.8)
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. tThe relationship between the Nursing Severity Index strata and in-hospital death rates is significant, p < 0.001.
184
Gary E. Rosenthal et al. Table 4. Mean length of stay and total hospital charges according to admission Nursing Severity Index scores among medical and surgical patients Medical patients Strata of Nursing Severity Index scores*
Mean length of stay (days)?
o-3 4-6 7-9 IO-12 13-15 >I6
3.6 4.6 6.0 7.4 9.5 16.0
Surgical patients
Mean charges (%)t 4245 5218 646 I 800 I 10,797 27,373
Mean length of stay (da&t 7.8 9.1 10.2 II.1 13.1 21.9
Mean charges (S)t 11,565 14,103 15.83 I 17,650 18,597 35.052
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. tThe relationship between the Nursing Severity Index strata and length of stay or total hospital charges is significant, p < 0.001. -
discharge; the model estimated that, among all patients, each of the six hierarchical strata was associated with a 1.6-fold difference in the risk of nursing home discharge (regression coefficient, 0.49 + 0.08; p < 0.0001). Relationship between the Nursing Severity Index and length of stay, charges and DRG outlier status Admission Nursing Severity Index scores were directly related to length of stay and total hospital charges. Among all patients, mean length of stay were 6.3, 7.8, 9.1, 10.3, 12.2 and 19.0 days in the six hierarchical strata, respectively 0, < 0.001); mean hospital charges were $8977, $11,496, $13,210, $15,633, $17,422 and $31,212 in the six strata, respectively @ < 0.001). These direct relationships were similar in medical and surgical patients (Table 4) and in groups stratified by DRG weight (see Methods) quartiles (Table 5). Among all patients, the Nursing Severity Index explained 9.8% of the variance in hospital charges and 8.8% of the variance in length of stay.
In multiple linear regression models, adjusting for DRG weight, age, sex, race, marital status and health insurance, the Nursing Severity Index remained independently related (p < 0.001) to length of stay and charges in both medical and surgical patients. Among medical patients, each Nursing Severity Index stratum was associated with a 25.4% increase in length of stay and a 21.3% increase in total hospital charges (e.g. the multivariate model estimated that, after adjusting for all other variables, charges were 79% greater for a patient with an admission score of 14 than for a patient with a score of five). Among surgical patients, each Nursing Severity Index stratum was associated with a 5.6% increase in length of stay and a 6.1% increase in charges. Finally, admission Nursing Severity Index scores were associated (p -C0.001) with having an excessively long length of stay, according to DRG length of stay outlier criteria; among all patients, rates of DRG outliers were 1.0, 1.O, 2.3, 2.3, 4.5 and 26.5%, respectively, in the six strata of Index scores. Associations were similar in medical [Table 2(A)] and surgical [Table 2(B)]
Table 5. Mean length of stay according to admission Nursing Severity Index scores in quartiles of expected resource use among all study patients. Patients were categorized into quartiles on the basis of DRG weights, as described in the Methods. Similar relationships (p < 0.001) in the DRG weight quartiles were found for hospital charges (results not shown)
Strata of Nursing Severity Index scores* O-3 4-6 7-9 IO-12 13-15 > 16
Quartile 1.1 (DRG weights 0.40&0.68) -
Quartile Ilt (DRG weights 0.69-l .08)
Quartile IlIt (DRG weights I .09-l .97)
Quartile IVP (DRG weights 1.984.15)
Mean length of stay (days) 3.1 3.8 4.7 5.4 9.1 17.5
3.9 5.1 5.7 6.8 8.7 II.7
8.4 9.2 10.3 10.0 10.5 20.1
13.4 13.7 14.4 14.4 15.8 26.7
*Nursing Severity Index scores were collapsed into six strata, as described in the Methods. tThe relationship between the Nursing Severity Index strata and length of stay is significant, p < 0.001.
Severity Ad.justment in Low Risk Patients
patients. In a logistic regression model, adjusting for potential confounders, the Nursing Severity Index remained an independent predictor; the multivariate model estimated that, among all patients, each of the six hierarchical strata was associated with a l&fold difference in the risk of being a DRG outlier (regression coefficient, 0.54 f 0.09; p 0.0001). DlSCUSSION
185
on acute pathophysiologic abnormalities and/or medical diagnoses [l&16]. The narrow spectrum of disease described by these methods may be particularly problematic for patients with musculoskeletal disease who are often affected by functional and psychosocial impairments. The 34 nursing diagnoses which comprise the Nursing Severity Index (see Appendix) assess functional status (e.g. continence, mobility and cognitive impairments) and psychosocial parameters (e.g. mood, family relationships and ability to cope with disease), as well as biologic aspects of disease (e.g. presence of infection and volume status). The Nursing Severity Index also differs from other instruments in that it utilizes simple clinical observations that can be recorded while providing care. Thus, the Index avoids the expense of retrospective review of medical records, required by several instruments our empirical analyses [lo]. Furthermore, indicate that the performance of the Nursing Severity Index among patients with musculoskeletal disease was similar to the performance of other severity measures in populations of general medical and surgical patients [IB, 19,291, with respect to mortality, length of stay and hospital charges. We suggest that future studies compare the predictive validity of the Nursing Severity Index and other generic severity of illness measures in patients with musculoskeletal disease.
Growing concern by the federal government and other payers about the cost of health care has led to a national effort to measure the effectiveness of hospital care through the use of patient outcomes data [l, 21. The yearly release of hospital mortality data by HCFA [3,4] and recent hospital comparative reports from Pennsylvania [5], Iowa [6] and New York [7] exemplify this effort. However, the interpretation of outcomes data and assessment of the hospital effectiveness requires valid and practical instruments for adjusting for differences in severity of illness between patients. Although several generic measures are currently in use, the validity of these instruments in specific patient groups has been poorly studied. This is particularly true for hospital patients in whom the overall risk of mortality is low. We, therefore, examined the predictive validity of a new instrument, the Nursing Severity Index, in 5347 medical and surgical patients with musculoskeletal disease admitted to an aca- Methodologic considerations In examining our findings, potential methoddemic medical center. Our analyses revealed that admission Nursing Severity Index assess- ologic limitations related to the assembly of the study sample, the classification of patients using ments accurately stratified patients according to rates of in-hospital mortality, discharge to the Nursing Severity Index, designation of the nursing homes, and of having excessively long ‘hospital admission’ period, and the generalizability of study findings should be considered. lengths of stay (i.e. DRG length of stay outFirst, selection bias in the assembly of the study liers); in multivariate analyses, adjusting for groups is possible because the groups were important covariates, the Nursing Severity Index remained a strong independent predictor non-random samples of consecutive admissions. of each of the three outcomes. In addition, we However, the study group included a large found that the Nursing Severity Index was an proportions of eligible patients (72%) and independent predictor of hospital charges and patients studied did not differ markedly from length of stay, even after adjusting for DRG those not studied. Second, although the nursing weight, a measure used to determine hospital diagnoses which comprise the Nursing Severity Index are widely used and are defined by explicit reimbursement, and other covariates. clinical criteria [30], misclassification of patients Comparison of the Nursing Severity Index to according to specific nursing diagnoses probinstruments ably occurs. However, inter-rater reliabilities of The Nursing Severity Index measures mulnursing diagnoses [29] are similar to physical tiple domains of health and, thus, is based on a examination findings, diagnostic test interpretbroader concept of illness than current severity ations, and medical diagnoses [35538]-data of illness instruments which are primarily based used in other severity measures. Third. desig-
186
Gary E. Rosenthal et al.
nation of the length of the ‘admission period’ is important to assessing the predictive validity of severity instruments because patients’ prognoses may change during hospitalization. In examining the Nursing Severity Index, we analyzed data collected on the first or second hospital day, a period that is similar to or shorter than that used in other methods [lo-121. Finally, the generalizability of our findings to other diagnostic categories in which the risk of mortality is low and to patients in other types of hospitals (e.g. community, non-teaching) should be established. Implications In recent years, efforts to measure the quality and efficiency of hospital care have focused on the analysis of patient outcomes [l, 21. For example, an analysis detailing ‘severityadjusted’ hospital mortality rates, lengths of stay, and charges for 66 specific DRGs among 26 hospitals was recently released by the Iowa Health Data Commission [6]; included are analyses of several musculoskeletal DRGs (209, 210, 215, 239 and 243). Proper interpretation of outcomes data, however, requires that methods used to adjust for differences in severity of illness be rigorously evaluated. Although generic severity of illness measures and arthritisspecific health status instruments have been developed, the current study represents the first critical evaluation of one such instrument in a large group of patients hospitalized with musculoskeletal diagnoses. In addition to establishing the predictive validity of the Nursing Severity Index, the current study demonstrates that the data necessary for severity adjustment can be accurately and concisely obtained during routine clinical practice. Furthermore, because Nursing Severity Index data can be gathered and used concurrently during hospitalization, the Index is complementary to the development of case managed care strategies which hospitals are implementing in an increasing fashion to improve patient outcomes and to deliver diagnostic, therapeutic and rehabilitative interventions more effectively [39]. As the demand by payers for information about the effectiveness of hospital care increases and as the proportion of hospital care that is reimbursed prospectively increases, the need for valid instruments that can account for differences in patient severity becomes more critical. To avoid the creation of disincentives for hospitals to care for sicker patients [40], it is impera-
tive that severity measures be implemented that have been validated in specific clinical subgroups. Finally, our finding that the Nursing Severity Index was an independent predictor of nursing home discharge in patients with musculoskeletal disease has implications for the future assessment of hospital outcomes in clinical subgroups, in whom the risk of mortality is low. To our knowledge, the predictive validity of severity measures with respect to nursing home placement has not previously been examined. Because nursing home discharge occurs more frequently than in-hospital death in patients with muscuioskeletal disease, it may represent a more sensitive outcome indicator. Indeed, data from patients with hip fracture suggests that hospital quality may be directly related to rates of nursing home discharge [41]. Future studies should address the relationship of severityadjusted rates of nursing home discharge to quality of care and to the assessment of hospital outcomes. Conclusion We conclude that in a large group of ‘low risk’ medical and surgical patients with musculoskeletal conditions, the Nursing Severity Index was easily measured during routine patient care and that it accurately predicted the risk of in-hospital death, the rate of discharge to nursing homes and hospital resource use. These results support the validity of the Nursing Severity Index as a method of assessing and comparing hospital outcomes in patients with musculoskeletal disease. Furthermore, these results suggest that the use of the Nursing Severity Index may avoid the creation of disincentives for hospitals to care for more severely ill patients. Therefore, we feel that the Nursing Severity Index merits further testing in patients with other diagnoses and in other hospitals. Acknowledgemenfs-We thank Paul Jones, Ph.D. and Roland W. Moskowitz, M.D. for their helpful reviews of this manuscript. Supported by NIH Grant AR-20618. U.S.P.H.S. Northeast Ohio Multipurpose Arthritis Center.
REFERENCES 1. Relman AS. Assessment and accountability: the third revolution in health care. New Engl J Med 1988; 319: 1220-1222. 2. Roper WL. Winkenwerder W. Hackbarth GM, Krakauer H. Effectiveness in health care: an initiative to evaluate and improve medical practice. New Engl J Med 1988; 319: 1197-1202.
Severity Adjustment
5. 6. 7. 8. 9.
10.
11.
12.
13. 14. 15. 16.
17.
18.
19. 20.
21. 22.
23.
New York Times. US releasing lists of hospitals with abnormal mortality rates. Wednesday, March 12, 1986. Sullivan LW, Toby W. Medicare Hospital Information Report 1988, 1989, 1990: Volumes l-55. Washington DC: Health Care Financing Administration, U.S. Department of Health and Human Services; 1992. Hospital Effectiveness Report. Harrisburg, PA: Pennsylvania Health Care Containment Council; 1991. Iowa Hospital Resource and Outcome Report, July 1990 through Juw 1991, DRG Specific Version. Des Moines, IA: Iowa Health Data Commission; 1992. Hannan EL, Kilburn H, O’Donnell JF, Lukacik G, Shields EP. Adult open heart surgery in New York state. JAMA 1990; 264: 2768-2774. Shaller DV, Woods P. Reforming the market for value in health care: the Cleveland experience. J Occup Med 1991; 33: 358-362. McMahon LF, Billi JE. Measurement of severity of illness and the Medicare prospective payment system: state of the art and future directions. J Gen Intern Med 1988; 3: 482490. Iezzoni LI. Measuring the severity of illness and case mix. In: Providing Quality Care: the Challenge to Clinicians Goldfield N, Nash DB (Eds). Philadelphia, PA. American College of Physicians; 1989. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system for severely ill patients. Crit Care Med 1985; 13: 818-829. Brewster AC, Karlin BG, Hyde LA, Jacobs CM, Bradbury RC, Chae YM. MEDISGROUPS: a clinically based approach to classifying hospital patients at admission. Inquiry 1985; 22: 3777387. Horn SD, Horn RA. The computerized severity index: a new tool for case-mix management. J Med Syst 1986; 10: 73378. Gonneila JS. Hornbrook MC. Staging a disease: a case-mix measurement. JAMA 1984; 251: 637Wj44. Young WW, Swinkola RB, Zorn DM. The measurement of hospital case mix. Med Care 1985; 20: 501-512. Daley J, Jencks S, Draper D, Lenhart G, Thomas N, Walker J. Predicting hospital-associated mortality for Medicare patients: a method for patients-with stroke, pneumonia, acute myocardial infarction, and congestive heart failure. JAMA 1988; 260: 3617-3624. Knaus WA, Wagner DP, Draper EA, et al. The APACHE III prognostic system: risk prediction of hospital mortality for critically ill hospitalized patients. Chest 1991; 100: 1619-1636. Thomas JW, Ashcraft MLF, Zimmerman J. An evaluation of alternative severity of illness measures for use by university hospitals. Ann Arbor, MI: Department of Health Services Management and Policy, School of Public Health, University of Michigan; 1986. Iezzoni LI, Moskowitz MA. A clinical assessment of MedisGroups. JAMA 1988; 260: 3 159-3 163. Alemi F, Rice J, Hankins R. Predicting in-hospital survival of myocardial infarction: a comparative study of various severity measures. Med Care 1990; 28: 762-775. Blumberg MS. Biased estimates of acute myocardial infarction mortality using MedisGroups admission severity groups. JAMA 1991; 265: 2965-2970. Lichtenstein MJ, Federspiel CF, Shaffner W. Factors associated with early demise in nursing home residents: a case control study. J Am Geriatr Sot 1985; 33: 315-319. Reuben DB, Mor V, Hiris J. Clinical symptoms and length of survival in patients with terminal cancer. Arch Intern Med 1988; 148: 15861591.
in Low Risk Patients 24.
25. 26.
27.
28. 29.
30. 31. 32. 33. 34. 35. 36. 37.
38.
39. 40.
41.
187
Narain P, Rubenstein LZ, Wieland CD, Rosbrook B, ef al. Predictors of immediate and 6-month outcomes in hospitalized elderly patients: the importance of functional status, J Am Geriatr Sot 1988; 36: 775-783. Greenfield S, Aronow HU, Elashoff RM, Watanabe D. Flaws in mortality data: the hazards of ignoring comorbid disease. JAMA 1988; 260: 2253-2255. Reuben DB, Rubenstein LV, Hirsch SH, Hays RD. Value of functional status as a predictor of mortality: results of a prospective study. Am J Med 1992; 93: 6633669. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chron Dis 1987; 40: 373-383. Kaplan MH, Feinstein AR. The importance of classifying initial comorbidity in evaluating the outcome of diabetes mellitus. J Chron Dis 1974; 27: 387404. Rosenthal GE, Halloran EJ, Kiley M, Pinkley C, Landefeld CS. Development and validation of the Nursing Severity Index: a new method for measuring severity of illness using nursing diagnoses. Med Care 1992; 30: 1127-I 141. Carpenito LJ. Nursing Diagnosis: Application to Clinical Practice, 3rd edn. Philadelphia, PA: JB Lippincott Co.; 1989. Gordon M. Nursing Diagnosis: Process and Application. New York: McGraw-Hill; 1982. Hanley JA, McNeil BJ. The meaning and use of the area nnder a receiver operating characteristic (ROC) curve. Radiology 1982; 148: 8399843. The LOGISTIC procedure. SAS/STAT Users Guide, Version 6, 4th Edn, Vol. 2. Cary, NC: SAS Institute Inc.; 1990. The GLM procedure. SAS/STAT Users Guide, Version 6, 4th Edn. Vol. 2. Carv, NC: SAS Institute Inc.; 1990. Koran LM. The reliability of clinical methods, data, and judgments (part one). New Engl J Med 1975; 293: 642. Koran LM. The reliability of clinical methods, data, and judgments (part two). New Engl J Med 1975; 293: 695. Iezzoni LI, Burnside S, Sickles L, Moskowitz MA, Sawitz E, Levine PA. Coding of acute myocardial infarction: clinical and policy implications. Ann Intern Med 1988; 109: 745. Anderson RE, Hill RB, Key CR. The sensitivity and specificity of clinical diagnostics during five decades: toward an understanding of necessary fallibility. JAMA 1989; 261: 1610. Zander K. Nursing case management: strategic management of cost and quality outcomes. J Nursing Admin 1988; 18: 23330. Horn SD, Bulkley G, Sharkey PD, Chamber AF, Horn RA, Schramm CJ. Interhospital differences in severity of illness: problems for prospective payment based on diagnosis-related groups (DRGs). New Engl J Med 1985; 313: 20-24. Fitzgerald JF, Moore PS, Dittus RS. The care of elderly patients with hip fracture: changes since implementation of the prospective payment system. New Engl J Med 1988; 319: 1392-1397.
APPENDIX The 34 individual nursing diagnoses which comprise the Nursing Severity Index are listed below. The diagnoses are grouped into five subscales, based on Gordon’s model of “dysfunctional health patterns” [31] which reflect specific aspects of nursing care. Each diagnosis is opera-
188
Gary E. Rosenthal et al.
tionalized by clinical nurses, using explicit and standardized criteria [30]. (I) Overall health and perceptions Potential for injury Infection/contagion Prolonged disease/disability Instability Impaired life support systems (II) Nutrition and metabolism Excess fluid volume Fluid volume deficit Bleeding Less nutrition than required Potential skin impairment Alterations in oral mucous membranes Altered body temperature (III) Urinary and fecal function Urinary incontinence Other altered urinary elimination Constipation
pattern
Diarrhea Bowel incontinence (IV) Activity and exercise Activity intolerance Ineffective airway clearance Altered breathing pattern Impaired gas exchange Decreased cardiac output Altered health maintenance Impaired mobility Self-care deficit ( V) Psycho -social concerns Disturbed self-concept Depression Grieving Altered family process Social isolation Impaired verbal communication Ineffective individual coping Potential for growth in family coping Spiritual distress