Insights from a machine learning model for predicting the hospital Length of Stay (LOS) at the time of admission

Expert Systems With Applications 78 (2017) 376–385

Contents lists available at ScienceDirect

Expert Systems With Applications journal homepage: www.elsevier.com/locate/eswa

Insights from a machine learning model for predicting the hospital Length of Stay (LOS) at the time of admission Lior Turgeman a,∗, Jerrold H. May a, Roberta Sciulli b a b

Joseph M. Katz Graduate School of Business, University of Pittsburgh, Mervis Hall, Pittsburgh, PA, 15260, USA Veterans Affairs Pittsburgh Healthcare System, Veterans Engineering Resource Center, Pittsburgh, PA, 15215, USA

a r t i c l e

i n f o

Article history: Received 10 August 2016 Revised 22 January 2017 Accepted 12 February 2017 Available online 16 February 2017 Keywords: Cubist decision tree Continuous association rule mining algorithm (CARMA) Support vector machine (SVM) Decision function Error distribution Length of Stay (LOS)

a b s t r a c t A model that accurately predicts, at the time of admission, the Length of Stay (LOS) for hospitalized patients could be an effective tool for healthcare providers. It could enable early interventions to prevent complications, enabling more efficient utilization of manpower and facilities in hospitals. In this study, we apply a regression tree (Cubist) model for predicting the LOS, based on static inputs, that is, values that are known at the time of admission and that do not change during patient’s hospital stay. The model was trained and validated on de-identified administrative data from the Veterans Health Administration (VHA) hospitals in Pittsburgh, PA. We chose to use a Cubist model because it produced more accurate predictions than did alternative techniques. In addition, tree models enable us to examine the classification rules learned from the data, in order to better understand the factors that are most correlated with hospital LOS. Cubist recursively partitions the data set as it estimates linear regressions for each partition, and the error level differs for different partitions, so that it is possible to deduce what are the characteristics of patients whose LOS can be accurately predicted at admission, and what are the characteristics of patients for whom the LOS estimate at that point in time is more highly uncertain. For example, our model indicates that the prediction error is greater for patients who had more admissions in the recent past, and for those who had longer previous hospital stays. Our approach suggests that mapping the cases into a higher dimensional space, using a Radial Basis Function (RBF) kernel, helps to separate them by their level of Cubist error, using a Support Vector Machine (SVM). © 2017 Elsevier Ltd. All rights reserved.

1. Introduction The hospital Length of Stay (LOS) and hospital readmissions play an important role in determining hospital resource utilization. In this paper, LOS is used to describe the duration of a single episode of hospitalization, calculated by subtracting the date and time of admission from the date and time of discharge. It is useful to predict a patient’s expected LOS, or to model LOS, in order to determine the factors that affect it. Accurately predicting the LOS would enable a hospital to predict the discharge dates of admitted patients with a high level of certainty. Doing so could improve the scheduling of elective admissions, leading to a reduction in the variance of hospital bed occupancy (Robinson, Davis, & Leifer, 1966). Accurately predicting the LOS could also allow hos-

∗ Corresponding author. Current address: IBM Research, University of Haifa Campus, Haifa, Israel. E-mail addresses: [email protected] (L. Turgeman), [email protected] (J.H. May), [email protected] (R. Sciulli).

http://dx.doi.org/10.1016/j.eswa.2017.02.023 0957-4174/© 2017 Elsevier Ltd. All rights reserved.

pitals to scale their capacity during their long term strategic planning, as well as to predict health care costs (Cosgrove, 2006). If accurately predicted at the time of a patient’s discharge, LOS could be used for accounting and reimbursement purposes, by comparing the predicted value to the actual LOS. There are ongoing efforts to force hospitals to constantly adopt policy changes designed to reduce both the LOS and readmission rates, which would require hospitals to review their processes to become more cost efficient and standardized (Carter & Potts, 2014). Congestive heart failure (CHF) is a progressive disorder, which, in many cases, requires intensive treatment and follow-up of the affected patients (Johansen, Strauss, Arnold, Moe, & Liu, 2003; McMurray et al., 1998). The CHF diagnosis includes some of the highest percentages of patients who are readmitted to a hospital (Aggarwal & Gupta, 2014; Islam, O’Connell, & Lakhan, 2013; Krumholz et al., 1997; Vinson, Rich, Sperry, Shah, & McNamara, 1990), and is also the leading cause of hospital admissions among patients over the age of 65 (Parmley, 1989). CHF is also known to be associated with high rates of mortality and morbidity (Schocken, Arrieta, Leaverton, & Ross, 1992). Whellan et al. studied the predictors of LOS for heart failure patients, using hospi-

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385

tal characteristics and clinical data, available at the time of admission. Based on a patient’s vital signs and laboratory values known at the time of admission, he concluded that patients with longer LOS had more comorbidities and a higher disease severity than did patients with shorter LOS (Whellan et al., 2011). Kossovsky et al. found that the probability of hospitalization for heart failure patients increases with age (Kossovsky et al., 20 0 0), whereas Wexler et al. demonstrated that greater age was associated with decreased costs for these patients (Wexler et al., 2001). Other studies have shown that patient characteristics, such as gender (Philbin & DiSalvo, 1998), race (Croft et al., 1997), and insurance coverage (Philbin, Rocco, Lynch, Rogers, & Jenkins, 1997) could also affect the LOS of CHF patients. There appears to be a general agreement among researchers that CHF-related hospitalization is a complex process, due to the interplay between aging, physical, and psychosocial factors affecting their LOS. Thus, the care of CHF patients in hospitals may be further enhanced by providing a framework that allows accurate prediction of their LOS. The actual LOS is typically highly skewed, so that statistical approaches that take the skewness into account are more likely to be successful than are those that do not do so (Carter & Potts, 2014; Turgeman, May, Ketterer, Sciulli, & Vargas, 2015). A variety of techniques have been used to model and to predict the LOS in different contexts. Regression methods used include linear, log-normal, and logistic regression (Mahadevan, Challand, & Keenan, 2010; Zheng, Cammisa, Sandhu, Girardi, & Khan, 2002). However, those regression approaches have been criticized, by some researchers, for not taking into account the skewness of the data (Carter & Potts, 2014; Faddy, Graves, & Pettitt, 2009). Carter et al. recommend, instead, the use of negative binomial regression (Carter & Potts, 2014). Verburg et al. predicted the LOS in an Intensive Care Unit (ICU) using eight regression models: ordinary least squares regression, a generalized linear model with a Gaussian distribution and a logarithmic link function, Poisson regression, negative binomial regression, Gamma regression with a logarithmic link function, and the original and recalibrated APACHE IV model. The predictive performance of the models was poor, perhaps due to the difficulty of predicting the ICU LOS for patients with unplanned admissions using only patient characteristics at ICU admission time, as well as the skewness of the data and the high mortality rate of ICU patients (Verburg, de Keizer, de Jonge, & Peek, 2014). The skewness of the LOS data could be taken into account by truncating the data at 30 days, or by fitting a Phase type (PH) distribution to the data, which is created by interrelated Poisson processes (Faddy & McClean, 1999; Turgeman et al., 2015). The PH distribution describes the time to absorption in a finite-state continuous time homogeneous Markov chain, in which there is a single absorbing state, and in which the stochastic process begins in a transient state. Covariates may be incorporated into such models, further enhancing their ability to describe complex processes (Faddy & McClean, 1999). McGrory et al. presented a reversible jump Markov chain Monte Carlo scheme for performing a fully Bayesian analysis to infer a Phase-type Coxian distribution with a Covariate Dependent Mean (McGrory, Pettitt, & Faddy, 2009). Marshall and McClean fit a conditional Phase type Coxian distribution to the LOS data for geriatric patients (Marshall & McClean, 2003). The data were first categorized by a Bayesian belief network, and then fitted using maximum likelihood estimation. The Bayesian belief network links various causal characteristics of the data, allowing categorization of patient LOS data into twelve groups before fitting the Coxian distribution. However, most of the models were limited by the extent of their ability to infer individual patient LOS from the fitted Phase type distribution, or the extracted Markov state, by taking into account patient characteristics, such as demographics, comorbidities, historical admis-

377

sions, lab and vitals values, inpatient procedures, and other health factors. Among those who used machine learning methods to model the LOS, Liu et al. combined a C5.0 tree (Quinlan, 2014) with a Naive Bayes (Murphy, 2006) classifier to predict the LOS in a geriatric hospital department. They improved the performance of their classifier by using a Naive Bayes model to impute values for missing data (Liu et al., 2006). Tanuja et al. compared a multilayer neural network (MLP), a Naive Bayes classifier, the k-nearest neighbors algorithm (KNN) (Larose, 2005), and a C4.5 decision tree (Quinlan, 2014), on the LOS data of an Indian hospital. The MLP algorithm was found to have better performance, compared to the other three techniques (Tanuja, Acharya, & Shailesh, 2011). Pendharkar and Khurana compared two regression tree methods, Classification And Regression Tree (CART) (Breiman, Friedman, Olshen, & Stone, 1984) and Chi-squared Automatic Interaction Detection (CHAID) tree (Kass, 1980), and support vector regression (Vapnik, 1998), using LOS data from 88 hospitals in Pennsylvania. Although there was no significant difference in the models’ performance, they noted that the CART tree was easier to understand and to interpret (Pendharkar & Khurana, 2014). Houthooft et al. compared different machine learning algorithms in order to predict the LOS of critically ill patients in the intensive care unit (ICU). The best performing model was support vector regression, achieving a mean absolute error of 1.79 days for those patients surviving a non-prolonged stay (Houthooft et al., 2015). The objective of this study is to describe the development and validation of a new machine learning model to predict the LOS of patients at the time of admission. A prediction of the length of stay, calculated at the time of admission, may be inaccurate for two reasons. One, features that significantly affect the length of stay may not be available until after the patient has been resident for at least 24 to 48 hours. Two, the values for features that are available at the time of admission, such as the principal diagnosis, may be revised during the course of the hospital stay. As a result, we believe it is important be able to categorize the error level of such a model, at the time of admission. Such a tool would allow the clinical teams that use the model to understand the validity of its predictions, as well as to apply the relevant interventions. In our study, we induce a Cubist (Quinlan, 1992) tree model to predict the LOS of congestive heart failure (CHF) patients at the time of admission, using de-identified administrative data from the Veterans Health Administration (VHA) facilities in Pittsburgh. The demographic, historical, and clinical predictors are static model inputs that are known at the time of admission, and do not change during patient’s stay. Cubist, like CART and CHAID, is a regression tree model, meaning that it recursively splits the data into appropriate subsets, and predicts a continuous value at the terminal leaf nodes. Unlike CART and CHAID, whose prediction is a single number, the mean of the subset of data included in the node, Cubist bases its estimate on an absolute-error-minimizing linear regression. As with other tree methods, the Cubist tree can also be represented by a series of decision rules, each of which has a linear regression as its consequent. We further investigate relations between patient characteristics within the groups of cases defined by the Cubist rules by using the CARMA (Continuous Association Rule Mining Algorithm) (Hidber, 1999) algorithm. We analyze the Cubist error distribution across the different extracted rules, and compare the Cubist performance with other machine learning algorithms. Mapping our database cases into a higher dimensional space using a Radial Basis Function (RBF) kernel helps to separate them by their level of Cubist error, based on a support vector machine algorithm.

378

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385

2. Materials and methods 2.1. Data We were granted access to the anonymized administrative records of 20,321 inpatient admissions, of 4840 patients, from the Veterans Health Administration (VHA) hospitals in Pittsburgh, for fiscal years (FY) 2006 through 2014. All patients in that dataset had been diagnosed with CHF during that time period, although the admissions were for all causes. The number of admissions for each patient ranged between one and 32. The following variables were considered: Patient demographics: Age Marital status Gender Relation to next of kin Power of attorney on file (yes/no) Previously diagnosed combordities (yes/no): Anemia Asthma Chronic Obstructive Pulmonary Disease (COPD) Chronic Renal Failure (CRF) Cerebrovascular Accident stroke (CVA) Diabetes Dementia Depression (DPRSN) Hypertension (HTN) Ischemic Heart Disease (IHD) Post Traumatic Stress Disorder (PTSD) Peripheral Vascular Disease (PVD) Outpatient history: Completed appointment rate Number of emergency visits (prior 3, 6, 9, and 12 months) Number of primary care visits (prior 30 days and prior year) Number of cardiology visits (prior 30 days and prior year) Number of pulmonary visits (prior 30 days and prior year) Number of mental health visits (prior 30 days and prior year) Number of telephone visits (prior 30 days and prior year) Number of other non-face visits (prior 30 days and prior year) Total number of visits (prior 30 days and prior year) Inpatient history: Number of admissions (prior 3, 6, 9, and 12 months) Number of all hospital stays (prior 30 days and prior year) Total LOS in all hospitals (prior 30 days and prior year) Number of mental hospital stays (prior 30 days) Number of bed days in all hospitals (prior year) Total LOS in all nursing homes (prior year) Number of bed days in all nursing homes (prior year) Medication history: Number of prescription drugs filled (prior year) Number of distinct drug types prescribed (prior year) Number of medication providers (prior year) Lab values and vital signs: Albumen (most recent reading) Blood Urea Nitrogen (BUN) (most recent reading) Creatinine (most recent reading)

Potassium (most recent reading) White Blood Cell (WBC) (most recent reading) Systolic blood pressure Diastolic blood pressure Pulse rate Respiration Information from the current admission: Source of admission Time elapsed since last discharge (in days) The LOS for each inpatient admission, for each patient, is calculated by subtracting the date and time of admission from the date and time of discharge. The time elapsed between the previous inpatient admission and the current index admission, for each patient, was calculated by the date difference between them. 2.2. Data pre-processing A few pre-processing steps were taken before analyzing the data: 1. Cohorts of CHF patients were identified based upon the ICD9 codes of the previous principal (or secondary) inpatient discharge diagnoses. Patients may have had hospital admissions prior to their first CHF diagnosis. 2. All records of the same patient were merged if the patient had multiple hospitalizations on the same day to the same medical unit. That applied to both medical and surgical inpatients. 3. Admissions that were recorded as having occurred within 24 hours of the immediately prior index discharge were excluded from the admission set. 4. Inconsistent and/or erroneous components, such as age discrepancies, or a discharge date that preceded the admission date, were excluded. 5. Missing values were imputed by the hot-deck method, using the same variable in prior or later observations of the specific patient in the data set. If the patient had no prior recorded inpatient admissions, his record was excluded. 6 The data set was separated randomly into a training set of 15,242 admissions (75%), and test (holdout/validation) set of 5078 admissions (25%). 2.3. Cubist extracted rules The Cubist model structure consists of decision tree, in which the splits at the nodes are defined by partitioning based on a single independent variable, combined with multiple linear regression models operating at the leaves (terminal nodes). Linear models are also calculated at each step of the tree. All the models are based on the variables used in previous splits. Both continuous and categorical variables can be used to split the data into more homogeneous sub-regions; continuous variables are optimally split into two ranges if they are used at a partitioning step. A prediction is made using the linear regression model at the terminal node of the tree, “smoothed” by taking into account the prediction from the linear model at the previous node of the tree. Generally, a Cubist regression model is easy to understand and interpret. Furthermore, the Cubist software provides the attribute usage (relative importance) of each variable in the model. The final tree is reduced to a set of rules that represent the paths from the top of the tree to the bottom (Quinlan, 1992). For example, a Cubist rule might look like: If (total LOS in all hospitals in the last 30 days ≤ 16) and (Number of all hospital stays in the last 30 days > 1) and (Number of bed days in all hospitals in the last year ≤ 9)

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385

Then LOS = 2.3–0.0183∗ time elapsed since last discharge (days) + 0.482∗ total LOS in all hospitals in the last 30 days 1.12∗ number of all hospital stays in the last 30 days + 0.012∗ Number of "Other" outpatient visits in the last 30 days. For each rule, the software provides the number of cases that were classified by that rule, the mean value of the target variable (LOS) for the records that are covered by that rule, and the estimated error for that rule. The coefficient of determination, R2 , represents the proportion of the variance in the dependent variable, LOS in our case, which is predictable from the independent variables. The Mean Absolute Error (MAE) measures how close are the LOS values predicted by the Cubist to the eventual LOS outcomes (Eq. 1).

MAE =

n  1    LOSi − LO Si  n

(1)

i=1

379

where p is the percentage error.A case falls into the category of being predicted with high error for a particular value of p if it satisfies Equation (2) for that value of p.Using that definition,we investigated whether it is possible to separate those cases with greater Cubist error from cases with lower Cubist error,using SVM. The SVM technique is based on the principle of structural risk minimization, in which the training set is mapped into a highdimensional feature space, using a nonlinear transformation, referred to as the kernel. We used the Radial Basis Function (RBF) kernel (Vapnik, 1998), and the LIBSVM package (IBM SPSS Modeler Algorithms Guide, 2016; Chang & Lin, 2011). Given a set of training vectors xi ∈ Rl , i = 1, ..., l, in two classes, and a vector y ∈ Rl , such that yi ∈ {−1, 1}, the LIBSVM algorithm solves the following optimization problem for the training set:

min f (α ) =

1 T α Q α − eT α 2

(3)

 where LO Si is the predicted value, LOSi is the observed value, and n is the sample size.

Subject to

2.4. Groups’ characteristics induced using CARMA (Continuous Association Rule Mining Algorithm)

The vector α = (α1 , α2 , .., αl )T contains the coefficients for the training samples (the coefficient is zero for non-support vectors). Q (xi · x j ) = yi y j K (xi , x j ). K(xi , xj ) is the transformation kernel. The decision function is defined as:

Each extracted Cubist rule covers a different portion of the dataset, defining a group of cases. We used CARMA to understand the associations among factors, such as demographics, comorbidities, vitals, and labs, within each of the groups. CARMA is used to continuously produce large itemsets, along with a shrinking support interval for each itemset (Hidber, 1999), allowing the user to change the support threshold anytime during the first scan, and always terminates in at most two scans. CARMA outperforms Apriori (Agrawal & Srikant, 1994) and dynamic itemset counting (DIC) (Brin, Motwani, Ullman, & Tsur, 1997) for low support thresholds, and readily computes large itemsets in situations that are intractable for Apriori and DIC. For more accurate prediction of whether an itemset is potentially large, it calculates an upper bound for the count of the itemset, which is the sum of its current count and an estimate of the number of occurrences before the itemset is generated. The estimate of the number of occurrences is computed when the itemset is first generated (Dunham, Xiao, Gruenwald, & Hossain, 2001). The confidence for the final prediction is the sum of the confidence values for rules triggered by the current record that give the winning prediction, divided by the number of rules that fired for that record (IBM SPSS Modeler Algorithms Guide, 2016).

0 ≤ αi ≤ C, and yT α = 0

f (x ) =

l 

yi

  αi K xi , x j + b

(4)

i=1

where b is a constant term. The posterior probabilities are approximated using the sigmoid function

PA,B (x ) =

1 1 + exp(A f (x ) + B )

(5)

where the optimal parameters A and B are estimated by solving a regularized maximum likelihood problem (Platt, 1999). 3. Results 3.1. Feature selection We ran a large number of experiments with the Cubist data mining package (Quinlan, 1992; www.rulequest.com). The top predictors of the created models are internally selected by the Cubist algorithm. Table 1 shows a list of the most significant attributes, sorted by their usage in the model.

2.5. Error classification by a Support Vector Machine (SVM)

3.2. Ensembles by committees and instance–based corrections

We measure the inaccuracy of our model as the percentage of the absolute error of the prediction, relative to the actual LOS. Our model produces its prediction based on information that is available at the time of admission. Therefore, the model may be inaccurate because its structure is inadequate to produce a precise estimate. It may also be inaccurate because the information available at the time of admission is inaccurate. The LOS is a function of the medical or surgical reason for the admission. If, for example, the admitting diagnosis is appendicitis, for which a typical LOS is three days, but the actual primary diagnosis at discharge is intestinal infection, for which a typical LOS is ten days, a prediction based on our model is likely to be judged to be highly inaccurate, even if the model would have produced a close estimate if it had been presented with the discharge diagnosis. Because we measure the amount of error in relation to the actual LOS, we determined whether the Cubist prediction error was high or low, for each case, by using the following condition:

The Cubist model can also use a boosting–like scheme, called committees, where iterative model trees are created in sequence. The first tree follows the procedure described in the model section. Subsequent trees are created using adjusted versions of the training set outcome. If the model over–predicted a value, the response is adjusted downward for the next model. Unlike traditional boosting, stage weights for each committee are not used to average the predictions from each model tree. The final prediction is a simple average of the predictions from each model tree. Cubist can also use nearest–neighbors to adjust the predictions from the rule– based model. First, a model tree (with or without committees) is created. Once a sample is predicted by this model, Cubist can find its nearest neighbors, and determine the average of those training set points (Quinlan, 1993). We applied 10-fold cross-validation to obtain an unbiased estimate of the model’s predictive accuracy, and to find the optimal number of committees, where the final decision regarding which model to use was based on the lowest MAE (Eq. (1)).

  LOSi −  LOSi  > p · LOSi

(2)

380

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385

Table 1 Variables used in the 20 committee, 0 neighbors Cubist model. Variable

% of cases classified by a rule that includes this variable

% of rules that include this variable

Total LOSa in all hospitals in 30 days Number of all hospital stays in 30 days Total number of visits in 30 days Time elapsed since last discharge (days) Number of admissions during the last 3 months Number of bed days in all hospitals in the last year Total LOSa in all hospitals in the last year Number of Other, Non-Face visits in the last 30 days Number of "Other" visits in the last 30 days Total number of visits in the last year Number of all hospital stays in the last year Source of admission is Out Clinic Number of "Other" visits in the last year Number of mental hospital stays in the last 30 days Source of admission is Nursing Care Number of Primary Care visits in the last 30 days

88%

94%

6.24

4

8.475

13.041

74%

80%

1.15

1

0.592

1.294

51%

41%

35.96

29

25.627

1.634

46%

19%

148.56

31

284.337

3.359

35%

9%

0.61

0

0.962

2.09

30%

46%

14.64

9

23.905

18.833

11%

22%

14.82

9

21.584

11.999

7%

12%

7.29

6

5.199

1.512

7%

22%

24.01

18

19.772

1.763

7%

24%

153.59

125.455

1.849

6%

10%

2.59

2

2.173

1.992

0.25

0

0.432

1.171

72

96.019

2.329

0.01

0

0.108

10.567

0.02

0

0.15

6.382

1.72

0

3.318

3.672

a

5% 4%

25%

3%

7%

1% 1%

3%

Mean value of this variable across all cases

100.42

Median value of this variable across all cases

120

Std. Deviation of this variable across all cases

Skewness of this variable across all cases

Length of Stay

Fig. 1. Diagnostic Odds Ratio (DOR) for the different quartiles of Length of Stay (LOS).

As an example of the Cubist output, Table 2 includes a set of 10 rules that are the most accurate in terms of their MAE. The estimated LOS for a patient is predicted from the linear regression in the terminal node to which the patient is classified by the Cubist tree. The performance measures for the candidate Cubist models, with different numbers of committees and nearest–neighbors, are shown in Table 3. Dividing our LOS data into quartiles yields the following cut off values: Q1 = 2 days, Q2 = 4 days, and Q3 = 7 days. Based on those cut off values, we separated our patients into four groups. We then calculated the Diagnostic Odds Ratio (DOR) from the confusion matrix of each group. Fig. 1 shows the odds ratio for predicting patients in each quartile of the LOS. The DOR for predicting patients in all quartiles is positive, which indicates good predictive performance for all quartiles. The DORs for predicting patients the topmost (Q4 ) and bottommost (Q1 ) quartiles are 140.9 and 209.5, re-

spectively. The DORs for predicting patients in the second and third quartiles are 32.4 and 45.8, respectively. That is, patients in the short LOS range (LOS ≤ 2), and in the long LOS range (LOS > 7), are characterized with patterns that are better described by the Cubist rules. We investigated whether the Cubist error appears to be uniformly distributed across the Cubist rules. Fig. 2 shows the error distribution for the Cubist rules, for the simple case of no committees and no neighbors. Table 4 shows the average error that is associated with different conditions within the Cubist rules, sorted by their mean MAE across all the rules in which they appear. Interestingly, rules containing the condition “Time elapsed since last discharge ≤ 3 days” had a MAE of 8.1 days. Rules with the condition “Time elapsed since last discharge > 16 days” had a MAE of 0.1 days. Rules containing the condition “Total LOS in all hospitals in 30 days > 16 had a MAE of 2.27 days”, whereas rules containing the condition “Total LOS in all hospitals in the last year ≤ 6 days” had a MAE of 0.3 days. Those results might indicate that the LOS of patients who had been very recently discharged is difficult to predict, as is the LOS of a patient who has recently had an extended time in the hospital. 3.3. Comparison to Poisson regression, and to other machine learning methods Our data could have been modeled with a single Poisson regression model, applied to the entire training set. As an alternative, we used Cubist regression trees, which are constructed by splitting the entire data into subsets, using all the independent variables, in order to produce terminal nodes that are as homogeneous as possible with respect to the linear relationship of the target vari-

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385

381

Table 2. Ten Cubist rules with smallest Mean Absolute Error (MAE). Rule

Consequent

Mean Absolute Error (MAE)

Total LOSa in all hospitals in 30 days ≤ 16 and Total number of visits in 30 days ≤ 61 and Number of all hospital stays in 30 days > 1

LOS = 5.1 - 0.0671 time elapsed since last discharge + 0.335 Total LOS in all hospitals in 30 days - 2.04 Number of all hospital stays in 30 days + 0.023 Number of bed days in all hospitals in a year + 0.021 Number of "Other" visits in 30 days + 0.003 Total number of visits in 30 days + 0.07 Number of admissions during previous 3 months LOS = 3.8 + 0.39 Total LOS in all hospitals in 30 days - 1.96 Number of all hospital stays in 30 days + 0.04 Number of Other, Non-Face visits in 30 days + 0.006 Number of bed days in all hospitals in a year + 0.006 Number of "Other" visits in 30 days 0.004 Total LOS in all hospitals in a year - 0.0 0 06 Total number of visits in a year + 0.03 Number of all hospital stays in a year - 0.0 0 06 Number of "Other" visits in a year + 0.002 Total number of visits in 30 days LOS = 22.4 - 17.59 Number of all hospital stays in 30 days + 0.672 Total LOS in all hospitals in 30 days + 0.078 Number of bed days in all hospitals in a year- 13.4 Number of mental hospital stays in 30 days - 0.049 Total LOS in all hospitals in a year - 0.0052 Number of "Other" visits in a year + 0.024 Number of "Other" visits in 30 days + 0.03 Number of Other, Non-Face visits in 30 days + 0.003 Total number of visits in 30 days LOS = 1.2 + 0.871 Total LOS in all hospitals in 30 days - 1.25 Number of all hospital stays in 30 days + 0.019 Number of "Other" visits in 30 days + 0.013 Number of bed days in all hospitals in a year - 2.6 Number of mental hospital stays in 30 days + 0.008 Total number of visits in 30 days - 0.002 Number of "Other" visits in a year - 0.005 Total LOS in all hospitals in a year + 0.02 Number of Other, Non-Face visits in 30 days LOS = 4.1 + 0.553 Total LOS in all hospitals in 30 days - 0.0088 time elapsed since last discharge - 2.61 Number of all hospital stays in 30 days + 0.022 Total number of visits in 30 days + 0.023 Number of bed days in all hospitals in a year + 0.015 Number of "Other" visits in 30 days - 0.0021 Number of "Other" visits in a year - 0.0014 Total number of visits in a year + 0.14 Number of admissions during previous 3 months - 1.2 Number of mental hospital stays in 30 days - 0.003 Total LOS in all hospitals in a year + 0.01 Number of Other, Non-Face visits in 30 days LOS = 4.6 + 0.608 Total LOS in all hospitals in 30 days - 0.0093 time elapsed since last discharge - 2.6 Number of all hospital stays in 30 days + 0.006 Total number of visits in 30 days - 0.001 Total number of visits in a year + 0.12 Number of admissions during previous 3 months + 0.0 0 07 Number of "Other" visits in a year LOS = 13.1 - 3.83 Number of all hospital stays in 30 days - 0.0122 Total number of visits in a year + 0.0134 Number of "Other" visits in a year + 0.11 Number of Other, Non-Face visits in 30 days + 0.49 Number of admissions during previous 3 months + 0.015 Total number of visits in 30 days + 0.008 Number of bed days in all hospitals in a year - 0.007 Number of "Other" visits in 30 days + 0.015 Total LOS in all hospitals in 30 days - 0.9 Number of mental hospital stays in 30 days LOS = 1.8 + 0.464 Number of bed days in all hospitals in a year + 0.507 Total LOS in all hospitals in 30 days - 1.29 Number of all hospital stays in a year - 0.002 time elapsed since last discharge - 0.49 Number of all hospital stays in 30 days + 0.003 Total number of visits in 30 days LOS = 6.8 + 0.81 Total LOS in all hospitals in 30 days - 0.0223 time elapsed since last discharge - 5.04 Number of all hospital stays in 30 days - 0.0024 Total number of visits in a year + 0.011 Total number of visits in 30 days + 0.0017 Number of "Other" visits in a year + 0.04 Number of all hospital stays in a year LOS = 3.5 + 0.818 Total LOS in all hospitals in 30 days - 2.14 Number of all hospital stays in 30 days - 0.0025 time elapsed since last discharge + 0.01 Total number of visits in 30 days + 0.01 Number of bed days in all hospitals in a year + 0.004 Number of "Other" visits in 30 days - 0.0 0 08 Number of "Other" visits in a year - 0.0 0 05 Total number of visits in a year

0.2

Total LOS in all hospitals in 30 days ≤ 16 and Number of all hospital stays in 30 days > 1

Total LOS in all hospitals in 30 days > 16 and Total LOS in all hospitals in 30 days ≤ 30 and Number of all hospital stays in 30 days = 1

Total LOS in all hospitals in 30 days ≤ 16 and Number of "Other" visits in a year > 200 and Total number of visits in 30 days > 61

Total LOS in all hospitals in 30 days ≤ 16 and Number of admissions during previous 3 months > 0 time elapsed since last discharge ≤ 28 and Total number of visits in 30 days ≤ 61 and Number of all hospital stays in 30 days > 0 and Number of bed days in all hospitals in a year ≤ 9

Total LOS in all hospitals in 30 days ≤ 8 and time elapsed since last discharge ≤ 28 and Number of all hospital stays in a year > 2 and Number of all hospital stays in 30 days > 0

Total LOS in all hospitals in 30 days > 16 and Number of all hospital stays in 30 days > 1

Total LOS in all hospitals in 30 days ≤ 16 and Number of admissions during previous 3 months = 0 and time elapsed since last discharge ≤ 28 and Number of all hospital stays in 30 days > 0 and Number of bed days in all hospitals in a year ≤ 9 Total LOS in all hospitals in 30 days ≤ 16 and time elapsed since last discharge ≤ 28 and Total number of visits in 30 days ≤ 61 and Number of all hospital stays in a year ≤ 2 and Total LOS in all hospitals in a year > 9 and Number of all hospital stays in 30 days > 0 and Number of bed days in all hospitals in a year > 9 Total LOS in all hospitals in 30 days > 8 and Total LOS in all hospitals in 30 days ≤ 16 and Total number of visits in 30 days ≤ 61 and Number of all hospital stays in a year > 2 and Number of all hospital stays in 30 days = 1

a

Length of Stay

0.3

0.4

0.7

1.1

1.5

1.5

1.6

1.8

2.1

382

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385 Table 3 Performance measures for the candidate Cubist models. Committees

Neighbors

MAEa (Training set)

R2 (Training set)

MAE (Testing set)

R2 (Testing set)

1 1 1 10 10 10 20 20 20

0 1 5 0 1 5 0 1 5

0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8 0.8

0.82 0.82 0.84 0.84 0.84 0.84 0.84 0.84 0.84

1 1 1 1 1 1 1 1 1

0.75 0.75 0.77 0.77 0.77 0.77 0.79 0.79 0.79

a

Mean Absolute Error

Fig. 2. Mean Absolute Error (MAE) for the Cubist rules (no committees, no neighbors).

Table 4 Mean Absolute Error (MAE) for different conditions within the Cubist rules. Condition

MAEa

Time elapsed since the last discharge ≤ 3 Number of bed days in all hospitals in the last year > 19 Total LOS in all hospitals in the last 30 days > 30 Total number of outpatient visits in the last 30 days > 61 Total number of outpatient visits in the last 30 days ≤ 44 Total LOS in all hospitals in the last year ≤ 6 Number of bed days in all hospitals in the last year ≤ 5 Time elapsed since last discharge > 16

8.1 6.3 4.76 2.12 0.4 0.3 0.1 0.1

a

Mean Absolute Error

able to a particular subset of the independent variables. If the entire data set shares a common Poisson regression model relationship between the target and the same set of independent variables, then the Poisson regression model should be superior to the Cubist model. The more heterogeneous the data are with respect to the relationship between the target and the independent variables, or if there is a nonlinear relationship between the target and the independent variables, and the form of that nonlinear relationship is unknown, so that a piecewise approximation to it would improve the goodness of fit, the Cubist model should be superior to Poisson regression model. Thus, a comparison of our Cubist model to the Poisson regression model might be indicative of the complexity of the LOS predictive problem. The MAE and R2 for Poisson regression model, the Cubist model, and other, alternative data mining methods, applied to the LOS data, are given in Table 5. 3.4. Cubist error classification by SVM Fig 3(a) and (b) show the Probability Mass Function (PMF) of the actual LOS for different SVM posterior probabilities (Eq. 5), for

the binary target value of having an error that is within 10% of the actual LOS (p = 0.1) (Case 1, displayed in Fig. 3(a)), and for the binary target value of having an error that is within 40% of the actual LOS (p = 0.4) (Case 2, displayed in Fig. 3(b)). The SVM predictions for the Case 1 target value are characterized by greater posterior probabilities confidence for a wider range of LOS values, as compared to its predictions for the Case 2 target value. For short LOS values (0–5 days), the probability of more confident SVM predictions is greater for Case 1 (Fig. 3(a)) than it is for Case 2 (Fig. 3(b)). Table 6 shows the precision, recall, and total accuracy for different error intervals, for the test set. The greatest area under the ROC curve (AUC), 0.83, occurs at p = 0.2. However, the greatest precision value is obtained for Case 1. Fig. 4 shows the receiver operating characteristic curve (ROC) curves for Case 1 (Fig. 4(a)), as compared to Case 2 (Fig. 4(b)). The precision value is important because it answers the question "If the Cubist error is within a specific error interval, how well does the SVM model predict the actual presence of such an error?”. 4. Discussion We used a Cubist tree to predict the hospital Length of Stay (LOS), estimated at the time of admission, using only information that would be available at that time. We induced the model, and tested it, using de-identified, administrative data from the VA hospital system in Pittsburgh, using records of 20,321 inpatient admissions of 4840 Congestive Heart Failure (CHF) patients. We tested the performance of the Cubist tree model by comparing it to a linear regression model and to other machine learning methods. Our results indicate that the LOS primarily depends on the number of previous admissions, the number of previous outpatient visits, the time that has elapsed since the last discharge, and the

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385

383

Table 5 Comparison of data mining methods. Method

MAEa (Training set)

R2 (Training set)

MAE (Testing set)

R2 (Testing set)

Neural network CARTb tree CHAIDc tree Generalized linear model (Poisson) Support vector machine Cubist (20 committees)

1.76 1.49 1.87 2.74 1.59 0.8

0.68 0.67 0.61 0.33 0.66 0.84

1.81 1.53 1.95 2.76 1.95 1

0.67 0.66 0.59 0.40 0.58 0.79

a b c

Mean Absolute Error Classification And Regression Tree Chi-squared Automatic Interaction Detection

Fig. 3. Probability Mass Function (PMF) of the actual Length of Stay (LOS) for different Support Vector Machine (SVM) confidence values (Eq. 5), for p = 0.1 of the actual LOS (a) and for p = 0.4 of the actual LOS (b).

Fig. 4. Receiver Operating Characteristic (ROC) curves of the Support Vector Machines (SVM) for errors of p = 0.1 (a) and p = 0.4 (b).

384

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385 Table 6 Support Vector Machine (SVM) classification performance.

Error Error Error Error a

is is is is

within within within within

10% 20% 30% 40%

Precision

Recall

Accuracy

AUCa

0.8214 0.72855 0.65995 0.56

0.616698 0.497992 0.258383 0.104218

0.838 0.831 0.825 0.844

0.827 0.83 0.79 0.743

Area Under the Curve

number of bed days the patient had in the year prior to the current index admission. We found that the groups of cases covered by the Cubist rules differ in their patient characteristics. Applying the CARMA algorithm allows us to discover some interesting relations among variables, such as comorbidities, demographics, lab values and readmission risk, of patients within those groups. For example, a high confidence is obtained for the following rule: patients with anemia and COPD (Chronic Obstructive Pulmonary Disease) who belong to the group of cases that are covered by the following Cubist rule (Group A): (Total LOS in all hospitals for discharges that occurred in the last 30 days ≤ 16) and (time elapsed since last discharge > 28) and (Total number of outpatient visits in 30 days ≤ 61) and (Number of all hospital admissions in the last 30 days > 0), are more likely to have hypertension, (confidence = 91.1%, support = 29%). On the other hand, patients with anemia and COPD who also have diabetes, or males with diabetes in their 50 s, are more likely to have hypertension, if they belong to group of cases defined by the following Cubist rule (Group B): (Total LOS in all hospitals for discharges that occurred in the last 30 days ≤ 16) and (time elapsed since last discharge ≤ 28) and (Number of all hospital admissions in the last 30 days > 0) and (Number of bed days in all hospitals in the last year ≤ 9), (confidence = 92.2%, support = 23% for males with diabetes in their 50 s, and confidence = 94.1%, support = 22.6% for patients with anemia, COPD, and diabetes). The strong association between anemia and COPD has already been noted by others (John et al., 2005; Mascitelli & Pezzetta, 2005). So has the association between CHF and hypertension (Ho, Anderson, Kannel, Grossman, & Levy, 1993; Wong et al., 2007). Others have pointed out the possibility of a strong association between hypertension and the 30day readmission rate for CHF patients. Our results indicate that the latter association depends not only on the specific combination of present comorbidities, but also on factors such as the total length of previous hospital stays, and age, as already discussed in the literature (Hamner & Ellison, 2005; Vinson et al., 1990). We found that Ischemic Heart Disease (IHD) is a dominant consequent of many CARMA rules, within both groups (A and B). As is already discussed in the literature, the presence of IHD may increase the chances of readmission for CHF patients (Krumholz et al., 20 0 0). However, our results indicate that males with IHD whose lab values and vitals are in the normal range are more likely to belong to Group A, that is, they have a greater elapsed time since their last discharge. The degree to which our suggested approach directly generalizes to the non-VA patient population may be affected by specific characteristics of the VA population. Previous studies have shown that the VA population exhibits a set of common chief complaints that is similar to that of the national emergency department (ED) population (and in similar proportions). However, the VA population is predominantly male and, on average, is older than the general population, and has a lower incidence of trauma and a greater rate of mental health problems (Kessler, Bhandarkar, Casey, & Tenner, 2011). In the VA, mental health problems account for a significant proportion of admissions (Seal, Bertenthal, Miner, Sen, & Marmar, 2007). The VHA is a fully integrated health system, so that complete outpatient and inpatient histories are available for our

model. Other large systems, such as Kaiser Permanente, the Mayo Clinic, and large medical and medical insurance systems, such as the University of Pittsburgh Medical Center, should be able to access similar historical information. If reliable similar data, supplied by patients or other providers, can be obtained at the time of admission, then the features we used for prediction could also be used in a non-integrated environment. The more incomplete the available histories, and the less reliable patient-supplied data are, the more challenging it might be to successfully apply our modeling approach. 5. Conclusions The advantage of Cubist tree is the interpretability of its predictions, allowing an examination of the logic that stands behind the classification rules applied to the Length of Stay (LOS) data, as well as enabling an understanding of the underlying factors which could affect the LOS. The Cubist error level seems to be nonuniformly distributed among the extracted rules, with greater prediction error for cases that are characterized with more frequent admissions and for those that are characterized by greater length of previous hospital stays. Mapping the cases into a higher dimensional space, by using a Radial Basis Function (RBF) kernel, enables a highly accurate separation of cases with lower Cubist error from those cases with higher Cubist error, by using a Support Vector Machines (SVM) algorithm. Acknowledgments This work was supported by the U.S. Department of Veterans Affairs, through master contract numbers VA244-13-C-0581 and VA240-14-D-0038 with the University of Pittsburgh. This work is an outcome of a continuing partnership between the Katz Graduate School of Business and the Pittsburgh Veterans Engineering Resource Center (VERC). Inpatient admissions data were pulled from the VA corporate data warehouse by Dr. Youxu C. Tjader. We thank the RRTICT project group of the Pittsburgh Veterans Engineering Resource Center (VERC) for helpful discussions. References Aggarwal, S., & Gupta, V. (2014). Demographic parameters related to 30-day readmission of patients with congestive heart failure: Analysis of 2,536,439 hospitalizations. International journal of cardiology, 176, 1343–1344. Agrawal, R., & Srikant, R. (1994). Fast algorithms for mining association rules. In Proc. 20th int. conf. very large data bases, VLDB: 1215 (pp. 487–499). Breiman, L., Friedman, J., Olshen, R., & Stone, C. (1984). Classification and regression trees. Cole advanced books Software. ISBN. Monterey, CA: Wadsworth Brooks. Brin, S., Motwani, R., Ullman, J. D., & Tsur, S. (1997). Dynamic itemset counting and implication rules for market basket data. ACM SIGMOD Record, 26, 255–264 ACM. Carter, E. M., & Potts, H. W. (2014). Predicting from an electronic patient record system: A primary total knee replacement example. BMC medical informatics and decision making, 14 1. Chang, C.-C., & Lin, C.-J. (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2 27. Cosgrove, S. E. (2006). The relationship between antimicrobial resistance and patient outcomes: Mortality, length of hospital stay, and health care costs. Clinical Infectious Diseases, 42, S82–S89. Croft, J. B., Giles, W. H., Pollard, R. A., Casper, M. L., Anda, R. F., & Livengood, J. R. (1997). National trends in the initial hospitalization for heart failure. Journal of the American Geriatrics Society, 45, 270–275. Dunham, M. H., Xiao, Y., Gruenwald, L., & Hossain, Z. (2001). A survey of association rules Retrieved January, 5, 2008. Faddy, M., Graves, N., & Pettitt, A. (2009). Modeling in hospital and other right skewed data: Comparison of phase-type, gamma and log-normal distributions. Value in Health, 12, 309–314. Faddy, M., & McClean, S. (1999). Analysing data on lengths of stay of hospital patients using phase-type distributions. Applied Stochastic Models in Business and Industry, 15, 311–317. Hamner, J. B., & Ellison, K. J. (2005). Predictors of hospital readmission after discharge in patients with congestive heart failure. Heart & Lung: The Journal of Acute and Critical Care, 34, 231–239. Hidber, C. (1999). Online association rule mining: 28. ACM.

L. Turgeman et al. / Expert Systems With Applications 78 (2017) 376–385 Ho, K., Anderson, K. M., Kannel, W. B., Grossman, W., & Levy, D. (1993). Survival after the onset of congestive heart failure in Framingham Heart Study subjects. Circulation, 88, 107–115. Houthooft, R., Ruyssinck, J., van der Herten, J., Stijven, S., Couckuyt, I., Gadeyne, B., et al. (2015). Predictive modelling of survival and in critically ill patients using sequential organ failure scores. Artificial Intelligence in Medicine, 63, 191–207. IBM (2016). SPSS Modeler algorithms guide. IBM Corporation. Islam, T., O’Connell, B., & Lakhan, P. (2013). Hospital readmission among older adults with congestive heart failure. Australian Health Review, 37, 362–368. Johansen, H., Strauss, B., Arnold, J., Moe, G., & Liu, P. (2003). On the rise: The current and projected future burden of congestive heart failure hospitalization in Canada. The Canadian Journal of Cardiology, 19, 430–435. John, M., Hoernig, S., Doehner, W., Okonko, D. D., Witt, C., & Anker, S. D. (2005). Anemia and inflammation in COPD. CHEST Journal, 127, 825–829. Kass, G. V. (1980). An exploratory technique for investigating large quantities of categorical data. Applied statistics, 119–127. Kessler, C. S., Bhandarkar, S., Casey, P., & Tenner, A. (2011). Predicting patient patterns in veterans administration emergency departments. Western Journal of Emergency Medicine, 12. Kossovsky, M. P., Sarasin, F. P., Perneger, T. V., Chopard, P., Sigaud, P., & Gaspoz, J.-M. (20 0 0). Unplanned readmissions of patients with congestive heart failure: Do they reflect in-hospital quality of care or patient characteristics. The American journal of medicine, 109, 386–390. Krumholz, H. M., Chen, Y.-T., Wang, Y., Vaccarino, V., Radford, M. J., & Horwitz, R. I. (20 0 0). Predictors of readmission among elderly survivors of admission with heart failure. American heart journal, 139, 72–77. Krumholz, H. M., Parent, E. M., Tu, N., Vaccarino, V., Wang, Y., Radford, M. J., et al. (1997). Readmission after hospitalization for congestive heart failure among Medicare beneficiaries. Archives of Internal Medicine, 157, 99–104. Larose, D. T. (2005). k-nearest neighbor algorithm. discovering knowledge. In Data: An introduction to data mining (pp. 90–106). Liu, P., Lei, L., Yin, J., Zhang, W., Naijun, W., & El-Darzi, E. (2006). Healthcare data mining: Predicting inpatient length of stay. Mahadevan, D., Challand, C., & Keenan, J. (2010). Revision total hip replacement: Predictors of blood loss, transfusion requirements, and length of hospitalisation. Journal of Orthopaedics and Traumatology, 11, 159–165. Marshall, A., & McClean, S. (2003). Conditional phase-type distributions for modelling patient in hospital. International Transactions in Operational Research, 10, 565–576. Mascitelli, L., & Pezzetta, F. (2005). Anemia and COPD. CHEST Journal, 128, 3084. McGrory, C. A., Pettitt, A. N., & Faddy, M. J. (2009). A fully Bayesian approach to inference for Coxian phase-type distributions with covariate dependent mean. Computational Statistics & Data Analysis, 53, 4311–4321. McMurray, J., McDonagh, T., Davie, A., Cleland, J., Francis, C., & Morrison, C. (1998). Should we screen for asymptomatic left ventricular dysfunction to prevent heart failure. European Heart Journal, 19, 842–846. Murphy, K. P. (2006). Naive Bayes classifiers. University of British Columbia. Parmley, W. W. (1989). Pathophysiology and current therapy of congestive heart failure. Journal of the American College of Cardiology, 13, 771–785. Pendharkar, P. C., & Khurana, H. (2014). Machine learning techniques for predicting hospital in Pennsylvania federal and specialty hospitals. International Journal of Computer Science and Applications, 2, 45–56.

385

Philbin, E. F., & DiSalvo, T. G. (1998). Influence of race and gender on care process, resource use, and hospital-based outcomes in congestive heart failure. The American journal of cardiology, 82, 76–81. Philbin, E. F., Rocco, T. A., Jr, Lynch, L. J., Rogers, V. A., & Jenkins, P. (1997). Predictors and determinants of hospital in congestive heart failure in ten community hospitals. The Journal of Heart and Lung Transplantation: The official publication of the International Society for Heart Transplantation, 16, 548–555. Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10, 61–74. Quinlan, J. R. (1992). Learning with continuous classes. In 5th Australian joint conference on artificial intelligence: 92 (pp. 343–348). Quinlan, J. R. (1993). Combining instance-based and model-based learning. In Proceedings of the tenth international conference on machine learning (pp. 236–243). Quinlan, J. R. (2014). C4. 5: Programs for machine learning. Elsevier. Robinson, G. H., Davis, L. E., & Leifer, R. P. (1966). Prediction of hospital. Health Services Research, 1, 287. Schocken, D. D., Arrieta, M. I., Leaverton, P. E., & Ross, E. A. (1992). Prevalence and mortality rate of congestive heart failure in the United States. Journal of the American College of Cardiology, 20, 301–306. Seal, K. H., Bertenthal, D., Miner, C. R., Sen, S., & Marmar, C. (2007). Bringing the war back home: Mental health disorders among 103 788 US veterans returning from Iraq and Afghanistan seen at Department of Veterans Affairs Facilities. Archives of Internal Medicine, 167, 476–482. Tanuja, S., Acharya, D. U., & Shailesh, K. (2011). Comparison of different data mining techniques to predict hospital. Journal of Pharmaceutical and Biomedical Sciences, 7. Turgeman, L., May, J., Ketterer, A., Sciulli, R., & Vargas, D. (2015). Identification of readmission risk factors by analyzing the hospital-related state transitions of congestive heart failure (CHF) patients. IIE Transactions on Healthcare Systems Engineering, 5, 255–267. Vapnik, V. N. (1998). Statistical learning theory. New York: Wiley. Verburg, I. W., de Keizer, N. F., de Jonge, E., & Peek, N. (2014). Comparison of regression methods for modeling intensive care. PloS one, 9 e109684. Vinson, J. M., Rich, M. W., Sperry, J. C., Shah, A. S., & McNamara, T. (1990). Early readmission of elderly patients with congestive heart failure. Journal of the American Geriatrics Society, 38, 1290–1295. Wexler, D. J., Chen, J., Smith, G. L., Radford, M. J., Yaari, S., Bradford, W. D., et al. (2001). Predictors of costs of caring for elderly patients discharged with heart failure. American Heart Journal, 142, 350–357. Whellan, D. J., Zhao, X., Hernandez, A. F., Liang, L., Peterson, E. D., Bhatt, D. L., et al. (2011). Predictors of hospital in heart failure: Findings from get with the guidelines. Journal of Cardiac Failure, 17, 649–656. Wong, N. D., Lopez, V. A., L’Italien, G., Chen, R., Kline, S. E. J., & Franklin, S. S. (2007). Inadequate control of hypertension in US adults with cardiovascular disease comorbidities in 20 03-20 04. Archives of Internal Medicine, 167, 2431–2436. Zheng, F., Cammisa, F. P., Sandhu, H. S., Girardi, F. P., & Khan, S. N. (2002). Factors predicting hospital stay, operative time, blood loss, and transfusion in patients undergoing revision posterior lumbar spine decompression, fusion, and segmental instrumentation. Spine, 27, 818–824.