READMIT: A clinical risk index to predict 30-day readmission after discharge from acute psychiatric units

Journal of Psychiatric Research 61 (2015) 205e213

Contents lists available at ScienceDirect

Journal of Psychiatric Research journal homepage: www.elsevier.com/locate/psychires

READMIT: A clinical risk index to predict 30-day readmission after discharge from acute psychiatric units* Simone N. Vigod a, b, c, d, *, Paul A. Kurdyak c, d, e, Dallas Seitz f, Nathan Herrmann d, g, Kinwah Fung c, Elizabeth Lin c, d, e, Christopher Perlman h, Valerie H. Taylor a, b, c, d, Paula A. Rochon a, b, c, d, Andrea Gruneir a, b, c, d, i a

Women's College Hospital, 76 Grenville Street, Toronto, Ontario, Canada Women's College Research Institute, 790 Bay Street, Toronto, Ontario, Canada Institute for Clinical Evaluative Sciences, 2075 Bayview Avenue, Toronto, Ontario, Canada d University of Toronto, 27 King's College Circle, Toronto, Ontario, Canada e Centre for Addiction and Mental Health, 250 College Street, Toronto, Ontario, Canada f Queens University, 99 University Avenue, Kingston, Ontario, Canada g Sunnybrook Health Sciences Centre, 2075 Bayview Avenue, Toronto, Ontario, Canada h University of Waterloo, 200 University Avenue West, Waterloo, Ontario, Canada i University of Alberta, 6-40 University Terrace, Edmonton, Alberta, Canada b c

a r t i c l e i n f o

a b s t r a c t

Article history: Received 11 August 2014 Received in revised form 28 October 2014 Accepted 4 December 2014

Our aim was to create a clinically useful risk index, administered prior to discharge, for determining the probability of psychiatric readmission within 30 days of hospital discharge for general psychiatric inpatients. We used population-level sociodemographic and health administrative data to develop a predictive model for 30-day readmission among adults discharged from an acute psychiatric unit in Ontario, Canada (2008e2011), and converted the final model into a risk index system. We derived the predictive model in one-half of the sample (n ¼ 32,749) and validated it in the other half of the sample (n ¼ 32,750). Variables independently associated with 30-day readmission (forming the mnemonic READMIT) were: (R) Repeat admissions; (E) Emergent admissions (i.e. harm to self/others); (D) Diagnoses (psychosis, bipolar and/or personality disorder), and unplanned Discharge; (M) Medical comorbidity; (I) prior service use Intensity; and (T) Time in hospital. Each 1-point increase in READMIT score (range 0e41) increased the odds of 30-day readmission by 11% (odds ratio 1.11, 95% CI 1.10e1.12). The index had moderate discriminative capacity in both derivation (C-statistic ¼ 0.631) and validation (C-statistic ¼ 0.630) datasets. Determining risk of psychiatric readmission for individual patients is a critical step in efforts to address the potentially avoidable high rate of this negative outcome. The READMIT index provides a framework for identifying patients at high risk of 30-day readmission prior to discharge, and for the development, evaluation and delivery of interventions that can assist with optimizing the transition to community care for patients following psychiatric discharge. © 2014 Elsevier Ltd. All rights reserved.

Keywords: Psychiatric readmission Psychiatric epidemiology Risk index

1. Introduction Worldwide, almost 1 in 7 individuals hospitalized for psychiatric reasons are readmitted within 30 days of discharge (Leslie and

* These results were presented at the Canadian Psychiatric Association meeting in Toronto, Ontario in September 2014. * Corresponding author. Department of Psychiatry, Women's College Hospital, 76 Grenville St. Rm. 7234, Toronto, Ontario, M5S 1B2, Canada. Tel.: þ1 416 323 6400x4080; fax: þ1 416 323 6356. E-mail address: [email protected] (S.N. Vigod).

http://dx.doi.org/10.1016/j.jpsychires.2014.12.003 0022-3956/© 2014 Elsevier Ltd. All rights reserved.

Rosenheck, 2000; Canadian Institute for Health Information and Statistics Canada, 2011; National Association of State Mental Health Program Directors Research Institute, 2012; OECD, 2013). This high readmission rate is a negative outcome from a clinical and public health perspective (Canadian Institute for Health Information and Statistics Canada, 2011), and is highly disruptive to patients and their families. It is also, at least to some extent, an avoidable outcome as supported by evidence that: (1) interventions of varying types have been evaluated in clinical trials and shown to reduce early readmission rates (Vigod et al., 2013a); and (2) some mental health care systems that have developed new

206

S.N. Vigod et al. / Journal of Psychiatric Research 61 (2015) 205e213

organizational and delivery models are demonstrating downward trends in readmission rates (OECD, 2013). To apply interventions efficiently and effectively, however, it is important to be able to identify the individuals who would benefit most. A risk score that quantifies the probability of early readmission for an individual patient could be used in a clinical setting to identify individuals who most require intervention; in a research setting to determine the appropriate target populations for specific interventions designed to reduce early readmission; and in hospital and public policy settings to align resources to areas of greatest need. While many studies have identified risk factors for early psychiatric readmission (Hendryx et al., 2003), few have focused on methods to quantify risk of psychiatric readmission for individuals in a clinical setting. Existing studies focus on specific populations (Gearing et al., 2009) such that their findings are not applicable to the majority of general adult inpatient settings that provide treatment to individuals with a range of diagnoses and needs. Our objective was to derive and validate a clinical risk index that predicts an individual's risk of psychiatric readmission within 30 days of hospital discharge from a general psychiatric inpatient setting. The specific intent of the risk index was for it to be used prior to discharge in any general psychiatric setting to help identify people who are at high risk of readmission so that they can receive services and supports during the hospitalization and postdischarge that may reduce that risk. We had access to a wide array of population-based, health administrative data that can be organized broadly into 4 categories: (1) sociodemographic variables; (2) prior health care utilization; (3) basic clinical and administrative information from a hospital admission; and (4) detailed psychiatric rating scales and metrics administered by clinicians during inpatient psychiatric admission. Accordingly, we systematically evaluated the relative contributions of these categories of information that are increasingly complex to measure on readmission risk prediction. This strategy allowed us to determine the added predictive capacity of including information that is more detailed, but would require greater effort and resources to collect. Therefore, in addition to predicting risk of readmission, we were also able to create a risk index that maximized both risk prediction and feasibility of data collection in a clinical setting. 2. Methods

number. The RPDB also contains age, gender and postal code. Index psychiatric admission information was obtained from the Ontario Mental Health Reporting System (OMHRS). OMHRS contains information on all hospital admissions for adults aged 18 and older admitted to psychiatric inpatient beds in Ontario. OMHRS data are derived from the Resident Assessment Instrument e Mental Health (RAI-MH), a comprehensive clinical assessment tool that is completed within 3 days of admission, at 90-day intervals during the admission (where applicable) and at discharge. The RAI-MH contains information on patient demographics, socioeconomic status (e.g. source of income, place of residence prior to admission, marital status), admission and discharge diagnoses according to Diagnostic and Statistical Manual of Mental Disorder, version IV (DSM-IV) criteria, measures of psychiatric symptoms, substance use, cognition and functional impairment (Hirdes et al., 2000). In inpatient settings, the items have adequate reliability (inter-rater agreement > 80%) and validity (Hirdes et al., 2002). Additional information was derived from other linked ICES datasets whose accuracy has been previously described (Williams et al., 1996). The Ontario Health Insurance Program (OHIP) database contains information on all physician outpatient and inpatient visits including procedures and diagnostic codes. The Canadian Institutes of Health Information - Discharge Abstract Database (CIHI-DAD) contains information on all non-OMHRS acute care hospitalizations and the CIHI National Ambulatory Care Reporting System (NACRS) contains information on all emergency room visits. The study received research ethics board approvals from Women's College Hospital and Sunnybrook Health Sciences Centre in Toronto, Ontario (ICES logged study: 2013 0904 301 000). 2.3. Cohort We identified all individuals aged 18 or older who were discharged from an Ontario APU between April 1, 2008 and March 31, 2011 using the OMHRS dataset, selecting an individual's first discharge during the study period as the index admission. Only individuals who were hospitalized for 72 h were included in our cohort because the characteristics of individuals hospitalized for shorter stays differ substantially from the characteristics of individuals who are hospitalized for at least 72 h, and the number of mandatory RAI-MH assessment variables is much lower for admissions <72 h than for longer-term admissions (Urbanoski et al., 2012).

2.1. Study design and setting 2.4. Outcome We performed a population-based cohort study using sociodemographic and health administrative data to derive and validate a clinical risk index predicting 30-day readmission among individuals discharged from acute psychiatric units (APUs) in Ontario, Canada between 2008 and 2011. Our health care system is a universal, government-funded insurance program that covers all Ontario residents and includes mental health care. Acute psychiatric units are general psychiatric units that provide the majority of inpatient psychiatric services for individuals with mental illness in Ontario. 2.2. Data sources Linked population health administrative databases at the Institute for Clinical Evaluative Sciences (ICES) in Toronto, Ontario were used for the current project. ICES is an independent, non-profit research organization that holds population-level data, including administrative data, for the purpose of evaluating health care services and their effectiveness in Ontario. Patient level records in these data are anonymously linked to each other through a unique identifier (ICES Key Number) using the Registered Persons Database (RPDB) for every Ontario resident with an assigned health care

The primary outcome was psychiatric readmission to any hospital in Ontario within 30 days of discharge from the index admission (Vigod et al., 2013b). The 30-day readmission outcome is used worldwide as a benchmark for “early readmission”, in that it is considered to be an indicator that reflects how patient needs are met in terms of coordination and continuity of services in the immediate period after discharge from hospital (Canadian Institute for Health Information and Statistics Canada, 2011; OECD, 2013). Readmissions that occur later are also of interest. However, predictors of readmissions that occur later may be different from those of early readmission, in that they are more likely to reflect how well chronic mental health care needs are met in the community and/or the course of the illnesses themselves (Canadian Institute for Health Information and Statistics Canada, 2011). In this study, the outcome was defined as either: (1) any readmission captured in the OMHRS dataset; or (2) any admission to an acute care hospital (as identified in CIHI-DAD) where the most responsible diagnosis was for a mental health condition (ICD-10CA: F00-F99) within 30 days from discharge. The latter includes psychiatric admissions to intensive care units after suicide attempts

S.N. Vigod et al. / Journal of Psychiatric Research 61 (2015) 205e213

and psychiatric admissions to overflow medical beds, which are not captured in the OMHRS dataset; these correspond to about 20% of psychiatric admissions in Ontario (Source: ICES unpublished data). Only unplanned admissions were counted to avoid planned hospitalizations (e.g. maintenance electroconvulsive therapy or other reasons). 2.5. Predictor variables The sociodemographic category of variables included age, sex and community size (rural <10,000; urban  10,000), as well as marital status, living situation and type of residence, source of income, and educational attainment. The prior health service utilization category included both mental health and non-mental health variables. Mental health service use variables measured prior to the index admission included the number of previous psychiatric hospital admissions (lifetime and within the 2 years prior to the index admission) as well as the number of psychiatric emergency department and outpatient mental health visits within the year prior to admission (Steele et al., 2004). We estimated overall medical comorbidity using a Charlson score (Deyo et al.,1992; Quan et al., 2005). The category of basic clinical and administrative information collected during the index admission comprised multiple variables: whether or not criteria were met for involuntary admission, whether there were concerns about harm to self, harm to others or inability to care for self, length of stay, whether discharge was planned or unplanned (i.e. against medical advice), DSM-IV psychiatric diagnoses and global assessment of functioning at admission and discharge (GAF score) utilized in the DSM-IV classification system (American Psychiatric Association, 2000). The final category of predictor variables included detailed rating scales and metrics contained in the RAI-MH. These RAI-MH scales and clinical assessment protocols (CAPs) are calculated at admission and discharge and reflect symptom, functional and behavioural domains (American Psychiatric Association, 2000; Hirdes et al., 2000; Hirdes et al., 2002). We considered each of 5 RAI-MH symptom scales: Depression Rating Scale (DRS), Positive Symptom Scale (PSS), Negative Symptom Scale (NSS), Mania Symptom Scale (MSS) and Cognitive Performance Scale (CPS). We summed 6 data elements representing symptoms of anxiety and considered additional RAI-MH scales that assess function: Activities of Daily Living Hierarchy Scale (ADLH), Aggressive Behaviour Scale, Risk of Harm to Others Scale; Severity of Self-harm Scale, and the Self-Care Index. We considered each of the twenty-one RAI-MH clinical assessment protocols (CAPs) that are derived from the data elements and scales contained in the RAI-MH, and are intended as feedback for the clinician in terms of areas of vulnerability for the patient that could be actioned to improve outcomes. Examples include issues related to smoking, sleep disturbance, selfcare, housing problems and risk of aggression. 2.6. Multivariable model derivation We described the distribution of possible predictor variables for individuals who were, and were not readmitted within 30 days of discharge using summary statistics. Multivariable logistic regression models were fit to determine the best predictive model for 30day psychiatric readmission. To build and internally validate the predictive model, we used a split-sample method in which the dataset was randomly divided in two. One part was used for building the model (the “derivation” dataset) and the other was used for validating the model (the “validation” dataset) (Altman and Royston, 2000). We created a series of 4 models by sequentially adding variables in order to identify those that best predicted the risk of 30-day psychiatric readmission (Hosmer and Lemeshow, 2000). This was done such that the models moved from the most

207

basic sociodemographic predictive variables (Model 1) to more comprehensive, considering the prior health service use variables (Model 2), the basic clinical and administrative data relevant to the index admission (Model 3), and then the specific RAI-MH scales and clinical assessment protocols (i.e. for symptom scales and measures of functional ability) that would be more complex to collect and standardize, possibly requiring a clinical information system (Model 4). We chose this strategy because we wanted to prioritize variables that could be collected most easily in a clinical setting, while still maximizing the predictive capacity of the model. Decisions about variable inclusion were made using the log likelihood test to determine if the addition of a risk factor, or a group of risk factors improved the model's overall ability to predict the outcome (readmission) compared to a simpler model. Specifically, chi-square tests were performed to test if the log-likelihood ratio was significant after a variable was dropped or added to the model. Because variables that were not previously significant might become important in the presence of others, once a variable was dropped, the effect of omitting each of the remaining variables in turn was examined. Dropped variables were therefore added to the model, one at a time, and any that changed the log-likelihood ratio significantly were retained in the model. This process went on until adding or removing a variable did not result in any significant loglikelihood ratio. For each model, we also examined the individual coefficients (Odds Ratios), stability of effect as variables were added, and precision (using 95% Confidence Intervals). This helped identify which variables were consistently strong predictors of readmission, and any potential confounding or collinearity (given the number of measures of patient complexity). Overall model fit was assessed using the HosmereLemeshow goodness-of-fit test and discrimination using the c-statistic. 2.7. Creation and validation of the risk index Once the final logistic regression model was established, we converted it into a risk index system employing the methods described by Sullivan and colleagues (described in detail elsewhere) (Sullivan et al., 2004). Briefly, this involved: (1) obtaining the estimates of the regression coefficients of the multivariable logistic regression model; (2) organizing the risk factors into categories to determine a reference value for each risk factor (for continuous variables we examined mid-points and distribution to determine how to generate appropriate categories); (3) determining the distance of each category from the reference category (in regression units); and (4) computing a point value for every category, where the reference category is assigned a point value of 0. We then generated the probability (or risk) for 30-day readmission for each score of the risk index. To demonstrate the ability of the model to discriminate between those who were and were not readmitted at 30 days post-discharge, we generated a C-statistic (and 95% confidence interval) for both the derivation and validation samples. To determine the calibration of the risk index, we described the expected and observed probabilities of readmission in both derivation and validation datasets. If the risk score were adequately calibrated to function in the derivation and validation samples, the observed probabilities of readmission would be similar to the expected values. All analyses were performed using SAS version 9.2 for UNIX (SAS Institute, Cary NC). 3. Results 3.1. Sample characteristics There were 65,789 index admissions during the study period and the risk of 30-day readmission was 9.19% (N ¼ 6044

208

S.N. Vigod et al. / Journal of Psychiatric Research 61 (2015) 205e213

Table 1 Selected socio-demographic, health service use and admission characteristics for 32,749 unique individuals in the derivation sample who were discharged from an acute psychiatric unit in Ontario (2008e2011), by 30-day readmission status, presented using N (%) unless otherwise specified. Variable Sociodemographic Sex Age in years, Median (IQR) Community size Neighbourhood income quintile Income source Education status Marital status Living situation Health service prior to index admission Lifetime psychiatric admissions Psychiatric admissions within 2 years Psychiatric ED visit within 1 yeara Self-injury within 1 year Community mental health within 1 year Psychiatrist visit within 1 year Charlson score within 2 years Index hospitalization Primary diagnosis (DSM-IV-TR)

Value

Readmitted (n ¼ 3022)

Not readmitted (n ¼ 29,747)

Female

1503 (50.1) 41 (28e53) 2671 (89.0) 936 (31.2) 643 (21.4) 788 (26.2) 1564 (52.1) 833 (27.7) 90 (3.0)

15,159 (51.0) 44 (31e55) 26,376 (88.7) 8460 (28.4) 7953 (26.7) 7454 (25.1) 14,899 (50.1) 9850 (33.1) 575 (1.6)

861 (28.6) 305 (10.2) 262 (8.7) 688 (22.9) 1665 (55.5) 1555 (51.8) 85 (2.8)

5569 (18.7) 1752 (5.9) 1823 (6.1) 6443 (21.7) 14,404 (48.4) 13,414 (45.1) 860 (2.9)

315 (10.5) 907 (30.2) 740 (24.7) 454 (15.6) 690 (23.0) 376 (12.5) 605 (20.2) 1572 (52.4) 1247 (41.5) 60 (50e65) 13 (7e22) 2642 (88.8)

3355 (14.6) 7093 (23.8) 8601 (28.9) 3781 (12.7) 5949 (20.0) 2840 (9.5) 4847 (16.3) 14,374 (48.3) 10,491 (35.3) 60 (50e65) 15 (8e28) 27,534 (92.6)

Urban (>10,000) Lowest Employed Disability or social assistance  high school Married/partner Homeless 4 3

3þ Alcohol or Substance Use Disorder Psychotic Disorder Major Depression Bipolar Disorder Other

Personality disorder Threat to others on admission Threat to self on admission Inability to care for self on admission GAF Score at discharge, Median (IQR) Length of stay in days, Median (IQR) Planned and regular discharge

GAF ¼ Global Assessment of Function; 95% CI ¼ 95% Confidence Interval; IQR ¼ Interquartile Range; ED ¼ Emergency Department; DSM-IV-TR ¼ Diagnostic and Statistical Manual for Mental Disorders e IV, Text Revision. a Without hospital admission.

readmissions). This was the highest risk period for readmissions within the first year after discharge, although readmissions continued to occur consistently thereafter, reaching 18.6% by 120 days; and 30.5% at 1 year post-discharge. The 30-day readmission rate varied minimally between years (8.42%e10.0%), and the patient demographic and clinical characteristics were similar in each fiscal year (data available upon request). About half of the cohort was female and the average age at admission was 44 years. Almost all (89%) lived in urban areas, with about one-quarter employed at the time of admission. Close to half of the sample (45%) had never been married, and 1.7% were homeless at discharge. The most common primary diagnoses during the index admission were major depressive disorder and psychotic disorders, accounting for >50% of all admissions, followed by bipolar disorder (~13%). Slightly less than 20% of the sample had been hospitalized for psychiatric reasons more than four times in their lifetime.

admission (i.e. excluding the RAI-MH symptom scale and CAP variables) resulted in improvement in discriminative capacity (Model 3: 12 variables, C-statistic ¼ 0.63, HL p ¼ 0.868). Consideration of the RAI-MH symptom scale and CAP variables resulted in a significantly improved -2LL (p ¼ 0.002), but the discriminative capacity increased only slightly (Model 4, 19 variables, Cstatistic ¼ 0.65, HL p ¼ 0.305) (Table 3). For variables common to both Model 3 and Model 4, the odds ratios were stable when moving from Model 3 to Model 4. As such, we determined that the increased resource capacity of Model 4 with 19 variables (7 of which were rating scales or clinical assessment protocols, CAPs) was not advantageous where the objective was to create a risk index that could be easily computed in most health care contexts, and chose Model 3 as the final logistic regression model with which to create the risk index. 3.3. Creation and validation of the risk index

3.2. Logistic regression models The derivation cohort was used to develop the logistic regression model and the risk index (N ¼ 32,749). Comparison of characteristics for individuals in the derivation cohort who were readmitted at 30 days versus those who were not is presented in Tables 1 and 2. In the logistic regression models, predictive capacity was poor when only socio-demographic characteristics were considered (Model 1: 3 variables, C-statistic ¼ 0.55, HosmereLemeshow (HL) p ¼ 0.945) and improved only slightly when prior health service use variables were considered (Model 2: 6 variables, C-statistic ¼ 0.58, HL p ¼ 0.582) (See Table 3). Adding basic clinical and administrative information from the index

We converted Model 3 into a risk index, and created the acronym “READMIT” as a useful mnemonic for remembering the variables in the risk index: history of repeat admissions, (R); emergent nature of the index admission (harm to self, harm to others, inability to care for self), (E); diagnoses of psychosis, bipolar disorder and personality disorder; and unplanned discharge, (D); medical comorbidity, (M); intensity of outpatient and emergency department use prior to admission, (I); and time in hospital, (T) (Table 4). The total number of possible points (or READMIT score) ranged from 0 to 41. The READMIT index was strongly associated with the outcome, where a 1-point increase in the READMIT score increased the odds

S.N. Vigod et al. / Journal of Psychiatric Research 61 (2015) 205e213

209

Table 2 Resident assessment instrument for mental health (RAI-MH) clinical characteristics at discharge for 32,749 unique individuals in the derivation sample who were discharged from an acute psychiatric unit in Ontario (2008e2011), by 30-day readmission status, presented using N (%) unless otherwise specified. Variable

Readmitted (n ¼ 3022)

Not readmitted (n ¼ 29,747)

Serious life event within past yeara (N, %) Assessment scales (Mean, 95% CI) Aggressive behaviour scale (range 0e12) ADL hierarchy scale (range 0e6) Cognitive performance scale (range 0e6) Depressive rating scale (range 0e15) Mania scale (range 0e18) Positive symptoms scale (short, range 0e12) Negative symptom scale Risk of harm to others scale (range 0e6) Self-care index (range 0e6) Severity of self-harm scale (range 0e6) Selected CAPs (N, % high risk)b Criminal activity Control interventions Education and employment Personal finances Harm to others Interpersonal conflict Medication adherence Pain Rehospitalization (90-days) Self care Suicidality and self-harm Sleep disturbance Social relationships Substance use

1378 (45.9)

14,393 (48.4)

0.7 0.2 0.5 1.8 1.9 1.1

0.4 0.2 0.4 1.5 1.1 1.9

(0.6e0.7) (0.2e0.2) (0.5e0.5) (1.7e1.9) (1.8e2.1) (1.0e1.2)

(0.3e0.4) (0.2e0.2) (0.4e0.5) (1.5e1.5) (1.1e1.1) (1.8e2.0)

1.3 (1.2e1.4) 1.2 (1.2e1.3) 1.8 (1.7e1.9)

1.0 (1.0e1.1) 0.9 (0.9e0.9) 1.6 (1.6e1.6)

823 (27.4) 95 (3.2) 603 (20.1) 337 (11.2) 152 (5.1) 342 (11.4) 593 (19.8) 166 (5.5) 166 (5.5) 54 (1.8) 275 (9.2) 13 (0.4) 120 (4.0) 1031 (34.3)

6756 (22.7) 506 (1.7) 6576 (22.1) 3422 (11.5) 1139 (3.8) 1954 (6.6) 4232 (14.2) 842 (2.8) 845 (2.8) 310 (1.0) 2713 (9.1) 136 (0.5) 1154 (3.9) 9739 (32.7)

a Serious life events include: Serious accident or physical impairment, Distressed about health of another person, Death of a close family member or friend, Child custody issues, Conflict-laden or severed relationship, Failed or dropped out of an education program, Major loss of income, Review hearings, Immigration, Lived in war zone, Witness to severe incident, Victim of Crime, Sexual Assault/Abuse, Physical Assault/Abuse, Emotional Abuse, Parental Abuse of Alcohol/Drugs. b For CAPs, no data available to calculate “Social Support” or “Social systems for discharge” CAP; CAP ¼ Clinical Assessment Protocol; 95% CI ¼ 95% Confidence Interval.

of 30-day readmission by 11% (odds ratio 1.11, 95% CI 1.10e1.12) and the relationship appeared to be linear (Fig. 1). The expected probability of 30-day readmission using the READMIT index ranged from 2% at a score of 0, to 49% at a score of 41, and the expected probability of readmission was within the 95% confidence interval of the observed probability for all scores in the derivation and validation samples, indicating adequate calibration (Table ST1). Very few individuals had scores at the extremes of the risk index, however, (i.e. < 5 or >33), resulting in wide confidence intervals around the estimates of the risk of readmission at these scores (Fig. 1 and Table ST1). The index had moderate discriminative capacity in both derivation (C-statistic ¼ 0.631) and validation (Cstatistic ¼ 0.630) datasets. 4. Discussion Using comprehensive population-level socio-demographic and health administrative data, we derived and internally validated a clinical risk index, the READMIT index, to quantify 30-day readmission risk after psychiatric hospitalization. To our knowledge, READMIT is the first published clinical risk score that can be used to quantify the risk of 30-day readmission at the patient-level for general psychiatric inpatients. The variables used to derive the score can be readily collected by a clinical team on an inpatient setting, or by providers caring for patients directly post-discharge. The variables in the READMIT risk score are highly consistent with existing literature regarding mental health re-hospitalization. A comprehensive review of 13 studies found that having one or more previous admissions was the most consistent predictor of readmission between 30 and 90 days after discharge (Durbin et al., 2007). Younger age, forensic history, low family support, severe mental illness, acuity of symptoms at admission or discharge, and discharge against medical advice were also identified as risk factors

(Durbin et al., 2007). Our study additionally identified medical comorbidity as a risk factor, perhaps reflecting that the overall complexity of the patient is important for successful transition to community care post-discharge. It was surprising that the detailed symptom scales and clinical assessment protocols from the RAI-MH in Model 4 only added marginally to Model 3 in terms of discriminative capacity. It is possible that the added variables in Model 4 (i.e. positive, negative and aggressive behaviour symptoms, use of restraint interventions and problems with self-care and interpersonal conflict) are measuring constructs that are similar to the variables measured by Model 3 such as primary diagnosis (e.g. psychotic disorder) and emergent nature of the admission (e.g. harm to self, harm to others, inability to care for self). While we have described the performance of READMIT model as “moderate”, its discriminative capacity is similar to that of other risk indices used in prediction of health and health care outcomes. Examples include the Framingham Risk Score (c-statistic ¼ 0.63) (Orford et al., 2002) and other medical models, including a recently published index to predict early death or unplanned readmission after discharge from hospital to the community for medical inpatients (van Walraven et al., 2010). It is not surprising that our model, like these other models, has only moderate discriminative capacity because events that occur after discharge can (and hopefully will) modify risk. The READMIT index allows for quantification of risk such that steps can be taken to reduce the chances that the negative outcome will occur. In clinical practice, the idea would be for clinicians to perform a detailed needs assessment for high-risk patients to identify unmet needs and inform follow-up care. In our systematic review of transitional interventions designed to reduce early psychiatric readmission, we found that such needs assessments had a very large effect on reducing readmission rates, in particular for patients at high risk of readmission (Vigod et al., 2013a).

210

S.N. Vigod et al. / Journal of Psychiatric Research 61 (2015) 205e213

Table 3 Logistic regression Models 1, 2, 3 and 4 using the derivation cohort (n ¼ 32,749), with contributing variables presented using odds ratios (OR) and 95% confidence intervals (CI). Variable Sociodemographic Age in years Education

Marital status Prior health service use Lifetime MH admissions

Outpatient MH visits (1 year)

MH emergency visits (1 year) Charlson Score (2 years) Index hospitalization Threat to self Threat to others Unable to care for self Primary diagnosis

Personality disorder Planned and regular discharge Length of stay (days)

RAI-MH clinical characteristics Stressful life events (1 year) Positive symptom scale

Negative symptom scale

Anxiety symptoms

Aggressive behaviour scale Self-care CAP

Interpersonal conflict CAP

Control interventions CAP

Value

Model 1 OR (95%CI)

Model 2 OR (95%CI)

Model 3 OR (95%CI)

Model 4 OR (95%CI)


0.99 1.00 0.87 0.90 1.00 1.18

0.99 1.00 0.91 0.95 NS

(0.99e0.99) (referent) (0.83e1.00) (0.84e1.07)

0.99 (0.99e0.99) NS

0.99 (0.99e0.99) NS

NS

NS

1.00 1.33 1.78 2.20 1.00 1.00 1.16 1.00 1.39 1.05

(referent) (1.22e1.46) (1.57e2.03) (1.95e2.47) (referent) (0.97e1.03) (1.05e1.28) (referent) (1.28e1.52) (1.00e1.10)

(0.99-0.99) (referent) (0.80e0.96) (0.79e1.01) (referent) (1.08e1.29)

None 1,2 or 3 4 to 6 6 or more 0 1 2 or more 0 1 or more

Not in Model 1

No Yes No Yes No Yes Psychosis Substance/Alcohol Depression Bipolar disorder Other No Yes No Yes 29 or more 15e28 10e14 0e9

Not in Model 1 or 2

Not in Model 1, 2 or 3 0 1 2 or more 0 1e3 4 or more 0 1 2 or more 0 1 or more Low risk Medium risk High risk Low risk Medium risk High risk Low risk Medium risk High risk

1.00 1.28 1.62 1.98 1.00 1.19 1.20 1.00 1.39 1.05

(referent) (1.17e1.40) (1.42e1.85) (1.75e2.24) (referent) (1.05e1.35) (1.10e1.33) (referent) (1.27e1.52) (1.00e1.10)

1.00 1.26 1.57 1.91 1.00 1.20 1.22 1.00 1.33 1.06

(referent) (1.15e1.39) (1.37e1.81) (1.67e2.18) (referent) (1.05e1.37) (1.10e1.35) (referent) (1.22e1.45) (1.01e1.11)

1.00 1.13 1.00 1.13 1.00 1.21 1.00 0.66 0.80 0.95 0.87 1.00 1.23 1.00 0.59 1.00 1.38 1.52 1.48

(referent) (1.04e1.22) (referent) (1.02e1.24) (referent) (1.12e1.32) (referent) (0.57e0.76) (0.72e0.90) (0.84e1.07) (0.77e0.98) (referent) (1.10e1.38) (referent) (0.52e0.67) (referent) (1.23e1.55) (1.35e1.72) (1.32e1.66)

1.00 (referent) 1.16 (1.07e1.27) NS 1.00 1.12 1.00 0.76 0.96 1.03 0.92 1.00 1.20 1.00 0.64 1.00 1.36 1.51 1.39

(referent) (1.02e1.22) (referent) (0.65e0.89) (0.85e1.09) (0.90e1.17) (0.80e1.05) (referent) (1.06e1.35) (referent) (0.55e0.75) (referent) (1.21e1.53) (1.34e1.71) (1.22e1.57)

0.98 1.00 1.03 1.22 1.00 0.97 0.96 1.00 1.23 1.23 1.00 1.25 1.00 1.07 1.24 1.00 1.13 1.30 1.00 1.12 1.04

(0.97e1.00) (referent) (0.84e1.26) (1.08e1.36) (referent) (0.87e1.09) (0.85e1.07) (referent) (1.04e1.45) (1.12e1.35) (referent) (1.11e1.42) (referent) (0.92e1.24) (1.11e1.39) (referent) (1.01e1.26) (1.12e1.50) (referent) (0.96e1.30) (0.82e1.33)

NS ¼ Not significant in this model; RAI-MH ¼ Resident Assessment Instrument for Mental Health; CAP ¼ Clinical assessment protocol; MH ¼ Mental Health; MH emergency visits are those that do not result in hospitalization.

A major strength of our study is that we used comprehensive population-level data in a universal health care system to identify predictors of early psychiatric readmission. The data included socio-demographic and health service utilization variables, as well as clinical symptom information. Importantly, the READMIT variables can be readily collected in most jurisdictions, such that this is likely to be a feasible tool for identifying individuals for targeted interventions and could be validated in other settings and

countries. Limitations are that we could not capture the details of the treatment provided during the index admission (e.g. physicianepatient interactions, nor the type and extent of psychotherapy). We were able to capture the level of symptoms and functional capacity at discharge, however (Table 2). Some of these variables (positive and negative symptoms of psychosis, anxiety symptoms) were significant in Model 4, and may reflect the success of treatments (e.g. ECT, medications, psychotherapy) applied in hospital.

S.N. Vigod et al. / Journal of Psychiatric Research 61 (2015) 205e213

211

Table 4 READMIT index (Range 0e41 points) for quantifying risk of 30-day readmission after discharge, with points assigned to values within each of the 12 variables in the index. Risk factor

Variable

Value

Points

“R” e Repeat admission (lifetime)

Number prior to index

“E” e Emergent admission

Threat to others

0 1 to 2 3 to 5 6 or more No Yes No Yes No Yes Older than 94 85 to 94 75 to 84 65 to 74 55 to 64 45 to 54 35 to 44 25 to 34 18 to 24 Alcohol or substance Depression Psychosis or Bipolar Other No Yes No Yes 0 1 to 2 3 or more Less than 2 2 or more None 1 or more More than 28 days 15 to 28 Less than 14

0 2 5 7 0 1 0 1 0 2 0 1 2 3 4 5 6 7 8 0 2 4 3 0 2 0 5 0 1 2 0 2 0 3 0 3 4 41

Threat to self Unable to care for self “A” e Age

Age group (years)

“D” e Diagnosis and discharge

Primary diagnosis

Any personality disorder Unplanned discharge “M” e Medical morbidity

Charlson comorbidity scorea

“I” e Intensity (past year)

Outpatient psychiatrist visits Emergency department visits

“T” e Time in hospital

Length of stay (Days)

Total possible score a

For Charlson comorbidity score, assign 1 point each for previous myocardial infarction, cerebrovascular disease, peripheral vascular disease, diabetes; 2 points each for heart failure, chronic obstructive pulmonary disease, mild liver disease, any tumor (including lymphoma or leukemia); 3 points each for dementia, connective tissue disease; 4 points each for AIDS and moderate or severe liver disease; and 6 points for metastatic solid tumour.

Unfortunately, the addition of these detailed variables added only marginally in terms of predictive capacity. Another potential limitation is that while the READMIT score helps quantify the risk of readmission, it is not perfectly discriminative. This is not surprising given that factors occurring post-discharge are likely to influence early readmission, such as availability and adequacy of communitybased resources to support patients outside of the hospital. In addition, while development of the most discriminative predictive model for readmission merits further attention, it is a different objective from the one that we intended for this project. The intent of the READMIT tool was for it to be a risk index that could be used in any general psychiatric setting to help identify people who are at high risk of readmission so that they can be flagged while still in hospital to receive services upon discharge that may reduce that risk. As such, inclusion of post-hospitalization variables, including region-specific health service availability might improve the predictive capacity of a risk index, but would not be practical to include because they are either not measurable by inpatient personnel administering the index or not likely to be generalizable across regions. There are multiple implications of our findings. First, the READMIT index is a systematic mechanism that could be used to flag atrisk individuals for additional assessment by inpatient, outpatient or transitional care teams. Some of the variables that we identified are potentially modifiable on an inpatient unit (e.g. trying to avoid an

unplanned discharge) and addressing these issues early in an admission may help to reduce the risk of early readmission. Others are not modifiable (e.g. age, diagnosis) but are still important because they contribute to the ability to flag high-risk individuals for added attention. Detailed risk assessments to determine the individual needs of each patient can then be performed to help clinicians guide patients toward specific transitional care pathways post-discharge. As noted above, there is strong evidence that such needs assessments can reduce readmission rates, particularly in patients at high risk for readmission (Vigod et al., 2013a). Varying cut-off scores (or risk levels) can be used to flag the most high-risk patients, depending on the types and availability of outpatient follow-up programs. Second, the READMIT index can be used in a research context to help identify target populations for specific transitional and postdischarge interventions designed to reduce early readmission and optimize transitions from inpatient to community-based care. In this setting, the READMIT score could be evaluated as a moderator of intervention outcome, and allow for a mechanism to ensure that the interventions are delivered to individuals in whom they are most likely to be effective when the research findings are translated into clinical practice. Third, the READMIT index can be used by hospitals and policy-makers to align resources for prevention of readmission with the areas of greatest need at a system-level. To date, the problem of early readmission has been relatively immutable, with high readmission rates that have not decreased

212

S.N. Vigod et al. / Journal of Psychiatric Research 61 (2015) 205e213

Fig. 1. Distribution of risk scores in the derivation cohort (n ¼ 32,749), and observed 30-day readmission rate and 95% confidence interval associated with each point on the risk index.

over time despite quality improvement efforts. We developed the READMIT index in an attempt to create a framework for the development and evaluation of interventions that can assist with optimizing the transition to community care for patients following psychiatric discharge. The ability to identify individuals at high risk of readmission is a critical step in efforts to address the potentially avoidable high rate of early psychiatric readmissions. Role of funder The project was funded by the Ontario Ministry of Health and Long-Term Care (MOHLTC) through the Alternate Funding Program Physician Innovation Fund (Grant title: “Risk of readmission to acute psychiatric units in Ontario”). In addition, this study was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the MOHLTC. The opinions, results, and conclusions reported in this paper are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript. Contributors Dr. Vigod conceptualized and designed the study, planned the analyses, drafted the initial manuscript, and approved the final manuscript as submitted. Dr. Vigod had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Dr. Kurdyak assisted with study design and interpretation, assisted with drafting the initial manuscript, reviewed and revised the manuscript, and approved

the final manuscript as submitted. Dr. Seitz and Dr. Herrmann assisted with study conceptualization, design and interpretation, reviewed and revised the manuscript, and approved the final manuscript as submitted. Ms. Fung assisted with study design, carried out the analyses in conjunction with Dr. Vigod, reviewed and revised the manuscript, and approved the final manuscript as submitted. Dr. Lin, Dr. Perlman and Dr. Rochon assisted with study design and interpretation, reviewed and revised the manuscript, and approved the final manuscript as submitted. Dr. Taylor helped conceptualize the study, reviewed and revised the manuscript, and approved the final manuscript as submitted. Dr. Gruneir assisted with study design and planning of analyses, as well as study conceptualization and interpretation. She reviewed and revised the manuscript and approved the final manuscript as submitted. Conflict of interest statement Dr. Vigod received a one-time consulting fee from MultiDimensional Health Care (MDH) consulting for the development of continuing health care activities related to perinatal mental health in 2011; Dr. Taylor receives funding from Bristol-Myers Squibb for an investigator initiated study and has been a speaker for Astra-Zeneca, Bristol-Myers Squibb, Eli Lilly and Lundbeck. None of the other authors have financial relationships with commercial interests within the preceding 36 months. Acknowledgements Dr. Vigod was supported by a New Investigator Fellowship from the Ontario Mental Health Foundation and by the Shirley Brown Clinician-Scientist award at the Women's College Research Institute during the development of this study.

S.N. Vigod et al. / Journal of Psychiatric Research 61 (2015) 205e213

Appendix A. Supplementary data Supplementary data related to this article can be found at http:// dx.doi.org/10.1016/j.jpsychires.2014.12.003. References Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med 2000;19(4):453e73. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. Arlington, VA: American Psychiatric Association; 2000. Canadian Institute for Health Information and Statistics Canada. Health indicators 2011. 2011. Ottawa. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol 1992;45(6):613e9. Durbin J, Lin E, Layne C, Teed M. Is readmission a valid indicator of the quality of inpatient psychiatric care? J Behav Health Serv Res 2007;34(2):137e50. Gearing RE, Mian I, et al. Developing a risk-model of time to first-relapse for children and adolescents with a psychotic disorder. J Nerv Ment Dis 2009;197(1): 6e14. Hendryx MS, Russo JE, Stegner B, Dyck DG, Ries RK, Roy-Byrne P. Predicting rehospitalization and outpatient services from administration and clinical databases. J Behav Health Serv Res 2003;30(3):342e51. Hirdes JP, Marhaba M, et al. Development of the Resident Assessment InstrumenteMental Health (RAI-MH). Hosp Q 2000;4(2):44e51. Hirdes JP, Smith TF, et al. The Resident Assessment Instrument-Mental Health (RAIMH): inter-rater reliability and convergent validity. J Behav Health Serv Res 2002;29(4):419e32. Hosmer DW, Lemeshow S. Applied logistic regression. New York: Wiley; 2000. Leslie DL, Rosenheck RA. Comparing quality of mental health care for public-sector and privately insured populations. Psychiatr Serv 2000;51(5):650e5.

213

National Association of State Mental Health Program Directors Research Institute. 30-day readmission rates. Using data, changing practice. Falls Church, Virginia: National Association of State Mental Health Program Directors Research Institute Inc.; 2012. p. 1e2. OECD. Unplanned hospital re-admissions for patients with mental disorders. Health at a glance 2013: OECD indicators. 2013. Orford JL, Sesso HD, et al. A comparison of the Framingham and European Society of Cardiology coronary heart disease risk prediction models in the normative aging study. Am Heart J 2002;144(1):95e100. Quan H, Sundararajan V, et al. Coding algorithms for defining comorbidities in ICD9-CM and ICD-10 administrative data. Med Care 2005;43(11):1130e9. Steele LS, Glazier RH, et al. Using administrative data to measure ambulatory mental health service provision in primary care. Med Care 2004;42(10):960e5. Sullivan LM, Massaro JM, et al. Presentation of multivariate data for clinical use: the Framingham study risk score functions. Stat Med 2004;23(10):1631e60. Urbanoski KA, Mulsant BH, Willett P, Ehtesham S, Rush B. Real-world evaluation of the Resident Assessment Instrument-Mental Health assessment system. Can J Psychiatry 2012;57(11):687e95. van Walraven C, Dhalla IA, Bell C, Etchells E, Stiell IG, Zarnke K, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ 2010;182(6):551e7. Vigod SN, Kurdyak PA, Dennis CL, Leszcz T, Taylor VH, Blumberger DM, et al. Transitional interventions to reduce early psychiatric readmissions in adults: systematic review. Br J Psychiatry 2013a;202(3):187e94. Vigod SN, Taylor VH, Fung K, Kurdyak PA. Within-hospital readmission: an indicator of readmission after discharge from psychiatric hospitalization. Can J Psychiatry 2013b;58(8):476e81. Williams JI, O'Brien BJ, Sellors C, Grootendorst P, Willison D. A summary of studies on the quality of health care administrative databases in Canada. In: Goel V, Williams JI, Anderson GM, Blacksterin-Hirsch P, Fooks C, Naylor CD, editors. Patterns of health care in Ontario. The ICES practice atlas. Ottawa: Canadian Medical Association; 1996. p. 339e45.