Design and rationale for the Acute Congestive Heart Failure Urgent Care Evaluation: The ACUTE Study

Trial Designs

Design and rationale for the Acute Congestive Heart Failure Urgent Care Evaluation: The ACUTE Study Douglas S. Lee, MD, PhD, a,b,d Jacques S. Lee, MD, MSc, c,d Michael J. Schull, MD, MSc, a,c,d,e Jeremy M. Grimshaw, MBChB, PhD, f Peter C. Austin, PhD, a and Jack V. Tu, MD, PhD a,c,d Toronto, Canada and Ottawa, Canada

Background Heart failure (HF) is one of the leading reasons for emergency department (ED) visits and hospitalization. However, externally validated risk algorithms for acute prognostication of heart failure patients are not available. Thus, many low-risk patients are hospitalized and some high-risk patients are discharged home, which, in some cases, may lead to death. Objectives The first objective of the ACUTE study is to perform a prospective validation of the Emergency Heart failure Mortality Risk Grade (EHMRG), which is a risk score derived to predict 7-day mortality in the ED setting. The second objective is to independently validate the 30-day model extension of the risk score (EHMRG30-ST) in the same cohort. Study design Patients with HF presenting to the ED will be recruited with a waiver of informed consent as a minimal risk study. The ED physician will calculate the EHMRG 7-day risk score, but treatment decisions will not be influenced by the predictive models. Follow-up will be obtained using probabilistic linkage with the Registered Persons Database of vital statistics, whereby deaths will be ascertained. We will examine mortality rates according to EHMRG and EHMRG30-ST algorithms. We will also compare physician-judged risk estimates, based on clinical judgment alone, with the EHMRG score. Conclusion The ACUTE study will determine if a retrospectively derived algorithm for simultaneous estimation of 7-day and 30-day mortality risk can accurately identify low- and high-risk patients with acute HF and improve upon physician-judged risk estimation. (Am Heart J 2016;181:60-65.)

Background Heart failure (HF) is a leading reason for hospitalization globally, affecting 26 million persons worldwide. 1 The costs of HF have been estimated to exceed $108 billion globally and much of these costs are attributable to hospital-based care. 2 Many patients with acute HF seek care in the emergency department (ED) and it has been estimated that there are approximately 1 million ED visits

From the aInstitute for Clinical Evaluative Sciences, University of Toronto, Toronto, Canada, b Division of Cardiology, Peter Munk Cardiac Centre of the University Health Network and Joint Department of Medical Imaging, University of Toronto, Toronto, Canada, cDepartments of Emergency Medicine and Cardiology, Sunnybrook Health Sciences Centre, d

University of Toronto, Toronto, Canada, Department of Medicine, University of Toronto, Toronto, Canada, eFaculty of Medicine, University of Toronto, Toronto, Canada, and f Ottawa Health Research Institute and University of Ottawa, Ottawa, Canada. RCT# NCT02634762. Submitted December 19, 2015; accepted July 22, 2016. Reprint requests: Douglas S. Lee, MD, PhD, Institute for Clinical Evaluative Sciences, Medicine, University of Toronto, Institute for Clinical Evaluative Sciences, 2075 Bayview Avenue, Room G-106, Toronto, Ontario M4N 3M5, Canada. E-mail: [email protected] 0002-8703 © 2016 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.ahj.2016.07.016

for HF annually in North America. Physicians working in this setting must decide whether to hospitalize or discharge the patient. In most cases, HF patients are admitted to hospital, and importantly, the decision to admit contributes to the high rates of hospitalization of this ambulatory care sensitive condition. 3 Decisions to hospitalize HF patients from the ED are influenced, in part, by the physician's estimate of prognosis. Although decisions to hospitalize patients with HF can roughly separate lower-risk from higher-risk patients, many low-risk patients are hospitalized and some high-risk patients are discharged home, which in some cases, may lead to death. 4 The Emergency Heart failure Mortality Risk Grade (EHMRG) is an acute risk stratification algorithm that calculates the estimated risk of death within 7 days for HF patients who present for care in the ED setting. 5 Seven-day mortality has been used previously in studies of ED outcomes. 6, 7 The EHMRG30-ST is a modification of the base model that provides an estimated risk of death within 30 days, by the addition of a single variable—the presence of ST-depression on the 12-lead electrocardiogram. 8 The models were derived and validated retrospectively in a cohort of 12,000 patients

American Heart Journal Volume 181, Number 0

and demonstrated high discriminative ability to stratify those who were low vs high risk. 5, 8 The ACUTE is a prospective, multicenter validation study of the EHMRG and EHMRG30-ST models conducted in the ED setting. Our study will validate this acute HF risk model system for early mortality outcomes. We will also determine whether physicians in the ED over- or under-estimate risk compared with model predictions in the acute emergency setting.

Methods Objectives The primary objectives of the ACUTE study are the following: 1. To determine if the EHMRG 7-day score stratifies the risk of death within 7 days among HF patients presenting to the ED 2. To determine if physicians' qualitative categorization or quantitative estimation of risk differs from model-derived predictions The secondary objective of this study is: 3. To determine if the EHMRG30-ST model stratifies the risk of death within 30 days among HF patients presenting to the ED A flow diagram of the study processes is depicted in Figure 1.

Patient population Patients with a diagnosis of HF presenting to one of 8 hospital EDs in Ontario will be eligible for inclusion. The clinical diagnosis of HF will be guided by the signs and symptoms of HF, specifically, the Framingham Heart Study criteria for acute HF, which are widely used. 9, 10 Brain natriuretic peptide or N-terminal pro-brain natriuretic peptide may be used to assist with the diagnosis of HF, but performance of this test is not a requirement for study entry. Those with another concomitant primary diagnosis (eg, pneumonia) with secondary HF occurring as a consequence of the former are excluded. We will exclude patients who are (a) dialysis dependent, (b) palliative or have a do-not-resuscitate order before ED presentation, and (c) nonresidents of Ontario who do not have a valid health card number under the Ontario Health Insurance Plan. There will be no restrictions based on sex, language spoken, or ethnicity. This study is registered on clinicaltrials.gov (NCT 02634762). Physician questionnaire At participating sites, ED physicians will respond to a questionnaire and document their plan for the patient before knowing the EHMRG score. Specifically, they will provide their plan to (a) discharge the patient home, (b)

Lee et al 61

refer to specialist for hospital admission, or (c) refer the patient for further specialist evaluation. When the plan is to discharge the patient home, physicians will describe the type of follow-up that they would prefer for the patient. We will obtain insights on the physician's estimation of the probability of 7-day death based on clinical estimation alone (ie, physician-estimated risk [PER]), as a numerical estimate (ie, the probability of the event occurring from 0% to 100%), and the category of risk. 11 To provide anchors for PER estimation, we will show the overall 7-day mortality rates as a benchmark for the province overall and for patients who were admitted vs discharged, using display methods described by Fischoff et al. 12 Similar methods have been used in prior studies that have obtained physician estimates of event probabilities. 13 Both the plan for the patient and PER will be surveyed before EHMRG calculation to ensure that the patient's score does not influence these responses.

Calculation of EHMRG 7-day risk score After completion of the questionnaire (above), physicians will enter the variables required to calculate the EHMRG: age, whether the patient arrived by ambulance, systolic blood pressure, heart rate, oxygen saturation, potassium concentration, creatinine concentration, troponin elevation above the upper threshold of normal, cancer, and use of metolazone before ED arrival. All of these variables will be entered into a programmable case report form, which will calculate the EHMRG risk score after completion of the physician questionnaire. The EHMRG score, probability of 7-day death, and the corresponding risk decile will be calculated based on the entered parameters. A sample output display is shown in Figure 2. Physician decision-making Because this is a prospective validation study, physicians are directed not to use the EHMRG 7-day score to make decisions about whether a patient should be discharged or admitted to hospital. Physicians will decide the disposition of the patient using their clinical judgment and not based on the score. This is a standard approach for clinical decision rule validation that has been used previously. 14 The 7-day time point was chosen as the primary outcome after consultation with ED physicians, who indicated that acute decision-making is better informed by the likelihood of early events. Physicians could choose not to calculate the risk score if desired, because an additional mouse click would be required to display the risk score and probabilities of 7-day death. Validation of the 30-day mortality model In a nested study, we will also prospectively validate the 30-day mortality model, EHMRG30-ST. The model provides estimated probabilities of 30-day death by the

American Heart Journal Month 2016

62 Lee et al

Figure 1

Yes

ED visit Dx with HF?

Yes No

Does not qualify for the ACUTE study

Exclusions? Dialysis, DNR or Invalid HCN No MD Questionnaire: 1) Estimate PER 2) Plan for patient

Completed Questionnaire? No

Yes Does MD want to view score results?

Enter parameters for EHMRG No

Yes Data linkage for mortality, admission, & ED discharge for passive follow-up

Display EHMRG score

Study flow diagram.

inclusion of the presence of ST-depression on the 12-lead electrocardiogram. A physician will determine the presence of ST depression or other conditions limiting the interpretation of the ST segment (eg, left bundle branch block, left ventricular hypertrophy, or ventricular paced rhythm). 8 Although the 30-day risk calculations will also be prospectively validated, predicted 30-day events are not included in the primary study, and will not be provided to physicians in the ED setting for comparison with PER.

Data handling All electronic case report forms will be collected in the ED and stored on encrypted computers. On a regular basis, electronic case report forms will be uploaded by virtual private network to secure servers at the Institute for Clinical Evaluative Sciences (ICES) for future data linkage and analysis. Outcomes determination Patients will be linked probabilistically to their encrypted provincial health card number. Patients who cannot be linked and those with incomplete physician

questionnaires or missing variables required to calculate the EHMRG score will be excluded. Using the encrypted health card number, we will link the patient to the National Ambulatory Care Reporting System database and the Canadian Institute for Health Information Discharge Abstract Database. The National Ambulatory Care Reporting System database records all ED visits and the Canadian Institute for Health Information Discharge Abstract Database includes all records of hospitalized patients. Using the patient's encrypted health card number we will also link to the Registered Persons Database, which provides vital status information including death.

Research ethics and funding Institutional review board approval will be sought at all study sites before study launch. We will seek ethical approval for a waiver of informed consent, given that this is an observational, minimal risk study. This study was funded by an operating grant from the Canadian Institutes of Health Research (CIHR). The authors are solely responsible for the design and conduct of this study, all study analyses, and the drafting and editing of the paper and its final contents.

American Heart Journal Volume 181, Number 0

Lee et al 63

Figure 2

Results display after EHMRG variable data entry.

Statistical analysis The primary outcomes are mortality within 7 and 30 days. Mortality rates at 7 days will be determined using EHMRG risk score strata and categorized into quintile thresholds from the original EHMRG derivation study. 5 Using the previously-derived β-coefficients and risk thresholds from the EHMRG30-ST model, we will also simultaneously determine the 30-day mortality rates in each of these a priori categories. 8 We will determine the odds ratios for 7-day and 30-day death compared with the lowest-risk categories. We will determine overall model discrimination using the area under the receiver operating characteristic curve (AUC). We will compare the c-statistics for the PER with the predicted probabilities derived from EHMRG and EHMRG30-ST. We will also determine the net reclassification improvement of adding the EHMRG and EHMRG30-ST to clinically estimated risk determined by PER. We will examine safety and efficiency where safety is defined as the proportion of patients who were both high risk and who died after ED discharge, and efficiency is defined as the proportion of admitted patients who were low risk and survived. 15 We will examine the tradeoff between efficiency and safety, 16, 17 expressed as ΔE (1 − P) ÷ ΔS (P), where ΔE is the increase in efficiency, ΔS is the change in safety, and P is the prevalence of the outcome. 15 Finally, in an exploratory analysis, we will determine if the planned decision to admit or discharge the patient differed from the observed disposition after calculation of the risk score by the physician in the ED.

Sample size We estimated the sample size required to validate an AUC of 0.81 for 7-day mortality, assuming an event rate of 2%, which were statistics reported previously in the primary EHMRG publication. With these assumptions, we will require 1,427 patients for an AUC estimate of 0.81 ± 0.10. 18 The analysis will be conducted in unique patients, and thus, we assumed a 25% repeat ED visit rate. Based on our prior work, we assumed that there would be at least 99% success in linking the clinical data with administrative databases for follow-up outcomes. 19 With the above assumptions, we would require a minimum final sample size of 1,885 patients. To validate a lower AUC of 0.75, with all other assumptions held constant, we would require a total of 2,069 patients to be recruited. Conservatively, we will aim to recruit a cohort that meets the latter, larger sample size estimate (ie, more than 2,069 ED patient visits) in this prospective study. Registry organization and funding This study was approved by the institutional review board at University Health Network and will have received approval at each participating site. The study is funded solely by the CIHR. The authors are solely responsible for the design and conduct of this study. The ICES is the study coordinating center.

Discussion Admissions and readmissions for HF are a high priority problem for the health care system. Although

American Heart Journal Month 2016

64 Lee et al

hospitalization rates for HF have decreased in recent years, 20 there remains substantial variation in admissions in population-based studies. 4 Currently, in a health care environment where risk stratification is not used routinely, low rates of hospitalization and high rates of ED discharge are associated with increased rates of adverse outcomes. 21 Strategies to improve decisionmaking in the ED setting are required to ensure that patients' prognosis is considered when making decisions to admit or discharge patients with acute HF. The availability of risk stratification methods that can be used in the ED setting may improve decision-making and care. 22 However, there are few methods of HF risk stratification that have been developed specifically for use in the acute ED setting before any decisions have been made regarding disposition. Many such risk scores were derived in cohorts consisting of patients who were already admitted to hospital, an often higher-risk group, in whom the decision has already been made to hospitalize. To date, none of these have been prospectively validated in the ED setting, further highlighting the importance of the ACUTE study. Prediction rules can significantly improve physicians' decisions and patient care. 23, 24 However, prospective validation is a necessary component before a predictive instrument can be broadly applied. 15 The EHMRG and EHMRG30-ST have been derived, but they have not been prospectively evaluated. After derivation of ED-based clinical decision rules, prospective explicit validation is the next logical step, 25 whereby the results of the decision rule are available to physicians during the decision-making process, but do not direct care (ie, results do not dictate a specified medical response). Prospective validation is an important step before broad implementation and is required before physicians will systematically use decision tools in practice. 26, 27 At the current time, risk stratification algorithms for acute HF are not systematically used, and there are problems of safety and inefficiency. 4 Safety is impacted by decisions made in the ED when higher-risk patients are unrecognized and they may be discharged home. In some cases, these discharges lead to death. Conversely, there may be issues of inefficiency, where lower-risk patients, who could also be managed in an outpatient setting, are instead admitted to hospital. When lower-risk HF patients who are dischargeable and who could be managed as an outpatient are hospitalized, potentially unnecessary costs are incurred to the health care system. In this way, hospital admission can be reserved for intermediate or higher-risk patients, and for those low-risk patients who have extenuating circumstances (eg, social reasons, reduced mobility, inability to care for self, etc). Using a risk stratification method such as EHMRG can potentially reduce the need for hospital admission, because many low-risk patients are often hospitalized. 4 As an alternative to hospital admission, low-risk patients could potentially

be discharged and receive outpatient care because HF is a condition that can be managed in ambulatory care settings. 28, 29 However, before such a strategy can be implemented, prospective validation of the predictive accuracy of the EHMRG prognostic tool is necessary.

Limitations Although the ultimate aim of the EHMRG is to lead to improved prognostication in the ED and better decision-making regarding hospital admission, our primary outcome is mortality and not admission or discharge home. Indeed, the primary outcome of this validation study is mortality (and not hospital admission) because (a) the EHMRG and EHMRG30-ST are models for prognostication of death, (b) in this study, the EHMRG is not intended to directly affect the decision to admit or discharge the patient, and (c) this outcome is a hard objective event and therefore not susceptible to subjectivity in ascertainment. However, prospective validation of a prognostic algorithm is required before it can be used in the clinical setting, necessitating a study such as ours to demonstrate external validity.

Conclusion The ACUTE prospective validation study is an important step in determining whether a retrospectively-derived clinical prognostic algorithm can be used more broadly to assist with decision-making in the ED. If predictive validity of the EHMRG and EHMRG30-ST can be demonstrated, they may be useful as adjuncts to clinical judgment in the acute care emergency environment.

Funding sources The ICES is supported in part by a grant from the Ontario Ministry of Health and Long Term Care. The opinions, results, and conclusions are those of the authors and no endorsement by the Ministry of Health and Long-Term Care or by the ICES is intended or should be inferred. This research was supported by an operating grant from the CIHR (CIHR MOP 114937). Dr D Lee is a clinician–scientist of the CIHR and is the Ted Rogers Chair in Heart Function Outcomes, a joint Hospital– University Chair of the University Health Network and the University of Toronto. Dr Austin is a career investigator of the Heart and Stroke Foundation of Ontario. Dr Tu is supported by a Canada Research Chair in Health Services Research and an Eaton Scholar Award. Dr Grimshaw holds a Canada Research Chair in Health Knowledge Transfer and Uptake.

References 1. Heidenreich PA, Albert NM, Allen LA, et al. Forecasting the impact of heart failure in the United States: a policy statement from the American Heart Association. Circ Heart Fail 2013;6:606-19.

American Heart Journal Volume 181, Number 0

2. Cook C, Cole G, Asaria P, et al. The annual global economic burden of heart failure. Int J Cardiol 2014;171:368-76. 3. Epstein AM, Jha AK, Orav EJ. The relationship between hospital admission rates and rehospitalizations. N Engl J Med 2011;365: 2287-95. 4. Lee DS, Schull MJ, Alter DA, et al. Early deaths in patients with heart failure discharged from the emergency department: a population-based analysis. Circ Heart Fail 2010;3:228-35. 5. Lee DS, Stitt A, Austin PC, et al. Prediction of heart failure mortality in emergent care: a cohort study. Ann Intern Med 2012;156:767-75. 6. Guttmann A, Schull MJ, Vermeulen MJ, et al. Association between waiting times and short term mortality and hospital admission after departure from emergency department: population based cohort study from Ontario, Canada. BMJ 2011;342:d2983. 7. Gabayan GZ, Derose SF, Chiu VY, et al. Emergency Department Crowding and Outcomes After Emergency Department Discharge. Ann Emerg Med 2015;66:483-92. 8. Greig D, Austin PC, Zhou L, et al. Ischemic electrocardiographic abnormalities and prognosis in decompensated heart failure. Circ Heart Fail 2014;7:986-93. 9. McKee PA, Castelli WP, McNamara PM, et al. The natural history of congestive heart failure: the Framingham study. N Engl J Med 1971;285:1441-6. 10. Lee DS, Pencina MJ, Benjamin EJ, et al. Association of parental heart failure with risk of heart failure in offspring. N Engl J Med 2006;355: 138-47. 11. Lee DS, Austin PC, Rouleau JL, et al. Predicting mortality among patients hospitalized for heart failure: derivation and validation of a clinical model. JAMA 2003;290:2581-7. 12. Fischhoff B, Bostrom A, Quadrel MJ. Risk perception and communication. In: Detels R, McEwen J, Reaglehole R, Tanaka H, eds. Oxford textbook of public health. 4th ed. New York: Oxford University Press; 2002. 13. Friedmann PD, Brett AS, Mayo-Smith MF. Differences in generalists' and cardiologists' perceptions of cardiovascular risk and the outcomes of preventive therapy in cardiovascular disease. Ann Intern Med 1996;124:414-21. 14. Stiell IG, Greenberg GH, Wells GA, et al. Prospective validation of a decision rule for the use of radiography in acute knee injuries. JAMA 1996;275:611-5. 15. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 2006;144:201-9.

Lee et al 65

16. Weinstein MC, Stason WB. Foundations of cost-effectiveness analysis for health and medical practices. N Engl J Med 1977;296:716-21. 17. Eddy DM. Clinical decision making: from theory to practice. Cost-effectiveness analysis. Is it up to the task? JAMA 1992;267: 3342-8. 18. Obuchowski NA, McClish DK. Sample size determination for diagnostic accuracy studies involving binormal ROC curve indices. Stat Med 1997;16:1529-42. 19. MacFadden DR, Crystal E, Krahn AD, et al. Sex differences in implantable cardioverter-defibrillator outcomes: findings from a prospective defibrillator database. Ann Intern Med 2012;156: 195-203. 20. Krumholz HM, Normand SL, Wang Y. Trends in hospitalizations and outcomes for acute cardiovascular disease and stroke, 1999–2011. Circulation 2014;130:966-75. 21. Bhatia RS, Austin PC, Stukel TA, et al. Outcomes in patients with heart failure treated in hospitals with varying admission rates: population-based cohort study. BMJ Qual Saf 2014;23:981-8. 22. Stiell IG, Clement CM, Brison RJ, et al. A risk scoring system to identify emergency department patients with heart failure at high risk for serious adverse events. Acad Emerg Med 2013;20:17-26. 23. Quinn JV, Stiell IG, McDermott DA, et al. The San Francisco Syncope Rule vs physician judgment and decision making. Am J Emerg Med 2005;23:782-6. 24. Reilly BM, Evans AT, Schaider JJ, et al. Impact of a clinical decision rule on hospital triage of patients with suspected acute cardiac ischemia in the emergency department. JAMA 2002;288: 342-50. 25. McGinn TG, Guyatt GH, Wyer PC, et al. Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 2000;284: 79-84. 26. Stiell IG, Wells GA, Vandemheen KL, et al. The Canadian C-spine rule for radiography in alert and stable trauma patients. JAMA 2001;286:1841-8. 27. Brehaut JC, Stiell IG, Graham ID. Will a new clinical decision rule be widely used? The case of the Canadian C-spine rule. Acad Emerg Med 2006;13:413-20. 28. Lee DS, Ezekowitz JA. Risk stratification in acute heart failure. Can J Cardiol 2014;30:312-9. 29. Dunbar-Yaffe R, Stitt A, Lee JJ, et al. Assessing risk and preventing 30-day readmissions in decompensated heart failure: opportunity to intervene? Curr Heart Fail Rep 2015;12:309-17.