Does the emergency surgery score accurately predict outcomes in emergent laparotomies?

ARTICLE IN PRESS

Does the emergency surgery score accurately predict outcomes in emergent laparotomies? Thomas Peponis, MD, Jordan D. Bohnen, MD, MBA, Naveen F. Sangji, MD, MPH, Anirudh R. Nandan, BA, Kelsey Han, BA, Jarone Lee, MD, D. Dante Yeh, MD, Marc A. de Moya, MD, George C. Velmahos, MD, PhD, David C. Chang, PhD, and Haytham M. A. Kaafarani, MD, MPH, Boston, MA

Background. The emergency surgery score is a mortality-risk calculator for emergency general operation patients. We sought to examine whether the emergency surgery score predicts 30-day morbidity and mortality in a high-risk group of patients undergoing emergent laparotomy. Methods. Using the 2011–2012 American College of Surgeons National Surgical Quality Improvement Program database, we identified all patients who underwent emergent laparotomy using (1) the American College of Surgeons National Surgical Quality Improvement Program definition of “emergent,” and (2) all Current Procedural Terminology codes denoting a laparotomy, excluding aortic aneurysm rupture. Multivariable logistic regression analyses were performed to measure the correlation (c-statistic) between the emergency surgery score and (1) 30-day mortality, and (2) 30-day morbidity after emergent laparotomy. As sensitivity analyses, the correlation between the emergency surgery score and 30day mortality was also evaluated in prespecified subgroups based on Current Procedural Terminology codes. Results. A total of 26,410 emergent laparotomy patients were included. Thirty-day mortality and morbidity were 10.2% and 43.8%, respectively. The emergency surgery score correlated well with mortality (c-statistic = 0.84); scores of 1, 11, and 22 correlated with mortalities of 0.4%, 39%, and 100%, respectively. Similarly, the emergency surgery score correlated well with morbidity (cstatistic = 0.74); scores of 0, 7, and 11 correlated with complication rates of 13%, 58%, and 79%, respectively. The morbidity rates plateaued for scores higher than 11. Sensitivity analyses demonstrated that the emergency surgery score effectively predicts mortality in patients undergoing emergent (1) splenic, (2) gastroduodenal, (3) intestinal, (4) hepatobiliary, or (5) incarcerated ventral hernia operation. Conclusion. The emergency surgery score accurately predicts outcomes in all types of emergent laparotomy patients and may prove valuable as a bedside decision-making tool for patient and family counseling, as well as for adequate risk-adjustment in emergent laparotomy quality benchmarking efforts. (Surgery 2017;j:j-j.) From the Department of Surgery, Division of Trauma, Emergency Surgery & Surgical Critical Care, Massachusetts General Hospital, Boston, MA

SIGNIFICANT PROGRESS has been made toward better and more objective assessment of the quality of surgical care. The need for robust risk-adjustment tools for accurate benchmarking has been largely recognized.1-6 Additionally, it is well known that emergency surgery (ES) has been a public health Accepted for publication March 18, 2017. Reprint requests: Haytham M.A. Kaafarani, MD, MPH, Division of Trauma, Emergency Surgery, and Surgical Critical Care, Massachusetts General Hospital & Harvard Medical School, 165 Cambridge Street, Suite 810, Boston, MA 02114. E-mail: [email protected]. 0039-6060/$ - see front matter Ó 2017 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.surg.2017.03.016

burden, with 27,668,807 admissions (7.1% of all hospitalizations) of ES patients in the United States from 2001 to 2010.7 Multiple recent studies have shown that ES is associated with higher morbidity and mortality rates compared to nonES, even after controlling for relevant preoperative and intraoperative confounders.8-10 However, benchmarking efforts in ES have been limited either by focusing on specific types of operation or by simply using a uniform “adjustment” variable for ES.11-14 A recent study by Bohnen et al15 showed that various comorbidities affect ES and non-ES differently. For example, the diabetesrelated relative increase in the risk of mortality is different based on whether the patient undergoes an elective versus an emergent colectomy. These SURGERY 1

ARTICLE IN PRESS 2 Peponis et al

findings strongly suggested the need to develop separate risk-adjustment models for ES, rather than include ES cases in the currently available models. As part of these efforts to improve benchmarking of care in ES, our group recently developed and validated a novel score, called the “emergency surgery score” (ESS; previously, ESAS).16 ESS is based on 22 independent predictors of mortality in ES patients, including 3 demographic variables, 10 comorbidities, and 9 preoperative laboratory variables. The score ranges from 0 to 29 and can be calculated from information obtained from a patient’s history and routine laboratory tests. ESS aims to accurately predict postoperative mortality in patients undergoing ES and has been suggested as a tool for surgical quality benchmarking, as well as for preoperative patient and family counseling. In the validation study, ESS had a c-statistic of 0.86, suggesting that it performs remarkably well as a mortality risk calculator.17,18 One of the shortcomings of the initial study was that all patients undergoing ES were included in the analysis, irrespective of the complexity of the operation. As a result, a large number of low-risk cases such as laparoscopic appendectomies and cholecystectomies were included. In the current study, we sought to investigate whether ESS can accurately predict postoperative mortality and morbidity in patients undergoing emergent laparotomies (EL), arguably the highest risk category of ES. METHODS The American College of Surgeons National Surgical Quality Improvement Program (ACSNSQIP) 2011 and 2012 databases were used to identify patients undergoing EL.19,20 The ACSNSQIP is a national database that contains more than 200 preoperative and intraoperative variables, as well as twenty-one 30-day postoperative outcome variables, including total morbidity and mortality. We identified emergent cases using the ACSNSQIP variable of “EMERGNCY.” According to the ACS-NSQIP user guide definitions, an emergent case is an operation “usually performed within a short interval of time from the diagnosis or the onset of related preoperative symptomatology.” Subsequently, we identified Current Procedural Terminology codes denoting a laparotomy. More specifically, we used the following codes: 38100, 38115, 43500 - 2, 43510, 43520, 43605, 43610, 43611, 43620 - 2, 43631 - 5, 43640, 43641, 43840,

Surgery j 2017

44005, 44050, 44110, 44111, 44120, 44121, 44125, 44130, 44140, 44141, 44143 - 6, 44150, 44151, 44155 - 8, 44160, 44310, 44320, 44602, 44604, 44800, 44950, 44960, 45540, 44550, 47350, 47600, 47605, 49000, 49002, 49020, 49560, 49561, and 49566. Laparoscopic, transplant, trauma, and ruptured aortic aneurysm operations were excluded. The correlation between ESS and mortality was assessed, and its ability to predict 30-day mortality in a stepwise progressive fashion was examined using the area under the receiver operator characteristic (ROC) curve or c-statistic. Similarly, the ability of ESS to predict 30-day morbidity in a stepwise progressive fashion was also evaluated using the area under the ROC curve or c-statistic of the correlation between the 2 variables. Subsequently, by using multiple regression models, we evaluated the ability of each of the 22 ESS components to independently predict 30-day postoperative mortality and morbidity after EL. We performed 2 separate sensitivity analyses to delineate the predictive accuracy of ESS across a wide range of operations and diagnoses. Based on prespecified Current Procedural Terminology codes, we defined 5 different subgroups delineating different types of operation: (1) gastroduodenal; (2) small and large intestinal; (3) hepatobiliary; (4) splenic; and (5) ventral hernia repair. Similar to the above, we studied the correlation between ESS and 30-day morbidity and mortality rates for the 5 subgroups using c-statistics. Using the International Classification of Diseases, Ninth Revision, diagnoses codes, we defined 8 subgroups: (1) appendicitis; (2) cholecystitis; (3) peptic ulcer disease; (4) mesenteric ischemia; (5) diverticulosis or diverticulitis; (6) incarcerated ventral hernia; (7) Clostridium difficile-associated colitis; and (8) bowel obstruction. Similar to the above, we studied the correlation between ESS and 30-day mortality rates for the 8 subgroups using the c-statistics. We performed the statistical analysis using STATA software (version 13.1). The variables are presented as frequencies and percentages. First, multiple logistic regression analyses were performed to measure the correlation (area under the ROC curve or c-statistic) of ESS to 30-day mortality and 30-day morbidity. This was repeated for the purpose of our sensitivity analyses, as described. Subsequently, we calculated odds ratios with 95% confidence intervals for each of the 22 ESS components. The study was performed after obtaining approval from our institutional review board.

ARTICLE IN PRESS Peponis et al 3

Surgery Volume j, Number j

Table I. Preoperative characteristics of patients who underwent an emergent laparotomy in 2011 and 2012 Variable Demographics Age >60 y White race Female Admitted from home Transferred from outside emergency department Transferred from other acute care hospital (inpatient) Baseline comorbidities Functional dependence Body mass index <20 kg/m2 Body mass index >35 kg/m2 >10% weight loss in the last 6 months Bleeding disorders Transfusion of more than 4 units of PRBCs up to 72 hours before surgery Dyspnea History of chronic obstructive pulmonary disease Current pneumonia Ascites Esophageal varices Steroid use Congestive heart failure up to 30 days before surgery Myocardial infarction up to 6 months before surgery History of angina up to 30 days before surgery Hypertension requiring medications History of percutaneous coronary intervention History of cardiac surgery Diabetes requiring oral agents or insulin Smoking up to 12 months before surgery History of disseminated cancer Chemotherapy up to 30 days before surgery History of revascularization/amputation due to peripheral vascular disease History of rest pain or gangrene Coma for more than 24 hours History of stroke with neurological deficit History of stroke without neurological deficit History of transient ischemic attacks Paraplegia Hemiplegia Quadriplegia Impaired sensorium History of central nervous system tumor Alcohol ingestion (>2 drinks per day, up to 2 weeks before admission) Radiotherapy up to 90 days before surgery Prior operation within 30 days Ventilator dependence Acute renal failure Laboratory tests Albumin <3.0 Alkaline phosphatase >125 Total bilirubin >1.0 Blood urea nitrogen >40.0 Creatinine >1.2 Hematocrit <38.0 International normalized ratio >1.5 Prothrombin time >35

n (%) 14,501 16,604 14,040 21,089 2,131 1,933

(54.9) (73.4) (53.2) (79.9) (8.1) (7.3)

2,984 2,444 3,840 1,008 3,239 2,034 2,797 2,369 336 1,240 32 2,095 715 197 113 13,054 549 657 4,201 5,758 1,255 336 272 26 40 357 262 267 57 146 34 502 21 478 91 782 1,864 895

(11.3) (10.1) (15.9) (3.8) (12.8) (7.7) (10.6) (9.0) (3.2) (4.7) (0.3) (7.9) (2.7) (1.9) (1.1) (49.4) (5.2) (6.3) (15.9) (21.8) (4.8) (3.2) (2.6) (0.3) (0.4) (3.4) (2.5) (2.6) (0.5) (1.4) (0.3) (4.8) (0.2) (4.6) (0.9) (7.4) (7.1) (3.4)

11,397 3,297 5,702 3,030 7,132 13,209 2,399 2,800

(43.2) (12.5) (21.6) (11.5) (27.0) (50.6) (14.1) (20.2)

(continued)

ARTICLE IN PRESS 4 Peponis et al

Surgery j 2017

Table I. (continued) n (%)

Variable Platelets <150,000 Serum glutamic-oxaloacetic transaminase >40 Sodium <135 Sodium >145 White blood cells <4,500 White blood cells >11,000 and <15,000 White blood cells 15,000–25,000 White Blood Cells (WBCs) > 25,000

2,800 3,999 5,727 682 1,580 6,304 6,107 1,288

(20.2) (15.1) (21.7) (2.6) (6.1) (24.2) (23.5) (5.0)

PRBC, packed red blood cells.

RESULTS Of the 986,034 patients in the 2011 and 2012 NSQIP databases, 26,410 met our inclusion criteria. The overall 30-day mortality rate was 10.2% and the overall 30-day morbidity rate was 43.8%. Table I summarizes the preoperative characteristics of these patients, including demographics and baseline comorbidities, as well as laboratory values. Table II describes their postoperative outcomes. ESS correlated very well with 30-day mortality, with a c-statistic of 0.84. A stepwise increase in mortality was observed for each additional point on the ESS scale; scores of 1, 11, and 22 correlated with mortalities of 0.4%, 39%, and 100%, respectively (Fig 1). Even though the maximum score that can be achieved using ESS is 29, none of the patients in our cohort scored higher than 22 and none of those scoring 22 survived. ESS correlated moderately well with 30-day morbidity also, with a c-statistic of 0.74. A stepwise increase in 30-day morbidity rates was observed, with scores of 0, 7, and 11 correlating with complication rates of 13%, 58%, and 79%, respectively. Above a score of 11, the correlation was a plateau, with the vast majority of those patients experiencing at least one complication in the postoperative period (Fig 2). In the multiple regression analyses performed, all ESS variables independently predicted mortality except for the variable “transfer status.” This variable indicates whether the patient was transferred from an outside hospital. The results are summarized in Table III. When we repeated the analyses using morbidity as the dependent variable, we found that 19 out of the 22 variables were independent risk factors of postoperative complications. The only 3 variables that did not show statistical significance were white race, body mass index less than 20, and transfer status. Tables IV and V summarize the findings of the sensitivity analyses. In summary, ESS was an

Table II. Postoperative outcomes of patients who underwent emergent laparotomy in 2011 and 2012 Variable Superficial surgical site infection Deep surgical site infection Organ space surgical site infection Abdominal wall dehiscence Pneumonia Unplanned intubation Pulmonary embolism Failure to wean off ventilator >48 hours after surgery Progression of baseline renal insufficiency Acute kidney injury Urinary tract infection Cerebrovascular accident with neurological deficits Coma lasting >24 hours Cardiac arrest requiring cardiopulmonary resuscitation Myocardial infarction Transfusion-requiring hemorrhage Deep venous thrombosis Sepsis Septic shock Mortality

n (%) 1,533 502 1,364 594 1,932 1,565 270 3,926

(5.8) (1.9) (5.2) (2.3) (7.3) (5.9) (1.0) (14.9)

287 (1.1) 687 (2.6) 940 (3.6) 152 (0.6) 29 (0.1) 550 (2.1) 389 5,516 639 1,660 2,017 2,700

(1.5) (20.9) (2.4) (6.3) (7.6) (10.2)

accurate predictor of mortality in all 5 procedurebased subgroups (gastroduodenal operation [n = 1,836], small and large intestinal operation [n = 19,892], hepatobiliary operation [n = 739], splenic operation [n = 196], and ventral hernia operation [n = 1,695]; c-statistic range, 0.79– 0.84), and in all 8 diagnosis-based subgroups (appendicitis [n = 2,076], cholecystitis [n = 310], peptic ulcer disease [n = 1,274], mesenteric ischemia [n = 707], diverticulosis or diverticulitis [n = 1,948], incarcerated ventral hernia [n = 1,909], C difficile-associated colitis [n = 180], and obstruction [n = 3,079]; c-statistic range, 0.75–0.90).

ARTICLE IN PRESS Peponis et al 5

Surgery Volume j, Number j

Fig 1. Observed 30-day mortality rates based on the ESS score.

Fig 2. Observed 30-day morbidity rates based on the ESS score.

DISCUSSION When applied to the higher risk ES patients undergoing EL, the ESS is an accurate 30-day mortality and morbidity predictor, and its predictive ability applies to many subgroups of EL performed for various clinical indications. As such, ESS may prove valuable as a bedside decision-making tool to help counsel patients and family members prior to EL. In addition, ESS can serve as a risk-adjustment tool for benchmarking the quality of care of ES.

Numerous studies have shown that ES is an independent predictor of poor postoperative outcomes.7-9 Surprisingly, efforts aimed at the development of a risk stratification system specific to ES (similar to the Trauma Quality Improvement Program for trauma patients or the ACS-NSQIP models specific for colorectal or bariatric operation) have been rather limited to date.21-28 One of those studies by Kwok et al25 compared a targeted “emergent colectomies” mortality prediction model to other well-established scoring systems

ARTICLE IN PRESS 6 Peponis et al

Surgery j 2017

Table III. Multivariable regression analysis of 22 ESAS components as independent risk factors for 30-day mortality among patients undergoing emergent laparotomy ESS Variables

Score

Odds ratio

Standard error

95% confidence interval

P value

Age >60 y White race Transfer status Transferred from outside emergency department Transferred from other acute-care hospital (inpatient) Functional dependence Body mass index <20 kg/m2 >10% weight loss in the last 6 months Dyspnea History of chronic obstructive pulmonary disease Ascites Steroid use Hypertension requiring medications Disseminated cancer Ventilator dependence Albumin <3.0 Alkaline phosphatase >125 Blood urea nitrogen >40.0 Creatinine >1.2 International normalized ratio >1.5 Platelets <150,000 Serum glutamic-oxaloacetic transaminase >40 Sodium (Na) >145 White blood cells <4,500 15,000–25,000 >25,000

2 1

2.46 1.29

0.18 0.10

2.12–2.85 1.12–1.50

<.001 <.001

1 1 1 1 1 1 1 1 1 1 3 3 1 1 1 2 1 1 1 1

1.12 1.10 1.68 1.65 1.52 1.51 1.53 1.57 1.23 1.22 2.29 2.53 1.50 1.46 1.30 2.06 2.06 1.61 1.63 1.42

0.11 0.10 0.12 0.15 0.18 0.11 0.13 0.15 0.11 0.08 0.23 0.20 0.10 0.11 0.10 0.14 0.14 0.11 0.11 0.19

0.93–1.36 0.92–1.31 1.46–1.94 1.38–1.96 1.21–1.91 1.30–1.74 1.30–1.79 1.30–1.90 1.03–1.45 1.07–1.39 1.87–2.80 2.16–2.96 1.33–1.70 1.26–1.69 1.12–1.51 1.80–2.36 1.80–2.37 1.40–1.85 1.43–1.87 1.09–1.83

.216 .302 <.001 <.001 <.001 <.001 <.001 <.001 .018 .003 <.001 <.001 <.001 <.001 .001 <.001 <.001 <.001 <.001 .008

1 1 2

1.63 1.18 1.68

0.17 0.09 0.18

1.34–1.99 1.02–1.36 1.37–2.07

<.001 .026 <.001

Table IV. Sensitivity analysis based on the Current Procedural Terminology codes Type of surgery Splenic Gastroduodenal Intestinal Hepatobiliary Ventral hernia

n 196 1,836 19,892 739 1,695

30-day mortality 9.7 11.5 9.8 5.7 2.3

% % % % %

Table V. Sensitivity analysis based on the International Classifications of Disease codes

c-statistic 0.81 0.80 0.83 0.84 0.79

such as the American Society of Anesthesiologists score, the Surgical Risk Scale, and the ACS Colorectal Surgery Risk Calculator. The targeted emergency colectomy score performed better. Such findings, like ours, are not surprising perhaps because the nonspecific scores, such as the ACS risk calculator, do not take into account the possibility that different preoperative variables could affect the outcome of ES and non-ES differently.15 For example, cirrhosis might increase the risk of mortality in ES more (or less), percentage-wise, than in non-ES. As a consequence, with this study

Diagnosis

n

Appendicitis Cholecystitis Peptic ulcer disease Mesenteric ischemia Diverticulosis or diverticulitis Incarcerated ventral hernia Clostridium difficile colitis Obstruction

2,076 310 1,274 707 1,948 1,909 180 3,079

30-day mortality c-statistic 0.63 4.19 11 29.99 7.34 3.04 36.11 5.49

0.90 0.83 0.86 0.75 0.80 0.81 0.85 0.81

as well as with our initial validation data, we envision ESS as a reliable mortality risk predictor in ES generally and EL specifically, similar to the Injury Severity Score that was developed in the 1970s and has since been established as the best risk stratification system in trauma patients.29,30 In addition, ESS can provide a reproducible and objective score to assist surgeons in preoperative

ARTICLE IN PRESS Peponis et al 7

Surgery Volume j, Number j patient and family discussions.31-34 Often, acutecare surgeons struggle with such discussions and usually convey their “gestalt” feeling about the eventual outcome of the operation to the patient and family. The ESS can help to more objectively inform this discussion by estimating mortality and morbidity. For example, in our study, patients who underwent an EL and had an ESS score of 22 or higher had a 100% mortality rate. There is no doubt that revealing such information can potentially avoid futile operation at the end of life, when used appropriately in preoperative discussions. ESS can also significantly assist in the benchmarking efforts of ES. Without adequate and convincing risk-adjustment specific to ES, areas of concern in the quality of care cannot be identified, targeted, or improved. With ESS being specifically designed for ES, acute care surgeons hopefully will come to trust its risk-adjusting abilities, which in turn will help in the buy-in of quality improvement efforts in clinical areas and hospitals with higher than expected mortality and morbidity rates. With EL having the highest rates of morbidity and mortality among surgical specialties, quality assessment and improvement that takes into consideration the acuity of disease and the patients’ metabolic derangements, which are prominent in EL patients, could result in substantial outcome improvement in the long-term. Our study has a few limitations that must be acknowledged. First, it is a retrospective analysis of a prospectively collected database. Second, the NSQIP definition of “emergent” differs from that of the AAST. However, NSQIP uses clinical reviewers who make the decision as to whether an operation should be coded as an emergency, whereas the AAST definition relies purely on administrative data and codes, which may in fact signify that our cohort better reflects “true” ES. Third, NSQIP captures data only up to postoperative day 30, whereas the 3- or 6-month endpoint outcome of these high-risk patients remains unknown. Finally, we were limited in our definition of outcome to mortality and complications, as defined by ACSNSQIP, and thus were not able to investigate the correlation of ESS with functional outcomes, which is equally important in ES patients. In conclusion, ESS accurately predicts postoperative 30-day mortality and 30-day morbidity in a wide variety of EL patients. It may prove valuable both as a bedside decision-making tool for the preoperative patient, for family counseling, and as a risk-adjustment tool in future quality benchmarking efforts in EL.

REFERENCES 1. Khuri SF, Daley J, Henderson WG, et al. The Department of Veterans Affairs NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998;228:491-507. 2. Napolitano LM, Bass BL. Risk-adjusted outcomes and perioperative care. Surg Clin North Am 2005;85:1341-6. 3. Khuri SF, Daley J, Henderson WG. The comparative assessment and improvement of quality of surgical care in the Department of Veterans Affairs. Arch Surg 2002;137:20-7. 4. Haider AH, Hashmi ZG, Gupta S, et al. Benchmarking of trauma care worldwide: the potential value of an International Trauma Data Bank (ITDB). World J Surg 2014;38: 1882-91. 5. Shafi S, Nathens AB, Parks J, Cryer HM, Flides JJ, Gentilello LM. Trauma quality improvement using riskadjusted outcomes. J Trauma 2008;64:599-606. 6. McMillan MT, Allegrini V, Asbun HJ, et al. Incorporation of procedure specific risk into the ACS-NSQIP surgical risk calculator improves the prediction of morbidity and mortality after pancreaticoduodenectomy. Ann Surg 2016;265: 978-86. 7. Gale SC, Shafi S, Dombrovskiy VY, et al. The public health burden of emergency general surgery in the United States: a 10-year analysis of the Nationwide Inpatient Sample-2001 to 2010. J Trauma Acute Care Surg 2014;77:202-8. 8. Ingraham AM, Cohen ME, Bilimoria KY, et al. Comparison of hospital perfomrance in nonemergency versus emergency colorectal operations at 142 hospitals. J Am Coll Surg 2010;210:155-65. 9. Ingraham AM, Cohen ME, Bilimoria KY, et al. Comparison of 30-day outcomes after emergency general surgery procedures: potential for targeted improvement. Surgery 2010;148:217-38. 10. Havens JM, Peetz AB, Do WS, et al. The excess morbidity and mortality of emergency general surgery. J Trauma Acute Care Surg 2015;78:306-11. 11. Shah AA, Haider AH, Zogg CK, et al. National estimates of predictors of outcomes for emergency general surgery. J Trauma Acute Care Surg 2015;78:482-90. 12. Savage SA, Klekar CS, Priest EL, et al. Validating a new grading scale for emergency general surgery diseases. J Surg Res 2015;196:264-9. 13. Shafi S, Priest EL, Crandall ML, et al. Multicenter validation of American Association for the Surgery of Trauma grading system for acute colonic diverticulitis and its use for emergency general surgery quality improvement program. J Trauma Acute Care Surg 2016;80:405-10. 14. Shafi S, Aboutanos M, Brown CV, et al. Measuring anatomic severity of disease in emergency general surgery. J Trauma Acute Care Surg 2014;76:884-7. 15. Bohnen JD, Ramly EP, Sangji NF, et al. Perioperative risk factors impact outcomes in emergency versus nonemergency surgery differently: Time to separate our national risk-adjustment models? J Trauma Acute Care Surg 2016;81:122-30. 16. Sangji NF, Bohnen J, Ramly EP, et al. Derivation and validation of a novel emergency surgery acuity score (ESAS). J Trauma Acute Care Surg 2016;82:213-20. 17. Kumar R, Indrayan A. Receiver operating characteristic (ROC) curve for medical researchers. Indian Pediatr 2011;48:277-87. 18. Linden A. Measuring diagnostic and predictive accuracy in disease management: an introduction to receiver operating

ARTICLE IN PRESS 8 Peponis et al

19.

20.

21.

22.

23. 24.

25.

characteristic (ROC) analysis. J Eval Clin Pract 2006;12: 132-9. American College of Surgeons National Surgical Quality Improvement Program. User guide for the 2012 ACS NSQIP Participant Use Data File. https://www.facs.org/ ;/media/files/quality%20programs/nsqip/ug12.ashx. American College of Surgeons National Surgical Quality Improvement Program. User guide for the 2011 Participant Use Data File [cited 2016 Nov 29]. Available from: https:// www.facs.org/;/media/files/quality%20programs/nsqip/ug 11.ashx Hashmi ZG, Dimick JB, Efron DT, et al. Reliability adjustment: a necessity for trauma center ranking and benchmarking. J Trauma Acute Care Surg 2013;75:166-72. Sutton R, Bann S, Brooks M, Sarin S. The Surgical Risk Scale as an improved tool for risk-adjusted analysis in comparative surgical audit. Br J Surg 2002;89:763-8. Copeland GP, Jones D, Walters M. POSSUM: a scoring system for surgical audit. Br J Surg 1991;78:355-60. Cohen ME, Bilimoria KY, Ko CY, Hall BL. Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. J Am Coll Surg 2009;208:1009-16. Kwok AC, Lipsitz SR, Bader AM, Gawande AA. Are targeted preoperative risk prediction tools more powerful? A test of models for emergency colon surgery in the very elderly. J Am Coll Surg 2011;213:220-5.

Surgery j 2017

26. Meredith JW, Evans G, Kilgo PD, et al. A comparison of the abilities of nine scoring algorithms in predicting mortality. J Trauma 2002;53:621-8. 27. Champion HR, Copes WS, Sacco WJ, et al. The Major Trauma Outcome Study: establishing national norms for trauma care. J Trauma 1990;30:1356-65. 28. Easterlin MC, Chang DC, Wilson SE. A practical index to predict 30-day mortality after major amputation. Ann Vasc Surg 2013;27:909-17. 29. Baker SP, O’Neill B, Haddon W Jr, Long WB. The injury severity score: a method for describing patients with multiple injuries and evaluating emergency care. J Trauma 1974; 14:187-96. 30. Semmlow JL, Cone R. Utility of the injury severity score: a confirmation. Health Serv Res 1976;11:45-52. 31. Bernat JL, Peterson LM. Patient-centered informed consent in surgical practice. Arch Surg 2006;141:86-92. 32. Jones JW, McCullough LB, Richman BW. A comprehensive primer of surgical informed consent. Surg Clin North Am 2007;87:903-18. 33. Pecanac KE, Kehler JM, Brasel KJ, et al. It’s big surgery: preoperative expressions of risk, responsibility, and commitment to treatment after high-risk operations. Ann Surg 2014;259:458-63. 34. Nabozny MJ, Kruser JM, Steffens NM, et al. Constructing high-stakes surgical decisions: it’s better to die trying. Ann Surg 2016;263:64-70.