- Email: [email protected]
Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator
The American Journal of Surgery xxx (xxxx) xxx
Contents lists available at ScienceDirect
The American Journal of Surgery journal homepage: www.americanjournalofsurgery.com
Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator Sina Khaneki a, Michael R. Bronsert a, c, William G. Henderson a, c, e, Maryam Yazdanfar a, Anne Lambert-Kerzner a, b, Karl E. Hammermeister a, c, d, Robert A. Meguid a, c, f, * a
Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO, USA Colorado School of Public Health, University of Colorado, Aurora, CO, USA Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA d Division of Cardiology, Department of Medicine, University of Colorado School of Medicine, Aurora, CO, USA e Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO, USA f Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA b c
a r t i c l e i n f o
a b s t r a c t
Article history: Received 1 May 2019 Received in revised form 24 July 2019 Accepted 27 July 2019
Background: The novel Surgical Risk Preoperative Assessment System (SURPAS) requires entry of five predictor variables (the other three variables of the eight-variable model are automatically obtained from the electronic health record or a table look-up), provides patient risk estimates compared to national averages, is integrated into the electronic health record, and provides a graphical handout of risks for patients. The accuracy of the SURPAS tool was compared to that of the American College of Surgeons Surgical Risk Calculator (ACS-SRC). Methods: Predicted risk of postoperative mortality and morbidity was calculated using both SURPAS and ACS-SRC for 1,006 randomly selected 2007e2016 ACS National Surgical Quality Improvement Program (NSQIP) patients with known outcomes. C-indexes, Hosmer-Lemeshow graphs, and Brier scores were compared between SURPAS and ACS-SRC. Results: ACS-SRC risk estimates for overall morbidity underestimated risk compared to observed postoperative overall morbidity, particularly for the highest risk patients. SURPAS accurately estimates morbidity risk compared to observed morbidity. Conclusions: SURPAS risk predictions were more accurate than ACS-SRC's for overall morbidity, particularly for high risk patients. Summary: The accuracy of the SURPAS tool was compared to that of the American College of Surgeons Surgical Risk Calculator (ACS-SRC). SURPAS risk predictions were more accurate than those of the ACSSRC for overall morbidity, particularly for high risk patients. © 2019 Published by Elsevier Inc.
Keywords: SURPAS Surgical risk prediction Postoperative outcomes Comparative effectiveness Accuracy Risk assessment
Introduction The SUrgical Risk Preoperative Assessment System (SURPAS), which has been developed over the past six years, requires the input of five preoperative predictor variables (the other three variables in the eight-variable model are automatically obtained from
* Corresponding author. Division of Cardiothoracic Surgery, Department of Surgery, University of Colorado Denver, Anschutz Medical Campus, 12631 E, 17th Avenue, C-310, Aurora, CO, 80045, USA. E-mail address: [email protected] (R.A. Meguid).
the EHR or a table look-up) to assess the patient's risk of 11 different postoperative adverse outcomes.1e5 The strengths of the SURPAS tool include: (1) It is based upon analysis of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) participant use file that includes operations in over 6 million patients from over 700 medical centers in the U.S. and abroad; (2) It uses one parsimonious model requiring manual input of only five of the eight required variables readily ascertained during the preoperative visit; (3) It is applicable to a broad range of surgical operations from nine surgical specialties; (4) It is built into the electronic health record (EHR) at the University of Colorado Health (UCHealth) System; (5) It compares the patient's individual
https://doi.org/10.1016/j.amjsurg.2019.07.036 0002-9610/© 2019 Published by Elsevier Inc.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
2
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
risks to the national average patient undergoing the same operation; (6) It automatically writes a preoperative risk discussion note into the patient's EHR; and (7) It produces a paper handout of the patient's risk compared to national averages that can be given to the patient. SURPAS has currently been used at the University of Colorado Hospital in over 3,000 patients by over 90 different providers, and we are working on further dissemination and implementation of SURPAS in the UCHealth system. Surgical providers have asked how the SURPAS tool compares with the ACS Surgical Risk Calculator (ACS-SRC).6,7 A typical outpatient surgical clinic schedule might include 20 patients in 5 h or less; this limited time has to be efficiently used to allow for high quality patient care and patient centered practice. The ACS-SRC requires manual entry of 21 variables including CPT code, age group, gender, functional status, emergency case, American Society of Anesthesiology physical status classification (ASA class), steroid use for a chronic condition, ascites within 30 days prior to surgery, systemic sepsis within 48 h prior to surgery, ventilation dependence, disseminated cancer, diabetes, hypertension requiring medication, congestive heart failure in 30 days prior to surgery, dyspnea, current smoker within one year, history of severe chronic obstructive pulmonary disease, dialysis, acute renal failure, height and weight. Chart review to gather these data items is a tedious task that is challenging in a busy practice. In comparison, SURPAS requires manual entry of only five variables (CPT code of primary operation from a drop-down listdthis provides workRVU of the operation and CPT-specific event rate from the ACS NSQIP database by table look-up), inpatient/outpatient setting, emergency status of the operation, ASA class, and functional health status) and the other variables (patient age and specialty of primary surgeon) are auto populated from the patient's electronic health record. In practice, we have found that SURPAS takes less than 1 min to accomplish compared to considerably more time for the ACS-SRC. SURPAS provides estimates of 11 30-day postoperative adverse outcomes including: mortality, overall morbidity, unplanned readmission, infectious, urinary tract infection (UTI), cardiac, bleeding, pulmonary, venous thromboembolic events (VTE), renal, and stroke complications. This consolidation of the 18 individual postoperative complications in the ACS NSQIP was done using factor analysis of the ACS NSQIP database. (2) The ACS-SRC provides risks for a slightly different set of 13 adverse postoperative outcomes: serious complication, any complication, pneumonia, cardiac complication, surgical site infection (SSI), UTI, VTE, renal failure, readmission, unplanned return to operating room, death, discharge to nursing or rehabilitation facility, and predicted length of stay. An important issue is, “How well do the SURPAS and ACS-SRC risk estimates compare?” The purpose of this study was to make this comparison in about 1,000 patients randomly selected from the ACS NSQIP database for two of the outcomes that are common between SURPAS and the ACS-SRC, postoperative mortality and overall morbidity. We hypothesized that SURPAS is as accurate at risk prediction as the ACS-SRC, while having the advantages of requiring manual entry of one-fourth of the predictor variables, being integrated into the EHR, comparing a patient's individual risks to national averages for patients undergoing the same operation, and automatically generating a preoperative note and a graphical display of the patient's risks for the patient to take home. Material and methods A limited sample size of approximately 1,000 patients was randomly selected from the ACS NSQIP database Participant Use File (PUF) from 2007 to 2016, which includes over 6 million patients from over 700 participating medical centers, for patients who had complete data for all of the variables needed to calculate risk from
both of the risk calculators. A simple random sample without replacement of the total database was used instead of randomly selecting patients by characteristics (for example, high risk patients or from certain types of operations or specialties) because the two risk calculators are intended to be used in any surgical patient. The random sample was obtained by using the SAS “proc surveyselect” statement. The outcome variables were limited to 30-day mortality and overall morbidity because these were identically defined in the SURPAS and ACS-SRC prediction systems and they have long been considered the two most important outcomes of the NSQIP.8,9 Overall morbidity was considered to be the primary outcome variable because it has the highest incidence of any of the NSQIP postoperative adverse outcomes; mortality was considered a secondary outcome. The ACS-SRC risk estimates were obtained by manually entering the 21 predictor variables into the ACS-SRC for each patient by two of the coauthors (SK, MY). The SURPAS risk estimates were obtained by computer calculation using the patient variables and the SURPAS models generated using logistic regression equations.4 The SURPAS model developed on ACS NSQIP data from 2012 to 2015 was used because it was a similar period to that reported for the current ACS-SRC (2012e2016).10 The risk differences were calculated by subtracting the SURPAS risk estimates (i.e., probabilities) from the ACS-SRC estimates, both rounded to the first decimal place. A positive value means that the ACS-SRC risk estimate is higher than the SURPAS risk estimate, while a negative value means that the SURPAS risk estimate is higher. Absolute values of the risk differences were also taken as an additional measure. Means, medians, and the interquartile ranges (IQRs) were calculated for ACS-SRC and SURPAS risk estimates and for the risk differences and absolute value of the risk differences. Histograms were used to plot the risk differences and BlandAltman plots,11,12 a graphical method to compare two measurement techniques. In this graphical method, the differences between the two measurements (along the vertical axis) are plotted against the average of the two measurements (along the horizontal axis). The ACS-SRC and SURPAS expected and observed events (30-day mortality and overall morbidity) were compared by quintiles of risk. A Hosmer-Lemeshow p-value, the c-index, and Brier scores between the ACS-SRC and SURPAS systems were also computed. We expected that the results for mortality would be tentative because the number of events would be low, but would be more reasonable for overall morbidity. The study was exempted by our institutional review board because the data used were deidentified. Results The target sample size of 1,000 patients was exceeded (n ¼ 1,006) due to the data being entered by two coauthors simultaneously. Characteristics of the 1,006 patients studied are presented in Table 1. This cohort included 589 females (58.6%) and 417 (41.4%) males. The majority of patients were non-Hispanic white (67.6%) followed by black (9.9%),Hispanic (5.7%), Asian or Pacific islander (3.6%), American Indian or Alaskan Native (0.3%) and unknown (12.9%). Mean age of this patient sample was 55.8 with the standard deviation of 16.5, 98% of whom were functionally independent at the time of surgery. Only 7.6% of patients underwent emergency operations. Patient's ASA class was predominately 2 and 3 (ASA 2: 49.0%, ASA 3: 39.1%). Surgical specialty representation was 52.1% from general surgery, 19.3% orthopedics, 8.5% gynecology, 5.9% vascular, and <5% for each of the four other specialties. The 30-day overall morbidity of the sample was 10.9%, and the mortality was 0.6%. Patient characteristics were very similar between the random sample and the total NSQIP population of 5.2 million patients. Because the annual intake of patients and
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
3
Table 1 Patient Characteristics of the study cohort and the National ACS NSQIP database, 2007e2016. Characteristics Demographics Gender Female Male Age, years, mean (SD) Race/Ethnicity White, not of Hispanic origin Black, not of Hispanic origin Hispanic origin Asian or Pacific Islander American Indian or Alaska Native Unknown Body mass index category Underweight (<18.5) Normal weight (18.5e24.9) Overweight (25.0e29.9) Obese class I (30.0e34.9) Obese class II (35.0e39.9) Obese class III (40.0) Comorbidities Functional health status before surgery Independent Partially dependent Totally dependent Steroid use for chronic condition Ascites (within 30 days) Systemic sepsis None Systemic inflammatory response syndrome (SIRS) Sepsis Septic Shock Ventilator dependent (within 48 h) Disseminated cancer Diabetes mellitus None Oral medication Insulin Blood pressure >140/90 Hg or taking antihypertensive medications Congestive heart failure (within 30 days) Dyspnea (within 30 days) None Moderate exertion At rest Cigarette smoker (within 1 year) Severe chronic obstructive pulmonary disease (COPD) Dialysis or hemofiltration (within 2 weeks) Acute renal failure (rising creatinine to >3 mg/dL within 24 h) Operative Emergency operation ASA class I (a normal healthy patient) II (a patient with mild systemic disease) III (a patient with severe systemic disease) IV (a patient with severe systemic disease that is a constant threat) V (a moribund patient) Primary surgeon specialty General surgery Orthopedic surgery Gynecologic surgery Vascular surgery Urologic surgery Neurosurgery Plastics Otolaryngology Thoracic surgery Work relative value unit, mean (SD) Outcomes 30-day mortality 30-day morbidity
Cohort (N ¼ 1,006) N (%)a
Nationalb(N ¼ 5,242,344) N (%)a
589 (58.6) 417 (41.4) 55.8 (16.5)
3,001,359 (57.2) 2,240,985 (42.8) 56.4 (16.8)
680 (67.6) 100 (9.9) 57 (5.7) 36 (3.6) 3 (0.3) 130 (12.9)
3,615,499 (69.0) 516,387 (9.9) 299,805 (5.7) 155,985 (3.0) 31,236 (0.6) 623,432 (11.9)
16 (1.6) 210 (20.9) 335 (33.3) 197 (19.6) 130 (12.9) 118 (11.7)
92,983 (1.8) 1,289,382 (24.6) 1,633,934 (31.2) 1,093,467 (20.9) 574,100 (11.0) 558,478 (10.7)
986 (98.0) 15 (1.5) 5 (0.5) 29 (2.9) 1 (0.1)
5,059,555 (96.5) 144,191 (2.8) 38,598 (0.7) 181,969 (3.5) 26,040 (0.5)
960 (95.4) 29 (2.9) 14 (1.4) 3 (0.3) 1 (0.1) 22 (2.2)
4,941,326 (94.3) 171,423 (3.3) 104,728 (2.0) 24,867 (0.5) 23,473 (0.5) 114,750 (2.2)
853 (84.8) 100 (9.9) 53 (5.3) 443 (44.0) 7 (0.7)
4,441,069 (84.7) 487,778 (9.5) 303,497 (5.8) 2,389,348 (45.6) 41,675 (0.8)
953 (97.7) 51 (5.1) 2 (0.2) 191 (19.0) 33 (3.3) 10 (1.0) 0 (0)
4,880,051 (93.1) 327,537 (6.3) 34,756 (0.7) 971,738 (18.5) 241,023 (4.6) 77,240 (1.5) 20,829 (0.4)
76 (7.6)
486,638 (9.3)
94 (9.3) 493 (49.0)
473,840 (9.0) 2,392,324 (45.6) 393 (39.1) 26 (2.6) 0 (0)
524 (52.1) 194 (19.3) 85 (8.5) 59 (5.9) 48 (4.8)
2,070,065 (39.5) 297,022 (5.7) 9,093 (0.2)
16.6 (8.6)
2,781,965 (53.1) 913,667 (17.4) 342,996 (6.5) 407,065 (7.8) 254,755 (4.9) 223,576 (4.3) 129,652 (2.5) 130,269 (2.5) 58,399 (1.1) 16.4 (9.1)
6 (0.6) 110 (10.9)
56,764 (1.1) 631,529 (12.1)
35 29 19 13
(3.5) (2.9) (1.9) (1.3)
Abbreviations: ASA class, American Society of Anesthesiology physical status classification. a All values are frequency and column percent unless otherwise specified. b All complete observations from the National Surgical Quality Improvement Participate User File from years 2007e2016.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
4
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
hospitals into the NSQIP database has increased each year, the vast majority of patients (84.9%) came from the 2012e2016 period, similar to when the ACS-SRC and SURPAS models for this paper were developed. The mean SURPAS risk for mortality was 0.61% (interquartile range (IQR): 0e0.3%) as compared to the ACS-SRC value of 0.59% (IQR: 0e0.5%) (Table 2). The mean difference between the two was 0.02% and mean absolute difference was 0.37%. The mean SURPAS and ACS-SRC risks were quite close for patients who died (12.6% vs. 13.7%) and did not die (0.54% vs. 0.51%). The mean SURPAS risk for overall morbidity was 10.97% (IQR: 2.6e12.5%) as compared to the ACS-SRC value of 7.73% (IQR: 2.8e9.0%). The mean difference between the two was 3.24% and mean absolute difference was 4.55%. The mean SURPAS risk for patients who did not have a complication was slightly more than the ACS-SRC risk (9.06% vs. 6.69%), but was much higher for patients who did have a complication (26.53% vs. 16.27%). Fig. 1a shows the distribution of differences in the individual patient 30-day mortality estimates between the ACS-SRC- and SURPAS-determined values. These most commonly were 0.0. When differences were nonzero, there were more negative (n ¼ 466) vs.
Table 2 30-day Mortality and Morbidity Risk Estimates and differences and absolute differences between ACS-SRC and SURPAS risk calculators (n ¼ 1,006). Risk Estimate 30-Day Mortality ACS-SRC risk SURPAS risk Difference between risk estimates Absolute difference between risk estimates
Did not die within 30-days ACS-SRC risk
Mean (SD)
Median (IQR)
0.59 (3.09) 0.61 (3.03) 0.02 (1.29) 0.37 (1.23)
0.1 (0e0.3)
0.51 (2.32)
0.1 (0e0.3)
SURPAS risk Died within 30-days ACS-SRC risk SURPAS risk
30-Day Morbidity ACS-SRC risk SURPAS risk Difference between risk estimates Absolute difference between risk estimates
Did not have a complication within 30-days ACS-SRC risk SURPAS risk Had a complication within 30-days ACS-SRC risk SURPAS risk
0.1 (0e0.5) 0 (0.2e0) 0.1 (0e0.3)
0.54 (2.12)
0.1 (0e0.5)
13.7 4.3 (0.5e7.6) (25.3) 12.6 1.6 (0.4e2.5) (27.9)
7.73 (8.06) 10.97 (12.84) 3.24 (7.21) 4.55 (6.46)
5.2 (2.8e9.0)
6.69 (6.63) 9.06 (10.04)
4.7 (2.6e7.9)
16.27 (12.46) 26.53 (20.47)
12.1 (7.0e23.9)
7.2 (2.6e12.5) 1.4 (4.8e0.5)
positive (n ¼ 154) differences, indicating that the SURPAS mortality risk more often had higher values compared to the ACS NSQIP values. Fig. 1b shows the Bland-Altman plot of these same differences plotted against the average of the mortality estimates by the ACS-SRC and SURPAS. With only a few exceptions, most of the points cluster close to zero and within two standard deviations of the mean difference of 0.1. Fig. 1c shows the distribution of differences in the individual patient 30-day morbidity estimates between the ACS-SRC and SURPAS-determined values. These had a broader range than that of the mortality differences. When the differences were nonzero, they also tended to be more frequently negative (n ¼ 645) vs. positive (n ¼ 21), indicating higher morbidity estimates for SURPAS compared to ACS-SRC. Fig. 1d shows the Bland-Altman plot of these same differences plotted against the average of the morbidity estimates by the ACS-SRC and SURPAS. Although most of the points again were within two standard deviations of the mean difference of 3.2, there was more scatter in the mean morbidity differences than in the mean mortality differences. Also, most of the points outside of the dense clustering were on the negative side, indicating a higher risk estimate for SURPAS compared to the ACS-SRC, mostly among the higher risk patients. Fig. 2 compares the goodness of fit analysis for 30-day mortality estimates between SURPAS and the ACS-SRC. The expected vs. observed event curves and p-values in the Hosmer-Lemeshow analysis are close for both SURPAS (p ¼ 0.12) and the ACS-SRC (p ¼ 0.29). Also, the Brier scores are quite close (SURPAS ¼ 0.0054 vs. ACS-SRC ¼ 0.0053). However, the c-index for the ACS-SRC was 0.937 vs. 0.853 for SURPAS (p ¼ 0.06). These results should be considered “tentative,” however, because they are based upon only 6 deaths in the cohort of 1,006 patients. In the same analysis for the 30-day overall morbidity presented in Fig. 3, the c-index of the SURPAS model was 0.805, with a Hosmer-Lemeshow p-value of 0.09, and Brier score of 0.0798. In comparison, the c-index of the ACS-SRC was 0.797, with a HosmerLemeshow p-value of 0.02, and Brier score of 0.0862. There was no significant difference between the c-indexes of the two (p ¼ 0.65). The ACS-SRC appeared to underestimate the morbidity risk for patients in the fourth and fifth quintiles compared to SURPAS (p ¼ 0.02). We performed one additional analysis to determine if the underestimation of overall morbidity risk by the ACS-SRC might be due to period effects (i.e., a risk calculator developed in a later period might underestimate risks of patients from an earlier period if actual morbidity is decreasing over time) rather than miscalibration. We compared mean patient overall morbidity risk for both the SURPAS and ACS-SRC vs. the mean overall morbidity risk for each patient as given in the NSQIP PUF. The results were: SURPAS 11.0; ACS-SRC 7.7; and NSQIP PUF 5.9. Thus, the underestimation does not seem to be due to period effects, but possibly due to miscalibration.
2.7 (0.8e5.3)
Discussion
6.5 (2.4e11.1)
23.2 (9.6e37.9)
*Abbreviations: SD, standard deviation; IQR, interquartile range; ACS-SRC, American College of Surgeons Surgical Risk Calculator; SURPAS, Surgical Risk Preoperative Assessment System.
The purpose of this study was to compare risk estimates for 30day postoperative mortality and overall morbidity for our SURPAS preoperative risk prediction tool to the ACS-SRC for a random sample of patients from the 2007e2016 ACS NSQIP PUF. The size of the sample (1,006 patients) was limited logistically because of the need to manually enter 21 data items for each patient into the ACSSRC. The SURPAS risk estimates were somewhat higher than the ACS-SRC risk estimates, particularly for overall morbidity. In comparing expected vs. actual events across quintiles of risk, the ACS-SRC appeared to underestimate morbidity risk, particularly for patients at the highest risk for morbidity (fourth and fifth
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
5
Fig. 1. a) Histogram for 30-day mortality. Values on the x-axis are ACS-SRC mortality estimates minus SURPAS mortality estimates for 1,006 patients. Negative numbers mean that the SURPAS estimates are higher than the ACS-SRC estimates; positive numbers mean that the ACS-SRC estimates are higher than the SURPAS estimates. Values on the y-axis are numbers of patients. b) Bland Altman Plot for 30-day mortality. Values on the x-axis are the average of the ACS-SRC and SURPAS mortality estimates for 1,006 patients. Values on the y-axis are differences in the ACS-SRC and SURPAS mortality estimates. Negative numbers mean that the SURPAS estimates are higher than the ACS-SRC estimates; positive numbers mean that the ACS-SRC estimates are higher than the SURPAS estimates. The plot shows the differences in the two estimates as a function of the magnitude of the estimates. c and d) Histogram and Bland Altman Plot for 30-day overall morbidity.
quintiles). Comparison of the goodness-of-fit statistics (c-index, Hosmer-Lemeshow p-value, and Brier score) between the ACS-SRC and SURPAS for mortality showed they were generally similar. The c-index for the ACS-SRC was a little higher than that for SURPAS, but results for mortality were tentative because they were based on only six deaths. In other SURPAS work, the findings indicated the SURPAS c-indexes for mortality to generally be > 0.90.4 Both prediction systems have the advantages that they use a large national database to generate their prediction models, and that one prediction model can be used for patients across a broad spectrum of different types of operations. From this limited work, the conclusion is that the SURPAS risk estimates are comparable to the ACS-SRC risk estimates, although it appears that the ACS-SRC may underestimate overall morbidity risk for high risk patients. The advantages of SURPAS are that it requires one-fourth of the number of predictor variables compared to the ACS-SRC, it has been successfully integrated into the EHR at the University of Colorado Health System, it compares the patient's individual risk to national averages for patients undergoing the same operation, and it automatically writes a preoperative note in the patient electronic health record and provides the patient with a graphical display of her/his
risks. SURPAS has been used in over 3,000 patients so far at our medical center, and we hope in the future for a wider dissemination and implementation of the tool. The authors do not know why the ACS-SRC appeared to underestimate morbidity risk for the higher risk patients. The ACSSRC uses many of the same predictor variables as does SURPAS, plus additional specific comorbidities, so one would expect that the ACS-SRC would be at least the same or superior. This could be an area for future research. The SURPAS tool was built into the UCHealth EHR by a consulting IT company (AgileMD, San Francisco, CA) under contract with UCHealth. Although it is currently available only at UCHealth, it might be possible for other interested surgical programs to develop similar arrangements with the vendor for them to build in the SURPAS tool into their center's EHR as well, to enable the SURPAS tool to be more widely available and utilized. The authors, who developed SURPAS, do not derive financial gain from its use. There are a number of important limitations of this study that could be addressed in future studies: (1) The results are applicable to a broad surgical population, but it would be useful to know how the risk calculators compare for individual types of operations, and
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
6
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
Fig. 2. Hosmer-Lemeshow Graphs and Fit Statistics for ACS-SRC (upper graph) and SURPAS (lower graph) estimates of 30-day postoperative mortality. Values on the x-axis are quintiles of risk. Values on the y-axis are average probability of 30-day mortality. The solid line represents the observed mortality rate for each quintile of risk. The dotted line represents the estimated mortality rate from the ACS-SRC and SURPAS risk calculators. In each graph, the closer the observed and expected graphs are to each other, the better the calibration of the ACS-SRC and SURPAS risk tools. A higher c-index, higher H-L p-value, and lower Brier score indicate better model fit* p value for the difference between C -index was 0.06 Values are sample sizes at each quantile.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
7
Fig. 3. Hosmer-Lemeshow Graphs and Fit Statistics for ACS-SRC (upper graph) and SURPAS (lower graph) estimates of 30-day overall morbidity. The interpretations of these graphs are the same as in Fig. 2. *p value for the difference between C -index was 0.65. Values are sample sizes at each quantile.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
8
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
also how they compare to more disease- and operation-specific risk calculators existing in the literature; (2) The analysis was limited to only two outcomes. SURPAS had four other outcomes in common with the ACS-SRC that were not studied due to the manual collection of the data (cardiac, UTI, VTE, and renal failure), and SURPAS has also recently added unplanned readmission and discharge destination. These could be the subject of future studies; (3) The sample size of the study was relatively small necessitated by the labor of manually entering 21 variables into the ACS-SRC for each patient. Studies with larger sample sizes would be possible if the details of the ACS-SRC models were published; and (4) The results for mortality were inconclusive because there were so few deaths in a sample of 1,006 patients. Future, much larger studies are needed for mortality. Conclusion The SURPAS risk estimates for postoperative mortality and overall morbidity appear to be comparable to those of the ACS-SRC. The ACS-SRC actually underestimates overall morbidity risk for the higher risk patients. The advantages of SURPAS are that it requires manual entry of only one-fourth the number of variables, is integrated into the EHR at UCHealth, provides a preoperative note in the patient record, and provides a graphical display of patient risk for the patients to take home. Funding This work was supported by a grant from the Agency for Healthcare Research and Quality (AHRQ 1R21HS024124-02). Conflicts of interest
References 1. Hammermeister KE, Henderson WG, Bronsert MR, Juarez-Colunga E, Meguid RA. Bringing quantitative risk assessment closer to the patient and surgeon: a novel approach to improve outcomes. Ann Surg. 2016;263(6): 1039e1041. 2. Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical risk preoperative assessment system (SURPAS): I. Parsimonious, clinically meaningful groups of postoperative complications by factor Analysis. Ann Surg. 2016;263(6):1042e1048. 3. Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical risk preoperative assessment system (SURPAS): II. Parsimonious risk models for postoperative adverse outcomes addressing need for laboratory variables and surgeon specialty-specific models. Ann Surg. 2016;264(1):10e22. 4. Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical risk preoperative assessment system (SURPAS): III. Accurate preoperative prediction of 8 adverse outcomes using 8 predictor variables. Ann Surg. 2016;264(1):23e31. 5. Lambert-Kerzner A, Ford KL, Hammermeister KE, Henderson WG, Bronsert MR, Meguid RA. Assessment of attitudes towards future implementation of the surgical risk preoperative assessment system (SURPAS) tool: a qualitative study. Patient Saf Surg. 2018;12:12. 6. Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217(5), 833-842 e831-833. 7. Cohen ME, Liu Y, Ko CY, Hall BL. An examination of American College of surgeons NSQIP surgical risk calculator accuracy. J Am Coll Surg. 2017;224(5): 787e795 e781. 8. Khuri SF, Daley J, Henderson W, et al. Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care. Results of the National Veterans Affairs Surgical Risk Study. J Am Coll Surg. 1997;185:315e327. 9. Daley J, Khuri SF, Henderson W, et al. Risk adjustment of the postoperative morbidity rate for the comparative assessment of the quality of surgical care. Results of the National Veterans Affairs Surgical Risk Study. J Am Coll Surg. 1997;185:328e340. 10. American College of Surgeons. American College of Surgeons Risk Calculator; 2018. https://riskcalculator.facs.org/RiskCalculator/. Accessed April 15, 2018. 11. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307e310. 12. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(2):135e160.
The authors report no conflicts of interest to disclose. The authors of this manuscript and the developers of SURPAS have no financial interests in the SURPAS product.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
Contents lists available at ScienceDirect
The American Journal of Surgery journal homepage: www.americanjournalofsurgery.com
Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator Sina Khaneki a, Michael R. Bronsert a, c, William G. Henderson a, c, e, Maryam Yazdanfar a, Anne Lambert-Kerzner a, b, Karl E. Hammermeister a, c, d, Robert A. Meguid a, c, f, * a
Surgical Outcomes and Applied Research Program, University of Colorado School of Medicine, Aurora, CO, USA Colorado School of Public Health, University of Colorado, Aurora, CO, USA Adult and Child Center for Health Outcomes Research and Delivery Science, University of Colorado School of Medicine, Aurora, CO, USA d Division of Cardiology, Department of Medicine, University of Colorado School of Medicine, Aurora, CO, USA e Department of Biostatistics and Informatics, Colorado School of Public Health, Aurora, CO, USA f Department of Surgery, University of Colorado School of Medicine, Aurora, CO, USA b c
a r t i c l e i n f o
a b s t r a c t
Article history: Received 1 May 2019 Received in revised form 24 July 2019 Accepted 27 July 2019
Background: The novel Surgical Risk Preoperative Assessment System (SURPAS) requires entry of five predictor variables (the other three variables of the eight-variable model are automatically obtained from the electronic health record or a table look-up), provides patient risk estimates compared to national averages, is integrated into the electronic health record, and provides a graphical handout of risks for patients. The accuracy of the SURPAS tool was compared to that of the American College of Surgeons Surgical Risk Calculator (ACS-SRC). Methods: Predicted risk of postoperative mortality and morbidity was calculated using both SURPAS and ACS-SRC for 1,006 randomly selected 2007e2016 ACS National Surgical Quality Improvement Program (NSQIP) patients with known outcomes. C-indexes, Hosmer-Lemeshow graphs, and Brier scores were compared between SURPAS and ACS-SRC. Results: ACS-SRC risk estimates for overall morbidity underestimated risk compared to observed postoperative overall morbidity, particularly for the highest risk patients. SURPAS accurately estimates morbidity risk compared to observed morbidity. Conclusions: SURPAS risk predictions were more accurate than ACS-SRC's for overall morbidity, particularly for high risk patients. Summary: The accuracy of the SURPAS tool was compared to that of the American College of Surgeons Surgical Risk Calculator (ACS-SRC). SURPAS risk predictions were more accurate than those of the ACSSRC for overall morbidity, particularly for high risk patients. © 2019 Published by Elsevier Inc.
Keywords: SURPAS Surgical risk prediction Postoperative outcomes Comparative effectiveness Accuracy Risk assessment
Introduction The SUrgical Risk Preoperative Assessment System (SURPAS), which has been developed over the past six years, requires the input of five preoperative predictor variables (the other three variables in the eight-variable model are automatically obtained from
* Corresponding author. Division of Cardiothoracic Surgery, Department of Surgery, University of Colorado Denver, Anschutz Medical Campus, 12631 E, 17th Avenue, C-310, Aurora, CO, 80045, USA. E-mail address: [email protected] (R.A. Meguid).
the EHR or a table look-up) to assess the patient's risk of 11 different postoperative adverse outcomes.1e5 The strengths of the SURPAS tool include: (1) It is based upon analysis of the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) participant use file that includes operations in over 6 million patients from over 700 medical centers in the U.S. and abroad; (2) It uses one parsimonious model requiring manual input of only five of the eight required variables readily ascertained during the preoperative visit; (3) It is applicable to a broad range of surgical operations from nine surgical specialties; (4) It is built into the electronic health record (EHR) at the University of Colorado Health (UCHealth) System; (5) It compares the patient's individual
https://doi.org/10.1016/j.amjsurg.2019.07.036 0002-9610/© 2019 Published by Elsevier Inc.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
2
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
risks to the national average patient undergoing the same operation; (6) It automatically writes a preoperative risk discussion note into the patient's EHR; and (7) It produces a paper handout of the patient's risk compared to national averages that can be given to the patient. SURPAS has currently been used at the University of Colorado Hospital in over 3,000 patients by over 90 different providers, and we are working on further dissemination and implementation of SURPAS in the UCHealth system. Surgical providers have asked how the SURPAS tool compares with the ACS Surgical Risk Calculator (ACS-SRC).6,7 A typical outpatient surgical clinic schedule might include 20 patients in 5 h or less; this limited time has to be efficiently used to allow for high quality patient care and patient centered practice. The ACS-SRC requires manual entry of 21 variables including CPT code, age group, gender, functional status, emergency case, American Society of Anesthesiology physical status classification (ASA class), steroid use for a chronic condition, ascites within 30 days prior to surgery, systemic sepsis within 48 h prior to surgery, ventilation dependence, disseminated cancer, diabetes, hypertension requiring medication, congestive heart failure in 30 days prior to surgery, dyspnea, current smoker within one year, history of severe chronic obstructive pulmonary disease, dialysis, acute renal failure, height and weight. Chart review to gather these data items is a tedious task that is challenging in a busy practice. In comparison, SURPAS requires manual entry of only five variables (CPT code of primary operation from a drop-down listdthis provides workRVU of the operation and CPT-specific event rate from the ACS NSQIP database by table look-up), inpatient/outpatient setting, emergency status of the operation, ASA class, and functional health status) and the other variables (patient age and specialty of primary surgeon) are auto populated from the patient's electronic health record. In practice, we have found that SURPAS takes less than 1 min to accomplish compared to considerably more time for the ACS-SRC. SURPAS provides estimates of 11 30-day postoperative adverse outcomes including: mortality, overall morbidity, unplanned readmission, infectious, urinary tract infection (UTI), cardiac, bleeding, pulmonary, venous thromboembolic events (VTE), renal, and stroke complications. This consolidation of the 18 individual postoperative complications in the ACS NSQIP was done using factor analysis of the ACS NSQIP database. (2) The ACS-SRC provides risks for a slightly different set of 13 adverse postoperative outcomes: serious complication, any complication, pneumonia, cardiac complication, surgical site infection (SSI), UTI, VTE, renal failure, readmission, unplanned return to operating room, death, discharge to nursing or rehabilitation facility, and predicted length of stay. An important issue is, “How well do the SURPAS and ACS-SRC risk estimates compare?” The purpose of this study was to make this comparison in about 1,000 patients randomly selected from the ACS NSQIP database for two of the outcomes that are common between SURPAS and the ACS-SRC, postoperative mortality and overall morbidity. We hypothesized that SURPAS is as accurate at risk prediction as the ACS-SRC, while having the advantages of requiring manual entry of one-fourth of the predictor variables, being integrated into the EHR, comparing a patient's individual risks to national averages for patients undergoing the same operation, and automatically generating a preoperative note and a graphical display of the patient's risks for the patient to take home. Material and methods A limited sample size of approximately 1,000 patients was randomly selected from the ACS NSQIP database Participant Use File (PUF) from 2007 to 2016, which includes over 6 million patients from over 700 participating medical centers, for patients who had complete data for all of the variables needed to calculate risk from
both of the risk calculators. A simple random sample without replacement of the total database was used instead of randomly selecting patients by characteristics (for example, high risk patients or from certain types of operations or specialties) because the two risk calculators are intended to be used in any surgical patient. The random sample was obtained by using the SAS “proc surveyselect” statement. The outcome variables were limited to 30-day mortality and overall morbidity because these were identically defined in the SURPAS and ACS-SRC prediction systems and they have long been considered the two most important outcomes of the NSQIP.8,9 Overall morbidity was considered to be the primary outcome variable because it has the highest incidence of any of the NSQIP postoperative adverse outcomes; mortality was considered a secondary outcome. The ACS-SRC risk estimates were obtained by manually entering the 21 predictor variables into the ACS-SRC for each patient by two of the coauthors (SK, MY). The SURPAS risk estimates were obtained by computer calculation using the patient variables and the SURPAS models generated using logistic regression equations.4 The SURPAS model developed on ACS NSQIP data from 2012 to 2015 was used because it was a similar period to that reported for the current ACS-SRC (2012e2016).10 The risk differences were calculated by subtracting the SURPAS risk estimates (i.e., probabilities) from the ACS-SRC estimates, both rounded to the first decimal place. A positive value means that the ACS-SRC risk estimate is higher than the SURPAS risk estimate, while a negative value means that the SURPAS risk estimate is higher. Absolute values of the risk differences were also taken as an additional measure. Means, medians, and the interquartile ranges (IQRs) were calculated for ACS-SRC and SURPAS risk estimates and for the risk differences and absolute value of the risk differences. Histograms were used to plot the risk differences and BlandAltman plots,11,12 a graphical method to compare two measurement techniques. In this graphical method, the differences between the two measurements (along the vertical axis) are plotted against the average of the two measurements (along the horizontal axis). The ACS-SRC and SURPAS expected and observed events (30-day mortality and overall morbidity) were compared by quintiles of risk. A Hosmer-Lemeshow p-value, the c-index, and Brier scores between the ACS-SRC and SURPAS systems were also computed. We expected that the results for mortality would be tentative because the number of events would be low, but would be more reasonable for overall morbidity. The study was exempted by our institutional review board because the data used were deidentified. Results The target sample size of 1,000 patients was exceeded (n ¼ 1,006) due to the data being entered by two coauthors simultaneously. Characteristics of the 1,006 patients studied are presented in Table 1. This cohort included 589 females (58.6%) and 417 (41.4%) males. The majority of patients were non-Hispanic white (67.6%) followed by black (9.9%),Hispanic (5.7%), Asian or Pacific islander (3.6%), American Indian or Alaskan Native (0.3%) and unknown (12.9%). Mean age of this patient sample was 55.8 with the standard deviation of 16.5, 98% of whom were functionally independent at the time of surgery. Only 7.6% of patients underwent emergency operations. Patient's ASA class was predominately 2 and 3 (ASA 2: 49.0%, ASA 3: 39.1%). Surgical specialty representation was 52.1% from general surgery, 19.3% orthopedics, 8.5% gynecology, 5.9% vascular, and <5% for each of the four other specialties. The 30-day overall morbidity of the sample was 10.9%, and the mortality was 0.6%. Patient characteristics were very similar between the random sample and the total NSQIP population of 5.2 million patients. Because the annual intake of patients and
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
3
Table 1 Patient Characteristics of the study cohort and the National ACS NSQIP database, 2007e2016. Characteristics Demographics Gender Female Male Age, years, mean (SD) Race/Ethnicity White, not of Hispanic origin Black, not of Hispanic origin Hispanic origin Asian or Pacific Islander American Indian or Alaska Native Unknown Body mass index category Underweight (<18.5) Normal weight (18.5e24.9) Overweight (25.0e29.9) Obese class I (30.0e34.9) Obese class II (35.0e39.9) Obese class III (40.0) Comorbidities Functional health status before surgery Independent Partially dependent Totally dependent Steroid use for chronic condition Ascites (within 30 days) Systemic sepsis None Systemic inflammatory response syndrome (SIRS) Sepsis Septic Shock Ventilator dependent (within 48 h) Disseminated cancer Diabetes mellitus None Oral medication Insulin Blood pressure >140/90 Hg or taking antihypertensive medications Congestive heart failure (within 30 days) Dyspnea (within 30 days) None Moderate exertion At rest Cigarette smoker (within 1 year) Severe chronic obstructive pulmonary disease (COPD) Dialysis or hemofiltration (within 2 weeks) Acute renal failure (rising creatinine to >3 mg/dL within 24 h) Operative Emergency operation ASA class I (a normal healthy patient) II (a patient with mild systemic disease) III (a patient with severe systemic disease) IV (a patient with severe systemic disease that is a constant threat) V (a moribund patient) Primary surgeon specialty General surgery Orthopedic surgery Gynecologic surgery Vascular surgery Urologic surgery Neurosurgery Plastics Otolaryngology Thoracic surgery Work relative value unit, mean (SD) Outcomes 30-day mortality 30-day morbidity
Cohort (N ¼ 1,006) N (%)a
Nationalb(N ¼ 5,242,344) N (%)a
589 (58.6) 417 (41.4) 55.8 (16.5)
3,001,359 (57.2) 2,240,985 (42.8) 56.4 (16.8)
680 (67.6) 100 (9.9) 57 (5.7) 36 (3.6) 3 (0.3) 130 (12.9)
3,615,499 (69.0) 516,387 (9.9) 299,805 (5.7) 155,985 (3.0) 31,236 (0.6) 623,432 (11.9)
16 (1.6) 210 (20.9) 335 (33.3) 197 (19.6) 130 (12.9) 118 (11.7)
92,983 (1.8) 1,289,382 (24.6) 1,633,934 (31.2) 1,093,467 (20.9) 574,100 (11.0) 558,478 (10.7)
986 (98.0) 15 (1.5) 5 (0.5) 29 (2.9) 1 (0.1)
5,059,555 (96.5) 144,191 (2.8) 38,598 (0.7) 181,969 (3.5) 26,040 (0.5)
960 (95.4) 29 (2.9) 14 (1.4) 3 (0.3) 1 (0.1) 22 (2.2)
4,941,326 (94.3) 171,423 (3.3) 104,728 (2.0) 24,867 (0.5) 23,473 (0.5) 114,750 (2.2)
853 (84.8) 100 (9.9) 53 (5.3) 443 (44.0) 7 (0.7)
4,441,069 (84.7) 487,778 (9.5) 303,497 (5.8) 2,389,348 (45.6) 41,675 (0.8)
953 (97.7) 51 (5.1) 2 (0.2) 191 (19.0) 33 (3.3) 10 (1.0) 0 (0)
4,880,051 (93.1) 327,537 (6.3) 34,756 (0.7) 971,738 (18.5) 241,023 (4.6) 77,240 (1.5) 20,829 (0.4)
76 (7.6)
486,638 (9.3)
94 (9.3) 493 (49.0)
473,840 (9.0) 2,392,324 (45.6) 393 (39.1) 26 (2.6) 0 (0)
524 (52.1) 194 (19.3) 85 (8.5) 59 (5.9) 48 (4.8)
2,070,065 (39.5) 297,022 (5.7) 9,093 (0.2)
16.6 (8.6)
2,781,965 (53.1) 913,667 (17.4) 342,996 (6.5) 407,065 (7.8) 254,755 (4.9) 223,576 (4.3) 129,652 (2.5) 130,269 (2.5) 58,399 (1.1) 16.4 (9.1)
6 (0.6) 110 (10.9)
56,764 (1.1) 631,529 (12.1)
35 29 19 13
(3.5) (2.9) (1.9) (1.3)
Abbreviations: ASA class, American Society of Anesthesiology physical status classification. a All values are frequency and column percent unless otherwise specified. b All complete observations from the National Surgical Quality Improvement Participate User File from years 2007e2016.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
4
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
hospitals into the NSQIP database has increased each year, the vast majority of patients (84.9%) came from the 2012e2016 period, similar to when the ACS-SRC and SURPAS models for this paper were developed. The mean SURPAS risk for mortality was 0.61% (interquartile range (IQR): 0e0.3%) as compared to the ACS-SRC value of 0.59% (IQR: 0e0.5%) (Table 2). The mean difference between the two was 0.02% and mean absolute difference was 0.37%. The mean SURPAS and ACS-SRC risks were quite close for patients who died (12.6% vs. 13.7%) and did not die (0.54% vs. 0.51%). The mean SURPAS risk for overall morbidity was 10.97% (IQR: 2.6e12.5%) as compared to the ACS-SRC value of 7.73% (IQR: 2.8e9.0%). The mean difference between the two was 3.24% and mean absolute difference was 4.55%. The mean SURPAS risk for patients who did not have a complication was slightly more than the ACS-SRC risk (9.06% vs. 6.69%), but was much higher for patients who did have a complication (26.53% vs. 16.27%). Fig. 1a shows the distribution of differences in the individual patient 30-day mortality estimates between the ACS-SRC- and SURPAS-determined values. These most commonly were 0.0. When differences were nonzero, there were more negative (n ¼ 466) vs.
Table 2 30-day Mortality and Morbidity Risk Estimates and differences and absolute differences between ACS-SRC and SURPAS risk calculators (n ¼ 1,006). Risk Estimate 30-Day Mortality ACS-SRC risk SURPAS risk Difference between risk estimates Absolute difference between risk estimates
Did not die within 30-days ACS-SRC risk
Mean (SD)
Median (IQR)
0.59 (3.09) 0.61 (3.03) 0.02 (1.29) 0.37 (1.23)
0.1 (0e0.3)
0.51 (2.32)
0.1 (0e0.3)
SURPAS risk Died within 30-days ACS-SRC risk SURPAS risk
30-Day Morbidity ACS-SRC risk SURPAS risk Difference between risk estimates Absolute difference between risk estimates
Did not have a complication within 30-days ACS-SRC risk SURPAS risk Had a complication within 30-days ACS-SRC risk SURPAS risk
0.1 (0e0.5) 0 (0.2e0) 0.1 (0e0.3)
0.54 (2.12)
0.1 (0e0.5)
13.7 4.3 (0.5e7.6) (25.3) 12.6 1.6 (0.4e2.5) (27.9)
7.73 (8.06) 10.97 (12.84) 3.24 (7.21) 4.55 (6.46)
5.2 (2.8e9.0)
6.69 (6.63) 9.06 (10.04)
4.7 (2.6e7.9)
16.27 (12.46) 26.53 (20.47)
12.1 (7.0e23.9)
7.2 (2.6e12.5) 1.4 (4.8e0.5)
positive (n ¼ 154) differences, indicating that the SURPAS mortality risk more often had higher values compared to the ACS NSQIP values. Fig. 1b shows the Bland-Altman plot of these same differences plotted against the average of the mortality estimates by the ACS-SRC and SURPAS. With only a few exceptions, most of the points cluster close to zero and within two standard deviations of the mean difference of 0.1. Fig. 1c shows the distribution of differences in the individual patient 30-day morbidity estimates between the ACS-SRC and SURPAS-determined values. These had a broader range than that of the mortality differences. When the differences were nonzero, they also tended to be more frequently negative (n ¼ 645) vs. positive (n ¼ 21), indicating higher morbidity estimates for SURPAS compared to ACS-SRC. Fig. 1d shows the Bland-Altman plot of these same differences plotted against the average of the morbidity estimates by the ACS-SRC and SURPAS. Although most of the points again were within two standard deviations of the mean difference of 3.2, there was more scatter in the mean morbidity differences than in the mean mortality differences. Also, most of the points outside of the dense clustering were on the negative side, indicating a higher risk estimate for SURPAS compared to the ACS-SRC, mostly among the higher risk patients. Fig. 2 compares the goodness of fit analysis for 30-day mortality estimates between SURPAS and the ACS-SRC. The expected vs. observed event curves and p-values in the Hosmer-Lemeshow analysis are close for both SURPAS (p ¼ 0.12) and the ACS-SRC (p ¼ 0.29). Also, the Brier scores are quite close (SURPAS ¼ 0.0054 vs. ACS-SRC ¼ 0.0053). However, the c-index for the ACS-SRC was 0.937 vs. 0.853 for SURPAS (p ¼ 0.06). These results should be considered “tentative,” however, because they are based upon only 6 deaths in the cohort of 1,006 patients. In the same analysis for the 30-day overall morbidity presented in Fig. 3, the c-index of the SURPAS model was 0.805, with a Hosmer-Lemeshow p-value of 0.09, and Brier score of 0.0798. In comparison, the c-index of the ACS-SRC was 0.797, with a HosmerLemeshow p-value of 0.02, and Brier score of 0.0862. There was no significant difference between the c-indexes of the two (p ¼ 0.65). The ACS-SRC appeared to underestimate the morbidity risk for patients in the fourth and fifth quintiles compared to SURPAS (p ¼ 0.02). We performed one additional analysis to determine if the underestimation of overall morbidity risk by the ACS-SRC might be due to period effects (i.e., a risk calculator developed in a later period might underestimate risks of patients from an earlier period if actual morbidity is decreasing over time) rather than miscalibration. We compared mean patient overall morbidity risk for both the SURPAS and ACS-SRC vs. the mean overall morbidity risk for each patient as given in the NSQIP PUF. The results were: SURPAS 11.0; ACS-SRC 7.7; and NSQIP PUF 5.9. Thus, the underestimation does not seem to be due to period effects, but possibly due to miscalibration.
2.7 (0.8e5.3)
Discussion
6.5 (2.4e11.1)
23.2 (9.6e37.9)
*Abbreviations: SD, standard deviation; IQR, interquartile range; ACS-SRC, American College of Surgeons Surgical Risk Calculator; SURPAS, Surgical Risk Preoperative Assessment System.
The purpose of this study was to compare risk estimates for 30day postoperative mortality and overall morbidity for our SURPAS preoperative risk prediction tool to the ACS-SRC for a random sample of patients from the 2007e2016 ACS NSQIP PUF. The size of the sample (1,006 patients) was limited logistically because of the need to manually enter 21 data items for each patient into the ACSSRC. The SURPAS risk estimates were somewhat higher than the ACS-SRC risk estimates, particularly for overall morbidity. In comparing expected vs. actual events across quintiles of risk, the ACS-SRC appeared to underestimate morbidity risk, particularly for patients at the highest risk for morbidity (fourth and fifth
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
5
Fig. 1. a) Histogram for 30-day mortality. Values on the x-axis are ACS-SRC mortality estimates minus SURPAS mortality estimates for 1,006 patients. Negative numbers mean that the SURPAS estimates are higher than the ACS-SRC estimates; positive numbers mean that the ACS-SRC estimates are higher than the SURPAS estimates. Values on the y-axis are numbers of patients. b) Bland Altman Plot for 30-day mortality. Values on the x-axis are the average of the ACS-SRC and SURPAS mortality estimates for 1,006 patients. Values on the y-axis are differences in the ACS-SRC and SURPAS mortality estimates. Negative numbers mean that the SURPAS estimates are higher than the ACS-SRC estimates; positive numbers mean that the ACS-SRC estimates are higher than the SURPAS estimates. The plot shows the differences in the two estimates as a function of the magnitude of the estimates. c and d) Histogram and Bland Altman Plot for 30-day overall morbidity.
quintiles). Comparison of the goodness-of-fit statistics (c-index, Hosmer-Lemeshow p-value, and Brier score) between the ACS-SRC and SURPAS for mortality showed they were generally similar. The c-index for the ACS-SRC was a little higher than that for SURPAS, but results for mortality were tentative because they were based on only six deaths. In other SURPAS work, the findings indicated the SURPAS c-indexes for mortality to generally be > 0.90.4 Both prediction systems have the advantages that they use a large national database to generate their prediction models, and that one prediction model can be used for patients across a broad spectrum of different types of operations. From this limited work, the conclusion is that the SURPAS risk estimates are comparable to the ACS-SRC risk estimates, although it appears that the ACS-SRC may underestimate overall morbidity risk for high risk patients. The advantages of SURPAS are that it requires one-fourth of the number of predictor variables compared to the ACS-SRC, it has been successfully integrated into the EHR at the University of Colorado Health System, it compares the patient's individual risk to national averages for patients undergoing the same operation, and it automatically writes a preoperative note in the patient electronic health record and provides the patient with a graphical display of her/his
risks. SURPAS has been used in over 3,000 patients so far at our medical center, and we hope in the future for a wider dissemination and implementation of the tool. The authors do not know why the ACS-SRC appeared to underestimate morbidity risk for the higher risk patients. The ACSSRC uses many of the same predictor variables as does SURPAS, plus additional specific comorbidities, so one would expect that the ACS-SRC would be at least the same or superior. This could be an area for future research. The SURPAS tool was built into the UCHealth EHR by a consulting IT company (AgileMD, San Francisco, CA) under contract with UCHealth. Although it is currently available only at UCHealth, it might be possible for other interested surgical programs to develop similar arrangements with the vendor for them to build in the SURPAS tool into their center's EHR as well, to enable the SURPAS tool to be more widely available and utilized. The authors, who developed SURPAS, do not derive financial gain from its use. There are a number of important limitations of this study that could be addressed in future studies: (1) The results are applicable to a broad surgical population, but it would be useful to know how the risk calculators compare for individual types of operations, and
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
6
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
Fig. 2. Hosmer-Lemeshow Graphs and Fit Statistics for ACS-SRC (upper graph) and SURPAS (lower graph) estimates of 30-day postoperative mortality. Values on the x-axis are quintiles of risk. Values on the y-axis are average probability of 30-day mortality. The solid line represents the observed mortality rate for each quintile of risk. The dotted line represents the estimated mortality rate from the ACS-SRC and SURPAS risk calculators. In each graph, the closer the observed and expected graphs are to each other, the better the calibration of the ACS-SRC and SURPAS risk tools. A higher c-index, higher H-L p-value, and lower Brier score indicate better model fit* p value for the difference between C -index was 0.06 Values are sample sizes at each quantile.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
7
Fig. 3. Hosmer-Lemeshow Graphs and Fit Statistics for ACS-SRC (upper graph) and SURPAS (lower graph) estimates of 30-day overall morbidity. The interpretations of these graphs are the same as in Fig. 2. *p value for the difference between C -index was 0.65. Values are sample sizes at each quantile.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036
8
S. Khaneki et al. / The American Journal of Surgery xxx (xxxx) xxx
also how they compare to more disease- and operation-specific risk calculators existing in the literature; (2) The analysis was limited to only two outcomes. SURPAS had four other outcomes in common with the ACS-SRC that were not studied due to the manual collection of the data (cardiac, UTI, VTE, and renal failure), and SURPAS has also recently added unplanned readmission and discharge destination. These could be the subject of future studies; (3) The sample size of the study was relatively small necessitated by the labor of manually entering 21 variables into the ACS-SRC for each patient. Studies with larger sample sizes would be possible if the details of the ACS-SRC models were published; and (4) The results for mortality were inconclusive because there were so few deaths in a sample of 1,006 patients. Future, much larger studies are needed for mortality. Conclusion The SURPAS risk estimates for postoperative mortality and overall morbidity appear to be comparable to those of the ACS-SRC. The ACS-SRC actually underestimates overall morbidity risk for the higher risk patients. The advantages of SURPAS are that it requires manual entry of only one-fourth the number of variables, is integrated into the EHR at UCHealth, provides a preoperative note in the patient record, and provides a graphical display of patient risk for the patients to take home. Funding This work was supported by a grant from the Agency for Healthcare Research and Quality (AHRQ 1R21HS024124-02). Conflicts of interest
References 1. Hammermeister KE, Henderson WG, Bronsert MR, Juarez-Colunga E, Meguid RA. Bringing quantitative risk assessment closer to the patient and surgeon: a novel approach to improve outcomes. Ann Surg. 2016;263(6): 1039e1041. 2. Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical risk preoperative assessment system (SURPAS): I. Parsimonious, clinically meaningful groups of postoperative complications by factor Analysis. Ann Surg. 2016;263(6):1042e1048. 3. Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical risk preoperative assessment system (SURPAS): II. Parsimonious risk models for postoperative adverse outcomes addressing need for laboratory variables and surgeon specialty-specific models. Ann Surg. 2016;264(1):10e22. 4. Meguid RA, Bronsert MR, Juarez-Colunga E, Hammermeister KE, Henderson WG. Surgical risk preoperative assessment system (SURPAS): III. Accurate preoperative prediction of 8 adverse outcomes using 8 predictor variables. Ann Surg. 2016;264(1):23e31. 5. Lambert-Kerzner A, Ford KL, Hammermeister KE, Henderson WG, Bronsert MR, Meguid RA. Assessment of attitudes towards future implementation of the surgical risk preoperative assessment system (SURPAS) tool: a qualitative study. Patient Saf Surg. 2018;12:12. 6. Bilimoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013;217(5), 833-842 e831-833. 7. Cohen ME, Liu Y, Ko CY, Hall BL. An examination of American College of surgeons NSQIP surgical risk calculator accuracy. J Am Coll Surg. 2017;224(5): 787e795 e781. 8. Khuri SF, Daley J, Henderson W, et al. Risk adjustment of the postoperative mortality rate for the comparative assessment of the quality of surgical care. Results of the National Veterans Affairs Surgical Risk Study. J Am Coll Surg. 1997;185:315e327. 9. Daley J, Khuri SF, Henderson W, et al. Risk adjustment of the postoperative morbidity rate for the comparative assessment of the quality of surgical care. Results of the National Veterans Affairs Surgical Risk Study. J Am Coll Surg. 1997;185:328e340. 10. American College of Surgeons. American College of Surgeons Risk Calculator; 2018. https://riskcalculator.facs.org/RiskCalculator/. Accessed April 15, 2018. 11. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307e310. 12. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(2):135e160.
The authors report no conflicts of interest to disclose. The authors of this manuscript and the developers of SURPAS have no financial interests in the SURPAS product.
Please cite this article as: Khaneki S et al., Comparison of accuracy of prediction of postoperative mortality and morbidity between a new, parsimonious risk calculator (SURPAS) and the ACS Surgical Risk Calculator, The American Journal of Surgery, https://doi.org/10.1016/ j.amjsurg.2019.07.036