- Email: [email protected]
Use of the American College of Surgeons NSQIP Surgical Risk Calculator for Laparoscopic Colectomy: How Good Is It and How Can We Improve It?
Use of the American College of Surgeons NSQIP Surgical Risk Calculator for Laparoscopic Colectomy: How Good Is It and How Can We Improve It? Kyle G Cologne, MD, Deborah S Keller, Anthony J Senagore, MS, MD, MBA, FACS
MS, MD,
Loriel Liwanag,
BA,
Bikash Devaraj,
MD,
The American College of Surgeons NSQIP risk calculator was developed from multiinstitutional clinical data to estimate preoperative risk. The impact of outliers has the potential to greatly affect predictions. Although the effect of outliers is minimized in large series, their impact on the individual provider or institution could be profound. No previous study has assessed the risk calculator for a single institution or provider, including outliers. Our goal was to evaluate the accuracy of the predicted outcomes at a single institution. STUDY DESIGN: Laparoscopic colectomies performed by two colorectal surgeons at a tertiary referral center were prospectively evaluated using the risk calculator. Predicted outcomes were compared with actual outcomes for length of stay (LOS), complications, return to the operating room, and death. Main outcomes measures were differences in actual vs predicted outcomes. RESULTS: One hundred and sixteen patients were included. Actual LOS was higher than predicted (mean SD 4.22 5.49 days vs predicted 4.11 1.18 days; p ¼ 0.0001). Four outliers with multiple complications had an LOS >3 SDs from the mean. After removing these, observed LOS was significantly shorter than predicted (adjusted LOS mean SD 3.31 2.30 days vs predicted 4.05 1.14 days; p ¼ 0.002). Occurrence of any complication was significantly lower than predicted (17.3% vs 19.4%; p ¼ 0.05). Rates of major complications (13.2% vs 19.4%; p ¼ 0.009) and surgical site infections (9.8% vs 11.8%; p ¼ 0.006) were also significantly lower than predicted. There were no significant differences in death, urinary tract infection, renal failure, and reoperation rates. CONCLUSIONS: Although the risk calculator was effective for evaluating average surgical-risk patients, it does not accurately predict outcomes in a small percentage of patients when one or more serious complications occur. Addition of surgeon- and patient-specific data via the American College of Surgeons case-logging system could better adjust for these areas. (J Am Coll Surg 2015; 220:281e286. 2015 by the American College of Surgeons)
BACKGROUND:
As patient outcomes are being linked to provider reimbursement, increased emphasis is placed on proper risk stratification. The Centers for Medicare and Medicaid Services’ programs, such as pay for performance and Physician Quality Reporting System, are being integrated into everyday clinical practice to drive reimbursement.1-3 In this health care environment, accurately defining risk is paramount to measuring quality and assessing outcomes, as not all patients are created equal.3 Several predictive models were developed to stratify patients undergoing colorectal surgery.4-8 However, targeted preoperative risk assessment tools customized to specific procedures and patient circumstances have been shown to be
CME questions for this article available at http://jacscme.facs.org Disclosure Information: Authors have nothing to disclose. Timothy J Eberlein, Editor-in-Chief, has nothing to disclose. Received November 1, 2014; Revised December 8, 2014; Accepted December 8, 2014. From the Division of Colorectal Surgery, Keck School of Medicine of the University of Southern California, Los Angeles, CA (Cologne, Liwanag, Devaraj), Colorectal Surgical Associates, Houston, TX (Keller), and Division of Colorectal Surgery, Case Medical Center, Cleveland, OH (Senagore). Correspondence address: Kyle G Cologne, MD, Division of Colorectal Surgery, Keck School of Medicine of the University of Southern California, 1441 Eastlake Ave, Suite 7418, Los Angeles, CA 90033. email: kyle. [email protected]
ª 2015 by the American College of Surgeons Published by Elsevier Inc.
281
http://dx.doi.org/10.1016/j.jamcollsurg.2014.12.007 ISSN 1072-7515/14
282
Cologne et al
American College of Surgeons NSQIP Calculator
more powerful than generic predictive models.9 Studies have shown that participating in NSQIP programs improves outcomes measures,10 but much of these data have been focused at a more aggregate level, and the effect of information on individual surgeons remains to be determined. The novel American College of Surgeons (ACS) NSQIP risk calculator was developed to aid risk stratification of patients undergoing major surgery, including colorectal resections.11,12 The ACS NSQIP risk calculator was created using aggregate multi-institutional NSQIP data. The clinician can input 21 patient-specific variables, in addition to the CPT code, to generate a predicted risk for a series of outcomes and the overall average risk for that CPT code. Its ability to accurately predict data for a single institution or surgeon has not been evaluated. The objective of this study was to evaluate the accuracy of the ACS NSQIP risk calculator in predicting outcomes for laparoscopic colectomy at a single institution. Our hypothesis was that the ACS NSQIP risk calculator does not appropriately adjust for risk at the individual provider level.
METHODS After obtaining IRB approval, consecutive laparoscopic colon resections performed on an elective basis from April 2011 through July 2014 by two board-certified colorectal surgeons at a tertiary referral center were prospectively risk stratified using the online ACS surgical risk calculator.13 Laparoscopic colectomies were defined using the Table 1. Demographics and Patient Characteristics Age, y, mean SD (range) 56 12.2 (26e89) ASA classification, mean SD 2.31 0.5 BMI, kg/m2, mean SD (range) 27.3 7 (21e56) Sex, % male 48 Hypertension, % of total 52 Coronary artery disease, % of total 7 Diabetes, % of total 16 (4 insulin dependent) Current (within 1 y) smoker, % of total 6 COPD, % 1 Dyspnea on moderate exertion, % 10 Acute renal failure (pre-existing), % 1 Hemodialysis, % 2 Recent steroid use, % 10 Wound class: clean contaminated, % 100 Functional status: independent, % 91 Ventilator dependent, % 0 Disseminated cancer, % 0 ASA, American Society of Anesthesiologists.
J Am Coll Surg
following CPT codes: 44204, 44205, 44206, 44207, 44208, 44210, 44211, 44212, and 44213. Patients were included if older than 18 years of age, if the procedure was performed by one of the two participating colorectal surgeons (KGC or AJS), if a preoperative ACS risk score was calculated, and if completed postoperative medical records were available. Patient demographic and outcomes variables were evaluated, including age, sex, race, smoking status, drinking status, transfer status, preoperative functional status, BMI, history of diabetes, severe COPD (resulting in functional disability, previous hospitalization, an FEV1 <75% of predicted, or requiring chronic bronchodilator therapy with oral or inhaled agents), congestive heart failure in the 30 days before surgery, hypertension requiring medication, peripheral vascular disease requiring revascularization or amputation, delirium, metastatic cancer, regular steroid use in the 30 days before surgery, bleeding disorders, any operation 30 days before surgery, and chemotherapy or radiotherapy within 30 days before surgery, length of stay (LOS), any complication, major complications, surgical site infections, renal failure, clinical leak rate, urinary tract infection, return to the operating room, and death. The “Surgeon Adjustment of Risks” was set at “1eNo adjustment necessary” for all cases. The prospectively predicted outcomes data were compared with actual data for the following patient outcomes variables: postoperative LOS, any complication, major complications, surgical site infections, renal failure, return to the operating room, urinary tract infection, and death. The main outcomes measures were the difference in actual vs predicted outcomes from the ACS risk calculator for each variable. Descriptive statistics were used to analyze results using mean SD for continuous variables. Categorical variables were compared using the Pearson chi-square tests, and continuous variables were compared using Wilcoxon rank sum tests. Regression analysis was used to examine the association between predicted outcomes and actual outcomes data. A p value <0.05 was considered statistically significant. All data were evaluated using STATA/IC statistical software, version 11.2 (Stata Corp).
RESULTS During the study period, 116 laparoscopic colon resections met inclusion criteria and were included in the analysis. The indications for colectomy included cancer (n ¼ 55), diverticular disease (n ¼ 21), large or endoscopically unresectable polyps or polyposis syndromes (n ¼ 18), inflammatory bowel disease (n ¼ 9), and other (endometriosis, stricture, volvulus, and other benign causes). The most commonly performed procedure was partial
Cologne et al
Vol. 220, No. 3, March 2015
Table 2. Patient no.
American College of Surgeons NSQIP Calculator
283
Length of Stay Outliers Length of stay, d
Details
1
37
2
22
3
22
4
15
Dialysis-dependent patient with colon cancer. Severe uremic platelet dysfunction with hypotension during dialysis postoperative. Subsequent leak requiring return to operating room, multiorgan system failure (resolved). Frail, elderly patient with near-obstructing colon cancer. Hemicolectomy þ end stoma. Ileus postoperative, also pneumonia, gastrointestinal bleed. BMI > 30 kg/m2, patient with locally advanced rectal cancer requiring resection and proximal diversion. Hematoma requiring transfusion and prolonged ileus/stoma dysfunction postoperatively. BMI > 40 kg/m2, patient with chronic diverticulitis, hypertension, and dyspnea on exertion. þAir-leak test requiring proximal diversion. Ileus/stoma dysfunction with resultant dehydration and acute renal insufficiency.
colectomy with anastomosis (CPT codes 44204, 44205; n ¼ 84). Distribution of other procedures included total abdominal colectomy (CPT codes 44210, 44211, 44212, 44213; n ¼ 18) and partial colectomy with low pelvic anastomosis (CPT codes 44207, 44208; n ¼ 14). Conversion rate to open procedure was 12%. All conversions were from straight laparoscopic to traditional open procedures. Tumor-related factors were the most common reason for conversion, followed by adhesions from previous surgery. For converted cases, there was no difference in predicted outcomes whether the laparoscopic or open CPT code was used in the calculator. Overall clinical characteristics of the patients are described in Table 1. Mean SD age was 56 12.2 years. Mean SD BMI was 27.3 7 kg/m2, and 48% were male. Most were American Society of Anesthesiologists class 2 (44%) or 3 (41%), and wound class 2 (100%). Forty-one percent of patients had previous abdominal surgery. Of 116 patients, there were 4 outliers, who experienced LOS >3 SD from the mean (Table 2). The observed LOS for the entire series was significantly longer than the LOS predicted by the calculator (4.22 5.49 vs 4.11 1.18; p ¼ 0.0001). After removal of the 4 outliers, the actual LOS was significantly shorter than that predicted by the risk calculator for our remaining 112 patients (3.31 2.30 vs 4.05 1.14; p ¼ 0.002). In examining all 116 patients, the occurrence of any complication was observed at a significantly lower rate than predicted by the risk calculator (17.3% vs 19.4%; p ¼ 0.05). The same was true of major complications (13.2% vs 19.4%; p ¼ 0.009), and surgical site infections (9.8% vs 11.8%; p ¼ 0.006). There was no difference in death, urinary tract infection, and reoperation rates among observed vs expected values (Table 3). Clinical leak rate (as defined by radiologic study demonstrating contrast extravasation and/or perianastomotic fluid collection or operative findings during repeat exploration) was 5.8%. Reoperation rate was 5.1%. Reasons for reoperation included anastomotic leak (n ¼ 3),
hematoma (n ¼ 2), and hernia (n ¼ 1). Readmission rate was 8%. Reasons for readmission included leak (n ¼ 4), ileostomy dysfunction (n ¼ 3), bowel obstruction (n ¼ 1), and bleeding (n ¼ 1).
DISCUSSION The ACS risk calculator is an important tool that allows surgeons to estimate patient-specific postoperative surgical risks for shared decision making and informed consent.11 The decision-support tool was developed from aggregate, multi-institutional clinical data, providing validation from the large sample size. However, the risk calculator has never been tested on a smaller scale and might lack validity for individual providers, where the impact of outliers has a greater impact on predicted outcomes. Our goal was to evaluate the accuracy of the ACS NSQIP risk calculator in predicting outcomes for a laparoscopic colectomy at a single institution. Our hypothesisdthat the ACS NSQIP risk calculator does not appropriately adjust for risk at the individual provider leveldwas true when outliers were included in data evaluated, particularly for predicted LOS. However, the risk calculator more accurately predicted the postoperative LOS, overall and major complication rates for average risk individual patients and surgeons who have low rates of major complications. The difficulty now is how to predict and accurately advise patients when rare events occur. Current studies had similar findings that the individual effect of an outlier is minimized in large series. One goal of NSQIP and quality programs in general has been to identify outliers and determine reasons for deviation and increase quality of care.10,14-16 This is possible with large datasets, but has proven an elusive goal with individual providers, as variations in patient care are more apparent in the small sample size.4 After using the calculator extensively, we have several observations. With the calculator, we found extremes of age caused significant increases in the predicted morbidity
284
Table 3.
Cologne et al
J Am Coll Surg
American College of Surgeons NSQIP Calculator
Predicted vs Actual Outcomes
Outcomes metric
Death Any complication Major complication SSI UTI Acute renal failure Reoperation Leak Readmission
Actual incidence,* %
Predicted incidence, %
p Value
1.07 17.3 13.2 9.83 2.85 1.90 5.1 5.8 8
0.83 19.4 19.4 11.8 2.90 0.90 5.60 NA NA
0.86 0.05 0.009 0.006 0.95 0.085 0.86 d d
*Mean for 116 patients. NA, not applicable; SSI, surgical site infection; UTI, urinary tract infection.
and mortality observations. Previous studies with the Colorectal Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (POSSUM) had similar findings; age causes a significant increase in the predicted morbidity and mortality and can result in overestimation of risk.6,17,18 Second, the predicted LOS does not significantly differ, despite the addition of multiple comorbidities. Even a morbidly obese, insulin-dependent diabetic who is a current smoker with a previous cardiac event has a predicted LOS of <6 days (less than the mean LOS in many studies).19-22 For this reason, this calculator will poorly predict outliers, as it is based on aggregate data. Even use of the built-in modifier to allow a provider to increase the perceived risk does not change the LOS calculation. This is why we chose to exclude 4 outliers, who had an average LOS of >3 SDs beyond the mean. Despite a lower than expected rate of complications and a low rate of leak and readmission, these outliers greatly affected our overall numbers (indicated by the large SD, which decreases from 5.49 to 2.30 by inclusion or not of these small numbers). Previous models of risk calculation have been developed in colorectal surgery. The POSSUM, Portsmouth POSSUM, and colorectal POSSUM scores were developed for comparative audit of colorectal resection, and were found to accurately predict mortality for emergent and elective colorectal surgery cases.5,23-26 However, the POSSUM, Portsmouth POSSUM, and colorectal POSSUM scores solely predict mortality. In addition, the scores were not accurate for laparoscopic colorectal resection, and require additional refinement for identification of potential outliers.17,18 Statistical Process Control, an industrial methodology developed for analyzing aggregate data in the setting of outliers, is one option for analyzing outliers. Statistical Process Control can identify outliers from expected outcomes, then evaluate these to assess the patient- and processlevel factors that affect postoperative outcomes.27
Statistical Process Control estimates where outliers would lie under a stable process bound by upper control and lower control limits. These control limits are usually set to >3 SDs, with outliers occurring outside of the set control limits.28,29 In our case, there were 4 outliers from the expected LOS that were removed, as these were affected by process anomalies and did not fall within predicted outcomes. The ACS risk calculator uses stable processes within the control limits for predicting outcomes, and additional methods should be implemented for analysis of outliers. Several concepts to improve performance of the risk calculator at the individual level were apparent from our study. First, a measure of the provider’s volume should be included in the calculation. There is a proposed “warranty cost” of care. That is to say that a surgeon’s potential complications must be included in payments for procedures, as these unexpected outliers must be “paid off” by the mean, and more specifically those without complications. The cost of complications has been shown to be incredibly expensive, especially for colectomy.3,4 Studies have shown surgical outcomes are related to surgical volume.30-33 Low-volume surgeons are harder to interpret and predict outcomes for using this type of risk calculator, as outliers will have a much more profound effect on the aggregate data. Second, a more precise method to adjust risk could help align the predicted and actual risk. The built-in adjustment score does result in significantly increased predicted morbidity, perhaps to an unrealistic level. In our study, the predicted incidence of complications goes up 1.5- to 2-fold, increasing the Surgeon Adjustment Score feature of the calculator for “somewhat higher.” The “significantly higher” than estimated risk changes the predicted incidence of complications by 2to 3-fold, and places all risk factors in the red category. Use of a frailty index34,35 might be an alternative method to more objectively grade the risk of the patient, and increase risk by more graded amounts. With respect to
Vol. 220, No. 3, March 2015
Cologne et al
colorectal procedures, the risk calculator also fails to capture individual preoperative risks that have a substantial impact on patient outcomes, such as radiation, chemotherapy, biologic therapy, or immunosuppression. In addition, it fails to capture anastomotic leak as an outcomes predictor. It also does not address the risk of readmission at allda factor proven to negatively impact patient and financial outcomes.36-38 As an added incentive for use as a routine part of the case-logging system, one possibility would be to exempt surgeons who consistently outperform anticipated measures to opt out of part IV of maintenance of certification requirements (ie, performance in practice). We recognize the limitations of this study. First, it is a small series based on patients treated at a single tertiary referral center. Therefore, the results might not be generalizable. Small differences in patient outcomes led to statistical significance, when the differences in outcomes might not be clinically significant. Consecutive elective patients were evaluated, so there might also be inherent bias in the study design. For all risk calculations, we set the Surgeon Adjustment Score to 1. By increasing the score with higher-risk patients, the prediction scores for complications and LOS might be more comparable with actual clinical data for outlier cases. However, the guidance of the current calculator is vague in determining when to adjust the score. Despite the limitations, our study is the first application of the calculator in routine clinical practice, and provides insight and direction for future study.
CONCLUSIONS We found the ACS risk calculator yielded accurate prediction results for average surgical risk patients for laparoscopic colectomy. However, it does not accurately adjust and predict outcomes when serious complications occur. The next step will be to provide surgeon- and institution-specific modification to better risk stratify patients. This will need to be done on a much larger scale and will require additional study. With adjustment for individual providers, the calculator can help surgeons improve the quality of care provided. The ideal risk predictor could be incorporated into the ACS case-logging system, allowing improved surgeon-specific data and risk prediction and the possibility to align incentives with quality surgical care. Author Contributions Study conception and design: Cologne, Senagore Acquisition of data: Cologne, Liwanag, Devaraj
American College of Surgeons NSQIP Calculator
285
Analysis and interpretation of data: Cologne, Keller, Liwanag, Devaraj, Senagore Drafting of manuscript: Cologne, Keller, Liwanag, Devaraj, Senagore Critical revision: Cologne, Keller, Liwanag, Devaraj, Senagore REFERENCES 1. Measure Application Partnership. MAP pre-rule making report 2013. Available at: http://www.qualityforum.org/Publications/ 2013/02/MAP_Pre-Rulemaking_Report_-_February_2013.aspx. Accessed September 14, 2014. 2. Khanduja K, Scales DC, Adhikari NK. Pay for performance in the intensive care unitdopportunity or threat? Crit Care Med 2009;37:852e858. 3. Cologne KG, Hwang GS, Senagore AJ. Cost of practice in a tertiary/quaternary referral center: is it sustainable? Tech Proctol 2014;18:1035e1039. 4. Schilling PL, Dimick JB, Birkmeyer JD. Prioritizing quality improvement in general surgery. J Am Coll Surg 2008;207: 698e704. 5. Tekkis PP, Prytherch DR, Kocher HM, et al. Development of a dedicated risk-adjustment scoring system for colorectal surgery (colorectal POSSUM). Br J Surg 2004;91: 1174e1182. 6. Bromage SJ, Cunliffe WJ. Validation of the CR-POSSUM risk-adjusted scoring system for major colorectal cancer surgery in a single center. Dis Colon Rectum 2007;50:192e196. 7. Fazio VW, Tekkis PP, Remzi F, Lavery IC. Assessment of operative risk in colorectal cancer surgery: the Cleveland Clinic Foundation colorectal cancer model. Dis Colon Rectum 2004; 47:2015e2024. 8. Cohen ME, Dimick JB, Billmoria KY, et al. Risk adjustment in the American College of Surgeons National Surgical Quality Improvement Program: a comparison of logistic versus hierarchical modeling. J Am Coll Surg 2009;209: 687e693. 9. Kwok AC, Lipsitz SR, Bader AM, et al. Are targeted preoperative risk prediction tools more powerful? A test of models for emergency colon surgery in the very elderly. J Am Coll Surg 2011;213:220e225. 10. Hall BL, Hamilton BH, Richards K, et al. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg 2009;250:363e376. 11. Bilmoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 2013;217:833e842. 12. Cohen ME, Bilmoria KY, Ko CY, Hall BL. Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. J Am Coll Surg 2009;208: 1009e1016. 13. American College of Surgeons. Surgical risk calculator. Available at: http://riskcalculator.facs.org/. Accessed August 1, 2014. 14. Ingraham AM, Richards KE, Hall BL, Ko CY. Quality improvement in surgery: the American College of Surgeons Surgical Quality Improvement Program approach. Adv Surg 2010;44:251e267.
286
Cologne et al
American College of Surgeons NSQIP Calculator
15. Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998;228:491e507. 16. Khuri SF, Daley J, Henderson WG. The comparative assessment and improvement of quality of surgical care in the Department of Veterans Affairs. Arch Surg 2002;137:20e27. 17. Law WL, Lam CM, Lee YM. Evaluation of laparoscopic colorectal resection with POSSUM, Portsmouth POSSUM, and Colorectal POSSUM. Br J Surg 2006;93:94e99. 18. Senagore AJ, Warmuth AJ, Delaney CP, et al. POSSUM, p-POSSUM, and Cr-POSSUM: implementation issues in a United States health care system for prediction of outcome for colon cancer resection. Dis Colon Rectum 2004;47: 1435e1441. 19. Delaney CP, Chang E, Senagore AJ, Broder M. Clinical outcomes and resource utilization associated with laparoscopic and open colectomy using a large national database. Ann Surg 2008;247:819e824. 20. Schwenk W, Haase O, Neudecker J, et al. Short term benefits for laparoscopic colorectal resection. Cochrane Database Syst Rev 2005;20:CD003145. 21. Guillou PJ, Quirke P, Thorpe H, et al. Short-term endpoints of conventional versus laparoscopic-assisted surgery in patients with colorectal cancer (MRC CLASSIC trial): multicentre, randomized controlled trial. Lancet 2005;365: 1718e1726. 22. Veldkamp R, Kuhry E, Hop WC, et al. Laparoscopic surgery versus open surgery for colon cancer: short-term outcomes of a randomized trial. Lancet Oncol 2005;6:477e484. 23. Al-Homoud S, Purkayastha S, Aziz O, et al. Evaluating operative risk in colorectal cancer surgery: ASA and POSSUMbased predictive models. Surg Oncol 2004;13:83e92. 24. Constantinides VA, Tekkis PP, Senapati A. Association of Coloproctology of Great Britain and Ireland. Comparison of POSSUM scoring systems and the surgical risk scale in patients undergoing surgery for complicated diverticular disease. Dis Colon Rectum 2006;49:1322e1331. 25. Copeland GP, Jones D, Walters M. POSSUM: a scoring system for surgical audit. Br J Surg 1991;78:355e360. 26. Poon JT, Chan B, Law WL. Evaluation of P-POSSUM in surgery for obstructing colorectal cancer and correlation of the
27. 28. 29.
30. 31.
32. 33. 34.
35.
36. 37.
38.
J Am Coll Surg
predicted mortality with different surgical options. Dis Colon Rectum 2005;48:493e498. Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003;12:458e464. Paulk MC. Software metrics and analysis: identifying outliers, understanding the process. Available at: http://home.comcast.net/ wmark.paulk/papers/p2009a.pdf. Accessed September 11, 2014. Romagnoli JA, Palazoglu A. Data processing and reconciliation. In: Romagnoli JA, Palazoglu A, eds. Introduction to Process Control. 2nd ed. Boca Raton, FL: CRC Taylor and Francis Group, LLC; 2012:515e539. Keller DS, Hashemi L, Lu M, Delaney CP. Short-term outcomes for robotic colorectal surgery by provider volume. J Am Coll Surg 2013;217:1063e1069. Harmon JW, Tang DG, Gordon TA, et al. Hospital volume can serve as a surrogate for surgeon volume for achieving excellent outcomes in colorectal resection. Ann Surg 1999;230: 404e411; discussion 411e413. Farjoodi P, Skolasky RL, Riley LH. The effects of hospital and surgeon volume on postoperative complications after Lumbar Spine surgery. Spine (Phila Pa 1976) 2011;36:2069e2075. Billingsley KG, Morris AM, Green P, et al. Does surgeon case volume influence nonfatal adverse outcomes after rectal cancer resection? J Am Coll Surg 2008;206:1167e1177. Keller DS, Bankwitz B, Nobel T, Delaney CP. Using frailty to predict who will fail early discharge after laparoscopic colorectal surgery with an established recovery pathway. Dis Colon Rectum 2014;57:337e342. Amrock LG, Neuman MD, Lin HM, Deiner S. Can routine preoperative data predict adverse outcomes in the elderly? Development and validation of a simple risk model incorporating a chart-derived frailty score. J Am Coll Surg 2014; 219:684e694. Wick EC, Shore AD, Hirose K, et al. Readmission rates and cost following colorectal surgery. Dis Colon Rectum 2011; 54:1475e1479. Lawrence JK, Keller DS, Samia H, et al. Discharge within 24 to 72 hours of colorectal surgery is associated with low readmission rates when using enhanced recovery pathways. J Am Coll Surg 2013;216:390e394. Keller DS, Khorgami Z, Swendseid B, et al. Identifying causes for high readmission rates after stoma reversal. Surg Endosc 2014;28:1263e1268.
MS, MD,
Loriel Liwanag,
BA,
Bikash Devaraj,
MD,
The American College of Surgeons NSQIP risk calculator was developed from multiinstitutional clinical data to estimate preoperative risk. The impact of outliers has the potential to greatly affect predictions. Although the effect of outliers is minimized in large series, their impact on the individual provider or institution could be profound. No previous study has assessed the risk calculator for a single institution or provider, including outliers. Our goal was to evaluate the accuracy of the predicted outcomes at a single institution. STUDY DESIGN: Laparoscopic colectomies performed by two colorectal surgeons at a tertiary referral center were prospectively evaluated using the risk calculator. Predicted outcomes were compared with actual outcomes for length of stay (LOS), complications, return to the operating room, and death. Main outcomes measures were differences in actual vs predicted outcomes. RESULTS: One hundred and sixteen patients were included. Actual LOS was higher than predicted (mean SD 4.22 5.49 days vs predicted 4.11 1.18 days; p ¼ 0.0001). Four outliers with multiple complications had an LOS >3 SDs from the mean. After removing these, observed LOS was significantly shorter than predicted (adjusted LOS mean SD 3.31 2.30 days vs predicted 4.05 1.14 days; p ¼ 0.002). Occurrence of any complication was significantly lower than predicted (17.3% vs 19.4%; p ¼ 0.05). Rates of major complications (13.2% vs 19.4%; p ¼ 0.009) and surgical site infections (9.8% vs 11.8%; p ¼ 0.006) were also significantly lower than predicted. There were no significant differences in death, urinary tract infection, renal failure, and reoperation rates. CONCLUSIONS: Although the risk calculator was effective for evaluating average surgical-risk patients, it does not accurately predict outcomes in a small percentage of patients when one or more serious complications occur. Addition of surgeon- and patient-specific data via the American College of Surgeons case-logging system could better adjust for these areas. (J Am Coll Surg 2015; 220:281e286. 2015 by the American College of Surgeons)
BACKGROUND:
As patient outcomes are being linked to provider reimbursement, increased emphasis is placed on proper risk stratification. The Centers for Medicare and Medicaid Services’ programs, such as pay for performance and Physician Quality Reporting System, are being integrated into everyday clinical practice to drive reimbursement.1-3 In this health care environment, accurately defining risk is paramount to measuring quality and assessing outcomes, as not all patients are created equal.3 Several predictive models were developed to stratify patients undergoing colorectal surgery.4-8 However, targeted preoperative risk assessment tools customized to specific procedures and patient circumstances have been shown to be
CME questions for this article available at http://jacscme.facs.org Disclosure Information: Authors have nothing to disclose. Timothy J Eberlein, Editor-in-Chief, has nothing to disclose. Received November 1, 2014; Revised December 8, 2014; Accepted December 8, 2014. From the Division of Colorectal Surgery, Keck School of Medicine of the University of Southern California, Los Angeles, CA (Cologne, Liwanag, Devaraj), Colorectal Surgical Associates, Houston, TX (Keller), and Division of Colorectal Surgery, Case Medical Center, Cleveland, OH (Senagore). Correspondence address: Kyle G Cologne, MD, Division of Colorectal Surgery, Keck School of Medicine of the University of Southern California, 1441 Eastlake Ave, Suite 7418, Los Angeles, CA 90033. email: kyle. [email protected]
ª 2015 by the American College of Surgeons Published by Elsevier Inc.
281
http://dx.doi.org/10.1016/j.jamcollsurg.2014.12.007 ISSN 1072-7515/14
282
Cologne et al
American College of Surgeons NSQIP Calculator
more powerful than generic predictive models.9 Studies have shown that participating in NSQIP programs improves outcomes measures,10 but much of these data have been focused at a more aggregate level, and the effect of information on individual surgeons remains to be determined. The novel American College of Surgeons (ACS) NSQIP risk calculator was developed to aid risk stratification of patients undergoing major surgery, including colorectal resections.11,12 The ACS NSQIP risk calculator was created using aggregate multi-institutional NSQIP data. The clinician can input 21 patient-specific variables, in addition to the CPT code, to generate a predicted risk for a series of outcomes and the overall average risk for that CPT code. Its ability to accurately predict data for a single institution or surgeon has not been evaluated. The objective of this study was to evaluate the accuracy of the ACS NSQIP risk calculator in predicting outcomes for laparoscopic colectomy at a single institution. Our hypothesis was that the ACS NSQIP risk calculator does not appropriately adjust for risk at the individual provider level.
METHODS After obtaining IRB approval, consecutive laparoscopic colon resections performed on an elective basis from April 2011 through July 2014 by two board-certified colorectal surgeons at a tertiary referral center were prospectively risk stratified using the online ACS surgical risk calculator.13 Laparoscopic colectomies were defined using the Table 1. Demographics and Patient Characteristics Age, y, mean SD (range) 56 12.2 (26e89) ASA classification, mean SD 2.31 0.5 BMI, kg/m2, mean SD (range) 27.3 7 (21e56) Sex, % male 48 Hypertension, % of total 52 Coronary artery disease, % of total 7 Diabetes, % of total 16 (4 insulin dependent) Current (within 1 y) smoker, % of total 6 COPD, % 1 Dyspnea on moderate exertion, % 10 Acute renal failure (pre-existing), % 1 Hemodialysis, % 2 Recent steroid use, % 10 Wound class: clean contaminated, % 100 Functional status: independent, % 91 Ventilator dependent, % 0 Disseminated cancer, % 0 ASA, American Society of Anesthesiologists.
J Am Coll Surg
following CPT codes: 44204, 44205, 44206, 44207, 44208, 44210, 44211, 44212, and 44213. Patients were included if older than 18 years of age, if the procedure was performed by one of the two participating colorectal surgeons (KGC or AJS), if a preoperative ACS risk score was calculated, and if completed postoperative medical records were available. Patient demographic and outcomes variables were evaluated, including age, sex, race, smoking status, drinking status, transfer status, preoperative functional status, BMI, history of diabetes, severe COPD (resulting in functional disability, previous hospitalization, an FEV1 <75% of predicted, or requiring chronic bronchodilator therapy with oral or inhaled agents), congestive heart failure in the 30 days before surgery, hypertension requiring medication, peripheral vascular disease requiring revascularization or amputation, delirium, metastatic cancer, regular steroid use in the 30 days before surgery, bleeding disorders, any operation 30 days before surgery, and chemotherapy or radiotherapy within 30 days before surgery, length of stay (LOS), any complication, major complications, surgical site infections, renal failure, clinical leak rate, urinary tract infection, return to the operating room, and death. The “Surgeon Adjustment of Risks” was set at “1eNo adjustment necessary” for all cases. The prospectively predicted outcomes data were compared with actual data for the following patient outcomes variables: postoperative LOS, any complication, major complications, surgical site infections, renal failure, return to the operating room, urinary tract infection, and death. The main outcomes measures were the difference in actual vs predicted outcomes from the ACS risk calculator for each variable. Descriptive statistics were used to analyze results using mean SD for continuous variables. Categorical variables were compared using the Pearson chi-square tests, and continuous variables were compared using Wilcoxon rank sum tests. Regression analysis was used to examine the association between predicted outcomes and actual outcomes data. A p value <0.05 was considered statistically significant. All data were evaluated using STATA/IC statistical software, version 11.2 (Stata Corp).
RESULTS During the study period, 116 laparoscopic colon resections met inclusion criteria and were included in the analysis. The indications for colectomy included cancer (n ¼ 55), diverticular disease (n ¼ 21), large or endoscopically unresectable polyps or polyposis syndromes (n ¼ 18), inflammatory bowel disease (n ¼ 9), and other (endometriosis, stricture, volvulus, and other benign causes). The most commonly performed procedure was partial
Cologne et al
Vol. 220, No. 3, March 2015
Table 2. Patient no.
American College of Surgeons NSQIP Calculator
283
Length of Stay Outliers Length of stay, d
Details
1
37
2
22
3
22
4
15
Dialysis-dependent patient with colon cancer. Severe uremic platelet dysfunction with hypotension during dialysis postoperative. Subsequent leak requiring return to operating room, multiorgan system failure (resolved). Frail, elderly patient with near-obstructing colon cancer. Hemicolectomy þ end stoma. Ileus postoperative, also pneumonia, gastrointestinal bleed. BMI > 30 kg/m2, patient with locally advanced rectal cancer requiring resection and proximal diversion. Hematoma requiring transfusion and prolonged ileus/stoma dysfunction postoperatively. BMI > 40 kg/m2, patient with chronic diverticulitis, hypertension, and dyspnea on exertion. þAir-leak test requiring proximal diversion. Ileus/stoma dysfunction with resultant dehydration and acute renal insufficiency.
colectomy with anastomosis (CPT codes 44204, 44205; n ¼ 84). Distribution of other procedures included total abdominal colectomy (CPT codes 44210, 44211, 44212, 44213; n ¼ 18) and partial colectomy with low pelvic anastomosis (CPT codes 44207, 44208; n ¼ 14). Conversion rate to open procedure was 12%. All conversions were from straight laparoscopic to traditional open procedures. Tumor-related factors were the most common reason for conversion, followed by adhesions from previous surgery. For converted cases, there was no difference in predicted outcomes whether the laparoscopic or open CPT code was used in the calculator. Overall clinical characteristics of the patients are described in Table 1. Mean SD age was 56 12.2 years. Mean SD BMI was 27.3 7 kg/m2, and 48% were male. Most were American Society of Anesthesiologists class 2 (44%) or 3 (41%), and wound class 2 (100%). Forty-one percent of patients had previous abdominal surgery. Of 116 patients, there were 4 outliers, who experienced LOS >3 SD from the mean (Table 2). The observed LOS for the entire series was significantly longer than the LOS predicted by the calculator (4.22 5.49 vs 4.11 1.18; p ¼ 0.0001). After removal of the 4 outliers, the actual LOS was significantly shorter than that predicted by the risk calculator for our remaining 112 patients (3.31 2.30 vs 4.05 1.14; p ¼ 0.002). In examining all 116 patients, the occurrence of any complication was observed at a significantly lower rate than predicted by the risk calculator (17.3% vs 19.4%; p ¼ 0.05). The same was true of major complications (13.2% vs 19.4%; p ¼ 0.009), and surgical site infections (9.8% vs 11.8%; p ¼ 0.006). There was no difference in death, urinary tract infection, and reoperation rates among observed vs expected values (Table 3). Clinical leak rate (as defined by radiologic study demonstrating contrast extravasation and/or perianastomotic fluid collection or operative findings during repeat exploration) was 5.8%. Reoperation rate was 5.1%. Reasons for reoperation included anastomotic leak (n ¼ 3),
hematoma (n ¼ 2), and hernia (n ¼ 1). Readmission rate was 8%. Reasons for readmission included leak (n ¼ 4), ileostomy dysfunction (n ¼ 3), bowel obstruction (n ¼ 1), and bleeding (n ¼ 1).
DISCUSSION The ACS risk calculator is an important tool that allows surgeons to estimate patient-specific postoperative surgical risks for shared decision making and informed consent.11 The decision-support tool was developed from aggregate, multi-institutional clinical data, providing validation from the large sample size. However, the risk calculator has never been tested on a smaller scale and might lack validity for individual providers, where the impact of outliers has a greater impact on predicted outcomes. Our goal was to evaluate the accuracy of the ACS NSQIP risk calculator in predicting outcomes for a laparoscopic colectomy at a single institution. Our hypothesisdthat the ACS NSQIP risk calculator does not appropriately adjust for risk at the individual provider leveldwas true when outliers were included in data evaluated, particularly for predicted LOS. However, the risk calculator more accurately predicted the postoperative LOS, overall and major complication rates for average risk individual patients and surgeons who have low rates of major complications. The difficulty now is how to predict and accurately advise patients when rare events occur. Current studies had similar findings that the individual effect of an outlier is minimized in large series. One goal of NSQIP and quality programs in general has been to identify outliers and determine reasons for deviation and increase quality of care.10,14-16 This is possible with large datasets, but has proven an elusive goal with individual providers, as variations in patient care are more apparent in the small sample size.4 After using the calculator extensively, we have several observations. With the calculator, we found extremes of age caused significant increases in the predicted morbidity
284
Table 3.
Cologne et al
J Am Coll Surg
American College of Surgeons NSQIP Calculator
Predicted vs Actual Outcomes
Outcomes metric
Death Any complication Major complication SSI UTI Acute renal failure Reoperation Leak Readmission
Actual incidence,* %
Predicted incidence, %
p Value
1.07 17.3 13.2 9.83 2.85 1.90 5.1 5.8 8
0.83 19.4 19.4 11.8 2.90 0.90 5.60 NA NA
0.86 0.05 0.009 0.006 0.95 0.085 0.86 d d
*Mean for 116 patients. NA, not applicable; SSI, surgical site infection; UTI, urinary tract infection.
and mortality observations. Previous studies with the Colorectal Physiological and Operative Severity Score for the Enumeration of Mortality and Morbidity (POSSUM) had similar findings; age causes a significant increase in the predicted morbidity and mortality and can result in overestimation of risk.6,17,18 Second, the predicted LOS does not significantly differ, despite the addition of multiple comorbidities. Even a morbidly obese, insulin-dependent diabetic who is a current smoker with a previous cardiac event has a predicted LOS of <6 days (less than the mean LOS in many studies).19-22 For this reason, this calculator will poorly predict outliers, as it is based on aggregate data. Even use of the built-in modifier to allow a provider to increase the perceived risk does not change the LOS calculation. This is why we chose to exclude 4 outliers, who had an average LOS of >3 SDs beyond the mean. Despite a lower than expected rate of complications and a low rate of leak and readmission, these outliers greatly affected our overall numbers (indicated by the large SD, which decreases from 5.49 to 2.30 by inclusion or not of these small numbers). Previous models of risk calculation have been developed in colorectal surgery. The POSSUM, Portsmouth POSSUM, and colorectal POSSUM scores were developed for comparative audit of colorectal resection, and were found to accurately predict mortality for emergent and elective colorectal surgery cases.5,23-26 However, the POSSUM, Portsmouth POSSUM, and colorectal POSSUM scores solely predict mortality. In addition, the scores were not accurate for laparoscopic colorectal resection, and require additional refinement for identification of potential outliers.17,18 Statistical Process Control, an industrial methodology developed for analyzing aggregate data in the setting of outliers, is one option for analyzing outliers. Statistical Process Control can identify outliers from expected outcomes, then evaluate these to assess the patient- and processlevel factors that affect postoperative outcomes.27
Statistical Process Control estimates where outliers would lie under a stable process bound by upper control and lower control limits. These control limits are usually set to >3 SDs, with outliers occurring outside of the set control limits.28,29 In our case, there were 4 outliers from the expected LOS that were removed, as these were affected by process anomalies and did not fall within predicted outcomes. The ACS risk calculator uses stable processes within the control limits for predicting outcomes, and additional methods should be implemented for analysis of outliers. Several concepts to improve performance of the risk calculator at the individual level were apparent from our study. First, a measure of the provider’s volume should be included in the calculation. There is a proposed “warranty cost” of care. That is to say that a surgeon’s potential complications must be included in payments for procedures, as these unexpected outliers must be “paid off” by the mean, and more specifically those without complications. The cost of complications has been shown to be incredibly expensive, especially for colectomy.3,4 Studies have shown surgical outcomes are related to surgical volume.30-33 Low-volume surgeons are harder to interpret and predict outcomes for using this type of risk calculator, as outliers will have a much more profound effect on the aggregate data. Second, a more precise method to adjust risk could help align the predicted and actual risk. The built-in adjustment score does result in significantly increased predicted morbidity, perhaps to an unrealistic level. In our study, the predicted incidence of complications goes up 1.5- to 2-fold, increasing the Surgeon Adjustment Score feature of the calculator for “somewhat higher.” The “significantly higher” than estimated risk changes the predicted incidence of complications by 2to 3-fold, and places all risk factors in the red category. Use of a frailty index34,35 might be an alternative method to more objectively grade the risk of the patient, and increase risk by more graded amounts. With respect to
Vol. 220, No. 3, March 2015
Cologne et al
colorectal procedures, the risk calculator also fails to capture individual preoperative risks that have a substantial impact on patient outcomes, such as radiation, chemotherapy, biologic therapy, or immunosuppression. In addition, it fails to capture anastomotic leak as an outcomes predictor. It also does not address the risk of readmission at allda factor proven to negatively impact patient and financial outcomes.36-38 As an added incentive for use as a routine part of the case-logging system, one possibility would be to exempt surgeons who consistently outperform anticipated measures to opt out of part IV of maintenance of certification requirements (ie, performance in practice). We recognize the limitations of this study. First, it is a small series based on patients treated at a single tertiary referral center. Therefore, the results might not be generalizable. Small differences in patient outcomes led to statistical significance, when the differences in outcomes might not be clinically significant. Consecutive elective patients were evaluated, so there might also be inherent bias in the study design. For all risk calculations, we set the Surgeon Adjustment Score to 1. By increasing the score with higher-risk patients, the prediction scores for complications and LOS might be more comparable with actual clinical data for outlier cases. However, the guidance of the current calculator is vague in determining when to adjust the score. Despite the limitations, our study is the first application of the calculator in routine clinical practice, and provides insight and direction for future study.
CONCLUSIONS We found the ACS risk calculator yielded accurate prediction results for average surgical risk patients for laparoscopic colectomy. However, it does not accurately adjust and predict outcomes when serious complications occur. The next step will be to provide surgeon- and institution-specific modification to better risk stratify patients. This will need to be done on a much larger scale and will require additional study. With adjustment for individual providers, the calculator can help surgeons improve the quality of care provided. The ideal risk predictor could be incorporated into the ACS case-logging system, allowing improved surgeon-specific data and risk prediction and the possibility to align incentives with quality surgical care. Author Contributions Study conception and design: Cologne, Senagore Acquisition of data: Cologne, Liwanag, Devaraj
American College of Surgeons NSQIP Calculator
285
Analysis and interpretation of data: Cologne, Keller, Liwanag, Devaraj, Senagore Drafting of manuscript: Cologne, Keller, Liwanag, Devaraj, Senagore Critical revision: Cologne, Keller, Liwanag, Devaraj, Senagore REFERENCES 1. Measure Application Partnership. MAP pre-rule making report 2013. Available at: http://www.qualityforum.org/Publications/ 2013/02/MAP_Pre-Rulemaking_Report_-_February_2013.aspx. Accessed September 14, 2014. 2. Khanduja K, Scales DC, Adhikari NK. Pay for performance in the intensive care unitdopportunity or threat? Crit Care Med 2009;37:852e858. 3. Cologne KG, Hwang GS, Senagore AJ. Cost of practice in a tertiary/quaternary referral center: is it sustainable? Tech Proctol 2014;18:1035e1039. 4. Schilling PL, Dimick JB, Birkmeyer JD. Prioritizing quality improvement in general surgery. J Am Coll Surg 2008;207: 698e704. 5. Tekkis PP, Prytherch DR, Kocher HM, et al. Development of a dedicated risk-adjustment scoring system for colorectal surgery (colorectal POSSUM). Br J Surg 2004;91: 1174e1182. 6. Bromage SJ, Cunliffe WJ. Validation of the CR-POSSUM risk-adjusted scoring system for major colorectal cancer surgery in a single center. Dis Colon Rectum 2007;50:192e196. 7. Fazio VW, Tekkis PP, Remzi F, Lavery IC. Assessment of operative risk in colorectal cancer surgery: the Cleveland Clinic Foundation colorectal cancer model. Dis Colon Rectum 2004; 47:2015e2024. 8. Cohen ME, Dimick JB, Billmoria KY, et al. Risk adjustment in the American College of Surgeons National Surgical Quality Improvement Program: a comparison of logistic versus hierarchical modeling. J Am Coll Surg 2009;209: 687e693. 9. Kwok AC, Lipsitz SR, Bader AM, et al. Are targeted preoperative risk prediction tools more powerful? A test of models for emergency colon surgery in the very elderly. J Am Coll Surg 2011;213:220e225. 10. Hall BL, Hamilton BH, Richards K, et al. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg 2009;250:363e376. 11. Bilmoria KY, Liu Y, Paruch JL, et al. Development and evaluation of the universal NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg 2013;217:833e842. 12. Cohen ME, Bilmoria KY, Ko CY, Hall BL. Development of an American College of Surgeons National Surgery Quality Improvement Program: morbidity and mortality risk calculator for colorectal surgery. J Am Coll Surg 2009;208: 1009e1016. 13. American College of Surgeons. Surgical risk calculator. Available at: http://riskcalculator.facs.org/. Accessed August 1, 2014. 14. Ingraham AM, Richards KE, Hall BL, Ko CY. Quality improvement in surgery: the American College of Surgeons Surgical Quality Improvement Program approach. Adv Surg 2010;44:251e267.
286
Cologne et al
American College of Surgeons NSQIP Calculator
15. Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998;228:491e507. 16. Khuri SF, Daley J, Henderson WG. The comparative assessment and improvement of quality of surgical care in the Department of Veterans Affairs. Arch Surg 2002;137:20e27. 17. Law WL, Lam CM, Lee YM. Evaluation of laparoscopic colorectal resection with POSSUM, Portsmouth POSSUM, and Colorectal POSSUM. Br J Surg 2006;93:94e99. 18. Senagore AJ, Warmuth AJ, Delaney CP, et al. POSSUM, p-POSSUM, and Cr-POSSUM: implementation issues in a United States health care system for prediction of outcome for colon cancer resection. Dis Colon Rectum 2004;47: 1435e1441. 19. Delaney CP, Chang E, Senagore AJ, Broder M. Clinical outcomes and resource utilization associated with laparoscopic and open colectomy using a large national database. Ann Surg 2008;247:819e824. 20. Schwenk W, Haase O, Neudecker J, et al. Short term benefits for laparoscopic colorectal resection. Cochrane Database Syst Rev 2005;20:CD003145. 21. Guillou PJ, Quirke P, Thorpe H, et al. Short-term endpoints of conventional versus laparoscopic-assisted surgery in patients with colorectal cancer (MRC CLASSIC trial): multicentre, randomized controlled trial. Lancet 2005;365: 1718e1726. 22. Veldkamp R, Kuhry E, Hop WC, et al. Laparoscopic surgery versus open surgery for colon cancer: short-term outcomes of a randomized trial. Lancet Oncol 2005;6:477e484. 23. Al-Homoud S, Purkayastha S, Aziz O, et al. Evaluating operative risk in colorectal cancer surgery: ASA and POSSUMbased predictive models. Surg Oncol 2004;13:83e92. 24. Constantinides VA, Tekkis PP, Senapati A. Association of Coloproctology of Great Britain and Ireland. Comparison of POSSUM scoring systems and the surgical risk scale in patients undergoing surgery for complicated diverticular disease. Dis Colon Rectum 2006;49:1322e1331. 25. Copeland GP, Jones D, Walters M. POSSUM: a scoring system for surgical audit. Br J Surg 1991;78:355e360. 26. Poon JT, Chan B, Law WL. Evaluation of P-POSSUM in surgery for obstructing colorectal cancer and correlation of the
27. 28. 29.
30. 31.
32. 33. 34.
35.
36. 37.
38.
J Am Coll Surg
predicted mortality with different surgical options. Dis Colon Rectum 2005;48:493e498. Benneyan JC, Lloyd RC, Plsek PE. Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care 2003;12:458e464. Paulk MC. Software metrics and analysis: identifying outliers, understanding the process. Available at: http://home.comcast.net/ wmark.paulk/papers/p2009a.pdf. Accessed September 11, 2014. Romagnoli JA, Palazoglu A. Data processing and reconciliation. In: Romagnoli JA, Palazoglu A, eds. Introduction to Process Control. 2nd ed. Boca Raton, FL: CRC Taylor and Francis Group, LLC; 2012:515e539. Keller DS, Hashemi L, Lu M, Delaney CP. Short-term outcomes for robotic colorectal surgery by provider volume. J Am Coll Surg 2013;217:1063e1069. Harmon JW, Tang DG, Gordon TA, et al. Hospital volume can serve as a surrogate for surgeon volume for achieving excellent outcomes in colorectal resection. Ann Surg 1999;230: 404e411; discussion 411e413. Farjoodi P, Skolasky RL, Riley LH. The effects of hospital and surgeon volume on postoperative complications after Lumbar Spine surgery. Spine (Phila Pa 1976) 2011;36:2069e2075. Billingsley KG, Morris AM, Green P, et al. Does surgeon case volume influence nonfatal adverse outcomes after rectal cancer resection? J Am Coll Surg 2008;206:1167e1177. Keller DS, Bankwitz B, Nobel T, Delaney CP. Using frailty to predict who will fail early discharge after laparoscopic colorectal surgery with an established recovery pathway. Dis Colon Rectum 2014;57:337e342. Amrock LG, Neuman MD, Lin HM, Deiner S. Can routine preoperative data predict adverse outcomes in the elderly? Development and validation of a simple risk model incorporating a chart-derived frailty score. J Am Coll Surg 2014; 219:684e694. Wick EC, Shore AD, Hirose K, et al. Readmission rates and cost following colorectal surgery. Dis Colon Rectum 2011; 54:1475e1479. Lawrence JK, Keller DS, Samia H, et al. Discharge within 24 to 72 hours of colorectal surgery is associated with low readmission rates when using enhanced recovery pathways. J Am Coll Surg 2013;216:390e394. Keller DS, Khorgami Z, Swendseid B, et al. Identifying causes for high readmission rates after stoma reversal. Surg Endosc 2014;28:1263e1268.