- Email: [email protected]
External validation of the Revised Cardiac Risk Index and National Surgical Quality Improvement Program Myocardial Infarction and Cardiac Arrest calculator in noncardiac vascular surgery
British Journal of Anaesthesia, 123 (4): 421e429 (2019) doi: 10.1016/j.bja.2019.05.029 Advance Access Publication Date: 27 June 2019 Cardiovascular
External validation of the Revised Cardiac Risk Index and National Surgical Quality Improvement Program Myocardial Infarction and Cardiac Arrest calculator in noncardiac vascular surgery rka1, R. A. Archbold5, B. Biccard6, E. Duceppe2,3, J. Fronczek1, K. Polok1, P. J. Devereaux2,3, J. Go 2,3,4 7 ska1 and W. Szczeklik1,* Y. Le Manach , D. I. Sessler , M. Duchin 1
Department of Intensive Care and Perioperative Medicine, Jagiellonian University Medical College, Krakow,
Poland, 2Population Health Research Institute, Hamilton, ON, Canada, 3Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada, 4Department of Anesthesia and Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada, 5General & Interventional Cardiology, Barts Heart Centre, St. Bartholomew’s Hospital, London, UK, 6Department of Anaesthesia and Perioperative Medicine, Groote Schuur Hospital and University of Cape Town, South Africa and 7Department of Outcomes Research, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA *Corresponding author. E-mail: [email protected]
Abstract Background: The National Surgical Quality Improvement Program Myocardial Infarction & Cardiac Arrest (NSQIP MICA) calculator and the Revised Cardiac Risk Index (RCRI) were derived using currently outdated methods of diagnosing perioperative myocardial infarctions. We tested the external validity of these tools in a setting of a systematic perioperative cardiac biomarker measurement. Methods: Analysis of routinely collected data nested in the Vascular Events In Noncardiac Surgery Patients Cohort Evaluation Study. A consecutive sample of patients 45 yr old undergoing in-hospital noncardiac surgery in a single tertiary care centre was enrolled. The predictive performance of the models was tested in terms of the occurrence of major cardiac complications defined as a composite of a nonfatal myocardial infarction, a nonfatal cardiac arrest, or a cardiac death within 30 days after surgery. The plasma concentration of high-sensitivity troponin T was measured before surgery, 6e12 h after operation, and on the first, second, and third days after surgery. Myocardial infarction was diagnosed according to the Third Universal Definition. Results: The median age was 65 (59e72) yr, and 704/870 (80.9%) subjects were male. The primary outcome occurred in 76/ 870 (8.7%; 95% confidence interval [CI], 6.9e10.8%) patients. The c-statistic was 0.64 (95% CI, 0.57e0.70) for the NSQIP MICA and 0.60 (95% CI, 0.54e0.65) for the RCRI. Predicted risks were systematically underestimated in calibration belts (P<0.001). The RCRI and the NSQIP MICA showed no clinical utility before recalibration. Conclusions: The NSQIP and RCRI models had limited predictive performance in this at-risk population. The recently updated version of the RCRI was more reliable than the original index. Keywords: cardiac arrest; myocardial infarction; perioperative care; risk assessment; vascular surgical procedures
Editorial decision: 03 May 2019; Accepted: 3 May 2019 © 2019 British Journal of Anaesthesia. Published by Elsevier Ltd. All rights reserved. For Permissions, please email: [email protected]
421
422
-
Fronczek et al.
Editor’s key points Surgical risk calculators have not kept abreast of the contemporary definition of postoperative myocardial infarction. The NSQIP MICA is prone to underestimation of risk for postoperative myocardial infarction, especially in highrisk settings. An update of risk models slightly increased their performance and clinical utility. Individual risk prediction for postoperative cardiac events is fraught.
Consecutive individuals scheduled for vascular surgery (n=1304)
The patient (n=372) or operating surgeon (n=12) refused to participate Missing preoperative serum creatinine level (n=9) Missing preoperative functional dependency status (n=12) ASA Physical Status class V (n=17; 8/17 with missing data) Reason for surgery (n=882) • Peripheral artery disease (n=639; 72%) • Abdominal aortic aneurysm (n=182; 21%) • Carotid artery disease (n=61; 7%)
Missing perioperative or 30-day follow-up outcome data (n=12)
Complete-case analysis (n=870)
Cardiac complications are the leading cause of death after noncardiac surgery.1 In vascular surgical population, invasive procedures are performed on patients with highly prevalent coronary artery disease and cerebrovascular disease.2e4 Accurate cardiac risk prediction can inform decisions about the appropriateness of surgery, guide perioperative management, and identify patients who require close monitoring after surgery.5 Two dedicated tools are recommended for preoperative cardiac risk assessment: the Revised Cardiac Risk Index (RCRI) and the National Surgical Quality Improvement Program (NSQIP) Myocardial Infarction and Cardiac Arrest (MICA) calculator.6,7 Both models had limited predictive performance in vascular surgery at the time of their derivation.8,9 The RCRI was later shown to underestimate the risk of perioperative cardiac events.10,11 The NSQIP MICA calculator has not been comprehensively tested outside of the NSQIP registry and requires external validation.12e14 Perioperative cardiac monitoring has seen major improvements in the past two decades.15,16 Systematic postoperative biomarker screening demonstrated that most perioperative myocardial infarctions (MIs) have no clinical symptoms, but they are associated with an equally poor prognosis as symptomatic MIs.17 Sensitivity of cardiac biomarkers used to detect MI has been increasing dramatically.18,19 In light of these changes, we aimed to study both models in a contemporary cohort of patients undergoing vascular surgery who had highsensitivity troponin measured routinely in the perioperative period.
Methods The Vascular Events In Noncardiac Surgery Patients Cohort Evaluation (VISION) Study was an international prospective cohort study evaluating major complications in a representative sample of patients undergoing noncardiac surgery conducted between October 2010 and November 2013 which enrolled individuals 45 yr of age who had noncardiac surgery under general or regional anaesthesia and stayed at least 1 night in the hospital after surgery. Previous reports from the VISION Study described in detail the screening, enrolment, and data collection processes.15,20 This substudy included patients from a single tertiary care centre who had surgery for peripheral artery disease, abdominal aortic aneurysm, or carotid artery disease (i.e. approximately 93% of all surgical procedures performed at our site as part of the VISION Study). Other types of procedures (e.g. urgent or emergent abdominal surgery) were not reported in this substudy. All patients included in this cohort were of Caucasian descent. The study
Fig 1. Study flowchart.
was approved by the local ethics committee (Jagiellonian University Bioethics Committee). The protocol complied with the provisions of the Helsinki Declaration. All participants provided written informed consent before enrolment. Research personnel performed clinical examination, interviewed patients, and reviewed their medical records to obtain information on preoperative baseline characteristics (e.g. age, history of coronary artery disease, congestive heart failure, cerebrovascular disease, diabetes mellitus, functional status before surgery). Variables used to calculate the RCRI and the NSQIP MICA are reported in Supplementary Table S1. The primary outcome was a composite of a nonfatal MI, a nonfatal cardiac arrest, or a cardiac death within 30 days after surgery. We defined MI according to the Third Universal Definition.21 The increase or decrease in plasma concentration of high-sensitivity troponin T (Elecsys 2010 analyser; Roche Diagnostics, Meylan, France) was detected using routine troponin measurements before surgery, 6e12 h after operation, and on the first, second, and third postoperative days, according to a predefined schedule.15,20 Patients with a troponin level of at least 14 ng L1 (i.e. a threshold considered abnormal by manufacturer) were assessed for ischaemic features (e.g. ischaemic symptoms, ischaemic electrocardiographic findings). All patients were followed throughout hospitalisation and contacted 30 days after surgery by phone to determine the occurrence of complications. Documentation was obtained if patients (or next of kin) indicated that they had experienced an outcome event. Outcomes were evaluated by expert unblinded physicians from the VISION Adjudication Committee before statistical analysis.15
Statistical analysis The report was written in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement and contained a checklist of required items.22 External validation of the models’ predictive performance included an evaluation of discrimination, calibration, and clinical utility.23 Discrimination was assessed using concordance statistics (c-statistics; areas under the receiver operating characteristic [ROC] curves), which were compared with a bootstrap procedure. We used benchmark values to assess the potential influence of differences between the derivation and external validation
External validation of the RCRI and NSQIP MICA
studies on the models’ discrimination.24 There are two types of benchmark values referred to as the case-mix corrected performance and refitted performance which address two issues that are commonly encountered at model validation: 1. The distribution of patient characteristics (case-mix heterogeneity) may be different in the external validation sample than in the development cohort. 2. The estimated effects of predictors may be different in the new setting. We obtained the case-mix corrected c-statistic after generating the outcome from predictions 1000 times in each subject. The refitted c-statistic was calculated by re-estimating regression coefficients in the validation sample and applying an optimism correction to prevent overfitting.24 We presented both calibration curves and calibration belts, the latter of which relate predicted probabilities with the observed number of events using a polynomial logistic
-
423
regression and provide a graphical representation of goodness-of-fit along with confidence intervals (CIs) and a likelihood ratio test for miscalibration.25 Clinical utility was evaluated using net benefit (NB), which is a performance measure that takes into account the consequences of using risk prediction models in making decisions based on risk thresholds.26 NB puts benefits and harms on the same scale and is conveniently expressed in the unit of true positives or true negatives.26,27 It is calculated as follows: 1. Choose a risk threshold: e Patients with predicted risk at or above the threshold are treated as high-risk. e Patients with predicted risk below the threshold are treated as low-risk. 2. Count the number of: e High-risk patients who had an event (true positives) and those who did not (false positives).
Table 1 Demographic and clinical characteristics of the current study’s cohort (i.e. external validation cohort) compared with development samples of the original models. Values are presented as percentages (%) for categorical variables and mean (SD) for continuous variables. COPD, chronic obstructive pulmonary disease; MI, myocardial infarction; MICA, myocardial infarction and cardiac arrest; NR, not reported; NSQIP, National Surgical Quality Improvement Program; RCRI, Revised Cardiac Risk Index; SD, standard deviation. Characteristics No. (%) with data
RCRI derivation cohort (n¼2893)
NSQIP derivation cohort (n¼211 410)
External validation cohort (n¼870)
Demographic Male sex Age, mean (SD), yr
1374 (47.5) 66.4 (10.1)
42.8 No MICA 55.1 (17.2) MICA 69.2 (13.5)
704 (80.9) 65.8 (8.5)
Cardiovascular Hypertension Coronary artery disease Diabetes Diabetes treated with insulin Smoker pack years, mean (SD)
NR 951 (32.9) NR 112 (3.9) NR
45.2 NR 14.6 NR No MICA 11.0 (22.7) MICA 24.4 (34.5) 0.7 2.9 4.5 NR
661 (76.0) 395 (45.4) 196 (22.5) 132 (15.2) 41.9 (28.6)
MI in past 180 days Prior transient ischaemic attack Prior cerebrovascular accident Congestive heart failure Other medical BMI, mean (SD), kg m2
19 (0.7) 175 (6) 155 (5.4) 434 (15.0)
COPD Elevated preoperative creatinine level Serum creatinine level >1.5 mg dl1 Serum creatinine level >2.0 mg dl1 Dialysis Functional status before surgery Independent Partially dependent Totally dependent Surgical Type of surgery Abdominal aortic aneurysm Other vascular Intraperitoneal, intrathoracic, or suprainguinal vascular procedures Emergency status ASA Physical Status class ASA 1 ASA 2 ASA 3 ASA 4 ASA 5
NR
10 21 93 97
(1.1) (2.4) (10.7) (11.1)
NR
No MICA 30.7 (8.8) MICA 28.9 (8.3) 4.7
25.3 (4.1) 341 (39.2)
NR 103 (3.6) NR
9.8 NR 2.4
38 (4.4) 10 (1.1) 4 (0.5)
NR
93.0 4.9 2.1
806 (92.6) 49 (5.6) 15 (1.7)
110 (3.8) 498 (17.2) 894 (30.9)
2.2 8.3 NR
181 (20.8) 689 (79.2) 379 (43.6)
0
12.6
0
140 (4.8) 1558 (53.9) 1078 (37.3) 81 (2.8) 0
10.4 45.2 37.2 6.9 0.3
0 179 (20.6) 628 (72.2) 63 (7.2) 0
424
-
Fronczek et al.
e Low-risk patients who had an event (false negatives) and those who did not (true negatives). 3. Calculate the number of net true positives and negatives per 1000 patients according to the following formulas, where N represents the total sample size:
Decision curves allow for a sensitivity analysis of many potentially meaningful risk thresholds. The curve with the highest NB achieved at reasonably chosen thresholds has the highest clinical utility.
Net true positives per 1000 patients ¼ 1000 * True positves False positives * N N Risk threshold 1 Risk threshold
Model updating
Net true negatives per 1000 patients ¼ 1000 * True negatives False negatives N N 1 Risk threshold Risk threshold
When a risk prediction model is validated in a new setting, its predictive performance may be worse than in the original derivation study. The basic aim of model updating is to improve the model’s predictions in the new sample by adjusting the model to the baseline risk of events in the external validation cohort (i.e. adjustment of the intercept in a calibration plot) or revising the estimated effects of predictors by applying an overall correction factor to regression coefficients (i.e. adjustment of the calibration slope). This updated (recalibrated) model is more likely to be generalisable to a new setting.22 The Canadian Cardiovascular Society (CCS) Guidelines on Perioperative Cardiac Risk Assessment and
Fig 2. Calibration belts. Predicted risks from a well-calibrated model should agree with the observed frequencies of outcome events, resulting in a calibration belt which approximates the diagonal red line of perfect calibration. The test indicates significant miscalibration when 80 (light grey area) and 95% (dark grey area) confidence bands deviate from this line (i.e. P<0.05). (a) RCRI; (b) CCS RCRI; (c) NSQIP MICA; (d) recalibrated NSQIP MICA. RCRI, Revised Cardiac Risk Index; CCS, Canadian Cardiovascular Society; MICA, Myocardial Infarction & Cardiac Arrest; NSQIP, National Surgical Quality Improvement Program.
External validation of the RCRI and NSQIP MICA
-
425
Management for Patients Who Undergo Noncardiac Surgery have recently updated risk estimates of the RCRI based on high-quality studies that measured troponin levels routinely in the perioperative period.28 We used these with no additional changes in our cohort. The NSQIP MICA required a dedicated logistic recalibration procedure (a detailed description of this method was provided in eMethods).29 The recalibrated NSQIP MICA decision curve was corrected for overoptimism using a 200-repetition 10-fold cross validation. The authors of the paper had a full access to all the data in the study and take responsibility for its integrity and the data analysis. All analyses were conducted in R Studio, packages: givitiR (calibration belts), rms (optimism-corrected calibration plot), Calibration Curves (flexible calibration curves), pROC (c-statistics), and rmda (decision curve analysis).25,30e34 Bootstrapped 95% CIs were obtained from 2000 samples.
Results Participant characteristics and the number of observed events The characteristics of the 870 patients included in the study (Fig. 1) are presented in Table 1 and compared with development samples of the original models. The case-mix severity of our sample appeared to be higher than that of both derivation sets in a crude comparison. The primary outcome occurred in 76 patients (8.7%; 95% CI, 6.9e10.8%). We observed 69 MIs, 2 concurrent cardiac arrests (non-fatal events in patients who suffered an MI), and 7 cardiac deaths.
Fig 3. Decision curve analysis. The curve with the highest value of net benefit at a particular risk threshold has the highest clinical utility. The grey curve and the black line assume that all patients are treated as high-risk or low-risk, respectively. RCRI, Revised Cardiac Risk Index; CCS, Canadian Cardiovascular Society; MICA, Myocardial Infarction & Cardiac Arrest; NSQIP, National Surgical Quality Improvement Program. Multiplying the value of net benefit by 1000 gives the number of net true positives per 1000 patients.
External validation The c-statistic for the prediction of the primary outcome was 0.64 (95% CI, 0.57e0.70) for the NSQIP MICA. The RCRI c-statistic was 0.60 (95% CI, 0.54e0.65). The difference in c-statistics between the two models was not statistically significant (P¼0.23). Calibration belts were above the line of perfect fit across the entire range of models’ predictions (P<0.001, Fig. 2). The models showed no clinical utility in a decision curve analysis (Fig. 3). Case mix-corrected benchmark values of cstatistics were 0.70 for the NSQIP MICA and 0.77 for the RCRI. The upper bound of discriminatory capacity achieved by the NSQIP MICA calculator after refitting corresponded to an optimism-corrected c-statistic of 0.64, whereas the RCRI yielded a c-statistic of 0.59.
Update of the models Power of individual predictors was not readjusted, therefore discrimination remained unchanged (Fig. 4). Calibration belts for the recalibrated NSQIP MICA and the CCS’s RCRI agreed with the line of perfect fit, (P¼0.98 and P¼0.51, respectively) (Fig. 2). Clinical utility was higher compared with the original iterations of the models, with a superiority of the recalibrated NSQIP MICA over the CCS’s RCRI (Fig. 3). We provided exact values of NB at different risk thresholds in addition to decision curves (Table 2).
Discussion The two models recommended by international clinical practice guidelines for preoperative cardiac risk assessment in noncardiac surgery poorly predicted which patients would suffer a cardiac death, a nonfatal MI, or a nonfatal cardiac
arrest within 30 days of vascular surgery. Using a systematic high-sensitivity troponin T surveillance to diagnose MIs according to the Third Universal Definition, we showed that the RCRI and the NSQIP MICA had low discriminatory capacities, systematically underestimated the risk for major cardiac complications, and achieved no clinical utility in the decision curve analysis. The latter two issues were to some extent corrected by models’ update. The reduced discrimination of the RCRI in vascular surgery has been previously reported. In the NSQIP MICA derivation study, the calculator’s c-statistic in vascular surgery was 0.75, whereas the RCRI applied to the same dataset yielded an area under curve of 0.59.9 In the original RCRI derivation study, the c-statistic in the vascular surgical subgroup was 0.77 (0.74e0.81), whereas Ford and colleagues35 in their metaanalysis showed a c-statistic for classifying patients undergoing vascular procedures as low risk vs intermediate to high risk of cardiac events equal to 0.65 (0.60e0.69).8 There is substantial evidence on the underestimation of risk by the RCRI, which ultimately led the CCS to issue an update of its risk estimates.28 Our results suggest that the updated risk categories better reflect the actual incidence of major cardiac complications in patients having noncardiac vascular surgery than those provided by the original RCRI. MI is a pivotal outcome for cardiac risk prediction models, yet both of the tested tools operate on currently outdated definitions of this outcome. There are large discrepancies in terms of cardiac biomarkers used to diagnose MI between the RCRI, the NSQIP MICA, and the current clinical practice. Most perioperative MIs are asymptomatic, but the prognosis associated with these events is poor regardless of their clinical
426
-
Fronczek et al.
Fig 4. Calibration curves for the original (left) and recalibrated (right) models. ABC for validation: A, alpha (intercept); B, beta (slope); C, cstatistic (area under the curve). (a) RCRI; (b) CCS RCRI; (c) NSQIP MICA; (d) recalibrated NSQIP MICA. *c-Statistics remained unchanged after recalibration. RCRI, Revised Cardiac Risk Index; CCS, Canadian Cardiovascular Society; MICA, Myocardial Infarction & Cardiac Arrest; NSQIP, National Surgical Quality Improvement Program.
Table 2 Net benefit values of the tested risk prediction models at different risk thresholds. aValues exceeding 1000 net true negatives per 1000 patients are possible in a penalised analysis and are associated with an extremely poor clinical utility of a model. CCS, Canadian Cardiovascular Society; CI, confidence interval; MICA, myocardial infarction and cardiac arrest; NSQIP, National Surgical Quality Improvement Program; RCRI, Revised Cardiac Risk Index. Prediction model
Original RCRI
NSQIP MICA Recalibrated CCS RCRI
NSQIP MICA after logistic recalibration
Risk threshold
Net true positives per 1000 patients (95% CI)
Net true negatives per 1000 patients (95% CI)
0.9% 6% 11% 1%
75 (56e92) 27 (11e42) 0 (e9 to 9) 36 (23e49)
e483 (e986 to 149) e30 (e227 to 182) 208 (55e371) e4131 (e5516 to e2733)a
6% 10.1% 15% 5% 10% 15%
33 (12e51) 2 (e7 to 10) 6 (e16 to 2) 44 (24e63) 10 (e4 to 23) 9 (e1 to 17)
59 (e11 to 147) 151 (e13 to 322) 381 (270e499) 93 (11e197) 215 (78e364) 470 (361e590)
External validation of the RCRI and NSQIP MICA
presentation.17 The NSQIP MICA model took into account only overt MIs that prompted an administration of a troponin measurement. Based on studies that included a systematic cardiac biomarker surveillance after surgery, predictions of this calculator are representative for only about 35% of the actual postoperative cardiac events.1 Furthermore, new troponin elevations lower than three times the upper reference range were considered minor and were excluded from reporting of the outcome in the NSQIP MICA, which dramatically contrasts to the standard 99th percentile diagnostic threshold retained in the Fourth Universal Definition of MI.36 This makes the NSQIP MICA prone to underestimation of risk for postoperative MI, especially in high-incidence scenarios.13,37 The RCRI was developed almost two decades ago and based on serial CK/CK-MB measurements with regard to the biomarker criterion of MI.8 This external validation study used high-sensitivity troponins, which are able to detect twice as many MIs as with non-high-sensitivity assays.18 Our definition of the composite endpoint was slightly different than in the original studies and included nonfatal MIs, nonfatal cardiac arrests, and cardiac deaths within 30 days after surgery. This should not threaten the validity of our findings as we were able to capture essentially all cases of perioperative myocardial ischaemia. A more detailed description of differences between the development and validation studies is provided in Supplementary Tables 2 and 3. Our study has several caveats. The RCRI and the NSQIP MICA were derived in a mixed surgical setting, and we attempted to externally validate them in a relatively small sample of patients having exclusively vascular procedures in one centre, which limits the generalisability of our findings. The number of events was smaller than the recommended 100, yet the original validation set of the RCRI comprised no more than 36 events, which is 50% less than we reported.38 One could argue that poor c-statistics shown in this study are a natural consequence of the sample’s homogeneity. However, the expected discrimination (i.e. case-mix corrected c-statistics simulated under the assumption that the tested models were statistically correct for the external validation cohort) was similar to that observed in subgroups of patients having vascular surgery in the original studies.8,9,24 Interestingly, the upper bound of the models’ discriminatory capacity calculated after re-estimating regression coefficients of predictors included in the RCRI and NSQIP MICA models in our sample was essentially the same as when we used original coefficients (i.e. refitted c-statistics were very similar to standard c-statistics). This suggests that additional predictors (e.g. natriuretic peptide measurements or standardised measures of functional capacity) may be needed to achieve better predictive performance in the future.39,40 The higher baseline risk of major cardiac complications in vascular surgical patients compared with the general surgery setting could have contributed to the understatement of predicted risks. We tried to account for this issue by recalibrating the models.31 The main strength of our study is that every participant had high-sensitivity troponin T levels measured multiple times in the perioperative period. The high quality of the outcome assessment was supported by an independent adjudication committee. Our study confirmed some of the previously raised concerns about the clinical utility of these tools and provided new evidence by independently testing the generalisability of the NSQIP MICA calculator to vascular surgery. The introduction of a systematic screening with highsensitivity cardiac biomarkers greatly improved the
-
427
detection of myocardial ischaemia in the perioperative period. The recent CCS Guidelines on Perioperative Cardiac Risk Assessment and Management for Patients Who Undergo Noncardiac Surgery highlighted this issue and recommended a daily troponin measurement for 48e72 h after surgery in patients who are at an increased risk of cardiac complications (e.g. those with an elevated natriuretic peptide measurement before surgery or an RCRI score 1).28 Our findings support the use of the updated version of the RCRI endorsed by the CCS which operates on higher risk thresholds than the original index and is more likely to produce reliable predictions.41 The NSQIP MICA requires further validation, preferably in multicentre studies incorporating a contemporary definition of MI. Large-scale studies using the VISION Study data are underway and should provide further advancements in this area.42
Authors’ contributions Conception of study and design, acquisition, analysis, and interpretation of data: all authors. Drafting and critical revision: all authors. Final approval of the version to be published: all authors. Accountable for all aspects of the work: all authors.
Acknowledgements The authors thank Pavel Roshanov and the three anonymous reviewers for their helpful and insightful comments on this work.
Declarations of interest ED is a co-applicant on investigator-initiated research grants received from Roche Diagnostics and Abbott Diagnostics. Abbott Diagnosticsdmodest. Roche Diagnosticsdmodest. PJD is a member of a research group with a policy of not accepting honorariums or other payments from industry for their own personal financial gain. They do accept honorariums/payments from industry to support research endeavours and costs to participate in meetings. Based on study questions PJD has originated and grants he has written, he has received grants from Abbott Diagnostics, Boehringer Ingelheim, Philips Healthcare, Roche Diagnostics. PJD has participated in an advisory board meeting for Boehringer Ingelheim. Abbott Diagnosticsdsignificant. Boehringer Ingelheimdsignificant. Philips Healthcaredsignificant. Roche Diagnosticsdsignificant.
Funding Roche Diagnostic provided the Troponin T assays for the VISION Study. The VISION Study funding sources had no role in the design and conduct of the study, in the collection, analysis, and interpretation of the data, or in the preparation, review or approval of the manuscript.
Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi.org/10.1016/j.bja.2019.05.029.
428
-
Fronczek et al.
References 1. Devereaux PJ, Sessler DI. Cardiac complications in patients undergoing major noncardiac surgery. N Engl J Med 2015; 373: 2258e69 2. Patel AY, Eagle KA, Vaishnava P. Cardiac risk of noncardiac surgery. J Am Coll Cardiol 2015; 66: 2140e8 3. Aboyans V, Ricco J-B, Bartelink M-LEL, et al. 2017 ESC guidelines on the diagnosis and treatment of peripheral arterial diseases, in collaboration with the European Society for Vascular Surgery (ESVS). Eur Heart J 2018; 39: 763e816 rka J, Polok K, Fronczek J, et al. Myocardial injury is 4. Go more common than deep venous thrombosis after vascular surgery and is associated with a high one year mortality risk. Eur J Vasc Endovasc Surg 2018; 56: 264e70 5. Devereaux PJ, Goldman L, Cook DJ, Gilbert K, Leslie K, Guyatt GH. Perioperative cardiac events in patients undergoing noncardiac surgery: a review of the magnitude of the problem, the pathophysiology of the events and methods to estimate and communicate risk. Can Med Assoc J 2005; 173: 627e34 6. Fleisher LA, Fleischmann KE, Auerbach AD, et al. 2014 ACC/AHA guideline on perioperative cardiovascular evaluation and management of patients undergoing noncardiac surgery: a report of the American College of Cardiology/American Heart Association task force on practice guidelines. Circulation 2014; 130: 2215e45 7. Kristensen SD, Knuuti J, Saraste A, et al. 2014 ESC/ESA Guidelines on non-cardiac surgery: cardiovascular assessment and management: the Joint Task Force on noncardiac surgery: cardiovascular assessment and management of the European Society of Cardiology (ESC) and the European Society of Anaesth. Eur Heart J 2014; 35: 2383e431 8. Lee TH, Marcantonio ER, Mangione CM, et al. Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation 1999; 100: 1043e9 9. Gupta PK, Gupta H, Sundaram A, et al. Development and validation of a risk calculator for prediction of cardiac risk after surgery. Circulation 2011; 124: 381e7 10. Davis C, Tait G, Carroll J, Wijeysundera DN, Beattie WS. The Revised Cardiac Risk Index in the new millennium: a single-centre prospective cohort re-evaluation of the original variables in 9,519 consecutive elective surgical patients. Can J Anaesth 2012; 59: 1013e22 11. Bertges DJ, Goodney PP, Zhao Y, et al. The vascular study group of new England cardiac risk index (VSG-CRI) predicts cardiac complications more accurately than the revised cardiac risk index in vascular surgery patients. J Vasc Surg 2010; 52: 674e83 12. Peterson B, Ghahramani M, Harris S, et al. Usefulness of the myocardial infarction and cardiac arrest calculator as a discriminator of adverse cardiac events after elective hip and knee surgery. Am J Cardiol 2016; 117: 1992e5 13. Juo YY, Mantha A, Ebrahimi R, Ziaeian B, Benharash P. Incidence of myocardial infarction after high-risk vascular operations in adults. JAMA Surg 2017; 152, e173360 14. Duncan D, Wijeysundera DN. Preoperative cardiac evaluation and management of the patient undergoing major vascular surgery. Int Anesthesiol Clin 2016; 54: 1e32 15. Devereaux PJ, Biccard BM, Sigamani A, et al. Association of postoperative high-sensitivity troponin levels with
16. 17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27. 28.
29.
30. 31.
32.
33.
myocardial injury and 30-day mortality among patients undergoing noncardiac surgery. JAMA 2017; 317: 1642e51 Biccard B. Proposed research plan for the derivation of a new cardiac risk index. Anesth Analg 2015; 120: 543e53 Devereaux PJ, Xavier D, Pogue J, et al. Characteristics and short-term prognosis of perioperative myocardial infarction in patients undergoing noncardiac surgery. Ann Intern Med 2011; 154: 523e8 Brown JC, Samaha E, Rao S, et al. High-sensitivity cardiac troponin T improves the diagnosis of perioperative MI. Anesth Analg 2017; 125: 1455e62 Twerenbold R, Jaeger C, Rubini Gimenez M, et al. Impact of high-sensitivity cardiac troponin on use of coronary angiography, cardiac stress testing, and time to discharge in suspected acute myocardial infarction. Eur Heart J 2016; 37: 3324e32 Vascular Events In Noncardiac Surgery Patients Cohort Evaluation (VISION) Study Investigators, Devereaux PJ, Chan MTV, Alonso-Coello P, et al. Association between postoperative troponin Levels and 30-day mortality among patients undergoing noncardiac surgery. JAMA 2012; 307: 2295e304 Thygesen K, Alpert JS, Jaffe AS, et al. Third universal definition of myocardial infarction. Eur Heart J 2012; 33: 2551e67 Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015; 350: g7594 Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 2014; 35: 1925e31 Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971e80 Nattino G, Finazzi S, Bertolini G. A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes. Stat Med 2014; 33: 2390e407 Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ 2016; 352: i6 Qian X, Lee W-C, Rousson V, et al. Decision curve analysis: a technical note. Ann Transl Med 2018; 6: 308 Duceppe E, Parlow J, MacDonald P, et al. Canadian Cardiovascular Society guidelines on perioperative cardiac risk assessment and management for patients who undergo noncardiac surgery. Can J Cardiol 2017; 33: 17e32 Van Calster B, Van Hoorde K, Vergouwe Y, et al. Validation and updating of risk models based on multinomial logistic regression. Diagn Progn Res 2017; 1: 2 Harrell Jr FE. rms: regression modeling strategies. R package version 5.1-3 2019 Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 2016; 74: 167e76 Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and Sþ to analyze and compare ROC curves. BMC Bioinform 2011; 12: 77 Brown M. rmda: risk model decision analysis 2017
External validation of the RCRI and NSQIP MICA
34. RStudio Team. RStudio: integrated development environment for R. Boston, MA 2016. Available from, http://www.rstudio. com/. [Accessed 30 March 2019]. accessed 35. Ford MK, Beattie WS, Wijeysundera DN. Systematic review: prediction of perioperative cardiac complications and mortality by the revised cardiac risk index. Ann Intern Med 2010; 152: 26e35 36. Thygesen K, Alpert JS, Jaffe AS, et al. Fourth universal definition of myocardial infarction (2018). Eur Heart J 2019; 40: 237e69 37. Polok K, Fronczek J, Szczeklik W. Myocardial infarction after vascular surgery: a systematic troponin surveillance and a uniform definition is needed. JAMA Surg 2018; 153: 496 38. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med 2016; 35: 214e26 39. Rodseth RN, Biccard BM, Le Manach Y, et al. The prognostic value of pre-operative and post-operative B-type
-
429
natriuretic peptides in patients undergoing noncardiac surgery: B-type natriuretic peptide and N-terminal fragment of pro-B-type natriuretic peptide: a systematic review and individual patient data meta-analysis. J Am Coll Cardiol 2014; 63: 170e80 40. Wijeysundera DN, Pearse RM, Shulman MA, et al. Assessment of functional capacity before major noncardiac surgery: an international, prospective cohort study. Lancet 2018; 391: 2631e40 41. MDCalc. Revised Cardiac risk index for pre-operative risk. Available from: https://www.mdcalc.com/revisedcardiac-risk-index-pre-operative-risk. [Accessed 30 March 2019]. accessed 42. Roshanov PS, Walsh M, Devereaux PJ, et al. External validation of the Revised Cardiac Risk Index and update of its renal variable to predict 30-day risk of major cardiac complications after non-cardiac surgery: rationale and plan for analyses of the VISION study. BMJ Open 2017; 7, e013510
Handling editor: H.C. Hemmings Jr
External validation of the Revised Cardiac Risk Index and National Surgical Quality Improvement Program Myocardial Infarction and Cardiac Arrest calculator in noncardiac vascular surgery rka1, R. A. Archbold5, B. Biccard6, E. Duceppe2,3, J. Fronczek1, K. Polok1, P. J. Devereaux2,3, J. Go 2,3,4 7 ska1 and W. Szczeklik1,* Y. Le Manach , D. I. Sessler , M. Duchin 1
Department of Intensive Care and Perioperative Medicine, Jagiellonian University Medical College, Krakow,
Poland, 2Population Health Research Institute, Hamilton, ON, Canada, 3Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada, 4Department of Anesthesia and Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, ON, Canada, 5General & Interventional Cardiology, Barts Heart Centre, St. Bartholomew’s Hospital, London, UK, 6Department of Anaesthesia and Perioperative Medicine, Groote Schuur Hospital and University of Cape Town, South Africa and 7Department of Outcomes Research, Anesthesiology Institute, Cleveland Clinic, Cleveland, OH, USA *Corresponding author. E-mail: [email protected]
Abstract Background: The National Surgical Quality Improvement Program Myocardial Infarction & Cardiac Arrest (NSQIP MICA) calculator and the Revised Cardiac Risk Index (RCRI) were derived using currently outdated methods of diagnosing perioperative myocardial infarctions. We tested the external validity of these tools in a setting of a systematic perioperative cardiac biomarker measurement. Methods: Analysis of routinely collected data nested in the Vascular Events In Noncardiac Surgery Patients Cohort Evaluation Study. A consecutive sample of patients 45 yr old undergoing in-hospital noncardiac surgery in a single tertiary care centre was enrolled. The predictive performance of the models was tested in terms of the occurrence of major cardiac complications defined as a composite of a nonfatal myocardial infarction, a nonfatal cardiac arrest, or a cardiac death within 30 days after surgery. The plasma concentration of high-sensitivity troponin T was measured before surgery, 6e12 h after operation, and on the first, second, and third days after surgery. Myocardial infarction was diagnosed according to the Third Universal Definition. Results: The median age was 65 (59e72) yr, and 704/870 (80.9%) subjects were male. The primary outcome occurred in 76/ 870 (8.7%; 95% confidence interval [CI], 6.9e10.8%) patients. The c-statistic was 0.64 (95% CI, 0.57e0.70) for the NSQIP MICA and 0.60 (95% CI, 0.54e0.65) for the RCRI. Predicted risks were systematically underestimated in calibration belts (P<0.001). The RCRI and the NSQIP MICA showed no clinical utility before recalibration. Conclusions: The NSQIP and RCRI models had limited predictive performance in this at-risk population. The recently updated version of the RCRI was more reliable than the original index. Keywords: cardiac arrest; myocardial infarction; perioperative care; risk assessment; vascular surgical procedures
Editorial decision: 03 May 2019; Accepted: 3 May 2019 © 2019 British Journal of Anaesthesia. Published by Elsevier Ltd. All rights reserved. For Permissions, please email: [email protected]
421
422
-
Fronczek et al.
Editor’s key points Surgical risk calculators have not kept abreast of the contemporary definition of postoperative myocardial infarction. The NSQIP MICA is prone to underestimation of risk for postoperative myocardial infarction, especially in highrisk settings. An update of risk models slightly increased their performance and clinical utility. Individual risk prediction for postoperative cardiac events is fraught.
Consecutive individuals scheduled for vascular surgery (n=1304)
The patient (n=372) or operating surgeon (n=12) refused to participate Missing preoperative serum creatinine level (n=9) Missing preoperative functional dependency status (n=12) ASA Physical Status class V (n=17; 8/17 with missing data) Reason for surgery (n=882) • Peripheral artery disease (n=639; 72%) • Abdominal aortic aneurysm (n=182; 21%) • Carotid artery disease (n=61; 7%)
Missing perioperative or 30-day follow-up outcome data (n=12)
Complete-case analysis (n=870)
Cardiac complications are the leading cause of death after noncardiac surgery.1 In vascular surgical population, invasive procedures are performed on patients with highly prevalent coronary artery disease and cerebrovascular disease.2e4 Accurate cardiac risk prediction can inform decisions about the appropriateness of surgery, guide perioperative management, and identify patients who require close monitoring after surgery.5 Two dedicated tools are recommended for preoperative cardiac risk assessment: the Revised Cardiac Risk Index (RCRI) and the National Surgical Quality Improvement Program (NSQIP) Myocardial Infarction and Cardiac Arrest (MICA) calculator.6,7 Both models had limited predictive performance in vascular surgery at the time of their derivation.8,9 The RCRI was later shown to underestimate the risk of perioperative cardiac events.10,11 The NSQIP MICA calculator has not been comprehensively tested outside of the NSQIP registry and requires external validation.12e14 Perioperative cardiac monitoring has seen major improvements in the past two decades.15,16 Systematic postoperative biomarker screening demonstrated that most perioperative myocardial infarctions (MIs) have no clinical symptoms, but they are associated with an equally poor prognosis as symptomatic MIs.17 Sensitivity of cardiac biomarkers used to detect MI has been increasing dramatically.18,19 In light of these changes, we aimed to study both models in a contemporary cohort of patients undergoing vascular surgery who had highsensitivity troponin measured routinely in the perioperative period.
Methods The Vascular Events In Noncardiac Surgery Patients Cohort Evaluation (VISION) Study was an international prospective cohort study evaluating major complications in a representative sample of patients undergoing noncardiac surgery conducted between October 2010 and November 2013 which enrolled individuals 45 yr of age who had noncardiac surgery under general or regional anaesthesia and stayed at least 1 night in the hospital after surgery. Previous reports from the VISION Study described in detail the screening, enrolment, and data collection processes.15,20 This substudy included patients from a single tertiary care centre who had surgery for peripheral artery disease, abdominal aortic aneurysm, or carotid artery disease (i.e. approximately 93% of all surgical procedures performed at our site as part of the VISION Study). Other types of procedures (e.g. urgent or emergent abdominal surgery) were not reported in this substudy. All patients included in this cohort were of Caucasian descent. The study
Fig 1. Study flowchart.
was approved by the local ethics committee (Jagiellonian University Bioethics Committee). The protocol complied with the provisions of the Helsinki Declaration. All participants provided written informed consent before enrolment. Research personnel performed clinical examination, interviewed patients, and reviewed their medical records to obtain information on preoperative baseline characteristics (e.g. age, history of coronary artery disease, congestive heart failure, cerebrovascular disease, diabetes mellitus, functional status before surgery). Variables used to calculate the RCRI and the NSQIP MICA are reported in Supplementary Table S1. The primary outcome was a composite of a nonfatal MI, a nonfatal cardiac arrest, or a cardiac death within 30 days after surgery. We defined MI according to the Third Universal Definition.21 The increase or decrease in plasma concentration of high-sensitivity troponin T (Elecsys 2010 analyser; Roche Diagnostics, Meylan, France) was detected using routine troponin measurements before surgery, 6e12 h after operation, and on the first, second, and third postoperative days, according to a predefined schedule.15,20 Patients with a troponin level of at least 14 ng L1 (i.e. a threshold considered abnormal by manufacturer) were assessed for ischaemic features (e.g. ischaemic symptoms, ischaemic electrocardiographic findings). All patients were followed throughout hospitalisation and contacted 30 days after surgery by phone to determine the occurrence of complications. Documentation was obtained if patients (or next of kin) indicated that they had experienced an outcome event. Outcomes were evaluated by expert unblinded physicians from the VISION Adjudication Committee before statistical analysis.15
Statistical analysis The report was written in accordance with the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Statement and contained a checklist of required items.22 External validation of the models’ predictive performance included an evaluation of discrimination, calibration, and clinical utility.23 Discrimination was assessed using concordance statistics (c-statistics; areas under the receiver operating characteristic [ROC] curves), which were compared with a bootstrap procedure. We used benchmark values to assess the potential influence of differences between the derivation and external validation
External validation of the RCRI and NSQIP MICA
studies on the models’ discrimination.24 There are two types of benchmark values referred to as the case-mix corrected performance and refitted performance which address two issues that are commonly encountered at model validation: 1. The distribution of patient characteristics (case-mix heterogeneity) may be different in the external validation sample than in the development cohort. 2. The estimated effects of predictors may be different in the new setting. We obtained the case-mix corrected c-statistic after generating the outcome from predictions 1000 times in each subject. The refitted c-statistic was calculated by re-estimating regression coefficients in the validation sample and applying an optimism correction to prevent overfitting.24 We presented both calibration curves and calibration belts, the latter of which relate predicted probabilities with the observed number of events using a polynomial logistic
-
423
regression and provide a graphical representation of goodness-of-fit along with confidence intervals (CIs) and a likelihood ratio test for miscalibration.25 Clinical utility was evaluated using net benefit (NB), which is a performance measure that takes into account the consequences of using risk prediction models in making decisions based on risk thresholds.26 NB puts benefits and harms on the same scale and is conveniently expressed in the unit of true positives or true negatives.26,27 It is calculated as follows: 1. Choose a risk threshold: e Patients with predicted risk at or above the threshold are treated as high-risk. e Patients with predicted risk below the threshold are treated as low-risk. 2. Count the number of: e High-risk patients who had an event (true positives) and those who did not (false positives).
Table 1 Demographic and clinical characteristics of the current study’s cohort (i.e. external validation cohort) compared with development samples of the original models. Values are presented as percentages (%) for categorical variables and mean (SD) for continuous variables. COPD, chronic obstructive pulmonary disease; MI, myocardial infarction; MICA, myocardial infarction and cardiac arrest; NR, not reported; NSQIP, National Surgical Quality Improvement Program; RCRI, Revised Cardiac Risk Index; SD, standard deviation. Characteristics No. (%) with data
RCRI derivation cohort (n¼2893)
NSQIP derivation cohort (n¼211 410)
External validation cohort (n¼870)
Demographic Male sex Age, mean (SD), yr
1374 (47.5) 66.4 (10.1)
42.8 No MICA 55.1 (17.2) MICA 69.2 (13.5)
704 (80.9) 65.8 (8.5)
Cardiovascular Hypertension Coronary artery disease Diabetes Diabetes treated with insulin Smoker pack years, mean (SD)
NR 951 (32.9) NR 112 (3.9) NR
45.2 NR 14.6 NR No MICA 11.0 (22.7) MICA 24.4 (34.5) 0.7 2.9 4.5 NR
661 (76.0) 395 (45.4) 196 (22.5) 132 (15.2) 41.9 (28.6)
MI in past 180 days Prior transient ischaemic attack Prior cerebrovascular accident Congestive heart failure Other medical BMI, mean (SD), kg m2
19 (0.7) 175 (6) 155 (5.4) 434 (15.0)
COPD Elevated preoperative creatinine level Serum creatinine level >1.5 mg dl1 Serum creatinine level >2.0 mg dl1 Dialysis Functional status before surgery Independent Partially dependent Totally dependent Surgical Type of surgery Abdominal aortic aneurysm Other vascular Intraperitoneal, intrathoracic, or suprainguinal vascular procedures Emergency status ASA Physical Status class ASA 1 ASA 2 ASA 3 ASA 4 ASA 5
NR
10 21 93 97
(1.1) (2.4) (10.7) (11.1)
NR
No MICA 30.7 (8.8) MICA 28.9 (8.3) 4.7
25.3 (4.1) 341 (39.2)
NR 103 (3.6) NR
9.8 NR 2.4
38 (4.4) 10 (1.1) 4 (0.5)
NR
93.0 4.9 2.1
806 (92.6) 49 (5.6) 15 (1.7)
110 (3.8) 498 (17.2) 894 (30.9)
2.2 8.3 NR
181 (20.8) 689 (79.2) 379 (43.6)
0
12.6
0
140 (4.8) 1558 (53.9) 1078 (37.3) 81 (2.8) 0
10.4 45.2 37.2 6.9 0.3
0 179 (20.6) 628 (72.2) 63 (7.2) 0
424
-
Fronczek et al.
e Low-risk patients who had an event (false negatives) and those who did not (true negatives). 3. Calculate the number of net true positives and negatives per 1000 patients according to the following formulas, where N represents the total sample size:
Decision curves allow for a sensitivity analysis of many potentially meaningful risk thresholds. The curve with the highest NB achieved at reasonably chosen thresholds has the highest clinical utility.
Net true positives per 1000 patients ¼ 1000 * True positves False positives * N N Risk threshold 1 Risk threshold
Model updating
Net true negatives per 1000 patients ¼ 1000 * True negatives False negatives N N 1 Risk threshold Risk threshold
When a risk prediction model is validated in a new setting, its predictive performance may be worse than in the original derivation study. The basic aim of model updating is to improve the model’s predictions in the new sample by adjusting the model to the baseline risk of events in the external validation cohort (i.e. adjustment of the intercept in a calibration plot) or revising the estimated effects of predictors by applying an overall correction factor to regression coefficients (i.e. adjustment of the calibration slope). This updated (recalibrated) model is more likely to be generalisable to a new setting.22 The Canadian Cardiovascular Society (CCS) Guidelines on Perioperative Cardiac Risk Assessment and
Fig 2. Calibration belts. Predicted risks from a well-calibrated model should agree with the observed frequencies of outcome events, resulting in a calibration belt which approximates the diagonal red line of perfect calibration. The test indicates significant miscalibration when 80 (light grey area) and 95% (dark grey area) confidence bands deviate from this line (i.e. P<0.05). (a) RCRI; (b) CCS RCRI; (c) NSQIP MICA; (d) recalibrated NSQIP MICA. RCRI, Revised Cardiac Risk Index; CCS, Canadian Cardiovascular Society; MICA, Myocardial Infarction & Cardiac Arrest; NSQIP, National Surgical Quality Improvement Program.
External validation of the RCRI and NSQIP MICA
-
425
Management for Patients Who Undergo Noncardiac Surgery have recently updated risk estimates of the RCRI based on high-quality studies that measured troponin levels routinely in the perioperative period.28 We used these with no additional changes in our cohort. The NSQIP MICA required a dedicated logistic recalibration procedure (a detailed description of this method was provided in eMethods).29 The recalibrated NSQIP MICA decision curve was corrected for overoptimism using a 200-repetition 10-fold cross validation. The authors of the paper had a full access to all the data in the study and take responsibility for its integrity and the data analysis. All analyses were conducted in R Studio, packages: givitiR (calibration belts), rms (optimism-corrected calibration plot), Calibration Curves (flexible calibration curves), pROC (c-statistics), and rmda (decision curve analysis).25,30e34 Bootstrapped 95% CIs were obtained from 2000 samples.
Results Participant characteristics and the number of observed events The characteristics of the 870 patients included in the study (Fig. 1) are presented in Table 1 and compared with development samples of the original models. The case-mix severity of our sample appeared to be higher than that of both derivation sets in a crude comparison. The primary outcome occurred in 76 patients (8.7%; 95% CI, 6.9e10.8%). We observed 69 MIs, 2 concurrent cardiac arrests (non-fatal events in patients who suffered an MI), and 7 cardiac deaths.
Fig 3. Decision curve analysis. The curve with the highest value of net benefit at a particular risk threshold has the highest clinical utility. The grey curve and the black line assume that all patients are treated as high-risk or low-risk, respectively. RCRI, Revised Cardiac Risk Index; CCS, Canadian Cardiovascular Society; MICA, Myocardial Infarction & Cardiac Arrest; NSQIP, National Surgical Quality Improvement Program. Multiplying the value of net benefit by 1000 gives the number of net true positives per 1000 patients.
External validation The c-statistic for the prediction of the primary outcome was 0.64 (95% CI, 0.57e0.70) for the NSQIP MICA. The RCRI c-statistic was 0.60 (95% CI, 0.54e0.65). The difference in c-statistics between the two models was not statistically significant (P¼0.23). Calibration belts were above the line of perfect fit across the entire range of models’ predictions (P<0.001, Fig. 2). The models showed no clinical utility in a decision curve analysis (Fig. 3). Case mix-corrected benchmark values of cstatistics were 0.70 for the NSQIP MICA and 0.77 for the RCRI. The upper bound of discriminatory capacity achieved by the NSQIP MICA calculator after refitting corresponded to an optimism-corrected c-statistic of 0.64, whereas the RCRI yielded a c-statistic of 0.59.
Update of the models Power of individual predictors was not readjusted, therefore discrimination remained unchanged (Fig. 4). Calibration belts for the recalibrated NSQIP MICA and the CCS’s RCRI agreed with the line of perfect fit, (P¼0.98 and P¼0.51, respectively) (Fig. 2). Clinical utility was higher compared with the original iterations of the models, with a superiority of the recalibrated NSQIP MICA over the CCS’s RCRI (Fig. 3). We provided exact values of NB at different risk thresholds in addition to decision curves (Table 2).
Discussion The two models recommended by international clinical practice guidelines for preoperative cardiac risk assessment in noncardiac surgery poorly predicted which patients would suffer a cardiac death, a nonfatal MI, or a nonfatal cardiac
arrest within 30 days of vascular surgery. Using a systematic high-sensitivity troponin T surveillance to diagnose MIs according to the Third Universal Definition, we showed that the RCRI and the NSQIP MICA had low discriminatory capacities, systematically underestimated the risk for major cardiac complications, and achieved no clinical utility in the decision curve analysis. The latter two issues were to some extent corrected by models’ update. The reduced discrimination of the RCRI in vascular surgery has been previously reported. In the NSQIP MICA derivation study, the calculator’s c-statistic in vascular surgery was 0.75, whereas the RCRI applied to the same dataset yielded an area under curve of 0.59.9 In the original RCRI derivation study, the c-statistic in the vascular surgical subgroup was 0.77 (0.74e0.81), whereas Ford and colleagues35 in their metaanalysis showed a c-statistic for classifying patients undergoing vascular procedures as low risk vs intermediate to high risk of cardiac events equal to 0.65 (0.60e0.69).8 There is substantial evidence on the underestimation of risk by the RCRI, which ultimately led the CCS to issue an update of its risk estimates.28 Our results suggest that the updated risk categories better reflect the actual incidence of major cardiac complications in patients having noncardiac vascular surgery than those provided by the original RCRI. MI is a pivotal outcome for cardiac risk prediction models, yet both of the tested tools operate on currently outdated definitions of this outcome. There are large discrepancies in terms of cardiac biomarkers used to diagnose MI between the RCRI, the NSQIP MICA, and the current clinical practice. Most perioperative MIs are asymptomatic, but the prognosis associated with these events is poor regardless of their clinical
426
-
Fronczek et al.
Fig 4. Calibration curves for the original (left) and recalibrated (right) models. ABC for validation: A, alpha (intercept); B, beta (slope); C, cstatistic (area under the curve). (a) RCRI; (b) CCS RCRI; (c) NSQIP MICA; (d) recalibrated NSQIP MICA. *c-Statistics remained unchanged after recalibration. RCRI, Revised Cardiac Risk Index; CCS, Canadian Cardiovascular Society; MICA, Myocardial Infarction & Cardiac Arrest; NSQIP, National Surgical Quality Improvement Program.
Table 2 Net benefit values of the tested risk prediction models at different risk thresholds. aValues exceeding 1000 net true negatives per 1000 patients are possible in a penalised analysis and are associated with an extremely poor clinical utility of a model. CCS, Canadian Cardiovascular Society; CI, confidence interval; MICA, myocardial infarction and cardiac arrest; NSQIP, National Surgical Quality Improvement Program; RCRI, Revised Cardiac Risk Index. Prediction model
Original RCRI
NSQIP MICA Recalibrated CCS RCRI
NSQIP MICA after logistic recalibration
Risk threshold
Net true positives per 1000 patients (95% CI)
Net true negatives per 1000 patients (95% CI)
0.9% 6% 11% 1%
75 (56e92) 27 (11e42) 0 (e9 to 9) 36 (23e49)
e483 (e986 to 149) e30 (e227 to 182) 208 (55e371) e4131 (e5516 to e2733)a
6% 10.1% 15% 5% 10% 15%
33 (12e51) 2 (e7 to 10) 6 (e16 to 2) 44 (24e63) 10 (e4 to 23) 9 (e1 to 17)
59 (e11 to 147) 151 (e13 to 322) 381 (270e499) 93 (11e197) 215 (78e364) 470 (361e590)
External validation of the RCRI and NSQIP MICA
presentation.17 The NSQIP MICA model took into account only overt MIs that prompted an administration of a troponin measurement. Based on studies that included a systematic cardiac biomarker surveillance after surgery, predictions of this calculator are representative for only about 35% of the actual postoperative cardiac events.1 Furthermore, new troponin elevations lower than three times the upper reference range were considered minor and were excluded from reporting of the outcome in the NSQIP MICA, which dramatically contrasts to the standard 99th percentile diagnostic threshold retained in the Fourth Universal Definition of MI.36 This makes the NSQIP MICA prone to underestimation of risk for postoperative MI, especially in high-incidence scenarios.13,37 The RCRI was developed almost two decades ago and based on serial CK/CK-MB measurements with regard to the biomarker criterion of MI.8 This external validation study used high-sensitivity troponins, which are able to detect twice as many MIs as with non-high-sensitivity assays.18 Our definition of the composite endpoint was slightly different than in the original studies and included nonfatal MIs, nonfatal cardiac arrests, and cardiac deaths within 30 days after surgery. This should not threaten the validity of our findings as we were able to capture essentially all cases of perioperative myocardial ischaemia. A more detailed description of differences between the development and validation studies is provided in Supplementary Tables 2 and 3. Our study has several caveats. The RCRI and the NSQIP MICA were derived in a mixed surgical setting, and we attempted to externally validate them in a relatively small sample of patients having exclusively vascular procedures in one centre, which limits the generalisability of our findings. The number of events was smaller than the recommended 100, yet the original validation set of the RCRI comprised no more than 36 events, which is 50% less than we reported.38 One could argue that poor c-statistics shown in this study are a natural consequence of the sample’s homogeneity. However, the expected discrimination (i.e. case-mix corrected c-statistics simulated under the assumption that the tested models were statistically correct for the external validation cohort) was similar to that observed in subgroups of patients having vascular surgery in the original studies.8,9,24 Interestingly, the upper bound of the models’ discriminatory capacity calculated after re-estimating regression coefficients of predictors included in the RCRI and NSQIP MICA models in our sample was essentially the same as when we used original coefficients (i.e. refitted c-statistics were very similar to standard c-statistics). This suggests that additional predictors (e.g. natriuretic peptide measurements or standardised measures of functional capacity) may be needed to achieve better predictive performance in the future.39,40 The higher baseline risk of major cardiac complications in vascular surgical patients compared with the general surgery setting could have contributed to the understatement of predicted risks. We tried to account for this issue by recalibrating the models.31 The main strength of our study is that every participant had high-sensitivity troponin T levels measured multiple times in the perioperative period. The high quality of the outcome assessment was supported by an independent adjudication committee. Our study confirmed some of the previously raised concerns about the clinical utility of these tools and provided new evidence by independently testing the generalisability of the NSQIP MICA calculator to vascular surgery. The introduction of a systematic screening with highsensitivity cardiac biomarkers greatly improved the
-
427
detection of myocardial ischaemia in the perioperative period. The recent CCS Guidelines on Perioperative Cardiac Risk Assessment and Management for Patients Who Undergo Noncardiac Surgery highlighted this issue and recommended a daily troponin measurement for 48e72 h after surgery in patients who are at an increased risk of cardiac complications (e.g. those with an elevated natriuretic peptide measurement before surgery or an RCRI score 1).28 Our findings support the use of the updated version of the RCRI endorsed by the CCS which operates on higher risk thresholds than the original index and is more likely to produce reliable predictions.41 The NSQIP MICA requires further validation, preferably in multicentre studies incorporating a contemporary definition of MI. Large-scale studies using the VISION Study data are underway and should provide further advancements in this area.42
Authors’ contributions Conception of study and design, acquisition, analysis, and interpretation of data: all authors. Drafting and critical revision: all authors. Final approval of the version to be published: all authors. Accountable for all aspects of the work: all authors.
Acknowledgements The authors thank Pavel Roshanov and the three anonymous reviewers for their helpful and insightful comments on this work.
Declarations of interest ED is a co-applicant on investigator-initiated research grants received from Roche Diagnostics and Abbott Diagnostics. Abbott Diagnosticsdmodest. Roche Diagnosticsdmodest. PJD is a member of a research group with a policy of not accepting honorariums or other payments from industry for their own personal financial gain. They do accept honorariums/payments from industry to support research endeavours and costs to participate in meetings. Based on study questions PJD has originated and grants he has written, he has received grants from Abbott Diagnostics, Boehringer Ingelheim, Philips Healthcare, Roche Diagnostics. PJD has participated in an advisory board meeting for Boehringer Ingelheim. Abbott Diagnosticsdsignificant. Boehringer Ingelheimdsignificant. Philips Healthcaredsignificant. Roche Diagnosticsdsignificant.
Funding Roche Diagnostic provided the Troponin T assays for the VISION Study. The VISION Study funding sources had no role in the design and conduct of the study, in the collection, analysis, and interpretation of the data, or in the preparation, review or approval of the manuscript.
Appendix A. Supplementary data Supplementary data to this article can be found online at https://doi.org/10.1016/j.bja.2019.05.029.
428
-
Fronczek et al.
References 1. Devereaux PJ, Sessler DI. Cardiac complications in patients undergoing major noncardiac surgery. N Engl J Med 2015; 373: 2258e69 2. Patel AY, Eagle KA, Vaishnava P. Cardiac risk of noncardiac surgery. J Am Coll Cardiol 2015; 66: 2140e8 3. Aboyans V, Ricco J-B, Bartelink M-LEL, et al. 2017 ESC guidelines on the diagnosis and treatment of peripheral arterial diseases, in collaboration with the European Society for Vascular Surgery (ESVS). Eur Heart J 2018; 39: 763e816 rka J, Polok K, Fronczek J, et al. Myocardial injury is 4. Go more common than deep venous thrombosis after vascular surgery and is associated with a high one year mortality risk. Eur J Vasc Endovasc Surg 2018; 56: 264e70 5. Devereaux PJ, Goldman L, Cook DJ, Gilbert K, Leslie K, Guyatt GH. Perioperative cardiac events in patients undergoing noncardiac surgery: a review of the magnitude of the problem, the pathophysiology of the events and methods to estimate and communicate risk. Can Med Assoc J 2005; 173: 627e34 6. Fleisher LA, Fleischmann KE, Auerbach AD, et al. 2014 ACC/AHA guideline on perioperative cardiovascular evaluation and management of patients undergoing noncardiac surgery: a report of the American College of Cardiology/American Heart Association task force on practice guidelines. Circulation 2014; 130: 2215e45 7. Kristensen SD, Knuuti J, Saraste A, et al. 2014 ESC/ESA Guidelines on non-cardiac surgery: cardiovascular assessment and management: the Joint Task Force on noncardiac surgery: cardiovascular assessment and management of the European Society of Cardiology (ESC) and the European Society of Anaesth. Eur Heart J 2014; 35: 2383e431 8. Lee TH, Marcantonio ER, Mangione CM, et al. Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation 1999; 100: 1043e9 9. Gupta PK, Gupta H, Sundaram A, et al. Development and validation of a risk calculator for prediction of cardiac risk after surgery. Circulation 2011; 124: 381e7 10. Davis C, Tait G, Carroll J, Wijeysundera DN, Beattie WS. The Revised Cardiac Risk Index in the new millennium: a single-centre prospective cohort re-evaluation of the original variables in 9,519 consecutive elective surgical patients. Can J Anaesth 2012; 59: 1013e22 11. Bertges DJ, Goodney PP, Zhao Y, et al. The vascular study group of new England cardiac risk index (VSG-CRI) predicts cardiac complications more accurately than the revised cardiac risk index in vascular surgery patients. J Vasc Surg 2010; 52: 674e83 12. Peterson B, Ghahramani M, Harris S, et al. Usefulness of the myocardial infarction and cardiac arrest calculator as a discriminator of adverse cardiac events after elective hip and knee surgery. Am J Cardiol 2016; 117: 1992e5 13. Juo YY, Mantha A, Ebrahimi R, Ziaeian B, Benharash P. Incidence of myocardial infarction after high-risk vascular operations in adults. JAMA Surg 2017; 152, e173360 14. Duncan D, Wijeysundera DN. Preoperative cardiac evaluation and management of the patient undergoing major vascular surgery. Int Anesthesiol Clin 2016; 54: 1e32 15. Devereaux PJ, Biccard BM, Sigamani A, et al. Association of postoperative high-sensitivity troponin levels with
16. 17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27. 28.
29.
30. 31.
32.
33.
myocardial injury and 30-day mortality among patients undergoing noncardiac surgery. JAMA 2017; 317: 1642e51 Biccard B. Proposed research plan for the derivation of a new cardiac risk index. Anesth Analg 2015; 120: 543e53 Devereaux PJ, Xavier D, Pogue J, et al. Characteristics and short-term prognosis of perioperative myocardial infarction in patients undergoing noncardiac surgery. Ann Intern Med 2011; 154: 523e8 Brown JC, Samaha E, Rao S, et al. High-sensitivity cardiac troponin T improves the diagnosis of perioperative MI. Anesth Analg 2017; 125: 1455e62 Twerenbold R, Jaeger C, Rubini Gimenez M, et al. Impact of high-sensitivity cardiac troponin on use of coronary angiography, cardiac stress testing, and time to discharge in suspected acute myocardial infarction. Eur Heart J 2016; 37: 3324e32 Vascular Events In Noncardiac Surgery Patients Cohort Evaluation (VISION) Study Investigators, Devereaux PJ, Chan MTV, Alonso-Coello P, et al. Association between postoperative troponin Levels and 30-day mortality among patients undergoing noncardiac surgery. JAMA 2012; 307: 2295e304 Thygesen K, Alpert JS, Jaffe AS, et al. Third universal definition of myocardial infarction. Eur Heart J 2012; 33: 2551e67 Collins GS, Reitsma JB, Altman DG, Moons KGM. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015; 350: g7594 Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J 2014; 35: 1925e31 Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010; 172: 971e80 Nattino G, Finazzi S, Bertolini G. A new calibration test and a reappraisal of the calibration belt for the assessment of prediction models based on dichotomous outcomes. Stat Med 2014; 33: 2390e407 Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ 2016; 352: i6 Qian X, Lee W-C, Rousson V, et al. Decision curve analysis: a technical note. Ann Transl Med 2018; 6: 308 Duceppe E, Parlow J, MacDonald P, et al. Canadian Cardiovascular Society guidelines on perioperative cardiac risk assessment and management for patients who undergo noncardiac surgery. Can J Cardiol 2017; 33: 17e32 Van Calster B, Van Hoorde K, Vergouwe Y, et al. Validation and updating of risk models based on multinomial logistic regression. Diagn Progn Res 2017; 1: 2 Harrell Jr FE. rms: regression modeling strategies. R package version 5.1-3 2019 Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 2016; 74: 167e76 Robin X, Turck N, Hainard A, et al. pROC: an open-source package for R and Sþ to analyze and compare ROC curves. BMC Bioinform 2011; 12: 77 Brown M. rmda: risk model decision analysis 2017
External validation of the RCRI and NSQIP MICA
34. RStudio Team. RStudio: integrated development environment for R. Boston, MA 2016. Available from, http://www.rstudio. com/. [Accessed 30 March 2019]. accessed 35. Ford MK, Beattie WS, Wijeysundera DN. Systematic review: prediction of perioperative cardiac complications and mortality by the revised cardiac risk index. Ann Intern Med 2010; 152: 26e35 36. Thygesen K, Alpert JS, Jaffe AS, et al. Fourth universal definition of myocardial infarction (2018). Eur Heart J 2019; 40: 237e69 37. Polok K, Fronczek J, Szczeklik W. Myocardial infarction after vascular surgery: a systematic troponin surveillance and a uniform definition is needed. JAMA Surg 2018; 153: 496 38. Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med 2016; 35: 214e26 39. Rodseth RN, Biccard BM, Le Manach Y, et al. The prognostic value of pre-operative and post-operative B-type
-
429
natriuretic peptides in patients undergoing noncardiac surgery: B-type natriuretic peptide and N-terminal fragment of pro-B-type natriuretic peptide: a systematic review and individual patient data meta-analysis. J Am Coll Cardiol 2014; 63: 170e80 40. Wijeysundera DN, Pearse RM, Shulman MA, et al. Assessment of functional capacity before major noncardiac surgery: an international, prospective cohort study. Lancet 2018; 391: 2631e40 41. MDCalc. Revised Cardiac risk index for pre-operative risk. Available from: https://www.mdcalc.com/revisedcardiac-risk-index-pre-operative-risk. [Accessed 30 March 2019]. accessed 42. Roshanov PS, Walsh M, Devereaux PJ, et al. External validation of the Revised Cardiac Risk Index and update of its renal variable to predict 30-day risk of major cardiac complications after non-cardiac surgery: rationale and plan for analyses of the VISION study. BMJ Open 2017; 7, e013510
Handling editor: H.C. Hemmings Jr